Deep Generative Clustering with Multimodal Diffusion Variational Autoencoders
Authors: Emanuele Palumbo
Laura Manduchi
Sonia Laguna
Daphné Chopard
Julia E Vogt
Venue: ICLR 2024 poster (2024)
Keywords: Generative Clustering
Multimodal VAEs
Variational Autoencoder
Multimodal Learning
Generative Models
Primary area: generative models

Abstract: Multimodal VAEs have recently gained significant attention as generative models for weakly-supervised learning with multiple heterogeneous modalities. In parallel, VAE-based methods have been explored as probabilistic approaches for clustering tasks. At the intersection of these two research directions, we propose a novel multimodal VAE model in which the latent space is extended to learn data clusters, leveraging shared information across modalities. Our experiments show that our proposed model improves generative performance over existing multimodal VAEs, particularly for unconditional generation. Furthermore, we propose a post-hoc procedure to automatically select the number of true clusters thus mitigating critical limitations of previous clustering frameworks. Notably, our method favorably compares to alternative clustering approaches, in weakly-supervised settings. Finally, we integrate recent advancements in diffusion models into the proposed method to improve generative quality for real-world images.

Published as a conference paper at ICLR 2024
DEEP GENERATIVE CLUSTERING WITH MULTIMODAL DIFFUSION VARIATIONAL AUTOENCODERS
Emanuele Palumbo':2 Laura Manduchi", Sonia Laguna", Daphné Chopard & Julia E. Vogt"
1 ETH AI Center 2 Department of Computer Science, ETH Zurich
3 Department of Intensive Care and Neonatology and Children's Research Center, University Children's Hospital Zurich, University of Zurich emanuele, umbo ai. ethz.ch
ABSTRACT
Multimodal VAES have recently gained significant attention generative mod- els for weakly-superrised learning with multiple heterogeneous modalities. In parallel, VAE-based methods have been explored probabilistic approaches for clustering tasks. At the intersection of these two research directions, we propose novel multimodal VAE model which the latent space is extended learn data clusters, leveraging shared information across modalities. Our experiments show that our proposed model improves generative performance over existing multi- modal VAEs, particularly for unconditional generation. Furthermore, we propose a post-hoe procedure to automatically select the number of true clusters thus miti- gating critical limitations of previous clustering frameworks. Notably, our method favorably compares to alternative clustering approaches, in -upervisee set- tings. Finally, integrate recent ad adancementss in diffusion models into the pro- posed method to improve generative quality for real-world images.
INTRODUCTION
Multimodal VAES are powerful generative models for weakly-supervised learning with multiple modalities. Different from initially proposed models (Suzuki al.] [0117] Vedantam et al. [2088), more recent scalable approaches (Wu & Goodman) Shi et al. 2019 Sutter et 2002 Hwang et al., 202T Palumbo et al., 2023) can handle large number modalities efficiently, thereby en- abling compelling applications in segmentation tasks or data integration in the healthcare domain (Lee & van der Schaar 2021 Dorent et al.) 2019). Multimodal VAEs have been mostly investi- gated in the realm of generative tasks, demonstrating significant success in cross-modal generation. despite less remarkable results for unconditional generation (Hwang et al., 2021 Palumbo et al.) 2023). However, VAE-based approaches generally fall behind in image synthesis applications, often yielding blurry generated images. Among several extensions proposed to address these limitations (Vahdat & Kautz] 2020 Bredell et al.] 2023), the most promising approaches integrate recent ad- vancements in Denoising Diffusion Probabilistic Models (DDPMs) (Sohl- Dickstein et al.) 20155] Ho ct al. 2020) into the VAE framework (Pandey al., 2022, Preechakul al. 2022). They retain meaningful and decodable representation of the input while leveraging the generative capability of diffusion models. Even though these methods have demonstrated greatly enhance the quality of generated samples, their adaptation to multimodal VAEs remains unexplored.
parallel line of research has explored VAE-based generative approaches for clustering tasks (Tang et al., 2017 Dilokthanakul et al., 2016). In particular, deep variational clustering methods have been employed to identify sub-groups of patients survival analysis (Manduchi et al., 2022). or to integrate domain knowledge from clinicians through prior probabilities (Manduchi et al. I02 Other works use VAE-based methods to learn interpretable representations for clustering time- sere-series (Fortuin et al. 2019), and hierarchical structures of latent semantic concepts of the data (Li et al. 2018).
With this work, we position ourselves at the intersection of these two lines of research by proposing a deep variational generative approach for clustering in a multimodal setting, where shared infor- mation across modalities is integrated model data clusters. In particular, introduce novel

---

Published as a conference paper at ICLR 2024
multimodal VAE model, called Clustering Multimodal VAE (CMVAE). An overview of the method is illustrated in Figure The proposed approach divides the latent space into shared and modality- specific embeddings (Palumbo et al. 2023). and imposes as prior a mixture distribution enforce a clustering structure in the shared latent representation of the data (see Figure| I aa). In our experi- ments, show that method represents an improvement over existing multimodal VAEs, partic- ularly for unconditional generation, where existing methods struggle to achieve satisfactory results. Moreover, we showcase the effectiveness of CMVAE for clustering weakly-superviser data, where unimodal approaches fail to achieve adequate performance, in comparison to alternative weakly- supervised methods. Notably, we introduce a post-hoe procedure to effectively infer the number of clusters at test time, without the need for training multiple instances of our model with differ- ent hyperparameters. The proposed algorithm selects the optimal configuration of latent clusters, to minimize the entropy of the posterior distribution of cluster assignments (see Figure TT (b)). Fi- nally, inspired by the work of Pandey et al. (2022), we propose to integrate DDPMs (Sohl-Dickstein et al. 2015: Ho et al. (2020) into the CMVAE framework, to further improve the generative qual- ity of reconstructed and generated images while retaining the clustered latent space of CMVAE. In particular, we train the diffusion process conditioned on the CMVAE reconstructions, using both the self and the cross-modal reconstructions, thus enhancing the model's capacity of generalization (see Figure[[()) Overall our approach significantly improves over alternative methods in realistic settings and is able to generate cluster-specine sharp images.
Our main contributions in this work can be summarized as follows:
we propose CMVAE, a novel multimodal VAE model designed to model data clusters in the latent space and we show that approach (a) outperforms existing multimodal VAEs, with a remarkable improvement in unconditional generation, and (b) outperforms alterna- live scalable weakly-supervised methods for clustering tasks in multimodal datasets;
we propose post-hoc procedure for selecting the optimal latent clusters at test time, in- spired by previous work, thereby avoiding the need to specify the correct number of clusters a-priori during training:
we propose the integration of DDPMs into our framework, yielding D-CMVAE, to improve the generative performances in multi-modal design of multimodal VAES for realistic applications.
settings, taking crucial step in the
2 RELATED WORK
Multimodal VAEs Multimodal VAEs extend the well-known VAE framework (Kingma Welling 2014) to handle data consisting of multiple modalities, leveraging the pairing across modalities as weak supervision. While early approaches (Suzuki et al., 2017 Vedantam et al.) 2018) faced scala- bility challenges due to inference requiring separate encoder network for each subset of modalities, more recent scalable approaches (Wu & Goodman 2018 Shi et al.
2019 Suttter et al.) 2020 202T Hwang et al. 2021) assume the joint encoder decomposes in terms of unimodal encoders. Despite promising applications (Lee & van der Schaar 2021 Dorent et al. 2019), recent work (Daunhawer et al. 2022) has uncovered important limitations for three main formulation of multimodal VAHS. In particular, these approaches exhibit a trade-off between generative quality (the similarity of gener- ated samples to real ones) and generative coherence (the semantic consistency in generated samples across modalities). Since then, attempts to enhance the performance of multimodal VAES involved additional regularization terms (Sutter al. 2020 Hwang et al., (002), or mutual supervision (Joy et al. (2023) proposed to model shared and private subspaces (Sutter al. 2020 Lee & Pavlovic 202i:lici22221]00 Wang et al.) 2016) and design an ELBO that exploits auxiliary distributions to facilitate the estimation of cross-modal likelihood terms. The resulting MMVAE+ model proves to achieve both high generative quality and high generative coherence.
et al. 2022).
Variational approaches for clustering Following the seminal work of Kingma & Welling (0014), variational autoencoders (VAEs) have been investigated for clustering with approaches such as GM- VAE (Dilokthanakul et al. 2016) and VaDE (Jiang et al.| 2017). More recently, there has been revived interest in these models. In particular, Manduchi et al. (2021) have proposed an ELBO for clustering data and incorporate prior knowledge in the form of prior probabilities. In another work, 2022) use a VAE approach to cluster patients in subgroups for survival analysis. Finally, Xu et al.
(Manduchi et al.
2021 propose variational approach cluster multi-view data, that however has
2

---

Published as a conference paper at ICLR 2024
X1
Gou, (M1)
((M)))
83-41
Rae-11
DDPM;
By Ciss Psp Rg- Bg
9
9
picte)
pOg Kx. R3)
Mios
XM
pouels, (b))
Po Wee
Axy
p(cte)
DDPM (c)
p(we)
4a (M9g (Rte)
Figure 1: (a) CMVAE divides the latent space into shared and sondality-speciffe embeddings. enforces mixture distribution as prior the shared latent subspace, create clustering structure in the shared representation. (b) A post-hoe procedure infers the number of clusters at test time efficiently by minimizing the entropy of p(c)z), penalized by the normalized joint log-likelihood. (c) D-CMVAE integrates DDPMs into the framework by conditioning the reverse process on the self and cross-modal CMVAE reconstructions. This enhances the generative performance while maintaining the clustered latent space.
the limitation of not handling missing modalities for inference, which impacts its scalability large number of modalities.
Scalable weakly-supervised clustering Weakly-supervised clustering refers clustering algo- rithms that integrate high-level and often noisy sources of supervision to improve the clustering performance. While several forms of weak supervision have been previously investigated in the lit- erature, such as parse-grained labels (Ni et al. 2022), bag level labels (Oner et al.) (219)), and pair- wise similarities (Manduchi et al., 2021). in this work we restrict ourselves to methods that exploit weak-supervision in the form of multiple modalities. In the multimodal setting, weakly-supervised clustering methods are often not designed to scale to a large number of modalities (Alwassel et al. 2020 Chen et al.) 2021 Zhou & Shen 2020). For instance, DeepCluster (Caron al. (018)) is unimodal clustering approach that iteratively learns cluster assignments and neural network param- eters. An adaptation of this method for modalities, XDC, was introduced by Alwassel et al. (2020). XDC relies on pseudo-labels from one modality to learn better feature representation for an- other modality, improving unimodal clustering for both, but not intended generalise number of modalities. Learning representations from datasets of large number of views has been investigated, for instance, the realm of contrastive approaches. As an example, (Tian et al. (2020) propose the CMC loss which maximizes mutual information between different views with a con- trastive objective, and can be extended to large number of views. While not specifically developed for clustering tasks, training K-means model the learned representations can be used as proxy the adaptability of the learned representations clustering.
larger
METHOD
3.1 A SCALABLE VAE OBJECTIVE FOR MODELING LATENT CLUSTERS IN MULTIMODAL DATA
We assume data consisting of M modalities X Ti...., RM is generated according to the following process. For each datapoint æi....,'', x where i E [1.....N} and N is the dataset size, a cluster assignment cf is drawn from a categorical distribution Px(c) with probabilities TI...., TK where K is the number of clusters. Then the M modalities are drawn ac- cording to wi....,.0 I'y Po, (xi ...,...,,, ,e (Z where the shared encoding z¹ ~ p(zlct) is generated conditioning on cluster assignment, while modality-specific encodings p(wM) are drawn from prior distributions. The resulting generative anf p(w)). (..)..... w model (e W, 2. P= (c)p(z(c) IIM Po... (wm, x)p(wm)) where priors and likeli- hoods are assumed to belong to specific family of distributions, e.g. Gaussian or Laplace, and likelihoods are parameterized by neural network decoders. Note that the shared encoding z and modality-specific encodings w,,... wM W are assumed to be independent.
3

---

Published as a conference paper at ICLR 2024
To obtain tractable objective, variational encoders qt. (x 9ow, (wi(x1).... 90wss (wM (xM). [<c|2, X) are introduced approximate posterior inference for each of the latent variables. In line with our generative assumptions, the shared and modality-specific encoders are assumed to be conditionally independent given the observed data. As in previous approaches (Shi al., 2019) Palumbo et al. 2023), to achieve scalability in the number of modalities, we model the joint encoder
for z as mixture of experts qo, (=|X)= M Σm=1 2 (z|am) ). In our objective, we incorporate two key ideas from previous related work. First, to accurately model both shared and modality- specific information in separate latent subspaces without conflicts, in (Palumbo al., (023), we use auxiliary distributions Ti (wi),... ... T' (Wm) for private features to estimate cross-moda reconstruction likelihoods (second summand in the right-hand side of Equation (2)). This leads to our proposed ELBO objective
M Σ
LCMVAE(X) = M
glclz.X) m== 9dam mmm)
Ge e c, 2, 2, wm)]
(I)
Sosem (10 mmmm)
where
Gr. ( (x,, 2, wm) = Pog (I 12 Wm) +
Σ retiir
Ee ~rn (w.) Pogpo, (x., (.r.,,,,......... ir,)]
Ps (2) qa (= [= Mo
+ 3 log
(Wm (m
and 3 hyperparameter weights latent space regularization. Furthermore, instead of learning additional encoder to approximate inference in c, we adopt the formulation for the approximate posterior of cluster assignments given z proposed in the work of (Jiang al. (017). which has the advantage of not requiring additional parameters
p(c)p(z/c)
q(c) z, x) = p(c|z)=
Ee=1
Different from the expectations with respect to the qu, (x|x), 9. (1),..),..... Adwx (wM (x encoders, that need be approximated via sampling with reparameterization, Eg(elz. x) can be com- puted exactly since assumes a discrete finite set of values. Finally, our proposed CM CMVAE objective is a valid evidence lower bound (ELBO), and Appendix A contains a formal proof.
3.2 ENTROPY OF POSTERIOR CLUSTER ASSIGNMENT DISTRIBUTION FOR POST-IICC LEARNING OF THE NUMBER OF CLUSTERS
A critical limitation of many existing methods for deep clustering is the need to specify the number of clusters a-priori. This can result in highly complex model selection procedures, in failure cases when proxy for this information cannot be obtained, or in limiting modelling capacity. As with other methods, CMVAE requires specifying a K value for the number of latent clusters assumed in the generative process for training. However, the number of clusters in the data K is in general unknown, and in practice K may differ from K. While few approaches use non-parametric prior distributions overcome this challenge (Goyal et al. 2017: Hu al., 2015 Griffiths et al.) (200)), we instead propose simpler post-hoc procedure to recover the optimal clusters in the data given an instance of model trained with an over-specified number of clusters, i.e. K K.
The proposed procedure is described in the pseudocode in Algorithm T and aims at obtaining a posterior cluster distribution p(c]z) where exactly K clusters have positive probability and each latent cluster correctly models a different true cluster of the data. In other words, it aims at recovering the true K clusters in the data without adding extra complexity the number of model parameters. In a nutshell, the procedure iterates over the data, ranking latent clusters by their importance. At each iteration, unimportant clusters are pruned by setting their prior probability to zero, and the entropy of p(c|z) is kept specifically, the clusters are ranked by the probability mass each one assigned to, and the average normalized entropy of p(c|z) ) is computed. Note that cluster assignments are determined by majority voting between modalities. Then, the latent cluster with the lowest probability mass is set null prior probability at the next iteration, with the remaining probabilities recomputed accordingly to
metric to select the optimal set of clusters to model the data. More
4

---

Published as a conference paper at ICLR 2024
maintain valid probability distribution. Iteratively, the latent clusters effectively pruned, while the average normalized entropy of p(c x) is calculated at every step. We select the optimal cluster configuration corresponding to the lowest value for the normalized entropy penalized by the normalized joint log-likelihood to avoid overly restricting modeling capacity (Biernacki et al. [2000) Baudry al., 2010). By minimizing the entropy overlapping clusters effectively pruned and the data are partition into meaningful subsets. In fact, as confirmed in our results in Figure 5a and Appendix when unimportant latent clusters are pruned, the normalized entropy term p z) decreases, as uncertainty over cluster assignments is reduced, up to a minimum, which is attained at the true number of clusters K. As clusters are further pruned, not all data clusters are represented in the latent space, which results in the entropy term increasing again. Note this procedure is performed on separate validation dataset our experiments (see Appendix D.2), and fully unsupervised. Finally, note this procedure obtains high clustering performance without limiting modelling capacity.
3.3 INTEGRATING DIFFUSION PROBABILISTIC MODELS INTO MULTIMODAL VAES
We hereby propose to incorporate Denoising Diffusion Probabilistic Models (DDPMs) into the CM- VAE framework. For this section will denote the input data X Xo == gl. 2: Mo Then, for each modality m we define a sequence of T noisy representations of the input am yelding 2mi. As in standard DDPMs (Sohl-Dickstein et al., 2015; Ho al., (2020) we define forward process (mmer mmo) that gradually destroys the structure of each data modality am.:
T
(Xmir [mmo) = II
(20m, 1mmm
(3)
t=1
q (xm. xm.-1)= =N (VI Bin Pm Bmat)
(4)
(mm. xmoo) = N (2 (V am, amm Tmo: (1 - am.) 1) where amm, = (1 - Bme), a.. = II amer
(5)
where Bm, for = 1, ...... T and I,..., M are the noise schedules. We then follow the work of Pandey et al. (2022) to define reverse process that is conditioned on the VAE reconstructions mng
P& (Xmorr #mo)) = (Xmm)) II Ps (xm.-1 (mm.)
(6)
Pe (Em--1 Dm. Dmm)) = N (pe (Irm, t, Lmo)) Egg. (I t, xmn))
(7)
In practice, the VAE reconstruction Imo are concatenated to the reverse process representation Im. at each time step to obtain Im.-1. While Diffuse VAE (Pandey et al.) (2022) is designed for a VAE with one data modality, we extend it to the multimodal setting and sample from both the self and cross-modal reconstructions of CMVAE with equal probability:
pe... (20 |z, wm),
zo-gosn (z) (m)). Wmn Coreem (w0) m.))
b
selffreconstruction
Pmo
zn-oon (* #n).
pa.. (xml (z, mm))
mo (10m)
if cross-reconssrrctionn
CCC )
where b sampled from a Bernoulli distribution with p 0.5. Using the cross-modal reconstruc- tions, which are usually noisier than the self-reconstructions, improves and stabilizes the training, making the diffusion process more robust at test time.
The new objective function is then defined as
M Σ
LD.CMAE((X)) (Xo) = LCMWAE Xo)
+
aoam (x am))) Tomm (mm) Pe (Emor
LDDPM... (Xo, 2, Wm)
(9)
M m=
#moo)
LDDPM... X 0 2, W.)) E (m mm))
(10)
q (Imir
Fmaa)
As Diffuse VAE (Pandey et al., 2022). we first train CMVAE using Equation I and then we freeze the CMVAE's weights and train the M diffusion processes.
5

---

Published as a conference paper at ICLR 2024
Algorithm Post-hoc selection of optimal latent clusters.
Hf denotes normalized entropy. |-] indicates the number of dimensions of given latent variable, assign, defines the cluster assignment step, and compute, the prior probability update operation.
Input: pwx (c).p(z(c), Aom, (z/a1) 40-3s (=|ær)) from trained CMVAE with K > K, data X 1:N
Output: P= with #g s.t. Σ AxeNU for K to do for æi...... V's in æ1 J:N ..... I:N do
zi...., Zir Zdz (z/ar)), A0mxe (2lorh)) for m to M do p(c|=m) = - Pmg (e' (2)mlc))) end for cu. = (psignn(p(c(21).... (.cc:))) p(c(22))
Pin (c)p(zmlle
hi >m=1 H(((c(zm)) log ((=.e)
end for h2 N Σm= h Tk compute, (claN (cls
TTk
end for
PiR where K- argming (/1,... le....., hx)
4 EXPERIMENTS
We test the performance of CMVAE, in comparison with alternative approaches, both semi- synthetic and real-world datasets. We first validate our contributions on the PolyMNIST dataset (Sutter et al.) 2021). semi-synthetic five-modality dataset depicting MNIST (LeCun et al., 2010) digits with modality-specife backgrounds, well-established as a benchmark for multimodal VAES
(Sutter al. 2021 Hwang et al.) 202T Palumbo et al. 2023). In Section 4.1
compare the
generative capabilities of our approach with alternative multimodal VAEs: CMVAE outperforms
existing approaches, particularly in unconditional generation. In Section 4.2 test the clustering
capabilities of CMVAE in comparison with alternative unimodal and scalable weakly-supervised approaches. In particular, we validate our proposed post-hoc procedure to determine the optimal set of latent clusters at inference time, hence avoiding the need for a- priori knowledge of the true number of clusters. Finally, in Section 4.3 we validate our contributions in a real-world experiment. We introduce a variation of the CUB Image-Captions dataset (Wah et al., 2011 Shi et al. 2019), which we name the CUB Image-Captions for Clustering (CUBICC) dataset. As the original CUB Image- Captions dataset consists of images of birds paired with matching captions, group sub-species of birds in the original dataset in eight single
Blackbird, Gull, Jay, Oriole, Tanager, Tern, Warbler, Wren obtaining challenging realistic multimodal clustering dataset illustrated in Figure 2b Details for the datasets are in Appendix D.I
Orlole
Hooded
(a) PolyMNIST dataset: Each column is a single data point in this dataset. consisting of five image modalities with matching digit. Each row show- cases samples from given modality.
Thknk Misciin with 1onlov200
Digh cirige unl
DM indvii
(b) CUBICC dataset: Example of the variability within single bird class, Oriole. Here show the images (modality 1) with their corresponding captions (modality 2).
Figure 2: Illustrative samples for the PolyMNIST (a) and CUBICC (b) datasets.
4.1 IMPROVED GENERATION PERFORMANCE OVER EXISTING MULTIMODAL VAES
In this section, we compare the generative capabilities of our model against the main existing formu- lations of multimodal VAEs the PolyMNIST dataset. Successful generative performance of mul-

---

Published as a conference paper at ICLR 2024
timodal VAES requires satisfying two criteria: achieving high semantic coherence across generated modalities (i.e., high generative coherence). as well as having high similarity between generated samples and real samples (i.e. high generative quality). Consistently with previous work (Sutter et al., 2021] Hwang et al., 2021 Palumbo et al., 2023). we compare models in terms of generative quality and generative coherence for both generation with latents sampled from prior distributions (i.e. unconditional generation) and cross-modal generation (i.c., conditional generation), and report our results in Figure 3 In line with prior research (Shi al. 2019 Daunhawer et al., 2022 Hwang et al. 2021 Palumbo et al., 2023). we adopt the FID score (Heusel et al. 2017) as a proxy for gen- erative quality, and use pre-trained digit classifiers assess generative coherence. For more details on these metrics, see Appendix D.3] Results show that CMVAE outperforms existing multimodal VAEs, across different hyperparameter values controlling latent space regularization, for both con- ditional and unconditional generation. While the improvement for conditional generation is mod- crate, CMVAE outperforms the other methods by a significant margin in unconditional generation, particularly for unconditional coherence. This is an important advancement over state-of- wif-the-at multimodal VAEs, where unconditional generation performance represents a critical weakness.
1.0
1.0
Uncomniionnl Coéerrnce
0.5
Canditonal Ecllerence
06
0.6
0.4
02-
0.2
60p
0.0
0.0
Unconditocaa FID
Conditionl FID
Figure 3: Unconditional (left) and conditional (right) generation performance of CMVAE compared with existing multimodal VAEs on PolyMNIST. Three independent runs for each model, with differ- ent symbols denoting different values of the 3 hyperparameter. In both plots, generative coherence measured on the y-axis (higher is better), while on the x-axis generative quality is assessed via the FID-score (lower is better). Therefore optimal performance is at the top-right corner of each plot.
0 0 G
2
7
9
F/
3
Figure 4: Generative performance for CMVAE and D-CMVAE on the PolyMNIST dataset. Each row showcases generated samples for each modality, with at the top CMVAE generations and cor- responding D- CMVAE generations below. We report three generations for all learned clusters, each one corresponding to a different column, across all modalities.
4.2 WEAKLY-SUPERRVISED CLUSTERING VIA POST-HOC SELECTION OF LATENT CLUSTERS AND SHARP GENERATIONS WITH DDPMS
In this section, we look at clustering results for our method on the PolyMNIST setting and validate the procedure described in Section 3.2 learn the optimal number of clusters in the data with CMVAE. Specifically, we train multiple instances of CMVAE with K = 40 clusters, markedly different from the true number clusters 10, corresponding to the ten digits. Once an instance of the model is trained, apply our proposed post-hoc procedure to select the optimal clusters in the latent space, whose effectiveness showeased in Figure S In particular, Figure 5a shows the trend of our proposed entropy- based criterion as the latent clusters pruned, reaching minimum when the true latent number of clusters is achieved, consistently across runs. In addition, Figures Scl and 5d show how clustering metrics evolve as clusters are pruned: the showcased trends show

---

Published as a conference paper at ICLR 2024
that the procedure gradually converges to the solution in which the true clusters modeled in the latent space, and clustering metrics, therefore, have the best results. These results also confirm that the procedure not only finds the correct number of clusters, but more to the point minimal set of latent clusters, each one modeling a true cluster in the data. Results in Figure 4 confirm our quantitative analysis from qualitative standpoint, and in addition show the effectiveness of incorporating DDPMs into our model to increase the quality of generations. Finally in Appendix by varying the number of modalities for inference, we show that CMVAE effectively exploits the presence of multiple modalities in computing cluster assignments at test time.
In Table we compare the performance of CMVAE in multimodal clustering with alternative meth- ods. A first natural baseline to report is VaDE (Jiang et al. (017). a well-established unimodal VAE- based clustering model. However, in order not to restrict ourselves to only variational approaches for unimodal clustering, we include in the comparison the well-known DeepCluster (Caron et al.) 2018]. In this setting, unimodal clustering approaches fail to achieve good performance (see also Appendix E.2. I we find they rather model background features, which are prominent pixel-wise in the images, instead the relevant digit content. As a baseline from the realm of contrastive learn- ing, we adopt CMC (Tian al., 2020), mainly due its scalability numerous modalities, with an instance of K -eans trained on the learned latent representations. Not surprisingly, this approach achieves comparable yet not superior performance to CMVAE in this setting, as it closely resembles a multi-view setting for which this approach is conceived. However, CMC achieves a comparable performance to our approach on the condition of having a-priori information on the true number of clusters in the data, which is not realistic in real-world scenarios. As shown in Appendix E.2. when a-priori information not available, our method has markedly better performance. Moreover, for more heterogeneous modalities, the gap in performance between the two methods is substantial, as the next section will show. More details about the baselines are reported in Appendix E.2
0:0
0.5-
0.0
0.4
30
25
20
15
10
35
15
10
15
15
10
15
15
10
15
20 15
(a) Entropy
(b) NMI
(c) ARI
(d) Accuracy
Figure 5: Post-hoc optimal latent cluster selection with the procedure in Algorithm | (a) Evolution of the entropy term (y-axis) as clusters (x-axis) are pruned. The minimal entropy recovers the true number of clusters. The trends for test NMI (b), ARI (c) and accuracy (d) support the validity of our procedure, showing the procedure selects a minimal set of latent clusters to correctly model the true data clusters. Additional evidence is shown qualitatively in Appendix B
PolyMNIST
CUBICC
K not given for training
NMI
ART
ACC
NMI
ART
ACC
VaDE
0.43 (0.04) 0.36 (0.04)
0.54 (0.05)
0.15 (0.01)
0.08 (0.01)
0.27 (0.01)
DeepCluster
0.12 (0.02)
0.08 (0.02)
0.26 (0.04)
0.19 (0.01)
0.10 (0.01)
0.29 (0.01)
CMC
0.97 (0.01)
0.97 (0.01)
0.99 (0.01)
0.37 (0.05)
0.10 (0.03)
0.31 (0.04)
X
CMVAE
0.97 (0.02)
0.97 (0.02)
0.99 (0.01)
0.67 (0.07)
0.59 (0.09) 0.76 (0.07)
Table 2: Quantitative comparison of clustering performance on PolyMNIST and CUBICC datasets.
4.3 CUBICC
In this section we test CMVAE on the realistic CUBICC dataset. Note that recent work shows the nature of such dataset is challenging for most existing multimodal VAEs (Palumbo et al.) (023)_ Additional challenges arise in modelling data clusters, corresponding to the eight bird species, due to similar appearances (e.g. Gull and Tern), and high intra-cluster variability result of the group-
8

---

Published as a conference paper at ICLR 2024
ing of sub-species (see Figure 2b). In spite of these challenges, results in Figure 6 and Table show that in this setting CMVAE accurately models the true latent clusters in the data, without prior knowledge of their number and in completely unsupervised fashion. In particular, results in Ap- pendix B validate our proposed procedure to find the optimal number of clusters in a real-world setting. A quantitative comparison with alternative approaches in Table 2 validates the effective- ness of CMVAE in this challenging setting, where it outperforms both unimodal and alternative weakly-supervised approaches. In Figure O we show the CMVAE and D-CMVAE generations for each learned cluster. As it can be seen, integrating DDPMs into the generative process is highly beneficial to generate sharp images. However, given the scarcity of the training samples and the di- versity of bird species, conditioning the diffusion process only on the self-reconstruutiions produces sub-optimal generative results (see Figure TS] in the Appendix). The integration of cross-modal re- constructions proves to be crucial for obtaining high-quality generations. Finally, in Figure 7 we showcase the performance of D-CMVAE in conditional generation on this dataset, where it gener- ates sharp images, that closely align with text prompts given as input. Notably, D- D-CMVAE achieves an average FID score of = 28 on this dataset, a tremendous improvement over generative quality of existing multimodal VAEs in related setting (see Table in Palumbo et al. (2023)
Eleobiit
Mer
Wedi
Figure 6: CMVAR (left) and D-CMVAE (right) generations on CUBICC dataset, for each learned cluster. Labels for the clusters are matched by looking at the most frequent label for the samples on the validation set assigned to the given cluster.
white andlan fad
bod
1d with long Seak small carved black fool und
tie With bacs
poirte
with rell put
fead
grop
relehvely ange feat
pointed boak
Figure 7: Five instances of conditional caption-to-image generation performance for CMVAE and D-CMVAE: CMVAE and D-CMVAE generations are shown in sequence.
5 CONCLUSION
In this work we introduce CMVAE, novel multimodal VAB, which extends recent advances for multimodal VAEs by enforcing a clustering structure in the latent space. Our experiments show that CMVAE outperforms existing multimodal VAEs in terms of generative performance, in particular showing a remarkable improvement in unconditional generation. Additionally, we show that our proposed model can accurately model data clusters in the latent space, without prior knowledge of the true number of clusters or any other form of supervision. This can be achieved by using a post-hoc procedure to accurately select the optimal number of clusters in the data, without requiring multiple trained instances of the model. Notably, we validate the effectiveness of our approach on the PolyMNIST and the CUBICC datasets. In the latter, introduced as realistic multimodal clustering setting, CMVAE achieves markedly superior clustering performance compared to both scalable weakly-supervised methods and unimodal approaches. Finally, by incorporating DDPMs into our proposed model we achieve outstanding generative results, previously unseen in multimodal VAEs, significantly boosting the applicability and relevance of our method in real-world scenarios.
Limitations and future work: While DDPMs are crucial for high-quality generations, their in- tegration results a computational overhead both at training and inference time. Indeed, each modality needs to be modeled by distinct diffusion process as the different modalities may have varying input dimensions and different features. However, simplification is feasible when the modal- ities share the same input space, by conditioning single DDPM on the different modalities. While we proved the effectiveness of our approach on challenging mynistyttettic and realistic datasets, the application real-world dataset with large number of modalities is main focus for future work.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
EP is supported by a fellowship from the ETH Al Center. EP and DC received funding from the grant #2021-911 of the Strategic Focal Area "Personalized Health and Related Technologies (PHRT)" of the ETH Domain (Swiss Federal Institutes of Technology). LM is supported by the SDSC PhD Fellowship #1-000 91-001568-033 0088-37. SL is supported by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number MB22.00047.
REFERENCES
Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems, 2020.
Jean- PatriPa Baudry, Adrian Raftery, Gilles Celeux, Kenneth Lo, and Raphaël Gottardo. Journal of computational and graphical statistics : joint publication of American Statistical Association, Institute of Mathematical Statistics, Interface Foundation of North America, 2010.
C. Biemackii, G. Celeux, and G. Govaert. Assessing a mixture model for clustering with the inte- grated completed likelihood. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000.
Gustav Bredell, Kyriakos Flouris, Krishna Chaitanya, Ertune Erdil, and Ender Konukoglu. Ex- plicitly minimizing the blur error of variational autoencoders. In International Conference Learning Representations, 2023.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. European Conference on Computer Vision (ECCV), 2018.
Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuchne, Samuel Thomas, Angie Boggust, Rameswar Panda, Brian Kingsbury, Rogerio Feris, David Harwath, et al. Multimodal clustering networks for sulf-supervised learning from unlabeled videos. In IEEE/CVF International Confer- ence on Computer Vision, 2021.
Imant Daunhawer, Thomas M. Sutter, Kieran Chin Chin-Cheong, Emanuele Palumbo, and Julia E Vogt. On the limitations of multimodal VAEs. International Conference on Learning Representa- tions, 2022.
Nat Dilokthanakul, Pedro AM Mediano, Marta Gamelo,, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. arXiv preprint /tt::///1.026444 2016.
Reuben Dorent, Samuel Joutard, Marc Modat, Sébastien Ourselin, and Tom Vercauteren. Hetero- modal variational encoder-decoder for joint modality completion and segmentation. Medical Image Computing and Computer Assisted Intervention. 2019.
Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. Som- vae: Interpretable discrete representation learning on time series. In International Conference on Learning Representations, 2019.
Prasoon Goyal, Zhiting Hu, Xiaodan Liang, Chenyu Wang, and Eric Xing. Nonparametric varia- tional auto-encoders for hierarchical representation learning. IEEE International Conference on Computer Vision (ICCV), 2017.
Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. Hierarchical topic models and the nested chinese restaurant process. Advances in Neural Information Processing Systems, 2003.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistte models. In Advances in Neural Information Processing Systems, 2020.
Zhiting Hu, Ho Qirong, Avinava Dubey, and Eric Xing. Large-scale distributed dependent nonpara- metric trees, In International Conference on Machine Learning, 2015.
HyeongJoo Hwang, Geon-Hyeong Kim, Seunghoon Hong, and Kee-Eung Kim. Multi-view repre- sentation learning via total correlation objective. In Advances in Neural Information Processing Systems, 2021.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: an unsupervised and generative approach to clustering. International Joint Con- ference Artificial Intelligence, 2017.
Tom Joy, Yuge Shi, Philip Torr, Tom Rainforth, Sebastian M Schmon, and Siddharth N. Learning multimodal VAEs through mutual supervision. In International Conference on Learning Repre- sentations, 2022.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. http://yann. http://ww....cc...oo////////siii 2010.
URL:
Changhee Lee and Mihaela van der Schaar. A variational information bottleneck approach to multi- omics data integration. In International Conference on Artificial Intelligence and Statistics, 2021.
Mihee Lee and Vladimir Pavlovic. Srivate-shared disentangled multimodal vae for learning of latent representations. CVPR Workshop on Multimodal Learning and Applications, 2021.
Xiaopeng Li, Zhourong Chen, Leonard K. M. Poon, and Nevin Lianwen Zhang. Learning latent superstructures in variational autoencoders for deep multidimensional clustering. In International Conference on Learning Representations, 2018.
Laura Manduchi, Kieran Chin- Chin-Cheong, Holger Michel, Sven Wellmann, and Julia E Vogt. Deep conditional Gaussian mixture model for constrained clustering. In Advances in Neural Informa- tion Processing Systems, 2021.
Laura Manduchi, Ričards Marcinkevičs, Michela C. Massi, Thomas Weikert, Alexander Sauter, Ver- ena Gotta, Timothy Müller, Flavio Vasella, Marian C. Neidert, Mare Pfister, Bram Stieltjes, and Julia Vogt. A deep variational approach to clustering survival data. In International Conference on Learning Representutions, 2022.
Jingchao Ni, Wei Cheng, Zhengzhang Chen, Takayoshi Asakura, Tomoya Soma, Sho Kato, and Haifeng Chen. superclass-conditoonn gaussian mixture model for learning fine-grained embed- dings. In International Conference on Learning Representations, 2022.
Mustafa Umit Oner, Hwee Kuan Lee, and Wing-Kin Kin Sung. Weakly supervised clustering by ex- ploiting unique class count. International Conference on Learning Representations, 2019.
Emanuele Palumbo, Imant Daunhawer, and Julia Vogt. MMVAE+: Enhancing the generative quality of multimodal VAEs without compromises. In International Conference on Learning Representations, 2023.
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. DiffuseVAE. Efficient, controllable and high-fidelity generation from low-dimensional latents. Transactions on Machine Learning Research, 2022.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakom. Diffu- sion autoencoders: Toward a meaningful and decodable representation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Yuge Shi, N. Siddharth, Brooks Paige. and Philip Torr Variational cixture--ffexppett autoencoders for multi-modal deep generative models. In Advances in Neural Information Processing Systems, 2019.
11

---

Published as a conference paper at ICLR 2024
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, 2015.
Thomas M. Sutter, Imant Daunhawer, and Julia E. Vogt. Multimodal generative learning utilizing Censen-Shannoon-dvvegeenee In Advances in Neural Information Processing Systems, 2020.
Thomas M. Sutter, Imant Daunhawer, and Julia E Vogt. Generalized multimodal ELBO. In Inter- national Conference on Learning Representations, 2021.
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep
generative models. arXiv preprint arXiv: 1,01911, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision (ECCV), 2020.
George Tucker, Dieterich Lawson, Shixiang Gu, and Chris J. Maddison. Doubly reparameterized gradient estimators for monte carlo objectives. In International Conference on Learning Repre- sentations, 2019.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems, 2020.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy, Generative models of visually grounded imagination. In International Conference on Learning Representations, 2018.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- UCSD birds-200-2011 1 dataset. 2011.
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu. Deep variational canonical correla- tion analysis. arXiv preprint arXiv:1610.03454, 2016.
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learn- ing. In Advances in Neural Information Processing Systems, 2018.
Jie Xu, Yazhou Ren, Huayi Tang, Xiaorong Pu, Xiaofeng Zhu, Ming Zeng, and Lifang He. Multi- vae: Learning disentangled view-common and view-peculiar visual representations for multi-view clustering. IEEE/CVF International Conference on Computer Vision (ICCV) 2021.
Runwu Zhou and Yi-Dong Shen. End-to-end adversarial-atteniin network for multi-modal cluster- ing. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
TABLE OF CONTENTS
CMVAE objective as lower bound on the log-evidence
14
Additional clustering results for CMVAE and deeper look into post-hoc selection of optimal clusters
Additional qualitative results for conditional and unconditional generation
17
D Details on datasets, metries and implementation
19
Datasets
19
D.2 Implementation details
19
D.3 Metries
20
E
Additional and extensive quantitative results
20
Quantiative results of generative performance multimodal VAES
................................
20
Baseline models in clustering experiments
21
E.2.1
PolyMNIST experiment
E.2.2
CUBICC experiment
23
13

---

Published as a conference paper at ICLR 2024
A CMVAE OBJECTIVE AS LOWER BOUND ON THE LOG-EVIDENCE
Lemma A.1. The CMVAE objective a valid lower bound on legpe(( (X)
Proof. We can prove the proposed CMVAE objective is a valid ELBO using arguments from the proof of Lemma I from Palumbo et al. 2023 As described in Section 3.1 we assume the data X = E1,-.. BM for M modalities generated according to the following process,
Pe p II Po... (m 2 2)p(Umm)
where the latent variables encode cluster assignment and shared information across modalities respectively. Modality-seecific latents W)...., . wM encode private information for the correspond- ing modality. Note that shared and modality-specifie latents
assumed to be independent. To
obtain a tractable lower bound on the log-evidence log pe (X), , we start by approximating posterior inference for 2, with an encoder qe_ (0_(z. c. x). line with our assumptions for the generative
model, we assume the encoder qa, (z, c| to factorize as q (z, c|X) q4 (z X )g(c))z X)).
To achieve scalability the number of modalities, as in previous works (Shi et al., 2019 Palumbo
et al. 2023). we assume mixture-of-expert encoder for the shared latent code z of the form
4 X Σ M
m== Ad.m. (z/xm). Hence,
derive the following ELBO
log 2 90 (= ( X)
log pe (X) > E x(cl=.X)
(11)
2o (z|x)
1
log pel X ,,
E
x(elx.X)
(12)
M
qa. (c//
90am (=/(m))
M
log Pr (c)(c)(22
40. (= X m= Pa... @m (m)
E
(13)
M
())(cl2 X)
m=1
9eam (=|xm))
which is a sum indexed by the unimodal encoders, where for each term given unimodal encoder is used for inference. Now, in (Palumbo et al.) 2023), we adopt two different estimators for the sefftreconstruction likelihood term Pe... [ [²), I.c. reconstruction
modality used for inference, and crossreccosttuuction likelihood terms po., (z, 22) i.e. reconstruction of modalities not used for
inference. Specifically, we estimate log po... (Imm 2) adopting an encoder qahnnn (mm xm) for the
private latents leading the lower bound
log pam (I0m Wmm, ()p())))))))
log po.. (mml2) 2 E gewa (wo (2m)
(14)
Johmm.
(wm (m
In contrast, to estimate cross-recoossrruction likelihoods, we adopt the lower-bound
log Pe, (z) log E,,,,r,,, (I (2, ".,. Ee (w..))) log Pe., (I,,) 2, w..
(15)
which uses auxiliary prior distribution T. (w,) specific to each target modality, and is derived by the definition of conditional expectation and Jensen's 's inequality.
Plugging the derived expressions in (14) and (15) in Eguation B recovers the CMVAE objective
in TT. The fact that (13) is an ELBO and (14) and (15) are lower bounds proves that the CMVAE
objective is valid ELBO. Note hyperparameter can be plugged in as in ( to weight latent
space regularization.
B ADDITIONAL CLUSTERING RESULTS FOR CMVAE AND DEEPER LOOK INTO POST-HOC SELECTION OF OPTIMAL CLUSTERS
In this section, show additional clustering results for CMVAE PolyMNIST and CUBICC, with a specific focus on two aspects: the entropy- based post-hoc selection of latent clusters described in Section 3.2 and clustering performance when modalities are missing test time. In particular,
14

---

Published as a conference paper at ICLR 2024
mentioned in Section 2 the efficient handling of missing modalities separates recent advancements in the class of multimodal VAEs from early approaches. In this section we further elaborate on the importance of these features for multimodal clustering models.
More results for post-hoe selection of optimal latent clusters Figurey complements the results shown in Figure 5a by displaying generation results from selected clusters, at different steps of our proposed post-hoc procedure, along with the associated values for entropy of posterior cluster assignments with likelihood penalization. The plot showcases how the minimal value for penalized entropy corresponds to true clusters being modeled in the latent space. In contrast, when K is over- specified certain true clusters are modeled by two or more latent clusters. Finally, when necessary clusters to model the data are pruned, uncertainty in cluster assignments augments, leading to an increase in penalized entropy.
Post-hoc selection of optimal latent clusters in realistic setting Figure 10 showcases the cor- responding quantitative analysis presented in Figure for for PolyMNIST on the CUBICC dataset. As such, the results in this section complement those shown in Section 4.3 demonstrating the effec- tiveness of our proposed entropy-based latent cluster selection procedure also realistic scenario. In particular, we train CMVAE on CUBICC with K 35 latent clusters, as assumed in our ex- periments, and Figure L0a illustrates the trend of our proposed catropy-based criterion as the latent clusters are pruned the CUBICC dataset. As explained in Appendix D.I this represents com- plex and realistic setting where intra-cluster variability can make it challenging to identify the latent clusters within the data. In spite of this, our post-hoc procedure consistently recovers the true num- ber of clusters, which are accurately modeled in the latent space, with only minor variations across five independent runs. Similar to Figure 5 the trends shown for test NMI, ARI, and accuracy in Figure Toal Figure Tob and Figure 10c respectively confirm that our procedure not only identifies the correct number of clusters, but also a minimal set of latent clusters necessary to correctly model the data.
CMVAE clustering performance varying number of modalities present at test time In this section, we analyze the clustering performance of CMVAE on PolyMNIST, varying the number of modalities available for inference. As mentioned in Section 2 scalability in the number of modal- ities is a crucial feature of our proposed approach, setting it apart from alternative methods (e.g. MultiVAE (Xu et al. 202T). XDC (Alwassel et al., (0200). From the results in Figure S and Table 3 draw the following important conclusions. First, as more modalities
present for infer- ence, CMVAE clustering performance improves. While this aspect might be easily overlooked, is crucial that for successful multimodal learning, particularly for the successful performance of mul- timodal VAFs, adding modalities does not degrade model performance (Shi et al. 2019 Palumbo
al. 2023)
Moreover, we report an additional comparison to validate the importance and effectiveness of clus- tering in the presence of weak-supervision. Specifically, we train VaDE (Jiang et al.) (017) on the MNIST dataset and compare the results with the ones obtained with CMVAE when a single modal- ity is present at test time. In fact, single modality of the PolyMNIST dataset essentially presents slightly more challenging version of the MNIST dataset, where instead of grayscale images, the background a random crop from a given colored image (Sutter et al. 202T). Notably, CMVAE's clustering performance when tested on a single PolyMNIST modality surpasses that of VaDE trained (and tested) on the grayscale MNIST dataset, where we control for latent space regularization, and VaDE is trained with the correct number of latent clusters, namely 10. This, despite the slightly more challenging nature of PolyMNIST modality compared to grayscale MNIST. This indicates that weak supervision during training can be beneficial even when testing is done single modal- ity.
15

---

Published as a conference paper at ICLR 2024
2.5
dusee disi
2.3
2.2
2.1
40
35
30
20
10
Number of chasters
2
Figure 9: Generated qualitative examples of the clustering capabilities of CMVAE on PolyMNIST dataset. Each column corresponds to the generated samples of one of the five modalities, while each row reports generation results for a different latent cluster.
135
0.8-
0.8-
0.8-
F 70.8-
XMT
0.1
0.1-
0.2
0.2
0.2
0
5
1
3
2
2
Number modalities
Number modalities
Number modalities
(a) NMI
(b) ARI
(c) Accuracy
Figure 8: CMVAE clustering results in terms of test NMI (a), ARI (b), and accuracy (c) on PolyM- NIST, varying the number of modalities present at test time. Results for five independent runs are shown.
# modalities|
Accuracy
ARI
NMI
Accuracy
ARI
NMI
5
0.985 (0.01) 0.969 (0.02) 0.971 (0.02)
VaDE
0.739 (0.07) 0.619 (0.07) 0.688 (0.04)
4
0.977 (0.01) 0.952 (0.02) 0.955 (0.01)
MNIST grayscule
3
0.962 (0.01) 0.920 (0.02) 0.924 (0.01)
CMVAE
2
00,99 (0.01) 0.893 (0.03) 0.900 (0.02)
PolyMNIST
0.900 (0.02) 0.790 (0.03) 0.807 (0.02)
1
(0.900 (0.02) 0.790 (0.03) 0.807 (0.02)
single medality
Table 3: Left: CMVAE clustering results trained on PolyMNIST, varying the number of modalities present at test time. Right: Comparison of CMVAF trained on PolyMNIST when a single modality present at test time, and VaDE trained (and tested) on the grayscale MNIST dataset.
16

---

Published as a conference paper at ICLR 2024
Narborofdata
of
(a) Entropy
(b) NMI
0.4
30
30
30
(c) ARI
(d) Accuracy
Figure 10: Post-hoc optimal latent cluster selection with the procedure in Algorithm on the CU- BICC dataset. (a) Evolution the entropy term (y-axis) as clusters (x-axis) are pruned. The minimal entropy recovers the true number of clusters with only minimal variations. The trends for test NMI (b), ARI (c) and accuracy (c) support the validity of our procedure, showing the procedure selects minimal set of latent clusters correctly model the true data clusters.
C ADDITIONAL QUALITATIVE RESULTS FOR CONDITIONAL AND UNCONDITIONAL GENERATION
In this section, we report qualitative results for CMVAE to complement the quantitative analysis Figure 3 We do this by providing qualitative examples for both unconditional and conditional generation in Figure 12 and Figure [1] respectively. In particular, here in Section 4.1 we report generations using CMVAE trained with 3 2.5, without resorting to the D-CMVAE extension, for fair comparison with alternative multimodal VAEs tested. Importantly, these results in line with our quantitative results in Figure 12 in particular the high semantic coherence unconditional generation, which separates our model from previous approaches evident. To have qualitative comparison with the other models reported in Figure S we refer to Appendix G.I in the work of Palumbo et al. (2023). In Figure [3] we show conditional generation results on PolyMNIST when adopting D-CM which boosts generative quality while at the same time generative coherence preserved.
6.
9
3
7
8
O
1
3
)
3
4"
7
&
3
7
3
%
J
7
9
(a) modality 2
(b) modality 3
(c) modality
(d) modality 5
Figure 11: Conditional generation from the first modality to the remaining ones for CMVAE trained with | 2.5 the PolyMNIST dataset. In each image, the top are starting samples, and below ten instances of conditional generation for the corresponding target modality. Qualitative results complement the analysis in Figure
17

---

Published as a conference paper at ICLR 2024
C
6
2
7
S
(a) modality 1
(b) modality 2
(c) modality 3
4
0
Q
8
6
3 3"
69
9
0
0
2
6
7
8
3
7
2
9
5
0
6
1
9
5
3
4
9
8
3
H
4
3
(d) modality 4
(c) modality 5
Figure 12: Uncondtional generation for CMVAR traind with 2.5 on the PolyMNIST dataset.
A hundred instances across modalities are shown. Qualitative results complement the analysis in Figure
2
5
6
9
2
8 9
Figure 13: D-CMVAE conditional generation the PolyMNIST dataset: the top row the starting samples from the first modality and on the four remaining rows are the conditional genera- tions in the remaining modalities.
we show CMVAE unconditional generations on the CUBICC dataset, jointly generat- ing images and captions. As expected, the results show high coherence across the two modalities. Also, as quality of the images VAE generations somehow limited, so quality of text, which could be improved in future work. However, these results favourably compare related results from recent work (Palumbo et al.] 2023). Finally, we perform an ablation on the D-CMVAE model and test the generative performance by conditioning the reverse diffusion process on only the self reconstructions of CMVAE As can be seen in Figure 15] this produces sub-optimal generative re- sults. The integration of the cross-modal reconstructions in the training procedure is crucial obtain high-quality generations.
In Figure 14,
breast Atricsed secher with
Uhe yallo with small brown ands ronrow
bed with Mlangdar colenve wnd
this partioler has pelooa gray ponchee and wings.
big booked small Made and the shest.
hox long. per
pointy
wings.
Figure 14: Six instances of CMVAE unconditional generation on the CUBICC dataset, where we jointly generate images and captions. We do not employ D-CMVAE in these results.
18

---

Published as a conference paper at ICLR 2024
Figure 15: D-CMVAE random generations of CUBICC data by conditioning the reverse diffusion process only the self reconstructions of CMVAE. This results in sub-optimal generative quality.
D DETAILS ON DATASETS, METRICS AND IMPLEMENTATION
D.I DATASETS
In this section, provide detailed information about the datasets used in this work. The first dataset we use is the PolyMNIST dataset (Sutter et al.) (021), a semi-synthetic yet challenging dataset consisting of five image modalities: each modality depicts MNIST (LeCun et al. 2010) digits patched on random crops from five distinct background images, one for each modality. Figure 2a showcases some illustrative samples from the dataset. Note that the digit label is the only shared information across modalities, while the background features and the handwriting style differ across modalities in each data point.
As a second experimental setting, we introduce modified version of the CUB Image- Saage-Captions dataset (Wah et al.) [20IT] [hhi et al.] 2019). This dataset originally consists of images of birds paired with corresponding descriptive captions. Our version, named the CUB Image-Caption for Clustering (CUBICC) dataset, serves as a benchmark for evaluating multimodal clustering methods under more realistic conditions. To create this dataset, grouped sub-species of birds into single species category. as illustrated in Figure I6 As a result, the CUBICC dataset consists eight classes, each representing different bird species. The grouping of sub-species into single class introduces significant variability within each class, posing a considerable modeling challenge.
Blackbird
Gull
Jay
Oriole
Blackbirdd
Rustty
Red-winged
Blackbird
Herring
Blue Jay
Florida Jay
Scott's Oriole
Hooded Oriale
Baltimore
Oriale
Blackbird
Tanager
Ter
Warbler
Wren
Scarlet Tanager
Summer Tanager
Elegant Term
Caspian Tarn
Pina Warbler
Vallow Warbler
House Wiren
Caroline Wren
Artic Tern
Kantucky Warbler
Marsh Wren
Figure 16: CUBICC dataset structure. Bird sub-species are grouped together single category.
D.2 IMPLEMENTATION DETAILS
To implement all multimodal VAEs Wu & Goodman 2018 Shi et al. 2019 Sutter et al. 2020
2021 Hwang et al. 202T] Palumbo et al. 2023) included in Section 4. for comparison in terms of
generative performance follow the same settings in Palumbo al. 2023 for training these
19

---

Published as a conference paper at ICLR 2024
models on the PolyMNIST dataset and get best performance. In particular, we use the same ResNet encoder and decoder networks for all compared models. CMVAE is trained for 250 epochs on this dataset, with 1e" learning rate. As done for MMVAE+, again following the original work of
Palumbo et al. 2023), ,. we set both the number of dimensions for the shared and modality-specific latent subspaces of CMVAE to 32 for experiments
PolyMNIST. To train CMVAE on the CU- BICC dataset, we use ResNet and convolutional encoder/decoder networks for the image and text modalities respectively. As in Palumbo al. (2023). on this dataset we use 10-sample estimator in our CMVAE ELBO objective and resort to the DReG estimator for gradients (Tucker et al., 2019). Again, we train the model for 250 epochs. We set 64 dimensions and 32 dimensions for shared and mppeiit latent spaces respectively. For the methods reported as baseline comparisons for clustering tasks, we ensure compatibility with the encoder networks and shared latent space size adopted by CMVAR for a fair comparison. For more details see also Appendix F.2 We follow best practices in their original work for training, but without resorting to any pre-training procedures (for c.g. VaDE (Jiang et al.) 2017)). again for fair comparison with CMVAE. On PolyMNIST and CUBICC, CMVAE is trained with K = 40 and K = 35 respectively, before the post-hoc procedure deseribed in Section 4.2 is applied. Experimental results across the paper are reported averaging over five independent seeds and we report standard deviations, with the exception of results in Fig- ure 3 and Table 4 where we use three independent seeds. In our clustering experiments for both PolyMNIST and CUBICC datasets we have independent training, validation, and test splits. For PolyMNIST there are 60000 samples for training, 5000 samples for validation, and 5000 samples for testing. For CUBICC, we have 1 11834 training, 638 validation, and 659 test samples. With CM- VAE, we perform the post-hoe procedure described Section fection[4.2]for optimal cluster selection on the validation datasets for both experiments, while clustering results for all models are always reported on the test set. We share the code for our model at https: // //g///ttttt/ /ggttuu......./ cot/ epall/ /CMVAE.
D.3 METRICS
Consistently with previous work (Sutter et al.) 2021] Daunhawer et al. 2022 Hwang et al. 2021 Palumbo et al. 2002), to assess generative quality in our experiments, we resort to the FID score Heusel et al. (2017). which is a well-established metric that has been shown to correlate well with human judgement. Moreover, again consistently with previous work (Shi et al. 2019 Palumbo et al., 2023). we assess generative coherence for our comparisons on the PolyMNIST dataset resorting to the usage of M pre-trained digit classifiers, each one trained on given modality. In particular, to evaluate my + m2 generative coherence, where m,, 72 are two distinct modalities, we feed conditionally generated samples using m as input and m2 as target modality, to the pre-trained digit classifier trained on modality m2, Then we compute the rate with which the given classifier predicts the correct label, i.e., the label from the input samples from modality mi. Finally, to obtain a single metric, we average results for conditional coherence across target and input modalities. To assess generative coherence for unconditional generation, we feed generated samples for each modality to the corresponding pre-trained classifier and compute the rate with which the predicted label matches across all modalities.
E ADDITIONAL AND EXTENSIVE QUANTITATIVE RESULTS
E.1 QUANTIATIVE RESULTS OF GENERATIVE PERFORMANCE OF MULTIMODAL VAES
Table provides a summary of generative performance, including both average and corresponding standard deviation, derived from three independent runs for the multimodal VAEs illustrated in Figure 3 Both conditional and unconditional generation performance across different values of the regularization hyperparameter 33 are reported.
20

---

Published as a conference paper at ICLR 2024
8= 1.0
Unconditional
Conditional
FID
Coherence
FID
Coherence
MVAE
50.65 (0.72)
0.007 (0.001)
82.59 (6.22)
0.093 (0.009)
MVTСAЕ
110.85 (2.61)
0,000 (0.000)
58.98 (0.62)
0.509 (0.006)
mmJSD
179.76 (2:97)
0.054 (0.011)
209.98 (1.26)
0.785 (0.023)
MoPoE- VAE
98.56 (1:32)
0.037 (0.002)
160.29 (4.12)
0.723 (0.066)
MMVAE
165.17 (3.40)
0,222 (0.019)
152,11 (4.11)
0.837 (0.004)
MMVAE+
86.64 (1.04)
0.095 (0.02)
80.75 (0.18)
0.796 (0.010)
CMVAE
78.52 (0.63)
0.560 (0.025)
74.53 (0.64)
0.806 (0.009)
2.5
Unconditional
Conditional
FID
Coherence
FID
Coherence
MVAE
58.53 (0.12)
0.080 (0.006)
85.23 (9.37)
0.298 (0.044)
MVTCAE
87.07 (0.89)
0.003 (0.000)
62.55 (1.30)
0.591 (0.004)
mmJSD
180.55 (8.67)
0.060 (0.010)
222.09 (5.34)
0.778 (0.003)
MoPoE- VAE
107.11 (0.780)
0.141 (0.05)
178.27 (2.01)
0.720 (0.008)
MMVAE
164.71 (3.17)
0.232 (0.010)
150.83 (2.69)
0.844 (0.010)
MMVAE+
96.01 (2.10)
0.344 (0.01))
92.81 (0.78)
0.869 (0.013)
CMVAE
85.68 (0.66)
0.781 (0.021)
85.12 (0.75)
0.897 (0.003)
B 5.0
Unconditional
Conditional
FID
Coherence
FID
Coherence
MVAE
61.25 (0.40)
0.112 (0.010)
90.37 (3.20)
0.301 (0.024)
MVTCAE
85,43 (2.80)
0.029 (0.001)
74.61 (3.41)
0.604 (0.004)
mmJSD
186.49
(2.89)
0.076 (0.018)
226.20 (2.91)
0.784 (0.029)
MoPoE- VAE
122.68
(1.96)
0.238 (0.001)
182.99 (1.96)
0.673 (0.002)
MMVAE
164.29
(2.97)
0.229 (0.017)
152.11 (3.18)
0.839 (0.010)
MMVAE+
109.08 (1.41)
0,421 (0.006)
107.78 (0.88)
0,836 (0.023)
CMVAE
103.95 (0.16)
0.775 (0.024)
102.36 (0.83)
0.882 (0.010)
Table 4: Conditional and unconditional generative performance of multimodal VAEs across different values of hyperparameter B.
E.2 BASELINE MODELS IN CLUSTERING EXPERIMENTS
In this section, we report further results for the baseline methods we include in our clustering ex- periments. Specifically, we compare our proposed CMVAE with existing unimodal clustering ap- proaches, namely VaDE (Jiang et al. 2017) and DeepCluster (Caron et al. 2018), as well as with weakly-supervised multimodal approaches, namely XDC (Alwassel et al. 2020) and CMC (Tian) et al. (020).. The reported results the main text correspond to giving as input the true a-priori number of clusters to the model. In particular for the unimodal approaches, formance across all modalities. It important to note that all these methods require a pre-specified value for the number of clusters K, unlike CMVAE. On PolyMNIST, we test these models using K = 100, K = 40, and K = 10. The rationale behind this choice is as follows: testing with K 40 allows for a fair comparison with CMVAE, which is trained with K 40 in PolyM- NIST experiments, providing a fair and realistic scenario where prior information on the true number of clusters is unavailable. Furthermore, we evaluate these methods with K 100 to give them large modeling capacity. Finally, we explore a facilitated setting where the true number of clusters K 10 given for training. Similarly, in the CUBICC dataset. we evaluate all corresponding baseline methods with K = 100, K = 35, and K 8 clusters, following the same rationale as in the PolyMNIST experiment.
select the best per-
E. POLYMNIST EXPERIMENT
Unimodal approaches Here present the results for the unimodal clustering approaches on the PolyMNIST dataset. We evaluate VaDE using different values of the regularization hy typrparraeeter in the VAE objective, specifically 1.0, 2.5, and 5.0, selecting the value corresponding to best
21

---

Published as a conference paper at ICLR 2024
performance for comparisons in the main text. The results for these approaches trained on each modality are reported in Table Values reported in the main text are underlined.
PolyMNIST: Unimodal clustering
m0
ml
m2
m3
m4
NMI
0.031 (0.01)
0.123 (0.05)
0.013 (0.01)
0.391 (0.04)
0.004 (0.00)
$=1.0
ARI
0.011 (0.00)
0.074 (0.03)
0,005 (0.01)
0.313 (0.04)
0.000 (0.00)
Acc
0.147 (0.01)
0.231 (0.04)
0.130 (0.01)
0.500 (0.05)
0.120 (0.00)
NMI
0.025 (0.01) 0.154 (0.05)
0.008 (0.01)
0.431 (0.04)
0.007 (0.00)
VaDE
B=2.5
ARI
0.010 (0.01)
0.094 (0.03)
0.002 (0.00)
0.364 (0.04)
0,002 (0.00)
Acc
0.141 (0.01)
0.262 (0.04)
0.124 (0.01)
0.544 (0.05)
0.127 (0.00)
NMI
0.031 (0.02) 0.101 (0.06)
0.035 (0.04)
0.415 (0.09)
0.004 (0.00)
8=5.0
ARI
0.014 (0.01) 0.054 (0.03)
0.018 (0.02)
0.355 (0.10)
0,000 (0.00)
Acc
0.145 (0.02)
0.208 (0.04)
0.147 (0.03)
0.538 (0.
0.120 (0.00)
NMI
0.023 (0.01) 0.038 (0.02)
0.023 (0.01)
0.122 (0.02)
0.006 (0.00)
K=10
ARI
0.009 (0.01)
0.015 (0.01)
0.010 (0.01)
0.079 (0.02)
0.002 (0.00)
Acc
0.140 (0.01)
0.5 (0.06)
0.143 (0.013)
0.258 (0.04)
0. 0.124 (0.00)
NMI
0.089 (0.01)
0.132 (0.03)
0.076 (0.02)
0,141 (0.02)
0.013 (0.00)
DeepCluster
K=35
ARI
0.024 (0.00)
0.047 (0.01)
0.023 (0.01)
0.051 (0.01)
0.001 (0.00)
Acc
0.096 (0.00)
0.138 (0.01)
0.100 (0.02)
0.145 (0.01) 0.060 (0.00)
NMI
0.163 (0.02)
0.210 (0.03)
0.105 (0.02)
0.180 (0.02) 0,033 (0.00)
K=100
ARI
0.026x0.00
0.039 (0.01)
0.014 (0.09)
0.031 (0.01) 0.001 (0.00)
Acc
0.069 (0.01) 0.088 (0.01) 0.054 (0.01) 0.081 (0.01) 0.031 (0.00)
Table 5: Baseline results for unimodal clustering on PolyMNIST dataset. Including ablation of B for VaDE and number of clusters for DeepCluster. Mean and standard deviation of five independent runs are reported. The underlined results correspond to those reported in Table 2 in the main text.
Multimodal approaches In addition, present results on the multimodal benchmarks. XDC is designed for two modalities and it does not directly scale to a larger number of modalities, hence, do not include it in the main text, but show it here for reference. To adapt XDC for PolyM- NIST, we select uniformly at random which other modality to extract the pseudo-labels from at each iteration. Nevertheless, its low performance highlights that the method is not properly scalable. The second baseline, CMC, is contrastive-based approach for multi-view representation learning. To evaluate its clustering capabilities, we train instance of K K-means on the embedding space. As discussed in Section 4.2] the high performance of CMC on PolyMNIST is on par with our proposed CMVAE. However, this performance obtained only when the model has a-priori knowledge the number of clusters, which is not realistic in practice. Particularly, in Table we can see how without a-priori information, i.e. being 40 as in CMVAE, CMC performance decreases significantly and no-longer par with CMVAE. For this analysis, explore values of K 10, 20, 35, 40, and 100 to illustrate how such high performance is only attainable with an inncee-nfforeed a-priori K. For both CMC and XDC, the final metries reported in Table O after averaging the embedding space generated by each modality and clustering using -means on the average space. The results used in the main text are underlined in Table
22

---

Published as a conference paper at ICLR 2024
PolyMNIST: Multimodal clustering
NMI
0.020 (0.01)
K=10
ARI
0.009 (0.01)
Acc
145 (0.01)
NMI
0.118 (0.13)
XDC
K=35
ARI
0.053 (0.05)
Acc
0.102 (0.05)
NMI
0.122 (0.07)
K=40
ARI
0.038 (0.03)
Acc
0.100 (0.02)
NMI
0.360 (0.08)
K=100
ARI
0.082 (0.02)
Acc
0.101 (0.02)
NMI
0.970 (0.01)
K=10
ARI
0.969 (0.02)
Acc
0.986 (0.01)
NMI
0.876 (0.01)
K=20
ARI
0.706 (0.02)
Acc
0.617 (0.02)
CMC
NMI
0,800 (0.00)
K=35
ARI
0.480 (0.01)
Acc
0.407 (0.01)
NMI
0.784 (0.00)
K=40
ARI
0.448 (0.01)
Acc
0.380 (0.01)
NMI
0.685 (0.00)
K=100
ARI
0.261 (0.11)
Acc
0.219 (0.09)
Table 6: Baseline results for multimodal clustering on the PolyMNIST dataset, including ablation of the number of clusters K. Underlined results are reported in the main text.
E.2.2 CUBICC EXPERIMENT
Unimodal and multimodal approaches In this section, we present results for the clustering ap- proaches the CUBICC dataset. As in the PolyMNIST dataset, we examine VaDE's performance using different values of the B hyperparameter. We also test DeepCluster with varying K. Results are reported in Table 7 Note that only the image modality reported for DeepCluster, method for image data. The multimodal results are displayed in Table 8 Unlike the PNISM dataset, CUBICC involves two modalities, aligning with XDC's original design, which reflected in the performance improvement observed. In both tables, the results reported in the main manuscript underlined.
it is
Latent space ablation Previous results in this experimental setting used a 64-dimmnsional latent space. Here increase the latent space size to 96 dimensions and show in Table 9 that this in- crease latent space capacity does not result significant changes in performance for the reported methods.
23

---

Published as a conference paper at ICLR 2024
CUBICC: Unimodal clustering
Image
Text
NMI
0.079 (0.02)
0.021 (0.00)
6=1.0
ARI
0.041 (0.01)
0.002 (0.00)
Acc
0.243 (0.03)
0.180 (0.00)
NMI
0.112 (0.03)
0.020 (0.00)
VaDE
B=2.5
ARI
0.063 (0.02)
0,000 (0.00)
Acc
0.259 (0.02)
0.174 (0.00)
NMI
0.155 (0.01)
0.019 (0.00)
B=5.0
ARI
0.084 0.01)
0.000 (0.00)
Acc
0.269
0.171 (0.00)
NMI
0.191 (0.01)
K=8
ARI
0.100 (0.01)
Acc
0.294 (0.01)
NMI
0.247 (0.00)
DeepCluster
K=35
ARI
0.058 (0.00)
Acc
0. 0.646 (0.00)
NMI
0.310 (0.01)
K=100
ARI
0.035 (0.00)
Acc
0.099 (0.01)
Table 7: Baseline results for unimodal clustering on test images from CUBICC dataset, including ablation of 3 for VaDE and number of clusters K for DeepCluster. Underlined results are the ones reported in Section in the main text, namely the ones corresponding to the best modality and the best for VaDE and to the true number of clusters for DeepCluster.
CUBICC: Multimodal clustering
NMI
0.218 (0.06)
K=8
ARI
0.109 (0.02)
Acc
0.281 (0.05)
NMI
0.262 (0.02)
XDC
K=35
ARI
0.116 (0.10)
Acc
0.224 (0.10)
NMI
0.323 (0.00)
K=100
ARI
0.035 (0.00)
Acc
0.091 (0.01)
NMI
0.374 (0.05)
K=8
ARI
0.097 (0.03)
Acc
0.310 (0.04)
NMI
0.586 (0.02)
CMC
K=35
ARI
0.313 (0.07)
Acc
0.385 (0.09)
NMI
0.556 (0.01)
K=100
ARI
0.180 (0.02)
Acc
0.211 (0.02)
Table 8: Baseline results for multimodal clustering on CUBICC dataset. Including ablation of
number of clusters. Underlined results the ones reported in Section 4.2 in the main text.
24

---

Published as a conference paper at ICLR 2024
CUBICC: 96-dimensional latent space
NMI
0.178 (0.01)
Vade 3=5,0
ARI
0.103 (0.01)
Acc
0.297 (0.00)
NMI
0. 195 (0.02)
DeepCluster K=8
ARI
0. 106 (0.02)
Acc
0.304 (0.02)
NMI
0.246 (0.04)
XDC K=8
ARI
142 (0.03)
Acc
0.339 (0.02)
NMI
0.313 (0.05)
CMC K=8
ARI
0. 124 (0.12)
Acc
0.268 (0.04)
Table Results clustering baselines on a latent space of 96 dimensions for the CUBICC dataset.
25


Reviews:
[{"review_id": "DIwyUduCsn", "reviewer": "Reviewer_i4Eb", "summary": "This work concerns multimodal VAE applied to clustering. In doing so the authors propose a three-fold approach: a VAE objective for latent clustering, an entropy based cluster number selection, and Diffuse VAE integrated to the multimodal setting. Furthermore the authors present the experiments using fairly common tasks for multimodal VAE plus for weakly supervised clustering.", "questions": "Can the proposed multimodal VAE be applied to more complex data (say datasets with higher resolution, like AFHQv2, FFHQ) and if so, how would this impact the visual quality of the results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, this paper is well-written and easy to read. It’s advocately positioning itself on the problem of clustering with multimodal VAE, and, through a neat combination of ideas that are slightly improved over previous works, it present a holistic solution that outperform previous works both in multimodal VAE and weakly supervised clustering.\n\nSpecifically, the objectives are heavily inspired by the previous works. The choice of selecting $k$ based on entropy is both intuitive \nincorporation of DiffuseVAE is direct yet impactful. Despite these individual components, what stands out is the overall pipeline being orchestrated coherently . The authors deserve commendation for doing so.\n\nThe experimental results are satisfactory and in line with common setups for multimodal. However this could be improved (see the weakness below).", "weaknesses": "As this paper is positioned as multimodal VAE for clustering, a broader empirical comparison with weakly supervised model would adds greatly to its value.\n\nWhile the current version compares the proposed method with  [1,2,3], there is an wide range of research in weakly supervised clustering, including but not limited to [4, 5]. Although these may not be latent models, they (with other works) demonstrates the big picture of  weakly supervised clustering and deserve comparison\n\n\nAdditionally, the quantitative results relegated to the Appendix are pivotal for a comprehensive grasp of the proposed work's empirical performance. It would be beneficial to include these findings in the main body of the text.\n\nFurthermore, a lot of quantitative results in Appendix is actually crucial in understanding the empirical performance of the proposed work. It would be beneficial to include these them in the main body of the text.\n\n\n---------\n\n[1] Jiang et al. Variational deep embedding: an unsupervised and generative approach to clustering IJCAI 2017\n\n[2] Caron et al. Deep clustering for unsupervised learning of visual features.  ECCV 2018\n\n[3] Tain et al. Contrastive multiview coding ECCV 2020\n\n[4] Oner et al. Weakly Supervised Clustering by Exploiting Unique Class Count ICLR 2020\n\n[5] Chang et al.  Deep adaptive image clustering ICCR 2017\n\n[6] Yang et al. Joint unsupervised learning of deep representations and image clusters CVPR 2016", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ouFmgebOK6", "reviewer": "Reviewer_XPst", "summary": "The authors present a neat extension of typical VAEs to the multi-modal settings. It results in a framework that can be used for weakly-supervised clustering. By further combining with Denoising Diffusion Probabilistic Models, it can generative better multi-modal", "questions": "1. Since the contribution of this work is the combination of several existing approaches based on VAEs. It is important for the authors to focus on to further justify the contribution of extension VAEs to multi-modal VAEs in these tasks are not trivial and require careful design. I suggest the authors conduct ablation studies of such designs e.g. Eq.8.\n\n2. To claim the scalability of the method, the authors may consider more experiments on realistic datasets with more than two modalities.\n\n### After rebuttal\nThanks for the response and addressing my concerns. I raised my score to borderline accept. I suggest the authors consider moving some content in the Appendix (e.g. Eq.8 ablation study) to the main manuscript.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The presentation is clear and well-structured.  \n- The authors report solid experimental results, both qualitatively and quantitatively on the selected two datasets. \n- The method is technically sound, and several prior works on VAEs are neatly integrated together.", "weaknesses": "1. I'm concerned about the technical novelty. The authors extend typical VAEs to multi-modal VAEs for two tasks. The first is the application of weakly-supervised clustering, and the second is to combine multi-modal VAEs with DDPM to improve its generative quality. However, both applications of VAEs are not novel and can be viewed as an extension from VAEs to multi-modal VAEs. Though some new tricks are proposed in the form of multi-modal VAEs like Eq.8, it is not properly discussed and evaluated with sufficient ablation studies. \n\n2. The experiments are only conducted on one realistic multi-modal dataset that only contains two modalities. The authors mention the state-of-the-art weakly supervised methods are not scalable to numerous modalities while the proposed method is. The prior work is either on image-text [1] datasets or video-audio [2] [3] datasets. In Table 2, on PolyMNIST the proposed method is not improved over non-VAE SOTA either.\nTherefore, I think the authors overclaim the contribution of the scalability of numerous modalities. Especially for weakly-supervised clustering, in my opinion, a realistic multi-modal dataset is essential to claim the contribution of scalability to more modalities.\n\n\n[1] Zhou et al., End-to-End Adversarial-Attention Network for Multi-Modal Clustering.\n\n[2] Chen et al., Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos.\n\n[3] Alwassel et al., Self-Supervised Learning by Cross-Modal Audio-Video Clustering.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sfL9yeXKvf", "reviewer": "Reviewer_GVdd", "summary": "This paper introduces a novel multimodal Variational Autoencoder (VAE) model that combines weakly-supervised learning and clustering for multiple heterogeneous modalities. The key contributions of the paper are as follows:\n\n**Multimodal VAE Model:** The paper presents a new multimodal VAE model that extends the latent space to simultaneously learn data clusters while leveraging shared information across different data modalities.\n\n**Improved Generative Performance:** Experimental results demonstrate that the proposed model outperforms existing multimodal VAEs in terms of generative performance, particularly for unconditional generation tasks.\n\n**Automatic Cluster Selection:** The paper introduces a post-hoc procedure to automatically determine the number of true clusters, addressing critical limitations of previous clustering frameworks. This enhances the model's ability to discover meaningful clusters in weakly-supervised settings.\n\n**Comparison to Alternative Clustering Approaches:** The proposed method is compared favorably to alternative clustering approaches in weakly-supervised settings, highlighting its effectiveness in clustering and learning representations from multimodal data.\n\nIntegration of Diffusion Models: The paper incorporates recent advancements in diffusion models into the proposed method to further enhance generative quality, particularly for real-world images. This integration helps improve the realism of generated images.", "questions": "The questions were asked in the weaknesses part.\n\nBTW, what are the cluster in CUBICC dataset? species or the subspecies?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I appreciate the authors' great effort in providing various experimental studies in the main part as well as the appendix.\n\n* The paper is well-organized.\n* The results show clear improvement over the baseline methods, especially for unconditional generation and realistic data clustering.\n* The idea is incrementally inspired from recent MVAE methods as well as DiffuseVAE; But, it is sound and interesting.\n* The automatic cluster selection on test data is interesting and novel, showing advantage to the prior work.", "weaknesses": "The proposed method needs better explanations.\n\n* What are the learned modules in CMVAE? Are all $q(z | X)$, $q(w_i | x_i)$, $p(w_i)$, $p(c)$, $p(z|c)$ learned during the training?\n* In figure 1, I cannot understand why there exist $z_1$, ..., $z_M$.\n* I cannot understand the cross-reconstruction formulation in Eq (8). I was thinking $p_{\\theta_m}$ can be only called with $w_m$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)