Unity by Diversity: Improved Representation Learning for Multimodal VAEs

Authors: thomas m sutter
yang  meng
andrea  agostini
daphné  chopard
norbert  fortin
julia e vogt
bahbak  shahbaba
stephan  mandt
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 17:09:02

Abstract: Variational Autoencoders for multimodal data hold promise for many tasks in data analysis, such as representation learning, conditional generation, and imputation.
Current architectures either share the encoder output, decoder input, or both across modalities to learn a shared representation. 
Such architectures impose hard constraints on the model. 
In this work, we show that a better latent representation can be obtained by replacing these hard constraints with a soft constraint. We propose a new mixture-of-experts prior, softly guiding each modality's latent representation towards a shared aggregate posterior.
This approach results in a superior latent representation and allows each encoding to preserve information better from its uncompressed original features. In extensive experiments on multiple benchmark datasets and two challenging real-world datasets, we show improved learned latent representations and imputation of missing data modalities compared to existing methods.

Main paper: 
 introduction: 
 The fusion of diverse modalities and data types is transforming our understanding of complex phenomena, enabling more nuanced and comprehensive insights through the integration of varied information sources. Consider, for instance, the role of a medical practitioner who synthesizes multiple tests and measurements during diagnosis and treatment. This process involves merging shared information across different tests and identifying test-specific details, both of which are critical for optimal patient care and medical decision-making.Among the existing methods, multimodal Variational Autoencoders (VAEs) have emerged as a promising approach for jointly modeling and learning from weakly-supervised heterogeneous data sources. While scalable multimodal VAEs utilizing a single shared latent space efficiently handle multiple modalities [Wu and Goodman, 2018 , Shi et al., 2019 , Sutter et al., 2021] , finding an optimal method to aggregate these modalities remains challenging. The aggregation methods and resulting joint representations are often suboptimal and overly restrictive [Daunhawer et al., 2022 , Sutter, 2023] , leading to inferior latent representations and generative quality. This trade-off between shared and modality-specific information in the latent representations of multimodal VAEs results in limited quality or coherence in generated samples, even in relatively simple scenarios.Figure 1 : Independent VAEs (Figure 1a ) provide reconstructions for individual modalities but lack information sharing across modalities. Multimodal VAEs with joint posterior approximation (Figure 1b ) aggregate unimodal posteriors into a joint posterior but may incur poor reconstruction quality. Our proposed MMVM VAE (Figure 1c ) enhances independent VAEs with a data-dependent prior, h(z | X), allowing soft-sharing of information between modalities while preserving modalityspecific reconstructions.In this work, we propose a novel multimodal VAE, termed the multimodal variational mixture-ofexperts prior (MMVM) VAE, to overcome the aforementioned limitations. Instead of modeling the dependencies between different modalities through a joint posterior approximation, we introduce a multimodal and data-dependent prior distribution (see Figure 1 ). Our proposed multimodal objective is inspired by the VAMP-prior formulation introduced by Tomczak and Welling [2017] , which is traditionally used to learn an optimal prior distribution between unimodal data samples, whereas we aim for an optimal prior between different modalities of the same data sample. The resulting regularization term in the VAE objective can be interpreted as minimizing the distance between positive pairs, similar to contrastive learning methods [Oord et al., 2019 , Tian et al., 2020] ; see Section 4 for details.We demonstrate the superior performance of the MMVM VAE on three multimodal benchmark datasets, comparing it to unimodal VAEs and multimodal VAEs with joint posterior approximations. Our evaluation focuses on the generative coherence and the quality of the learned latent representations. While independent unimodal VAEs fail to leverage additional modalities during training, they avoid multimodal aggregation disturbances in data reconstruction. On the other hand, multimodal VAEs with a joint posterior approximation must combine both shared and modality-specific information. Previous work by Daunhawer et al. [2022] has shown that this approach results in a trade-off between reconstruction quality and learned latent representation. In contrast, the MMVM VAE accurately reconstructs all modalities and learns meaningful latent representations.In more practical settings, we address two challenging tasks from the neuroscience and medical domain. First, we analyze hippocampal neural activities from multiple subjects in a memory experiment [Allen et al., 2016] . By treating each subject as a modality, our MMVM VAE enables the description of underlying neural patterns shared across subjects while quantifying individual differences in brain activity and behavior, thereby providing potential insights into the neural mechanisms underlying memory impairment. Second, we tackle identifying cardiopulmonary diseases from chest radiographs using the MIMIC-CXR dataset [Johnson et al., 2019b] , which reflects real-world conditions with images of varying quality. By leveraging both frontal and lateral X-ray views as distinct modalities, our MMVM method learns representations that consistently improve disease classification compared to existing VAEs. This paper advances multimodal machine learning by providing a robust framework for integrating diverse data types and improving the quality of learned representations and generative models. 
related work: 
 Multimodal Learning. While there is a long line of research on multimodal machine learning (ML) [Baltrušaitis et al., 2018 , Liang et al., 2022] , multimodal generative ML has gained additional attraction in recent years [Manduchi et al., 2024] , driven by impressive results in text-to-image generation [Ramesh et al., 2021 , 2022 , Saharia et al., 2022] . Unlike these methods, we focus on scalable methods that are designed for a large number of modalities to generate any modality from any other modality without having to train a prohibitive number of different models2 .Multimodal VAEs. Scalable multimodal VAEs using a joint posterior approximation are based on aggregation in the latent space 3 . Multimodal VAEs that learn a joint posterior approximation of all modalities [e.g., Wu and Goodman, 2018 , Shi et al., 2019 , Sutter et al., 2021] require restrictive assumptions, which lead to inferior performance. Daunhawer et al. [2022] show that aggregationbased multimodal VAEs cannot achieve the same generative quality as unimodal VAEs and struggle with learning meaningful representations depending on the relation between modalities. If we can predict one modality from another, mixture-of-experts-based posterior approximations perform best if only a single modality is given as input, while product-of-experts-based approximations excel if the full set of modalities is available. Extensions [Sutter et al., 2020 , Daunhawer et al., 2020 , Palumbo et al., 2023] have introduced modality-specific latent subspaces that lead to improved generative quality but cannot completely overcome these limitations. In contrast, the proposed MMVM method uses neither an aggregated latent space nor modality-specific latent subspaces as in previous works. It only leverages a data-dependent prior distribution to regularize the learned posterior approximations. A related line of work with different constraints is multiview VAEs [Bouchacourt et al., 2018 , Hosoya, 2018] . In contrast to multimodal VAEs, multiview VAEs often use a single encoder and decoder for all views (thereby sharing the parameter weights between views). While initial attempts also assume knowledge about the number of shared and independent generative factors, extensions [Locatello et al., 2020 , Sutter et al., 2023a,b] infer these properties during training.Role of Prior in VAE Formulations. Tomczak and Welling [2017] first incorporated data-dependent priors into VAEs by introducing the VAMP-prior. In contrast to Tomczak and Welling [2017] , who are primarily interested in better ELBO approximations, our focus is on learning better multimodal representations and overcoming the limitations faced in previous multimodal VAE works. Sutter et al. [2020] used a data-dependent prior combined with a joint posterior approximation defining a Jensen-Shannon divergence regularization based on the geometric mean. However, their work lacks a rigorous derivation and relies on the suboptimal conditional generation during training [Daunhawer et al., 2022] . Joy et al. [2021] also presented a multimodal VAE inspired by the VAMP-prior VAE. They leverage the VAMP prior to model missing modalities rather than using it as a regularization objective between multimodal samples, as we do in this work. An additional line of work [e.g., Bhattacharyya et al., 2019 , Mahajan et al., 2020] leverages normalizing flows to increase the expressivity of the multimodal prior distribution, but this sacrifices the method's scalability. 
background on multimodal vaes: 
 Problem Specification. We consider a dataset X = {X (i) } n i=1 where eachX (i) = {x (i) 1 , . . . , x (i) M } is a set of M modalities x m with latent variables z = {z (i) 1 , . . . , z (i) M }. The modalities x (i)m could represent images of the same object taken from different camera angles, text-image pairs, or-as in this paper-neuroscience data from different animal subjects with shared experimental conditions and multiple medical measurements of a patient. When contextually clear, we remove the superscript (i) to improve readability.Inspired by variational autoencoders [VAEs, Kingma and Welling, 2014] , we aim to learn an objective for representation learning while sharing information from different data modalities. For example, we would like to embed neuroscience data into a shared latent space to make brain activations comparable across subjects. At the same time, we want to avoid imposing assumptions on information sharing that are too strong to be able to take individual traits of the data modalities into account. As is typical in VAEs, this procedure involves a decoder (or likelihood) p θ (X | z), an encoder (or variational distribution) q ϕ (z | X), and a prior h(z|X) that we allow to depend on the input.Data-Dependent Prior and Objective. The VAE framework allows us to derive an ELBO-like learning objective E as followsE(X) = E q ϕ (z|X) log p θ (X | z) -log q ϕ (z | X) h(z|X) .Above, θ and ϕ denote the learnable model variational parameters. Importantly, our approach allows for an input-dependent prior h(z | X). Data-dependent priors can be justified from an empirical Bayes standpoint [Efron, 2012] and enable information sharing across data points with an intrinsic multimodal structure, as in our framework. They effectively amortize computation over many interrelated inference tasks. We stress that by making the prior data dependent, our model no longer allows for unconditional generation; however, this property can be restored by incorporating pseudo inputs [Tomczak and Welling, 2017] , hyper-priors [Sønderby et al., 2016] , or ex-post density estimation techniques [Ghosh et al., 2019] . We discuss the objective in more detail in Appendix A, where we prove that the resulting objective is upper bounded by the mean squared reconstruction error, ensuring the existence of (local) optima and thus tractable optimization.Encoder and Decoder. We now specify our encoder and decoder assumptions. A simple encoder choice relies on a single neural network encoder that expects multi-modal inputs, but this approach fails if one or more modalities are missing [Suzuki and Matsuo, 2022] . This shortcoming has motivated multiple approaches [Wu and Goodman, 2018 , Shi et al., 2019 , Sutter et al., 2021] with separate encoders q m ϕ (z m |x m )-one for each modality m-that are then aggregated in the latent space, e.g., by using a product or mixture distribution. Samples drawn from the joint distribution, e.g., q ϕ (z| X) = 1 M M m=1 q m ϕ (z | x m ), reconstruct all modalities:EQUATION)As argued and discussed in this paper, such aggregation can be overly restrictive. Instead, this paper explores a different aggregation mechanism that preserves the individual encoders and decoders for each modality. Hence, we assume independent decoders p m θ (x m |z m ) for every modality m, assuming conditional independence of each modality given their latent representation [see also Wu and Goodman, 2018 , Shi et al., 2019 , Sutter et al., 2021] .Following this assumption, we rewrite the objective E asE(X) = E q ϕ (z|X) [log h(z | X)] + M m=1 E q ϕ (zm|xm) log p m θ (x m | z m ) q m ϕ (z m | x m ).(2)Our assumptions imply that the likelihood and posterior entropy terms (the second term in Equation (2)) decouple across modalities, i.e. q ϕ (z| X) = M m=1 q m ϕ (z m | x m ) and p θ (X | z) = M m=1 p θ (x m | z m ).In contrast, the cross-entropy between the encoder and prior (the first term in Equation (2)) does not decouple and may result in information sharing across modalities. We specify further design choices in the next section. 
multimodal variational mixture vae: 
 We propose the multimodal variational mixture-of-experts prior (MMVM) VAE, a novel multimodal VAE. The main idea is to design a mixture-of-experts prior across modalities that induces a softsharing of information between modality-specific latent representations rather than hard-coding this through an aggregation approach.VAEs are an appealing model class that allows us to infer meaningful representations and preserve modality-specific information due to the reconstruction loss. Contrastive learning approaches, on the other hand, have shown impressive results on representation learning tasks related to extracting shared information between modalities by maximizing the similarity of their representations [Radford et al., 2021] . Contrastive approaches focus on the shared information between modalities, neglecting potentially useful modality-specific information. We are interested in preserving modality-specific information, which is necessary to generate missing modalities conditionally. Therefore, we leverage the idea of maximizing the similarity of representations for generative models. We propose a prior distribution that models the dependency between the different views and a new multimodal objective that encourages similarity between the unimodal posterior approximations q m ϕ (z m | x m ) using the regularization term in the objective as a "soft-alignment" without the need for an aggregation-based joint posterior approximation. We discuss objectives based on data-dependent priors in more detail in Appendix A.To this end, we define a data-dependent MMVM prior distribution in the form of a mixture-of-experts distribution of all unimodal posterior approximationsh(z | X) = M m=1 h(z m | X) where h(z m | X) = 1 M M m=1 q m ϕ (z m | x m), ∀ m ≤ M. (3)This notation implies that we use the variational distributions of all modalities m to construct a mixture distribution and then use the same mixture distribution as a prior for any modality m. Finally, we build the product distribution over the M components.Our construction of a variational mixture of posteriors is similar to the VAMP-prior of Tomczak and Welling [2017] that proposes the aggregate posterior q(z) ≡ 1N n i=1 q ϕ (z | x (i)) of a unimodal VAE as a prior. Note, however, that our approach considers mixtures in modality space and not data space. In contrast to Tomczak and Welling [2017] , our variational mixture is conditioned on a specific instance X and, therefore, does not share information across different instances X (i) ∈ X. Rather, we share information across the different modalities x (i) m ∈ X (i) within a given instance. Intuitively, we build the aggregate posterior in modality space and replicate this aggregate posterior over all modalities. We stress that this aggregate posterior differs from the standard definition as an average of variational posteriors over the empirical data distribution. Even though the prior appears factorized over the modality space, each factor still shares information across all data modalities by conditioning on the multimodal feature vector X (Equation (3)).Figure 1 graphically illustrates the behavior of the proposed MMVM VAE compared to a set of independent VAEs and an aggregation-based multimodal VAE. A set of independent VAEs (Figure 1a ) cannot share information among modalities. Aggregation-based VAEs (Figure 1b ), in contrast, enforce a shared representation between the modalities. The MMVM VAE (Figure 1c ) enables the soft-sharing of information between modalities through its input data-dependent prior h(z | X).Minimizing Jenson-Shannon Divergence. The "rate" term R in the objective, i.e., the combination of variational entropy and cross-entropy, reveals a better understanding of the effect of the mixture prior. Defining R = KL(q ϕ (z | X)||h(z|X)) where KL denotes the Kullback-Leibler divergence, the factorization in Equation (3) implies thatR = M m=1 KL q m ϕ (z m |x m )|| 1 M M m q m ϕ (z m |x m) = M • JS(q 1 ϕ (z 1 |x 1 ), . . . , q M ϕ (z M |x M )),where JS(•) is the Jensen-Shannon divergence between M distributions [Lin, 1991] . Hence, maximizing the objective E(X) of the proposed MMVM VAE is equal to minimizing M times the JS divergence between all the unimodal posterior approximations q m ϕ (z m | x m ). Minimizing the Jensen-Shannon divergence between the posterior approximations is directly related to pairwise similarities between posterior approximation distributions of positive pairs, similar to contrastive learning approaches but in a generative approach.Lemma 4.1 shows that Equation (3) is optimal in the sense that it is the unique minimizer of the cross entropy between our chosen variational distribution and an arbitrary prior. Proof. Since the cross-entropy term in Equation (2) involves an expectation over the data X and both q ϕ (z | X) and h(z | X) depend on X, we can prove the identity for a given value of X.We exploit the factorization of both the variational posterior and the prior over the modalities. Interpreting the cross-entropy between the variational distribution and prior as a functional F of the prior h, we haveF [h(z|X))] ≡ E q ϕ (z|X) [log h(z|X))] = E M m=1 q m ϕ (zm|xm) log M m=1 h(z m |X)) = M m=1 E q m ϕ (zm|xm) [log h(z m |X))] = M • E 1 M M m=1 q m ϕ (zm|x m) [log h(z m |X))] .As a result, we see that F [h(•)] is an expectation over a mixture distribution. We can solve for the optimal distribution h(•) by adding a Lagrange multiplier that enforces h(•) normalizes to one:max h(zm|X) F [h(z m | X)] + γ h(z m | X)dz m -1 = max h(zm|X) L[h, γ]To maximize the Lagrange functional L[h, γ], we compute its (functional) derivatives with respect to h(z m |X) and γ.∂L[h(z m |X), γ] ∂h(z m |X) = 1 M M m=1 q m ϕ (z m |X) h(z m |X) + γ ! = 0 ∂L[h(z m |X), γ] ∂γ = zm h(z m |X)dz m -1 ! = 0The first condition implies that for any value of z m , the ratio between the mixture distribution and the prior is constant, while the second condition demands that the prior be normalized. These conditions can only be met if the prior equals the mixture distribution, which proves the claim. 
experiments: 
 We evaluate the proposed MMVM VAE on three benchmark datasets and two challenging real-world applications. ). An optimal model would be in the top right corner.We first compare the proposed method against five strong VAE-based learning approaches on three frequently used multimodal benchmark datasets4 .Datasets. We perform benchmark experiments on the translated PolyMNIST [Daunhawer et al., 2022 , Sutter et al., 2021] , the bimodal CelebA [Sutter et al., 2020] , and the CUB image-captions [Shi et al., 2019] dataset. The translated PolyMNIST dataset uses multiple instances of the MNIST dataset [LeCun et al., 1998 ] with different backgrounds but shared digits. The digits of the different modalities are randomly translated, so we cannot predict their location across modalities. Bimodal CelebA extends the CelebA dataset [Liu et al., 2015] with an additional text modality based on the attributes describing the faces. Similarly, the CUB image-captions dataset extends the Caltech bird dataset [Wah et al., 2011] with human-generated captions describing the images. Please see Appendix B for more details regarding the datasets.Baselines. We evaluate our proposed method against a set of jointly-trained independent VAEs [independent, Kingma and Welling, 2014] , different aggregation-based multimodal VAEs, and an aggregation-based multimodal VAE with additional modality-specific latent spaces. For the set of independent VAEs, there is no interaction or regularization between the different modalities during training. For the aggregation-based multimodal VAEs, we use a multimodal VAE with a joint posterior approximation function. We evaluate four different aggregation functions: a simple averaging [AVG, Hosoya, 2018] , a product-of-experts [PoE, Wu and Goodman, 2018] , a mixture-of-experts [MoE, Shi et al., 2019] , and a mixture-of-products-of-experts [MoPoE, Sutter et al., 2021] . For the multimodal VAE with modality-specific subspaces, we use MMVAE+ method [MMVAE+, Palumbo et al., 2023] .We train all VAE methods as β-VAEs [Higgins et al., 2016] , where β is an additional hyperparameter weighting the rate term R of the VAE (see Section 4). Appendix B.4 provides the implementation details of the proposed method and the baseline alternatives.Evaluation. We test the methods' ability to infer meaningful representation when only a subset of modalities is available. In addition, we evaluate all methods in terms of their data imputation performance, where we withhold a subset of modalities at test time and conditionally generate them from the shared latent representations. In this imputation task, we assess whether the generated modalities are both of high quality and display the expected shared information, which we refer to as coherence. We assess the quality of the learned latent representations using linear classifiers trained on representations of the training set and the coherence using nonlinear classifiers trained on original samples of the training set5 . We use the reconstruction error as a proxy for how well each method learns the underlying data distribution. We assess each method by relating their achieved reconstruction error to either the learned latent representation classification or the coherence (Figure 2 ). We evaluate all methods for multiple values of β, where the average performance over multiple seeds with a single β value leads to a single point in Figure 2 . Evaluating the methods for different values of β considers that the optimal β value is model-and data-dependent. In addition, increasing β emphasizes a more structured latent space [Higgins et al., 2016] . Hence, highlighting the dynamics between reconstruction error and classification performance for different multimodal objectives provides additional insights. We chose β ∈ {2 -8 , . . . , 2 3 } on the PolyMNIST dataset, β ∈ {2 -5 , . . . , 2 4 } on the CelebA dataset, and β ∈ {2 -2 , . . . , 2 2 } on the CUB dataset. In all figures, the arrows go from small to large values of β. See Appendix B.2 for more details on the evaluation metrics.Results. Figure 2 shows that the proposed MMVM VAE consistently outperforms the other VAEbased methods (independent, AVG, MoE, PoE, MoPoE) on all datasets and both tasks. We can show that our method overcomes the limitations of aggregation-based multimodal VAEs on translated PolyMNIST described in Daunhawer et al. [2020] . Also, MMVM VAE can learn meaningful representations and generate coherent samples across different modalities while achieving high reconstruction quality for both text-image datasets bimodal CelebA and CUB image-captions. The coherent conditional generation is especially surprising as the proposed MMVM VAE decoder is never confronted with a sample from another modality. For all benchmark datasets, the proposed MMVM either achieves better latent representation classification and coherence performance with a similar reconstruction loss or lower reconstruction loss with a similar classification performance than other multimodal VAE approaches. In summary, we can show that the newly proposed MMVM VAE overcomes the limitations of previous aggregation-based approaches to multimodal learning (see Section 2) and outperforms previous works on all three benchmark datasets. We provide more results on the benchmark datasets in Appendix B.Dataset. Temporal organization is crucial to memory, affecting various perceptual, cognitive, and motor processes. While we have made progress in understanding the brain's processing of the spatial context of memories, our knowledge of their temporal structure is still very limited. To this end, neuroscientists have recorded neural activity in the hippocampus of rats performing a complex sequence memory task [Allen et al., 2016 , Shahbaba et al., 2022] . More specifically, this study investigates the temporal organization of memory and behavior by recording neural activity from the dorsal CA1 region of the hippocampus. Briefly, the task involves presenting rats with a repeated sequence of non-spatial events (four stimuli: odors A, B, C, D) at a single port [Shahbaba et al., 2022] .Since the same experimental setup was conducted across all rats, we consider the rats as different "modalities" and apply our proposed MMVM method to extract meaningful latent representations. While the existence (and importance) of subject-specific effects is well-known in neuroscience, such effects tend to be treated as unexplained variance because of the lack of the required analytical tools to extract and utilize this information properly. Results. Our proposed MMVM method outperforms6 most previous works regarding learned latent representations and conditional generation coherence. Only the MoPoE VAE achieves a classification performance comparable to the MMVM method but with a higher reconstruction loss. Figure 3a shows the separation of the latent representation (measured by the accuracy of a multinomial logistic regression classifier) against the reconstruction loss. Similar to the results on the benchmark datasets, the proposed MMVM VAE outperforms previous works by providing a clear separation of odors in the latent space while maintaining a low reconstruction loss. Figure 3b compares the coherence of conditional generation accuracy against the reconstruction loss. As before, our proposed approach outperforms the alternatives. The proposed MMVM method allows learning an aligned latent representation across different modalities. We show the 2-dimensional latent representations for every rat and four VAE encoders in Figure 4 . Each dot is the two-dimensional latent representation of a 100 ms sub-window of one odor trial for one rat and is colored according to its ground truth odor value. Figure 4 shows the odor stimuli separation on the latent space and how good MMVM VAE is in separating the odors. At the same time, two baseline models fail to extract the shared information between rats. Although it shows separation in some views, the independent model does not provide a connection between views. The five tiny clusters in Figure 4b show that, instead of showing a clear odor separation on the latent space, the AVG VAE separated the data by rats. In other words, the five rats' latent representations were far from each other, so the AVG VAE failed to connect the five views. See also Appendix B.8 for more results.Dataset. To assess the performance of our approach in a real-world setting, we evaluate the proposed MMVM method on the automated analysis of chest X-rays, a common and critical medical task.For this purpose, we use the MIMIC-CXR dataset [Johnson et al., 2019b] , a well-established and extensive collection of chest X-rays. The dataset reflects real clinical challenges with varying image quality due to technical issues, patient positioning, and obstructions. The dataset includes different views, which provide complementary information valuable for improving diagnostic [Raoof et al., 2012] . In this work, we consider frontal and lateral images as two modalities (see Appendix B.7 for further details). Each set of X-rays is labeled with different cardiopulmonary conditions, which have been automatically extracted from the associated radiology reports [Irvin et al., 2019] . This results in instances with incomplete label sets [Haque et al., 2023] , which presents a challenge for fully supervised approaches and motivates the need for self-supervised methods instead.Results. We evaluate 7 the quality of the unimodal latent representations of the MMVM VAE by comparing them with those learned by a set of jointly-trained independent VAEs (independent) as well as with representations from other multimodal VAEs that use aggregation-based approaches (AVG, MoE, MoPoE, and PoE) (see Section 5.1 for more details). We do this by training binary random forest classifiers independently for each method and all labels on the inferred representations of a subset of the training set. Table 1 shows the AUROC for these classifiers, averaged over three seeds and both unimodal representations for a subset of labels. In addition, we also report the performance of a deep nonlinear network trained in a fully supervised manner (supervised) on the same train/test split for reference purposes. Detailed experiment information can be found in Appendix B.7.3, with extensive results for each modality and label available in Table 2 and Table 3  
broader impact & limitations: 
 This paper aims to advance the field of Machine Learning by providing a natural and fundamental solution for integrating data across modalities. The proposed approach can be applied to various scientific and engineering problems with a potentially significant societal impact. In the field of neuroscience specifically, our method could allow neuroscientists to leverage individual differences in brain activity and behavior to understand basic information processing in the brain, as well as to capture distinct longitudinal changes to understand how it is affected in disease. In translational research, it could help identify subjects more susceptible to disease or potentially more responsive to treatment. While we can show that the proposed method learns better representations and generates more coherent samples, we cannot directly generate random samples anymore (see also Section 4).Although we show results on two real-world datasets, Sections 5.2 and 5.3, additional experiments on even larger scale multimodal datasets would help further evaluate the proposed method, e.g. [Damen et al., 2018 , Wu et al., 2023] . However, training and evaluating our methods on such datasets requires immense computing resources. 
discussion & conclusion: 
 In this work, we presented a new multimodal VAE, called MMVM VAE, based on a data-dependent multimodal variational mixture-of-experts prior. By focusing on a multimodal prior, the proposed MMVM VAE overcomes the limitations of previous methods with over-restrictive definitions of joint posterior approximations. The proposed objective leveraging the MMVM prior takes inspiration from contrastive learning objectives, where the goal is to minimize the similarity between positive pairs while maximizing the similarity between negative pairs [see, e.g., Chen et al., 2020 , Tian et al., 2020] . In the MMVM objective, we minimize the similarity between different modalities of the same sample (positive pairs) via the regularizing term in the objective, whereas the second part of the objective, the reconstruction loss, prevents degenerate solutions.In extensive experiments on three different benchmark datasets, we show that MMVM VAE outperforms previous works in terms of learned latent representations as well as generative quality and coherence of missing modalities. We also demonstrate its efficacy on two challenging real-world applications and show improved performance compared to previous VAEs and even a fully-supervised approach. Future research could involve studying the representation-distortion tradeoff from an information-theoretical perspective [Yang and Mandt, 2022, Yang et al., 2023]  
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We completely follow the Neurips Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the potential broader impact of the proposed work alongside its limitations in Section 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not use any pretrained language models, image generators, or scraped datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite all the papers we took inspiration from or we used code from. We include licenses for all datasets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We will publicly release our code upon acceptance and document it accordingly.We attach the code to the submission as supplementary files.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Bound on the proposed objectiveThe objective E(x m ) for a single modality m is given byEQUATION)EQUATIONEQUATIONwhere µ m (x m ) = f m (x m ) is the output of the optimized (mean) encoder f m (•) of modality m. Equation ( 5) follows from the non-negativity of the KL divergence. Without regularization term, the maximizing distribution is a delta distribution with zero variance (Equation ( 6)). Equation ( 6) equals the maximum-likelihood version of the proposed MMVM VAE (for a single modality). Put differently, the MSE of a "vanilla" autoencoder is an upper bound on the objective E(x m ) for any prior distribution p θ (z m ), hence, also for the newly introduced MMVM prior distribution h(z m | X).Let us have a look at when the KL term actually vanishes given a MMVM prior distribution h(z m | X). The KL term can only vanish if all posterior approximationEQUATIONandKL q m ϕ (z m | x m ) || h(z m | X) = 0.Another scenario is whenq m ϕ (z m | x m ) and h(z m | x m ) have disjoint modes. Hence, q m ϕ (z m | x m) is only a match to itself. In this case, we haveEQUATION= -log M + E q m ϕ (zm|xm) log q m ϕ (z m | x m ) (9) So, our objective will still be (up to a constant) the maximum likelihood objective, leading the variances to shrink to zero. In this case, the objective will reduce to the limiting independent VAE, where the modalities do not "talk" to each other and there is no multimodal alignment.In the most interesting case, there will be a non-trivial overlap between q m ϕ (z m | x m ) and h(z m | x m ), i. e. between the different unimodal posterior approximations, leading to a multimodal alignment through the soft sharing that we wish to see.In addition, Figure 5 empirically shows that the negative mean squared error (MSE) of the vanilla autoencoder (AE) upper bounds the proposed objective E. We show that lowering the β value of the regularizer R (see Section 4) approximates the negative MSE of the vanilla AE.• PolyMNIST: originally published in [Sutter et al., 2021] , downloaded the data from https: //drive.google.com/drive/folders/1lr-laYwjDq3AzalaIe9jN4shpt1wBsYM? usp=sharing and the code from https://github.com/thomassutter/MoPoE, published under the MIT license • Bimodal CelebA: originally published in [Sutter et al., 2020] , downloaded from https: //drive.google.com/drive/folders/1lr-laYwjDq3AzalaIe9jN4shpt1wBsYM? usp=sharing, licensed under the MIT license. The original CelebA dataset was published in [Liu et al., 2015] , license not found. • CUB image-captions: originally published in [Shi et al., 2019] , downloaded from http: //www.robots.ox.ac.uk/~yshi/mmdgm/datasets/cub.zip, licensed under GPL 3.0. We evaluate the different methods based on the coherence of their imputed samples, the quality of their latent representations, and their reconstruction error. We assume access to the full set of modalities during training, but we do not make this assumption at test time. Hence, there is a need for methods that can conditionally generate samples of these missing modalities, given the available modalities. In other words, we want to be able to impute missing modalities. Imputed modalities should not only be of high generative quality but also display the same shared information as the available modalities (i.e., be coherent). For example, if we generate a sample of modality x 2 based on modality x 1 from the PolyMNIST dataset (see Figure 6 ), the generated sample of modality x 2 should contain the same digit information as modality x 1 but show the background of modality x 2 .We assess the learned representations based on subsets of modalities and not the full set. The quality of the representations serves as a proxy of how useful the learned representations are for additional downstream tasks that are not part of the training objective. Hence, high-quality representations of subsets of modalities are also the basis for conditionally generating coherent samples. We assess their quality by using the classification performance of linear classifiers. We train independent classifiers on the unimodal representations of the training set and evaluate them on unimodal test set representations. To assess the learned latent representation, we train a logistic regression classifier on 10000 latent representations of the training set. The accuracy is computed on all representations of the test set.The coherence of conditionally generated samples shows how well the content of the imputed modalities aligns with the content of the available modalities in terms of the shared information. We evaluate the coherence using ResNet-based classifiers [He et al., 2016] that are trained on samples from the original training set of every modality. Using the described procedure, generated samples have to be visually similar to the original samples to have high coherence. Otherwise, the nonlinear classifier will not be able to predict digits correctly.To compute the coherence of conditionally generated samples, we train additional deep classifiers on original samples of the training set. We use a ResNet-based non-linear classifier that is trained on the full original training set. The prediction of this classifier is then used to determine the class (on PolyMNIST: digit) of the conditionally generated sample of a missing modality. The nonlinear classifier reaches an accuracy of above 98% on the original test set. For other datasets, the nonlinear classifier is trained to predict the shared information of the multimodal dataset. Hence, it serves as a good oracle for determining the digit of generated samples.Conditional Generation with the MMVM VAE and the independent VAEs To generate modality x m conditioned on modality x m , we proceed as follows:1. We encode modality x m using the encoder q m ϕ (z m | x m ) 2. We sample a latent vector z ∼ q m ϕ (z m | x m ) 3. We input the latent representation z into the decoderp m θ (x m | z) of modality x mTo compute the coherence numbers reported, we perform the above steps for every modality x m where m ∈ {1, . . . , M } and average the achieved coherence accuracies.Conditional Generation with the aggregated multimodal VAEs All aggregation-based multimodal VAEs conditionally generate samples in the same way. Hence, the following procedure applies to the multimodal VAEs used in Section 5 ([AVG, Hosoya, 2018] , [PoE, Wu and Goodman, 2018] , [MoE, Shi et al., 2019] , and [MoPoE, Sutter et al., 2021] ). To generate modality x m conditioned on modality x m , we proceed as follows:1. We encode modality x m using the encoder q m ϕ (z m | x m ) 2. We sample a latent vector z ∼ q m ϕ (z m | x m ) 3. We input the latent representation z into the decoder p m θ (x m | z) of modality x mIf we would have access to a multimodal subset X A consisting of more than one modality, i.e.|A| > 1, we would have the joint posterior approximation distribution q ϕ (z | X A ) of the subset X A , sample a latent vector from that distribution, i.e. z ∼ q ϕ (z | X A ), and generate modality x m using the decoder p m θ (x m | z) of modality x mThe reconstruction error is a proxy for how well every method learns the underlying data distribution. We evaluate the different VAE models by their achieved reconstruction error against either the learned latent representation classification or coherence (e.g., Figure 2 ). We do this for multiple values of β, where the average performance over multiple seeds with a single β value leads to a scatter point in the figures. This way, we can assess the trade-off between accurately reconstructing data samples and inferring shared information. In addition, we compute the FID [Heusel et al., 2017] values for modalities that can be summarized as natural images in the appendix. A low FID score correlates with a high-quality generated image.We use the scikit-learn [Pedregosa et al., 2011] package for the linear classifiers to evaluate the learned latent representations. All code is written using Python 3.11, PyTorch [Paszke et al., 2019] and Pytorch-Lightning [Falcon and The PyTorch Lightning team, 2019]. We base the implementations of the aggregation-based VAE methods on the official implementations. Hence, we base our implementations on the following repositories:• AVG: https://github.com/HaruoHosoya/gvae • PoE: https://github.com/mhw32/multimodal-vae-public The dataset is based on the original MNIST dataset [LeCun et al., 1998 ]. Compared to the original dataset, the digits are scaled down by a factor of 0.75 such that there is more space for the random translation. In its original form, the PolyMNIST consists of 5 different modalities. We only use the first three modalities in this work. The background of every modality x m consists of random patches of size 28 × 28 from a large image. The digit is placed at a random position of the patch. We refer to the original publication [Sutter et al., 2021] for details on the background images. Using this setup, every modality has modality-specific information given by its background image and shared information given by the digit, which is shared between all modalities. An additional difficulty compared to the original PolyMNIST is the random translation of the digits. The dataset can be found at https://github.com/thomassutter/MoPoE.We use the same network architectures for all multimodal VAEs. Every multimodal VAE consists of a ResNet-based encoder and a ResNet-based Decoder [He et al., 2016] . All modalities share the same architecture but are initialized differently. We assume Gaussian distribution for all unimodal posterior approximations, i.e.EQUATIONwhere the parameters µ m and σ m are inferred using neural networks such that we haveEQUATIONThe conditional data distributions p θ (x m | z m ) are modeled using the Laplace distribution, where the location parameter is modeled with a neural network (decoder) and the scale parameter is set to 0.75 [Shi et al., 2019] , i.e.EQUATIONwhere L(•) defines a Laplace distribution. It follows thatEQUATIONWe use the method of Hosoya [2018] for the implementation of the aggregated VAE. In this approach, a simplistic version of the joint posterior distribution is chosen where for Gaussian distribution joint posterior approximation N (µ s , σ s I) we have the following distribution parameters µ s and σ sEQUATIONµ m and σ m are the distribution parameters of the unimodal posterior approximations N (µ m , σ m I).During training and evaluation, no weight-sharing takes place, i.e. every modality has its own encoder and decoder. We use the same architectures as in [Daunhawer et al., 2022] . For all experiments on this dataset, we use an Adam optimizer [Kingma and Ba, 2014] with an initial learning rate of 0.0005, and a batch size of 256. We train all models for 500 epochs.We use NVIDIA GTX 2080 GPUs for all our runs. Each experiment can be run with 4 CPU workers and 16 GB of memory. An average run takes around 7 hours. To train all methods used in this paper, we had to train 11 × 5 × 6 = 330 different models: 11 different β values, 5 different seeds, and 6 different methods. Hence, the total GPU compute time used to generate the results for the PolyMNIST dataset is equal to 330 × 7 = 2310 hours. We-of course-also had to invest development GPU time, which we did not measure.We generated Figures 2a and 2d by plotting the classification accuracy of a linear classifier, which we trained on the learned latent representation, against the reconstruction error on the test set. For the learned latent representation, we train a classifier on the unimodal latent representations. For the aggregated VAE, this means that we train the classifier on samples of the unimodal posterior approximations and not the joint posterior approximations. Using this procedure, we test the different methods according to their performance in case of missing data, e.g., we only have access to a single modality instead of the full set at test time. For the reconstruction loss, however, we computed the error given the full set of modalities. The idea for Figures 2a and 2d is to compare the reconstruction quality (i.e., how well can we learn the data distribution?) against metrics that are related to the "generative factors" of the data and relate the different modalities to each other, i.e. the shared information of a multimodal dataset.In Figure 7 , we evaluate the performance of individual modalities in case of missing modalities.For that, we reconstruct every modality if it was the only modality available at test time. Hence, the modalities in the aggregated VAE have to be reconstructed based on the unimodal posterior approximations and not the joint posterior approximation. For the independent VAEs and the MMVM-VAE, the reconstruction of a modality is only based on its own unimodal posterior approximation. Hence, for the latter two methods, nothing changes in this setting. The performance of the learned latent representation and the coherence of generated samples are evaluated in the same way as in Figures 2a and 2d .Figure 7 shows that the reconstruction error of the aggregated VAE increases a lot if every modality needs to reconstruct itself. Interestingly, we can see that the "self-reconstruction error" (the x-axis in Figures 7a and 7b ) decreases with an increasing β-value, which is different to the other two methods and also different to the aggregated VAE's behavior in Figures 2a and 2d .In addition, we also show the conditional FID values the different multimodal VAEs achieve. Conditional FIDs come from the conditional generation of modality x m given x m . From all the conditionally generated samples of modality x m , we then compute the conditional FID values. Figure 8 shows the aggregated values of all conditional FIDs. We again show the FID values in relation to the downstream task performance and the conditional generation coherence. We see that only the proposed MMVM VAE reaches FID values similar to the ones of a set of independent VAEs. However, the set of independent VAEs cannot achieve the same latent representation classification nor coherence as the proposed MMVM VAE.We take the dataset from Sutter et al. [2020] . It extends the original CelebA dataset [Liu et al., 2015] to the multimodal setting by creating a second modality based on the attributes. In the original dataset, every image comes with a set of 40 labels, such as blond hair, smiling, etc. The difficulty of the individual attributes is not only due to their visual appearance but also their frequency in the dataset.In the multimodal extension, Sutter et al. [2021] created a text string from the attribute words. Unlike the label vector, absent attributes are dismissed in the text string instead of negated. Additionally, the attributes are randomly ordered. We use ResNet-based encoders and decoders for this experiment as well [He et al., 2016] , similar to the ones in the PolyMNIST experiment. The image encoder and decoder consist of 2D convolutions, while the text encoder and decoder consist of 1D convolutions. We use a character-level encoding and not a word or token-level encoding because of the synthetic nature of the text modality. We also use an Adam optimizer [Kingma and Ba, 2014] with a starting learning rate of 0.0002 and a batch size of 128. We train all models for 400 epochs and 3 seeds. The implementation follows the one described in Appendix B.4.We use NVIDIA GTX 2080 GPUs for all our runs. Each experiment can be run with 4 CPU workers and 16 GB of memory. An average run takes around 24 hours. To train all methods used in this paper, we had to train 10 × 3 × 6 = 180 different models: 10 different β values, 3 different seeds, and 6 different methods. Hence, the total GPU compute time used to generate the results for the PolyMNIST dataset equals 180 × 24 ≈ 4320 hours. We-of course-also had to invest GPU time to develop the method, which we did not measure.Given the multilabel nature of the CelebA dataset, we evaluate the learned latent representation with respect to the individual attributes and not only the average performance across all attributes.Figure 14 shows the detailed results according to the full set of 40 attributes for the three methods: independent VAEs, aggregated VAE, and MMVM VAE. We again train linear binary classifiers on inferred representations of the training set and evaluate them on representations of the test set. However, we now report the individual performance of every classifier. In the main text (see Figures 2b and 2e ), we report the average performance of the 40 binary classifiers.The dataset contains 11,788 images of birds and 117,880 (10 times as many) captions, each image with 10 fine-grained captions describing the bird's appearance characteristics collected through Amazon Mechanical Turk (AMT). We performed a 75-25 training-test split, with 8,855 and 2,933 images, and 88,550 and 29,330 corresponding captions in the training set and test set, respectively. We resized the images to 3 channels and 64 by 64 pixels and modeled the captions using embeddings.The labels applied to downstream tasks are based on the bird's primary colors. Based on the primary color attribute, we remapped the original 15-class primary color label set into a 6-class label set to address the sparsity issue. The labels and mappings are blue-to-red (blue, iridescent, purple, green, pink, red), brown (brown), grey (grey), yellow (yellow, olive, orange, buff), black (black), and white (white).The network structures are similar to the ones in other experiments. For both image and caption data, we employed a CNN encoder and decoder. We used a 128-dimensional latent space with a Laplace likelihood on images and a Categorical likelihood for captions. The activation function was chosen to be ReLU between layers. For all experiments on this dataset, we used an Adam optimizer with an initial learning rate of 0.001 and a batch size of 128. We trained all models for 100 epochs.The dataset we use in our experiment is a multimodal interpretation of the original MIMIC-CXR dataset. The original MIMIC-CXR dataset comprises high-resolution chest X-ray images related to imaging studies. A study may include multiple chest X-ray images captured from several view positions. We categorized these views into two primary modalities: frontal (including "AP" and "PA" views) and lateral (including "LL" and "Lateral" views). For each study, we pair every frontal image with every lateral image in all possible combinations. Studies lacking at least one frontal and one lateral image are excluded. This approach formalizes a new dataset composed of image pairs, thus offering a bimodal interpretation of the original MIMIC-CXR dataset. More rigorously, we define a dataset X = {X (i) } n i=1 where eachEQUATIONl } is a bimodal tuple composed of one frontal image and one lateral image of the same study. An image may appear in multiple tuples, but we ensure that each tuple is unique by having at least one different image. Examples of these bimodal tuples are illustrated in Figure 17 .We use the JPG version of the MIMIC-CXR dataset, namely the MIMIC-CXR-JPG dataset [Johnson et al., 2019a] . In our preprocessing pipeline, we apply center cropping and downscale the images to a resolution of 224 × 224. We utilize the labels from the MIMIC-CXR-JPG dataset which are obtained using the CheXpert tool [Irvin et al., 2019] . All non-positive labels (including "negative," "non-mentioned," or "uncertain") were combined into an aggregate "negative" label following the approach adopted by Haque et al. [2023] . Each imaging study is connected to a subject. We split the dataset into distinct training (80%), validation (10%), and test (10%) sets based on subjects, thus ensuring that the same image or study cannot be present in multiple sets.Multimodal VAEs Implementation & Training We use ResNet-based encoders and decoders for this experiment [He et al., 2016] , similar to those used in the PolyMNIST and the Bimodal CelebA experiments. The image encoder and decoder consist of 2D convolution layers. The architectural design of the image encoders and decoders is uniform for both frontal and lateral modalities. We use an Adam optimizer [Kingma and Ba, 2014] with a learning rate of 0.00005 and a batch size of 32. Different to the training, we provide only the image to every model and based on the latent representation of that image, we generate a sample. We see that the aggregated VAE (Figure 15b ) is not able to conditionally generate visually pleasing samples compared to the independent VAEs (Figure 15a ) and the MMVM VAE (Figure 15c ). We train all methods for 240 epochs and 3 seeds. The implementation follows the one described in Appendix B.4.We use NVIDIA A100-SXM4-40GB GPUs for all our runs. An average run, evaluating one method on one seed, takes approximately 45 hours. To train all methods evaluated in this experiment, we had to train 3 × 3 = 9 different models: 3 seeds and 3 methods. Hence, the total GPU compute time used to generate the VAE results for the MIMIC-CXR experiment is around 45 × 9 = 405 hours. We also had to use GPU time in the development process, which we did not measure.We use supervised classifiers based on Resnet. The classifier architecture is derived from the VAEs encoder we use in this experiment. To adapt the encoder for classification tasks, we added a linear layer equipped with 14 neurons corresponding to the number of labels in the MIMIC-CXR dataset. We use an Adam optimizer with a learning rate of 0.0001 and a batch size of 256. We train distinct models for both frontal and lateral modalities, training each for 30 epochs and 3 seeds.We train the classifiers using NVIDIA A100-SXM4-40GB GPUs. To train one classifier on one seed takes approximately 1 hour. We train a total of 3 × 2 = 6 classifiers: 3 seeds and 2 modalities, resulting in about 6 hours of GPU compute time. We also had to use GPU time in the development process, which we did not measure.In our main experiment, we evaluate the quality of the learned latent representations using Random Forest (RF) classifiers. Specifically, we train independent RF binary classifiers for each model and each label on the inferred representations of the training set and evaluate them on the representations of the test set. The RF classifiers are configured with 5,000 estimators and a maximum depth of 30.In Table 1 in Section 5.3, we report for each model the average performance over the two modalities and three seeds, totaling six RF classifiers per model: one for each modality and each of the three different seeds.Here, we provide detailed insights into the capabilities of the models to leverage different modalities during training to improve the unimodal representations by reporting the performance of the latent representation classification for each modality separately in Table 2 and Table 3 . These results allow us to highlight the strengths of the MMVM model in improving unimodal representations with additional modalities. For instance, in the main text we mentioned that in the Cardiomegaly classification task, the MMVM lateral representations z L lead to substantial improvements over the frontal representations z F of the other VAEs, even though the former modality x L is less informative than the latter x F as indicated by the respective performance of the supervised model on each modality. This behavior is not specific to Cardiomegaly and can be observed for other labels such as Atelectasis, Lung Opacity, No Finding, Pleural Effusion, Pneumonia, and Support Devices. Even though the difference between modalities for these labels is sometimes small, the consistency of those results demonstrate the soft-sharing capabilities of the MMVM approach. Conversely, in tasks where the frontal view is less informative, such as Consolidation, Enlarged Cardiomediastinum, and Pneumothorax, the MMVM's lateral representations outperform the frontal ones of the other VAEs.The efficacy of the soft sharing mechanism in the MMVM VAE is also reflected in the average performance for all labels. The performance difference between the two unimodal latent representations produced by the MMVM VAE is substantially smaller (0.9 percentage points on average) compared to the independent VAEs (2.5 percentage points on average) and the PoE VAE (2.2 percentage points on average). The training data was collected from 250 ms length time frames after the port entry. Due to the behavior difference from each rat (some rats react faster to the odors while some others react slower), the training time frames of the five rats started from 250 ms, 250 ms, 500 ms, 500 ms, and 250 ms, respectively. During training, we treated each data point as independent and trained all the VAE models based on sliding windows (100 ms sub-window, 10 ms steps; 16 data points per window on each trial). The 100 ms sub-windows constituted the input data, with the dimension equal to the rat's number of neurons multiplied by 10, as the data was further binned into 10-ms increments.We use the same network architectures for all multimodal VAEs. Each of the autoencoders includes its unique encoder and decoder, both containing two hidden layers, without weight-sharing during training and evaluation. All modalities share the same architecture but the layers' dimensions are different, with 920, 790, 1040, 490, 460 dimensional input and hidden layers, respectively. The activation function was chosen to be LeakyReLU with a 0.01 negative slope. For all experiments on this dataset, we use an Adam optimizer with an initial learning rate of 0.001, a batch size of 128. We train all models for 1000 epochs.We show the 2-dimensional latent representations for every rat and the six VAE encoders in Figure 4 and Figure 20 . In these two figures, each dot is the two-dimensional latent representation of a 100 ms sub-window of one odor trial for one rat. Figure 4 is colored by 4 odors, and Figure 20 is colored by 5 modalities (rats). Figure 4 shows the odor stimuli separation on the latent space and how good MMVM VAE is in separating the odors. Figure 20 shows that the proposed MMVM VAE can best utilize the shared information between the five rats by pulling the latent representations together. At the same time, the independent AVG and PoE baseline models fail to extract the information shared between rats. Although it shows separation in some views, the independent model does not provide a connection between views. The five tiny clusters in Figure 4 and Figure 20 show that, instead of showing a clear odor separation on the latent space, the AVG model separated the data by rats. The results went against the intention to share information across views. In other words, the five rats' latent representations were far away from each other, so the aggregated VAE completely failed to connect the five views. Justification: We report the compute time for the experiments conducted (including a statement that more compute time was needed for the development) and the used GPU models.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).There are 2 M -1 different subsets for a dataset of M modalities and, hence, paths for any-to-any mappings.Sutter et al. [2021] describe how different implementations of joint posterior multimodal VAEs relate to different abstract mean definitions.The code for the experiments on the benchmark datasets can be found here: https://github.com/ thomassutter/mmvmvae.We use the same evaluation procedure as in previous work [e.g.,Shi et al., 2019, Sutter et al., 2021]The code for the hippocampal neural activity experiments can be found here: https://github.com/ yangmeng96/mmvmvae-hippocampalThe code for the MIMIC-CXR experiments can be found here: https://github.com/agostini335/ mmvmvae-mimic 


Decision: The reviewers acknowledge the soundness and the technical novelty of the paper. The rebuttal has a positive impact on the final ratings and all the reviewers scored weak accept or higher. The ACs followed the reviewers' recommendation.

Reviews: review: paper_summary: paper_summary: In this work, the authors proposed a new multimodal VAE architecture which, quickly summarized, both encoders and decoders factorized across modality latent variables, and the prior for each modality latent, $z_m$, is a mixture-of-experts composed of all the encoders. With this probabilistic model choice, the authors derive a number of interesting results, and show that this choice of prior is optimal to optimize the corresponding ELBO. Then, the authors empirically validate their findings by comparing their approach with existing methods in the PolyMNIST, Bimodal CelebA, and CUB datasets. Finally, the authors evaluate the quality of the latent samples on the Hippocampal Neural Activities dataset, showing that it can separate different odor types within the data.
strength_weakness: strength_weakness: - **W1.** The experiments should compare with more modern approaches like [MMVAE+](https://openreview.net/forum?id=sdQGxouELX).
- **W2.** The presentation could be improved, there is a clear lack of derivations, and some details are hard to grasp (it took me several reads to fully understanding the proposed probabilistic model).
- **W3.** There are plenty of (to my eyes) over-statements and vague statements, depending on what the text requires. E.g.:
  - Line 68. "Require restrictive assumptions" Which ones? Why are they restrictive?
  - Lines 72-73. "cannot ... and require stronger assumptions". Why? Which ones?
  - Line 52: "thereby providing potential insights into the neural mechanisms underlying memory impairment"
  - Line 125: "As argued and discussed in this paper, such aggregation can be overly restrictive."
  - Line 183: "is directly related to pairwise similarities between posterior approximation distributions of positive pairs, similar to contrastive learning approaches but in a generative approach" 
- **W4.** Similarly, conclusions from the experiments look over-optimistic to me. It is true that the proposed method work as- or better than the baselines, but the results are pretty close (see Fig 3) without standard deviations. Also, saying that the samples in Fig. 4d are separated seems like an over-stretch to me (I'd at least add transparency to the points).


PS: I'll happily increase my score once a newer method like MMVAE+ is included, which I find completely necessary given the scope of the paper.
questions: questions: - **Q1.**  Why only independent and PoE were selected for the experiments in Table 1? I understand that MMVAE should have similar complexity to PoE.
limitations: limitations: Limitations are properly discussed.
paper_summary: paper_summary: The paper introduces the multimodal variational mixture-of-experts prior (MMVM) VAE, a new approach to enhance representation learning for multimodal Variational Autoencoders (VAEs). The MMVM VAE overcomes the limitations of the alternatives that parameterize joint posteriors with restrictive assumptions by using a data-dependent prior that softly guides each modality’s latent representation towards a shared aggregate posterior, instead of imposing hard constraints. The experimental analysis on benchmarks and real datasets assesses the quality of the latent representations and preservation of modality-specific information.
strength_weakness: strength_weakness: - The experimental analysis presented in the main text is not fully convincing. While the evaluation is properly conducted on several datasets, the performance is assessed with only a single quantitative experiment in each case. The extensive analysis provided in the Appendix could be referenced in Section 5 to better support and strengthen the discussion.
- The paper would benefit from a more specific analysis of the limitations of related work in multimodal VAEs. This would enhance the motivation and context for the proposed approach.
- The paper does not explore potential solutions to mitigate the inability to perform unconditional sampling. Addressing this limitation would significantly improve the overall contribution and applicability of the work.
questions: questions: **Major concerns**
- I found that Figures 2 and 3 are unclear in demonstrating the benefits of the proposed models due to the following reasons:
   - Showing several values of $\beta$ appears irrelevant. Fixing $\beta = 1$ would likely provide a fairer comparison and clearer results.
   - In representation learning literature, assessing meaningful representations is a challenging task with many proposed metrics and evaluations. Why do predicting metrics on linear/non-linear classifiers assess the quality of latent representations and coherence, respectively? Without references or explanations for these metrics, their validity in supporting the claims remains unclear.

**Minor concerns**

- I would move the “Encoder and Decoder” paragraph from Section 3 to the beginning of Section 4.

**Found typos**

- Line 98: “patient , .” $\rightarrow$ “patient.”.
limitations: limitations: The primary limitation of this work is its inability to perform unconditional sampling, which the authors acknowledge and discuss appropriately. However, while they mention several potential solutions to address this issue, they do not explore any of these alternatives. Efficient unconditional and conditional sampling are among the key advantages of VAEs. Incorporating a solution for unconditional sampling would significantly enhance the paper, allowing the model to generate unconditional samples and fully leverage the strengths of VAEs.
paper_summary: paper_summary: This paper proposes a multimodal variational mixture-of-experts prior (MMVM) VAE to address challenges in learning representations of multimodal data. Instead of modeling the latent representation as a joint posterior, this work introduces a multimodal data-dependent prior to regulate the latent representation learned from the VAE for each modality. Experiments on multiple benchmarks demonstrate that the proposed method is promising.
strength_weakness: strength_weakness: (1) There is no direct evaluation of the learned representation and the data-dependent prior to support the "soft-sharing of information between modalities" proposed by this work. For example, a naive investigation could examine the differences and overlaps among each $H(z_m|𝑋)$, and how similar each learned $q^{m}_{\phi} (z_m|x_m)$ is to their prior.

(2) In experiments, the exclusion of two multimodal VAE baselines, Daunhawer et al., 2020 and Palumbo et al., 2023, which are mentioned in the related work section, weakens the evaluations of the proposed work.
questions: questions: (1) Regarding the metrics and classifiers mentioned in lines 233-235, are they the same as those defined in [1] and [2]? If yes, please cite them. If not, please provide a more detailed description.

[1] Shi, Y., Siddharth, N., Paige, B., Torr, P.: Variational mixture-of-experts autoencoders for multi-modal deep generative models. In: Advances in Neural Information Processing Systems (2019)

[2] I. Daunhawer, T. M. Sutter, K. Chin-Cheong, E. Palumbo, and J. E. Vogt. On the Limitations of Multimodal VAEs. International Conference on Learning Representations, 2022.
limitations: limitations: N/A

Comments: title: Rebuttal feedback
comment: Dear reviewer eP7z,

Thank you for your response and your recommendation of acceptance.

We agree that a more thorough discussion of our results is valuable and will improve the paper. We will add this to the camera-ready version of our paper.
With that said, it would be highly appreciated if your recommendation of acceptance would be reflected in an increased score. 

Thanks again for the valuable discussion, we do not take it for granted and it is appreciated.
Dear Authors,

Thank you for your response. I understand that highlighting this dynamic or trade-off can be valuable. However, in my opinion, a more thorough discussion of the plot and a clearer visual justification of why the proposed method is superior are necessary. In the original submission, the discussion was somewhat unclear.

With this point addressed, my concerns are mostly resolved, and I will recommend acceptance.
Dear reviewer 56cj,

thank you for appreciating our effort and updating your score.

We will add a probabilistic model to the updated manuscript and improve the clarity of the derivations.

Please let us know if there are more points that are unclear or could be improved.
Dear reviewer 6TPP,

thank your for appreciating our rebuttal effort and updating your score.

Please let us know if other points remain unclear or could be improved.
Dear reviewer eP7z,

thank you for your detailed reply. We appreciate your effort.

We agree that the $\beta$-parameter has several effects on the learned model and its interpretation may vary depending on the context.

From our experiments, please see the rebuttal PDF or the experiments in the submitted manuscript, we could clearly see that training several multimodal VAE implements with multiple $\beta$ values resulted in models that seem to trade off the classification accuracy and the reconstruction loss in different ways. We can see that high classification accuracy does not necessarily result in low reconstruction error on different datasets and that different methods seem to have different "$\beta$-dynamics. As you pointed out, the optimal model depends on the context and from our experiments, we can see that it is not always straightforward to choose the best model according to latent representation classification accuracy *and* reconstruction error. 

Showing this dynamic or trade off is the reason behind our evaluation with multiple $\beta$ values. However, we agree and thank you for it that highlight the models where $\beta=1$ improves our plots. We will formulate this more clearly and rephrase the mentioned sentence in the updated version our manuscript.

Please let us know if additional points remain unclear.
Dear authors,

Thank you for the detailed response. I am mostly satisfied with the answers provided to mine and the other reviews, and I will thus update my score to reflect this.

As of the presentation, my main suggestion is to have clearly in a single spot of the main text a couple equations clearly stating the probabilistic model (i.e., likelihood, prior, and posterior).
I appreciate the authors' efforts in the detailed response. My concerns have been addressed, and I have increased my score.
Dear authors,

Thank you for addressing my concerns, specifically by strengthening the motivation, highlighting the results for $\beta=1$, adding more references to support the chosen metrics, and incorporating additional references to highlight the limitations of previous works. However, I still have a few concerns:

- Regarding the experimental material in the Appendix, my suggestion was not to move a specific result to the main text but rather to incorporate references to the Appendix material within the main analysis and discussion. This could enhance the overall analysis and discussion, thereby strengthening the paper.

- While I acknowledge that showing results for varying $\beta$ can be insightful, without sufficient motivation, it became difficult for me to visually understand the gains of the proposed method. Why does “Figure 3 clearly show […] without trading off reconstruction quality?” Perhaps I am missing something—could you please clarify this point? 

- Additionally, including more references to other works that have performed similar analyses on $\beta$-grids could better ground the evaluations.


I strongly disagree with your statement about the $\beta$-VAE. The phrase “trading off the learned representation against the reconstruction quality” is vague and imprecise, and it is repeated several times throughout the paper. I do not believe that “learned representation” is a well-defined measure in itself, especially considering the extensive work and effort from the community in this area.

The $\beta$ factor primarily controls the latent overlap [1], and under certain circumstances [2], it can favor latent disentanglement, i.e., representations aligned with the underlying generative factors of variation in the data. This might be a desirable property or not, depending on the context—such as when accurate reconstruction is prioritized, as in [3].

That said, I appreciate the inclusion of results for $\beta$=1, as it helped to more clearly highlight the contributions, and I thank the authors for making this adjustment.

[1] Mathieu, E., Rainforth, T., Siddharth, N., & Teh, Y. W. (2019, May). Disentangling disentanglement in variational autoencoders. In International conference on machine learning (pp. 4402-4412). PMLR.

[2] Burgess, C. P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., & Lerchner, A. (2018). Understanding disentangling in $\beta $-VAE. arXiv preprint arXiv:1804.03599.

[3] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).
Dear reviewer 6TPP,

please let us know if we could address your concerns with our rebuttal or if there are other question remaining.
Dear reviewer eP7z,

please let us know if we could address your concerns with our rebuttal or if there are other question remaining.
Dear reviewer 56cj,

please let us know if we could address your concerns with our rebuttal or if there are other question remaining.