Unity by Diversity: Improved Representation Learning for Multimodal VAEs
Authors: Thomas M. Sutter
Yang Meng
Andrea Agostini
Daphné Chopard
Norbert Fortin
Julia E Vogt
Babak Shahbaba
Stephan Mandt
Venue: NeurIPS 2024 poster (2024)
Keywords: multimodal generative learning
VAE
representation learning
data-dependent prior
mimic-cxr
Primary area: generative_models
TL;DR: We propose a novel VAE that learns from multimodal data using a mixture-of-experts prior for aggregation

Abstract: Variational Autoencoders for multimodal data hold promise for many tasks in data analysis, such as representation learning, conditional generation, and imputation.
Current architectures either share the encoder output, decoder input, or both across modalities to learn a shared representation. 
Such architectures impose hard constraints on the model. 
In this work, we show that a better latent representation can be obtained by replacing these hard constraints with a soft constraint. We propose a new mixture-of-experts prior, softly guiding each modality's latent representation towards a shared aggregate posterior.
This approach results in a superior latent representation and allows each encoding to preserve information better from its uncompressed original features. In extensive experiments on multiple benchmark datasets and two challenging real-world datasets, we show improved learned latent representations and imputation of missing data modalities compared to existing methods.

Unity by Diversity: Improved Representation Learning for Multimodal VAEs
Thomas M. Sutter', Yang Meng Andrea Agostini', Daphné Chopard 1.2 Norbert Fortin Julia E. Vogt", Babak Shahbaba', Stephan Mandt
Department of Computer Science, ETH Zurich
2 Department of Intensive Care and Neonatology, University Children's Hospital Zurich DDeparmment of Statistics, UC Irvine
Department of Neurobiology and Behavior, UC Irvine Department of Computer Science, UC Irvine
Abstract
Variational Autoencoders for multimodal data hold promise for many tasks data analysis, such as representation learning, conditional generation, and imputation. Current architectures either share the encoder output, decoder input, or both across modalities to learn a shared representation. Such architectures impose hard con- straints on the model. In this work, we show that a better latent representation can be obtained by replacing these hard constraints with soft constraint. We propose a new mixture-of ofpexterss prior, softly guiding each modality's latent representation towards shared aggregate posterior. This approach results in superior latent representation and allows each encoding to preserve information better from its uncompressed original features. In extensive experiments multiple benchmark datasets and two challenging real-world datasets, we show improved learned latent representations and imputation of missing data modalities compared existing methods.
Introduction
The fusion of diverse modalities and data types is transforming our understanding of complex phenomena, enabling more nuanced and comprehensive insights through the integration of varied information sources. Consider, for instance, the role of a medical practitioner who synthesizes multiple tests and measurements during diagnosis and treatment. This process involves merging shared information across different tests and identifying test-specific details, both of which are critical for optimal patient care and medical secision-making
Among the existing methods, multimodal Variational Autoencoders (VAEs) have emerged as promising approach for jointly modeling and learning from weakly-supervised heterogeneous data sources. While scalable multimodal VAEs utilizing single shared latent space efficiently handle multiple modalities [Wu and Goodman, 2018, Shi al., 2019, Sutter al., 2021], finding an optimal method to aggregate these modalities remains challenging. The aggregation methods and resulting joint representations are often suboptimal and overly restrictive [Daunhawer al., 2022, Sutter, 2023], leading to inferior latent representations and generative quality. This trade-off between shared and moodality-specifii information the latent representations multimodal VAEs results limited quality coherence in generated samples, even in relatively simple scenarios.
Corresponding author: thomas. sutterrinf ethz.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Mr (X0
xi
E
bal (xx)
Di
+
x-
Fi
16/2 [x))
Agonogotion
D:
+3
&
*alEe xx)
1g: xx)
[x)
a,t xx)
(a) Independent VAEs
(b) Aggregated VAE
(c) MMVM VAB
Figure/ 1: Independent VAEs (Figure Ia) provide reconstructions for individual modalities but lack information sharing across modalities. Multimodal VAEs with joint posterior approximation (Figure lb) aggregate unimodal posteriors into a joint posterior but may incur poor reconstruction quality. Our proposed MMVM VAE (Figure Ic) enhances independent VAEs with data-dependent prior, h(z X), allowing soft-sharing of information between modalities while preserving modality- specific reconstructions.
In this work, we propose novel multimodal VAF, termed the multimodal variational mixture-of- experts prior (MMVM) VAE, to overcome the aforementioned limitations. Instead of modeling the dependencies between different modalities through a joint posterior approximation, we introduce multimodal and data-dependent prior distribution (see Figure 1). Our proposed multimodal objective is inspired by the VAMP-prior formulation introduced by Tomczak and Welling [2017]. which is traditionally used to learn an optimal prior distribution between unimodal data samples, whereas aim for optimal prior between different modalities of the same data sample. The resulting regularization term in the VAE objective can be interpreted as minimizing the distance between positive pairs, similar contrastive learning methods [Oord et al., 2019, Tian et al., 2020]; see Section 4 for details.
We demonstrate the superior performance of the MMVM VAE on three multimodal benchmark datasets, comparing it to unimodal VAEs and multimodal VAEs with joint posterior approximations. Our evaluation focuses on the generative coherence and the quality of the learned latent represen- tations. While independent unimodal VAES fail to leverage additional modalities during training, they avoid multimodal aggregation disturbances in data reconstruction. On the other hand, multi- modal VAEs with a joint posterior approximation must combine both shared and codality-speciffi information. Previous work by Daunhawer et al. [2022] has shown that this approach results in trade-off between reconstruction quality and learned latent representation. In contrast, the MMVM VAE accurately reconstructs all modalities and learns meaningful latent representations.
In more practical settings, we address two challenging tasks from the neuroscience and medical do- main. First, we analyze hippocampal neural activities from multiple subjects in memory experiment [Allen et al., 2016]. By treating each subject as a modality, our MMVM VAE enables the description of underlying neural patterns shared across subjects while quantifying individual differences in brain activity and behavior, thereby providing potential insights into the neural mechanisms underlying memory impairment. Second, tackle identifying cardiopulmonary diseases from chest radiographs using the MIMIC-CXR dataset [Johnson et al., 2019b]. which reflects real-world conditions with images of varying quality. By leveraging both frontal and lateral X-ray views as distinct modalities, our MMVM method learns representations that consistently improve disease classification compared to existing VAEs.
This paper advances multimodal machine learning by providing a robust framework for integrating diverse data types and improving the quality of learned representations and generative models.
2 Related Work
Multimodal Learning. While there is long line of research on multimodal machine learning (ML) [Baltrušaitis et al., 2018, Liang et al., 2022], multimodal generative ML has gained additional attraction in recent years [Manduchi et al., 2024], driven by impressive results in text-to-image generation [Ramesh et al., 2021, 2022, Saharia et al., 2022]. Unlike these methods, we focus on scalable methods that are designed for large number of modalities to generate any modality from any other modality without having to train a prohibitive number of different models?
There are 2M 1 different subsets for dataset of M modalities and, hence, paths for any-to-an mappings.

---

Multimodal Vs Scalable multimodal VAEs using joint posterior approximation are based on aggregation in the latent space'. Multimodal VAEs that learn a joint posterior approximation of all modalities [e.g.. Wu and Goodman, 2018, 8, Shi et al., 2019, Sutter et al., 2021] require restrictive assumptions, which lead to inferior performance. Daunhawer et al. [2022] show that aggregation- based multimodal VAEs cannot achieve the same generative quality as unimodal VAEs and struggle with learning meaningful representations depending on the relation between modalities. we can predict one modality from another, extture-o--xppert--bseeeee only a single modality is given as input, while product-of-expeets-bbeee approximations excel if the full set of modalities is available. Extensions [Sutter et al., 2020, Daunhawer et al., 2020, Palumbo et al., 2023] have introduced modality speciifi latent subspaces that lead to improved generative quality but cannot completely overcome these limitations. In contrast, the proposed MMVM method uses neither an aggregated latent space nor modality-specifie latent subspaces as in previous works. It only leverages data-dependent prior distribution to regularize the learned posterior approximations. related line work with different constraints is multiview VAEs [Bouchacourt et al., 2018, Hosoya, 2018]. In contrast multimodal VAEs, multiview VAEs often use single encoder and decoder for all views (thereby sharing the parameter weights between views). While initial attempts also assume knowledge about the number of shared and independent generative factors, extensions [Locatello et al., 2020, Sutter et al., 2023a,b] infer these properties during training.
posterior approximations perform best
Role of Prior in VAE Formulations. Tomezak and Welling [2017] first incorporated data-dependent priors into VAEs by introducing the VAMP- prior. In contrast to Tomezak and Welling [2017], who primarily interested in better ELBO approximations, our focus is learning better multimodal representations and overcoming the limitations faced in previous multimodal VAE works. Sutter et al. [2020] used data-dependent prior combined with joint posterior approximation defining a Jensen- Shannon divergence regularization based on the geometric mean. However, their work lacks a rigorous derivation and relies on the suboptimal conditional generation during training [Daunhawer et al., 2022]. Joy et al. [2021] also presented a multimodal VAE inspired by the VAMP-prior VAE. They leverage the VAMP prior to model missing modalities rather than using it regularization objective between multimodal samples, as we do in this work. An additional line of work [e.g.. Bhattacharyya et al., 2019, Mahajan et al., 2020] leverages normalizing flows to increase the expressivity of the multimodal prior distribution, but this sacrifices the method's scalability.
3 Background on Multimodal VAEs
Problem Specification. We consider a dataset X = (x (i) 1 where each (0) {a
wr
(4)
is a set of M modalities Imn with latent variables = == = [z
(1) The modalities am could M
represent images of the same object taken from different camera angles, text-image pairs, or-as in this paper- neuroscicee data from different animal subjects with shared experimental conditions and multiple medical measurements of a patient. When contextually clear, we remove the superscript (i) to improve readability.
Inspired by variational autoencoders [VAEs, Kingma and Welling, 2014], we aim learn an objective for representation learning while sharing information from different data modalities. For example, we would like embed neuroscience data into a shared latent space to make brain activations comparable across subjects. At the same time, we want to avoid imposing assumptions on information sharing that are too strong to be able take individual traits of the data modalities into account. As is typical in VAES, this procedure involves decoder (or likelihood) (x z), an encoder (or variational distribution) g X), and prior h((z) X) that we allow to depend on the input.
Data-Dependent Prior and Objective. The VAE framework allows us to derive an ELBO-like learning objective & as follows
E E(X) = Egs() bogpo((X (X x) - log g = XX) h( X)
Above, 0 and & denote the learnable model variational parameters. Importantly, our approach allows for an input-dependent prior h(z X). Data-dependent priors can be justified from an empirical Bayes standpoint [Efron, 2012] and enable information sharing across data points with
SSutter et al. [2021] describe how different implementations of joint posterior multimodal VAEs relate to different abstract mean definitions.
3

---

an intrinsic multimodal structure, as in our framework. They effectively amortize computation over many interrelated inference tasks. We stress that by making the prior data dependent, our model no longer allows for unconditional generation: however, this property can be restored by incorporating pseudo inputs [Tomezak and Welling, 2017], hyper-priors [Sønderby et al., 2016], or ex-post density estimation techniques [Ghoss et al., 2019]. We discuss the objective in more detail in Appendix A, where we prove that the resulting objective is upper bounded by the mean squared reconstruction error, ensuring the existence of (local) optima and thus tractable optimization.
Encoder and Decoder. We now specify our encoder and decoder assumptions. simple encoder choice relies on single neural network encoder that expects multi-modal inputs, but this approach fails if one or more modalities are missing [Suzuki and Matsuo, 2022]. This shortcoming has motivated multiple approaches [Wu and Goodman, 2018, Shi et al., 2019, Sutter et al.. 2021] ]] with separate encoders 9- (zm mnn for each modality m-thht are then aggregated in the latent space, e.g., by using product mixture distribution. Samples drawn from the joint distribution, e.g., (= egg.4s((
x) M EM
46 (z (m)). reconstruct all modalities:
Pg
22)
E(X) (x)() [logh(z) X)] + (=)(x)
log
(I)
(z X)
As argued and discussed in this paper, such aggregation can be overly restrictive. Instead, this paper explores different aggregation mechanism that preserves the individual encoders and decoders for each modality. Hence, we assume independent decoders P/ (æml (m)) for every modality m, assuming conditional independence of each modality given their latent representation [see also Wu and Goodman, 2018, Shi et al., 2019, Sutter et al., 2021].
Following this assumption, we rewrite the objective & as
M
p6 (Im mm) 9% (zm
((X) = (= [logh(z h(z X)]
+
Σ m==1
E (= (mm))
log
(2)
3mm)
Our assumptions imply that the likelihood and posterior entropy terms (the second term in Equa-
tion (2)) decouple across modalities, i.e. (a(2 X) = I g (zm (m) and po [ =)
M Im= Pe(2m Zm) In contrast, the cross-entropy between the encoder and prior (the first term in Equation (2)) does not decouple and may result in information sharing across modalities. We specify
further design choices in the next section.
4 Multimodal Variational Mixture VAE
We propose the multimodal variational mixture-of-experts prior (MMVM) VAF, novel multimodal VAE. The main idea to design mixture-of-eeppers prior across modalities that induces soft- sharing of information between modality-speciffi latent representations rather than hard-coding this through an aggregation approach.
VAES are an appealing model class that allows us to infer meaningful representations and preserve modality-specine information due to the reconstruction loss. Contrastive learning approaches, the other hand, have shown impressive results on representation learning tasks related to extracting shared information between modalities by maximizing the similarity of their representations [Radford et al., 2021]. Contrastive approaches focus on the shared information between modalities, neglecting potentially useful modality-spechie information. We interested in preserving modality-specine information, which is necessary to generate missing modalities conditionally.
Therefore, we leverage the idea of maximizing the similarity of representations for generative models. We propose a prior distribution that models the dependency between the different views and a new multimodal objective that encourages similarity between the unimodal posterior approximations 18 (zm Zmm) using the regularization term in the objective as a "soft-alignment" without the need for an aggregation-basse -based joint posterior approximation. We discuss objectives based on data-dependent priors in more detail in Appendix A.

---

To this end, we define data-dependent MMVM prior distribution in the form of a mixture-of-expprn distribution all unimodal posterior approximations
h(2
X
II m==1
h(z..
X)
where
hlZm
X)
1 M
gø 2mm
Friel, V/ S M. (3)
m=1
This notation implies that we use the variational distributions of all modalities m to construct mixture distribution and then use the same mixture distribution prior for any modality Finally, we build the product distribution over the M components.
Our construction of variational mixture of posteriors is similar to the VAMP-prior of Tomczak and Welling [2017] that proposes the aggregate posterior ((z) = * > 4s x(()) of unimodal
VAE prior. Note, however, that our approach considers mixtures modality space and not data space. In contrast to Tomczak and Welling [2017], our variational mixture is conditioned on specific instance X and, therefore, does not share information across different instances (4) E X. Rather, we share information across the different modalities I' E X (i) within given instance. Intuitively, we build the aggregate posterior in modality space and replicate this aggregate posterior over all modalities. We stress that this aggregate posterior differs from the standard definition an average of variational posteriors over the empirical data distribution. Even though the prior appears factorized over the modality space, each factor still shares information across all data modalities by conditioning on the multimodal feature vector X (Equation (3)).
Figure 1 graphically illustrates the behavior of the proposed MMVM VAE compared to set of independent VAEs and an aggregation- based multimodal VAE. A set independent VAEs (Figure la) cannot share information among modalities, Aggregation-baser VAEs (Figure lb), in contrast, enforce shared representation between the modalities. The MMVM VAE (Figure Ic) enables the soft-sharing of information between modalities through its input data-dependent prior h(2 X).
Minimizing Jenson-Shannon Divergence. The "rate" term R in the objective, i.e., the combination of variational entropy and cross-entropy, reveals better understanding of the effect of the mixture prior. Defining R = K L= [X] [/[]]] X)) where K L denotes the Kullback-Leibler divergence, the factorization in Equation (3) implies that
M Σ m=1
M
R
KL
(Z (mmm/mmm]]
1
Σ
Zm [mm)
MJJ ... (zM (xM)).
M
where JS(-) is the imizing the objective {(X) of the proposed MMVM VAE is equal to minimizing M times the IS divergence between all the unimodal posterior approximations (2m #mn). Minimizing the Jensen-Sihe -hasennShannon divergence between the posterior approximations is directly related to pairwise similarities between posterior approximation distributions of positive pairs, similar to contrastive learning approaches but in generative approach.
divergence between M distributions [Lin, 1991]. Hence, max-
4.1 Optimality of the MMVM Prior
Lemma 4.1 shows that Equation (3) is optimal in the sense that it is the unique minimizer of the cross entropy between our chosen variational distribution and an arbitrary prior.
Lemma 4.1. The expectation on the right-hand side of Equation (2) is maximized when for each m E {1.... M, the prior hl ()) ) is equal to the aggregated posterior of a multimodal sample given on the first line of Equation (3).
Proof. Since the cross-entropy term in Equation (2) involves an expectation over the data X and both (=(2 X) and h(z X) depend X, we can prove the identity for given value of X.
We exploit the factorization of both the variational posterior and the prior over the modalities. Interpreting the cross-entropy between the variational distribution and prior functional F of the
5

---

independent
AVG
MoE
PoE
MoPoE
MMVAE+
MMVM
1.0
0.6
0.7
Average Prection
Averago Precison
0.6
0.5
0.5
6.2
0.2
0.1
03-
0.0
50.00
4,000
8000
3500
3000
20000
15000
10000
03
20000
15000
10000
Reconstruction
Reconstruction
Reconstruction
(a) LR: translated PoyMNIST
(b) Bimodal CelebA
(c) CUB
0.6
0.5
Avergee Precison
Average Prockion
Q4
0.3
0.3
0.2
0.2
0.2
8000
4,500
8000
3500
3000
20000
15000
I0600
20000
15000
10000
Reconstruction Error
Reconstruction Error
Reconstruction Errrr
Coh: translated POYMNIST
(e) Coh: Bimodal CelebA
(f) Coh: CUB
Figure 2: Results on the benchmark datasets translated PolyMNIST, bimodal CelebA, and CUB. An optimal model would be in the top right corner with low reconstruction error and high classification performance. The proposed MMVM method either achieves higher classification performance, latent representation (LR, Figures 2a 2c) or coherence of generated samples (Coh, Figures 2d 2f), with the same reconstruction loss or the same classification performance with lower reconstruction loss. Every point averages runs over multiple seeds and specific value (see Section 5.
prior h, we have
M log II h(zm X))
[[](2X))) = [xx(= [log hz|[X))] = -BIIM = o% (2m (m)))
M
Σ (x (m))
loog h( X)) M E 37 EM o3 (xm/am)) log 1/21/)220 ((2m) [))]
As result, we see that F(h(-) ((())]] is an expectation over a mixture distribution. We can solve for optimal distribution h(-)) by adding a Lagrange multiplier that enforces h(--) normalizes one:
Fl x)] + ( hh(zm X ))dzm - -1)
maax ( (
max Chh, 7
M(#m XX)
To maximize the Lagrange functional LTh, 7), we compute its (functional) derivatives with respect to f(( .m ()) and y.
(c
X),7]
ΣM
mm=1 9% (zm
(X)
+ _
ah(zm! X)
h h(zmml X
achh(zm X),7l
h(zm X)dZz
1 -
Ory
Sm
The first condition implies that for any value of Zm the ratio between the mixture distribution and the prior is constant, while the second condition demands that the prior be normalized. These conditions can only be met if the prior equals the mixture distribution, which proves the claim.
Experiments
We evaluate the proposed MMVM VAE three benchmark datasets and two challenging real-world applications.

---

independent
AVG
MoE
PoE
MoPaE
MMVM
0.6
0.6
Accuracy
Accuracy
0.4
0.4
0.2
0.2
0.0
0.0
0.19
0.18
0.17
0.16
0.15
0.19
0.18
0.17
0.16
0.15
Reconstruction Error
Reconstruction Error
(a) Latent Representation Classification
Conditional Generation Coherence
Figure 3: Results based on a memory experiment conducted on five rats, each regarded as separate modality. We report the performance of the latent representation classification and the conditional generation coherence against the reconstruction loss for different & values for the different VAE methods. Every point in the figures represents a specific 3 value, where 3 = (10-5, 10-1, 10-1,10-3 10,3003,000 2.5x 10-$,5 X 10-3 10-2). An optimal model would be in the top right corner.
5.1
Benchmark Datasets
We first compare the proposed method against five strong VAE-based learning approaches on three frequently used multimodal benchmark datasets*.
Datasets. We perform benchmark experiments on the translated PolyMNIST [Daunhawer al., 2022, Sutter et al., 2021J, the bimodal CelebA [Sutter et al., 2020J, and the CUB image-captions (Shi et al., 2019] dataset. The translated PolyMNIST dataset dataset [LeCun et al., 1998] with different backgrounds but shared digits. The digits of the different modalities are randomly translated, we cannot predict their location across modalities. Bimodal CelebA extends the CelebA dataset [Liu et al., 2015] with an additional text modality based on the attributes describing the faces. Similarly, the CUB image-captions dataset extends the Caltech bird dataset [Wah et al., 2011] 1] with human-gen generatte captions describing the images. Please see Appendix B for more details regarding the datasets.
multiple instances of the MNIST
Baselines. We evaluate our proposed method against set of jointly-trained independent VAEs (independent, Kingma and Welling, 2014], different aggregation- multimodal VAEs, and an aggregation- based multimodal VAE with additional modality-specific latent spaces. For the set of independent VAEs, there no interaction or regularization between the different modalities during training. For the aggregation-basee based multimodal VAEs, we use multimodal VAE with joint posterior approximation function. We evaluate four different aggregation functions: a simple averaging [AVG, Hosoya, 2018]. a prodect-of-experts [PoE, Wu and Goodman, 2018], a miituree-of-exper [MoE, Shi et al., 2019], and mixture-of products-of- VAE with modality-specin We train all VAE methods as B- BAVAS [Higgins al., 2016], where 3 an additional hyperparameter weighting the rate term R of the VAE (see Section 4). Appendix B.4 provides the implementation details of the proposed method and the baseline alternatives.
[MoPoE, Sutter et al., 2021]. For the multimodal subspaces, we use MMVAE+ method [MMVAE+ Palumbo et al., 2023).
Evaluation. We test the methods' ability to infer meaningful representation when only subset of modalities is available. In addition. we evaluate all methods terms of their data imputation performance, where we withhold subset of modalities at test time and conditionally generate them from the shared latent representations. In this imputation task, we assess whether the generated modalities are both of high quality and display the expected shared information, which we refer to as coherence. We assess the quality of the learned latent representations using linear classifiers trained on representations of the training set and the coherence using nonlinear classifiers trained on original samples of the training set use the reconstruction error as a proxy for how well each method learns the underlying data distribution. We assess each method by relating their
'The code the experiments on the benchmark datasets can be found here: thomassutter /ewtteaeemmmaeeeeeeeeeeee 'We use the same evaluation procedure in previous work [e.g., Shi et al., 2019, Sutter et al., 2021]
7

---

Odor A
Odor
Odor C
Odor
-10
-34 -10
30
(a) independent
(b) AVG
(c) MoPoE
(d) MMVM
Figure 4: Latent neural representation during a memory experiment. Each model's performance is evaluated based on its own optimal 3 value (0.00001, 0.01, 0,00001, 0.001 for independent, AVG, MoPoE, and MMVM respectively) in terms of the unimodal latent representation classification accuracy according to Figure 3a. Our method can distinguish the odor stimuli in the latent space with clear separation of odors similar to MoPoE VAE (4 different colors). Conversely, unimodal and AVG models failed to combine multi-views as the odor separation only occurred within single views.
achieved reconstruction error to either the learned latent representation classification or the coherence (Figure 2). We evaluate all methods for multiple values of 3, where the average performance over multiple seeds with a single value leads to a single point in Figure 2. Evaluating the methods for different values of 3 considers that the optimal / value is model- and data-dependent. In addition, increasing 3 emphasizes a more structured latent space [Higgins et al., 2016]. Hence, highlighting the dynamics between reconstruction error and classification performance for different multimodal objectives provides additional insights. We chose 3 E (2-8 3.... 2 } on the PolyMNIST dataset, E [2-5 ...., 2¹) on the CelebA dataset, and [2-2 22) on the CUB dataset. In all figures, the arrows go from small to large values of B. See Appendix B.2 for more details on the evaluation metries.
Results. Figure shows that the proposed MMVM VAE consistently outperforms the other VAE- based methods (independent, AVG, MoE, PoE, MoPoE) on all datasets and both tasks. We can show that our method overcomes the limitations of aggregation- PolyMNIST described in Daunhawer et al. [2020]. Also, MMVM VAE can learn meaningful representations and generate coherent samples across different modalities while achieving high reconstruction quality for both text-image datasets bimodal CelebA and CUB image-captions. The coherent conditional generation is especially surprising as the proposed MMVM VAE decoder is never confronted with sample from another modality. For all benchmark datasets, the proposed MMVM either achieves better latent representation classification and coherence performance with similar reconstruction loss lower reconstruction loss with similar classification performance than other multimodal VAE approaches, In summary, we can show that the newly proposed MMVM VAE overcomes the limitations of previous aggregation-based approaches multimodal learning (see Section 2) and outperforms previous works on all three benchmark datasets. We provide more results on the benchmark datasets in Appendix B.
multimodal VAEs on translated
5.2 Hippocampal Neural Activities
Dataset. Temporal organization is crucial to memory, affecting various perceptual, cognitive, and motor processes. While we have made progress in understanding the brain's processing of the spatial context of memories, our knowledge of their temporal structure is still very limited. To this end, neuroscientists have recorded neural activity in the hippocampus of rats performing a complex sequence memory task [Allen et al., 2016, Shahbaba et al., 2022]. More specifically, this study investigates the temporal organization of memory and behavior by recording neural activity from the dorsal CA1 region of the hippocampus. Briefly, the task involves presenting rats with a repeated sequence of non-spatial events (four stimuli: odors A, B, C, D) at a single port [Shahbaba al., 2022]. Since the same experimental setup was conducted across all rats, we consider the rats as different "modalities" and apply our proposed MMVM method to extract meaningful latent representations. While the existence (and importance) of subject-specift effects is well-known in neuroscience, such effects tend to be treated as unexplained variance because of the lack of the required analytical tools to extract and utilize this information properly.
8

---

Table 1: VAE latent representation quality evaluation. Average AUROC [in %] over three seeds of the two unimodal latent representations (2r and 2T,) on subset of MIMIC-CXL labels. The latent representations learned by the MMVM VAE lead to better classification performance compared to the other VAFs and are competitive with the fully-supervised method. Full results in Appendix B.7.3.
labels
No Finding
Cardiomegaly
Edema
Lung Lesion
Pneumonia
supervised
70.5 1
73.0 #1.4
80.3 #
87.1 #0,9
54.8 #2,5
61.3 +0.4
independent
68.0 =3.3
75.3 +1.4
73.5 82.8
79.2 #3,9
60.1 #1.2
55,8 +008
AVG
69.8 88.5
76.3 +1.5
76.1 +24
81.3 #3.3
60.4 +1.4
57.3
MoE
68.9 =8.7
76.5 #0.7
74,9 1
80.2 +2.3
59.6 #13
56.9 1
MoPoE
70.3 =8.9
77.2 +002
76.3 +:088
82.1 +12
60.8 10.6
57.8 1:0.7
PoE
70.4 #8.3
75.9 +1.
76.7 +1.9
81.8 +2.7
61.3 #2.1
57.8 +0,4
MMVM
73.1 4888
78.7
10.4
79.6 +:0.9
85.3 +1.0
63.6 +0.7
59.5 #0.7
Results. Our proposed MMVM method outperforms most previous works regarding learned latent representations and conditional generation coherence. Only the MoPoE VAE achieves classification performance comparable to the MMVM method but with a higher reconstruction loss. Figure 3a shows the separation of the latent representation (measured by the accuracy of multinomial logistic regression classifier) against the reconstruction loss. Similar to the results on the benchmark datasets, the proposed MMVM VAE outperforms previous works by providing clear separation of odors in the latent space while maintaining a low reconstruction loss. Figure 3b compares the coherence of conditional generation accuracy against the reconstruction loss. As before, our proposed approach outperforms the alternatives. The proposed MMVM method allows learning an aligned latent representation across different modalities. We show the 2-dimensional latent representations for every rat and four VAE encoders in Figure 4. Each dot is the wwoddimensional latent representation of 100 ms sub-window of one odor trial for one rat and is colored according to its ground truth odor value. Figure 4 shows the odor stimuli separation on the latent space and how good MMVM VAE is in separating the odors. At the same time, two baseline models fail to extract the shared information between rats. Although it shows separation in some views, the independent model does not provide connection between views. The five tiny clusters in Figure 4b show that, instead of showing a clear odor separation on the latent space, the AVG VAE separated the data by rats. In other words, the five rats' latent representations were far from each other, so the AVG VAE failed to connect the five views. See also Appendix B.8 for more results.
5.3 MIMIC-CRR
Dataset. To assess the performance of our approach in real-world setting, evaluate the proposed MMVM method on the automated analysis of chest X-rays, a common and critical medical task. For this purpose, we use the MIMIC-CXE dataset [Johnson et al., 2019b], a well-established and extensive collection of chest X-rays. The dataset reflects real clinical challenges with varying image quality due to technical issues, patient positioning, and obstructions. The dataset includes different views, which provide complementary information valuable for improving diagnostic [Raoof et al., 2012]. In this work, we consider frontal and lateral images as two modalities (see Appendix B.7 for further details). Each set of X-rays is labeled with different cardiopulmonary conditions, which have been automatically extracted from the associated radiology reports [Irvin et al., 2019]. This results in instances with incomplete label sets [Haque et al., 2023], which presents a challenge for fully supervised approaches and motivates the need for self-supervised methods instead.
Results. We evaluate? the quality of the unimodal latent representations of the MMVM VAE by
comparing them with those learned by a set of jointly-trained independent VAEs (independent) as well as with representations from other multimodal VAËS that use aggregation- MoE, MoPoE, and PoE) (see Section 5.1 for more details). We do this by training binary random forest classifiers independently for each method and all labels on the inferred representations of
approaches (AVG,
'The code for the hippocampal neural activity experiments can be found here: https: httpss://github..com y yangmeng96/mmvarn mmvvvvae- tumawaae-hippoaamee hippocampal 'The code for the MIMIC-CX CXR experiments can be found here: https://github.rr wtt.:occooaaaaaeeeee http:/ttthbbc.coo///ooii33333111 mmvnvae Dovmvae-mmiic

---

subset of the training set. Table shows the AUROC for these classifiers, averaged over three seeds and both unimodal representations for subset of labels. In addition, we also report the performance of deep nonlinear network trained in a fully supervised manner (supervised) on the same train/test. split for reference purposes. Detailed experiment information can be found Appendix B.7.3, with extensive results for each modality and label available in Table 2 and Table 3. Overall, our approach shows performance improvements across all labels compared to the other VAEs and is highly competitive with the fully-supervised method, surpassing it in average performance over all labels. Examining each unimodal representation separately provides further insights into the VAEs' ability leverage information from other modalities during training. For example, in the Cardiomegaly prediction task, the MMVM VAE's s lateral representations 21 slightly outperform the PoE VAE's s frontal representations 2F (MMVM Z1: 78.7%, PoE 2F: 78.5%), even though the lateral modality seems generally less informative (supervised Fy: 79.0%, Fr: 81.7%) for this task. The same observation can be made for other labels (see detailed results and discussion in Appendix B.7.3). This illustrates the MMVM VAE'S ability to soft-share information between modality-speciff latent representations during training. thereby enhancing the representation quality of the weaker modality.
6 Broader Impact & Limitations
This paper aims to advance the field of Machine Learning by providing a natural and fundamental solution for integrating data across modalities. The proposed approach can be applied to various scientific and engincering problems with a potentially significant societal impact. In the field of neuroscience specifically, our method could allow neuroscientists to leverage individual differences in brain activity and behavior to understand basic information processing in the brain, as well as to capture distinct longitudinal changes to understand how it is affected in disease. In translational research, it could help identify subjects more susceptible to disease or potentially more responsive to treatment. While we can show that the proposed method learns better representations and generates more coherent samples, we cannot directly generate random samples anymore (see also Section 4). Although we show results on two real- world datasets, Sections 5.2 and 5.3, additional experiments on even larger scale multimodal datasets would help further evaluate the proposed method, e.g. [Damen al., 2018 Wu al., 2023]. However, training and evaluating our methods on such datasets requires immense computing resources.
7 Discussion & Conclusion
In this work, presented a new multimodal VAE, called MMVM VAE, based on data-dependent multimodal variational mixture-of-experts prior. By focusing on a multimodal prior, the proposed MMVM VAE overcomes the limitations of previous methods with over-restrictive definitions of joint posterior approximations. The proposed objective leveraging the MMVM prior takes inspiration from contrastive learning objectives, where the goal is to minimize the similarity between positive pairs while maximizing the similarity between negative pairs [see, c.g., Chen et al., 2020, Tian et al., 2020]. In the MMVM objective, we minimize the similarity between different modalities of the same sample (positive pairs) via the regularizing term in the objective, whereas the second part of the objective, the reconstruction loss, prevents degenerate solutions.
In extensive experiments on three different benchmark datasets, we show that MMVM VAE outper- forms previous works in terms of learned latent representations well as generative quality and coherence of missing modalities. We also demonstrate its efficacy on two challenging real-world ap- plications and show improved performance compared to previous VAEs and even a fully-supervised approach. Future research could involve studying the representatioon-ditoriiee perspective [Yang and Mandt, 2022, Yang et al., 2023] and applying similar ideas to more powerful multimodal generative models and representation learning methods. We see a lot of potential in applying the MMVM regularization to other multimodal and multiview objectives, e.g., as an additional guidance signal for diffusion models. While masked modeling has shown impressive results as an objective for representation learning, current multimodal masked modeling objectives concatenate the embedding tokens coming from different modalities [see, e.g., Bachmann et al., 2022]. Adding the MMVM regularization objective would offer an interesting alternative to sharing information from different modalities.
tradeoff from an
information-theoretiee
10

---

Acknowledgements
TS, AA, and DC are supported by the grant #2021-911 of the Strategic Focal Area "Personal- ized Health and Related Technologies (PHRT)" of the ETH Domain (Swiss Federal Institutes of Technology). BS and NF received funding from the NIH award R01-MHI 15697 and NSF award NCS-FR-23 2319618. SM acknowledges support from the National Science Foundation (NSF) under an NSF CARBER Award 2047418 and IIS- 2007719, the NSF LEAP Center, by Department of Energy under grant DE-SC0022331, the IARPA WRIVA program, the Hasso Plattner Research Center at UCI, and by gifts from Qualcomm and Disney.
References
A. Allen, D. M. Salz, S. McKenzie, and N. J. Fortin. Nonspatial sequence coding in CAI neurons. Journal of Neuroscience, 36(5):1547-1563, 2016. ISSN 0270-6474. doi: 10.1523JNEUROSCI. 2874-15.2016 URL http://wwwr http:///www.jneurossc org/ /tg/conteent///5/55/4/
R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir. Multimae: Multi-modal multi-task masked autoencoders, 2022. URL https::/aaxxiei
wwwggabs//2004.000
T. Baltrušaitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423 443, 2018.
A. Bhattacharyya, M. Hanselmann, M. Fritz, B. Schiele, and C. Straehle. Conditional flow variational autoencoders for structured sequence prediction. CoRR, Abs/1908.09009 2019. URL http:
//arxiv. org/
D. Bouchacourt, R. Tomioka, and S. Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A Simple Framework for Contrastive Learning of Visual Representations, June 2020. URL
D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018.
I. Daunhawer, T. M. Sutter, R. Marcinkevics, and J. E. Vogt. Self-supervised Disentanglement of Modality-speciff and Shared Factors Improves Multimodal Generative Models. German Conference on Puttern Recognition, 2020.
I. Daunhawer, T. M. Sutter, K. Chin-Cheong, E. Palumbo, and J. E. Vogt. On the Limitations of Multimodal VAEs. International Conference on Learning Representations, 2022.
B. Efron. Large-scale inference: empirical Bayes methods for estimation, testing, and prediction, volume 1. Cambridge University Press, 2012.
W. Falcon and The PyTorch Lightning team. PyTorch Lightning, Mar. 2019. URL htLptttps://ittht http/LiighnnnngAA//iigee. AI/iightnin.....
Ghosh, M. S. Sajjadi, A. Vergari, M. Black, and B. Schölkopf. From variational to deterministic autoencoders. arXiv preprint arXiv: 12436, 2019.
M. U. Haque, A. K. Dubey, I. Danciu, A. C. Justice, O. S. Ovchinnikova, and J. D. Hinkle. Effect of image resolution on automated classification of chest X-rays. Journal of Medical Imaging, 10 (4)044503-004503, 2023.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.

---

I. Higgins, L. Matthey, A. Pal. C. Burgess, and X. Glorot. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016. URL / wttpr//vviii.eeeeeotnoiiiaaaaa Sy2fzU9g1.
H. Hosoya. A simple probabilistic deep generative model for learning generalizable disentangled representations from grouped data. CoRR, abs//8009.n 2018.
Irvin, Rajpurkar, M. Ko, Yu, S. Ciurea-lleus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597,40 2019. 2019.
A. Johnson, M. Lungren, Y. Peng, Z. Lu, R. Mark, S. Berkowitz, and S. Horng. MIMIC-CXR-JGG- chest radiographs with structured labels. PhysioNet, 2019a.
A. E. Johnson, T. J. Pollard, J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, and S. Horng. MIMIC-CRR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019b.
T. Joy, Y. Shi, P. H. S. Torr, T. Rainforth, S. M. Schmon, and N. Siddharth. Learning multimodal vaes through mutual supervision. CoRR, 015222006.155700 2021. URL https://arxiv, www.:o..gga/aao...o / 2106. 12570.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998 2014.
D. Kingma and M. Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations, 2014.
Y. LeCun, C. Cortes, and C. J. Burges. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, volume 86, pages 2278-2324, 1998.
Liang, A. Zadeh, and L.-P. Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint 20X1v:2209.034300, 2022.
J. Lin. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37(1):145-151 Jan. 1991. ISSN 1557-9654. doi: 10.1109/18.61155. URL https:
//ieeexplore ieee. httg/document/6111
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep Learning Face Attributes in the Wild. In The IEEE International Conference on Computer Vision (ICCV), 2015.
F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. Weakly-supervised disentanglement without compromises. International Conference on Machine Learning, pages PMLR, 2020.
S. Mahajan. I. Gurevych, and S. Roth. Latent normalizing flows for many-to-many cross-domain mappings. CoRR, abs/2002.06661. 2020. URL. aattps://arxiv. org//bs////0......
L. Manduchi, K. Pandey, R. Bamler, R. Cotterell, S. Däubener, Fellenz, A. Fischer, T. Gärtner, M. Kirchler, M. Kloft, al. On the challenges and opportunities in generative ai. arXiv preprint arXiv:2403.00025, 2024.
A. v. d. Oord, Li, and O. Vinyals. Representation Learning with Contrastive Predictive Coding, Jan. 2019. URL attp://arxiin org/ haaaab///10 1807 1807.03748.
E. Palumbo, I. Daunhawer, and J. E. Vogt. MMVAE+: Enhancing the generative quality of multimodal vaes without compromises. In The Eleventh International Conference on Learning Representations. OpenReview, 2023.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. Advances Neural Information Processing Systems 32, pages 8024-8035.
Curran Associates, Inc., 2019.
12

---

F. Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel. M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and H. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2882 2011.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning Transferable Visual Models From Natural Language Supervision, Feb. 2021. URL http:://aars org/ abs//10///033 00020.
A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 2021.
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 07Xiv:2204.061255, 2022.
S. Raoof, D. Feigin, A. Sung, S. Raoof, L. Irugulpati, and E. C. Rosenow III. Interpretation of plain chest roentgenogram. Chest, 141(2)555--558 2012.
C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi. Photorealistic Text-to- -exttto-Image Diffusion Models with Deep Language Understanding, CoRR, abx/2205.1, 2022. doi: 10.48550/axiiv22205. 1487. URL https: / /doi. org/ ftrg/10.ao 48550//aXiv......//////// 2205...............mm.. 11487.
B. Shahbaba, L. Li, F. Agostinelli, M. Saraf, K. W. Cooper, D. Haghverdian, G. A. Elias, P. Baldi, and N. J. Fortin. Hippocampal ensembles represent sequential relationships among tended sequence of nonspatial events. Nature Communications, 13(1):787, Feb. 2022. ISSN 2041-1723. doi: 10. 10.1038/s41460 841467-022-28057
URL https: nature. com/articles
Y. Shi, N. Siddharth, B. Paige, and P. Torr. Variational Sexture-of-Expertt Autoencoders for Multi- Modal Deep Generative Models. In Advances in Neural Information Processing Systems, pages 15692-15703, 2019.
C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29, 2016.
T. M. Sutter. Imposing and Uncovering Group Structure in Weakly-Superrvssed Learning. Doctoral Thesis, ETH Zurich, 2023. URL http::://wt.rrsearrccooooollee ethz. www.woe..wt.......ettteeeeeaaeae ch/handle/ 20.500. 01850/634822.
T. M. Sutter, I. Daunhawer, and J. E. Vogt. Multimodal Generative Learning Utilizing Jensen- Shannon Divergence. Advances Neural Information Processing Systems, 2020. URL https: //arxiv..v.////////// wrw. 2b0/ 2aa.b///000002.......
T. M. Sutter, I. Daunhawer, and J. F. Vogt. Generalized Multimodal ELBO. International Conference on Learning Representations, 2021. URL.
M. Sutter, L. Manduchi, A. Ryser, and J. E. Vogt. Learning Group Importance using the Differ- entiable Hypergeometric Distribution. In International Conference on Learning Representations, 2023a.
T. M. Sutter, A. Ryser, J. Liebeskind, and J. E. Vogt. Differentiable Random Partition Models. In Advances in Neural Information Processing Systems, 2023b.
M. Suzuki and Y. Matsuo. A survey of multimodal deep generative models. Advanced Robotics, 36 (5-6):261-278, 2022.
Y. Tian, D. Krishnan, and P. Isola. Contrastive Multiview Coding, Dec. 2020. URL httt:://axiii org/abs/1906. https//1006.00.999. 05849.
J. M. Tomczak and M. Welling. VAE with a VampPrior. arXiv preprint arXIv:1705.07122 2017.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-UCSD Birds-200-2011 (CUB-200-201I). Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
13

---

M. Wu and N. Goodman. Multimodal Generative Models for Scalable
Learning. In Advances in Neural Information Processing Systems 31: Anmual Conference on Neural Infor- mation Processing Systems 2018, NewrlPS 2018, 3-8 December 2018, Montreal, Canada, pages 5580-5590 5590, Feb. 2018. URL httpt/t://arate /tt/////ww....rra/////////////// 05335.
Wu, Fei, L. Qu, W. Ji, and T.-S. Chua. NEXT-GPT: Any-to-Any Multimodal LLM, 2023.
Y. Yang and S. Mandt. Towards empirical sandwich bounds on the rate-distortion function. In International Conference Learning Representations, 2022.
Y. Yang, S. Eckstein, M. Nutz, and S. Mandt. Estimating the rate-distortion function by wasserstein gradient descent. Advances in Neural Information Processing Systems, 36, 2023.
14

---

A MMVM VAE
A.1 Bound on the proposed objective
The objective (ar)) for a single modality m is given by
(m = Bug (xm (mm)
[log Po (mim
zmm] - KTT. [gp(2m Fmm) pa (2())
(4)
S Bg (xm1)2m)) (mm)
[log Po (30m 2mn)
(5)
< log (2 (2mm)
(6)
where um (m)m = fm is the output of the optimized (mean) encoder f.(() of modality m.
Equation (5) follows from the non-negativity of the KL divergence. Without regularization term, the maximizing distribution is a delta distribution with zero variance (Equation (6)). Equation (6)
equals the maximum-likelihood version of the proposed MMVM VAE (for single modality). Put differently, the MSE of "vanilla" autoencoder is an upper bound on the objective {xxm) for any prior distribution pe (.m). hence, also for the newly introduced MMVM prior distribution h(zm. X).
Let us have look at when the KL term actually vanishes given a MMVM prior distribution h(zm X). The KL term can only vanish if all posterior approximation g g (zm Emm.) map single mode 1 (zm Dm.) In that
M
h(zm X))
L (zm (.m..)
43 (zm
(7)
and KL 9% (zm xm) || h(=m
x)
O.
Another scenario is when 1% (zm Imm) and 12m Fmm) have disjoint modes. Hence, L (zm only a match to itself. In this case, we have
Xm))
))
Egg (# (oe..)
log
1 M
Ls
(zm. mmm))
Boy (am lamm)
log
M 26 Fm
(2m
(8)
m=
log M Eggg (xm(am)) [log %% (Zm
[m)
(9)
So. our objective will still be (up to a constant) the maximum likelihood objective, leading the variances shrink to zero. In this case, the objective will reduce to the limiting independent VAE, where the modalities do not "talk" to each other and there is no multimodal alignment.
In the most interesting case, there will be non-trivial overlap between 13 (2m Fm) and 112m 2,), i. between the different unimodal posterior approximations, leading to multimodal alignment through the soft sharing that we wish to see.
In addition, Figure empirically shows that the negative mean squared error (MSE) of the vanilla autoencoder (AE) upper bounds the proposed objective E. We show that lowering the value the regularizer R (see Section 4) approximates the negative MSE of the vanilla AE.
B
Experiments
B.1
Dataset Licences
originally published in [Sutter et al., 2021], downloaded the data from https: / /drrive.gogll. com/drivee/ollerrs [al==-aayyjjgakaaaaaaleeeaaaaaaa usp=sharing and the code from https: https://github: com http////ammsttttee/ooooo... published under the MIT license
PolyMNIST:
Bimodal CelebA: originally published in [Sutter al., 2020], downloaded from https: //drive.googe google. com/ /t//driie///e// wwweeeefolerrreeeeeeeeeeeeeeeese jala--aaaaipgAAzzzaaaaaeeealaaa JAst otttr/ffooodrrss//aaaaaaaaa.... /arraaaaaaaaja/zaaaaaaa
usp=sharing, licensed under the MIT license. The original CelebA dataset was published in [Liu et al., 2015], license not found.
CUB image-captions: originally published [Shi et al., 2019], downloaded from http: //www.robott.. ox. wwsuyshi//////////p vyshi/mmdgul http///ddnnddadaeddee/eee hwtp//dtatttst/eeeeeaaaaaaeeeeet cub. zip, licensed under GPL 3.0.
15

---

-0.76
Negative MSE of Vanilla Autoencoder, the Bound
Objective
-0.78
-0.80
-0.82
-084
-10
8
log(3)
Figure 5: We compare the achieved values of the proposed objective & to the vanilla Autoencoder's negative mean squared error (MSE). Lowering the /3 value of the regularizer R in the objective (see Section 4) approximates the negative MSE bound provided by the vanilla AE. This proves empirically that the negative MSE of the vanilla AE indeed upper bounds the proposed objective E.
MIMIC-CXR:
originally published in [Johnson al., 2019b], downloaded from https: cettant/mmiiccceeeeee 0/. licensed under PhysioNet Creden-
/ //physionet. h/tg/conten//miin tialed Health Data License 1.5.0 (see https: httpps/:///ysiontt...pm wwtg/onnten//mmiicc. cxr/
vien-1icensee2...00
MIMIC-CXR-JPG: originally published in [Johnson et al., 2019a], downloaded from https://physsionet. org/ content/mimii exr- stt-jjp/2.1.00 licensed under Phys- ioNet Credentialed Health Data License 1.5.0 (see https: httt::::/pphsiooet... wwtpeett.rr///ooeeeeeeee org/content/////ooooooogooo.. mimic-cxr-jpg/2.. .0).
Hippocampal Neural Activity data: originally published in [Shahbaba et al., 2022], downloaded from https:/ www.oorg/sssaseee maashhdataset//aee 7280/D14X30 licensed under CCO 1.0 Universal (CCO 1.0) Public Domain Dedication (see https:
//creativecommons http:///moo..ooopppooooooooo//aa
B.2 Evaluation Details
We evaluate the different methods based on the coherence of their imputed samples, the quality of their latent representations, and their reconstruction error. We assume access the full set of modalities during training, but we do not make this assumption at test time. Hence, there is a need for methods that can conditionally generate samples of these missing modalities, given the available modalities. In other words, we want to be able to impute missing modalities. Imputed modalities should not only be of high generative quality but also display the same shared information as the available modalities (i.e., be coherent). For example, if we generate sample of modality 22 based on modality æj from the PolyMNIST dataset (see Figure 6), the generated sample of modality x2 should contain the same digit information as modality æ1 but show the background of modality x2.
B.2.1 Latent Representation Evaluation
We assess the learned representations based on subsets of modalities and not the full set. The quality of the representations serves as a proxy of how useful the learned representations are for additional downstream tasks that are not part of the training objective. Hence, high-quality representations of subsets of modalities are also the basis for conditionally generating coherent samples. We assess their quality by using the classification performance of linear classifiers. We train independent classifiers on the unimodal representations of the training set and evaluate them on unimodal test set representations. To assess the learned latent representation, we train a logistic regression classifier on 10000 latent representations of the training set. The accuracy is computed on all representations of the test set.
B.2.2 Coherence Evaluation
The coherence of conditionally generated samples shows how well the content of the imputed modalities aligns with the content of the available modalities in terms of the shared information. We evaluate the coherence using basNet-based classifiers [He et al., 2016] that are trained samples
16

---

from the original training set of every modality. Using the described procedure, generated samples have be visually similar the original samples have high coherence. Otherwise, the nonlinear classifier will not be able to predict digits correctly.
To compute the coherence of conditionally generated samples, we train additional deep classifiers on original samples of the training set. We use a ResNet-based non-linear classifier that is trained on the full original training set. The prediction of this classifier is then used determine the class (on PolyMNIST: digit) of the conditionally generated sample of a missing modality. The nonlinear classifier reaches an accuracy of above 98% on the original test set. For other datasets, the nonlinear classifier is trained to predict the shared information of the multimodal dataset. Hence, it serves as good oracle for determining the digit of generated samples.
Conditional Generation with the MMVM VAE and the independent VAEs zm conditioned on modality Dm. we proceed as follows:
To generate modality
1. We encode modality om using the encoder (Zm @m))
2. We sample latent vector ~ 4% (Zm mm)
3. We input the latent representation z into the decoder pe (#tm. =) of modality ain.
To compute the coherence numbers reported, we perform the above steps for every modality a. where m E (I,...,M, MM and average the achieved coherence accuracies.
Conditional Generation with the aggregated multimodal VAEs All aggregation-baser multi- modal VAEs conditionally generate samples in the same way. Hence, the following procedure applies the multimodal VAEs used in Section ([AVG, Hosoya, 2018], [PoE, Wu and Goodman, 2018], [MoE, Shi et al., 2019], and [MoPoE, Sutter et al., 2021J). To generate modality om conditioned on modality am we proceed follows:
1. We encode modality om using the encoder (zm (m))
2. We sample latent vector ~ 4% (Zm @m)
3. We input the latent representation z into the decoder po (æta. z) of modality æn
If would have access to multimodal subset XA consisting of than one modality, i.e. |A| >> 1, we would have the joint posterior approximation distribution go(2 Xx) of the subset X A- sample a latent vector from that distribution, i.c. g0(= Xa). and generate modality anh. using the decoder p (a 2) of modality æi.
B.2.3 Generative Quality
The reconstruction error is a proxy for how well every method learns the underlying data distribution. We evaluate the different VAE models by their achieved reconstruction eтrоr against either the learned latent representation classification or coherence (e.g., Figure 2). We do this for multiple values of B where the average performance over multiple seeds with single value leads scatter point in the figures. This way, we can assess the trade-off between accurately reconstructing data samples and inferring shared information. In addition, we compute the FID [Heusel al., 2017] values for modalities that can be summarized as natural images in the appendix. A low FID score correlates with high-quality generated image.
B.3 Implementation (General)
We use the scikit-learn [Pedregosa et al., 2011] package for the linear classifiers to evaluate the learned latent representations. All code is written using Python 3.11, PyTorch [Paszke et al., 2019] and Pytorch-Lightning [Falcon and The Py'oorch Lightning team, 2019]. We base the implementa- tions of the aggregation-based VAE methods the official implementations. Hence, base implementations on the following repositories:
AVG: htttps//ggituuu..mm///rroololloe
PoE: https: /tt:://gittttt http://.....////////mmmmmaaaaaaa public
17

---

2
3
4
7
2
3
4
5
7
9
2
$
4
5
8
Figure 6: PolyMNIST (translated, scale=75%): every column is a multimodal tuple X, and every row shows samples of single modality Em- We see the random translation between samples by looking at images from a single row or column.
MoE:
MoPoE:
B.4 PolyMNIST
B.4.1
Dataset
The dataset is based on the original MNIST dataset [LeCun et al., 1998]. Compared to the original dataset, the digits are scaled down by factor of 0.75 such that there more space for the random translation. In its original form, the PolyMNIST consists of 5 different modalities. We only use the first three modalities in this work. The background of every modality &m consists of random patches of size 28 x 28 from a large image. The digit is placed at random position of the patch. We refer to the original publication [Sutter et al., 2021] for details on the background images. Using this setup, every modality has modality-specifff information given by its background image and shared information given by the digit, which is shared between all modalities. An additional difficulty compared to the original PolyMNIST is the random translation of the digits. The dataset can be found at https: //t///gittt githuu................... com/ http///oomssutteer///o /MoP.E..
B.4.2 Implementation & Training
We use the same network architectures for all multimodal VAEs. Every multimodal VAE consists of a ResNet-based encoder and a ResNet-basee Decoder [He et al., 2016]. All modalities share the same architecture but are initialized differently. We assume Gaussian distribution for all unimodal posterior approximations, i.e.
g (Zm xm) (zmi Mms Tmd))
(10)
where the parameters Um and om are inferred using neural networks such that we have
(zm xmm) 1% (Zmi Mm (Xm), Fm (xx() )) = (2m Jm (mm), am (Xm))
(11)
The conditional data distributions pe (xm =m) are modeled using the Laplace distribution, where the location parameter is modeled with neural network (decoder) and the scale parameter is set to 0.75 [Shi et al., 2019], i.e.
pa (x
2m. L(x Mm, Tm);
(12)
where C(-) defines a Laplace distribution. It follows that
Po (mm
Zm) = Po (Dai Pm (zm), Fm) = L(xm ; frm (Zm) a.
(13)
We use the method of Hosoya [2018] for the implementation of the aggregated VAE. In this approach, a simplistic version of the joint posterior distribution is chosen where for Gaussian distribution joint posterior approximation [[,,, oI we have the following distribution parameters μ,d and o,
M Σ
14,
Pm
(14)
M
M
M
o
(15)
M
m
18

---

Um and om are the distribution parameters of the unimodal posterior approximations N Npm: m
During training and evaluation, no weight-sharing takes place, i.e. every modality has its own encoder and decoder. We use the same architectures as in [Daunhawer et al., 2022]. For all experiments on this dataset, we use an Adam optimizer [Kingma and Ba, 2014] with an initial learning rate of 0.0005, and a batch size of 256. We train all models for 500 epochs.
We use NVIDIA GTX 2080 GPUs for all our runs. Each experiment can be run with 4 CPU workers and 16 GB of memory, An average run takes around 7 hours. To train all methods used in this paper, we had to train 11 x 5 x 6 330 different models: 11 different 3 values, 5 different seeds, and 6 different methods. Hence, the total GPU compute time used to generate the results for the PolyMNIST dataset is equal to 330 x 7 = 2310 hours. We-of of course also had to invest development GPU time, which we did not measure.
B.4.3 Additional Results
We generated Figures 2a and 2d by plotting the classification accuracy of linear classifier, which we trained on the learned latent representation, against the reconstruction error on the test set. For the learned latent representation, we train a classifier on the unimodal latent representations. For the aggregated VAE, this means that we train the classifier on samples of the unimodal posterior approximations and not the joint posterior approximations, Using this procedure, we test the different methods according their performance in case of missing data, e.g., we only have access modality instead of the full set at test time. For the reconstruction loss, however, we computed the error given the full set of modalities. The idea for Figures 2a and 2d is to compare the reconstruction quality (i.e., how well can we learn the data distribution?) against metrics that are related to the "generative factors" of the data and relate the different modalities to each other, i.e. the shared information of multimodal dataset.
single
In Figure 7, we evaluate the performance of individual modalities in case of missing modalities. For that, we reconstruct every modality if it was the only modality available at test time. Hence, the modalities in the aggregated VAE have to be reconstructed based on the unimodal posterior approximations and not the joint posterior approximation. For the independent VAEs and the MMVM- VAE, the reconstruction of a modality is only based on its own unimodal posterior approximation. Hence, for the latter two methods, nothing changes in this setting. The performance of the learned latent representation and the coherence of generated samples are evaluated in the same way as in Figures 2a and 2d.
Figure shows that the reconstruction error of the aggregated VAE increases lot if every modality needs to reconstruct itself. Interestingly, we can see that the "self-reconstruucton error" (the x-axis in Figures 7a and 7b) decreases with an increasing 3-value, which is different to the other two methods and also different to the aggregated VAE's behavior in Figures 2a and 2d.
In addition, we also show the conditional FID values the different multimodal VAEs achieve. Con- ditional FIDs come from the conditional generation of modality Fm given Imm From all the condi- tionally generated samples of modality E.n, we then compute the conditional FID values. Figure 8 shows the aggregated values of all conditional FIDs. We again show the FID values in relation to the downstream task performance and the conditional generation coherence. We see that only the proposed MMVM VAE reaches FID values similar to the ones of a set of independent VAEs. However, the set of independent VAEs cannot achieve the same latent representation classification nor coherence as the proposed MMVM VAE.
B.5 Bimodal CelebA
B.5.1
Dataset
We take the dataset from Sutter et al. [2020]. It extends the original CelebA dataset [Liu et al., 2015] to the multimodal setting by creating a second modality based on the attributes. In the original dataset, every image comes with set of 40 labels, such as blond hair, smiling, etc. The difficulty of the individual attributes is not only due to their visual appearance but also their frequency the dataset. In the multimodal extension, Sutter et al. [2021] created text string from the attribute words. Unlike the label vector, absent attributes are dismissed in the text string instead of negated. Additionally, the attributes are randomly ordered.
19

---

independent
AVG
MoE
MoPoE
MMVAE+
MMVM
1.0
0.6
0.8
Accuracy
0.6
Accuracy
0.4
0.4
0.2
0.2
0.0
0.0
8000
6000
4000
8000
6000
4000
Conditional Reconstruction Error
Conditional Reconstruction Error
Latent Representation Classification
(b) Conditional generation coherence
Figure 7: Results on the PolyMNIST dataset for different VAE methods. We report the performance of the latent representation classification and the conditional generation coherence against the conditional reconstruction loss for different 3 values. Every point in the figures above is the average of five runs over different seeds and specific 3 value where 3 2 for (-8...., ,3.. Different to Figures 2a and 2d, the x-axis is the sum of the self-reconstruction losses if only single modality given as input. Hence, for the aggregated VAE methods, every modality is decoded by own unimodal posterior approximation instead of the joint posterior approximation.
independent
AVG
MoE
PoE
MoPoE
MMVAE+
MMVM
1.0
0.6
0.8
Accuracy
0.6
Accuracy
0.4
0.4
0.2
0.2
#0-046
0.0
0.0
250
200
150
100
50
250
200
150
100
50
FID
FID
(a) Latent Representation Classification
(b) Conditional generation coherence
Figure Results on the PolyMNIST dataset for different VAE methods. We report the performance of the latent representation classification and the conditional generation coherence against the conditional FID values. Every point the figures above is the average of five runs over different seeds and specific B value where 3 = 2 for {-8..... 3). Different to Figures 2a and 2d, the x-axis is not reconstruction error but the average FID value computed from conditionally generated samples.
20

---

8
8
7
0
7
3
8
3
9
(a)
(b) m0 ml
(c) m0 -> m2
Figure 9: Qualitative results for the conditional generation task for the set of unimodal VAEs.
8
3
7
8
(a) m0 > m0
(b) m0 - ml
(c) m0 -> m2
Figure 10: Qualitative results for the conditional generation task for the aggregation- based multimodal VAE (AVG).
21

---

17
8
(a)
m0
(b) m0 ml
(c) m0 + m2
Figure 11: Qualitative results for the conditional generation task for the proposed MMVM VAE.
independent
AVG
MoE
PoE
MoPoE
MMVAE+
MMVM
0.6
0.6
0.5
0.5
Accuracy
Accuracy
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
250
200
150
100
50
250
200
150
100
50
FID
FID
(a) Latent Representation Classification
(b) Conditional generation coherence
Figure 12: Results the bimodal CelebA dataset for the different VAE methods. We report the performance of the latent representation classification and the conditional generation coherence against the conditional FID values. Every point in the figures above is the average of five different seeds and specific 3 value where = 2* for k E { 4]. Unlike in Figures 2a and 2d, the x-axis the reconstruction error but the average FID value computed from conditionally generated samples.
heapy makeup big los bange cwering meckiick, young. beard.
playy cydaroess byg rose,
bags
holdiis moin.
2hi umiinng musodde
loettik bangs, blond lose vovy nage under syee, high chweddvies,
Figure 13: Bimodal CelebA. Three samples of image-text pairs. To introduce another level of difficulty to the text modality, added random translation to the starting point of the text attributes.
22

---

B.5.2 Implementation & Training
We use ResNet-based encoders and decoders for this experiment as well [He et al., 2016), similar to the ones in the PolyMNIST experiment. The image encoder and decoder consist of 2D convolutions, while the text encoder and decoder consist of ID convolutions. We use character-levee encoding and not a word or token-level encoding because of the synthetic nature of the text modality. We also use an Adam optimizer [Kingma and Ba, 2014] with starting learning rate of 0.0002 and batch size of 128. We train all models for 400 epochs and 3 seeds. The implementation follows the one described in Appendix B.4.
We use NVIDIA GTX 2080 GPUs for all our runs. Each experiment can be run with CPU workers and 16 GB of memory. An average run takes around 24 hours. To train all methods used in this paper, we had to train 10 x 3 6 = 180 different models: 10 different 3 values, 3 different seeds, and 6 different methods. Hence, the total GPU compute time used to generate the results for the PolyMNIST dataset equals 180 x 24 x 4320 hours. We-of course-also had to invest GPU time to develop the method, which we did not measure.
B.5.3 Additional Results
Given the multilabel nature of the CelebA dataset, we evaluate the learned latent representation with respect to the individual attributes and not only the average performance across all attributes. Figure 14 shows the detailed results according to the full set of 40 attributes for the three methods: independent VAEs, aggregated VAE, and MMVM VAE. We again train linear binary classifiers on inferred representations of the training set and evaluate them representations of the test set. However, we now report the individual performance of every classifier. In the main text (see Figures 2b and 2e), we report the average performance of the 40 binary classifiers.
B.6
CUB Image-Captions
B.6.1
Dataset
The dataset contains 11,788 images of birds and 117,880 (10 times many) captions, each image with 10 fine-grained captions describing the bird's appearance characteristies collected through Amazon Mechanical Turk (AMT). We performed a 75-25 training-test split, with 8,855 and 2,933 images, and 88,550 and 29,330 corresponding captions in the training set and test set, respectively. We resized the images 3 channels and 64 by 64 pixels and modeled the captions using embeddings.
The labels applied to downstream tasks are based on the bird's primary colors. Based on the primary color attribute, we remapped the original 15-class primary color label set into a 6-class label set to address the sparsity issue. The labels and mappings are blue-to-red (blue, iridescent, purple, green, pink, red), brown (brown), grey (grey), yellow (yellow, olive, orange, buff), black (black), and white (white).
B.6.2 Implementation Details & Training
The network structures are similar to the ones in other experiments. For both image and caption data, we employed CNN encoder and decoder. We used 128 22-diimesioonal latent space with Laplace likelihood on images and a Categorical likelihood for captions. The activation function was chosen to be ReL.U between layers. For all experiments on this dataset, we used an Adam optimizer with an initial learning rate of 0.001 and batch size of 128. We trained all models for 100 epochs.
B.7 MIMIC-CXR
B.7.1
Dataset
The dataset we use in our experiment is multimodal interpretation of the original MIMIC-XXL dataset. The original MIMIC-CXR dataset comprises high-resolution chest X-ray images related to imaging studies. A study may include multiple chest X-ray images captured from several view positions. We categorized these views into two primary modalities: fronial (including "AP" and "PA" views) and lateral (including "LL." and "Lateral" views). For each study, we pair every frontal image with every lateral image in all possible combinations. Studies lacking at least one frontal and one
23

---

nndpeecenn
Neeggs brecsonn
06-
0.4-
Bak
Bango
Big yp-
Bigg Nous
Black that
Drond lor
Blery
Eyetroue
Double Chi
Eygghvors
Gontee
Cody Hat
Henry wonyyMakeeo
High Ohokkton
Mile
Mouth Siighly Open
Eyeo
Pale Shis
Paincy Nose
Decdd Horiie
Royy Clasko
Sibbumu
Smiing
Stsight Har
Ware Hai
Westnn Earingss
Lipuid
Bags Undee
Brows
Werrine
Clock
Viewing
(a) Image Representations
Avenge Presees
66
5. Clock Shatto
Archel Exirron
Bogs Mtede Eyre
Bald
Blcck Har
Dond. lar
Blony
Brown Har
Busty Eyybrros
Childty
Doulke Chi
Fynghnes
Gostee
Guyy Har
High Chelkhoree
Male
Mocarb
Narro Exta
No. Moad
Ovel Foot
Palk Skin
Poircy Mose Mose
Beceiing Harkee
Ruy Creede
Siddburs
Hair
Weotin Eartngs
Waating Hac
Wosting Lipoiic
Worting Neddace
Weeting Neckis
Yourd
Mouth Sightyy
Aot Butee
(b) Text Representations
Figure Attribute-level results on the bimodal CelebA datasets for the latent representation classification. The MMVM VAE outperforms the independent VAEs and the aggregated VAE on most attributes.
lateral image are excluded. This approach formalizes dataset composed of image pairs, thus offering a bimodal interpretation the original MIMIC-CXR dataset. More rigorously, we define dataset X {X (6) y where each X {x/ (c) ,x(") is a bimodal tuple composed of one frontal image and one lateral image of the same study. An image may appear in multiple tuples, but we ensure that each tuple unique by having at least one different image. Examples of these bimodal tuples are illustrated in Figure 17.
We the JPG version of the MIMIC-CXR dataset, namely the MIMIC-CXR-JPC dataset [Johnson et al., 2019a]. In our preprocessing pipeline, we apply center cropping and downscale the images resolution of 224 x 224. We utilize the labels from the MIMIC-CXR-JE dataset which are obtained using the CheXpert tool [Irvin et al., 2019]. All non-positive labels (including "negative," " "non- mentioned," or "uncertain")) were combined into an aggregate "negative" label following the approach adopted by Haque et al. [2023]. Each imaging study is connected to subject. We split the dataset into distinct training (80%), validation (10%), and test (10%) sets based on subjects, thus ensuring that the same image or study cannot be present in multiple sets.
B.7.2 Implementation & Training
Multimodal VAEs Implementation & Training We use ResNet-based encoders and decoders for this experiment [He et al., 2016), similar to those used in the PolyMNIST and the Bimodal CelebA experiments. The image encoder and decoder consist of 2D convolution layers. The architectural design of the image eneoders and decoders is uniform for both frontal and lateral modalities. We use an Adam optimizer [Kingma and Ba, 2014] with learning rate of 0.00005 and a batch size of 32.

---

(a) Independent VAEs
(b) AVG VAE
(c) MMVM VAE
Figure 15: Qualitative Results for CelebA dataset on the image-to-image generation task. The first 10 rows of every subplot show the input image and the bottom 10 rows its conditional generation. Different to the training, we provide only the image to every model and based on the latent represen- tation of that image. generate sample. We that the aggregated VAE (Figure 15b) is not able to conditionally generate visually pleasing samples compared to the independent VAEs (Figure 15a) and the MMVM VAE (Figure 15c).
(a) Independent VAEs
(b) PoE VAE
(c) MMVM VAE
Figure 16: Qualitative results for the MIMIC-CXI dataset the conditional generation task lateral to frontal. Results are aligned with the other datasets.
25

---

Figure 17: MIMIC-CXE experiment dataset: every column is bimodal tuple X, the top row shows samples of the frontal modality z, and the bottom row shows samples of the lateral modality æı- The first two tuples are linked No Findings, indicative of healthy conditions. Tuples three and four are labeled with Consolidation disease. The tuple five is labeled with Arelectasis disease. We can observe that tuples three and four share the same frontal image, but they differ due having distinct lateral images.
We train all methods for 240 epochs and seeds. The implementation follows the one described in Appendix B.4.
We use NVIDIA 40GB GPUs for all our runs. An average run, evaluating one method on one seed, takes approximately 45 hours. To train all methods evaluated in this experiment, we had to train 3 x
different models: seeds and methods. Hence, the total GPU compute time used to generate the VAE results for the MIMIC-CXR experiment is around 45 x 9 405 hours. We also had to use GPU time the development process, which we did not measure.
Supervised classifier Implementation & Training We use supervised classifiers based on Resnet. The classifier architecture is derived from the VAEs encoder we use in this experiment. To adapt the encoder for classification tasks, we added linear layer equipped with 14 neurons corresponding to number labels the MIMIC-CXR dataset. We use an Adam optimizer with a learning rate of 0.0001 and batch size of 256. We train distinct models for both frontal and lateral modalities, training each for 30 epochs and seeds.
We train the classifiers using NVIDIA A A100-SXM4-40GB GPUs. To train one classifier on one seed takes approximately hour. We train total of 6 classifiers: seeds and 2 modalities. resulting in about 6 hours of GPU compute time. We also had to use GPU time in the development process, which we did not measure.
B.7.3 Additional Results
In our main experiment, we evaluate the quality of the learned latent representations using Random Forest (RF) classifiers. Specifically, we train independent RF binary classifiers for each model and each label on the inferred representations of the training set and evaluate them on the representations of the test set. The RF classifiers are configured with 5,000 estimators and a maximum depth of 30. In Table in Section 5.3, we report for each model the average performance over the two modalities and three seeds, totaling six RF classifiers per model: one for cach modality and each of the three different seeds.
Here, we provide detailed insights into the capabilities of the models to leverage different modalities during training to improve the unimodal representations by reporting the performance of the latent representation classification for each modality separately in Table and Table 3. These results allow us to highlight the strengths of the MMVM model in improving unimodal representations with additional modalities. For instance, in the main text we mentioned that in the Cardiomegaly classification task, the MMVM lateral representations 21. lead to substantial improvements over the frontal representations z the other VAEs, even though the former modality Jy. less informative than the latter zF as indicated by the respective performance of the supervised model each
26

---

Table Evaluation of the VAEs' frontal latent representations 2F classification performance on the test split. The performance of fully-supervised non- linear deep network is included for reference. The average AUROC [%] and standard deviation over three seeds are reported. Eni. Cardiom, stands for Enlarged Cardiomediastinun and Support Dev. for Support Device.
supervised
independent
AVG
MoE
MoPoE
PoE
MMVM
Atelectasis
79.5 1003
73.1
75.2
73.0
74.2 10.
75.7
77.6
Cardiomegaly
81.7 +0.1
76.3 +0.4
78.5 #02
76.5 #0.6
77.1 #0.1
78.5 #0.3
80.5 #0.1
Consolidation
65.3 +0.7
62.4 +04
66.0 +0.8
62.9 #06
63.9 #0.3
66.7 #0.8
69.1 #0.6
Edema
1:0.2
83.0 10.3
84.6 10.3
82.4 10.6
83.1 10.6
84.5 10.3
86.3 1..1
Enl. Cardiom.
57.9 +1.3
59.5 +12
64.9 +0.8
61.7 #0.5
64.1 #0.5
66.4 #0.5
68.6 #1.1
Fracture
51.4 1.0.5
56.0
58.4
57.3
57.4 10.7
58.8
58.6
Lung Lesion
52.4 +0.4
61.3 +0.4
61.8 #0.2
60.6 #0,9
60.7 #0.8
63.4 #0.4
64.1 #0.2
Lung Opacity
69.5 +0.2
63.8 +0.3
65.7 +0.3
63.5 #0.2
64.7 =0.0
66.3 #0.2
68.1 #0.1
No Finding
73.9 11.3
76.6 10.3
77.8 1.0.0
77.1 110.2
77.4 10.11
77.2 102
79.1 1.0.1
Plcural Effusion
88.0 +0.0
81.2 #0,6
82.8 +0.0
81.6 #0.4
82.5 #0.5
82.8 #0,2
85.7 #0.3
Pleural Other
53.9
67.8
68.9
67.5
68.3
68.7
70.0
Pncumonia
61.3 +0.3
55.3 +0.5
57.8 #0.4
56.4 #0.4
57.3 #044
57.5 #04
60.0 #0.6
Pneumothorax
76.9
75.3 1100
78.3 #0.6
77.7 #0.4
78.8 =0.7
78.5 #0.7
81.9 #04
Support Dev.
77.7 1.0.1
70.8 1.0.3
73.1 1.0.4
72.7 1.0.7
73.7 1.0.6
73.8 1.0.1
76.6 1.0.2
All Labels
69.8 1.166
68.7 1..0
71.0 1.8.6
69.4 1.8.8
70.2 118.8
71.3 .18.4
73.3 1.8.
Table 3: Evaluation of the VAEs' lateral latent representation 2z classification performance on the test split. The performance of fully-supervised non-linear deep network is included for reference. The average AUROC [%] and standard deviation over three seeds are reported. Enl. Cardiom. stands for Enlarged Cardiomediastinum and Support Dev. for Support Device.
supervised
independent
AVG
MoB
MoPoE
PoE
MMVM
Atelectasis
78.0 10.
70.7 #0.3
73.5 10.4
72.8 10.1
74.7 #0.2
73.7 #0.2
77.0 :02
Cardiomegaly
79.0 +0.2
70.8 209
73.7 +0.1
73.3 102
75.5 10.1
74.8 10.1
78.7 10.0
Consolidation
68.6 11.4
64.4 11.
65.4 1115
64.9
65,8 10.8
66.7 10.9
70.2
Edema
86.2 +0.1
75.4 +0.9
78.0 1:.3
78.0 10.5
81.1 10.8
79.1 10.1
84.3 10.3
Enl. Cardiom.
61.5 11.0
60.1 20.7
62.0 11.0
60.5 10.5
64.2 1009
63.5 10.8
69.0 1007
Fracture
+02
57.9 #0.6
58.3 +0.7
56.8 10.8
58.6 +0.8
59.0 +0.5
+0.3
Lung Lesion
57.3 +.04
58.9 20.2
59.0 40.
58.6 1008
60.8 110.3
59.3 10.3
63.0 10.7
Lung Opacity
68.9 +0.2
61.9 #05
63.4
63.9 10.1
65.4 :004
64.1 #0.4
68.1
No Finding
72.0 10.8
73.9 40.
74.8 10.
75.9 10.2
77.1 110.1
74.6 10.11
78.3 10.1
Pleural Effusion
91.0 10.3
80.2 402
82.0 10.1
82.0 1:03
84.3 11022
82.1 10.1
85.7 1:0.1
Pleural Other
61.5
62.8
64.3 +0.7
62.7 +1.7
63.6 11.0
63.9 +1.0
68.5 11.
Pneumonia
61.2 1.0.4
56.4 10.55
56.9 103
57.5 1:12
58.3 110.5
58.2 1.0.1
59.0 1:02
Pneumothorax
82.4 =0.8
75.6 +0.5
77.8 +0.3
76.9 #0.7
79.2 #0.6
78.6 #0.2
81.7 10.3
Support Dev.
77.3 +0.1
71.9 +0.6
72.9 +0.5
73.6 +0.7
75.9 +0.4
74.7 +0.5
77.1 +:03
All Labels
71.2 +11.3
67.2 47.6
68.7 48.1
68.4 48.4
70.3 #866
69.4 +8.0
73.0 +8.5
modality. This behavior is not specific to Cardiomegaly and can be observed for other labels such as Atelectasis, Lung Opacity, No Finding. Plcural Effusion, Pncumonia, and Support Devices. Even though the difference between modalities for these labels is sometimes small, the consistency of those results demonstrate the soft-sharing capabilities of the MMVM approach. Conversely, in tasks where the frontal view is less informative, such as Consolidation. Enlarged Cardiomediastinum. and Pneumothorax, the MMVM's lateral representations outperform the frontal ones of the other VAEs.
The efficacy of the soft sharing mechanism in the MMVM VAE is also reflected in the average perfor- mance for all labels. The performance difference between the two unimodal latent representations produced by the MMVM VAE is substantially smaller (0.9 percentage points average) compared the independent VAEs (2.5 percentage points on average) and the PoE VAE (2.2 percentage points on average).
27

---

independent
PdE
0.80
0.75
AUROC
0.70
0.65
0.60
60000
50000
40000
30000
Reconstruction Error
Figure 18:
Latent representation classification for the MIMIC-CXR dataset. The mean AUROC over all labels and averaged over three seeds is reported.
Figure 18 illustrates the performance of the average unimodal latent representation classification described above against the reconstruction loss for the three different VAE methods introduced in the main text (independent, aggregate PoE VAE and the proposed MMVM VAE). As reminder, the AUROC performance [in %] is averaged over three seeds and the two modalities.
B.8
Hippocampal Neural Activities
B.8.1
Dataset
The training data collected from 250 length time frames after the port entry. Due the hehavior difference from each rat (some rats react faster to the odors while some others react slower), the training time frames of the five rats started from 250 ms, 250 ms, 500 ms, 500 ms, and 250 ms, respectively. During training, we treated each data point as independent and trained all the VAE models based on sliding windows (100 ms sub-window, 10 steps; 16 data points per window on each trial). The 100 ms sub-windows constituted the input data, with the dimension equal to the rat's number of neurons multiplied by 10, as the data was further binned into 10-ms increments.
B.8.2 Implementation & Training
We use the same network architectures for all multimodal VAEs. Each of the autoencoders includes its unique encoder and decoder, both containing two hidden layers, without weight-sharing during training and evaluation. All modalities share the same architecture but the layers' dimensions are different, with 920, 790, 1040, 490, 460 dimensional input and hidden layers, respectively. The activation function this dataset, we use an Adam optimizer with an initial learning rate 0.001, a batch size 128. We train all models for 1000 epochs.
chosen to be LeakyReLU with 0.01 negative slope. For all experiments
B.8.3 Additional Results
We show the 2-dimensional latent representations for every rat and the six VAE encoders in Figure and Figure 20. In these two figures, each dot is the two-dimensional latent representation of 100 sub-window of one odor trial for one rat. Figure is colored by 4 odors, and Figure 20 is colored by modalities (rats). Figure 4 shows the odor stimuli separation on the latent space and how good MMVM VAE is in separating the odors. Figure 20 shows that the proposed MMVM VAE can best utilize the shared information between the five rats by pulling the latent representations together. At the same time, the independent AVG and PoE baseline models fail to extract the information shared between rats. Although shows separation in some views, the independent model does not provide a connection between views. The five tiny clusters in Figure 4 and Figure 20 show that, instead of showing a clear odor separation on latent space, the AVG model separated the data by rats. The results went against the intention to share information across views. In other words, the five rats' latent representations were far away from each other, so the aggregated VAE completely failed to connect the five views.
28

---

Odar
Odor
Odor
Odor
-13
03
(a) independent
(b) AVG
(c) MoE
-12
-24
02
(d) PoE
(c) MoPoE
(f) MMVM
Figure 19: Latent neural representation during a memory experiment. Each model's performance is evaluated based its optimal 3 value (0.00001, 0.01, 0.001 for independent, aggregated, and MMVM respectively) in terms of the self-conditioned latent representation classification accuracy according Figure 3a. Our model can distinguish the odor stimuli in the latent space with clear separation of odors (4 different colors).
Rat 1
Rat
Rat
Rat
Rat 5
-28
-13
0.5
(a) independent
(b) AVG
(c) MoE
60
-12
-24
62
(d) PoE
(e) MoPoE
MMVM
Figure 20: Latent Representation of Rats Brain by Each Rat. In our proposed MMVM model, the five views shared latent representations the latent representation of all five views (colors) gathered together, while the two baseline models failed to combine multi-views.
29

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the paper, we introduce novel multimodal method that we evaluate against previous multimodal approaches. In the experiments, show improved performance multiple datasets.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the paper and the proposed method in the limitations section (see Section 6)
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
30

---

Answer:
[Yes]
Justification: We describe the assumptions to Lemma 4.1 in Sections 3 and and the proof in Section 4.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe all the experiments in full detail in the appendix (see Appendix B) such that all the results on all datasets can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
31

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the full code to reproduce all results, including README files and environment requirements. All the data is publicly available, and we provide information on how to access the data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ cablic/guides///
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips. httpp//cccpbbbcccguuuuuc wwwpc///w//////eee/eeeecee Ctte//Cooeeuiiiiic wwwssssiiccpiyiyo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the necessary information either in the appendix (see Ap- pendix B) or link to references, where the necessary information is provided.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report average results for all experiments performed over at least seeds. For smaller experiments, we average over 5 seeds. We report the number of random seeds used per experiment in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the compute time for the experiments conducted (including statement that more compute time was needed for the development) and the used GPU models.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //terrip/////......// ctteriis...pppppee httpppllc//ticsuuiiiddeep
Answer:
[Yes]
Justification:
We completely follow the Neurips Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential broader impact the proposed work alongside its limitations in Section 6.
Guidelines:
33

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
We do not use any pretrained language models, image generators, or scraped datasets.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all the papers we took inspiration from or we used code from. We include licenses for all datasets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
34

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com dataeets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will publicly release our code upon acceptance and document it accordingly, We attach the code the submission supplementary files.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
35

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "6vfB4pChpy", "reviewer": "Reviewer_56cj", "summary": "In this work, the authors proposed a new multimodal VAE architecture which, quickly summarized, both encoders and decoders factorized across modality latent variables, and the prior for each modality latent, $z_m$, is a mixture-of-experts composed of all the encoders. With this probabilistic model choice, the authors derive a number of interesting results, and show that this choice of prior is optimal to optimize the corresponding ELBO. Then, the authors empirically validate their findings by comparing their approach with existing methods in the PolyMNIST, Bimodal CelebA, and CUB datasets. Finally, the authors evaluate the quality of the latent samples on the Hippocampal Neural Activities dataset, showing that it can separate different odor types within the data.", "questions": "- **Q1.**  Why only independent and PoE were selected for the experiments in Table 1? I understand that MMVAE should have similar complexity to PoE.", "limitations": "Limitations are properly discussed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **S1.** The idea of the MMVM model is interesting, and it conceptually makes sense, overcoming the issue of combining the latent space that exists in current architectures. As so, it is significant to the community.\n- **S2.** I've checked most derivations, and they look correct and sound.\n- **S3.** The experiments show that the MMVM is at-good or better than existing works.", "weaknesses": "- **W1.** The experiments should compare with more modern approaches like [MMVAE+](https://openreview.net/forum?id=sdQGxouELX).\n- **W2.** The presentation could be improved, there is a clear lack of derivations, and some details are hard to grasp (it took me several reads to fully understanding the proposed probabilistic model).\n- **W3.** There are plenty of (to my eyes) over-statements and vague statements, depending on what the text requires. E.g.:\n  - Line 68. \"Require restrictive assumptions\" Which ones? Why are they restrictive?\n  - Lines 72-73. \"cannot ... and require stronger assumptions\". Why? Which ones?\n  - Line 52: \"thereby providing potential insights into the neural mechanisms underlying memory impairment\"\n  - Line 125: \"As argued and discussed in this paper, such aggregation can be overly restrictive.\"\n  - Line 183: \"is directly related to pairwise similarities between posterior approximation distributions of positive pairs, similar to contrastive learning approaches but in a generative approach\" \n- **W4.** Similarly, conclusions from the experiments look over-optimistic to me. It is true that the proposed method work as- or better than the baselines, but the results are pretty close (see Fig 3) without standard deviations. Also, saying that the samples in Fig. 4d are separated seems like an over-stretch to me (I'd at least add transparency to the points).\n\n\nPS: I'll happily increase my score once a newer method like MMVAE+ is included, which I find completely necessary given the scope of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 56cj,\n\n\nThank you again for your insightful review and feedback. In the following, we will address and answer your comments and questions.  \n\n---\n> The experiments should compare with more modern approaches like MMVAE+.\n\nWe show the results for the MMVAE+ in the rebuttal PDF. We excluded the results in the submitted manuscript because the more complicated MMVAE+ method could not outperform the (stronger) aggregation-based baselines used in our manuscript.\n\n---\n> The presentation could be improved, there is a clear lack of derivations, and some details are hard to grasp (it took me several reads to fully understanding the proposed probabilistic model).\n\nThank you for highlighting this. We will improve the presentation for the camera-ready version of our paper. In particular, please point us to specific sections that you would like to see improved; we appreciate the suggestions. \n\n---\n> There are plenty of (to my eyes) over-statements and vague statements, depending on what the text requires. E.g.:\n\nThank you for these remarks. We will formulate them more precisely in the camera-ready version of our paper. \n\nRegarding the difficulties of defining joint posterior approximation distribution: We explain in lines 44 and 45 why it is challenging to design good aggregation functions that fulfill all the required properties. Due to space constraints, we do not repeat the statements in section 2 (lines 68, 72-73) or section 3 (line 125). However, reiterating this could improve clarity, and we will include it in the camera-ready version.\n\nRegarding reducing the pairwise similarities: In the first paragraph of section 3, we describe in our dataset assumption that the different modalities of a multimodal sample describe the same underlying phenomenon, making them positive pairs using contrastive learning terminology. We derive in lines 178 and 179 why the regularizer in the proposed objective is equal to minimizing the Jensen-Shannon divergence between M distributions. Hence, we accurately describe what we mean by minimizing the distance between positive pairs. We will add more clarification to the camera-ready version of our manuscript.\n\n---\n> Similarly, conclusions from the experiments look over-optimistic to me. It is true that the proposed method work as- or better than the baselines, but the results are pretty close (see Fig 3) without standard deviations. Also, saying that the samples in Fig. 4d are separated seems like an over-stretch to me (I'd at least add transparency to the points).\n\nFigure 3 shows a clear outperformance of our method compared to previous work. We highlight that the proposed MMVM VAE achieves a higher representation quality or lower reconstruction loss (or both). In the uploaded rebuttal PDF, we also show the outperformance compared to MMVAE+ on the translated PolyMNIST and CelebA datasets. Table 1 also shows that the proposed MMVM outperforms previous works (incl. a supervised method) on the challenging MIMIC-CXR dataset (please see the additional results in the rebuttal PDF).\nWe acknowledge that the performances on the hippocampal neural activities dataset are more comparable across the different methods. We will revise that part of our conclusion accordingly. Thank you for highlighting this.\n\n---\n> Why only independent and PoE were selected for the experiments in Table 1? I understand that MMVAE should have similar complexity to PoE.\n\nAlthough we did not state it explicitly, we reported the results for the PoE VAE to highlight the performance of aggregation-based methods, as it is the best-performing approach in the MIMIC-CXR experiment. However, we understand the reviewers' feedback and will include an updated table with the performance of all baseline methods in the camera-ready version of our manuscript. The updated table is available in the rebuttal PDF.\n\n---\n> PS: I'll happily increase my score once a newer method like MMVAE+ is included, which I find completely necessary given the scope of the paper.\n\nWe added the results of the MMVAE+ to the rebuttal PDF. In the submitted manuscript, we focus on comparing the newly introduced MMVM VAE to independent and aggregation-based VAEs. As the results of the MMVAE+ cannot outperform stronger aggregation-based multimodal VAEs, we did not include them in the evaluation. However, we are happy to include the MMVAE+ results in the camera-ready version of the paper.\n\n---\nThank you again for your feedback. We welcome any additional suggestions, questions, or requests for information and encourage further discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jrnhFPSoLk", "reviewer": "Reviewer_eP7z", "summary": "The paper introduces the multimodal variational mixture-of-experts prior (MMVM) VAE, a new approach to enhance representation learning for multimodal Variational Autoencoders (VAEs). The MMVM VAE overcomes the limitations of the alternatives that parameterize joint posteriors with restrictive assumptions by using a data-dependent prior that softly guides each modality’s latent representation towards a shared aggregate posterior, instead of imposing hard constraints. The experimental analysis on benchmarks and real datasets assesses the quality of the latent representations and preservation of modality-specific information.", "questions": "**Major concerns**\n- I found that Figures 2 and 3 are unclear in demonstrating the benefits of the proposed models due to the following reasons:\n   - Showing several values of $\\beta$ appears irrelevant. Fixing $\\beta = 1$ would likely provide a fairer comparison and clearer results.\n   - In representation learning literature, assessing meaningful representations is a challenging task with many proposed metrics and evaluations. Why do predicting metrics on linear/non-linear classifiers assess the quality of latent representations and coherence, respectively? Without references or explanations for these metrics, their validity in supporting the claims remains unclear.\n\n**Minor concerns**\n\n- I would move the “Encoder and Decoder” paragraph from Section 3 to the beginning of Section 4.\n\n**Found typos**\n\n- Line 98: “patient , .” $\\rightarrow$ “patient.”.", "limitations": "The primary limitation of this work is its inability to perform unconditional sampling, which the authors acknowledge and discuss appropriately. However, while they mention several potential solutions to address this issue, they do not explore any of these alternatives. Efficient unconditional and conditional sampling are among the key advantages of VAEs. Incorporating a solution for unconditional sampling would significantly enhance the paper, allowing the model to generate unconditional samples and fully leverage the strengths of VAEs.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Effective modeling technique: the paper presents an effective modeling technique that enhances representation learning by incorporating inductive biases into the joint prior of the latent space in multimodal VAEs.\n- The contributions are grounded by solid theoretical foundations.", "weaknesses": "- The experimental analysis presented in the main text is not fully convincing. While the evaluation is properly conducted on several datasets, the performance is assessed with only a single quantitative experiment in each case. The extensive analysis provided in the Appendix could be referenced in Section 5 to better support and strengthen the discussion.\n- The paper would benefit from a more specific analysis of the limitations of related work in multimodal VAEs. This would enhance the motivation and context for the proposed approach.\n- The paper does not explore potential solutions to mitigate the inability to perform unconditional sampling. Addressing this limitation would significantly improve the overall contribution and applicability of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer eP7z,\n\nThank you again for your insightful review and feedback. In the following, we will address and answer your comments and questions.  \n\n---\n> The experimental analysis presented in the main text is not fully convincing. While the evaluation is properly conducted on several datasets, the performance is assessed with only a single quantitative experiment in each case. The extensive analysis provided in the Appendix could be referenced in Section 5 to better support and strengthen the discussion.\n\nThank you for the comment. We like the idea of referencing the additional results provided in the appendix. We will include this reference in the camera-ready version of our manuscript. However, we are open to suggestions on what could be moved to the main text to make the core analysis more self-sufficient. \n\nWe assess three different metrics in the main text of the paper: the quality of the learned latent representation, the quality of the conditional generation, and the reconstruction error. We evaluate the quality of the learned representations (Figures 3a-3c, 4a) using the classification performance of a linear classifier. The quality of the conditionally generated samples using pretrained nonlinear classifiers and the reconstruction error using the output negative log-probability of reconstructed samples. We describe the evaluation in more detail in Appendix B.2.\n\n---\n> I found that Figures 2 and 3 are unclear in demonstrating the benefits of the proposed models due to the following reasons:\nShowing several values of beta appears irrelevant. Fixing beta=1 would likely provide a fairer comparison and clearer results.\n\nWe disagree that comparing the different methods concerning multiple $\\beta$ values is irrelevant. Figure 3 clearly shows that the proposed method can outperform previous works regarding the quality of latent representation and conditional generation without trading off reconstruction quality. The $\\beta$-VAE paper [1] and earlier work on multimodal VAEs [2] already showed the regularizing effect of the $\\beta$ parameter regarding trading off the learned representation against the reconstruction quality. Hence, evaluating different methods regarding multiple $\\beta$-values approximates the (whole) range of representation-reconstruction trade-offs, thus showing the models’ robustness concerning $\\beta$.\n\nBased on the feedback, we've created an improved plot version highlighting models with $\\beta=1$ with a star (see rebuttal PDF).\nWhy would a comparison based on a single $\\beta$ value be more fair (in your opinion)? The rebuttal PDF highlights that not all baseline methods have optimal performance at $\\beta=1.0$ and benefit from evaluating a set of different $\\beta$ values. \n\n---\n> In representation learning literature, assessing meaningful representations is a challenging task with many proposed metrics and evaluations. Why do predicting metrics on linear/non-linear classifiers assess the quality of latent representations and coherence, respectively? Without references or explanations for these metrics, their validity in supporting the claims remains unclear.\n\nThank you for highlighting the missing references for our evaluation procedure. Using linear/non-linear classifiers is a standard procedure [3, 4, 5] for assessing learned representations, not only for VAEs. While linear classifiers should highlight the simplicity of the learned representations, non-linear classifiers should indicate the information present in the representations; see also [3, 4, 5]. We will add more references for the evaluation procedure to the camera-ready version of our manuscript.\n\n---\n> I would move the “Encoder and Decoder” paragraph from Section 3 to the beginning of Section 4.\n\nThank you for the suggestion. We like the idea of moving the mentioned paragraph to section 4. We will incorporate the changes in the camera-ready version of our paper.\n\n---\n> The paper would benefit from a more specific analysis of the limitations of related work in multimodal VAEs. This would enhance the motivation and context for the proposed approach.\n\nThank you for pointing this out. Adding a more specific discussion of the limitations of previous works will improve the paper. We will add the following paragraph to the camera-ready version of our paper:\n\nDaunhawer et al. [2022] show that aggregation-based multimodal VAEs cannot achieve the same generative quality as unimodal VAEs and struggle with learning meaningful representations depending on the relation between modalities. If we can predict one modality from another, mixture-of-experts-based posterior approximations perform best if only a single modality is given as input but struggle to leverage the complete information coming from all modalities. In contrast, product-of-experts-based approximations excel if the complete set of modalities is available. However, their difficulty with optimizing unimodal approximations lets them underperform if they only receive a single modality as input.\n\n\n---\n\n> The paper does not explore potential solutions to mitigate the inability to perform unconditional sampling. Addressing this limitation would significantly improve the overall contribution and applicability of the work.\n\nWe would like to highlight that the proposed method is not generative by design, as conditional generation of missing modalities and representation learning are arguably the main use cases of multimodal VAEs. We outline approaches for adding generative capabilities in future work. While presenting a solution for the inability to perform unconditional generation would further strengthen the paper, we see it beyond the scope of this paper. Introducing the MMVM prior distribution and presenting convincing experimental results is a substantial and strong contribution.\n\n---\nThank you again for your feedback. We welcome any additional suggestions, questions, or requests for information and encourage further discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iNQyyLeqU0", "reviewer": "Reviewer_6TPP", "summary": "This paper proposes a multimodal variational mixture-of-experts prior (MMVM) VAE to address challenges in learning representations of multimodal data. Instead of modeling the latent representation as a joint posterior, this work introduces a multimodal data-dependent prior to regulate the latent representation learned from the VAE for each modality. Experiments on multiple benchmarks demonstrate that the proposed method is promising.", "questions": "(1) Regarding the metrics and classifiers mentioned in lines 233-235, are they the same as those defined in [1] and [2]? If yes, please cite them. If not, please provide a more detailed description.\n\n[1] Shi, Y., Siddharth, N., Paige, B., Torr, P.: Variational mixture-of-experts autoencoders for multi-modal deep generative models. In: Advances in Neural Information Processing Systems (2019)\n\n[2] I. Daunhawer, T. M. Sutter, K. Chin-Cheong, E. Palumbo, and J. E. Vogt. On the Limitations of Multimodal VAEs. International Conference on Learning Representations, 2022.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The proposed data-dependent prior is novel and very inspiring, providing a different direction for investigating multimodal representation learning.\n\n(2) The experiments conducted on two medical datasets demonstrate that the proposed methods are beneficial and promising for real-world applications.", "weaknesses": "(1) There is no direct evaluation of the learned representation and the data-dependent prior to support the \"soft-sharing of information between modalities\" proposed by this work. For example, a naive investigation could examine the differences and overlaps among each $H(z_m|𝑋)$, and how similar each learned $q^{m}_{\\phi} (z_m|x_m)$ is to their prior.\n\n(2) In experiments, the exclusion of two multimodal VAE baselines, Daunhawer et al., 2020 and Palumbo et al., 2023, which are mentioned in the related work section, weakens the evaluations of the proposed work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 6TPP,\n\nThank you again for your insightful review and feedback. In the following, we will address and answer your comments and questions.  \n\n---\n> There is no direct evaluation of the learned representation and the data-dependent prior to support the \"soft-sharing of information between modalities\" proposed by this work. For example, a naive investigation could examine the differences and overlaps among each , and how similar each learned is to their prior.\n\nThank you for this comment. We like the idea of directly comparing the latent representations of different modalities. In the table below, we compare the symmetric Kullback-Leibler (KL) divergence between the posterior approximations of the frontal and the lateral view of the MIMIC-CXR dataset to assess the differences between latent representations of different modalities.\nWe compare the KL divergence of the independent VAEs to the KL divergence of the proposed MMVM VAE. Suppose the KL divergence of the MMVM VAE is lower than the KL divergence of the independent VAEs. In that case, it implies that the latent representations align and that the soft-sharing of information between modalities works.\n\nThe table below shows that the symmetric KL divergence of the proposed MMVM is (much) lower than the KL divergence of the independent VAEs. We compute the symmetric KL divergence as\n$$KL = 0.5 \\cdot KL[q(z | x_F) || q(z | x_L)] + 0.5 \\cdot KL[q(z | x_L) || q(z | x_F)]$$\nusing 10000 multimodal samples from the MIMIC-CXR dataset and across two seeds.\n\n| Method | KL divergence | \n| ------------- | -------------------- |\n| Independent | 26693.488 &plusmn; 11558.155 |\n| MMVM | 576.698 &plusmn; 48.882 | \n\n\nIn the manuscript, we support the claim of the soft-sharing of information by evaluating conditional generation coherence. The difference in conditional generation coherence between the independent VAEs and the proposed MMVM is achieved through the different prior distribution between the two methods. Figures 3d-3f show that the proposed MMVM can conditionally generate coherent samples; hence, the soft-sharing of information between the modalities via the proposed prior distribution works. We will emphasize this more in the camera-ready version of our paper.\n\n---\n> In experiments, the exclusion of two multimodal VAE baselines, Daunhawer et al., 2020 and Palumbo et al., 2023, which are mentioned in the related work section, weakens the evaluations of the proposed work.\n\nWe show the results for the MMVAE+ in the rebuttal PDF. We excluded the results in the submitted manuscript because the more complicated MMVAE+ method could not outperform the aggregation-based baselines used in our manuscript.\n\n---\n> Regarding the metrics and classifiers mentioned in lines 233-235, are they the same as those defined in [1] and [2]? If yes, please cite them. If not, please provide a more detailed description.\n\nThank you for highlighting the missing references regarding the evaluation procedures. We use the same metrics as in [2, 3,6] and will add the corresponding references to the manuscript.\n\n---\nThank you again for your feedback. We welcome any additional suggestions, questions, or requests for information and encourage further discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to thank all reviewers for providing comprehensive and valuable feedback. We particularly value the reviewers' recognition of our effective, interesting, and inspiring idea [**56cj**, **eP7z**, **6TPP**], the solid theoretical framework [**56cj**, **eP7z**], and the strong results from MMVM [**56cj**, **6TPP**].

Your thorough reviews have helped us identify potential areas for improvement and gain valuable perspectives that strengthen the contribution of our research. We have carefully addressed each of your comments. Based on the feedback [**56cj**, **eP7z**, **6TPP**], we include additional results and experimental evaluation in the rebuttal PDF, confirming the strong performance of the proposed MMVM VAE.

We include the results for MMVAE+ as an additional baseline [**56cj**, **6TPP**], which were excluded in the submitted manuscript as the MMVAE+ could not outperform other aggregation-based multimodal VAEs.
To address the feedback regarding our evaluation procedure using multiple $\beta$ values [**eP7z**], we specifically show the performance of the models, which were trained with $\beta=1$, and discuss their performance differences.
For the MIMIC-CXR experiment, we include more aggregation-based multimodal VAEs, which we did not include in the submitted manuscript [**56cj**] as we only showed the performance of the best-performing aggregation-based multimodal VAE.

We welcome any additional suggestions, questions, or requests for information and encourage further discussions.
Once again, we thank the reviewers for their time and effort in evaluating our work.

-----
## References

[1] Higgins et al., “Learning Basic Visual Concepts with a Constrained Variational Framework”, ICLR 2017

[2] Sutter et al., “Generalized Multimodal ELBO”, ICLR 2021

[3] Shi et al., “Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models“, Neurips 2019

[4] Oord et al., “Representation learning with contrastive predictive coding”, 2018

[5] Chen et al., “A Simple Framework for Contrastive Learning of Visual Representations”, 2020

[6] Daunhawer et al., “On the Limitations of Multimodal VAEs”, ICLR 2022

Meta-review: The reviewers acknowledge the soundness and the technical novelty of the paper. The rebuttal has a positive impact on the final ratings and all the reviewers scored weak accept or higher. The ACs followed the reviewers' recommendation.
Decision: Accept (poster)