Score-Based Multimodal Autoencoder
Authors: Daniel Wesego
Pedram Rooshenas
Venue: Accepted by TMLR ()

Abstract: Multimodal Variational Autoencoders (VAEs) represent a promising group of generative models that facilitate the construction of a tractable posterior within the latent space given multiple modalities. Previous studies have shown that as the number of modalities increases, the generative quality of each modality declines. In this study, we explore an alternative approach to enhance the generative performance of multimodal VAEs by jointly modeling the latent space of independently trained unimodal VAEs using score-based models (SBMs). The role of the SBM is to enforce multimodal coherence by learning the correlation among the latent variables. Consequently, our model combines a better generative quality of unimodal VAEs with coherent integration across different modalities using the latent score-based model. In addition, our approach provides the best unconditional coherence.

Published in Transactions on Machine Learning Research (12/2024)
Score-Based Multimodal Autoencoder
Daniel Wesego
dueseg2@uic.cc edu
Deportment of Computer Science University of Mlinois Chicago
Pedram Rooshenas
pedrom Quic. edu
Department of Computer Science University of Ilinois Chicago
Reviewed on OpenReview: https: // openreview. net/ mett forum? id- JbuP6UV3Fk
Abstract
Multimodal Variational Autoencoders (VAEs) represent promising group of generative models that facilitate the construction of tractable posterior within the latent space given multiple modalities. Previous studies have shown that as the mumber of modalities increases, the generative quality of each modality declines. In this study, we explore an alternative approach to enhance the generative performance of multimodal VAEs by jointly modeling the latent space of independently trained unimodal VAEs using score-based models (SBMs). The role of the SBM is to enforce multimodal coherence by learning the correlation among the latent variables. Consequently, VAEs with coherent integration across different modalities using the latent score-based model. In addition, our approach provides the best unconditional coherence. The code can be found at
model combines better generative quality of unimodal
1 Introduction
The real-world data often has multiple modalities such as image, lext, and audio, which makes learning from multiple modalities an important task. Multimodal VAEs are class of multimodal generative models that are able to generate multiple modalitics jointly. Learning from multimodal data is inherently more challenging than from unimodal data, as involves processing multiple data modalities with distinet characteristics.
In order to learn the joint representation of these modalities, previous approaches generally preferred encode them to latent distribution that governs the data distribution across different modalitics. In general, expect the following properties from multimodal generative model:
Multiway conditional generation: Given the presence of certain modalities, it should be feasible to generate the absent modalities based on the existing ones (Shi et al.) 2019 Wu & Goodman, 2018). The conditioning process should not be limited to specific modalities; rather, any modality should serve as a basis for generating any other modality.
Unconditional generation: If we have modality present to condition on, we should be able to sample from the joint distribution that the generated modalities are coherent (Shi et al. 2019). Coherence in this case means that the generated modalities represent the same concept that is expressed in the different. modalities we have.
Conditional modality gain: When additional information is provided to the model through the observation of more modalities, there should be an enhancement in the performance of the generated absent modalities. In other words, the performance should consistently improve the number of observed (given) modalities increases.

---

Published in Transactions on Machine Learning Research (12/2024)
Scalability: The model should scale as the number of total modalities increases. Moreover, the model complexity shouldn't become computationaly inefficient when adding more modalities (Sutter et al. 2020
Other properties like joint representation where we want a representation that takes into account the statistics and properties of all the modalities (Srivastava & Salakhutdinov 2014 Suzuki et al., 2017 can be easily learned from a multimodal model obtaining the above properties. Another property is weak supervision where we make use of multimodal data that isn't paired together (Wu & Goodman 2018
Naively using multimodal VAEs by training all combinations of modalities becomes easily intractable as the number of models to be trained increases exponentially. We will need to train 2M -1 different combinations of models for each subset of the modalities. Since this is not a scalable approach, previous works have proposed different methods of avoiding this by constructing a joint posterior of all the modalities. They achieve this by modeling the joint posterior over the latent space z: q(z)X1:M)) where X1:M is the set of modalities. To ensure the tractability of the inference network q, prior works have proposed using a product of experts (Wu & Goodman 2018), a mixture of experts (Shi et al. 2019), or in the generalized form, a mixture of the product of experts (MoPoE) (Sutter et al. 2021 1; among others.
After selecting how to fuse the posteriors of different modalities, these approaches then construct the joint multimodal ELBO and use it to train multimodal data. At inference time, they use the modalities that are observed to generate the missing modalities. Wu & Goodman (2018) intentionally adds ELBO subsampling during training to increase the model's performance on generating missing modalities at inference time. Mixture of experts and mixture of products of experts subsample modalities as part of their training process because the posterior model is mixture model. Subsampling of the modalities, as pointed out by DDunhawer ct al. 2022), results in generative discrepancy among modalitics. We also observe that conditioning on more modalities doesn't t improve the performance of the generated modality on some multinodal VAEs which goes contrary to one's expectation. As a model receives additional information, it should perform better or should show no decrease in performance. Daunhawer et al. (2022) further concludes that these models cannot. be useful for real-world applications at this point due to these failures.
To overcome these issues, instead of constructing a joint posterior, we try explicitly model the joint latent space of individual VAEs: ()()).)) The joint latent model learns the correlation among the individual latent space in separate score- model trained independently without constructing a joint posterior as the previous multimodal VAFss. The joint model can ensure prediction coherence by sampling from the score model while also maintaining the generative quality that is close to a unimodal VAE. And by doing so, we avoid the need to construct a joint multimodal ELBO. We use the independently trained unimodal VAFs to generate latent samples, and use those samples to train a score network that models the joint latent, space. This approach is scalable as it only uses M unimodal VAEs and one score model. Unconditional generation from the joint distribution can also be done by sampling from the score model respecting the joint distribution. Conditional generation is also possible by fixing the observed modalities and generating the rest. It also avoids the subsampling problems discussed by Daunhawer et al. (2022) as it doesn't 't need to train a joint multimodal ELBO. In addition to that, it provides the best unconditional results compared with the baselines while at the same time having a competitive conditional performance. Figure [ [describes the overall architecture of the model.
Our contributions include 1) we propose novel generative multimodal autoencoder approach that satisfies most of the appealing properties of multimodal VAEs, supported using extensive experimental studies. 2) We introduce coherence guidance to improve the coherence of observed and unobserved modalities. 3) Our experimental result demonstrates that the proposed model performs well unconditionally and conditionally and, in addition, exhibits enhanced robustness against adversarial perturbations compared to baseline methods.
2 Methodology
Assuming the each data point consists of M modalitios: XIMM- (x1, X2, XM), our latent variable model describes the data distribution as p(X1:a)) = = p(x ( (PZ..M) where Z1:M5 = (Zz, 22. Zxz) and 2g is the latent vector corresponding to the kth modality. In contrast to common multimodal VAE setups
2

---

Published in Transactions on Machine Learning Research (12/2024)
Brown. Hair
Heavy,
Mouth, Sightly, Open
Reced ng_ Heirliee
Smiling Straight_Hait
Image Decoder
Mask Decoder
Attribute Decoder
SBM
Zi
m
Za
Image Encoder
Mask Encoder
Attribute Encoder
Heavy Makeup Mouth. www.h_liggtpppppee
Heavy_Makeup
Receding, Hairline Sm ling
Stroight_Hair
Figure 1: A variational regularized auto-encoder will be used for each individual modality to get the latent representation which then will be used to train the scoro-based model which will allow the prediction of any modality given some or none. The auto-encoders are trained independently in the first stage and the respective of each modality will be used to train the score network.
[Wu & Goodman modalities, and we assume each latent variabld Zy only captures the modality-speeific representation of the corresponding modality k. Therefore, the variational lower bound on log p(x1.M) can be written as:
2018 Shi et al. 2019), we don't consider any shared latent representation among different
1oga(X):a)) 2 (o(Raaa/xaa)) log p(x Z1:M (((EM)
(1)
9 (Z X1.Mr)
To simplify the joint generative model ((xim/ZzaM)) and the joint recognition model q(Z1:M/ (1:M), we assume
two conditional indenpendencies: 1) Given an observed modality Xk, its corresponding latent variable Zą
is independent of other latent variables, i.e., Xş have enough information describe Zạ: Z 1 Z- x 2)
Knowing the latent variable 2ạ of kth modality is enough to reconstruct that modality: X 1. X- Z Zk. Using these conditional independencies the generative and recognition models factorize as:
M
M
P(XIMM (Z:M) = II p(xulz)) and / = II -=1
(Z Xxx)
(2)
Therefore we can rewrite the variational lower bound as:
logp(x1n>)) 2 Σ Bgs, (na (xc) log Per (xxlzz) DKL I aou xxx (p
(3)
where &e and t are the parameterizations of the recognition and generative models, respectively, and 0 parameterizes the prior. If we assume the prior po(21:M) factorizes as Ihe Po, (æ&), then the variational lower bound of 1(x becomes Ee ELBOR, where ELBO, is the variational lower bound of individual modality. However, such an assumption ignores the dependencies among latent variables and results in a lack
For simplicity "variable" refer the group of variahles that describe latent representation modality
3

---

Published in Transactions on Machine Learning Research (12/2024)
of coherence among generated modalities when using prior for generating multimodal samples. To benefit from the decomposabi ELBO but also benefit from the joint prior, we separate the training into two steps. In step 1, we maximize the ELBO with respect to 6 and to assuming prior p(Z).M) = (I, which only regularizes the recognition models. In step II, we optimize ELBO with respect to assuming a joint prior over latent variables. Therefore, in step II maximizing the ELBO reduces lo: ming DKL (IIx tos (2x xxxxxx) (ppg and since recognition model is constant w.r.t. 8, the step II becomes maxØ pp(x) B II ((me/xx) log Po (0(:)))))
which is equivalent to maximum likelihood training of the parametric prior using sampled latent variables for each data points. And during inference, we only need samples from pa (B(M)))) thus we parameterize sp(Z1:M0 = V VRnAr log(p(Z):r)) as the score model and train sa (z(M) using score matching (Hyvärinen & Dayan 2005). In practice, the score matching objective does not scale to the large dimension which is required in our sctup, and other alternatives such as denoising score-matching (Vincent 2011] and sliced score-matching (Song et al. 2020ah have been proposed.
Here we use denoising score matching in the continuous form that diffuses the latent samples into noise distribution and can be written in stochastic differential equation (SDE) form (Song al. 2020b). The forward process will be an SDE of dz [z, 1)dl + g(O)w where is the Brownian motion. To reverse the noise distribution back to the latent distribution, we use the trained score model and sample iteratively from the reverse SDE dz = [f v(t)3v, log (x() dt + g())dW. The score network which approximates Vz (z) is trained using eq. We use the unweighted version where X(t) =.
0r = arg minE, (0)) E2(
[| (]so(((0))) - Vz(e) log (0((()((())))) 20))
(4)
2.1 Inference with missing modalities
The goal of inference is sample unobserved modalities (indexed by u S (1,.... M)) given the observed modalities (indexed by (1,... ,.)) .M)) from p(xul>x). We define variational lower bound log-probability
log Dogp(xxlee (x.) using posterior q(zu|xo) the latent variables of the unobserved modalities Zu:
log (xx])x)) log Be (xxu/xxXxx>>>>>
Xo: Zu)
Eg(uulxo) log p(xu Xo: Zu)
[x.l/x))) log
(5)
where p(x, = Il AikuuP Pvr. (xxl (x)) and Pvy (xx/zx) is the generative model for modality k. We define the posterior distribution q(zu|xo) as the following:
q Xo) = Σ
go. (zi|xi) II Lico
po po (zo),
(6)
where 4or (zz) xx) is the recognition model for modality k.
In order to sample from q(zul following eq. 6 we first sample zo for all observed modalities, and then sample 2n from Po (z (o)) by fixing zo which is the latent representation of the observed modalities and updating the unobserved ones during sampling. When all modalities are missing, we will update all modalities in the sampling step.
A After rumning the sampling for T steps we use the generative models p(x, (.) IIxe Pve (xx Pr,, the unobserved modalities. We use the Peddictor-Correttor (PC) sampling algorithm (Song et al. 2020b)
(zx) to sample
which is mix of Euler-Maruyama and Langevin Dynamics (Welling & Teh 2011 to sample from the score model.
2.2
Coherence guidance
2.2.1
Energy-based coherence guidance
The score- bcore-based model improves the coherence among predicted modalities, however, when the number of unobserved modalities increases, it is more likely that. the predicted modalities are not aligned with the
4

---

Published in Transactions on Machine Learning Research (12/2024)
observed modalities. To address this issue, we extra conditional guidance during the reverse process for sampling (Dhariwal& & Nichol 2021]. In particular, train an energy-based model that assigns low energy to a coherent pair of modalities and high energy to incoherent ones. We define the energy-based model (zo, zu) over the latent representations of an observed modality 2, and an unobserved modality Zu- Here we assume that the latent. representation of all modalities has the same dimension but in general, it can work with any dimension. During training, we randomly select two modalities (Xo; x.) as a positive pair and substitute the second modality with an incoherent example X in training data to construct a negative pair Xn). We train E using noise contrastive estimation (Gutmann & Hyvärinen 2010):
max (e w
log
(7)
1 + exp (Ew ((zo, zu)))
1
(1 log
1 exp (-Ew kn))
We also perturb the latent representations with the same perturbation kernel used to train the score model. The final score function with coherence guidance has the following form:
8(2x) so(2a) - VV Eu zu),
(8)
where controls the strength of guidance. During each step of inference, we randomly select To from observed modalities and randomly select Iu from unobserved modalities and update the score of zu using eq. g
2.2.2 Contrastive guidance
Another way to enhance the conditional performance of the score-based model is to leverage contrastive guidance from an auxiliary encoder work F(x).. This encoder maps the input data x from all modalities to common embedding space, enabling conditional alignment through contrastive learning objectives (Rombach ct al. 2021 Tang et al. 2023). The resulting embeddings capture semantic information that can guide the score- based model during the sampling process.
Specifically, we incorporate secondary encoder embeddings E(x) as an additional conditioning input to the score model (z(z(t), E(x)). E(()). The model can use this aligned representation which will be useful in providing guidance. We show this method supportive as an additional guidance method on the datasets in the experiments section. Compared to the energy-based coherence guidance, which is trained independently of the score model and added during sampling time, this method requires the secondary encoders be present when training the score model. We train the secondary encoders, E(x), using contrastive learning similar to Radford et al. (2021) h where the similarity between representations from the same pairs of data are maximized and the similarity between representations from unrelated pairs are minimized.
3 Related Works
Our work heavily inspired by earlier works on deep multimodal learning (Ngiam et al. 2011 Srivastaval & Salakhutddino 2014). Ngiam et al. (201T) use a set of autoencoders for each modality and shared representation across different modalities and trained the parameters to reconstruct the missing modalities given the present one. (Srivastava & Salakhutdinov (2014) define deep Boltzmann machine to represent. multimodal data with modality-speeific hidden layers followed by shared hidden layers across multiple modalities, and use Gibbs sampling to recover missing modalities.
Suzuki et al. (2017) approached this problem by maximizing the joint ELBO and additional KL terms between the posterior of the joint and the individuals to handle missing data. Tsai et al. (2019) propose a factorized model a supervisod setting over model-spocific representation and label represcntation, which capture the
shared information. The proposed factorization q(Z1:M 2ylxi.as.)) = g(2,) X1.M) I q(x((x)), where zy is the latent variable corresponding to the label.
5

---

Published in Transactions on Machine Learning Research (12/2024)
Most current approaches define multimodal variational lower bound similar to variational autoen- coders Kingma & Welling (201)) using shared latent representation for all modalities:
logp(X1:a) E. (a) (logoo(x..lz)) DKL( (x
(9)
Similar to our setup Po (x IIe Per Por (xx|z), but qo (z(2xxi:M) is handled differently. Wu & Goodman (2018) use a product of experts to describe q: groz (Z/X1:M) ) = p(z) I qe, (z|xx)- Assuming p(z) and cach of go, (z)(())))) xx. follow a Gaussian distribution, (PoR can be calculated a closed form, and we can optimize the
multimodal ELBO accordingly. To get a good performance on generating missing modality, [Wu & Goodman (2018) sample different ELBÖ combinations of the subset of modalities. Moreover, the sub-sampling proposed
by Ww & Goodman (2018) results in invalid multimodal ELBO (Wu & Goodman 2019). The MVAE proposed by (Wu & Goodman 2018) generates good-quality images but sulfers from low cross-modal coherence, To address this issue Shi et al. (2019) propose constructing (s(z)x as a mixture of experts: (MoR(2/x1:M) = t Ex Idh (z|xx). However, as pointed out by Daunhawer et al. (2022), sub-sampling from the mixture component results in lower generation quality, while improving the coherence.
Sutter et al. (2021) propose a mixture of the product of experts for q by combining these two approaches: N" Ex.. (z))xm), where q(z)xm) II g((|xx) and m is a subset of modalities. the number of modalities increases. MoPoE
[M
The number of mixture components grows exponentially has better coherence than PoE, but as discussed by Daunhawer et al. (2022) sub-sampling modalities in mixture-based multimodal VAEs result in loss of generation quality of the individual modalities. To address this issue, more recently and in parallel to our work, PPalumb et al. (2023) introduce modality- specific latent variables in addition to the shared latent variable. In this setting the joint probability p(z) II p(xx/z, (s))(Ws)) and q factorizes as (MMVAE+ (z, W WI:ar ( 2o. (z|x) II fou (w& (xx). Using modality-specile
model over all variables factorizes as p(XI:M, z, WI:M)
representation is also explored by Lee & Pavlovic (021], however, the approach proposed by Palumbo et al. (2023) is more robust to controlling modality-speciff representation vs shared representation. But since the shared component is a mixture of experts of individual components, it can only use one of them at time during inference which limits its ability to use additional observations that are available as the number of given modalities increase.
Sutter et al. (2020) propose an updated multimodal objective that consists a JS-divergence term instead of the normal KL term with a mixxure-offexxpere posterior. They also add additional modality-specifn terms and a dynamic prior to approximate the unimodal and the posterior term. Though these additions provide some improvement, there is still need for a model that balances coherence with quality (Palumbo al. 2023).
Wolff et al. (0022) propose a hierarchical multimodal VAËs for the task where a generative model of the form p(z, g, 2) and an inference model 4(9, 2 (1:as)) containing multiple hicrarchics of where g holds the shared structures of multiple modalities a mixlure-of-experts form (gg]=rr)) They argue that the modality-exclusit hierarchical structure helps in avoiding the modality sub-sampling issue and can capture the variations of cach modality. Though the hicrarchy gives some improvement in results, the model is still restricted in capturing the shared structure in g discussed in their work.
Suzuki& & Matsuo 2023) introduce a multimodal objective that avoids sub-sampling during training by not using a muixture-f-expeer posterior to avoid the main issue discussed by Daunhawer et al. (2022). They propose a product-of-experts posterior multimodal objective with additional unimodal reconstruction terms to facilitate cross-modal generations.
Hwang al. 2021 propose an ELBO that is derived from an information theory perspective that encourages finding representation that decreases the total correlation. They propose another ELBO objective which is a convex combination of two ELBO terms that are based on conditional VIB and VIB. The VIB term decomposes to ELBOs of previous approaches. The conditional term decreases the KL between the joint posterior and individual posteriors. They use product-of-experts as their joint posterior.
Finally, parallel to our work, Xu et al. (2023) propose diffusion models to tackle the multimodal generation which uses multi-flow diffusion with data and context layer for each modality and shared global layer. Also, concurrent to our work, Bounoua et al. (2024) proposed latent diffusion model for multimodal data using
6

---

Published in Transactions on Machine Learning Research (12/2024)
score-based diffusion model. They autoencoders to project the modalities into latent space and train joint model similar to ours while training the score model differently by masking some of the modalities in the diffusion process to facilitate conditional performance. [Tngg et al. (2023) introduced any-to-any generation, but contrast to our work, their approach involves training a separate latent diffusion model for each modality. Diffusion models are then conditioned on an aligned latent space (they have used lext modality as the anchor) for conditional generation and are trained jointly to offer joint modality prediction. Our approach is based on learning joint latent space and avoiding aligument learning common latent space, while their approach is based learning an aligned latent space for conditioning independent modality generation, which is highly dependent on finding the aligned latent space for different modalities.
C
2
3
1
4
4
4
6
7 8
7
8
(a) SBM-VAE (b) SBM-RAE (c) MMVAE+ (d) MVTCAE
(e) MoPoE
(I) MMVAE
(g) MVAE
Figure Multiple conditionally generated samples for cach digit from the third modality. Each column shows samples, from to generated conditionally given the remaining modalities.
4
Experiments
We study our proposed methods and selected baselines using extended version of PolyMNIST (Sutter et al. 2021) as well as high-dimensional CelebAMask-HOC, (Lee et al. 2020 datasets. Extended PolyMNIST is idcal for observing the performance patterns in the presence a different number modalities, which is also used by previous works, while CelebAMMskk- HQ tests the behavior of the methods in high-dimenssonal setting. the CelebAMask-HHC demonstrate that
2
2
2
2
2
2
3
7
4
4
4
4
7
3
3
3
3
3
3
3
O
0
7
0
O
The experiments multimodal VAFs, in general, can be used to solve structured prediction problems. Finally, we study the robustness of mul- timodal VAEs in the presence of adversarial attacks. Other
S
5
5
5
5 5
5
Figure 2: Extended PolyMnist Dataset
experiments including an audio modality are available the Appendix section A.6
4.1 Extented PolyMnist
The original PolyMNIST introduced by /Sttter et al. (2021) has five modalities and order to study the behavior of the methods on a larger number modalitics, we extended the number of modalitics ten. Figure shows samples of the Extended PolyMNIST data.
We compare methods SBM-VAE and SBM-RAE, which substitute individual variational autoencoder with regularized deterministic autocncoder [Ghosh et al. 2020); see Appendix A.1 for the details of SBM-RAE, as well as their counterparts with energy-based coherence guidance SBM-VAE-C and SBM-RAE-C plus SBM- VAE-T which uses contrastive guidance with baselines of MVAE (Wu & Goodman 2018 MMVAE Shi et al. 2019); MoPoE. (Sutter al. 2021). MVTCAE (Hwang et al. 2021), and MMVAE+ (Palumbo al. 2023). We added SBM-RAE to show that our approach can be used
with any auto-encoder structure and is not limited to only variational auto encoder, unlike the baselines.
7

---

Published in Transactions on Machine Learning Research (12/2024)
Hyperparameters and neural network design are discussed in detail in Appendix A.2 Conditional generation is done for MVAE and MVTCAE using PoE of the posteriors of the given modalities, and for the mixture models, a mixture component is chosen uniformly from the observed modalities.
We evaluate all methods on both prediction coherence and generative quality. To measure the coherence, we use a pre-trained classifier to extract the label of the generated output and compare with the associated label of the observed modalities Shi al. (2019). The coherence of the unconditional generation is evaluated by counting the number of consistent predicted labels from the pre-defined classifier. We also mcasure the generative quality of the generated modalities using the FID score (Heusel et al. 2017 All the results that are shown are run for at least 3 times and the mcan is shown. The standard deviation is shown as shades under the curves in each figure. Figure 3 shows the generated samples from the third modality given the rest. The SBM models generate high-quality images with considerable variation, very close to the original data, while most of the baselines generate more blurry images with a lower variation.
Unconditional generation. We first study unconditional generation. Showws the generated samples using SBM-VAE. For the full sample sets for all baseline methods please see Figure 26 in Appendix. We evaluate unconditional coherence by counting the number of consistent labels across different modalities after classifying the generated images using a pre-trained classifier. In other words, the unconditionally generated images should represent the same number expressed in the different modalities. Since no modalities have been observed, we don't use coherence guidance for unconditional generation. The results have been reported in Figure 5 Approximately 71% of the generated images using SBM models are coherent which is considerably better than the other baselines. The coherence percentage gap between SBM models and the next best model, MMVAE, is much larger for nine out of ten, 92.5% vs 74.9%, respectively. To ensure that our models' high performance is not due to mode drop, we generate 10000 unconditional samples and calculate the mode coverage for all digits. Our methods and MMVAE+ uniformly cover all the models, while MVTCAE coverage is not uniform. See Figure 20 in Appendix for mode coverage.
2
3
0
9
8 8
4
8
Further, we study the scalability of methods by changing the number of trained modalities from two to ten and then evalu- ating unconditional FID. Similar to unconditional coherence, for unconditional FID we don't use coherence guidance. The outcome is shown in Figure [ The performance of our proposed methods is consistent and superior in the presence of different numbers of data modalities.
9
2
3
9
7
3
2
9
2
0
9
?
9 9
8
8
2
9
2
3
0
8
8
2
9
Z
3
0
8
9
8
9
7
3
O
8
9
8
Conditional generation. We run similar experiments to study the behavior of the methods for conditional generation. In the conditional generation, we assume we are generating the first modality given the rest. The selection of the first modality is to remain consistent as we increase the number of modalities from two to ten. Figure 7 shows the oulcome. The SBM models achieve the best performance in generation quality in both conditional and unconditional generations as we scale the nunber of modalities. This shows that our approach can scale to a high number of modalities without compromise. Specifically, the SBM- VAE variants show consistent FID and coherence, for which the complexity of the sampling from prior remains the same as we increase the number of data modalities.
9
2
3
6
A
/
9
6
8
2
4
4
8
8
2
X
9
2
3
0
9
7
8
8
2
9
2
3
0
8
9
7
8
Figure 4: Unconditional samples from 10 modalities using SBM-VAE, Columns repro- sent different samples from each modality.
We also study, how the conditional coherence and FID change with the number of the observed modality in single model trained with 10 modalities. For an accurate conditional model, we expect as we observe more modalities, the accuracy of conditional generation will improve without decrease in quality. Figure shows the FID score of the last modality as increase the given modalities and Figure 9 demonstrates the conditional coherence of the predicted images on the last modality given a different number of observed modalities. Our models show consistent FID while also having increasing accuracy. The figure shows the
8

---

Published in Transactions on Machine Learning Research (12/2024)
200
180
10
Unc Coherence
0.8
160
140
0.4
100
0.2
S0P AVA
80
MANAE
0.0
Number of Simil ar Modalities
Number of Trained Modalittes
Figure 5: Unconditional coherence. The x-axis shows the number of coherent modalities and the y-axis shows the percentage of such coherent predicted modalities in the generated output
Figure 6: Average unconditional FID the mum- ber of modalities the models are trained on in- crease as shown on the x-axis
180
160
140
Accuracy
100
80
60
Number Trained Modallties
Number of Trained Modalities
(a) Conditional FID
(b) Conditional accuracy
Figure 7: a) Conditional FID and b) Conditional accuracy of the generated first modality given the rest of modalities the number of modalities the models are trained on increases as shown on the x-axis.
deterioration in quality quantified by increasing FID more modalitics observed for the compared methods except for the SBM models and MMVAE+. SBM- multimodal VAEs and MMVAE+ both are equipped with way to capture modality-speciffc representation, so increasing the number of observed modalities does not affect their generation quality. The SBM exploits the additional modalities to generate better and more accurate images as we increase the number of modalities without losing quality. Similar experiments have been reported for predicting the first modality in the Appendix section (see Figure 12 and
Figure 13
The plain score model suffers from low coherence when have only two observed modalities out of the ten. But with the help of the coherence guidance, and that helps increase the coherence of SBM-VAE and SBM-RAE when the number of observed modalities is low. In addition to that, increasing the number of sampling steps from 100 to 1000 also helps when a few modalities are observed. We demonstrate the effect of coherence guidance and the number of inference steps in Figure [1 The SBM-VAF-T model also shows a high coherence even when the number of observed modalities low. Further details as well more generated samples and more results on other modalities have been reported in Appendix A.4
guide the model towards the correct output

---

Published in Transactions on Machine Learning Research (12/2024)
200
050 -00-
1.0
175
Accuracy
MiK
125
0.6
SENONE
100
SEMAKO
75
1
Number Observed Modalities
Modalities
Figure The conditional accuracy of the last modality generated by incrementing the given modality at time. The x-axis shows how many modalities are given generate the modality and the y-axis shows the accuracy of the generated modality.
Figure 8: The conditional FID of the last modality generated by incrementing the given modality at a time. The x-axis shows how many modalities are given to generate the modality and the y-axis shows the FID score of the generated modality.
Syshome
Vide-
Moark Highty
Opee
Jewes It.r
Harry Makio
Eotrs Niboo
Dlind Hur Moank Siight..
Mirah Skigt
Opci Mave Haa
Fach Way. Tax
Shelii
Sundl
Vill
Mill
Open
beer Mikrop
Stmegh
2bck
Malcpp
-Ops
Ober
Figure 10: Unconditional generation using SBM-VAE, Left) images, Center) masks, and Right) attributes.
1.0
Accuracy
SENAOE-C TENARI
0.2
L8MANN-CAA
Number of Observed Modalities
Figure 11: Effect of having the energy-basel coherence guidance model and number of sampling steps
4.2 CelebAMask-HO
The images, masks, and attributes of the CelebAMask-HO datasct can be treated as different modalities of expressing the visual characteristies of person. sample from the dataset is shown in Figure [ The images and masks resized to 128 by 128. The masks either white black where all the masks given
10

---

Published in Transactions on Machine Learning Research (12/2024)
in the CelebAMask-HO except the skin mask are drawn top of each other as one image. We follow the pre-processing of Wu & Goodman (2018) by selecting 18 out of 40 existing attributes as described.
We compare SBM- SBM-VAE-C, SBM-VAE-T, SBM-RAE, SBM-RAE-C, SBM-RAE-T from the SBM
variants and MoPoE by Sutter et al. 2021 MVTCAE by Hwang et al. (2021). MMVAE+ by Palumbo
jet al. (022), and ML.D by [Bonnua et al. (2024) as baselines. See Appendix A.9 for the experimental setups of these methods.
We evaluate the generation quality of the image modality using FID score and generation accuracy of mask and attribute modalities using sample-average F1 score. Table I shows how our methods compare with the baselines in the presence of one or two observed modalities. We have also reported the performance of a supervised model trained to predict altributes and masks directly given the images. Table 2 shows the unconditional image generation performance the models. SBM-VAB and SBM-RAE variants generate high-quality images compared to the basclines both conditional and unconditional settings, while on the mask modalities, MoPoE and MVTCAE achieve a better Fi score on mask prediction. We can note that the contrastive guidance is very helpful and the rop-performing model when predicting the attributes conditionally, We can also observe that conditional guidance consistently improves the coherence of score-based models. The SBM models achieve superior performance when predicting the image modality consistently and also perform comparatively or better in the other modalities, showing the effectiveness of the method in different. modalities.
Finally, we show the unconditional generation across different modalities using SBM-VAL in Figure 10 Please see Appendix A.10 for more experiments and qualitative samples the CelebAMask-HI dataset.
4.2.1 Robustness to Adversarial Attacks
In this section, we study how multimodal VAEs and SBM-VAEs perform in the presence of blackbox adversarial attack. We choose blackbox attack because the inference algorithm for SBM is not end-to-end differentiable. We apply the blackbox method by Papernot et al.] (2017) to generate the adversarial examples using a supervised model that classifies the attributes from the images of the CelebAHQ dataset and then feed these perturbed images to both multimodal VA based on MoPoF, MVTCAF, MMVAF+ as well as our SBM--VAE- and the contrastive guided SBM-RAE-T. The perturbed images are created using the Fast Gradient Sign Method (FGSM) Goodfellow et al. (0155) with e=0.55. We then use the adversarial images instead of the original images and prediet the attributes.
As reported in Table 3 SBMVAE-C is much less affected by the adversarial attacks as its performance decreased by approximately 4% while MoPoE's and MVTCAE's performance dropped by 19.1% 1% and 13.7%, respectively. This shows that our method is more robust to blackbox adversarial attacks. The robustness of SBM-basod VAEs because of the inference procodure which causes purification Nie et al. (2022) on the latent representation of the perturbed modality, by mapping back the noisy latent representation to the original latent manifold. On the other hand, SBM-RAF-T is affected the most the secondary conditioning encoder directly takes the images which will be affected when given an adversarial input.
5 Conclusion and Discussion
Multimodal VAEs are an important tool for modeling multimodal data. In this paper, we provide a different multimodal posterior using score-based models. Our proposed method learns the correlation of latent spaces of unimodal representations using a joint score model in contrast to the traditional multimodal VAE ELBO. We show that our method can generate better-quality samples and, at the same time, preserve coherence among modalities. We have also shown that our approach is scalable to multiple modalities. We also introduce coherence guidance to improve the coherence of the generated modalities. The coherence guidance effectively improves the coherence of our score-based models. In conclusion, while our score-based multimodal VAE approach offers certain advantages, it has a trade-off. Unlike traditional multimodal VAEs, our model requires a computationally expensive sampling procedure during inference. This ultimately forces a balance between the quality and coherence of predictions and the computational resources needed. This limitation highlights area for future research. Broader Impact: We believe this model can be used in real-world applications
11

---

Published in Transactions on Machine Learning Research (12/2024)
Table 1: CelebAMask-HH generation performance for each modality. Image quality is measured using FID score, and the quality Mask and Attribute modalities are measured using Fi score. The second row indicates the observed modalities.
Attribute
Image
Mask
GIVEN
Both
Img
Both
Mask
Attr
Both
Img
F1
F1
FID
FID
FID
F1
SBM-RAE
(200003)
0.6
81.9(10.19)
(6.4(20.02)
85.6(20.48)
(.83(20.001)
0.82(20.001)
SBM-RAE-C C
(10.003)
(.64(400084)
83.6(20.05)
82.8
83.1(=0.44)
(2..04)
(.82(20.004)
SBM-RAE-T
0.77 (#0.001)
(t0.000)
83. (3:00000
82.6(20.45)
00.20=)
0.8 (20.00))
0.84(=0.001)
SBM-VAE
0.62 (10.003)
(.58(20.005)
81 81.6(10010)
81.9
78. (.20))))
0.83 (200000)
(2.00))
SBM-VAE-C
+0.005))
0.66(+
82.4(+0.1)
81.7 (++.29)
(+.77)
(.840)))00)
0.84(=0.001)
SBM-VAF-T
(.70(10-001)
(20.022)003)
80.9
83.0(20.02)
73.8: (22.11)))
0.8 (220000)))
(20002))001)
MLD
0.71 (+0.005)
(+0.00))
81.7(+0.25)
82.4
80, (+0.6)
0. (6
=0.00))))
MoPoE
(8100002)
0.7 (20.0004
114.9(10.52)
(01.1
(80.7()
(2850200002)
0.92(1:0.00
MVTCAE
0.7 0.71 (11)001)
0. (200000)
94(20.48)
84.2(20)32)
87 (=0.s
0.89
0.89t20.00g
MMVAE+
0.64 (10.003)
0.61
133422)
97.3
153 4
(20.00))
(.89 (0.000))
Supervised
0.79(20.001)
(.94
Table 2: Unconditional performance on the CelebAMask-HO dataset
SBM-VAE SBM-RAE
MLD
MoPoE
MVTCAE
MMVAE+
FID
79.1(-0.07)
84.2(+0.26)
82.8c+o.0s)
164.8
162.2
103.7(+0.01)
Table 3: Performance on Adversarial Attacks by predicting the attributes given image from CelebAMask-HO
datasot
Model
F1 with Ady Inputs
Percentage drop
MoPoE
0.574
19.1
MVTCAF
0.595 002)
13.7
MMVAE+
0.576
5.6
SBM-RAE-T
0.490 001)
36.3
0.634 (+0.002)
3.9
because of its advantages and improve diverse applications like healtheare and other areas that rely on multiple modalities.
References
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett. (eds.), Advances in Neural Information Processing Systems 29: Annual Conference Neurol Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spoin, pp. 892-900, 2016. URL https
//prrocediing..eueee
Mustapha Bounoua, Giulio Franzese, and Pietro Michiardi. Multi-modal latent diffusion. Entropy, 26(4), 2024. ISSN 1099-4300. doi: 10.3390/
URL
Tal Daniel and Aviv Tamar. Soft-introvac: Analyzing and improving the introspective variational autocncoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognétion (CVPR), pp. 4391-4400, June 2021.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Imant Daunhawer, Thomas M. Sutter, Kieran Chin-Cheong, Emanuele Palumbo, and Julia E Vogt. On the limitations of multimodal VAFs. In International Conference on Learning Representations, 2022. URL
https://openrevierr wwt.:e/wweeeoooomonnoooomoa iid=:miidiiiiiie CPUXX-A]
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Mare' Aurelio Ranzato, Alina Beygekimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 8780 8794, 2021. URL https aetaaa-bsttttt..te html
\/proceedings betpeediingn.nuurrie neurips..... http:////aaaee./////////////a///
Partha Ghosh, Mchdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From variational to deterministic auloencoders. In International Conference on Learning Represenlations, 2020.
URL
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, Son Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: 7/arxiv. //arxiv.org/abs... 2ttg/abs/1412..577
Michael Gutmann and Aapo Hyvärinen. Noise-contraatiiee estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statisties, volume 9 of Proceedings of Machine Learning Research, pp. 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL httts::/ mlr. ptwps/v9/ pptps///9/gutmaan11100 gut/anmmmmnnnnnn html
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a. local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: httttp:://proceeddngg wtw.:oceediinnnnn.nis..... www.ineurrccom cc/ paper_
Papere pdf
HyeongJoo Hwang, Geon-Hyeong Kim, Seunghoon Hong, and Kee-Eung Kim. Multi-view representation learning via total correlation objective. In M. Ranzato, A. Boygelzimer, Y. Dauphin, P.S. Liang, and
J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 12194 12207. Curran Associates, Inc., 2021. URL https///pprrooeeng.....rroooopop pcppppppppcrppeeeer.ie/////pppre 202.///i064a0000r1111feeee0000a0
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, Son Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL.
org/ absa /atss//41...990
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (ods.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http::/aaxiiv.rr org/ wwtpaab/../1111// 1312.6114
Cheng-Han Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Mihee Lee and Vladimir Pavlovic. Privale-shared disentangled multimodal vae for learning of latent representations. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1692-1700, 2021. doi: 111100//CVPRW3300222220000
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. Multimodal deep learning. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML/11, pp. 689-696, 696, Madison, WI, USA, 2011. Omnipress. ISBN 9781 450306193.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. orXiv preprint urXiv:2205.0744 07460, 2022.
Emanuele Palumbo, Imant Daunhawer, and Julia E Vogt. MMVAF+: Enhancing the generative quality
of multimodal VAEs without compromises. In The Eleventh International Conference on Learning Representations, 2023. URL
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents. CoRR, abs/ hbbs/2201.00300 2022. URL https
Nicolas Papernot, Patrick D. McDaniel, lan J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Ramesh Karri, Ozgur Sinanoghu, Ahmad- Reza Sadeghi, and Xun Yi (eds.), Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, AsiaCCS 2017, Abu Dhabi, United Arab Emirates, April 2-6, 2017, pp. 506-519. ACM, 2017. doi: 001145/305297330553000 URL https://doi..c. org/ 10. 1145/3052/3/11111 3053009
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Tlya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 2021. URL
attp://proceeddings.uu mlr. wwwppeeeeseee 2at39g/radfoddd22
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esscr, and Björn Ommer. High-resolution image synthesis with latent diffussio models, 2021.
Yuge Shi, Siddharth Narayanaswamy, Brooks Paige, and Philip H. S. Torr. Variational mixture-of-experts autoencoders for multi-modal deep generative models. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Newral Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouwer, BC, Canada, pp. 15692-1570s 2019. URL https: http://p....pppppp/.///aaaa/aaaa htt.aaaaa...........eee-eeoa.... Abstratt.tttttatthht
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Bue, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 11895-11907 2019. URL https:://proceedings. www.:eddiiinn..uuis..oo html
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp. 574-584. PMLR, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diedcrik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Basee-based generative modeling through stochastic differential equations. CoRR, abs/2011.133/ 2020b. 13456
URL http:://aaxxvv.org/aa/
Nitish Srivastava and Ruslan Salakhutdinov. Multimodal learning with deep boltzmann machines. Journal of Machine Learning Research, 15(84):2949 2980, 2014. URL http:://wwwprooorr/ppppere/t////
html
Thomas M. Sutter, Imant Daunhawer, and Julia E. Vogt. Multimodal generative learning utilizing jensen- Mhannon-divergeen In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (ods.), Advances Neural Information Processing Systems 39: Annual Conference on Neu- ral Information Processing Systems 2020, NewrIPS 2020, December 62, 2020, virtual, 2020. URL (ttpss/
14

---

Published in Transactions on Machine Learning Research (12/2024)
Thomas M. Sutter, Imant Daunhawer, and Julia. E. Vogt. Generalized multimodal ELBO. In 9th Inter- national Conference on Learning Representalions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
Masahiro Suzuki and Yutaka Matsuo. Mitigating the limitations of multimodal VAEs with coordination-baset approach, 2023. URL
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.nne,, 2017. URL htttps://openreviee.eee net/forum? 7d=BAL7
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. CoRR, abs/2305. //////230..1111 2023. doi: (0.4850/ARXXI..235...0 2306. URL https
48550/ arxiv 2305 11846
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis- Pouis-Philippe Morency, and Ruslan Salakhutdi- nov. Learning factorized multimodal representations. In 7th International Conference Learning Represenlations, TCLR 2019, New Orleans, LA, USA, May 6-9, 2019. Revieweil, 2019. URL
https://openrevier wttp:/oowwidddrryyoo.a...e
Miguel Vasco, Hang Yin, Francisco S Melo, and Ana Paiva. Leveraging hierarchy in multimodal generative models for effective cross-modality inference. Newral Networks, 146:238 255, 2022.
Pascal Vincent. A connection between score matching and denoising autoenccoesss..... Neural Comput., 23(7): 1661-1674, 2011. doi: 10.116//NECO 00142. URL https://dod..org/100 11w.:/EC12e/1.ee 1a0DE000__0011__3
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICMI-11), pp. 681-688. Citescer, 2011.
Jannik Wolff, Rahul G Krishnan, Lukas Ruff, Jan Nikolas Morshuis, Tassilo Klein, Shinichi Nakajima, and Moin Nabi. Hierarchical multimodal variational autoencoders, 2022. URL htttps://openreieev.neee
formm?1d=4V4720777
Mike Wu and Noah D. Goodman. Multimodal generative models for scalable weakly-supervised learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Nicolò Cesa-Biannn Bianchi, and Roman Garnett (eds.), Ad- vonces Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Camada, pp. 5580 5590, 2018. URL httpps
Mike Wu and Noah D. Goodman. Multimodal generative models for compositional representation learning. CoRR, abs/19122 abs/1912.05075, 2019. URL http:///axxiv.rg/abbb/////
Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model, 2023.
A Appendix
A.1 SBM-RAE
Regularized autoencoders (RAEs) (Ghss et al. 2020 be used instead of VAEs in our setting, RAEs assume a deterministic encoder and regularize the latent space directly by penalizing L2 norm of the latent
variables:
CRAE = CREC + B||2|| + XLREG:
(10)
where LREC is the reconstruction loss for deterministic autoencoder and LEEG the decoder regularizer. our setup, we don't. any decoder regularizer.
15

---

Published in Transactions on Machine Learning Research (12/2024)
In order to generate a sample from the latent space, RAEs require to fit separate density estimators the latent variables. In our case, the score models are responsible for generating samples from the latent space, which makes RAE a compelling choice for our setup. RAEs are capable of learning more complex latent structures, and expressive generative models such as score models can effectively learn that structure to generale high-quality samples.
A.2 Model Architectures and Experimental setups for Extended PolyMNIST
The extended PolyMnist dataset was updated from the original PolyMnist dataset by /Stttrr al. (2020) with different background images and ten modalities. It has 50,000 training set, 10,000 validation set, and 10,000 test set. The VAEs for each modality are trained with initial learning rate of 0.001 using B value of 0.1 where all the prior, posterior, and likelihood are Gaussians. This also applies to all multimodal VAEs except MMVAE+ which uses Laplace distribution instead of Gaussian. The RAE for each modality was trained using the mcan squarod error loss with the norm of | regularized by a factor of 10-6 and a Gaussian noise added to z before feeding to the decoder with mean 0 and variance of 0.01 where the hyperparameler values were tuned using the validation set. The encoders and decoders for all models use residual connections improve performance and are similar in structure to the architecture used in Daniel & Tamar (2021) except for MMVAE+ were used the original model because the model's performance doesn't ' generalize to different neural net architecture. For MMVAF+, modality-speciffc and shared latent sizes are each 32 and the model was trained similarly to the code provided by the paper F with the IWAE estimator with K-1. The detailed architecture can be found by referring to the code that is attached. We used latent size of 64 for our models and MMVAE+ and we increased the latent dimension of the other baselines to 64 * n where n is the number of modalitics because they don't 't have any modality specific representation. We chose the best / value for each model using the validation set. For MoPoE and MMVAE, of 2.5 was chosen, for MMVAE+ 5, for MVAE 1 and for MVTCAE 0.1.
The neural net of SBM-VAE and SBM-RAE is UNET network where we resize the latent size to 8x8. For SBM-VAE, samples are laken from the posterior at training time and the mean of posterior is taken at inference time. For SBM-RAE, the zs are taken directly. We use a learning rate of 0.0002 with the Adam optimizer (Kingma & Ba 2015). The detailed hyperparameters are shown in table We use the VPSDE with Bu 0.1 and 31=5, with N- 100 and the PC sampling technique with Buler-Maruyane sampling and langevin dynamics. For modalities less than 10, we use Bo of 1, the others hyperparameters remain the same. The energy-based model is a simple MLP with the softplus activation. We follow equation to train the models.
The classifier used for evaluation is composed of three convolutional layers followed by ReLU activations and two fully connected layers where the last one outputs 10 logits for classification. It's trained using the cross-entropy loss using the training data composed of all modalities and achieves an average accuracy 98.7 on the test set of all modalities.
Table 4: Score Hyperparameters Mnist
Model
Bmin
Smax
N
LD per step
EBM-scale
batch-size
SBM 10mod
0.1
5
100
1
1000
256
SBM below 10mod
1
5
100
1
1000
256
A.3 Training and Inference Algorithm
We follow the Song et al. (2020b) to train the score models using the latent representation from the encoders. The following algorithms show the training and inference algorithm we use.
MMMAAF+ code taken and updated from the official repo provided https://gtthaa.ccom/ppaa//l
16

---

Published in Transactions on Machine Learning Research (12/2024)
Algorithm Training
Require: M, N,
M modality, N -epochs
2
for = to M do
Get
Get the input x from modality Sample from the encoder
Sample from (())x,)
Append z, to end for for epoch = to N do
Diffuse
Compute Sg(2, 1)
Calculate loss using equation
Backpropagate and update the weights of the SBM
end for
Algorithm Inference
if Unconditional then
Sample z from (O, I) and stack to get Z
else if Conditional then
Sample z from (O I) if missing
Sample z from encoder g(z)x;) if present
Stack all z end if
for i= to do
is number of sampling iterations Update missing zs by the following equation using the PC sampling
if use guidance then
Add gradient from EBM to the score end if end for
Feed the z the respective decoder get output
23WAIE 53P
1.-
0.8
Accuracy
0
100
SENWAET
0.4
75
50
MATICA
0.2
2
Number of Observed Modalities
Number Observed Modalities
Figure 13: The conditional accuracy of the first modality generated by incrementing the given modality at time. The x-axis shows how many modalities are given to generate the modality and the y-axis shows the accuracy of the generated modality.
Figure 12: The conditional FID of the first modality generated by incrementing the given modality at a time. The x-axis shows how many modalities are given generate the modality and the y-axis shows the FID score of the generated modality.
17

---

Published in Transactions on Machine Learning Research (12/2024)
A.4 Ablation study
A.4.1 Extended PolyMnist results
Here we show additional graphs for different modalities in addition to the ones shown in the main paper. Figure 12 shows the same setup as figure in the main paper but for another modality.
We also show the average conditional coherence and average conditional FID for cach model averaging over the modalities generated given the rest. Table 5 shows the result of that. Figure 14 and per modality.
15 show the results
A.4.2 3 Ablation
In Table G we discuss how of the unimodal VAE aflects the conditional FID and the conditional coherence of SBM-VAF, As the score model is trained the samples from the posterior, samples that have good reconstruction are important for the score model to learn well. This is also evident in the result as lower B values have better result than higher 3 values as VAEs with lower & values have better reconstruction. The score model also learns the gap created in unconditional sampling due to this trade-off as the score model transforms normal Gaussian noise to posterior samples during inference. The table shows a score model on the first two modalities of Extended PolyMnist trained using different 3 values of 0.1,0.5, and 1 and their average conditional result.
In figure 16
also show results of different values for MMVAE+ and MVTCAЕ predicting the last modality the number of observed modalities is increased from to 9.
17
19 and
16
A.5 Mode coverage
In this section, in addition to FID, discuss an evaluation technique for unconditional generation. We generate 10,000 unconditionally generated images from the multimodal Extended PolyMnist dataset and count how many digits are generated from the 10,000 images for each digit. A good model should generate all digits uniformly covering all the distribution of the dataset. Figure 20 shows the counts of each modality
Model
Avg Coherence
Avg FID
MVAE
0.751 (00.00)
239.47(40.0cm)
MMVAE
0.969
17 76.49((+.188)
MoPoE
0.982
163.3L (+0..000
МVTСАЕ
0.983
77.69 (7.69ctt.200)
MMVAE+
0.937
110.65 0005)
SBM-VAE
0.967 (+0.001)
(3.29 (0.000)
SBM-VAE-C
0.989
(20.6)))
SBM-VAE-T
0.894(=0.001)
77. (+0.05) (0.00))
SBM-RAE
0.968(22.001)
81. $1.70 100.1880)
SBM-RAE-C
0.988
83.52
SBM-RAE-T
0.940(z0.0o1)
$6.48
Table Conditional Performance
Avg Coherence
Avg FID
0.1
0.905.+a.002
52.1 02.12(+0.045)
0.5
(800)..000)
102.85 00.00))
0.71 (3.114:4..o3)
108.35
Table 6: Efffec of 3 on SBM-
18

---

Published in Transactions on Machine Learning Research (12/2024)
200
HVTCAE
160
SBM VAE
0.9
SBM-VAE-T
SOM-RAE
0.8
SBM RAE
120
0.7
MMVAE
SBM-VAE
SEM
80
SBM-VAETT
0.6
60
SBM RAE-C
RAE-T
40
0.5
10
Figure 14: Conditional Accuracy (Coherence) of each modality given the rest..
Figure 15: Conditional FID of each modality given the rest.
fea-Ci
200
175
0.8
Accuracy
0.4
Bea-05
5
Number of Observed Modalities
Number of Observed Modalities
Figure 16: The conditional accuracy of the last modality generated by incrementing the given modality at time for different As of MMVAE+
Figure 17: The conditional FID of the last modality generated by incrementing the given modality at a time for different As of MMVAE+.
1.0
150
Accuracy
0.6
125
100
0.4
75
0,2
teae1
50
6
Number Observed Modalities
Number Observed Modalities
Figure 18: The conditional accuracy of the last modality generated by incrementing the given modality at a time for different As of MVTCAE
Figure 19: The conditional FID of the last modality generated by incrementing the given modality at a time for different 3s of MVTCAE.
in bar graph for SBMVAE, SBMRAE, MMVAE+, and MVTСAE, Our models and MMVAE+ cover most modes almost uniformly where MVTСAA has some non-uniformity. Our models also have the highest.
19

---

Published in Transactions on Machine Learning Research (12/2024)
unconditional coherence which means they also generate coherent outputs at the same time which is an ideal property.
A.5.1 Fine-tunning the generative models using missing modalites
We can further finetune the generative model (decoder) to increase the overall coherence. During training, we assume all the modalities are present. This condition is necessary for training Pe in the described setup. However, we are interested conditional queries p(. xo), we can achieve tighter lower bound by further optimizing the (x, z2)) for sample from (o: Xo). We update the parameters of decoders to maximize the conditional log- likeliiiohoo::
max Eg(zulze xx))) log Peo (Xu (x))zu) (u))
K
max
Σ
Pog Pxulez (x =u ~ Xo)
(11)
K
1=1
For each training example in the batch, randomly drop each modality with probability p. Eq. 11 will increase the likelihood of the true assignment of the dropped modalities given the observed modalities. We
evaluate this experiments on a different score model trained using NCSN Song & Ermon (2019) and with unimodal VAEs with f=0.5.
Our experiments on fine-tuning show that the model's conditional performance increases very slightly, but the quality, measured by FID, drops significantly.
A.5.2 Additional evaluation metrics
In this section, want to include additional metric for the extended PolyMnist. In addition to using classifier and comparing accuracy predicted using the classifier, here will use cosine similarity and measure how latent variable z of each modality is recovered. We will the cosine similarity of each recovered with true encoded z for SBM-VAE-C. For comparison, we also include MVTCAE, which uses the product of experts. Table 7 shows the result of this section. The SBM model recovers the latent z that is conditionally generated and it has a higher cosine similarity with the ground truth encoded z.
A.5.3 Qualitative Results for Extended PolyMNIST
Here we show some ionally generated samples and unconditional generation from each model. Condi- tional samples are shown in figures 24 and
25 and unconditional samples in figure 26
Modality
SBM-VAE cos sim
MVTСAE cos sim
0.157
0.184
0.164
0.161
0.165
0.138
0.180
0.063
0.187
0.125
0.229
0.131
0.078
0.158
7
0.107
0.158
8
0.183
0.040
0.192
0.175
Avg
0.164
0.118
Table 7: Cosine similarity between the encoded of the test data with the conditionally generated given the rest for each modality
20

---

Published in Transactions on Machine Learning Research (12/2024)
10000
8000
6000
4000
2000
1-5BMVAE
1-SBMRAE
1-MMVAE+
1-MVTCAE
2-SBMVAE
2-SBMRAE
2-MMVAE+
2-MVTCAE
3BMBAA
3-SBMRAE
--MMAE++
3-MVTCAE
4-SBVAA
SBMMAA
+MMVAE+
MMVTCA
5-SBMVAE
5-SBMRAE
5-MMVAE+
5-MVTCAE
6-SBMVAE
6-SBMRAE
6-MMVAE+
6-MVTCAE
7-SBMVAE
7-BBRRAE
7-MMVAE+
7-MVTCAE
8-5BMVAE
8-SBMRAE
+-MMAAE+
8-MYTCAE
9-SBMVAE
9-5BMRAE
9-MMAAE+
9-MVTCAE
10-SBMVAE
(0-SBMRAE
1
10-MVTCAE
Figure 20: Mode Coverage
21

---

Published in Transactions on Machine Learning Research (12/2024)
1.0
250
10: SA ANC-0CR
0.9
225
200
Accuracy
0.7
175
0.6
150
0.5
125
0.4
100
0.2
75
8
1
2
3
5
Number of Observed Modelities
Observed
Plot of finetuning experiment using different values on how it affects conditional co- herence (different score model). The conditional accuracy of the last modality generated by incre- menting the given modality a time. The x-axis shows how many modalities are given to generale the modality and the y-axis shows the accuracy of the generated modality.
Figure 22:
Figure 21: Plot of finetuning experiment using different p values on how affects conditional FID (different score model). The conditional FTD of the last modality generated by incrementing the given modality at time. The x-axis shows how many ies are given to generate the modality and the y-axis shows the FID of the generated modality,
O
O
O
C 1
2 2 3 3
4 4
4
6
6
6
7
7
7
7
7
2
?
(a) SBM-VAE
(b) SBM-RAE
(c) MMVAE+
(d) MVTCAE
(e) MoPoE
(I) MMVAE
(g) MVAE
Figure 23:
Conditional Samples from the third modality given the rest
0
O
0 0 $
0
0
0
0 0 0
0
7
/
1
/
(
9
5
2
A
2
3
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
4/
4
4
5
5
5
5
5
5
G
6
6
6
6
6
6
6
6
6
6
4
6
5
6
6
6
6
6
7
7
7
7
7
7 8
7
7
7
7
7
7
7
0
8
3
0
8
8
8
8
G
7
9
SBM-VAF
(b) SRM-RAF
(c) MMVAF+
(d) MVTCAE
(e) MoPoF
(f) MMVAF
(g) MVAE
Figure 24: Conditional Samples from the first modality given the rest
A.6 MHD Dataset
In this section, we discuss an additional experiment consisting of an audio modality. We use the audio and image modalities from the MHD dataset by Vasco al. (2022). The image modality is from the MNIST
22

---

Published in Transactions on Machine Learning Research (12/2024)
O
O
O
0
0
0
O
0
0
O
0
0
/
1
1
7
9
2 3
2
2 8
2
3
2
2
2
2 3
2
2
2
2
2
2 3 4 5 6 7 8
7
2
2
3
3
3
3
5
3
3
5
3
3
3 4
3
3
3
3
3
4
Y
4
4
4
4
4
4
4
4
4
5
5
6
5
5
5
5
6
6
6
6
6 6 7 2
6 7
6
6
6
6
6
6
6
6
6 7
6
6
6 7
6
O
6
9
6
0
G
7
7
7 8 4
7
7
7 8
7
7 7
7
7
7
7
7 2
7
7 7
9
9
9
7
8
8
8
0
9
7
9
9
9
9
9
9
9
0
9
9
(a) SBM-VAE
(b) SBM-RAE
(c) MMVAE+
(d) MVTСAE
(e) MoPoE
(f) MMVAE
(g)
MVAE
Figure 25: Conditional Samples from the sixth modality given the rest
9
8
0
9
9
4
6
3
9
8
2
/ 3
4
9
7
3
3
0
2
8
/
O
6
4
6
O
3
2
3
0
8
9
3
8
2
O
/
2
6
4
9
4
7
6
O
3
3
9
9
0
7
8
9
8
8
1
2
0
4
9
7
4
4
7
6
3
3
9
7
O
7
8
9
8 6
8 3
O
2
0
4
7
/
7
6
6
3
7
0
0
7
9
4
6
3
3
9
4
C
2 8
X
3
6
9
5
T
6
0
3
3
9
7
8
-
/
6
3
3
7
7
8
0
9
9
4
3
SBM- VA
(b) SBM-RAL
(c) MMVAF+
8
3
6
7
8
5
2
6
3
O
Ø
9
7
7
S
7
8
7
6
7
2
8
2
8
7
0
6
2
6
6
2
8
6
8
8
2
O
7
O
8
8
6
2
6
7
6
85
8
6
7
(d) MVTСАЕ
(e)
MoPoE
(f)
MMVAE
H
8
6
9
2
/
6
5
6
8
0
4
2
6
6
6
4
7
4
8
O
6
7
9
9
3
0
7
2
7
9
9
7
<
4
3
0
5
4
5
5
6
Q
7
4
/
2
8
(g)
MVAE
Figure 26: Unconditional Samples where each of the columns are unconditional samples from each modality
dataset and the audio is recording of each digit. We use the same encoder-decoder architecture from Vasco et al. (0222 with latent size 64. We conduct experiments three models: SBMVAE, MVTCAF, and
23

---

Published in Transactions on Machine Learning Research (12/2024)
MMVAE+ and we report the result in Table The result shows our approach works in the presence of an audio modality.
Table 8: Generation Coherence on MHD Imago-Audio
Model
AIdae-+Image
Image- AAuioo
Unc
SBMVAE
0.853 (+..00)
0.861 (+0.04)
0.839(=0.00
МVTСАЕ (.6.4(20.004)
0.817 (+t.002)
(2065
MMVAE+ (..00)+
0.442 ++0.002)
0.248 (+2.000
A.7 Computational Efficiency
As discussed in the conclusion section as one limitation, SBM variant models lake longer time during inference compared to the other baselines. To show how the models are different during inference in terms of computational efficiency, exlended Mnist data. We first start with a model trained on 2 modalities and increase the modalities the model is trained on to ten modalities. We compare MoPoE which uses mixture of product of experts for inference and SBMVAE. We use A100 GPU for computing the time the models take. Figure [7 shows the time the models take in seconds. SBM-VAE takes almost a constant time for all modalities, while MoPoE starts increasing exponentially as the number of modalities increases. For the CelebMaskHQ dataset, a similar batch of 256 takes approximately 95 seconds for inference while MoPoE takes approximately 0.05 seconds. Here MoPoE has clear edge as we use 1000 timesteps for this dataset the score-based model. But one advantage of the score-based model is that the inference timesteps can be significantly decreased without compromising that much quality. If we use 100 timesteps, the time required will decrease 10 folds to around 9.5 seconds, but the performance drops very slightly. For 100 timesteps, the model retains the same F1 performance for the attribute and mask, and only the FID of the image modality drops by a few points.
evaluate the time it takes to generate conditional samples from batch of 256
10
MoPoE
SBM-AE
Seconds
2-
X
X
X
X
10
Number of Modalities
Figure 27: Time takes for batch 256 on models trained from modality to 10 modality
A.8
Dataset
In this section, we want to show that our model can be scaled to higher dimensional datasots and modalities. We use partial samples, approximately 100K, from the soundnet dataset (Aylar el al. 2016) and train it on two modalities by separating the video and the audio as two modalities from each video. We follow the setup of CoDi (Tang et al., 2023) and use pre-trained image and audio auto-cncoders. The audio is changed melspectogram and encoded using an encoder to get 2aud- We also use an autoencoder for the image and encode the image to Zing from size of 256x256. To reconstruct the audio from the latent. representation,
24

---

Published in Transactions on Machine Learning Research (12/2024)
we first decode it and pass the audio melspectrogram to Hifi-GAN model to generate audio samples. We train the joint SBM-VAE model using a UNET architecture from scratch on the combination of the two latent modalities, Zand and 2mmg- It's important to note that due to computation limitations, we only trained our model on the partial dataset, and generating good-quality video samples requires hundreds of millions of datasets. For reference, CoDi used approximately 112 Million dataset samples for video generation. In contrast, we are only using 100K samples of videos from soundnet and we split cach one into a 2-socond video with 8 frames per second and feed the corresponding audio as another modality. Figure 28 29 30
and 31 show the frames of the 2-second video generated and the audio can't be displayed here. Please
look at the attached assets folder for more samples. Even though our model is not on par with current. state-of-thee-rt video-audio generative models, partially because it's trained on a few datasets, it can be seen that it has the potential to be scaled to high-dimensional modalities. We have also added conditional and unconditional video and audio generation quantitative results in Table g We use FID to measure the quality of the generated image frames and the FAD metric using VGG features to evaluate the quality of the audio on 2000 held-out samples. CoDi, shows the result of the CoDi model on the same dataset. CoDi generates images and audios that are more general to this dataset, hence the lower result, but in general, the CoDi model performs much better, as shown in CoDig, which is evaluated on CoCo text-to-image FID and AudioCaps FAD where the results are taken from the paper Tang et al. (2023). In addition, CoDi can't perform unconditional generation, which is a disadvantage of the model. The unconditional results for this model are left empty because of that.
Table 9: Video generation Performance using SBM-VAE and CoDi
SBMVAE
CoDi,
CoDig FID
FID
FAD
FID
FAD
FAD
Unconditional
99.02
3.31
Conditional
101.19
8.23
100.4
14.9
11.26
1.80
A.9 CelebMaskHQ Experimental Setup
The CelebMaskHQ dataset is taken from Lee et al. (0020) where the images, masks, and attributes are the three modalities. All face part masks were combined into single alack-aad-white image except the skin mask. Out of the 40 attributes, 18 were taken from it similar to the setup of Wu & Goodman (2018). The encoder and decoder architectures are similar to Daniel & Tamar (022)) except MMVAE+ for the same reason as in PolyMNIST. A latent size of 256 was used for SBM models and MMVAE+, and a latent size of 1024 was used for MVTCAE and MoPoE. For MMVAE+, modality-specifie and shared latent sizes are each 128 trained with IWAE estimator with K=1. For SBM-VAE, the image VAE was trained using Gaussian likelihood, posterior, and prior with 8 = 0.1. The same applies for the mask VAE but with B = 1. The attribute VAE uses Gaussian prior and posterior with bernoulli likelihood. This applies to all other baselines with the exception of MMVAF+ which uses laplace likelihood and prior. We select the best / for the baselines from (0.1,0.5,1,2.555,) MoPoE use 3 of 0.1, MVTCAE 0.5 and MMVAE+ 5. For SBM-RAE, 8 values of 10-4, 10-5, 10 4 and Gaussian noises of mean O and variance of 0.001, 0.001, 0.1 were added to a before feeding to the decoder for the image, mask and attribute modality respectively and the best performing one was selected.
The score-based models use a UNET architecture with the latent size reshaped into a size of 16x16. We take the mean of the posterior during training and inference time for SBM-VAE where as the z were taken during both times for SBM-RAE. Table shows the detailed hyperparameters used for the score models. DiffuseVAE hyperparametors and models are the same ones used in Pandey et al. (202)) with formulation for the 128x128 CelebHQ dataset. The energy-based model uses an MLP worw We train 3 pairs of models for each combination of modality. The supervised classifier for the attribute modality put for reference has a similar architecture to the encoder, and the features are projected for classification and trained with cross-cntropy loss. The mask classifier uses UNET architecture with an input dimension of 3 for the image and an output dimension of 1 to predict the masks.
25

---

Published in Transactions on Machine Learning Research (12/2024)
Figure 28: 16 video frames (ordered sequentially from left to right and top to bottom) generated unconditionally of a view of trees taken by somcone moving along with audio modality (can't be shown here) of the background noise a road. Please see the attachment for video samples.
Figure 29: 16 video frames (ordered sequentially from left to right and top to bottom) generated unconditionally of water wave with audio modal- ity (can't be shown here) of a background noise of water wave. Please see the attachment for video samples.
Figure 30: 16 vidco frames (ordered sequentially from left. to right and lop bottom) generated unconditionally of a view of road (audio can't be shown here). Please see the attachment for vidco samples.
Figure 31: 16 vidco frames (ordered soquentially from left to right and top to bottom) generated unconditionally of a snowy area with trees (audio can't be shown here). Please see the attachment for video samples.
For the contrastive guidance case of SBM-VAE-T or SBM-RAE-T, we trained the auxiliary conditioning encoders using contrastive learning. Their architecture similar the encoder of the VAEs plus a linear project head that. projects the modalities to latent size of 512. We align the representations of each
26

---

Published in Transactions on Machine Learning Research (12/2024)
auxiliary encoder using contrastive learning objective so that the similarity between the coming from the same pair of multimodal data are maximized and the similarity of the pairs of zs coming from unrelated
pairs are minimized, MLD by [Bounoua et al. (2024) is trained using the same experimental setup used by the SBM models, the same train/val/tost split, and the same architecture using the code provided herd
Model
8min
8max
N
LD per step EBM-scale
batch-size
SBM
0.1
20
1000
2000
256
Table 10: Score Hyperparameters CelebMaskHQ
A.10 CelebA Extended Experiments
A.10.1
Training with missing modaliites
Multimodal models trained paired data coming from all modalities. Sometimes, some modality may be missing from data sample. In those cases, it's important to make multimodal data trainable in these circumstances. We add this capability to the score-based model by using masking. Specifically, we mask out parts of the output score produced from the missing data. The missing data will be initially filled with noise. To simulate the performance the model on missing training data, we take the MelebMaskHH datasot and removing some portion of the data from the datasct. For example, we remove 20% of the data from each modality randomly and mix them. Note that multimodal data points that are not full are much larger than 20% we first remove 20% from each modality randomly and then mix them. We evaluate the performance of the model as shown in Table I Overall, MVTСAE has good performance when training data is incomplete. SBMVAE has comparative outputs with the best Image FID.
Model
Mask F1
Attr F1
Img FID
SBMVAE 0.837 (20.02)
(19.0)
(0.96
MoPoE to.00)
(...0)+a.02)
116.5.+0.0)
MVTСAE
0.895 (2: 04)
0.695(2..00)
99.49(a..4)
Table 11: Missing training data performance evaluated given the other modalitics on the whole test set
3Code for MT.D http://///aeeteeeeee//e//eseeeer
27

---

Published in Transactions on Machine Learning Research (12/2024)
A.10.2 High Quality Image generation for CelebAHQ
our setup, we use a normal variational autoencoder which makes suitable for baseline comparison and training compute requirements. But since variational autoencoders suffer from low-quality and blurry images, compared other SOTA generative models such as diffusion models (Dhariwal & Nichol 2021p, ; in order to get higher-quality,
can do two things. The first is to independent two phase training stages. The train the score model on it which is possible because we have second is to further increase the quality of the output of the decoder using DiffuseVAE model (Pandey et al., 2022). We illustrate the latter case here where the generated samples from SBM-VAE are fed into DiffuseVAE as shown in Figure 32 The DiffuseVAE helps in generating high-quality images from the low-quality ones without changing the image characteristics. As the figure shows, the quality of the images much better while preserving the attributes and the masks given generate them.
high-quality pro-trained auto-encoder and
Male Mouth Slightyy Open Straight
Male
Gray Haiz
Straight
(a) SBM-VAE outpul
(b) DiffuseVAE output
Figure 32: Higher quality image generation using DiffuseVAE given mask and attribute shown in the first two columns
A.10.3 CelebMaskHQ Qualitative Result
In this section, we show samples from the CelebHQQAASK dataset where the generated images are conditioned on different modalities. Figure 33 shows unconditionally generated outputs from each modality. Figure 34 shows different samples where only the image is given, Figure 35 shows different, samples where the mask is given, Figure 37 shows different samples where the attribute is given.
28

---

Published in Transactions on Machine Learning Research (12/2024)
(a) SBM-RAE image
(b) SBM-RAE mask
(c) SBM-RAE attribute
DEIP PORE MELLE
4
Flan PAR
(d) SBM- VAE image
(e) SBM-VAL mask
(f) SBM-VAE attribute
4
Al
4
(g)
MoPoE image
(h) MoPoE mask
ii) MoPoE attribute
Figure 33: Unconditional generation
29

---

Published in Transactions on Machine Learning Research (12/2024)
GRAM
BLOK FAST
GARE AET
00002 2017
1480
1488000 FaRleo
1887 C8R03
(b) SBM-RAE-C attribute
(a) SBM-RAE-C mask
1487 2WRNE
Bland
100 2017
6007
(d) SBM-RAE-T attribute
(c) SBM-RAE-T mask
Hourh SESPRTA
Paash Sioperyyyoooo
Fland Pale
Oreun 1027
ELAME 1857
Clend LET
100 2017
(I) SBM-VAE-C attribute
(e) SAE-V mask
Figure 34: Mask and Attribute generation given Image
30

---

Published in Transactions on Machine Learning Research (12/2024)
Man PALE
Mav 2011
Paary
Hay 22T.1/...p
ELAME 1AST
ETond 190
Stand SET
1007 1817
1907
6WV.
(h) SBM-VAE-T attribute
(g) SBM-VAE-T mask
(i) MoPoE attribute
(i) MoPoE mask
Figure 34: Mask and Attribute generation given Image
31

---

Published in Transactions on Machine Learning Research (12/2024)
sangt
Pale
Puto
Sangt
(b) SBM-RAE-C attribute
(a) SBM-RAE-C image
Hala
08'ge 2017
Stralge
Straggh
(d) SBM-RAE-T attribute
(c) SBM-RAE-T image
Dangs
Sangs
7514 22cc
(n) SBM-VAE-C attribute
(e) SBM-VAE-C image
Figure 35: Image and Attribute generation given Mask
32

---

Published in Transactions on Machine Learning Research (12/2024)
COMTON
1811
(h) SBM-VAE-T attribute
(g) SBM-VAE-T image
(j) MoPoE attribute
(i) MoPoE image
Figure 35: Image and Attribute generation given Mask
33

---

Published in Transactions on Machine Learning Research (12/2024)
(a) SBM-RAE-C image
(b) SBM-RAE-C mask
Black_
Busky Byebrows
Male
Mustache
(c) SBM-RAE-T image
(d) SBM-RAE-T mask
(e) SBM-VAE-C image
(I) SBM-VAE-C mask
Figure 36: Image and Mask generation given Attribute
34

---

Published in Transactions on Machine Learning Research (12/2024)
Black_
Busky Byebrows
(a) SBM-VAE-T image
(b) SBM- VAE-T mask
Male
Mustache
(c) MoPoE image
(d) MoPoE mask
Figure 37: Inage and Mask generation given Autribute
35


Reviews:
[{"review_id": "R0CtYzBg4Z", "reviewer": "Reviewer_Rnrd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of this work is that the authors' algorithm is simple and scales well to a large number of modalities. We know how to train unimodal autoencoders well, and training the joint prior is a relatively simple generative modeling problem with low-dimensional (compared to the original inputs) observations. The coherence guidance is also a good idea, although it can likely be improved (see requested changes.)\n\nThe weakness of this work are the empirical results. The authors experiment with two toy MNIST-based problems (one in the appendix), and one realistic dataset. The paper would be stronger with an additional realistic dataset, as is fairly standard with recent empirically-oriented ML papers. Also, this is highly subjective, but the datasets that the authors work with are not canonical examples of multimodal learning. This is especially the case with PolyMNIST, where the \"modalities\" are what people typically call \"style.\" I argue this is also the case for the image and mask in CelebAMask-HQ. I believe the paper would be much stronger with an additional dataset such as VQAv2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a way to fuse together a set of unimodal autoencoders into a multimodal autoencoder, using score matching to train the joint prior over the latent variables for all modalities. They use contrastive learning to classify when a pair of latent variables from two modalities correspond to the same example, and use this to improve the coherence across modalities. Their algorithm is empirically validated using two toy datasets (MNIST-based), and one realistic dataset (CelebAMask-HQ), where they compare against various multimodal VAE baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "You might be able to improve the coherence guidance by using multiple positive pairs, as done in supervised contrastive learning (Khosla 2020). Given an anchor modality, the positive pairs are all other modalities from the same example, while the negative pairs are all other modalities from different examples.\n\nI would be more in favor of acceptance if this paper included an additional realistic dataset such as VQAv2.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WCT5G20bhw", "reviewer": "Reviewer_dCDR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThis paper tackles the challenge of multimodal training from a novel and interesting perspective. Instead of directly learning the joint posterior, which becomes easily intractable in high dimensions, this approach smartly divides the problem into two sub-problems and solves them one by one in two different stages. It also avoids modality sub-sampling, which fails to provide modality coherence, as shown by [1].\n\n**Weaknesses**\n\nWhile this paper provides a practical algorithm demonstrating promising performance on the considered datasets, some theoretical assumptions seem incoherent with their use cases. For example, at the beginning of section 2, it is assumed that the latent variables only consist of modality-specific latent variables in the sense that none of them are shared between different modalities. On the other hand, this paper has proposed using contrastive guidance for score matching by incorporating an aligned representation learned using a shared encoder (section 2.2.2).  If, as stated in Section 2, no latents are shared between modalities, which kind of representation do we expect from this secondary contrastive encoder? And why would it then help guide score matching?\n\n**Reference**\n\n[1] Imant Daunhawer, Thomas M. Sutter, Kieran Chin-Cheong, Emanuele Palumbo, and Julia E Vogt. On the limitations of multimodal VAEs. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=w-CPUXXrAj.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel approach to learning multimodal VAEs by first training individual unimodal VAEs for each modality and then modeling the intra-modality dependencies (i.e., the joint latent distribution) using score matching approaches. Their experiment results on extended PolyMNIST and CelebAMask-HQ have demonstrated a series of desired properties of a multi-VAE model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In the first row of page 4, is there a particular reason that the score is defined as an approximation of the log derivative of the distribution instead of just equality? One short sentence to clarify would be appreciated.\n\n2. The latent coherence is evaluated solely based on label prediction in a discrete space. It would be great to have an additional continuous measure for the latent coherence. For this, one could design a synthetic numerical experiment by sampling a set of correlated modality-specific latents and generating the corresponding observations that form different modalities.  After training, one can compare the correlation of the predicted latents and the ground truth ones to see how well the proposed model preserves the latent coherence.\n\n3. Prior works have studied provable latent variable identification under the multi-modality setting (for example, [1, 2]). Identified latents have shown a series of desired properties that align with the expected properties discussed in this work. Is there any identifiability guarantee in this proposed approach? In general, I think it would be beneficial to include a discussion on (1) the identifiability of the proposed approach and (2) a connection with other identifiable frameworks under a multimodal setting.\n\n\n**Reference**\n\n[1] Imant Daunhawer, Thomas M. Sutter, Kieran Chin-Cheong, Emanuele Palumbo, and Julia E Vogt. On the limitations of multimodal VAEs. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=w-CPUXXrAj.\n\n[2] Dingling Yao, Danru Xu, Sebastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=OGtnhKQJms.", "broader_impact_concerns": "There are no significant ethical concerns from my point of view", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WWDr0PMVyM", "reviewer": "Reviewer_aRPf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Overall, the methodology is well defined with the appropriate equations. Introducing the coherence guidance makes sense.\n- Related works provide a thorough and detailed summary of previous research.\n- The variants of proposed method achieve superior performance overall.\n\nWeaknesses\n- As the authors mentioned in Conclusion and Discussion, the proposed method is compute heavy. However, the authors did not compare computational efficiency between SBM variants and previous approaches, including inference time.\n- I acknowledge that the experiments on PolyMNIST and CelebAMask-HQ prove the effectiveness of the proposed method. However, it makes the proposed method more practical if the method achieves superior performance on more high-dimensional data. SBM models are trained on images with a resolution up to 128x128 while vision community usually uses high-resolution images nowadays. Also, the connection between the audio and image modalities of the MHD dataset is quite unnatural. I don't think there is high correlation between written digits and the voice that reads it aloud.\n\nMissing details\n- The authors does not provide the details of the classifier to be used for measuring accuracy, including model architecture, training data, training recipes, etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Previous multimodal variational autoencoders (VAEs) models the joint posterior over the shared latent space with multimodal ELBO. This might result in generative discrepancy among modalities and no gain in additional conditional modalities. Instead, this paper proposes learning the correlation of individual latent spaces of unimodal representations using a score-based model (SBM). Furthermore, to increase coherence between the predicted modalities and the observed modalities, it provides coherence guidance by either using an energy-based model or aligning modalities through contrastive learning objectives. The resulting model achieves competitive or better performance compared to existing approaches on the the extended PolyMNIST and CelebAMask-HQ, in terms of both prediction coherence and generative quality. It is also more robust to adversarial attacks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the above concerns. Especially, it would be great to provide the results of training the proposed method on video data, using visual frames and corresponding audio signals.", "broader_impact_concerns": "Please include the broader impact statement in the final draft.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommend acceptance in their final decision. Nevertheless, Reviewer aRPf requests the authors to include the results of baseline video generation models in the additional results of Appendix A.8. The AE acknowledges that such a comparison should be taken carefully, as the different models would be trained with different amounts of data, but this could be highlighted in the table and in the text.
Decision: Accept (poster)