Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders
Authors: Rogelio A. Mancisidor
Robert Jenssen
Shujian Yu
Michael Kampffmeyer
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Learning
Product of Experts
Generative Models
Variational Autoencoders
Primary area: probabilistic_methods->variational_inference
TL;DR: Multimodal learning with VAEs where the dependence between expert distributions is taken into account.

Abstract: Multimodal learning with variational autoencoders (VAEs) requires estimating joint distributions to evaluate the evidence lower bound (ELBO). Current methods, the product and mixture of experts, aggregate single-modality distributions assuming independence for simplicity, which is an overoptimistic assumption. This research introduces a novel methodology for aggregating single-modality distributions by exploiting the principle of *consensus of dependent experts* (CoDE), which circumvents the aforementioned assumption. Utilizing the CoDE method, we propose a novel ELBO that approximates the joint likelihood of the multimodal data by learning the contribution of each subset of modalities. The resulting CoDE-VAE model demonstrates better performance in terms of balancing the trade-off between generative coherence and generative quality, as well as generating more precise log-likelihood estimations. CoDE-VAE further minimizes the generative quality gap as the number of modalities increases. In certain cases, it reaches a generative quality similar to that of unimodal VAEs, which is a desirable property that is lacking in most current methods. Finally, the classification accuracy achieved by CoDE-VAE is comparable to that of state-of-the-art multimodal VAE models.

Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders
Rogelio A Mancisidor! Robert Jenssen 234 Shujian Yu 25 Michael Kampffmeyer
Abstract
1. Introduction
Motivated by the observation that learned representations from multimodal data tend to be more generalizable (Wu & Goodman, 2018; 2019), variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) have been used to learn representations from multiple data modali- ties as they are capable of simultaneously generating new observations and inferring joint representations. However, during model inference, is not guaranteed that we will have access to all modalities, multimodal data is expen- sive to obtain (Wu & Goodman, 2018; 2019; Sutter al., 2021). Therefore, multimodal VAEs should be able to gen- erate representations even when some modalities are miss- ing. This fact has led to a line of research on scalable mul- timodal generative models with VAEs (Wu Goodman, 2018; Shi et al., 2019; Sutter et al., 2021; Hwang et al., 2021; Palumbo al., 2023) in which the number of single- modality distributions, called experts, linear to the num- ber of modalities M.
Multimodal learning with variational autoen- coders (VAEs) requires estimating joint distri- butions to evaluate the evidence lower bound (ELBO). Current methods, the product and mix- ture of experts, aggregate single-modality dis- tributions assuming independence for simplicity, which is an overoptimistic assumption. This re- search introduces a novel methodology for ag- gregating single-modality distributions by ex- ploiting the principle of consensus of dependent experts (CoDE), which cireumvents the afore- mentioned assumption. Utilizing the CODE method, we propose a novel ELBO that approx- imates the joint likelihood of the multimodal data by learning the contribution of each sub- set of modalities. The resulting CoDE-VAE model demonstrates better performance in terms of balancing the trade-off between generative coherence and generative quality, as well as generating more precise log-likelihood estima- tions. CoDE-VAR VAE further minimizes the gen- crative quality gap as the number of modali- ties increases. certain cases, reaches a generative quality similar to that of unimodal VAEs, which is desirable property that is lacking in most current methods. Finally, the classification accuracy achieved by CoDE-VAE is comparable to that state-of-the-art multi- modal VAE models. CoDE- VAE is available at: http:///tttbb..oooonnm/oooaacccd
The product of experts (PoE) (Hinton, 2002) and mixture of experts (MoE) are currently the predominant methods to estimate joint variational posterior distributions in mul- timodal VAEs. However, both assume independence be- tween the experts for simplicity. This is overoptimistic assumption, the data modalities are simply different sources of information on the same object. Besides, both have flaws in the estimation of the joint posterior. PoE can produce posterior distributions with lower density if any of the experts also has low density, and if the precision of the experts is miscalibrated, the results can be biased (Shi et al., 2019). The weights in MoE, on the other hand, can be seen probabilities that the distribution is correct. However, the notion of a "correct model" is not reasonable in this context (Winkler, 1981), the objective is aggregate es- timates from expert distributions. Furthermore, MoE is in- efficient high-dimensional spaces, as the joint posterior cannot be sharper than any of the experts (Hinton, 2002). In addition to the aforementioned limitations in estimating the joint variational posterior distributions, the optimiza- tion of multimodal VAEs requires valid evidence lower bound (ELBO). In the past, this has been obtained by sum- ming the ELBOs corresponding to each of the k-th subsets
Department of Data Science, BI Norwegian Business School, Oslo, Norway Depaarment of Physics and Technology, UIT The Aretic University of Norway, Tromso, Norway 'Department of Computer Science, University of Copenhagen. Copenhagen, Den- mark 'Dept. BAMJO, Norwegian Computing Center, Oslo, Nor- way 'Department of Computer Science, Vrije Universiteit Ams- terdam, Amsterdam, Netherlands. Correspondence to: Rogelio A Mancisidor
bi.no>.
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Consensus of Dependent Experts Multimodal VAE
Xx PX)) (Sutter et al., 2021), where X is the multimodal data, P(X) is its powerset, and where each k-th ELBO con- tributes equally to optimization. We argue that not all sub- sets provide the same information and, therefore, should not contribute equally to optimization.
generative coherence and generative quality of generated modalities, as well likelihood approximations in mul- timodal VAEs, are higher when the dependence between experts is considered and the contribution of ELBO terms to optimization is learned.
Methods for aggregation of expert distributions, such as PoE or MoE, are called consensus of experts (Winkler, 1981). Although the information provided by expert distri- butions can exhibit dependence, limited attention has been paid to it. Therefore, this research advances the method in- troduced in (Winkler, 1981) by extending it to multivariate data and applying it to the approximation of joint varia- tional distributions in multimodal VAEs, which is a non- trivial task. Our proposed consensus of dependent experts (CoDE) method expresses the dependence between experts through the experts' error of estimation and estimates the joint variational distributions with a principled Bayesian method, which circumvents the notion of a correct model and the approximation of vaguer posterior distributions than any of the experts. Furthermore, when the expert dis- tributions are miscalibrated, or uncertain, the joint distri- bution estimated by CoDE leans towards expert distribu- tions that are more certain. Based on CoDE, we introduce novel multimodal VAE model BODDE-VABI that not only takes into account the dependence between expert distribu- tions when estimating joint posterior variational distribu- tions, but also learns the contribution of each k-th ELBO term in optimizing the overall ELBO, An important dis- tinction in estimating joint posterior distributions and op- timizing the CoDE-VAE model is that it does not rely sub-sampling' (Daunhawer et al., 2022), which has been shown to harm the approximation of the joint distribution of the data and yet is used by some multimodal VAEs.
2. Related Work
The initial research on multimodal learning with VAEs fo- cused on bimodal data, e.g., (Suzuki et al., 2017; Vedan- tam et al., 2018) where the consensus distribution is ap- proximated by concatenating the two modalities or (Wang et al., 2017) where the authors simply approximate the two expert distributions, assuming that one modality is avail- able during training and test time. Since then, there have been different lines of research to improve performance in multimodal VAEs. For example, the information the class labels has been used to improve the performance in discriminative tasks (Tsai et al., 2019; Mancisidor et al., 2024) and to disentangle sources of variation (Ilse et al., 2020); informatioonthhorett approaches have modeled the total correlation (Hwang et al., 2021), common informa- tion (Kleinman et al., 2023), and mutual information (Man- cisidor et al., 2024) in the objective function; latent distri- butions have been aligned by minimizing Wasserstein dis- tances (Theodoridis et al., 2020) and using adversarial net- works (Chen & Zhu, 2021); (Joy et al., 2022) use mutual supervision to combine information from modalities; hier- archical representation levels of latent representations are used to improve generative properties (Vasco et al., 2022); modeling modality-specific and shared latent distributions has been widely used (Hsu & Glass, 2018; Sutter et al., 2020; Lee & Pavlovic, 2020; Palumbo et al., 2023); diffu- sion decoders and clustering techniques have been coupled with multimodal VAEs (Palumbo et al., 2024).
The empirical analysis in this research shows that, when dependence between experts is considered, CoDE-VAR PADE-VABes ex- hibits better performance in terms of balancing the trade- off between generative coherence and generative quality, as well as generating more precise log- liglliiel estimations. Furthermore, CoDE-VAF minimizes the generative quality gap as the number of modalities increases, achieving un- conditional FID scores similar to unimodal VAEs, which a desirable property that is lacking in most current mod- els. Finally, CoDE-VAE achieves classification accuracy that is comparable that of current state-of-the-e multi- modal VAEs. The contributions of this research are: (1) the development of novel consensus of experts method that takes into account the stochastic dependence between ex- perts; (2) the use of this novel consensus of experts method to derive a new ELBO that learns the contribution of each subset of modalities to optimization; (3) to show that the
One of the fundamental problems in multimodal VAEs is how to approximate joint variational posterior distribu- tions. Joint representations must capture the abstract com- positional structure of the underlying object (Wu & Good- man, 2019), and the approximation method must be flex- ible to scale to a large number of modalities. The pre- dominant methods for this task are PoE (Wu & Goodman, 2018; Sutter et al., 2021; Hwang et al., 2021) and MoE (Shi et al., 2019; Sutter et al., 2020; Palumbo al., 2023), where the joint distributions are approximated q(=lX) IIm 9m (z|=z.) and (x2[X) = 1/M E.m 9m (z) (m.). re- spectively. However, none of these methods take into ac- count the dependence between expert distributions, and failure to do so may harm the performance of multimodal VAES. In addition, the MoE method introduces a sub- sampling of modalities that has negative impact on the likelihood estimation and the generative quality of multi- modal VAEs (Daunhawer et al., 2022). Instead, this re- search introduces efficient consensus of experts method
"Sub-sampling in this research refers only to the definition of consensus distributions as a mixture model and the training of some, but not all, Le terms (defined in Section 3).
2

---

Consensus of Dependent Experts Multimodal VAE
to model the dependence between experts via errors of es- timation.
VAEs. The CoDE method, therefore, represents the first principled Bayesian approach capable of approximating joint variational posterior distributions ((z(Xx) E P(X) in multimodal VAEs.
3. Methods
We observe multimodal data X composed of M modal-
All M distributions {< 1), where Xx denotes the cardinality of Xx: are considered expert dis- tributions, estimating the remaining unknown distributions g(= X 1), which are called consensus distributions.
Definition 1.
ities .., aM that are governed by a latent vari-
able through the conditional distribution p(X]z) =
(xz=)=))((a))).....pb))) p(x.pppa////)) (z). In this context, assum-
ing prior distribution p(z), the marginal p(X)
J p(x)=)p((zdzz and posterior p(=|X) distributions are in- tractable. Therefore, variational inference (VI) approxi- mates the true and unknown posterior distribution p(z[X) with the parametric variational distribution [(2[X). It
Definition introduces expert and consensus distributions, also referred to as single-modality and joint posterior distri- butions in previous research. Each expert distribution pro- vides information to estimate all consensus distributions in P(X). In the remainder of this section, we use standard Bayesian notation where 0 denotes unknown variables for which we compute posterior distributions.
seems natural to minimize the Kullback-Leibler (KL) di-
vergence [[]g==/X)][[[== X)]: however, contains the intractable marginal distribution [[]]
Given that the KI. divergence is strictly positive, it is straight- forward to derive a lower bound on log (Xx) as fol- lows (X.) 2 [x(=[x) - [L[g(=) { [=))) However, we are interested in approximating the true pos-
log (0)p(())BBB [xxx [lo +]]+ (
Let O2 (0), 02)) denote the la- tent variable z E I 2 for the k-th consensus distribution ((2[Xx). Each j-th expert distribution provides a point es- timate 2 on the d-th dimension 0g. and the uncertainty about the estimation is expressed in the parameter of of each expert distribution. Therefore, the error of estimation of the j-th expert in the d-th dimension is of = 14 0% and the error of estimation of all experts in the d'th dimen- sion is ed = (ef. eq. ... crrr. )", where M is the number of experts who evaluate the subset Xg. The overall error of estimation on 0₂ is e = (e' e?, eD)T.
Definition 2.
terior distribution p(z[X) even when only subset X E P(X) is available. In the case of missing modalities, as shown by (Sutter et al., 2021), valid lower bound for the available subset Xx E P(X) is
(1) To consider all scenarios of missing modalities, we opti- mize all lower bounds L& (X) governed by its variational distribution g=)))x). The next section introduces our pro- posed CoDE method for estimating joint variational pos- terior distributions taking into account the stochastic de- pendence between ainle-modality distributions. The main point to note is that the single-modality distributions have access to the same source of information. Therefore, as- suming independence between them, as PoE and MoE do, an overoptimistic assumption.
L (x) = ( log p( X [ K KT(a(=
According to Definition 2, all expert distributions provide estimates of the unknown variable 0, as well a mea- surement of the uncertainty on their estimation. Then, af- ter considering the estimates of all relevant expert distri- butions, we can calculate the unknown consensus distri- butions. Note that only the consensus distribution corre- sponding to the subset with all elements, i.e. q(2 = M). contains estimates of all M expert distributions. In general, consensus distributions are derived from different numbers of expert distributions, depending on the cardinality of the subset on which they are conditioned. Therefore, the size of the vector e₁ depends on the cardinality of XT.
3.1. Consensus of Experts with Stochastic Dependence
In multimodal learning we are interested in generating la- tent variables z even when multimodal data X have missing modalities. Under this premise, there are K 2M - dis- tributions g(z (xx) to be learned given M modalities, where k = 1, 2, ....., K denotes a subset of the powerset P(x)? Since the number of distributions is exponential in the num- ber of modalities, we need methods that scale linearly in This research advances
Lemma 1. Assume that the error of estimation es of the k-th consensus distribution a random variable with mul- livariate Gaussian distribution es ~ NNO, Ex), where is a [M' D x 1] vector, M' is the number of experts who are elements of the subset Xe, and D is the dimensionality of The covariance matrix is defined as
M, more details in Appendix the consensus of experts method in (Winkler, 1981) by ex- tending it to multivariate data, considering dependence be- tween expert distributions, and applying it to multimodal
Σ
0
0
0
In this research, we do not consider the empty of P(X) since there are no learnable parameters in g(2).
0
>>
3

---

Remark 1. Lemma subsumes PoE for Σn with diagonal matrices Ed in which case h(0x(pa) ~ See Appendix for Consensus of Dependent Experts Multimodal VAE
044
exenti exent Cade
p = 0.6, 01 v3, and 02 vi. The estimated consen- sus distribution by CoDE recovers the unknown parameter 0, by leaning even more towards expert 2. Note that MoE estimates a consensus distribution that is not sharper than any of the experts and does not recover, in expectation, the parameter. PoE underestimates the variance of the con- sensus distribution, as neglects the dependence between experts. Consequently, the consensus distribution for PoE is sharper than that of any of the experts", even when it does not recover 0. See Appendix B for additional details.
625
0.23
602
000
-30
-25
25
30
14
250
20.5
(a) Expert distributions
(b) Consensus distributions
Figure I. Two univariate expert distributions with estimates jn 4 and μl2 = 8 on the unknown variable 0 = 8. Expert 2 is more certain (o² = 1) than expert 1 (oî = 3).
Lemma 2. Assuming a flat prior distribution on the pa-
where
rameter O% of the k-th consensus distribution and known covariance matrix En. the posterior distribution is
0
1,2
M.
2
02
M..M'
h(0 ~ (AA Ax
(2)
Σd
where A u² "Z' u, BR u² > 146, Dx is the inverse matrix of En, and u and pa are defined in Lemma I.
OM
M'
a is a [M's x XM') matrix. Let pe = (ML. p2 p )T be a vector with estimates of all expert distributions Me the d-th dimension and fu = vector containing estimates all expert distributions about all dimensions of 2x Therefore, the distribution of all es- timates of the k-th consensus distributions is the multivari- ate Gaussian distribution pag ~ NNuOk, Ex), where u is a [M' - x D) design matrix with size M' vectors of Is along the diagonal and Os elsewhere
".f.....,PP) PT be the
details.
Lemma (proof in Appendix A.2) shows that we can es- timate the posterior distribution of the unknown param- eter Oş after observing its estimates μl from the expert distributions. Recall that Definition 2 refers to the latent variable with Og. having CoDE approximates consensus distributions in a principled Bayesian manner with the posterior distribution g(>| Xx) ~ (AA, Br, A, )4, and can be used in any multimodal VAE that uses PoE as an approximation method. It noteworthy that CoDE does not rely on sub-sampling tech- niques, which have been shown to harm the performance of multimodal VAEs (Daunhawer et al., 2022). Furthermore, note that choosing a diagonal matrix Σd, as PoE does (see Remark 1), assumes that expert estimates are not correlated and, therefore, the variance of the consensus distribution is underestimated.
There are a couple of interesting results according to Lemma I (see Appendix A. for the proof). First, the point estimates μe provided by the expert distributions are ran- dom Gaussian variables. If N N(uO,, Ex) is viewed as function of 02 instead of j4g, it can be interpreted as like- lihood function and we can calculate the posterior distribu- tion of 0s in principled Bayesian manner. Second, each covariance matrix Ed expresses both the uncertainty in the expert error of estimation on the d-th dimension of x (main diagonal) and the dependency between expert errors (off- diagonal). seems reasonable assume that expert distri- butions have overlapping information, positive correla- tion, since the data modalities are simply different sources of information about the same object.
g(2[Xx). Therefore,
Dependency between expert distributions: Lemma 2 shows that the covariance matrix of the consensus distribu- tions is a function of Eg, which is composed of nd on its main diagonal. The off-diagonal elements of Ed capture the dependency between the error of estimation of expert distributions. Therefore, in the forward pass the only un- known parameters in Ar and B, to calculate the consen- sus distribution are the off- off-diagonal elements djys in Σd,
Toy example: To gain insight into the aggregation of ex- pert distributions with CoDE, PoE, and MoE, Fig. shows two experts with dependence on their assessment of the unknown parameter f. Experts 1 and provide estimates I1 and #2 as well as a degree of uncertainty about their estimates of 3 and of 1. This informa- tion is summarized in expert distributions, assumed to be Gaussian. Taking estimates (#1, p2) as samples from the likelihood function (uu8 E), we can calculate the pos- terior (consensus) distribution of 6. The dependence be- tween experts is considered in the off-diagonal of Σ using
NNote that TPoE = Ti + ........ TTy' where T is the inverse of the variance, so TroB > T, or oo < a for 1,.... M'. 'Conditioning μt or X2 have the same meaning, as pt are estimates from all experts the subset X,.
4

---

Consensus of Dependent Experts Multimodal VAE
as (1i, of) outputs of the encoder networks for all di- mensions. We specify Jij as pajo; and the unknown cor- relation parameter p is found by cross-validation over the values [0, 0.2, 0.4, 0.6, 0.8). Note, CoDE does not impose any restriction on specifying E differently, e.g. different or negative p values, as long as it is an invertible matrix.
weights of the neural networks. The weight coefficients π are learnable parameters optimized by entropy maximiza- tion. See Algorithm I in Appendix D.I for details.
4. Experiments
Following the standard experimental setup in this domain
3.2. Evidence Lower Bound
(Sutter et al., 2021; Daunhawer et al., 2022; Palumbo et al.,
2023), performance is evaluated using the following multi- modal datasets: MNIST-SVHN-TER dataset (Sutter al., 2020) composed of matching MNIST and SVHN digits and a text describing the digit; PolyMNIST (Sutter et al., 2021) composed of MNIST images of the same digit, but dif- ferent background and handwriting style; and The Caltech Birds (CUB) dataset (Wah et al., 2011; Daunhawer et al., 2022), which is composed of images of birds paired with captions describing each bird. This is significantly more challenging version compared to the one used in (Shi et al., 2019), where they use ResNet embeddings rather than ac- tual images making the learning task significantly easier. Note that these datasets have different levels of complexity due to the shared and modality-speciff information across modalities. MNIST-SVHN- and CUB have a large amount of modality-specinf information given the hetero- geneity across modalities. On the other hand, PolyMNIST contains a moderate amount of modality tion, but is a suitable dataset for analyzing the quality gap as the number of input modalities for model training in- creases.
To leverage the proposed CoDE for multi-modal VAEs, we follow an approach similar to that of (Sutter et al., 2021), where the objective function is the sum of all ELBOs (x) in Eq. 1 that arises from all subsets Xx P(X). In (Sutter al., 2021) each k-th ELBO term contributes equally to optimization. In this work, however, we argue that if tr( (r( > tr(Ex), where tr(-) is the trace of a ma- trix and Σ is the covariance matrix of expert or consensus distributions, the distribution associated with the k-th sub- set is more confident in its estimate, indicating that it has relatively more reliable information than the j-th subset. Similarly, if x > X2, the j-th subset contains relatively more information and it should contribute more to the op- timization of the objective function. To avoid using equal weights in the weighted sum of all ELBOs, we introduce an indicator vector& & that is drawn from the categorical dis- tribution Cat( rr), where π (m Tx) is probability vector, and Ki is the number of subsets in P(X) (excluding the empty set). In our context, the indicator vector maps each k-th subset X E P(X) to either or O, and each event occurs with probability Ty- For example, the first subset in P(X) occurs with probability Pr(E[X1) = Mi, where { = [1,00.....00 0].
informa-
We compare the performance of the CoDE-VAE model with that of MVAE (Wu & Goodman, 2018), MMVAE (Shi et al., 2019), mmJSD (Sutter et al., 2020), MoPoE (Sutter et al., 2021), MVTCAE (Hwang et al., 2021). and MMVAE+ (Palumbo et al., 2023). Note that MMVAE+ is the only model in the comparison that utilizes modality- specific and shared latent variables the decoder to im- prove the quality of the generated modalities. We measure the performance of all models in terms of the generative herence and the generative quality of the generated modali- ties conditioned on both latent variables from the joint pos- terior and prior distributions. In addition, to assess the qual- ity of the approximation of joint posterior distributions, we measure the tightness of the lower bound by log- likellhoood estimations. Finally, for completeness, the discriminative power of the joint latent representations z using linear clas- sifiers is shown in the appendix. Limitations of our pro- posed model are provided in Appendix C, while details on datasets, evaluation criteria, network architectures, training procedure, and extra results are in Appendix D.
Assuming the generative model p(X, E) E) (()))))(=)p({), where p({) ~ Cat(1) is a prior categorical distribution with equal probabilities = (1/K,--. .... 1/K/), and the inference model q(2, ((x,) = (= (E( where g({(x)) ~ Cat(Tr) is the posterior distributions of e. the variational lower bound of the CADE-VAE model for single data point is given in Lemma 3.
Lemma 3. The concave foncaveffunction
((X) = E Th [ log (og (X|z)]
- -KL (=x [[b(3)) + H((o(() + C. (3)
where H denotes the entropy function and C constant term, is variational lower bound on log r(X) that opti- mizes all k-th ELBOs La with weight coefficients Tg.
All experiments report the average performance over three different runs, and the hyperparameters 3 (Higgins et al.,
The ELBO of the CoDE-VAE model in Eq. 3 (proof in Appendix A.3) parameterizes the conditional likelihoods ( ))) expert and consensus distributions (x({x) with neural networks, where 0 and & denote the learnable
5

---

Consensus of Dependent Experts Multimodal VAE
0.8
MVAE
MHVAE
Carditional Coltrence
Uncordmtonal Coharence
MMgDD
+
MVTCCE
0.5
0.3
MMVAE+
0.4
NIVAE
0.3
0.2
MoPNE-VAL VAE
0.2
MYTCAE
0.1
+
0.1
CoDEVAE
0.0
-2100
-2050 -2000 1955 -1900 -1850 -1800
-2100 -2050 -2000 -1950 -1900
-1850
Log-llkelhood
Log-llkelthood
Figure 2. Tradc-off between generative coherence (1) and joint log-likelihoods (f) on the MNIST-SVHN- test set.
Table 1. Generative quality measured by FID scores (L) and generative coherence (1) on the MNIST-SVHN- test set.
Conditional Coherence
Unconditional Coherence Conditional FID
Unconditional FID
MVAE
0.38 10.042
0.18 10.047
41.09 12.57
44.96 1:326
MMVAE
0.72 #0.014
0.34 #0.031
153.96 +7.73
111.72 #5.07
mmJSD
0.74 #0.013
0,02 =0.001
148.47 +4.38
291.59 +1.38
MoPoE-VAB
0.79 #0.015
0,46 =0.014
105.36 +1.49
101.43 +7.49
МVTСАE
0.66 10.031
0.15 20.040
38.34 #1.85
52.86 20.48
MMVAF+
0.53 10.012
0.20 10.005
54.69 10.86
77.35 1.337
CoDEVAE
0.82 #0.022
0.51 =0.015
79.11 #2.73
71.40 #1.93
2016) and are found by cross-validation over the val- ues 10.1, 1, 5, 10, 15, 20] and 00,0.2, 0.4, 0.6, 0.8]. respec- tively. addition, we consider / 2.5 the experiments on the PolyMNIST data to be consistent with the setup in (Palumbo et al., 2023). Given that the parameter p is clusive to CoDE-VAL VAE, all experiments present results that correspond to the value demonstrating the strongest over- all performance. We compare the performance of all mod- els across the entire range of 3 values. When only single values are reported, e.g., in tables or figures, the results correspond to the optimal for each method to ensure fair comparison different models achieve optimal per- formance different values (Daunhawer et al., 2022).
However, simple models such as MVAE, trades off tighter log-likelihoods for generative coherence. The CoDE-VAE models finds a balance between generative coherence and the approximation the joint likelihood, showing superior performance than all benchmark models for all beta values.
While generative coherence is important, the generated modalities should be of high-quality. Table compares generative coherence with generative quality measured by FID scores (Heusel et al., 2018). Both the MVAE and MVTСAE models are able to generate high-quality im- age modalities, but this is accomplished by compromis- ing generative coherence. In the family of models using MoE, MMVAE+ achieves the highest conditional quality, which is an obvious result of the modality- ables. However, this is achieved by trading off generative coherence. On the other hand, the CoDE- VAE model is able to generate highly coherent images without too much compromise of the generative quality.
vari-
4.1. MNIST-SVHN-TeS
An important aspect of multimodal VAES is their ability approximate the joint likelihood of multimodal data with- out sacrificing generative coherence. Fig. 2 shows the trade-off between generative coherence and log-likelihood estimation for all 33 values. Simple models based on MoE, e.g. mmJSD and MMVAE, show poor log-likelshoods, which is result of modality sub-sampling and clearly pre- vents a tight approximation as there is large amount of modality-specific information in the data. Even though MMVAE+ approximates the consensus distributions with the MoE, its log-likelihood estimations are tighter due to the factorization of the latent space that learns the modality- specific information. On the other hand, methods based on PoE all have relatively tight log-like ligeeiilddood estimations.
4.2. PolyMNIST
Generative quality gap: The generative quality gap (Daunhawer et al., 2022) defined as the difference mea- sured by FID scores between modalities generated with unimodal VAEs and multimodal VAEs. Methods that rely sub-sampling of modalities suffer from it and the gap increases with the number of modalities used for model training (Daunhawer et al., 2022). To assess the genera- tive quality gap, we train all models with 2-5 modalities in the PolyMNIST data, always using the modalities my and
6

---

Consensus of Dependent Experts Multimodal VAE
mmjSD MoPuF
MVTCAE MMVAF
CoDEVAE
minjSD MaPaF
MVTCCE
CoDEVAE
MMVAF
VAE
MNVAF
MMVAF
VAF
175
275
Conditiona FID
Uncondtional FID
225
125
75
75
Number of modalities
Number of modalities
Figure 3. Generative quality gap for modality mg a function of the number of modalities used to train the model.
0.9
0.6
Conditornl Cohereecc
Untrditional Coherence
MMVAE
MVAE
Conditional CatereeCCCChhrcc
Caherence
Uncoddtional Caherence
1.2
MMVAE
MVAE
0.6
MVAE
MNVVE
mijSiD
MoPoE VAE
0.6
NIAE
MMVAE
MmijDD
MoPaE VAE
0.3
MIjjSD
MVTCNEE
MAE
0.3
CODEVAE
MVTCCCE
MNVAE+
0.3
mmj5D
NVTCAE
MoPoEMAE
G.3
0.6
MVTCCAE
CODEVAE
MMVAE++
CaDEVAF
0.0g
0.0
CODEVAE
0.0
-0000
-5000
7000 -5000
-6000
1000-8000-7500-121
7000--700--60000 6500-6000
250 200
50
200
Logillkllhood
Log-llkelihodd
Conditiona FID
Unconditional FID
Figure 4. Trade-off between generative coherence (1) and likelihddd estimation (1), and between generative coherence and genera- tive quality (4) for 8 E 21, 5] on the PolyMNIST dataset.
mi and generating modality mo. For each of these four training scenarios, Fig. 3 shows that the generative quality gap increases with the number of modalities in all models using sub-sampling of modalities. Only the MVAE, MVT- CAE, and CoDE-VAE models, which do not rely on sub- sampling of modalities and MoE, have a linear or decreas- ing trend as the number of modalities increases. Only the CoDE-VAE model achieves the same unconditional gener- ative quality as unimodal VAEs when the number of modal- ities is increased to 4 and 5.
to the log- lig-likelihood estimations, MMVAE+ and CoDE- VAE find balance in the generative quality and gener- ative coherence of modalities. However, we would like to emphasize that MMVAE+ is not able to achieve sig- nificantly better FID scores than CoDE-VAE despite us- ing modality-specifft variables that improve the generative model.
noteworthy that MMVAE+ shows higher co- herence due to the low values in this experiment, which replicate the setup in (Palumbo et al., 2023). CoDE-VAE performs better at higher / values, achieving coherence of 0.38 at B = 10 for example.
Log-likelihoods, generative quality, and coherence: Fig. 4 shows the same pattern as for MNIST-SVHN- Text, where the MVAF and mmJSD models have a poor approx- imation of the joint distribution of the data. However, due to the moderate amount of modality-specifet information in PolyMNST, the generative coherence for these models not compromised, specially conditional coherence. We can corroborate that models using the PoE to approximate consensus distributions have higher log- likeliioisoo estima- tions. However, models like MVAE and MVTСAE trade off generative coherence. Both the MMVAE+ and CoDE- VAE models are able to balance the trade-off between gen- erative coherence and log- likelihoodd estimations.
4.3. CUB
To validate the performance of our proposed model on complex real-world data, we test the generative coherence and generative quality on the CUB data. Table 2 shows the quantitative results for the caption-to-image generative quality and coherence, while the qualitative results can be found in Appendix D.5. Conditional coherence is calcu- lated using the approach introduced in (Palumbo et al., 2023), as the captions describing the CUB data do not necessarily describe the shared content between modali- ties. The results obtained by MVAE, mmJSD, MoPoE, and MVTCAE reflect the complexity of the CUB data when real images are considered. While MMVAE is able to ob- tain relatively high coherence, its performance totally de- pends on the estimation of the variance parameter in the prior distribution. CoDE-VAE achieves relative high herence without compromising the quality of the generated images, which is not far away from MMVAE+ that uses
The pattern in generative quality is similar to that in the log-likelihoods estimation, except for the relatively poor FID scores achieve by the MoPoE model, specially in the conditional scenario. Both MVAE and MVTСAE gener- ate high-quality modalities but at the expense of genera- tive coherence, which is clearly not desirable. Similarly
7

---

Consensus of Dependent Experts Multimodal VATE
0.25
700
0.3
0.20
0.03
Aug 2.
0.02
500
0.1
400
0.00
0.00
0.0
(a) MNIST-SVHN -Txt
(b) PolyMNIST
(c) CUB
Figure 5. The bars show the learned coefficients (left axis), while red lines show the average trace of the covariance matrix of the distribution q( Xx) (right axis). All values are averages over the subsets with the same cardinality.
Uncendtoonal Debererce
0.5
23
leamable and o¹ pand
120
6.5
4
0.4
80
100
equal it and p
and p
-6.6
0.3
learnable p
0.2
MSF Optt-pt
40
equal it and p²
learnable a rand p
Op
6.8
7.0
Baseline
63
0.1
1900
1850
1800
20
2
3
0.4
Log iikellhood
number of modalities
number of modalities
(a) Optimal vs Baseline.
(b) Modality mg
(c) Modality m7
(d) Log-likelihoods
Figure 6. a) Contribution of learning Tu and cross- validatnin p (8 values at the top): b) and c): Effect of the coefficients Tu and correlation p on FII scores on PolyMNIST; Log-likelihoods (in thousands) for different (I's as function of p (3 0.1 the right axis).
Table 2. Caption-to-ina conditional generative quality and gen- erative coherence on the CUB dataset.
k ELBO term's 's contribution to optimization, since distri- butions for which experts are certain should have a greater contribution.
Conditional Coherence
Conditional FID
MVAE
0.27 #0.00 40.007
172.21 #39.61
4.5. Ablation Experiments: the effect of π and p
MMVAE
0.71 10.057
232.20 12.14
mmJSD
0.56 +0.158
262.80 +6.93
6a shows that considering dependencies between ex- perts and learning म yields higher performance across all beta values (blue error bars), compared assuming equal weights and independent expert distributions (orange eπrоr bars), using the MNIST-SVHN-Tess dataset. See Appendix F for results on PolyMNIST. Furthermore, we train the CoDE-VAE model optimizing Eq. 3 and using 2-5 modal- ities of the PolyMNIST data in the following scenarios: learning the weights æ and i) cross-validating p (denoted as p²) or ii) using p O; alternatively using equal weights Th. for all ELBOs and iii) cross-validating P iv) using 0. Figs. 6b and 66 reveal that learning and consid- ering dependence among experts improve FID scores for the generated modalities. Finally, Fig. 6d shows that aver- age joint log-likelihoods increase with higher p and lower B values.
Fig.
MoPoE-VAR
0.58 10.158
265.55 1:401
MVTСAE
0.22 10.007
208.43 11.0
MMVAE+
0.72 #0.090
164.94 +1.50
CoDEVAE
0.75 +0.050
175.97 #0.30
mopelity-specific variables to improve precisely this met- ric.
4.4. Contribution of each Lg to optimization
Fig. 5a shows the learned coefficients Ty (left axis) asso- ciated with each k-th ELBO (x(X) in the overall ELBO in Eq. 3, as well as the average trace (right axis) of the covari- ance matrix of the k-th distribution. In both cases, averaged over subsets with equal cardinality. The average My for sub- sets x
is lower than that of subsets X, > 2, and the ELBO that dominates optimization on the MNIST-SVHN- Text data is X 3. On the other hand, as the average trace increases, meaning that the distributions are relatively uncertain about their estimates, the average weight тę de- creases. Similarly, Fig. 5b and 5c show that for the PolyM- NIST and CUB datasets, subsets with more modalities con- tribute more to optimization and have smaller average trace values, indicating decreased uncertainty as modalities in- crease. The findings validate our motivation learn each
5. Conclusion
This research introduces CoDE, novel consensus of ex- perts method that, unlike existing approaches, takes into account the stochastic dependence between expert distri- butions through the experts' eгror of estimation. Based on CoDE, we introduce CoDE-VAE, an innovative mul- timodal VAE that optimizes each ELBO associated with every k-th subset within P(X) by learning its contribution
8

---

Consensus of Dependent Experts Multimodal VAE
T.- Extensive experimentation shows that CoDE-VAE ex- hibits better performance in terms of balancing the trade- off between generative coherence and generative quality, as well as generating more precise log- logllikelhhood estima- tions. As the CoDE-VAE optimization does not rely on sub-sampling of modalities, it reduces the generative qual- ity gap as modalities increase, which desirable prop- erty that is missing in most current methods. Further- more, CoDE- VAE achieves classification accuracy com- parable to that of current models. Finally, ablation exper- iments show the benefit of modeling the dependence be- tween expert distribution and learning the contribution of each ELBO term to optimization.
References
Allingham, J. U. and Nalisnick, E. A product of experts approach to early-exit ensembles. In Workshop at Inter- national Conference on Machine Learning, 2022.
Brown, A. D. and Hinton, G. E. Products of Hidden Markov Models. In International Workshop on Artificial Intelligence and Statistics, pp. 21-28. PMLR, 2001.
Chen, W. and Zhu, J. Multimodal Adversarially Learned Inference with Factorized Discriminators. In Proceed- ings of the AAAI Conference on Artificial Intelligence, 2021.
Acknowledgments
Cormen, T. H., Leiserson, C. E., Rivest, R. L., and Stein, C. Introduction to Algorithms. MIT press, 2022.
RJ, SY and MK affiliated with Visual Intelligence, Centre for Research-based Innovation funded by the Re- search Council of Norway (RCN) and consortium partners, grant no. 309439. RJ is affiliated with Pioneer Centre for AI, DNRF grant number PI. The work was further supported by RCN FRIPRO grant no. 315029 and RCN IKTPLUSS grant no. 303514. Finally, we acknowledge Sigma2 (Norway) for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Un- dertaking, hosted by CSC (Finland) and the LUMI consor- tium through project no. NN10040K.
Daunhawer, I,, Sutter, T. M., Chin-Cheong, K., Palumbo, E., and Vogt, J. E. On the Limitations of Multimodal VAEs. In International Conference on Learning Repre- sentations, 2022.
He, K., Zhang, X., Ren, S., and Sun, Deep Residual Learning for Image Recognition. In 2016 IEEE Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs Trained by a Two Time-Scale Up- date Rule Converge to a Local Nash Equilibrium. In International Conference on Learning Representations, 2018
Impact Statement
This research presents work whose main objective is to ad- vance the field of multimodal VAEs by introducing a con- sensus of experts method, which leverages the dependence between expert (single-modality) distributions, to approxi- mate consensus (joint posterior) distributions of any com- bination of different modalities. This is not trivial task and the existing literature usually assumes the indepen- dence between expert distributions for simplicity, which is an over-optimistic assumption in practice. Although the fo- cus of our work is on multimodal VAEs, our proposed con- sensus of dependent experts (CoDE) method has broader impact, previous research in other domains has aggre- gated information from expert distributions, such as natu- ral language processing to combine distributions of hidden Markov models (Brown & Hinton, 2001), image synthesis to combine modalities and generate images using genera- tive adversarial networks (Huang al.. 2022), large lan- guage models for comparative assessment of texts (Liusie et al., 2024), in early-exit ensembles (Allingham & Nalis- nick, 2022), or in diffusion models to aggragate distillation of diffusion policies (Zhou et al., 2024). There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Higgins, I., Matthey, I.., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. Veta-VAE: Learning Basic Visual Concepts with Con- strained Variational Framework. In International Con-
ference on Learning Representations, 2016
Hinton, G. E. Training Products of Experts by Minimiz- ing Contrastive Divergence. Neural Computation, 14(8): 1771-1800, 2002.
Hsu, W.-. and Glass, J. Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data, 2018
Huang. X., Mallya, A., Wang, T.-C., and Liu, M.-Y. Mul- timodal conditional image synthesis with product-of- experts gans. In European conference on computer vi- sion, pp. 91- Springer, 2022.
Hwang, H., Kim, G.-H., Hong, S., and Kim, K.-E Multi- View Representation Learning via Total Correlation OB- jective. In Advances in Neural Information Processing Systems, 2021.

---

Consensus of Dependent Experts Multimodal VAE
lsse, M., Tomezak, J. M., Louizos, C., and Welling, M. DIVA: Domain Invariant Variational Autoencoders. In Proceedings of the Third Conference on Medical Imag- ing with Deep Learning, pp. 322-348. PMLR, 2020.
Sutter, T. M., Daunhawer, I., and Vogt, J. E. Multi- modal Generative Learning Utilizing Jensen-Shannon- Divergence. In Conference on Neural Information Pro- cessing Systems, 2020.
Joy, T., Shi, Y., Torr, P. H. S., Rainforth, T., Schmon, S. M., and Siddharth, N. Learning Multimodal VAEs through Mutual Supervision. In International Confer- ence on Learning Representations, 2022.
Sutter, T. M., Daunhawer, I., and Vogt, J. E. General- ized Multimodal ELBO. In International Conference on Learning Representations, 2021.
Suzuki, M., Nakayama, K., and Matsuo, Joint Multi- modal Learning with Deep Generative Models. In Inter- national Conference on Learning Representations, 2017.
Kingma, D. P. and Ba, J. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2017.
Theodoridis, T., Chatzis, T., Solachidis, V., Dimitropou- los, K., and Daras, P. Cross-modal Variational Align- ment of Latent Spaces. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4127-4136, Seattle, WA, USA, 2020.
Kingma, D. P. and Welling, M. aato-EEcodiig Variational Bayes. In International Conference on Learning Repre- sentations, 2014.
Kleinman, M., Achille, A., Soatto, S., and Kao, J. Gaes- Korner Common Information Variational Autoencoder. In Conference on Neural Information Processing Sys- tems, 2023.
IEEE ISBN 978-
Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., and Salakhutdinov, R. Learning Factorized Multimodal Rep- resentations. In International Conference on Learning Representations, 2019.
Lee, M. and Pavlovic, V. Private-Shared Disentangled Mul- timodal VAE for Learning of Hybrid Latent Represen- tations. In Advances in Neural Information Processing Systems, 2020.
Vasco, M., Yin, H., Melo, S., and Paiva, A. Leveraging hierarchy in multimodal generative models for effective mrodstyy inference. Neural Networks, 146:238- 255, 2022.
Liusie, A., Raina, V., Fathullah, Y., and Gales, M. Effi- cient Ilm comparative assessment: product of experts framework for pairwise comparisons. arXiv preprint 2xXiv:2405.05894, 2024.
Vedantam, R., Fischer, I., Huang, J., and Murphy, K. Gen- erative Models of Visually Grounded Imagination. In International Conference Learning Representations, 2018.
Mancisidor, R. A., Kampffimeyer, M., Aas, K., and Jenssen, R. Discriminative multimodal learning via conditional priors in generative models. Neural Networks, 169:417- 430, 2024.
Wah, C., Branson, S., Welinder, P., Perona, P., and Be- longie, S. The caltech-ucsd birds-200-2011 dataset. 2011.
Palumbo, E., Daunhawer, I., and Vogt, J. E. MMVAE+: Enhancing the Generative Quality of Multimodal VAES without Compromises. In The Eleventh International Conference on Learning Representations, 2023.
Wang, W., Yan, X., Lee, H., and Livescu, K. Deep Varia- tional Canonical Correlation Analysis, 2017.
Winkler, R. L. Combining Probability Distributions from Dependent Information Sources. Management Science, 27(4):479 488, 1981.
Palumbo, E., Manduchi, L., Laguna, S., Chopard, D., and Vogt, J. E. Deep generative clustering with multimodal diffusion variational autoencoders. In The Twelfth Inter- national Conference on Learning Representations, 2024.
Wu, M. and Goodman, N. Multimodal Generative Models for Scalable Weakly-Supervised Learning. In Advances in Neural Information Processing Systems, volume 31, 2018
Rezende, D. J., Mohamed, S., and Wierstra, D. Stochas- tic Backpropagation and Approximate Inference in Deep Generative Models. In International Conference on Ma- chine Learning, 2014.
Wu, M. and Goodman, N. Multimodal Generative Models for Compositional Representation Learning, 2019.
Zhou, H., Blessing, D., Li, G., Celik, O., Jia, X., Neumann, G., and Lioutikov, R. Variational distillation of diffusion policies into mixture of experts. In Advances in Neural Information Processing Systems, 2024.
Shi, Y., N. S., Paige, B., and Torr, P. Variational Mixture- of-Exxerts Autoencoders for Multi-Modal Deep Gener- ative Models. In Advances in Neural Information Pro- cessing Systems, volume 32, 2019.
10

---

Consensus of Dependent Experts Multimodal VA
Appendix
A. Proofs
A.1. PROOF OF LEMMA
Proof. Let us rewrite the eπror of estimation as Ex = (Ms - utx), where u is a [M' D x D] design matrix with size M' vectors of along the diagonal and Os elsewhere. Furthermore, assume that the error of estimation has a Gaussian distribution ex (0, Ex). Therefore, the expectation of μau
E El =Blex u0
=Blea] + ul,
u0
(4)
Similarly, the covariance matrix
[[ Eluu))( [[(x))* =B((e + u0; - [[px))(ex + u0x E(ux])²]
=E((ex (exx)((ex [F(c))"
=
(5)
Given that is a linear function of random variable with multivariate Gaussian distribution, it follows that is also multivariate Gaussian, i.e, 14 ~ NuuO,, Ex).
A.2. PROOF OF LEMMA 2
Proof. Assuming an improper flat prior distribution on O2. and dropping the subscript k to avoid cluttering the nota- tion, the posterior distribution of the k-th consensus distribution proportional to the likelihood function, i.e. h(llp)) x N N(uO,. Ex). Therefore.
h(@ (4) exp
(p u0)² E- u01]
1 T Eー'1E -11² 1 - up 0 o-u'I-1 g-'nx'u u0]
exp
0 A0 2200000 B + +c]
cxp
10 AAA-AAA-- 0 -20" AA B + BdT A BBA-BB + C
exp
|AA0 T) A- (A0 Bi]
exp
which is an exponential form on 0. Taking the first derivative of log h(O μ1) wrt and solving for 0, we obtain the
expectation of the posterior distribution.
i log h(0))) -0A + B8
00
Similarly, the variance of the posterior distribution is given at -( 000 log h(0111))
0²
log (0) = A
ag2
Therefore, the posterior distribution
h(0)))) NAA 'B 1
(6)
also Gaussian, where A = B. B- I- the inverse matrix of Σ, u is [M D x D] design matrix with size M vectors of Is along the diagonal and Os elsewhere. and a [M' D x 1] vector with point estimates from relevant expert distributions on all dimensions of the k-th consensus distribution.

---

Consensus of Dependent Experts Multimodal VAEE
A.3. PROOF LEMMA
Proof. We introduce an indicator vector & that maps each k-th subset Xx E PXX) to I or O. We assume that both prior and posterior distributions of the indicator vector categorical, i.e. Cat( Therefore, the probability that the subset Xx. occurs is Th, where Ty π. Furthermore, assume the inference model q(z, { q(= (g)((( ({))).) where (xx
and (({ are variational distributions, which are conditionally independent given Xt.
Minimizing the sum of all the Kullback-Leible divergences between the intractable posterior distribution p(z, {)) p p( and the inference model, we obtain the following.
(())
log ( Elx,,) + log (x)
Σ KLLa(z, (x)]pl Xx
Σ
Ey(z xx)
X (1)(()(())((()))
%
(22, E\%{,)
(t Xx)
log
+2 log (XX)
p X 2))((z))((())
X
q 2 {(%A,
x By(x Xx)
log
+ log ((X),
p X (()
X
where in line we factor out 2M Assuming categorical prior distribution for p(E) with probability mass function v (1/K, 1/K), where K
a constant, which should not affect the optimization (Wu & Goodman, 2019).
2d - i.e. equal prior probabilities for all subsets, we obtain
log x(
lxg x
Σ
B (((x) %4
Σ
log p(X|z)p( (p() PP())
log p(X)
<(2 Xv ( (E)
EE x, [((1(1)))
log logp(X|>) + poz) - log g(2 - log ({E|Xx)
Σ
Buceixe Dlo P(E))
Xx
( + H H((((E)x)))
Xx
Note that the expected value of the indicator variable & iff the k-th subset occurs is
E[[]%
= E n = =10, ... 1k: ".,00 .0 [m], "r Tk. Tk.-..)
=
(Cormen et al., 2022) (Lemma 5.1 130), and that Ex. ( = logg p(E) is just a constant term Therefore, the variational lower bound for a single data point
logp(2 (X)
E{ ne
KLlg,( H((x(((Xx)) C,
Xx
where is the entropy function, A and & are the learnable weights of the neural networks parameterizing generative and variational distributions.
It is noteworthy that the entropy term (( (E)x)) decreases during optimization of the ELBO, as it is not optimized in isolation. Therefore, CODE-VAE learns uneven parameters m, shown in Fig. 5.
Alternatively, the lower hound could contain the Xxx)(((Xx))lpp divergence term. However, minimizing this divergence tilt the parameters vą towards 1/ K. Instead, we let the data speak and optimize the entropy the posterior distribution to learn æ.
12

---

Consensus of Dependent Experts Multimodal VAE
B. CoDE A simple example
At this point it is important note that all X in En are guaranteed to be full rank by construction, as oa > for all i, where o2e of. To see this, we need to show that the quadratic form (3 E"E'B O, is only satisfied for zero-vector (3. Further, let be the smallest Ou value, which is positive by construction. Therefore, Σi S 33 R >; E, B,B,. Since K. > O, the only solution that satisfies K: Σ, Ej 8rB, = the zero-vector B. Since X = 0 only for the zero-vector, Ed is positive definite and therefore invertible, and so is Σn- Ed is still invertible even if its off-diagonal elements are 0 given that of > O for all i. Without loss of generality, assume that we observe bimodal data X = (I1, with unimodal Gaussian data modalities. Therefore, the powerset is P(x) = ((a1), (I2), (11,,2)) implying that there are two expert distributions q( (<1[z1) and g(z|22), and one unknown consensus distribution ((](r1, i.c., K = 2 distributions in total, where M is the number of modalities. Note that there are exactly K scemarios where set of modalities may not be available test time. Therefore, we are interested in learning all K distributions, so that at test q(2 Kg), for ho 1...,. regardless if there unavailable sets. In what follows, drop the k
time we can draw subscript and superscript in the formulae. Furthermore, assume that the unknown parameter is 0 8, the expert estimates and their uncertainty are μ1 = 4. p2 = 8, of = 3, and o 1, and that the estimates have correlation p = 0.6. Therefore, we have that
1
4 0
cel c
3 0.6 v3 v1
0.6 v3 V1
/
and E
8
0
Then we can calculate the consensus distribution using Lemma 2 as follows:
A
A=
[1
1]
[an,1
01,2
01,1 + 0221 + 01,2 + 02,2
Q2,1
02,2
B= [1
1]
01,1
0122
μ1
= (a1,1 + a2,)) + (001 + (222)
02,1
02,2
[2]
and ~ (( [ara (antta Dan 2
where Σ = as...
01.10112 -02.2
CoDE subsumes PoE: Now, let us assume that p = 0. Hence,
0 0
1/02 0 0
Σ 2
Σ
0
o
1/02]
A = T1,1 + T2,2, B T1,1/1 + T2,2/2,, and the consensus distribution is g(2|01, T, = 1/0?, like in (Wu & Goodman, 2018). Therefore, for = 0 the CoDE parameters are simply PoE parameters. As shown by (Winkler, 1981), we can write the mean of the consensus distribution as
where
|CoDE = w1 + wa/22,
where W1 (03-po101))
(17-)))19)) show that the posterior parameter MCoDE a weighted average of+o3 o2 and W2 of of
of the expert parameters 11 and μı2 Note that the weights can be negative depending on the value of with respect to 01 and 02. In the above example, for p 0.6, the posterior mean is PCoDE = -0.02 * 4 + 1.02 * 8 and for p = 0 the mean becomes (CODE = 0.25 * 4 + 0.75 * In both cases, the weights sum up to 1. However, when the correlation between expert estimates is taken into account and the correlation is relatively high, the posterior mean leans even more towards the more accurate (with less variance) estimate. Fig. 7 shows both weights the weight for the less accurate estimate becomes negative. On the other hand, for p 0 the weights are always positive and (CCDE is convex function of the expert mean parameters. Similarly, we can write the variance of the consensus distribution as
function of p and, only for & greater than 0.5,
(1 -1-p1 11-p2)of03
d CDDE of + 0 2p00002
Note that when the dependence between experts is neglected, i.c. p = O, the variance of the consensus distribution is higher than it should be, as the denominator increases.
13

---

Consensus of Dependent Experts Multimodal VA
2.0
1.5
0.5
0.0
-15
Figure 7. The consensus parameter J&CoDE can be estimated as JCoDE = w1i + wap2, where the weights w1 and w2 are functions of the correlation parameter
Two expert distributions and 2D joint representation =: In this case, 0 10',02) and the matrices for the consensus distributions are
0
F
Hi 01
0
H1 01
e1
Σ
u
1
1x2
e
11
02
e2
Σ
0
1
p
02
where
Ed
of
po
10001
02
for d 1,2 the covariance matrix of the d-th dimension, and a 2x2 zero matrix. Therefore,
01,1
01
0
1
07
A = u u
0
07
Q2,1
0222
0
1
0
1
0
0
01,1
01,2
0
02211
02,,2
and
a1,1
01,2
0
0
p1
B=
1 1
0
07
02,1
0222
0
0
0
0
1
0
01,1
01,2
16
0
02,1
02,2
[1
where Σ-1 = Oiij for d d= 1,2.
C. Limitations
The dependence between expert distributions is captured in the parameter, which is found by cross-validation. For large and complex data, cross-valides validatiig can be computationally costly. Moreover, although CoDE is a principled Bayesian method, in which is possible to choose different prior distributions and likelihood functions, it not clear how multimodal VAEs can trained with different choices than the ones we make for these. Finally, although CoDE-VAN VAE has a relatively high computational cost O(2 1), model training is feasible even for 5-modality datasets on single GPU.
Overhead added by CoDE: To calculate consensus distributions with the CoDE method, we only need to find the inverse of each d matrix, which is an affordable computation. The average processing time for one batch of the MNIST- SVHN-Text data using CoDE is 46 milliseconds, which is not significantly higher than using PoE (Wu & Goodman, 2018) (36 milliseconds). For PolyMNIST data, where E 5 the average processing time for one batch is 1050 and 790 milliseconds using CoDE and PoE, respectively. Models are trained on single A100 GPU.
14

---

Consensus of Dependent Experts Multimodal VAE
Algorithm Minibatch version of the CoDB-VAE algorithm.
0, Φ. π <- initialize network parameters randomly and ny = 1/(2 - 1). Define by cross-validation.
repeat
X Random minibatch with all data modalities
Random samples from (V
for x E P(X') do
Σx + Define E using the encoder outputs of and pa;o for d = ...,.. 1,..,, D.
Ag. Bx f Use encoder outputs 14, and find the inverse of Σx
(= x + Expert and consensus distributions
>Aply Lemma 2
VLn (0, Φ TT, X E) + Get gradients
AAply Lemma 3
end for
0, Φ π f- Update parameters with the Adam optimizer until Convergence of 0, Φ π return 0, Φ.
D. Experiments
D.1. COMMON MODEL TRAINING
We optimize Eq. 3 by stochastic gradient descent using the reparameteriization trick (Kingma & Welling, 2014) for es- timating gradients (see Algorithm 1), and the Adam optimizer (Kingma & Ba, 2017) is used on all datasets. Models are trained on single A100 GPUs with AMD EPYC Milan processors with 24 cores. We find the optimal correlation parameter p by cross-validation using the values [0, 0.2, 0.4, 0.6, 0.8] and the off-diagonal in the covariance matrices d Lemma 1 are specified as Jz = 0z We cross- cross- validate only positive p values, as it reasonable to infer that the dependency between experts arises from common information about their underlying object. Following previous re- search, scale the Kullback-Leible divergence terms in Eq. 3 by regularization coefficient 3 (Higgins et al., 2016), Le.. [KL ( xxXx,)) [p))]] as it has significant impact on the evaluation of the model. The B value is found by cross- validation using the values [0.1. 1,5, 5, 10, 15, 20). We also consider / = 2.5 in the PolyMNIST data to replicate the setting in (Palumbo et al., 2023). For all datasets, we assume that the prior distribution is an isotropic Gaussian distribution, and the expert distributions are assumed to be multivariate Gaussian distributions with diagonal covariance matrix. All consen- sus distributions are approximated using Lemma 2. Finally, all experiments report the average performance and standard deviations of different runs and for the benchmark models use the following original implementation codes: MMVAE MVAE, mmJSD, and MoPoE http://gghhbbcoomooooommsstttttt MMVAE+ http://ghhhhoooommmmmllmmmmmeeeh
http://wwwwoomfsssddddeeeeeeeeee CAE hatp:///ggialloooeeefooo////t/ts
MVTT
D.2. EVALUATION CRITERIA
class in sklearn to fit linear classifiers with 500 latent representations of all subsets X E P(X). The only parameters that we specify are solver=' ibāgs', multi_class- auto' and max max_iter=3000. To test the performance of the classifiers on the entire test set, we use accuracy_score which is available in sklearn, for the MNIST-SVHNNTex and PolyMNIST data. See the released code for further details.
Linear classification: We use the
For each the modalities in the datasets, we train classifier networks that have the same architecture as their encoder, using original observations of the same modality. Then generate modalities conditioned on latent representa- tions of each subset X E PX)), including the empty set, in which case ~ p(z). To evaluate unconditional coherence, we classify the generated modalities conditioned on representations of the prior, calculate the number of generated modal- ities classified as having the same label, and divide it by the number of total modalities generated. To measure conditional coherence, we classify generated modalities conditioned on representations of subsets where the modality being classified not present, e.g., MNIST images can only generated conditioned on subsets containing SVHN Text, and SVHN& Text modalities, and calculate the accuracy_score
Coherence:
score e for the LINIST-SVNNTTen and PolyMNIST data.
Fréchet inception distance (FID): We use a TensorFlow pre-trained inception network to calculate the FID scores. Note that we tested our TensorFlow implementation with that of (Daunhawer al., 2022), which coded in PyTorch, and for
15

---

Consensus of Dependent Experts Multimodal VAEE
Table 3. MNIST encoder and decoder layers. All layers linear with ReLU activations. Finally, the number of input and output
dimensions in each layer is shown in the columns #F.In and Out,ut, respectively.
Encoder
Decoder
Layer
Type #F
#F.Out
Layer
Type
#F.In
#F.Out
linear 784
400
linear
20
400
2a
linear
400
20
2
linear
400
784
2b
linear
400
20
Table SVHN encoder and decoder layers. The last column for each model specifies the kernel size, stride, padding, and dilation. All layers are 2D convolutional (conv) and upconvolutional (upconv) in the encoder and decoder, respectively, with ReLU activations. Finally, the number of input and output dimensions each layer is shown in the columns #F and #F. #F.Out, respectively.
Encoder
Decoder
Layer
Type
#F.In
#F.Out
Spec.
Layer
Type
#F.In
#F.Out
Spec.
conv
3
32
14,,,1,1)
linear
20
128
2
conv
32
64
(4,2,1,1)
2
upсonv
128
64
(4,2,0,1)
cony
64
64
(4,2,1,1)
3
upсony
64
64
(4,2,1,1)
conv
64
128
(4,2,0,1)
upconv
32
(4,2,1,1)
5a linear
128
20
upconv
32
3
(4,2,1,1)
5b linear
128
20
a sample of PolyMNIST images, we obtained the same values. See the released code for details of the implementation. The conditional FID scores are calculated using generated modalities conditioned on representations of all subsets, while the unconditional FID scores are calculated using generated modalities conditioned on representations from the prior distribution. In both cases, generated images are compared with original ones. The conditional FID scores reported in Table and in Fig. are averages of generated images conditioned on all subsets and all modalities, while the conditional FID scores in Fig. 3 for modality mo are averages of generated images conditioned on all subsets. Finally, the scores in Table 2 correspond to generated images conditioned on the subset containing the caption modality.
D.3. MNIST-SVHN-
Data and training details: The dataset consists handwritten digits, images, and a text strings corresponding to an underlying digit. The triples are created in a many-to-many mapping. therefore, there are 1,121,360 and 200,000 obser- vations in the train and test sets, respectively. The digit text has 8 characters and its initial position chosen randomly, leaving the remaining characters blank. We train CoDE-VAE model with the Adam optimizer with default values and learning rate of 0.001, using mixed-precision to speed up model training. Both image modalities are assumed to have Laplace likelihoods, whereas the text modality is assumed to have a categorical likelihood. The dimension of the latent space is set to 20, as in (Sutter et al., 2020; 2021; ; Palumbo et al., 2023; Mancisidor et al., 2024). For a fair comparison of all models, we follow a similar approach as (Palumbo et al., 2023) to select the dimension of the modality latent space in MMVAE+. That is, divide the total number of dimensions in the latent space assumed by models without modality-specifi variables by the number of modalities.
We weights scale the decoder networks, as the distributions for the modalities have different scales. The weights calculated as in (Sutter et al., 2021), where the modality with the highest dimension is set to I and all the others are scaled by their relative ratio to the dimension of the data. Hence, the weights we use are 1, 3.9, and 384 for the SVHN, MNIST, and text decoders, respectively. Similarly, we scale the entropy loss by 1,000, a value which we found by cross-validation and that works well for all datasets in this research. The architectures that we use in the encoder and decoder networks are shown in Tables 3-5.
Classification and qualitative results: Fig. 8 shows the classification accuracy of linear classifier trained with latent representations for all subsets X e P()), averaged over equal cardinality subsets. We omit standard deviations as they are smaller than 0.01 for all models. Note that only the models using PoE and CoDE are able to achive higher classification accuracy as the number of expert increases. On the other hand, models using MoE achieves a flat performance despite having more modalities approximate the joint posterior distributions. The CoDE- CoDE-VAE model achieves slightly higher
16

---

Consensus of Dependent Experts Multimodal VAE
Table 5. Text encoder and decoder layers. The last column for each model specifics the kernel size, stride, padding, and dilation. All layers are ID convolutional (conv) and upconvolutional (upconv) in the encoder and decoder. respectively, with ReLU activations. Finally, the number of input output dimensions each layer is shown in the columns #F.In #F.Out, respectively.
Encoder
Decoder
Layer
Type
#F.In
#F.Out
Spec.
Layer
Type
#F.In
#F.Out
Spec.
conv
71
128
(1,1,0,1)
linear
20
128
2
conv
128
128
(4,2,1,1)
2
upconv
128
128
(4,1,0,1)
3
conv
128
128
(4,2,0,1)
3
upсonv
128
128
(4,2,1,1)
4a
linear
128
20
conv
128
71
(1,1,0,1)
4b linear
128
20
classification accuracy in subsets with 2 and modalities compared to that of all benchmark models. Finally, Fig. shows samples of modalities that are conditionally generated on representations of the prior distribution (top row), and on representations of the consensus distribution (= 3) (bottom row).
MVAE
mMjSD
MVTCAE
CODEVAE
MMVAE
MaPoE-VAE
MMVAE+
0.95
Classffication Accuracy
0.90
0.85
0.75
0.70
Cardinality of subset X
Figure 8. Classification accuracy of linear classifier trained with latent representations for all subsets X2 P(X). averaged over equal
cardinality subsets.
Table 6. Optimal parameters for all models in the classification MNIST-SVHN-TeN data.
MVAE
MMVAE
mmJSD
MoPoE-VAL
МVTСАЕ
MMVAE+
CoDE-VAE
5
0.1
5
10
20
5
0.4
D.4. POLYMNIST
Data and training details: the PolyMNIST dataset is built upon MNIST by varying the original background. A random 28x28 crop from 5 different images is used as the background to form a 5-modality dataset. Links to the images are provided in (Sutter et al., 2021), where the dataset was first introduced. Note that we keep the same order the links of the images define modality mo, m,, etc. Training and test sets have 60,000 and 10,000 images, respectively. We use the Adam optimizer with default values and learning rate of 0.001 to train CoDE-VAL models using mixed- preeediireiiionn.... Since all modalities have the same dimension, it is not necessary to scale the decoders in this case, but the entropy term is again scaled by 1,000. The architectures of the encoder and decoder are shown in Table 8, which are similar to the architectures in (Sutter et al., 2021) and not in (Daunhawer et al., 2022). We assume Laplace likelihoods. The dimensionality of the latent representations is set to 512 as in (Sutter et al., 2021: Daunhawer et al., 2022; Palumbo et al., 2023).
The results in Fig. are based on all 5 modalities of the PolyMNIST data, while the results in Fig. 3 and in Fig. 6 show
17

---

Consensus of Dependent Experts Multimodal VAE
0
3
3
4
6
7
9
2
9
9
3
6
6
6
9
7
OOP
8
0
4
4
9
0
/
9
4
2
3
0
9
9
9
5
7
3
5
8
7
7
2
2
2
9
0
2
2
5
/
O
6
2
6
9
4
3
2
9
6
2
7
7
7
5
0
7
0
5
/
0
3
5
4
8
8
3
9
2
8
5
8
2
4
8
2
/
6
8
5
8
0
6
5
3
3
7
0
6
6
9
5
2
6
3
0
5
2
5
6
9
4
8
4
3
2
/
0
2
4
9
5
8
9
7
Figure 9. Random generation of digits corresponding the MNIST, SVHN, and Text modalities. The decoders are conditioned on representations from the prior (top row) and the consensus q(= 3) distribution.
MVAE
MMJSD
MVTCAE
CoDEVAE
MoPoE
MMVAE+
1.00
Classification Accuracy
0.95
0.90
0.85
5
Cardinality subset x
Figure 10. Classification accuracy of a linear classifier trained with latent representations for all subsets X5 E P(X), averaged over equal cardinality subsets.
Table 7. Optimal parameters for all models in the classification PolyMNIST data.
MVAE
MMVAE
mmJSD
MoPoE-VAR
MVTСАЕ
MMVAE+ CoDE-VAE
5
5
0.4
FID scores obtained by the CoDE- VAE model trained with 2, 3, 4, and 5 modalities, i.e. we train our proposed model with four versions the data depending on the number modalities considered, which we ordered as follows:
2 modalities + {mo, m1}
18

---

Consensus of Dependent Experts Multimodal VAE
5.3
4
26
S
3
4/ :
DJ5
1
6 3
G
4
2
8
x9
U
7
2
1
e
19 R6
9
4
S. 42
0
O
4
v 8
2
6
3 Z
V 64 W81
8803
3 F 3035
Figure 11. Random generation of digits for all modalities. The decoders are conditioned representations from the prior (top row) and the consensus distribution g( X 5).
modalities + {mo, m1, m2)
4 modalities + {mo, m1, m2, m3}
modalities {mo. m1, m2, mg, m4)
Given that we cross-validate $, p. and x (for the ablation experiments) we trained 120 different configurations assuming 2, 3, 4, and 5 modalities, i.e. 480 experiments that we run 3 different times each. Following a similar approach to (Daunhawer al., 2022), compared the quality of the generative model choosing the architecture with the smallest FID scores.
Classification results are shown in Fig. 10 where standard deviations are omitted they smaller than 0.01 for all models. We observe similar trend in the classification results for the MNIST- SVHN-Text data. Models using MoE are not able to learn extra information that is useful for classification the number of modalities increases. Note that our results for the mmJSD model do not agree with those presented in (Huang et al., 2022) that show increasing accuracy similar to that of MVTCAE. Such a trend is not common in models using MoE and, therefore, we believe that they are wrong. The MVTCAE model achieves a perfect classification accuracy for subsets with and 5 models, followed by MoPoE, CoDE-VAB and MVAE.
Classification and qualitative results:
Fig. 11 shows samples of all 5 modalities that are conditionally generated representations of the prior distribution (top row), and on representations of the consensus distribution g(>|X 5) (bottom row). In Fig. 12, we show random generated digits corresponding to modality 1 (top row) and modality (bottom row), as a function of the number of modalities in the dataset, i.e. the first column shows generated images with CoDE-VAE model trained with modalities, while the last column shows images generated with model trained with 5 modalities. Finally, Fig. 13 shows the average coherence and classification accuracy a function of correlation p. obtained with CoDE-VAL models trained with 2, 3, 4, and 5 modalities. The average is calculated over all subsets in each case, i.c. 3. 7, 15, and 31 subsets, respectively. In all cases, the coherence and classification is higher when the correlation between experts is taken into account, except when CoDE-VAE is trained with modalities.
Finally, Fig. 14 shows random generations of the modality my by unimodal VAE (top row) and CoDE-VAR which were used to calculate the FID scores in Section 4.2 Generative quality gap. The generations of both models are visually indistinguishable.
D.5. CUB
Data and training details: the CUB data used in this research are composed of two modalities: bird images and 10 different detailed descriptions for each image. Therefore, there are in total 1 17,880 pairs of image-captions, where 88,550 are used for model training and 29,330 for testing. This bi-modal version of the CUB dataset has been used in (Shi et al., 2019 Daunhawer et al., 2022; Palumbo et al., 2023). However, (Shi et al., 2019) uses a simplified version that replaces bird images with ResNet embeddings. The architectures for the encoder and decoder caption modality are in Table 14, and
19

---

Consensus of Dependent Experts Multimodal VAE
3
Y
a
y
5
6
5
9
5
P
9
7
4
3
8
2
5
9
9 7
4
4
7
3
Y
2
?
7
3
7
Figure 12. Random generation of digits in modality mao (top row) and modality my (bottom row). The first column shows images of digits generated with CoDE-VAE model trained with 2 modalities. Similarly, the second, third, and last columns show images generated by CoDE-VAE trained with 3, 4, and 5 modalitics.
Table 8. PolyMNIST encoder and dccoder layers. The last column for each model specifies the kernel size, stride, padding, and dilation. All layers 2D convolutional (conv) and upconvolutional (upconv) in the encoder and decoder, respectively, with ReLU activations. Finally, the number of imput and output dimensions in each layer is shown in the columns #F. In and #F.Out, respectively.
Encoder
Decoder
Layer
Type
#F.In
#I.Outt
Spec.
Layer
Type
#I.In
#F.Out
Spec.
conv
32
(3,2,1,1)
linear
512
2048
conv
32
64
(3,2,1,1)
2
upconv
2048
64
(3,2,0,1)
conv
64
128
(3,2,1,1)
3
upconv
64
32
(3,2,1,1)
4a
linear
128
512
4
uрсoпи
32
3
(3,2,1,1)
4b
linear
128
512
mod
3 mod
med
mod
mod
mod
mod
5 mod
0.85
0.90
0.80
Caherence Acc
0.80
0.75
0.75
0.70
0.65
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Figure 13. Average coherence and classification accuracy as function of correlation p. obtained with CoDE-VAE models trained with 4, and modalities. The average calculated over all subsets each case, 3, 7, 15, and subsets, respectively.
for the image modality please see the released code. The dimensionality of the latent space is set to 64 as in (Daunhawer et al., 2022; Palumbo et al., 2023) and the weights to scale the decoders are 0.0026, and 1.0 for the image and caption modalities, respectively.
We follow the approach introduced in (Palumbo et al., 2023) to calculate the conditional coherence. First, we construct the following caption "this bird is completely <color>" where color any color (white, yellow, red, blue, green, gray,
20

---

Consensus of Dependent Experts Multimodal VAE
4
4
7
7
9
6
6
7
9
14
9
4
/
5
8
7
3
2
4
4
7
4
3 7
7
0
3
3
3
8 1
6
5
7
9
8
9
J
3
9
6
7 4
4
S
3
O
2
6
6
(a) Unconditional generation of modality ma by VAE.
(b) Conditional generation of modality mo by VAE.
f
f
0
6
3
9
2
1
3
E
6
S
9
5
2
9
3
7
2
6
2
9
4
/
3
2
5
57 8
7
2
M
7
6
9.
f
&
9
3
6
4
3
5
+
6
4
4
3
5 6
R
1
O
6 7
9
2
3
Z
A
S
3
0
7
6
8
3
2
3
2
9
2
3
(c) Unconditional generation of modality me by CADE-VAE. (d) Conditional generation of modality mg by CoDE-VAE.
Figure 14. The top row shows random generations by unimodal VAE, while the bottom row corresponds to generations by CoDE-VAE.
The conditional generations by CoDE-VAE are based on the full subsets.
brown, black). Then, for each of the eight captions, we generate ten images (eighty in total). Finally, we count the number of pixels that are within the boundaries of hue, saturation, and value (HSV) colors in Table 13, and an image is said to be coherent if any of the two classes of the highest pixel count the same as the color the initial caption. Note that we use the library 2v2 and the method cvtColor obtain the HSV values of the generated images.
Qualitative results: Fig. 15 shows the generated images for the coherent test previously explained. The row at the top shows the captions that are used to generate the image modality. On the otter hand, Fig. 16 shows images generated from original images in the CUB test sets, which are in the first row. Both grid of plots show that the generated images are able capture the details in the modality that they are conditioned on. Finally, Fig. 17 shows generated images from original captions in the test set.
E. Additional Experiments
Section 4 presents the main relevant benchmark models and the results against which the performance of these models should be compared. However, for completeness, this section analyses the performance of CoDE-VAN challenging bi-modal CelebA data (Sutter et al., 2020), and compares the generative quality and generative coherence of CoDE-VAE with that of the clustering multimodal VAE (CMVAE) model introduced in (Palumbo et al., 2024). Although the contribution of the CMVAE model is relevant to the field of multimodal VAES, its line of research is orthogonal to CoDE-Ve VAE, as CMVAE leverages clustering structures in the latent space by introducing a flexible prior distribution based on a mixture model and uses diffusion decoders.
the real and
21

---

Consensus of Dependent Experts Multimodal VA
this thi this this
completely completely completely completety completely completely completety completttly yellow
red
green gray
bronn
black
Figure 16. Image-to-image generation. Original images the first row.
Figure 15. Caption-to to image generation used in the coherence test.
and has
with grey wings
breast and
brown beily
color colored beat
Figure Caption-to-image generation for the captions in the first row.
E. CELEBA
We test the performance of CoDe-VAE and MMVAE+ on the bi-modal CelebA data which are composed of images and text descriptions for each image. The description modality is based on 40 attributes that describe each image and is fixed at 256 characters. For images with shorter descriptions, the remaining spaces are filled with a "*** character. The train and set sets have 162,560 and 19,712 observations, respectively. We train the CoDE VAE model with default values in the Adam optimizer and a learning rate of 0.0001, using mixed precision and assuming a Laplace likelihood for the image modality and categorical likelihood for the text modality. Furthermore, assume p 0.6, 3 1, and latent space assume 3 and both modality-seecific and common latent variables have 16
with 32 dimensions. For MMVAE+, dimensions each. Hence, the decoders in both models generate modalities based on 32 dimensions. The architectures for the encoder and decoder are based on residual networks (He et al., 2016) and are shown in Tables and 10.
Table 11 shows the performance of CoDE- VAE and MMVAE+ in terms of generative quality, generative coherence, and classification accuracy. CoDE-VAE achieves better generative quality and classification accuracy even when the param- eter was not cross-validated. Fig. 18 shows some random faces, which are conditionally generated on representations of the prior distribution (left plot) and on representations of the consensus distribution q(2) X = 2) (right panel). Note that some of the images conditionally generated on the consensus distribution q(> x = 2), have not only sharper faces, but also sharper backgrounds. On the other hand, the top row of Figure 19 shows faces conditionally generated on representations
22

---

Consensus of Dependent Experts Multimodal VAE
Table CelchA encoder and decoder layers for images. The last column for cach model specifies the kernel size, stride, padding. and dilation. Layers are 2D convolutional (conv), upconvolutional (upconv), residual blocks (res), and upconvolutional residual blocks (res upeonv), with ReLU activations. Finally, number of input and output dimensions each layer shown the columns #F.In and #F.Out, respectively.
Encoder
Decoder
Layer
Type
#F.In
#F.Out
Spec.
Layer
Type
#F.In
#F.Out
Spec.
conv
3
128
(3,2,1,1)
1
linear
32
640
res
256
(4,2,1,1)
2
res res_upconv
640
512
(4,1,0,1)
3
res
256
384
(4,2,1,1)
3
res upсonv
512
384
(4,1,1,1)
4
res
384
512
(4,2,1,1)
4
res_upconv
384
256
(4,1,1,1)
5
512
640
(4,2,1,1)
5
res res_upcony
256
128
(4,1,1,1)
6a
linear
640
32
6
upconv 128
3
(3,2,1,1)
6b
linear
640
32
Table 10. CelebA encoder and decoder layers for text descriptions. The last column for each model specifies the kernel size, stride, padding, and dilation. Layers are ID convolutional (conv), upconvolutional (upconv), residual blocks (res), and upconvolutional residual blocks (res upconv), with ReLU activations. Finally, the number of input and output dimensions in each layer is shown in the columns #F.In and #F.Ott respectively.
Encoder
Decoder
Layer
Type
#F.In
#F.Out
Spec.
Layer
Type
#F.In
#F.Out
Spec.
71
128
(4,2,1,1)
linear
32
640
2
128
256
(4,2,1,1)
2
res_upconv
640
640
(4,1,0,1)
3
res
256
384
(4,2,1,1)
3
res upconv
640
640
(4,2,1,1)
4
res
384
512
(4,2,1,1)
4
res res_upconv
640
512
(4,2,1,1)
5
res
512
640
(4,2,1,1)
5
res upponv
512
384
(4,2,1,1)
res
640
640
(4,2,1,1)
5
res res uppoonv
384
256
(4,2.1,1)
7
res
640
640
(4,2,0,1)
5
res upconv
256
128
(4,2,1,1)
8a
linear
640
32
6
upconv
128
71
(4,2.1,1)
8b
linear
640
32
Table 11. Model performance of CoDE-VAE and MMVAE+ the CelebA data.
CoDE-VAB
MMVAE+
Conditional FID (4)
92.11 10.61
97.30 10.40
Unconditional FID (L)
87.41 20.36
96.91 10.42
Conditional Coherence (f)
0.38 40.001
0.46 10.001
Unconditional Coherence (t)
0.23 +0.003
0.31 #0.030
Classification (t)
0.38 1.0002
0.37 10.003
of the expert distribution q=2(Xtext) Note that some attributes, such gender, smile, o' o'clock shadow, relatively easy to capture in the generated face. The bottom row of Figure 19 shows faces conditionally generated representations of the consensus distribution q(= [ x 2). For both cases, we added the text modality that describes the face attributes.
E.2. POLYMNIST
We follow the same experimental setup in Scction 4.2 and compare the generative quality and generative coherence of all models, including CMVAE, which is shown in Fig. 20. CMVAE achieves significantly higher performance in unconditional coherence, as it uses flexible prior distribution composed of a mixture model (one component for each cluster in the latent space), Such prior distribution is certainly more informative compared to the non- con-informative isotropic Gaussian used in CoDE-VA VAE and most of benchmark models. However, CoDE VAE achieves about the same performance in conditional coherence and conditional FID scores. Interestingly, despite its more complex architecture, CMVAE not able to achieve significantly better unconditional and conditional FID scores.
VODE-VAE
23

---

Consensus of Dependent Experts Multimodal VA
Figure 18. Random generation faces. The decoder is conditioned on representations from the prior (left plot) and the consensus
qx
X = 2) distribution (right plot).
begrundemyye
begrundereyes
attractive
bignose
lighose
Mondhar
bangs
bagoundereyes
troenher
bageunderoyes highcheelioones
attractive heawymakee
bnownthai
grayhaii
lagios
big
Bualyyeycbross
tearymakkuu
tearyymakuup
mouthsiigtyyppe
hesvymakeup
rebeard
moudtnulgghtlysseen
mnuuhniighhlysppn
tecenthar
malls
Soclodahhatoo
male
Fag highchookbbnees
moudtniigghtyypee
smillig
pointyngge
Ovaiface
male
ovaiface
Isgoundertyes
sarraecyee
rebeard
ovalface
mhaiiigharrinee
weinngllpttek
wavyyhai
rebeard
Ivallace
draaghthar
iddebmm
navyyaair
Mraaghtthat
wearinghedüace
straighthii
lestingllstick
smilling
smiling
wearingllssiik
dtraigthair
wearinghot
wearing lastick
wearingllpbbik
beg Bagsundereyes
tigigs
attractive brownhaii
Mackhair
eegrundereyes
attractive heavymakeuu
Highcheesbones
bald
bengs
meutthslghelyypee
bignoss
bignose
Bagsundereyes
highcheokoones
rebcard
dbablechin eveglassess
brownhair
highcheekloone
nglicheekbones
bangs
ricedinhairtine
tearymakeup
male
male
trownhair
ovaifacc
highcheelboees
mouthssighhhygee
mouthstigttyrpen
pointynose
highcheesbones
robeard
pointynase
waryhair
seatthgearrngs
male
nobeard
polntynose
dtriighthair young
wearthglesttck young
wearingllostiic
gllootik young
costtnglasticc young
male young
voortegnedkaaee
rebeard
wearingnediiaee
smling
young
young
young
Figure 19. Conditionally generated faces on representations of the expert distribution q(> (eee)) (top row) and the consensus distribution
q(2 x 2) (bottom row), where added the text modality that describes the face attributes.
F. Effect of weights and correlation P
We train the CoDE-VAE model using equal weights π and assuming p 0 (Baseline), and learning the weights and cross validating E [0.2, 0.4, 0.6, 0.8] (Optimal), to assess its effect on unconditional coherence and log-likelihoods across different 3 values. Fig. 21 shows that for all 3 values, but 3 = 1, the Optimal models achieve higher coherence, higher log-likes lieelike,,ood, or both, relative to the Baseline models when using the PolyMNIST data. For 8 = 1, the coherence of the Baseline model is slightly higher, but the Optimal model has higher likelihood.
24

---

Consensus of Dependent Experts Multimodal VAE
Unconditional Coherence
MVAE
Conditional Coherence
1.2
MVAE
MMVAE
MMVAE
mMjSD
0.9
0.6
mmjSD
MoPoE-VAE
MoPoE-VAE
MVTCAE
0.6
MVTСAE
MMVAE+
0.3
MMVAE+
0.3
CMVAE
CMVAE
CoDEVAE
0.0
0.0
CoDEVAE
200
150
100
50
300
250 200
150
100
50
Unconditional FID
Conditional FID
Figure Trade-off between generative coherence (f) generative quality (L) for 9 E [1, 2.5, 5] on PolyMNIST dataset.
0.5
PolyMNIST Optimal
PolyMINST Baseline
Unconditional Coherence
0.4
20
15
0.3
0.2
0.1
0.1
-6600
-6600
-6400
-6200
Log-likelhhood
Figure 21. Effect of learning the my weights and cross-validating E 0.44 0.6, across different 9 values, which are annotated the top-right seatters.
Table 12. Effect of edge case correlated modalities, assuming p 0 and 0.9, on generative quality measured by FID scores (L).
0%
25%
95%
0.0
29.00
31.27
48.22
0.9
26.12
29.90
53.68
Table 15 shows the generative coherence, generative quality, and classification aceuracy as function of p on the MNIST- SVHN-Text dataset. All values are caleulated at B 20. All metries show significant improvement for p 0.
Edge case correlated modalities: To better understand the behavior CoDE-VAE. use the modality m, PolyM- NIST in the following way. We apply three different levels of noise the modality my: 0 %, 25%, and 95%. Then, paired the noisy version with the original modality
to obtain bi-modal data. For each of these data, we train CADE-VAE assuming = 0 and 0.9 and generate the non-noisy version of m, When CoDE-VAL is trained with the data with 0% noise, both modalities are the same and we expect that the model assuming generative quality. On the other hand, when CoDE- VAE is trained with the data with 95% noise, the modalities are uncor- related and using captures the dependency between experts distributions through the p parameter.
0.9 will have relatively high
expected to have relatively high generative quality. Table 12 shows that CoDE- VAE correctly
25

---

Consensus of Dependent Experts Multimodal VA
Table 13, HSV color boundarics used in the coherence test of the CUB data.
White
Yellow
Blue
Red
lower bound upper bound
(0,0,120)
[25,50,70]
[90,50,70] (158,255,255]
[0,50,70] [15,255,255]
[159,50,70]
1880,1
[35,255,255]
[180,255,255]
Green
Gray
Brown
Black
lower bound upper bound
[36,50,70]
[0,0,50]
[24,255,255]
[0,0,0]
[89,255,255]
(80,18,120)
[16,50,70]
[180,255,50]
Table 14. Encoder and decoder layers for captions the CUB data. The last column for each model specifies the kernel size, stride, padding, and dilation. Layers are dense with linar activations (linear), 2D convolutional (conv), and upconvolutional (upconv) with ReL.U activations. Finally, number of input and output dimensions each layer is shown the columns
#F.Out, respectively.
Encoder
Decoder
Layer
Type
#F.In
#F.Out
Spee.
Layer
Type
#F.In
#F.Out
Spec.
linear
1590
128
upconv
64
512
(4.1,0,1)
conv
128
32
(4,2,1,1)
2
uрсonv
512
256
(11,4),)))))11111
conv
32
64
(4,2,1,1)
3
upconv
256
256
(3,1,1,1)
conv
64
128
(4,2,1,1)
upconv
256
128
(1,,4)
conv
128
256
(1,)
upconv
128
128
(3,1,1,1)
conv
256
512
(1.2).)))))
upсonv
128
64
(4.2,1,1)
7a
512
64
(4,1,0,1)
upconv
64
32
(4,2,1,1)
7b
conv
512
64
(4,1,0.1)
7
upconv
32
1
(4,2,1,1)
8
linear
1590
Table 15. Comparison of generative coherence, generative quality, and classification accuracy as function of the MNIST-SVHN-
Text dataset.
Coherence f
FID
P
Unconditional
Conditional
Unconditional
Conditional
Classification 1
0.0
0.52
0.70
78.2
95.3
0.88
0.2
0.53
0.78
77.5
87.4
0.93
0.4
0.51
0.82
76.7
83.6
0.95
0.6
0.55
0.81
79.4
87.0
0.95
0.8
0.54
0.77
77.0
91.2
0.95
26


Reviews:
[{"review_id": "7CmAEMVi1x", "reviewer": "Reviewer_FV9z", "summary": "This paper introduces the Consensus of Dependent Experts (CoDE) in the context of multimodal learning with Variational Autoencoders (VAEs). Current approaches for this task, such as: (i) the product of experts; or (ii) the mixture of experts assume cross-modal independence which is restrictive. Towards this end, the current work proposes a novel Empirical Lower Bound (ELBO) that estimates the joint likelihood by learning the contribution of each modality. The proposed method can strike a balance between generative coherence and generative quality. Empirical evaluations are conducted on several datasets.\n\n## Update after rebuttal\nConvinced with the responses to my questions by the authors. Hence, raising my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful and comprehensive comments. We will address your concerns and questions, point by point:\n\n## How accurately is ELBO minimized: \nWe are not completely sure if we follow your concern. If the comment refers to how the ELBO is maximized (the loss), we train the CoDE-VAE model until the ELBO converges, confirmed by visual inspection. [These plots](https://anonymous.4open.science/r/codevae_icml-27EA/) ([backup](https://anonymfile.com/LNara/elbo-cub.png) [backup](https://anonymfile.com/Vxpdx/elbo-mmnist.png) [backup](https://anonymfile.com/8p9jO/elbo-mst.png)) show the convergence of the ELBO. If your concern is about how close the ELBO is to the intractable marginal log-likelihood, we calculate log-likelihoods on the test sets using importance-sampling (shown in Figures 2 and 4 for all models). Please let us know If we are misunderstanding your concern. \n\n## $\\Sigma_d^{-1}$ costly in high-dimensional data.\n$\\Sigma_d$ is not a sample covariance matrix and its size depends on the number of expert distributions assessing consensus distributions (CD). So, it is limited by the number of modalities M, which typically is small and only one CD is conditioned on all modalities. In our research the largest M is 5 (PolyMNIST). Therefore, the computational costs to find the inverse of $\\Sigma_d$ are affordable.\n\n## $\\Sigma_d$ if it is not full rank.\n$\\Sigma_d$ is guaranteed to be full rank by construction, as $\\sigma_{i,j}>0$ for all $i,j$, where $\\sigma_{i,i}=\\sigma^2_i$. To see this, we need to show that the quadratic form $\\beta^T\\Sigma_d\\beta = 0$, is only satisfied for a zero-vector $\\beta$. Let $\\kappa$ be the smallest $\\sigma_{i,j}$ value, which is positive by construction. Therefore, $\\sum_i \\sum_j \\beta_i \\sigma_{i,j} \\beta_j > \\kappa \\sum_i \\sum_j \\beta_i \\beta_j$. Since $\\kappa>0$, the only solution that satisfies $\\kappa \\sum_i \\sum_j \\beta_i \\beta_j=0$ is the zero-vector $\\beta$. We will add this discussion in the revised version.\n\n## More complex datasets\nWe added a new section in the appendix with experiments on CELEB-A for CoDE-VAE and MMVAE+, which is the model that stands out in the other experiments. Due to the limited time, we evaluate both models only for $\\beta=1$, assuming a latent space with 32D. For CoDE-VAE we assume $\\rho=0.6$, as we observed in other experiments to be a value that performs consistently well. We obtained the following results:\n\n|                           | CoDE-VAE  | MMVAE+    |\n|---------------------------|-----------|-----------|\n| Conditional FID           | 92.11 (0.61)      |  97.30 (0.40)    |\n| Unconditional FID         | 87.41 (0.36)      |  96.91 (0.42)    |\n| Conditional Coherence     | 0.38 (0.001)      |  0.46 (0.001)    |\n| Unconditional Coherence   | 0.23 (0.003)      |  0.31 (0.030)    |\n| Classification            | 0.38 (0.066)      |  0.37 (0.003)    |\n\nwhich may improve further by cross-validating $\\rho$ in CoDE-VAE.\n\n## Generation Quality and Classification accuracy\nMultimodal VAEs trade high generative quality with reduced generative coherence [1]. The experimental setup in our research shows that CoDE-VAE performs as well as or better than SOTA multimodal VAEs in terms of balancing the trade off between generative coherence and generative quality. When we assess in isolation whether multimodal VAEs are able to improve generative quality as the number of modalities increases, CoDE-VAE clearly shows a higher performance (Figure 3). It is possible to add modality-specific (MS) latent variables to CoDE-VAE to further improve the generative quality, which requires a careful design of the number of dimensions in the common and MS variables to avoid the short-cut problem [1]. When it comes to the classification results, Figures 8 and 10 in the appendix show that models using the mixture-of-experts are not able to achieve higher classification accuracy as latent representations are learned from subsets with more modalities. On the other hand, CoDE-VAE ranks 1st and 3rd, while balancing the trade-off between generative quality and generative coherence at the same time.\n\n[1] Palumbo et al. Enhancing the Generative Quality of Multimodal VAEs Without Compromises, ICLR, 2023.\n\n## Typos and L89, Col 1\nYou are correct that there is a typo in L89, Col 1. The sentence should read *\"Furthermore, CoDE-VAE minimizes the generative quality gap as the number of modalities increases...\"*, referring to the results in Figure 3. We will fix this in our revised version of the paper.\n\n## Softmax instead of categorical\nWe assume that you are referring to the Gumbel-Softmax distribution, which is useful when we need to sample and backpropagate at the same time. As our main interest is learning the $\\pi$ parameters, we do not expect a significant different behavior by using the Gumbel-Softmax distribution. The main concern when using a different distribution, is whether the entropy term in the ELBO of the CoDE-VAE model can be evaluated, ideally in closed form.", "claims_and_evidence": "Generally, the claims make sense. However, the chief concern with the claims is:\n1. The paper does not show how accurately the ELBO is minimized for the different datasets. This is an important shortcoming of the present work.", "methods_and_evaluation_criteria": "While overall the method is intuitive, the chief concern is as follows:\n1. L201, Col 2: Estimating the off-diagonal elements in $\\Sigma_d$ in the forward pass could be computationally expensive for high-dimensional scenarios. \n2. Also, how does the model deal with cases, where $\\Sigma_d$ is not full rank.", "theoretical_claims": "Generally, the theoretical claims seem accurate.", "experimental_designs_or_analyses": "Overall, the experimental evaluation is pretty broad but has the following shortcomings:\n1. Some of the more complex real-world image datasets, CELEB-A [a], CELEB-HQ [b] have not been experimented with.\n2. Generation Quality (as measured by FID scores) and Classification accuracy are not the best.\n\nReferences:\n[a] Liu, Z., Luo, P., Wang, X. and Tang, X., 2018. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018), p.11.\n[b] Karras, T., Aila, T., Laine, S. and Lehtinen, J., 2018, February. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In International Conference on Learning Representations.", "relation_to_broader_scientific_literature": "The idea of combining multimodal distributions relate to prior works that combine hidden markov model distributions (Brown & Hinton, 2001), image synthesis to combine modalities and generate images using generative adversarial networks (Huang et al., 2022), large language models for comparative assessment of texts (Liusie et al., 2024), in early-exit ensembles (Allingham & Nalis\u0002nick, 2022), or in diffusion models to aggregate distillation of diffusion policies (Zhou et al., 2024).", "essential_references_not_discussed": "Most important related works have been discussed.", "other_strengths_and_weaknesses": "Other strengths:\n1. The proposed model can deal with scenarios when one or more modalities are missing.\n2. The current formulation allows for the estimation of uncertainties of each of the experts.\n\nOther weaknesses:\nSee previous sections.", "other_comments_or_suggestions": "The following are some additional comments:\n1. L98, Col 1: \"....method, the derivation...\" -> \"....method, for the derivation...\"\n2. L376, Col 1: \"...significant...\" -> \"...significantly...\"", "questions_for_authors": "The following are some questions for the authors:\n1. L89, Col 1: Does the proposed approach really \"minimize the generated quality as the number of modalities increase\". This seems counter-intuitive. Or is this a typo? \n2. What would happen if instead of a Categorical distribution, a softmax distribution is used to weigh the experts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8MnseFXRQC", "reviewer": "Reviewer_kGwL", "summary": "The paper introduces a new method for aggregating multimodal expert distributions in Variational Autoencoders (VAEs) by incorporating the dependence between experts, which has traditionally been ignored in models like the product of experts (PoE) and mixture of experts (MoE).  This method, called Consensus of Dependent Experts (CoDE), aims to improve the estimation of joint likelihoods in multimodal data by accounting for the dependencies between different modality-specific distributions.  The paper proposes the CoDE-VAE model, which enhances the trade-off between generative coherence and quality, improving log-likelihood estimations and classification accuracy.  The authors claim that CoDE-VAE performs better than existing multimodal VAEs, especially as the number of modalities increases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and detailed comments. We will address your concerns and questions point by point.\n\n## Experiments and analysis on edge cases\nWe agree that analyzing CoDE-VAE on edge cases helps to understand its behavior, and makes our research more robust. We believe that the experiments in Section 4.2 Generative quality gap (Figure 3), in Appendix D.4. PolyMNIST (Figure 13), and the classification results in Figures 8 and 10 provide a good indication that the performance of CoDE-VAE improves with the number of modalities and with the cardinality of the subset on which consensus distributions are conditioned on. \n\nTo address the scenario where the assumption about dependent experts may not hold, we trained the CoDE-VAE model on PolyMNIST using the modality $m_1$ in the following way. We apply 3 different levels of noise to the modality $m_1$, 0 \\%, 25\\%, and 95\\%. Then, we paired the noisy version with the original modality to obtain a bi-modal data. For each of these data, we train CoDE-VAE assuming $\\rho=0$ and $\\rho=0.9$, and generate the non-noisy version of $m_1$. When CoDE-VAE is trained with the data with 0\\% noise, both modalities are the same and we expect that $\\rho=0.9$ will have relatively high generative quality. On the other hand, when CoDE-VAE is trained with the data with 95\\% noise, the modalities are uncorrelated and $\\rho=0$ is expected to have relatively high generative quality. We obtain the following average FID scores ([link1](https://anonymous.4open.science/r/codevae_icml-27EA/), [backup_link](https://anonymfile.com/k0W28/uncorrelated-experts.pdf)) :\n\n|               | 0\\%       | 25\\%      | 95\\%      |\n|---------------|-----------|-----------|-----------|\n| $\\rho=0$      | 29.0      | 31.27     | 48.22     |\n| $\\rho=0.9$    | 26.12     | 29.90     | 53.68     |\n\nshowing that CoDE-VAE correctly captures the dependency between experts distributions through the $\\rho$ parameter.\n\n## CoDE-VAE in cases with missing data or when modalities are not available.\nThe evaluation setup in our research follows the standard method in multimodal VAEs where all possible combinations of missing modalities are evaluated at test time (Appendix B). To handle missing data is not trivial, as we need to estimate consensus distributions. To estimate $q(z|x_1,x_2)$, any aggregation method would require the same number of observations $x_1$ and $x_2$. This problem could be overcome by using only complete pairs $(x_1,x_2)$, and re-weighting the ELBO terms with fewer samples and could be an interesting direction to pursue in future work.\n\n## CoDE-VAE when one modality is much more informative?\nCoDE-VAE learns the contribution of each k-th ELBO term to the optimization, balancing the importance of relatively more informative modalities. The empirical results of Section 4.4 show that the text modality in the MNIST-SVHN-Text is relatively more important to the optimization of the ELBO as shown by the weight learned for the subset containing that modality. This result seems reasonable, as there is more noise in the MNIST and SVHN modalities. The ablation experiments of Section 4.5 show that CoDE-VAE achieves higher performance (coherence and FID), when the contribution of each subsets to the optimization, and so each modality, is learned. \n\n## CoDE where the assumption of expert dependence does not hold?\nCoDE is a flexible approach that does not impose any restriction on the way $\\Sigma_d$ is specified as long as it is invertible. For independent modalities, it should be enough to use $\\rho=0$ (see answer on edge cases). \n\n## CoDE-VAE generative quality. Could you provide more concrete examples?\nWe recognize that the wording of this claim could be improved and made it more concrete. We have replaced the original sentence with *\"CoDE-VAE minimizes the generative quality gap as the number of modalities increases, achieving quality similar to unimodal VAEs measured by unconditional FID scores.\"*, which corresponds to the experiments in Section 4.2 Generative quality gap. Furthermore, Figure 3 shows that CoDE-VAE achieves higher generative performance as the number of modalities used for model training increases, something most of the benchmark models are not able to achieve. We have [added figures](https://anonymous.4open.science/r/codevae_icml-27EA/) ([backup](https://anonymfile.com/rN124/polymnist-gen.pdf)) that show the generated modality $m_0$  as a function of input modalities, as well as generated samples by the unimodal VAE for qualitative comparisons. \n\n## Computational complexities of CoDE-VAE\nWe agree that this is an important aspect to be considered. Therefore, in Appendix C we mention that CoDE-VAE has a relatively high computational cost $\\mathcal{O}(2^M-1)$, where $M$ is the number of modalities. However, given that the size of the matrix $\\Sigma_d$ depends only on $M$ (see the question from reviewer FV9z), model training is feasible on a single GPU even for 5-modality datasets.", "claims_and_evidence": "The paper provides clear empirical evidence to support the claims about the CoDE-VAE’s superior performance.  The experimental results on datasets such as MNIST-SVHN-Text, PolyMNIST, and CUB support the assertion that CoDE-VAE balances generative coherence and quality better than existing models.  Additionally, the paper argues that CoDE’s consideration of expert dependence leads to better log-likelihood estimations and reduced generative quality gaps compared to models relying on modality sub-sampling.  However, the discussion could benefit from more in-depth comparisons in specific edge cases where other models might outperform CoDE-VAE.", "methods_and_evaluation_criteria": "The methodology behind CoDE is sound, introducing a principled Bayesian approach to account for expert dependence.  The CoDE-VAE model builds on existing multimodal VAEs, addressing key challenges like missing modalities and the imbalance in the contribution of different ELBO terms.  The evaluation criteria, including generative coherence, log-likelihood estimation, and classification accuracy, are appropriate for comparing multimodal models.  However, the explanation of how the model behaves in extreme cases (e.g., when only one modality is available) is not fully addressed.", "theoretical_claims": "The theoretical claims regarding the new ELBO formulation and the aggregation method using CoDE are well-supported by the paper’s derivations and lemmas.  The paper provides a solid mathematical foundation for the method, with proofs of key results, such as the posterior distribution and consensus distributions.  There are no apparent issues with the correctness of these proofs.", "experimental_designs_or_analyses": "The experimental design is robust, with comprehensive comparisons to multiple baseline models.  The use of multiple datasets (MNIST-SVHN-Text, PolyMNIST, and CUB) provides a good cross-section of real-world multimodal problems.  However, more detailed ablation studies or analyses of edge cases where the assumptions about expert dependence may not hold could further strengthen the paper.", "relation_to_broader_scientific_literature": "The paper does a good job of relating its contributions to the broader literature on multimodal VAEs and expert aggregation methods.  The work is clearly motivated by existing challenges in multimodal learning, such as missing modalities and independent expert assumptions.  The comparison with PoE and MoE methods, along with references to key multimodal VAE papers, establishes the novelty of the approach.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The CoDE-VAE method is a novel and theoretically sound approach that addresses the challenge of dependent expert distributions.\n2. The experimental results are convincing, showing that CoDE-VAE outperforms existing methods in key areas like generative coherence and log-likelihood estimation.\n\nWeaknesses:\n1. Some parts of the experimental setup could be explained more clearly, particularly regarding the optimization process for learning the contribution of each ELBO term.\n2. More detailed comparisons with edge cases or failures of the model would help to solidify the generalizability of the results.", "other_comments_or_suggestions": "1. Consider adding more analysis on how CoDE-VAE behaves in cases with missing data or when some modalities are not available.\n2. A clearer distinction between the CoDE-VAE approach and similar models would benefit readers unfamiliar with multimodal VAEs.", "questions_for_authors": "1. How does the performance of CoDE-VAE change when there is a significant imbalance between the available modalities (e.g., when one modality is much more informative than the others)?\n2. Could you elaborate on how the CoDE method handles scenarios where the assumption of expert dependence does not hold (e.g., in highly independent modalities)?\n3. The paper mentions that CoDE-VAE reaches generative quality similar to unimodal VAEs in certain cases—could you provide more concrete examples of these cases, and how the model behaves with increasing modality count?\n4. What are the computational complexities of CoDE-VAE compared to existing models like PoE and MoE, particularly as the number of modalities increases?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "khEdrKaGKk", "reviewer": "Reviewer_kHs1", "summary": "The authors propose to challenge the assumption of independence between unimodal experts in computing the joint posterior in multimodal VAEs. Therefore they propose the CoDE-VAE that uses a Bayesian approach to compute the joint posterior between unimodal experts, modelling the dependence between them. Experimental results show that their idea is effective and are positive, despite not significantly outperforming the most recent alternative approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful and detailed comments. We will respond to your concerns and questions point by point.\n\n## Confusion about the concept of sub-sampling modalities\nThank you for pointing this out. In the paper, we use the concept of sub-sampling to refer to ELBO sub-sampling and to the use of mixture distributions to approximate consensus distributions. We will make this clear in the revised version.  \n\n## Alternative Datasets\nBased on your and Reviewer FV9Z comment, we provide results on the CELEB-A data. Due to the limited time, we only considered a latent space with 32D. For CoDE-VAE we assume $\\rho=0.6$, as we observed to be a value that performs well in other experiments. We obtained the following results\n\n|                           | CoDE-VAE  | MMVAE+    |\n|---------------------------|-----------|-----------|\n| Conditional FID           | 92.11 (0.61)      |  97.30 (0.40)    |\n| Unconditional FID         | 87.41 (0.36)      |  96.91 (0.42)    |\n| Conditional Coherence     | 0.38 (0.001)      |  0.46 (0.001)    |\n| Unconditional Coherence   | 0.23 (0.003)      |  0.31 (0.030)    |\n| Classification            | 0.38 (0.066)      |  0.37 (0.003)    |\n\nwhich may improve further by cross-validating $\\rho$ in CoDE-VAE.\n\n## Comparison to Deep Generative Clustering with Multimodal Diffusion Variational Autoencoders\nThank you for pointing out this paper, which we were not aware of. We acknowledge the importance of the paper in the field of multimodal VAEs and will discuss it in the related work section and will include the experimental comparison on PolyMNIST in the appendix of the revised version of the paper, as the Clustering Multimodal VAE (CMVAE) model introduced in [2] is not 100\\% comparable with our proposed CoDE-VAE model. The main focus of our research is to present a novel approach to estimate consensus distributions and to learn the contribution of each ELBO subset, while the goal in CMVAE is to couple multimodal VAEs with clustering tasks by leveraging clustering structures in the latent space and to introduce diffusion decoders, which is certainly a novel and relevant line of research. CMVAE captures clustering structures using a mixture model as a prior, and we hypothesized that this flexible prior in CMVAE plays an important role in the performance of unconditional generative tasks. We will include this discussion and the relation between the methods in the updated manuscript.\n\n## Notation and Clarity\nThank you for pointing this out. We will change the notation in the revised version to ensure consistency in the use of subscripts and superscripts, which will improve the clarity of Section 3.\n\n## Thorough comparison on the generative quality gap for PolyMNIST\nWe agree that an average generative quality gap would provide a robust comparison. However, the computational costs of such experiment is significant, as to assess the generative quality for each of the 5 modalities as a function of the number of input modalities requires to train at least 12*3=36 times each model (considering 3 different runs to report standard deviations). This will require 252 runs in total, as there are 7 different models in the evaluation of the quality gap (without considering the unimodal VAEs). Given that our research includes 3 data sets, 6 benchmark models, and several ablation experiments, we let such a robust comparison for future research.\n\n## Add grid-search hyperparameters\nThank you for pointing this out. We will add to the Appendix the hyperparameters for the grid search and their optimal values, including the ones for the new experiments on the CELEB-A data.\n\n## Is the dependency between expert errors assumed to be constant across the D latent dimensions? \nYes, for simplicity, we assume a common $\\rho$ parameter for all dimensions. However, the CoDE is a flexible approach that does not impose any restriction on the way $\\Sigma_d$ is specified. We let future research explore whether model performance can be improved by using different correlation values for different dimension. \n\n## Beta values and latent space for MMVAE+ on the MNIST-SVHN-Text data.\nFor all models in Section 4.1, we cross-validate $\\beta$ using the grid $[0.1,1,5,10,15,20]$. All models, but MMVAE+, assume that the latent space has 20 dimensions as in previous research. To select the dimensionalty of common and modality specific (MS) variables in MMVAE+, we follow a similar approach as in the original paper, where the authors choose the dimensions of these two to be equal to the dimension of the latent space in MMVAE (a model without MS variables) divided by the number of modalities. Therefore, MMVAE+ assumes that both common and MS variables have 7 dimensions. These details are explained in Appendix D3. We also tested to use 10 dimensions in both common and MS variables, so the decoders in the MMVAE+ model would generate modalities based on 20 dimensions, just as the other models. We did not observe significant differences.", "claims_and_evidence": "- Empirical results are positive and show the effectiveness of the approach, despite not outperforming SOTA in the multimodal VAE literature. \n- I think there seems to be some confusion in the paper about the concept of subsampling modalities in the ELBO, related to the limitations highlighted in [1]. The authors state \"It is noteworthy that CODE-VAE does not rely on sub-sampling techniques, which have been shown to harm the performance of multimodal VAEs\", but in the CoDE-VAE ELBO in eqn 3 subsampling actually happens. To see it, it is sufficient to notice that computing a given term of the sum in the ELBO requires reconstruction of all modalities given only a subset used for inference, hence sub-sampling of modalities happens and the CoDE-VAE is also subject to the theoretical limitations outlined in [1], as also confirmed in the experimental results. \n\n[1] Daunhawer et al On the limitations of multimodal VAEs, ICLR, 2022.", "methods_and_evaluation_criteria": "As outlined also below the chosen datasets to benchmark the approach are sensible, while already well-studied in the multimodal VAE literature. As for the proposed method, challenging the assumption of independence between unimodal experts in approximating joint posterior inference  for multimodal VAEs is a valuable research direction. Moreover, the proposed method appears to be effective.", "theoretical_claims": "The authors justify their approach as a Bayesian method to approximate the joint posterior assuming a dependence between unimodal experts. The derivations to be seem correct and back up their theoretical claims.", "experimental_designs_or_analyses": "The datasets chosen for the experiments are fairly standard in the multimodal VAE literature and existing models already achieve convincing results in these setups. While the comparisons on these datasets are valid, I think the authors could have picked a novel more challenging dataset to outline the benefit of their proposed model. I think experiments on the chosen datasets are properly conducted, and the results are properly commented. While model performance does not surpass certain recent approaches (e.g. MMVAE+), the results are still somehow positive and show that the suggested idea works. I'd strongly suggest the authors to compare with a recent paper [2] that proves to outperform alternative multimodal VAEs. While the proposed model has the option to be equipped with diffusion decoders, which would make an unfair comparison in terms of generative quality with CoDE-VAE, the authors show that the proposed ELBO without diffusion decoders still improves over alternative multimodal VAEs. Hence it seems relevant to include it in the comparisons in this paper. \n\n[2] Palumbo et al. Deep Generative Clustering with Multimodal Diffusion Variational Autoencoders, ICLR, 2024.", "relation_to_broader_scientific_literature": "The idea discussed in this paper fits nicely in the literature of multimodal VAEs as it explores the direction of modelling dependence between unimodal experts in computing the joint posterior, which was not explored thus far to my knowledge.", "essential_references_not_discussed": "Recent relevant work is not discussed. Specifically, [2]. As I mentioned above I strongly suggest the authors to at least discuss this paper in the related work, and advise also to include it in the experimental comparisons. \n\n[2] Palumbo et al. Deep Generative Clustering with Multimodal Diffusion Variational Autoencoders, ICLR, 2024.", "other_strengths_and_weaknesses": "Weaknesses:\n- Notation is sometimes confusing. E.g in section 3 the dimension $d$ appears at times as superscript and at times as subscript, even in the same equation ( $e^d_j = \\mu^d_j - \\theta_d^k$). \n- Certain experimental comparisons could be more thorough. For instance, on PolyMNIST it would be more appropriate to assess the generative quality gap on all modalities (and possibly do an average in performance), instead of only focusing on generating modality $m0$.", "other_comments_or_suggestions": "- I think clarity in the paper could be improved in section 3. \n- I suggest that, when grid-searching hyperparamters, the authors make explicit (e.g. in the MNIST-SVHN-Text experiment) which hyperparameters achieve best performance for each model, and hence are used to report the results.  These info can also be left to the Appendix.", "questions_for_authors": "- Is the dependency between expert errors assumed to be constant across the D latent dimensions? Why do the authors make this assumption?\n- Which beta values and latent space dimensions are chosen to get the results for the MMVAE+ model on the MNIST-SVHN-Text dataset, reported in the main text? It is not really clear to me, even after having a look at the Appendix. It seems somehow strange that the model achieves a relatively low performance in this dataset, keeping in mind  the results on PolyMNIST and CUB datasets.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work  introduces CoDE-VAE, a model that challenges the common assumption of independence between unimodal experts in multimodal Variational Autoencoders by modeling their dependencies. Using a Bayesian approach, CoDE-VAE aims to improve joint posterior estimation and enhance generative coherence and classification accuracy. Experimental results show that while CoDE-VAE performs well, its improvements over the latest alternatives are modest.

While this work has some limitations, like including more thorough experiments, improving clarity of the paper overall, and more detailed comparisons with edge cases or failures of the model would help to solidify the generalizability of the results, the strength outweigh the limitations. The authors propose a method that is novel and theoretically sound, and it addresses the challenge of dependent expert distributions.  I think this work is of interest to the ML community
Decision: Accept (poster)