ICML 2025

Aggregation of Dependent Expert Distributions
in Multimodal Variational Autoencoders
Rogelio A Mancisidor 1 Robert Jenssen 2 3 4 Shujian Yu 2 5 Michael Kampffmeyer 2 4
Abstract
Multimodal learning with variational autoen-
coders (VAEs) requires estimating joint distri-
butions to evaluate the evidence lower bound
(ELBO). Current methods, the product and mix-
ture of experts, aggregate single-modality dis-
tributions assuming independence for simplicity,
which is an overoptimistic assumption. This re-
search introduces a novel methodology for ag-
gregating single-modality distributions by ex-
ploiting the principle of consensus of dependent
experts (CoDE), which circumvents the afore-
mentioned assumption.
Utilizing the CoDE
method, we propose a novel ELBO that approx-
imates the joint likelihood of the multimodal
data by learning the contribution of each sub-
set of modalities.
The resulting CoDE-VAE
model demonstrates better performance in terms
of balancing the trade-off between generative
coherence and generative quality, as well as
generating more precise log-likelihood estima-
tions.
CoDE-VAE further minimizes the gen-
erative quality gap as the number of modali-
ties increases.
In certain cases, it reaches a
generative quality similar to that of unimodal
VAEs, which is a desirable property that is
lacking in most current methods.
Finally, the
classiﬁcation accuracy achieved by CoDE-VAE
is comparable to that of state-of-the-art multi-
modal VAE models. CoDE-VAE is available at:
https://github.com/rogelioamancisidor/codevae.
1Department of Data Science, BI Norwegian Business School,
Oslo, Norway 2Department of Physics and Technology, UiT The
Arctic University of Norway, Tromsø, Norway 3Department of
Computer Science, University of Copenhagen, Copenhagen, Den-
mark 4Dept. BAMJO, Norwegian Computing Center, Oslo, Nor-
way 5Department of Computer Science, Vrije Universiteit Ams-
terdam, Amsterdam, Netherlands. Correspondence to: Rogelio A
Mancisidor <rogelio.a.manc...