Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives
Authors: Marcel Hirt
Domenico Campolo
Victoria Leong
Juan-Pablo Ortega
Venue: Accepted by TMLR ()

Abstract: Devising deep latent variable models for multi-modal data has been a long-standing theme in machine learning research. Multi-modal Variational Autoencoders (VAEs) have been a popular generative model class that learns latent representations that jointly explain multiple modalities. Various objective functions for such models have been suggested, often motivated as lower bounds on the multi-modal data log-likelihood or from information-theoretic considerations. To encode latent variables from different modality subsets, Product-of-Experts (PoE) or Mixture-of-Experts (MoE) aggregation schemes have been routinely used and shown to yield different trade-offs, for instance, regarding their generative quality or consistency across multiple modalities. In this work, we consider a variational objective that can tightly approximate the data log-likelihood. We develop more flexible aggregation schemes that avoid the inductive biases in PoE or MoE approaches by combining encoded features from different modalities based on permutation-invariant neural networks. Our numerical experiments illustrate trade-offs for multi-modal variational objectives and various aggregation schemes. We show that our variational objective and more flexible aggregation models can become beneficial when one wants to approximate the true joint distribution over observed modalities and latent variables in identifiable models.

Published in Transactions on Machine Learning Research (09/2024)
Learning multi-modal generative models with permutation invariant encoders and tighter variational objectives
Marcel Hirt
marellnndr..hrtt@ntu..ccm
School of Social Sciences
Nangang Techmological Universiby
Singapore
Domenico Campolo
d. compolo@ntu.edu..o.
School of Mechanical and Aerospace Engineering Nangang Technological Universily
Singapore
Victoria Leong
VictoriaLeong Ontu. edu.sg
School of Social Sciences
Nanyang Technological Universily
Singapore
Juan-Pablo Ortega
juan-pablo. honn-pabbl..rteeau@tt.... edu..g
School of Physical and Mathematical Sciences
Nanyong Technological University
Singapore
Reviewed on OpenReview: https: openreview. net/ forum? id= LMInHnaCfI
Abstract
Devising deep latent variable models for multi-modal data has been a long-standing theme in machine lcarning research. Multi-modal Variational Autoencoders (VAEs) have been a pop- ular generative model class that learns latent representations that. jointly explain multiple modalities. Various objective functions for such models have been suggested, often moti- vated lower bounds on the multi-modal data log-likelihood from informattion-theorelie considerations. To encode latent variables from different modality subsets, Product-of- Experts (PoF) or Mixture-of- (MoE) aggregation schemes have been routinely used and shown to yield different trade-offs, for instance, regarding their generative quality or consistency across multiple modalities. In this work, we consider a variational objective that can tightly approximate the data log- likelhoo.. We develop more flexible aggregation schemes that avoid the inductive biases in PoE or MoE approaches by combining encoded features from different modalities based on permuuattion-invaiiaa neural networks. Our nu- merical experiments illustrate trade-offs for multi-modal variational objectives and various aggregation schemes. We show that our variational objective and more flexible aggregation models can bocome beneficial when one wants to approximate the true joint distribution over observed modalities and latent variables in identifiable models.
1 Introduction
Multi-modal data sets where each sample has features from distinct sources have grown in recent years. For example, multi-omics data such genomics, epigenomics, transcriptomics, and metabolomics provide

---

Published in Transactions on Machine Learning Research (09/2024)
Audio
Image
Text
Image
7/
Toxt
Audic
Image
Text
Audio
Image
Xy
Xis
X
(a)
(b)
(c)
(d)
Figure 1: Reconstruction or cross- cross-prediction of modalities in la and 1b for a mixture-based bound and our objective, respectively. The mixture-basse bound resorts single latent variable Z ~ (s(-lzg) that encodes information from modality subset ss and is trained to reconstruct the conditioning modalities as; well as to predict the masked modalities sis- Our objective relies two latent variables Zs go(-|as) and ZM- go(- where Zs is learned to reconstruct all its conditioning modalities, with Z learned to reconstruct the remaining modalities. KL regularization terms in Ic and [d] for a mixture-basee bound and our objective, respoctively. The mixture-based bound aims to minimize the KL divergence between the encoding distribution given modality subset is and prior distribution. Our objective additionally aims lo minimize the KL divergence between the encoding distribution given all modalities relative the encoding distribution of modality subset Eg--
more comprehensive understanding of biological systems if multiple modalities are analyzed in integrative framework (Argelaguct et al. 2018 Lee and van der Schaar 2021 Minoura et al. 2021 In neuroscience, multi-modal integration of neural aclivity and behavioral data can help to learn latent neural dynamics [hhou and Wei 2020 Schneider et al. 2023). However, annotations or labels in such data sets are often rare, making unsupervised or semi-supervised generative approaches particularly attractive as such methods can be used in these settings to (i) generate data, such as missing modalities, and (ii) learn latent representations that useful for down-stream analyzes that of scientifie interest. themselves. The availability of heterogeneous data for different modalities promises to learn generalizable representations that can capture shared content across multiple modalities in addition to modality-specifie information. A promising class of weakly-superrised generative models is modal VAEs (Suzuki et al. 2016 Wu and Goodman 2019 Shi et al.] 2019; Sutter et al., 2021) that combine information across modalities in an often-shared low- dimensional latent representation. A common route for learning the parameters latent variable models is via maximization of the marginal data likelihood with various lower bounds thereof, as suggested in previous work.
Setup. We consider a set of M random variables {X X with empirical density Pa, where each random variable X,, M = (1....,.M M), can be used to model different data modality taking values in X. With some abuse of notation, X (Xs, x) for two partitions of the random variables into Xs = (Xs)ses and X = (Xx =Xx]eeMM We pursue a latent variable model setup, analogous to uni-modal VAËs (Kingma and Ba 2014 Rezende
write X = [XI...., XM) and for any subset S C CM, we set
et al. 2014). For a latent variable po(2, = po(2) I (o(x,12) where po(x,1z) is commonly referred to as the decoding distribution for modality 8. Observe that all modalities are independent given the latent variable shared across all modal- ities. However, one can introduce modality-specifie latent variables by making sparsity assumptions for the decoding distribution. Intuitively, this conditional independence assumption means that the latent variable Z captures all unobserved factors shared by the modalitics.
Z with prior density (z)),, we posit a joint generative mode|
We usually denote random variables using upper-case letters, and their realizations by the corresponding lower-case letter. We assume throughout that = RD, and that pa(z) is Lehesgue density, although the results can he extended to more general settings such discrete random variables Z with appropriate adjustments, instance, regarding the gradient istttors..
2

---

Published in Transactions on Machine Learning Research (09/2024)
Multi-modal variational bounds and mutual information. Popular approaches to train multi-modal
models are based on mixture-based variational bound (Daunhawer et al. 2022 Shi et al. 2019) given by CMix (0,xx0,,c (0,0, 3) = p( (I, 0,0. 3)dS, where
C (z, 0. 0. = (x(x)))) polgoo(z/2)) dz (KL(
(1)
and p is some distribution on the power set P(M) of M and 3 0. For 3 1, one obtains the bound C 0, o, 3) log)o(z). However, shown in Daunhawer et al. (2022) there is gap between the
variational bound and the log-likelihood given by the conditional entropies that cannot be reduced even for flexible encoding distributions. More precisely, it holds that
f (og)))ogoo(()dde > (allt) 0,,,,, 1)dx + I s(s))
where M(pa(XxslX3) is the entropy of the conditional data distributions. Intuitively, in Q one tries to reconstruct or predict all modalities from incomplete information using only the modalities S which leads to learning an inexact, average prediction (Daunhawer et al. 2022p. In particular, it cannot reliably predict mppelity- information that is not shared with other modality subsets, as measured by the conditional entropies ss
For an illustration, see Figure Ia. where the latent variable Z encodes information from a text and audio modality and is tasked to both reconstruct the text and audio modalities and to predict unobserved image. Information that is specific to the image modality only thus cannot be recovered. A related observation has been made for Masked AutoEncoders (He et al. 2022) that correspond to the limiting case 3 + 0, where the crossteeconttruction objective leads to learning features that are invariant to the masking of modalitics (Kong and Zhang, 2023) and allows for the recovery of latent variables that represent maximally shared information between the unmasked and masked modality (Kong et al. 2023
We will illustrate that maximizing Cilix can be interpreted as the information-theorett objective of
maximizing { {ilh (X, Zs) sigh go (Xs, K
(2)
where juh and fhb are variational upper, respectively, lower bounds of the corresponding mutual information 1,X, Y) q(z, y) log g(a.)) g(z)(()) ( dody of random variables X and Y having marginal and joint densities q- In order to emphasize that the latent variable Z is conditional on Xs under the encoding density 9a. we write Zs instead of Z. Variations of ( have been suggested (Sutter et al. 2020), such as by replacing the prior density pe in the KI-term by a weighted product of the prior densily pe and the uni-modal encoding distributions go(2|x,) for all s E M. Likewise, the multi-view variational information bottleneck approach developed in Lee and van der Schaar (2021) for predicting Xis given Xs can be interpreted as maximizing illb (Xis, Zs) - (Xx, 7s).
Hwang et al. (2021) suggested a related bound that aims to maximize the reduction of total correlation of X when conditioned on a latent variable. Similar bounds have been suggested in Sutter et al. (2020) and Suzuki et al. (0166) by considering different KL-regularisation terms; see also Suzuki and Matsuo (2022). Shi et al. (2020) add a contrastive estimate Ino of the point- wise mutual information to the maximum likelihood objective and minimize - -logPo(2) - -i (#s, s,s)). Optimizing variational bounds of different mutual information terms such as (2 yield latent representations that have different trade-offs in terms of either (i) reconstruction or (ii) cross- cross-prediction of modal data from a rate-distortion viewpoint (Alemi et al. 2018).
Multi-modal aggregation schemes. To optimize the variational bounds above or to allow for flexible conditioning at test time, we need to learn encoding distributions (=(=))) for any S E P(M). The typical aggregation schemes that are scalable to large number of modalities are based on a choice of uni-modal encoding distributions go. (z|r.) for any 8 E M, which are then used to define the multi-modal encoding distributions as follows:
Mixture of Experts (MoE), see Shi el al.
2019
g
MoE
(z (2|05)
To. (2(xgg)
S
3

---

Published in Transactions on Machine Learning Research (09/2024)
Text
Audio
Image
Text
Audio
Image
Text
Audio
Image
Text
Audio
Image
Bitcy
Dnty
BYC,
Bnce
Bnd
Bmdg
My.
Ha. MA
Pi,
Rry Of
Mai MA
Po.
Rp
by
Ny
hy
8
POë Aggregation
Nott oft / of) NNN,
#
8h
gi
97
N(v.a))
N(A.0*)
N(L
(a) PoE
(b) MoE
(c) Sum-Pooling
(d) Atffntttnntoon
Figure 2: Illsstaation multi-modal aggregation schemes. All encoding schemes first apply modality-peecffi encoders each individual modality. A POE model (2a) aggregates the outputs from the modality-specific encoders into single Gaussian distribution that results from multiplication of the corresponding uni- modal Gaussian densities. An MoE model (bb assumes
equally weighted Gaussian mixture distribution comprised of the uni-modal Gaussian densities. Our new aggregation schemes allow for learning permutation- invariant fusion models: A Sum-Pooling or Deep Set model (2c) applies the same function to the encoded features has s E {T, before summing them up and using a non-linear projection p to the parameters of Gaussian distribution. A Self-Attention model (2d) differs from the Sum-Pooling approach by applying self-attention layers or transformer layers before summing up the features, thereby accounting for pairwise interactions between the encoded modalities. Our newly introduced schemes allow for encoding only modality subset by using standard masking operations.
Product of Experts (PoE), see Wu and Goodman 2018
46 (2|0s) x po(=) II #ES
PoB
go. (z|xx).
illustrates these previously considered aggregation schemes. While these schemes do not require learning the aggregation function, introduce aggregation schemes that, as illustrated in 2c 2d involve learning additional neural network parameters that specify learnable aggregation function.
Figure 2a 2b
Contributions. This paper contributes (i) new variational objective as an approximation of lower bound on the multi-modal log-likelihood (LLH). We avoid a limitation of mixture-based bounds [ which may not provide tight lower bounds on the joint LLH there considerable (Daunhawer et al.] 2022), even for flexible encoding distributions. The novel variational objective contains lower bound of the marginal LLH log po (æs) and a term approximating the conditional log po ((s)) for any choice of S P(M), provided that we learn flexible multi-modal encoding distribution. This paper then contributes (ii) new multi-modal aggregation schemes that yield more expressive multi-modal encoding distributions when compared to MoEs or PoEs. These schemes are motivated by the flexibility of (PI) architectures such as DeepSets (Zaheer et al. 2017) or attention models (Vaswani ct al. 2017 Lee et al. (2019). We illustrate that these innovations (iii) are beneficial when learning identifiable models, aided by using llexible prior and encoding distributions consisting of mixtures, and (iv) yield higher LLH in experiments.
moddlltyysspeciff variation
coumutation-nvvaiian
Further related work. Canonical Correlation Analysis (Hotelling 1936 Bach and Jordan 2005 a clas-
sical approach for multi-modal data that aims to find projections of two modalities by maximally correlating
4

---

Published in Transactions on Machine Learning Research (09/2024)
and has been extended to include more than two modalities Archambeau and Bach
2008 Tenenhaus and
Tenenhaus 2011) to allow for non-linear transformations (Akaho, 2001 Hardoon et al. 2004 Wang et al. 2015 Karami and Schuurmans 2021). Probabilistic CCA can also be seen as multi-battery factor analysis (MBFA) [Browne 1980 Klami et al. 2013). wherein a shared latent variable models the variation common to all modalities with modality-specific latent variables capturing the remaining variation. Likewise, latent fac-
for regression or classification models (Stock and Watson 2002) assume that observed features and response are driven jointly by latent variable. |Vedantam et al. (2018) considered tiple-ELBO for two modalities, while (Sutter et al. (2021) introduced generalized variational bound that involves summation over all modality subsets. A series of work has developed multi-modal VAFs based shared and private latent
variables (Wang et al. 2016 Lee and Pavlovic 2021; Lyu and Fu 2022 Lyu et al. 2021 Vasco et al. 2022;
objective and mini- mized an approximation of the Wasserstein distance between the generated and observed multi-modal data. Joy et al. (2021) consider a semi-supervised sctup of two modalities that requires no explicit multi-modal aggregation function. Extending the Info-Max principle (Linsker) 1988). maximizing mutual information (9 (X))) S I,((X,, X2), (Z1. Z2)) based representations Z, (xx)) for modality-specinn en- coders 9s from two modalities has been motivation for approaches based on (symmetrized) contrastive objectives (Tian et al. 2020; Zhang et al. 2022c Daunhawer et al. 2023) such as InfoNCE (Oord et al. 2018 Poole et al. 2019 Wang and Isola, 2020 as a variat ional lower bound on the mutual information
Palumbo et. al. 2023 ). Tsai el. al. (2019a) proposed a hybrid generalive-discriinnall
between Z1 and Z2. Recent work (Bounoua et al. 2023
Bao et al. 2023 considered score- bosee diffusion
models on auto-encoded private latent variables.
2 A tighter variational objective with arbitrary modality masking
For C.M and 0, we define
Cs(es.0.0 0,0,)) - lo (00)=/(002// (2((5))) [log (o(ss (KL(g ( Po())).
This simply a standard variational lower bound (Jordan et al. 1999 Blei et al. 2017 restricted to the
subset. S for = 1, and therefore (s(ss, 0, Φb, 1) S log (xs)) One can express the variational bound information-theorred (Alemi al., 2018 terms
J ((lss) = DDs -RRs
for the rate
Rs = f pu(
measuring the information content that is encoded from the observed modalities in S by qs into the latent reprosentation, and the distortion
Ds s==
pal pa(as))oos)(ss) log po (1ss)2))dddd
given as the negative reconstruction log-likelihood of the modalities in S. While the latent variable Zs that is encoded via. (){2)=)) from Xs can be tuned via the choice of B 0 to tradeoff compression and reconstruction of all modalities in S jointly, it does not explicitly optimize for cross-modal prediction of modalities not in S. Indeed, the mixture- based variational bound differs from the above decomposition exactly by an additional cross-modal prediction or cross-distortion term
Dis -
pa(x)) P log (od.
thereby explicitly optimizing for both recnsssccottinii of modality subset and cross-modal prediction within a single objective:
J ( -Ds DCs -3Rs.
(4)
5

---

Published in Transactions on Machine Learning Research (09/2024)
Instead of adding an explicit cross-modal prediction term, consider an additional variational objective with a second latent variable ZM that encodes all observed modalities = and tries to reconstruct the remaining modality subset. Unlike the latent variable Zs in (T) and (3) that can only encode incomplete information using the modalities S, this second latent variable ZM can encode modality-specifie information from all observed modalities, thereby avoiding the averaging prediction in the mixture-baser bound, see
Figure 1b
Ideally, we may like to consider an additional variational objective that lower bounds the conditional log- likelihood log ( so that maximizing the sum of both bounds maximizes lower bound of the multi-modal log-likel log-likelihood log = log po(xs) + log pa sxs) To motivate such conditional variational objective, note that maximizing the variational bound (3 with infinite capacity encoders yields (o(2|zs) = po(z|zs). This suggests replacing the intractable posterior p=(2)xs) with the encoder (z(2( for the probabilistic model when conditioned on rg. A variational objective under this replacement then becomes
(1s(2,0,003) = (0(2/x)) [logoo dz (KL(@o(2)
(5)
However, the above only approximates lower bound on the conditional log-likelihood log ( skss, and Lis is a lower bound only under idealized conditions. We will make these approximations more precise in Section 2.1 where we illustrate how those bounds yield to a matching of different distributions in the latent or data space, while Section 2.2 provides an information-theorttii interpretation of these variational objectives. The introduction ion of a second latent variable leads to two KT. regularization terms, see Figure ld. that do not satisfy a triangle inequality
(=))))))()))(())))) + (L(((())=<))lp((() 전 (L((s<)\=)loo(2))
In summary, for some fixed density P on P(M), we suggest to maximize the overall bound
C(x,0,0,00 = / ((S) [Ls(as, 0, 0, + Lis( 0, 0,
with respect to 0 and o, which is generalization of the bound suggested in Wu and Goodman 2019 to an arbitrary number of modalities. This bound can be optimized using standard Monte Carlo techniques, for example, by computing unbiased pathwise gradients (Kingma and Ba 2014 Rezende et al. 2014 Titsias and Gredill 2014) using the reparameterization trick. For variational familics such as Gaussian mixtures P one can employ implicit reparameteriization (Figurnov et al. 2018). is straightforward to adapt variance reduction techniques such as ignoring the scoring term of the multi-modal encoding densities for pathwise gradients (Roeder et al., 2017), see Algorithm in Appendix H for pscudo-code. Nevertheless, scalable approach requires an encoding technique that allows to condition on any masked modalities with computational complexity that does not increase exponentially in M. We will analyse scalable model architectures in Section 3
2.1 Multi-modal distribution matching
Likelihood-based lcarning approaches aim to match the model distribution po(z) to the true data distribution pa(x). Variational approaches achieve this by matching in the latent space the encoding distribution to the true posterior as well as maximizing a tight lower bound on logro(#)), see Rosca et al. (2018). These types of analyses have proved useful for uni-modal VAEs as they can provide some insights as to why VAEs may lead to worse generative sample quality compared to other generative models such as GANs (Goodfellow et al., 2014) or may fail to learn useful latent representations (Zhao et al. 2019 Dieng al. 2019). We show similar results for the multi-modal variational objectives. This suggests that limitations from uni-modal VAËs also affect multi-modal VAEs, but also that previous allempts to address these shortcomings in uni-modal VAEs may benefit multi-modal VAEs. In particular, mismatches between the prior and the aggregated prior for uni-modal VAEs that result in poor unconditional generation have a natural counterpart for cross-modal generations with multi-modal VAEs that may potentially be reduced using more flexible conditional prior distributions, see Remark 3 or via adding additional mutual information regularising terms (Zhao et al.
FFor MoE aggregation schemes. Shi al. 2019 considered stratified ELBO estimator as well tighter bound based importance sampling, see also Morningstar et. al. 2021 ), that do not pursue here for consistency with other aggregation schemes that can likewise be optimized based importance sampling ideas.
6

---

Published in Transactions on Machine Learning Research (09/2024)
2019
Remark Given these results. it is neither surprising that multi-modal diffusion models such
88 Bounoua et al. (0023) Bao et al. (2023) yield improved sample quality, nor that sample quality can be
im improved by augmenting multi-modal VA with diffusion models (Pandey et al. 2022 Palumbo et al.
2024
We consider the densities
po(2, c
and
(= x) = p
The latter is the encoding path comprising the encoding density conditioned on and the empirical density Pa- We write
(
for the aggregated prior (Makhzani et al. restricted on modalities from and (*
2016
Hoffman and Johnson, 2016 Tomezak and Welling 2017]
Is
(TS () 2)/ g (z) and likewise consider its conditional version,
/
q (=(ass
for aggregated encoder conditioned on æs. We provide multi-modal ELBO surgery, summarized in Proposition I below. It implies that maximizing f (a(=s) 0, ø)drs drives
the submodalitics to the joint the joint inference distribution q0(2, vs) = pa( generative distribution po(2,x5) = po (1)) 2) and
the generative marginal pe(xs) its empirical counterpart pa(æs).
Analogously, maximizing J p xs) Cis (2, 8, d) drives, for fixed rs.
1. the distribution pa(a,s| |zs (oprr)) to the distribution no(m)ss=)) and
2. the conditional pa so( to its empirical counterpart pa(z)s(as).
Furthermore, it shows that maximizing (is(z, 0 Φ) minimizes cossstcon matching term (1 dgg \s (=) for the multi-modal encoders where a mismatch can yield poor gens-generation, as an analog of the prior not matching the aggregated posterior leading to poor unconditional generation, see Remark f
Proposition (Marginal and conditional distribution matching) For any S E P(M), we have
/
pa(ss) 0, + H(pa(es))
(L(p0(,,assllp((z..s))) (L(pd(xs))po(<s))
ZX (Xmarginat)
/
(X (Xmarginal)
(Zaarinnat)
where q- (xs|=) = 4 (=s, 2)/ Moreover, for fixed as,
/ pa(msslas) ddn + H(pa(r))((ss)
= KL (aatttttiiiiaaaagsssoooooooooaa
(ZX condittion)))
= (Ls)
(Xconditional)
f pa(m)))os) KL (L((a(=))))lo((laa) / (=(x)))) log (o(2|zs) dz dxys
po(2|os)
(elg (
dz, (Zconditiona)

---

Published in Transactions on Machine Learning Research (09/2024)
where q²
2g9 o (2(2s)
If (0(2|o)) approximates po(=|zs) exactly, Proposition implics that Lis( 0, is lower bound of logpa(zree ( sss. More precisely, we obtain the following log-lioe approximation.
Corollary 2 (Multi-modal log-likelihood approximation) For any modality mask S, have
f pa(x) [Es(as, 0, o, 1) + (z 0, Φ, 1)] dx - | pa(x) [log (x)) dæ
= -- Pa(ms) (KL(gg(llls)lll(llrrr dx pa(x)) (KL(a (z((=)) dr
- / (a(2))
log ( Po (z) x
dzdæ.
Proof This follows from
X Xmarginal and
Fonditional
Our approach recovers meta-learning with (latent) Neural processes (Garnelo et al. 2018b) when one op- timizes only Lis with S determined by context-target splits, ef. Appendix [ Our analysis implies that Ls + Ls is an approximation of a lower bound on the multi-modal log-likelihood that bocomes tight for infinite-capacity encoders so that (o(=/as) = po(z(as) and (0(2(x) = p=(2))), see Remarks 3 and 5 for details.
Remark 3 LLog-Likelihood approximation and Empirical Bayes) The term
(=(x)))(=1x) log go (=(2 (o(2|rs)
dædæ
arising in Corollary and in (conditionall is not necessarily negative. Analogous to other variational ap- proaches for learning conditional distributions such latent. Neural processes, our bound becomes ap- proximation of lower bound. Note that Ls is maximized when g0(2|zs) po (s),, see Xmarginal which implies lower bound in Corollary 2 of
f pala) (,,(1),0,6,1) Gis(x, o, / pa(x) [log (L((a([ll))
We can re-write the conditional expectation of Lis for any fixed as as
I [alansses) 1)da = / pa p (o sleddades + (((anglas))((((((l)))))
/ a55 (2 log (og
Whenever (2(2)))) can be learned independently from (2)2), the above maximized for
( / pa(
pez (2(zs)
From a different perspective, we can consider an Empirical Bayes viewpoint (Robbins 1992 2019b) wherein one chooses the hyperparameters of the (conditional) prior so that maximizes an approxi- mation of the conditional log-likelihood log soss) The conditional prior p3(n)))) in the corresponding conditional ELBO term
Wang et al.
Fsslx,,,0000 = (0(2)²) (oog dz (KL(q
(6)
can thus be seen as a learned prior having the parameter (D) that is learned by maximizing the above variational approximation of log over pd for & 1, and as such depends the
8

---

Published in Transactions on Machine Learning Research (09/2024)
empirical multi-modal dataset D. While the aggregated prior 2 (2)rs) is the optimal learned prior when maximizing Lis: this choice can lead to overfitting. Moreover, computation of the aggregated prior, or sparse approximations thercof, such variational mixture of posteriors prior (VampPrior) in Tomczak and Welling (2017), are challenging in the conditional selup. Previous constructions in this direction (Joy el al. 2021) for learning priors for bi-modal data only considered unconditional versions, wherein pseudo-samples are not dependent on some condition zs. While our permutation-inaant
architectures introduced below may be used for flexibly paramcterizing conditional prior distributions (z as a function of as with a model that different from the encoding distributions (=())xs), we contend ourselves with choosing the same model for both the conditional prior and the encoding distribution, P (=| = (2]2z)) Note that the encoding distribution then features both bounds, which encourages learning encoding distributions that perform well as conditional priors and as encoding distributions. In the ideal scenario where both the generative and inference models have the flexibility to satisfy pa (ms) = po ( (0(2)z)) = po and (o(2(r)) = pa (z)æs), then the optimal conditional prior distribution is
2 (2|2s) = / pe polassslss)pol(lldddss = po(c))s/z)) ( drys = po(2|as) = (2(2)))) pa (((
Remark 4 (Prior-holc problem and Bayes or conditional consistency) In the uni-modal sctting, the mismatch between the prior and the aggregated prior can be large and can lead to poor uncondi- tional generative performance because this would lead to high-probability regions under the prior that have not been trained due to their small mass under the aggregated prior (Hoffman and Johnson 2016 Rosca et al. 2018). Equation [marginal) extends this to the multi-modal case, and we expect that unconditional generation can be poor if this mismatch is large. Moreover, (conditiional extends this conditioned on some modality subset, and we expect that cross-gencration for xy conditional on Is can be poor if the mismatch between ags (2(rs)) and (o(s( is large for us ~ pa, because
regions under (=(5( will not have been trained via optimizing L(s(r) to model Viss conditional on æs, duc to their small mass under 40 (z((s)) The mismatch will vanish when the encoders are consistent. and correspond to a single Bayesian model where they approximate the true posterior distributions. A potential approach to reduce this mismatch may be to include as a regulariser the divergence between them that can be optimized by likelihood-fres techniques, such as the Maximum-Mean Discrepancy (Gretton et al. 2006), as in Zhao et al. (2019) for uni-modal or unconditional models. For the mixture-based bound, the same distribution mismatch affects unconditional generation, while both the training and generative sampling distribution is (=(()))) for eross-generation.
#gg
Remark 5 (Variational gap for mixture-based bounds) Corollary shhows that the variational objec- tive can become a tight bound in the limiting case where the encoding distributions approximate the true posterior distributions. A similar result does not hold for the mixture-based multi-modal bound. More- over, our bound can be tight for an arbitrary number of modalities in the limiting case of infinite-capacity encoders. In contrast, Daunhawer et al. (2022) show that for mixture- based bounds, this variational gap increases with each additional modality if the new modality is 'sufficicntly diverse', even for infinito-capacity encoders.
Remark 6 (Optimization, multi-task learning and the choice of p) For simplicity, we have chosen to sample S ~ " in our experiments via the hicrarchical construction 7 ~ U(, 1), my ~ Bern(y) iid for all j [M] and setting S = (s E [M]: my = 1]. The distribution P for masking the modalities can be adjusted to accommodate various weights for different modality subsets. Indeed, (2) can be seen as linear scalarization of multi-task learning problem (Fliege and Svaiter 2000 Sener and Koltun 2018 We aim to optimize loss vector (Ls + Lis)scM, where the gradients for cach S C CM can point in different directions, making it challenging to minimize the loss for all modalities simultaneously. Consequently, [Javaloy et al. (2022) used multi-task learning techniques (e.g., as suggested in Chen et al. (2018): Yu et al. (2020)) for adjusting the gradients mixturo-based VAEs. Such improved optimization routines are orthogonal to our approach. Similarly, we do not analyze optimization issues such as initializations and training dynamics that
have been found challenging for multi-modal learning (Wang et al. 2020: Huang et al. 2022
9

---

Published in Transactions on Machine Learning Research (09/2024)
2.2
citimattion-hhorrttii perspective
Beyond generative modeling, V-VAEs Higgins et al. 2017 have been popular for representalion learning and data reconstruction. Alemi et al. 2018) suggest learning a latent representation that achieves certain mutual information with the data based on upper and lower variational bounds of the mutual information. A Legendre transformation thereof recovers the 3-VAE objective and allows a trade-off between information content or rate versus reconstruction quality or distortion. We show that the proposed variational objective gives rise an analogous perspective for multiple modalities. Recall that the mutual information on the inference path is given by
(o( "B8 po( 19.5 (zs)
Lo (Xs, ao(os)2s)
degdrs,
can be bounded by standard Barber and Agakov 2004
Alemi et al. 2016 2018 lower and upper bounds:
Hss - Ds I H. Ds + A 14 (Xs, Es) = Rs A S Rs.
(7)
with A,, A2 2 0 and Hs S Rs Ds. For details, see Appendix C Consequently, by tuning 8, we vary upper and lower bounds of (Xs. Zs) to tradeoff between compressing and reconstructing Xs.
To arrive at a similar interpretation for the conditional bound Lis that involves the conditional mutual information
Lo0 (X,s, Zm|Xs) httls///sssaaaaaaaaaaaesstttssss
recalling that 2 (=Ml=s) = ( ( we set
R / pcce)K((aataaa))lettsssl
for conditional or cross rate. Similarly, set
D.s - - pa
One obtains the following bounds, see Appendix
Lemma 7 (Variational bounds on the conditional mutual information) It holds that
-f / 0, ()p(ddr) Dis + +RR
and for A,s. 20, His Dis A (X XM) R.s A
Consoquently, by tuning 8, we can vary upper and lower bounds of Ig. (X Zs) to tradcoff between compressing relative to (sss) and reconstructing X.s- Using the chain rules for entropy, we obtain that the suggested bound can be seen relaxation of bounds on marginal and conditional mutual information.
Corollary 8 (Lagrangian relaxation) It holds that
Ds Dis S I, (Xs, Zs) + Iga ZM|Xs) S Rs + RAs
and mazimizing C for fixed B D(D+DD.s) minimizes the rates Rs + R.ss and distortions Ds + D R(
We include the conditioning modalities index the latent variable Z when the conditioning set unclear.
10

---

Published in Transactions on Machine Learning Research (09/2024)
Remark 9 (Mixture-based variational bound) We show in Appendix C
also Daunhawer et al.
2022), that
HM - Ds D5 - HM - Ds D3 + Ai = Iop (XM: Zs).
where A 1 while Rs is variational upper bound Tuo (XM; Zs); which establishes 2 Maximizing the mixture- based bound thus corresponds to encoding single latent variable Zs that maximizes the reconstruction of all modalities while at the saine time being maximally compressive relative to the prior.
(x)2)) (
0. Consequently,
Ds D variational lower bound,
Remark 10 (Optimal variational distributions) Consider the annealed likelihood p x po(zs|z) 1/8 as well as the adjusted posterior (3.0(2(as)) & (3.0(es(2)p((s). The minimum the bound J (s()) is attained at any as for the variational density
q (2((2)) exp ( log [ (logpo(2))) x (o.0(1/zs).
(8)
see also Huang et al. (2020) and Remark 18 Similarly, if (8) holds, then it is readily seen that the minimum
of the bound Pu(d))4 L
(r) is attained at any æ for the variational density g (x(xx) = (p.,(=/x)) In contrast,
as shown in Appendix 19 the optimal variational density for the mixturo-based
multi-modal bound is
attained at
q- (z|æs) x (2.0( exp (f (dalersses)) log ts.o (
The optimal variational density for the mixture- based bound thus tilts the posterior distribution to points that achiove highor cross-modal predictions.
3 Permutation-invariaant modality encoding
Optimizing the above multi-modal bounds requires learning variational densities with different conditioning sets. We write he : X, + R for some modality-specific feature function. We recall the following multi- modal encoding functions suggested in previous work where usually hz,p = [Ma.p (ax)), vx.(E..p(z.)) (xx))) with Ma.p and Es.p being the mean, respectively the (often diagonal) covariance, of a uni-modal encoder of modality Accommodating more complex variational families, such mixture distributions for the uni- modal encoding distributions, can be more challenging for these approaches.
MoE: 4 (2l2s) = [ Exes (2(2) (x.)), where qv(2/p, Σ) is Gaussian density with mean a and covariance E.
PoE: uf PoB (z(us) p=) ILeS IN (=(2 (Ma), (x)()))), for some E R. For Gaussian priors pa(2) = (z/ ig) with mean ne and covariance Le, the multi-modal distribution de Poo (2|2s) is Gaussian with mean
(Mo2D + Eu (2s)) (x,)) + (2)
#ES
SES
and covariance
(Σ + Σ (x
SES
Mixture of Product of Experts (MoPoE), see Sutter et al.
2021
MoPoE
(2|xM) 1
Σ
PoB (z|Ts))
g
2M
#gEP(zM)
11

---

Published in Transactions on Machine Learning Research (09/2024)
3.1 Learnable permutation-invariae aggregation schemes
We aim to learn a more flexible aggregation scheme under the constraint that the encoding distribution is invariant (Bloem-Reddy and Tch: 2020) with respect to the ordering of encoded features of each modality. Put differently, for all (Hs)sss E RS and all permutations π E Sş of S, we assume that the conditional distribution is Sg-invariant, i.. (2(2/h) (2( (2|=Thh) for all & E RD, where x acts on H = (Ha)aes via x x- H = & = (e, 0) and remark that the encoding distribution is
(Hx()))sss- We set (o(2|rs) = not invariant with respect to the modalities, but becomes only invariant after applying modality-speeifie encoder functions ha Observe that such constraint is satisfied by the aggregation schemes above for ha.g
being the uni-modal encoders.
A variety of invariant (or equivariant) functions along with their approximation propertics have been consid-
ered previously, see for instance Santoro et al. (2017) Zaheer et al. (2017); Qi el al. (2017) Lee et al.
(0019)
Segol and Lipman 2019); Murphy et al. (2019); 1; Maron et al. (2019); Sannai et al. (2019): Yun et al.
2019
Bruno ct al. (2021) Wagstaff ct al. (2022); ; Zhang et al. (2022b); Li al. (2022):
Bartunov et al. (2022
and
applied in different contexts such as meta-learning (Edwards and Storkey 2016 Garnelo el al.] 2018b
Kim
et al. 2018 Hewitt et al. 2018 Giannone and Winther (022), reinforcement learning (Tang and Ha
2021
Zhang et al.
2022a) or generative modeling of (uni-modal) sets (t al., 2018 2020 Kim et al. 2021
Bilos
and Günnemann 2021 Li and Oliva 2021). We can use such constructions to parameterize more flexible
encoding distributions. Indeed, the results from Bloem-Reddy and Teh (2020) imply that for an exchangable
sequence Hs = (H.)ses E RlSiiDe and random variable Z, the distribution g((2(4s) is Sg-invariant if and only if there is a measurable function f [0, x M(RPs) + IR such that
2) B.S. (Hs, f' )), where E 1 210 1] and E L Hs
with -+ óH, being the empirical measure of hs, which retains the values of hss but discards their order. For variational densities from location-scale family such Gaussian Laplace distribution, we find it more practical to consider a different reparameterization in the form Z = j(hs) + a(hs) @ E, where E is sample from paramcter-free density such as standard Gaussian and Laplace distribution, while [u(hs), log (hss]] = ((bs) for a PI function f: RI - RRD Likewise, for mixture distributions thereof, assume that for a PI function f.
[m (hs), log 01 (hs)), ................................ ux (hs) log lx log whs)) = f(hs) E 222K*K
and Z= = (bs) + or. (hs) o with L ~ Chs(w(hs)) denoting the sampled mixture component out of K mixtures. For simplicity, we consider here only two examples of PI functions f that have representations with parameter in the form
fo(hs) = pa Σ go((s))
WES
for function Po: Rr + Do and permutation-equivarian function go: R. NxDE + RNDP.
Example 1 (Sum Pooling Encoders) The Deep Set ZZahoer et al. 2017 construction fo(hs) Po (Exes Xo(h)) applies the same neural network X#: RP RDr each encoded feature h We assume that Xo is feed-forward neural network and remark that pre-activation ResNets (He et al.) 2016) have been advocated for deeper Xa- For exponential family models, the optimal natural parameters of the posterior solve an optimization problem where the dependence on the generative parameters from the different modalities decomposes as a sum, see Appendix
Example 2 (Set Transformer Encoders) Let MTB, be multi-head pre-layer-norm transformer block
Wang et al. 2019a Xiong el al. 2020), see Appendix D for precise definitions. For some neural network Xa: R + RP, set d = Xa(hs) and for k E {I,..., set ott = ,
We then consider
fo(hs) Pa (Exes ). This can be seen as a Set Transformer (Lee et al. 2019 Zhang et al. 2022a) model
"The function f" generally depends the cardinality S. Finite-length exchangeable sequences imply a de Finetti latent. variable representation only up to approximation errors DDicconis and Freedman 1980).
12

---

Published in Transactions on Machine Learning Research (09/2024)
without any inducing points as for most applications, computational complexity that scales quadratically in the number of modalities can be acceptable. In our experiments, use layer normalization Ba et al. 2016) within the transformer model, although, for example, set normalization (Zhang et al. 2022a could be used alternatively.
Note that the PoE's aggregation mechanism involves taking inverses, which can only be approximated by the learned aggregation models. The considered permutation-invaiian models can thus only recover a PoE scheme under universal approximation assumptions.
Remark 11 (Miitture--fPrrodct------eeeee or MoPoEs) Sutter et al. (2021) introduced MoPoE aggregation scheme that extends MoE PoE schemes by considering mixture distribution of all 2M modality subsets, where each mixture component consists of PoE model, i.e.,
MoPoE (2|x.)
Σ
PoE 4
(2|zs).
2M
#8EP(=M)
This also be another PI model. While it does not require learning separate encoding models for all modality subsets, it, however, becomes computationally expensive to evaluate for large M. Our mixture models using components with SumPooling or SelfAttention aggregation can be seen as an alternative that allows one to choose the number of mixture components K to be smaller than 2 with non-uniform weights, while the individual mixture components are not constrained to have a PoE form.
Remark 12 (Pooling expert opinions) Combining expert distributions has a long tradition decision theory and Bayesian inference; see Genest and Zidek 1986) for early works, with popular schemes being linear pooling (i.e., MoE) or log-linear pooling (i.e., PoB with tempered densities). These are optimal schemes for minimizing different objectives, namely weighted (forward reverse) KL-divergence between the pooled distribution and the individual experts (Abbas 2009 Log-linear pooling operators are externally Bayesian, allowing for consistent Bayesian belief updates when each expert updates her belief with the same likelihood function Genest et. al. 1986).
3.2 Permutation--qquivrrianee and private latent variables
In principle, the general permulation invariant aggregation schemes that have been introduced could also be used for learning multi-modal models with private latent variables. For example, suppose that the generative model factorizes as
po(2,x)= = p(z) II (o(x, SEM
(9)
for (2., Z, for shared latent variables Z' and private latent variable 20 for each s E M. Note that for s * [M],
X. L 2. Z.
(10)
Consequently,
(0 &s, slsss) po(2', Es, / (=
(11)
An encoding distribution (=(=)))) that approximates (=(a should thus be unaffected by the inputs æs when encoding 2. for $ S, provided that, a priori, all private and shared latent variables are independent. Observe that for fo with the representation (3.1) where Ps has aggregated inputs y, and that parameterizes the encoding distribution of (x', ăs, values x, is
s)s) the gradients of its i-th dimension with respect to the modality
Opy.i Dy
a Eso(hs(rs))))
lfo (fo(hs(zs))d)
ggo(hs
Or.
dx, les
In the case of a SumPooling aggregation, the gradient simplifies to (h.(x:))
Dpa.i
Exo(h((t>))
0xe
(h.(I,) dz.
Oy
Zh
rES
13

---

Published in Transactions on Machine Learning Research (09/2024)
Suppose that the i-th component of pa maps to the mean log-standard deviation of some component of 7, for some E S. Notice that only the first factor depends on i so that for this gradient to be zero, Pe,i has to be locally constant around y = E.es (xx)) if some other components have non-zero gradient with respect to xz. It it thus very likely that inputs X, for s E S can impact the distribution of the private latent variables 3,s-
However, the specific generative model also lends itself an alternative parameterization that guarantees that cross-modal likelihoods from X,s do not affect the encoding distribution of Zs under new variational into
objective. The assumption of private latent variables suggests additional permutation-equuiariin the encoding distribution that approximates the posterior in 11 in the sense that for any permutation π E Ss, it holds that
dás/sT hp(#s), = aa . as (zs))
assuming that all private latent variables are of the same dimension D Indeed, suppose we have modality- specific feature functions hp such that { [H, = h (X,)) is exchangcable. Clearly, (10) implics for any * t that
h (X,) 1 Ze z', 2,.
The results from Bloem-Roddy and Tch (0200) then imply, for fixed [8], the existence of a function f such that for all 8 E S, almost surely,
(Hss, 2.) = (Hs. Z', Ha, M))), where E. ~ 41 1] iid and E. 1 Hs.
(12)
This fact suggests an alternative route to approximate the posterior distribution in 11 First, po po( (Es)
can often be computed analytically based on the learned or fixed prior distrrbuttin. Second, a permutation- invariant scheme can be used to approximate pa (= Finally, a permutation-equivarian scheme can be employed to approximate po(ăs z') with a reparamcterization in the form of 12 The variational objective that explicitly uses privale latent variables is detailed Appendix permutation-equivalat
Three examples of such
schemes are given below with pseudocode for optimizing the variational objective given in Algorithm 2 Note that the assumption (áslaa, as5) = (= æs)) an inductive bias that generally decreases the variational objective it imposes a restriction
the encoding distribution that only approximates the posterior where this independence assumption holds. However, this independence assumption allows us to respect the modality-specific nature of the private latent variables during encod- ing. In particular, for some comuuaation-ivvarian encoder (=(2"(=s) for the private latent variables and permutation -quivarrann encoder (0(/) æs) for the private latent variables of the observed modalities, we can encode via
qs(=', {m]=s) - (=ss( (ā lz' (as(Esl as)
that the mopality-speeific information of xs encoded via Es is not impacted by the realisation 7 of modality-specifie variation from the other modalities.
Example 3 (Permutation-cquivaiiae PoΣ) Similar to previous work Wang et al. 2016 Lee and
Pavlovic 2021 Sutter et al. (2020), consider encoding density of the form
g0(2', oe
Po (='(as) II
exr(Es E (xx)) SES
II po(Es), 8EM1S
where
d PoE (2'
P (2'(0s)
Po
1
II avr (xx), (x.p(())) SES
is a coamutattioninvariinnt PoE aggregation, and we assumed that the prior density factorizes over the shared and different private variables. For cach modality s, we encode different features N.,pp = (K'a Ex,p)) and hawp = (Na.e. )) for the shared, respectively, private, latent variables. We followed previous works Tsai et al. 2019b Lee and Pavlovic 2021 Sutter et al. 2020 in that the encodings and prior distributions for the modalltyyspecilic latent variables are independent of the shared latent variables. However, this assumption can be relaxed, as long as the distributions remain Gaussian.
5Te effective dimension can vary across modalities in practice if the decoders are mask redundant latent dimensions.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Example 4 (Permutation-eequivarin Sum-Pooling) We consider encoding density that is written g0(2', = SumP (z' _ (2/(xs) |
Equiv-SumP (8s|=',2)) I&)
II sEMYS
po(2,12').
Here, we use (permutation-invaaiin)) Sum- Mho-Pnoling aggregation scheme for constructing the shared latent variable Z' (hs) to (hs) @h') ds SumP (='(ss), where E' ~ p and fo R| RD given as in Example Equiv-SumP (žs|=',as) .s)) consider functions Xj.0: RD
1 with (h), log o' (h)) = fo(h). To sample Øs Os DD j E [3] and Po: Dr - Po, e.g., fully-connected neural networks. We define Sg Equiv-SumP, RiS|DDs - IR Do via
8 Equiv. (z', = xana(ht) ([ + xx,o(=z) + xeal(hs) LIES
With (ā(hs) log T(hs) TI h = (x,)) for modality-speciffc feature functions hpps X, + RR,
fa Equiv-SumP ks), we then set - o(hs), OE, for E, iid,
Example 5 (Permutation-oquuvvrinn colf-Attention) Similar to Sum-Pooling approach, we con- sider an encoding density that is written
(0 &m(=s) = 9o SA (2(12))9
Equiv-SA (ăs)z' xs)
po(Ez|z'). .EMiS
II
Here, the shared latent variable Z' sampled via the permutation-invariane aggregation above by summing the elements of permutation-eqquiarriaan transformer model of depth L'. For encoding the private latent variables, we follow the example above but set
[A(hs) log (hs) = = EEqiv (2', = =95
with d MTB:(gg=)) an 9 (ha, + x2,0(x))
4
Identifiability and model extensions
4.1
Identifiability
Identifiability of parameters and latent variables in latent structure models is classic problem (Koopmans 1976 Allman el al. 2009), that has been studied increasingly for non-linear
and Reierso] 1950: Kruskal
latent variable models, e.g., for ICA [Hyarrinnn and Morioka 2016 Hälvä and Hyvarinen] 2020 Hälvä et al.
2021 VAEs [Khemakhem et al. 2020a Zhou and Wei 2020 Wang ct al. 2021 [ et al. 2021 Lu et al.
2022 Kim et al. 2023) EBMs Khemakhem et al. 2020b), flow-based [Sorrenson et al. 2020 or mixture
models (Kivaa al. 2022).
Non-linear generative models are generally unidentifiable without imposing some structure (Hyvärinen and Pajunen 1999 Xi and Bloem-Reddy 2022). Yet, identifiability up to some ambiguity can be achieved in some conditional models based on observed auxiliary variables and injective decoder functions wherein the prior density is conditional on auxiliary variables. Observations from different modalities can act as auxiliary variables to obtain identifiability of conditional distributions given some modality subset under analogous assumptions.
Example 6 (Auxiliary variable as modality) In the iVAE model [Khemakhem et al. 2020a). the latent variable distribution p=(2(21) is independently modulated via an auxiliary variable Xi U. Instead of interpreting this distribution as a (conditional) prior density, we view it as a posterior density given the first modality X,- Khemakhem et al. (2020a) estimate a model for another modality X₂ by lower bounding log po (z lx,) via Lx(1) under the assumption that (2(22) is given by the prior density po(2(æ1). Similarly, Mita et al. 2021 optimize log po (2) by double VAE bound that reduces L for masking
15

---

Published in Transactions on Machine Learning Research (09/2024)
distribution 82) (61 @ (s) 52) that always masks the modality X2 and choosing to parameterize separate encoding functions for different conditioning sets. Our bound thus generalizes these procedures to multiple modalities in scalable way.
We are interested in identifiability, conditional on having observed some non-empty modality subset 8 C M. For illustration, we Iranslate an identifiability result from the uni-modal iVAE setting in Lu el al. (2022) which does not require the conditional independence assumption from Khemakhem et al. (2020a). We assume that the encoding distribution (o(2[xg) approximates the true postcrior po(2|ag) and belongs strongly exponential family, i.e,
po(2(0s) qs (z( PVo.s (z/zs),
(13)
with
PVs. (2|xs) = m(z) exp (Vs(>), log logTs(As(rs))),
where μ is base meastre, V's: Z 1R* is the sufficient statistics, (s R the natural parameters and ss normalizing term. Furthermore, one can only reduce the exponential component to the base mcasure sets having measure zero. In this section, we assume that
(=) Ps,c(1, f2))
(14)
for some fixed noise distribution Pa.e with a Lebesgue density, which excludes observation models for discrete modalitics. Let Og be the domain of the parameters fg = (fis, Vs, with h,s: Z (fs(z))sEMiss e XxEMixX Xys. Assuming 13 note that
Pos (as(ois(os)
-
p - fis(z))d2,
with P.S.e SIEM)SPar We define equivalence relation on Os by (iis, Vs, ~As (hss Vs,As) As) ifl there exist invertible As R and cs E R such that
(s(fis((ass)) AsVs +
for all Iis Xys-
Proposition 13 (Weak identifiability) Consider the data generation mechanism x)) po(z) IT Po (2) where the observation model satísfies (14) for an injective fis- Suppose further that po(=(as)) is strongly exponential and (13) holds. Assume that the set (ri E X (P)ss)( 0} has measure zero, where s.s.e is the characteristic function of the density Pis.e. Purthermore, suppose that there exiit k + points I²) u.......' E Ex such that
L= [>>(s''s As(az)) As
{s(x))] E R
is invertible. Then Pas (nys xs) Pas (1))) for all x E X implies 0 ~As 0.
This result. follows from Theorem 4 in Lu al. (2022). Note that Pas (m)s(es) Pog (s)s) for all x E X implics with the regularity assumption on P.s.c that the transformed variables = (X and 2 Fis (X have the same density function conditional on Xs.
Remark 14 (Conditional identifiability) The identifiability result above is about conditional models and does not contradict the un-identifiability of VAEs: When S = 0 and we view ZM one modality, then the parameters of pe, (x) characterized by the parameters V/ and Ap of the prior pe, (2(ra) and the encoders fM will not be identifiable as the invertibility condition will not be satisfied.
Remark 15 (Private latent variables) For models with private latent variables, we might not expect that conditioning on Xs helps to identify Zis as
po(z', âs. ((ssss) = po(z', 8sl &s))
Indeed, Proposition 13 will not apply in such models fis will not be injective.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Remark 16 (Data supported on low-dimensional manifolds) Note that 14 and 13 imply that
each modality has a Lebesgue density under the generative model. This assumption may not hold for some
modalities, such as imaging data that can be supported (closely) on a lower-dimensional manifold Roweis
and Saul 2000 causing issucs in likelihood-based methods such as VAEs (Dai and Wipf 2018 Loaiza-
Ganem et al.) 2022). Moreover, different conditioning sels or modalities may result in different dimensions
of the underlying manifold for conditional data (Zheng et al., 2022). Some two-step approaches (Dai and
Wipf 2018 Zheng et al., 2022) first estimate the dimension of the ground-truth manifold as function of the encoder variance relative to the variance under the (conditional) prior for each latent dimension i, E [P], with S D. It would, therefore, be interesting to analyze in future work if more flexible aggregation schemes that do not impose strong biases on the variance components of the encoder can better learn the manifold dimensions in conditional or multi-modal models following an analogous two-step approach.
Recall that the identifiability considered here concerns parameters of the multi-modal posterior distribution and the conditional generative distribution. It is thus preliminary to estimation and only concerns the gen- erative model and not the inference approach. However, both the multi-modal posterior distribution and the conditional generative distribution are intractable. In practice, we thus replace them with approxima- tions. We believe that our inference approach is beneficial for this type of identifiability when making these variational approximations bocause (a) unlike some other variational bounds, the postcrior is the optimal variational distribution with (x)()) being an approximation of a lower bound on log po s see Remark I0 and (b) the trainable aggregation schemes can be more flexible for approximating the optimal encoding distribution.
4.2
Mixture models
An alternative to the choice of uni-modal prior densities has been to use Gaussian mixture priors (Johnson
ct al. 2016 Jiang et al. 2017 Dilokthanakul et al., 2016) or more flexible mixture models (Falck et al. 2021j. Following provious work, we include a latent cluster indicator variable c E [K] that indicates the mixture component out of K possible mixtures with augmented prior po(e, z) po(c)po(zle). The classic example is pe(c) being categorical distribution and pe(zlc) a Gaussian with mean Me and covariance matrix Σc. Similar to Falck et al. (2021) that use optimal variational factor in mcan-field model, we use an optimal factor of the cluster indicator in structured variational density go(c, =|xs) = (((=(=)))((()))) (c),,,s)))
with go (c(2,, =s) = (cl=). Appendix G details how one can optimize an augmented multi-modal bound.
Concurrent work (Palumbo et al. 2024 considered similar optimal variational factor for discrete mixture
model under a MoE aggregation.
4.3 Missing modalities
In practical applications, modalities can be missing for different data points. We describe this missing- ness pattern by missingness mask variables m, E 10 where m, 1 indicates that observe modal- ity 8, while m, = 0 means it is missing. The joint generative model with missing modalities will be of the form Po(2, x, m) = po(z) II (=,11=)))(()))
for some distribution (m(()) over the mask variables m = (m,)seM- For S C M,, we denote by x3 = [x.: m. = 1,8 E S) and x"'h = {x.: m. = 0, s E S} the set of observed, respectively missing, modalities. The full likelihood of the observed and missingness masks becomes then po(=3, m) = f (o()) IL Po(xx]/ )) po
(m| (x) dz. If (m()aa .r) does not depend on the observations, that is, observations are missing completely random (Rubin 1976). then the missingness mechanisms po(m) for inference approaches maximizing Po(=", m) can be ignored. Consequently, one can instead concentrate maximizing log po(")) only, based on the joint generative model po(2, = p=(z) II m,=1) Po (z (z). In particular, one can employ the variational objectives above by considering only the observed modalities. Since masking operations are readily supported for the considered permulation-invariinl models, appropri- ate imputation strategies (Nazabal et al.] 2020 Ma et al., 2019) for the encoded features of the missing modalities are not ncoessarily required. Settings allowing for not (completely) at random missingness have been considered in the uni-modal case, for instance, in [psen al. (2021): Ghalebikesabi et al. (2021); Gong et al. (2021). and we leave multi-modal extensions thereof for future work for a given aggregation approach.
17

---

Published in Transactions on Machine Learning Research (09/2024)
5 Experiments
We conduct series of numerical experiments to illustrate the effects of different variational objectives and aggregation schemes. Recall that the full reconstruction log-likelihood is the negative full distortion -DM based all modalities, while the full rate RM is the averaged KL between the encoding distribution of all modalities and the prior. Note that besture--based bounds maximize directly the cross-modal log- liglliielhoood - -Dis, see ( and do not contain a cross-rate term Ris, i.e. the KL between the encoding distribution for all modalities relative to a modality subset, a regulariser, in contrast to our objective (Lemma f and Corollary
8 The log-likelihood should be higher if generative model is able to caplure modality-speeifee information for models trained with 8 = 1. For arbitrary 3, we can take a rate-distortion perspective and look at how different generative models self-rcconstruct all modalities, i.c., the full reconstruction term - -DM; relative to the KL-ddvergence between the multi-modal encoding distribution and the prior, i.e. RM- This corresponds to a rate-distortion analysis of a VAE that merges all modalities into a single modality, A high full-reconstruction term thus indicative of the encoder and decoder being able to reconstruct all modalities preciscly so that they do not produce average prediction. Note that neither our objective nor the mixture-based bound optimize for the iill-reoonstucttoo lerm directly.
5.1 Linear multi-modal VAEs
The relationship between uni-modal VAEs and probabilistic PCA (ippping and Bishop, 1999) has been studied in previous work (Dai et al. 2018 Lucas et al. 2019 Rolinek et al. 2019 Huang et al. 2020:
Mathieu et al. 2019). We analyze how different multi-modal fusion schernes and multi-modal variational objectives affect (a) the learned generative model in terms of its true marginal log- liglikelihood (LLH) and (b) the latent representations in terms of information-thcoreee (weak) identifiability of the method, we follow Khemakhem et al. (2020a b) to compute the mean correlation coefficient (MCC) between the true latent variables Z and samples from the variational distribution go( (a.) after an affine transformation using CCA.
quantitics and identifiability. To evaluate the
Generative model. Suppose that a latent variable Z laking values in R.P is sampled from slandard Gaussian prior (z(2) = (0, generates M data modalities X, E R., D < D., based on a linear decoding model po [2),1=)== NV, + ba, I) for a factor loading matrix W, E D.xD bias b, E D. and observation scale > 0. Note that the annealed likelihood function (2,6(=..22) = N(W,= + ba, 3o corresponds to a scaling of the observation noise, so that we consider only the choice = 1, set 0g 0 and vary 8 > O. It is obvious that for any S C M, it holds that (3.0(=5(2) = NWWs2 + bs, 03 Is), where Ws and 6y are given by concatenating row-wise the emission or bias matrices for modalities in S, while o3 Is is the diagonal matrix of the variances of the corresponding observations. By standard properties of Gaussian distributions, it follows that (=5) = [bs Cs) where Cs = WsWJ +o Is is the data covariance matrix. Furthermore, with Ks WJW Ws + o3 La, the adjusted posterior is (3.e(= (N(K'' (as bs). o3 La La ") If we sample orthogonal rows of W, the posterior covariance becomes diagonal that it can in principle be well approxinated by encoding distribution with diagonal covariance matrix. Indeed, the inverse of the posterior covariance matrix is only a function of the generative parameters of the modalities within S and can be written as the sum o3 I+W Ws = o3 I E.es W.W,, W., while the postcrior mean function is zs 7 (0) 1+ Eves W. W.) E.es Wo( b.).
Illustrative example. We consider a bi bi-lodal setup comprising a less noisy and more noisy modality. Coneretely, for latent variable Z (Z1, Z2, Za) RR3, assume that the observed modalitics can be repre- sented as
X, = Zo + 2, + U1
X₂ Zo + 107 + U2,
for standard Gaussian prior Z ~ (O, and independent noise variables U1, ~ ~N(00,) 1). Note that the second modality is more noisy compared to the first one. The results in Table T for the obtained log-likelihood
18

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Gaussian model with a noisy and less noisy modality. Relative difference of the true MLE vs the (analytical) LLH from the learned model in the first two columns, followed by multi-modal information theoretic quantities.
Relative LLH gap
Full Roconstraation
Full Rotes
Cross Prediction
Cross Rates
Aggregation
our obj.
mixture hound
our obj.
mixture bound
our obj.
mixturre bound
our ob.
mixture bound
our obj.
mixtare bound
1.29
2.30 10
2.2 10
1025
2.0. 1035
10*
10%
1.4 10
1.7. 10
McE
0.11
0.6
-32.07
-30.09
1.02
2.84
-33.27
-28.52
2.37
19.33
SumPooling
10
-2.84
-3.23
2.88
2.82
-32.58
-27.26
1.42
27.35
SelAAttention
3.4 10
0.06
-2.85
-3.23
2.87
2.82
-52.59
-27.25
1.42
27.41
Table 2: Gaussian model with five modalities: Relative difference of true LLH to the lcarned LLH. MCC true latent. The generative model for the invariant aggregation schemes uses dense decoders, whereas the ground truth model for the permutation-eqquiaaiia
encoders uses sparse decoders to account for private latent variables. We report mean values with standard deviations in parentheses over five independent runs.
Invariant aggregation
Equivariant aggregation
Proposed objective
Mixture bound
Proposed objective
Mixture bound
Aggregation
LLH Gap
MCC
LLH Gap
MCC
PoE
0.03 (0.058)
0.75 (0.20)
0.04 (0.074)
0.77 (0.21)
0.00 (0.000)
0.91 (0.016)
0.01 (0.001)
0.88 (0.011)
MoE
0.01 (0. (0.005)
0.82 (0.04)
0.02 (0.006)
0.67 (0.03)
SumPooling
0.00 (0.000)
0.84 (0.00)
0.00 (0.002)
0.84 (0.02) 0.00 (0,000)
0.85 (0.004)
0.00 (0.000)
0.82 (0.003)
SelfActention
0.00 (0.003)
0.84 (0.00)
0.02 (0.007)
0.83 (0.00)
0.00 (0.000)
0.83 (0.006) 0.00 (0.000)
0.83 (0.003)
values show first that learnable aggregation models yield higher log-likelihoodd") and second that our bound yiclds higher log-likelihood values compared to mixture-based bounds for any given fixed aggregation model. We also compule various information theoretic quantities, confirming that our bound leads to higher full reconstructions at higher full rates and lower cross predictions at lower cross rates compared to mixture- based bounds. More flexible aggregation schemes increase the full and cross predictions for any given bound while not necessarily increasing the full or cross rates, i.c., they can result in improved point within a rale-distortion curve for some configurations.
Simulation study. We consider M- = 5 modalities following multi-variate Gaussian laws. We consider generative models, where all latent variables are shared across all modalities, as well as generative models, where only parts of the latent variables are shared across all modalities, while the remaining latent vari- ables are modality-speeific. The setting of private latent variables can be incorporated by imposing sparsity structures on the decoding matrices and allows to analyze scenarios with considerable modality-specifie variation described through private latent variables. We provide more details about the data generation mechanisms in Appendix J For illustration, invariant aggregations in the first case and multi-modal encoders that utilize additional equivariant aggre- gations for the private latent variables in the second case. Results in Table 2 suggest. that more flexible aggregation schemes improve the LLH and the identifiability for both variational objectives. Furthermore, our new bound yields higher LLH for a given aggregation scheme.
use multi-modal encoders with shared latent variables using
5.2 Non-linear identifiable models
We construct artificial data following Khemakhem et al. (2020a), with the latent variables E RDD being conditionally Gaussian having means and variances that depend on an observed index value X2 E [K]. More precisely, po(2|12) = -Nuua Eny), where Pe ~ @ U(-,,5) and E. diag(Ac), Ae @ u(005,3 3) tid for e E [K]. The marginal distribution over the labels is uniform ((K]) so that the prior density Po(z) = JKK] (a(=)a2))o((
Auxiliary labels as modalities.
becomes a Gaussian mixture. We choose an injective decoding function fi: D + RDi, D < DI, as a composition of MLPs with LeakyReL.US and full rank weight matrices having monotonically increasing row dimensions [Khemakhem et al. 2020b), with iid randomly sampled entries. We assume 2 ~ (Z) o² I) and set a = 0.1, D = D = 2. fi has a
We found that PoE model can have numerical issues here.
19

---

Published in Transactions on Machine Learning Research (09/2024)
(a) Data X
(b) Proposed objec-(c) Proposed objec-(d) Mixture
bound Mixture bound +PoE
tive +SumPool
tive +PoB
+SumPool
(I) True
(H) Proposed objec-(li) Proposed objec-(i) Mixture
bound
Mixture bound +PoE
tive +SumPool tive +PoE
+SumPool
Figure 3: Continuous data modality in (a) and reconstructions using different bounds and fusion models in (b)-(e). The true latent variables are shown in (f), with the inferred latent variables in (g)-(i) with linear transformation indeterminacy. Labels are color-coded.
single hidden layer of size Dj 2. One realization of bi-modal data X, the true latent variable Z, as well inferred latent variables and reconstructed data for selection of different bounds and aggregation schemes, are shown in Figure 3 with more examples given in Figures G and 7 We find that learning the aggregation model through SumPooling model improves the data reconstruction and better recovers the ground truth latents, up to rotations, in contrast. to a PoF. model. Simulating five different such datasets, the results in Table 3 indicate first that our bound obtains better log-likelihood estimates for different fusion schernes. Second, demonstrates the advantages of our new fusion schemes that achieve better log-likelihoods for both bounds. Third, it shows the benefit of using aggregation schemes that have the capacity to accommodate prior distributions different from single Gaussian. Also, MoE schemes lead to low MCC values, while PoE schemes have high MCC values.
Table 3: Non-linear identifiable model with one real-valued modality and an auxiliary label acting as a second modality: The first four rows use a fixed standard Gaussian prior, while the last four rows use a Gaussian mixture prior with 5 components. Mean and standard deviation over repetitions. Log-likelhhoods are estimated using importance sampling with 64 particles.
Proposed objective
Mixture bound
Aggregation
LLH (8 1)
MCC (3 1)
MCC (8 0.1)
LLH (3
MCC (3
MCC (3 0.1)
PoE
-43.4 (10.74)
0.98 (0.006)
0.99 (0.003)
-318 (361.2)
0.97 (0.012)
0.98 (0.007)
MoF:
-20.5 (6.18)
0.94 (0.013)
0.93 (0.022)
-57.9 (6.23)
0.93 (0.017)
0.93 (0.025)
SumPooling
-1.9 (3.92)
0.99 (0.004)
0.99 (0.002)
-18.9 (4.09)
0.99 (0.005)
0.99 (0.008)
SelfAttention
-18.2 (4.17)
0.99 (0.004)
0.99 (0.003)
-18.6 (3.73)
0.99 (0.004)
0.99 (0.007)
SumPooling
-15.4 (2.12)
1.00 (0-001)
0.99 (0.004)
-18.6 (2.36)
0.98 (0.008)
0.99 (0.006)
SelfAttention
-15.2 (2.05)
1.00 (0.001)
1.00 (0.004)
-18.6 (2.27)
0.98 (0.014)
0.98 (0.006)
MumooolingMistture
-15.1 (2.15)
1.00 (0.001)
0.99 (0.012)
-18.2 (2.80)
0.98 (0.010)
0.99 (0.005)
SelfAttennionMiitture
-15.3 (2.35)
0.99 (0.005)
0.99 (0.004)
-18.4 (2.63)
0.99 (0.007)
0.99 (0.007)
Multiple modalitics. Considering the same generative model for Z with a Gaussian mixture prior, suppose now that instead of observing the auxiliary label, we observe multiple modalities X, RP.
1 (N(Ja(Z), for injective MLPs fa constructed as above, with D 10, D. 25, a 0.5
20

---

Published in Transactions on Machine Learning Research (09/2024)
Table 4: Partially observed (1 = 0.5) and fully observed (7 = 0) non-linear identifiable model with 5 modalities: The first four rows use a fixed standard Gaussian prior, while the last four rows use Gaussian mixture prior.
Partially observed
Fully observed
Proposed objective
Mixture bound
Propoeed objective
Mixture bound
Aggregation
LLH
MCC
LLH
MCC
LLH
MCC
LLH
MCC
PoE
-250.9 (5.19)
0.91 (0.015)
-288.1 (8.53)
0.93 (0.018)
-173.6 (9.01)
(0.005)
-197.7 (11.26)
0.97 (0.008)
MoE
-250.1 (4.77)
0.92 (0.022)
-286.2 (7.63)
0.90 (0.019)
-47..9 (8.50)
0.91 (0.014)
-494.6 (9.20)
0.92 (0.004)
SumPooling
-219.6 (1.85)
0.95 (0.016)
-275.6 (7.35)
0.92 (0.031)
-4714 -4714(88.99
0.99 (0.004)
-480.5 (8.81)
0.98 (0.005)
SelfArteation
-219.7 (1.83)
0.95 (0.014)
-275.5 (7.15)
0.93 (0.022)
-171.4 (8.97)
0.99 (0.002)
-182.8 (10.51)
0.98 (0.004)
SumPooling
-247.3 (4.23)
0.95 (0.009)
-269.6 (7.42)
0.94 (0.018)
-465.4 (8.16)
(0.002)
-475.1 (7.54)
(0.003)
SelfAstetion
-247.5 (4.22)
0.95 (0.013)
-269.9 (6.06)
0.93 (0.022)
-469.3 (4.76)
0.95 (0.003)
-474.7 (8.20)
0.95 (0.002)
Sum PomPolinngMixtture
-244.8 (4.44)
0.95 (0.011)
-271.9 (6.54)
0.93 (0.021)
-464.5 (8.16)
0.99 (0.003)
-474.2 (7.61)
0.98 (0.004)
SellAdeeniooMiiture
-245.4 (4.55)
0.96 (0.010)
-270.3 (5.96)
0.94 (0.016)
-464.4 (8.50)
0.99 (0.003)
-473.6 (8.24)
0.95 (0.002)
Table 5: Test LLH estimates for the joint data (M+S+T) and marginal data (importance sampling with 512 particles). The first part of the table is based on the same generative model with shared latent variable Z R:40 while the second part of the table is based on restrictive generative model with shared latent variable Z' E I0 and modality-speciffe latent variables Z, E I.
Proposed objective
Mixture bound
Aggregation
M+S+T
M
S
T
M-S+T
M
S
T
PoE+
6872 (9.62)
2599 (5.6)
4317 (1.1)
-9 (0.2)
5900 (10)
2449 (10.4)
3443 (11.7)
-19 (0.4)
PoF.
6775 (54.9)
2585 (18.7)
4250 (8.1)
-10 (2.2)
5813 (1.2)
2432 (11.6)
3390 (17.5)
-19 (0.1)
MoE+
5428 (73.5)
2391 (104)
3378 (92.9)
-74 (88.7)
5420 (60.1)
2364 (33.5)
3350 (58.1)
-112 (133.4)
MoE
5597 (26.7)
2449 (7.6)
3557 (26.4)
-11 (0.1)
5485 (4.6)
2343 (1.8)
3415 (5.0)
-17 (0.4)
SumPooling
7056 (124)
2478 (9.3)
46400 (114)
-6 (0.0)
6130 (4.4)
2470 (10.3)
3660 (1.5)
-16 (1.6)
SelfAtiention
7011 (57.9)
2508 (18.2)
4555 (38.1)
-7 (0.5)
6127 (26.1)
2510 (12.7)
3621 (8.5)
-13 (0.2)
PoE+
6549 (33.2)
2509 (7.8)
4095 (37.2)
-7 (0.2)
5869 (29.6)
2463 (4.3)
3431 (8.3)
-19 (1.7)
SumPooling
6337 (24.0)
2483 (9.8)
3965 (16.9)
-6 (0.2)
5930 (23.8)
2468 (16.8)
3491 (18.3)
-7 (0.1)
SelEAtiention
6662 (20.0)
2516 (8.8)
4247 (31.2)
-6 (0.4)
6716 (21.8) 2430 (26.9)
4282 (49.7)
-27 (1.1)
and M 5. We consider semi-supervised setting where modalities are missing completely at random, as in Zhang et al. (2019), with a missing rate 1) as the sample average of 1 that using the new variational objective improves the LLH and the identifiability of the latent representation. Furthermore, using learnable aggregation schemes benefits both variational objectives.
LM Exem(1 M,). Table shows
5.3 MNIST-SVHN-Ten
Following previous work (Sutter et al. 2020 2021; Javaloy et al. 2022), we consider tri-modal dataset based augmenting the MNIST-SVHN dataset (Shi et al. 2019) with text-based modality. Herein, SVHN con- sists of relatively noisy images, whilst MNIST and text are clearer modalities. Multi-modal VAFs have been shown to exhibit differing performances relative to their multi-modal coherence, latent classification accuracy or test LLH, see Appendix]] [ for definitions. Previous works often differ in their hyperparameters, from neural network architectures, latent space dimensions, priors and likelihood families, likelihood weight- ings, decoder variances, etc. We have chosen the same hyperparameters for all models, thereby providing a clearer disentanglement of how either the variational objective or the aggregation scheme affects different multi-modal evaluation measures. In particular, we consider multi-modal generative models with (i) shared latent variables and (ii) private and shared latent variables. We also consider PoE MoE schemes (denoted PoE+, resp., MoE+) with additional neural network layers in their modality-specific encoding functions so that the number of parameters matches or exceeds those of the introduced PI models, see Appendix [M.5 for details. For models without private latent variables, estimates of the test LLHs in Table suggest that our bound improves the LLH across different aggregation schemes for all modalitics and different |3s (Table 7. with similar results for PE schemes, except for a Self-Attention model. More flexible fusion schemes yield higher LLHs for both bounds. Qualitative results for the reconstructed modalities are given in Figures
21

---

Published in Transactions on Machine Learning Research (09/2024)
with shared latent variables, in Figure 10 for different 8-hyperparametenss and in Figure 11 for models with private latent variables. Cross-generation of the SVHN modality is challenging for the mixture-based bound with all aggregation schemes. In contrast, our bound, particularly when combined with learnable aggregation schemes, leads to more realistic samples of the cross-generated SVHN modality. No variational objective or aggregation scheme performs best across all modalities by the generalive coherence measures (see Table for uni-modal inputs, Table for bi-modal ones and Tables 12 for models with private latent variables and different Bs), along with reported results from external baselines (MVAE, MMVAE, MoPoE, MMJSD, MVTCAE). Overall, our objective is slightly more coherent for generating SVHN Text, but less coherent for MNIST. The mixture-based bound tends to improve the unsupervised latent classification accuracy across different fusion approaches and modalities, see Table To provide complementary insights into the trade-ofls for the different objectives and fusion schemes, we consider a multi-modal rale-distortion evaluation Figure [ Ignoring MoE where reconstructions are similar, our bound improves the full recon- struction with higher full rates and across various fusion schemes. The mixture-bass bound yiclds improved cross- cross-prediciee presspreeistions for all aggregalion models, with increased cross-rate terms. Flexible PI architectures for our bound improve the full reconstruction, even at lower full rates.
/
/
/
i
2
7
7
7
7
6
6
8
6
6
6
6
6
6
2
4
4
4
4
/
7
F
1
PMM
7
9
FIRE
5
V
5
6
2
3
2
2
2
/
NNF
Q
Hume
4/
eRC
IRC
one
ONe
esC
Seve
7
7
7
SEVE
2
(a) Proposed objective
(b) Mixture- based bound
Figure 4: Conditional generation for different aggregation schemes and bounds and shared latent variables. The first column the conditioned modality. The next three columns are the generated modalities using a SumPooling aggregation, followed by the three columns for a SelfAttention aggregation, followed by PoE+, and lastly MoE+.
5.4 Summary of experimental results
We presented a series of numerical experiments that illustrate the benefits of learning more flexible aggre- gation models and that optimizing our variational objective leads to higher log-likeldoood values. Overall, find that for given choice of aggregation scheme, our objective achieves higher log-likelihood across the different experiments. Likewise, fixing the variational objective, Attention encoders achieve higher multi-modal log-likclihoods compared to MoE or PoE schemes. Morcover, we demonstrate that our variational objective results in models that diller in their information theoretic quantities compared to those models trained with a mixture-based bound. In particular, our variational ob- jective achieves higher full-reconstruction terms with higher full rates across different data sets, aggregation schemes, and beta values. Conversely, the mixture-base bound improves the predictto while having higher cross-rate terms.
observe that Sum-Pooling or Self-
22

---

Published in Transactions on Machine Learning Research (09/2024)
Table 6: Conditional coherence with shared latent variables and uni-modal inputs. The letters on the second line represent the generated modality based on the input modalities on the line below it.
Proposed objective
Mixture bound
M
Aggregation
T
S
M
T
PoE
0.97
0.83
PoE+
0.97
0.15
0.63
0.24
0.63
0.42
0.79
0.35
1.00
0.96
0.53
0.99
0.11
0.59
0.11
0.45
0.39
1.00
MoE
0.96
0.80
0.99
0.11
0.59
0.11
0.11
0.37
1.00
0.91
0.81
0.97
0.10
0.51
0.10
0.45
0.39
1.00
MoE++
0.93
0.77
0.95
0.11
0.54
0.10
0.44
0.33
0.98
0.91
0.80
0.98
0.10
0.53
0.10
0.45
0.39
1.00
SumPooling
0.97
0.25
0.36
0.86
0.10
0.45
SelfAttention
0.97
0.44
0.79
0.20
0.71
0.36
0.61
0.43
1.00
0.97
0.86
0.99
0.10
0.63
0.11
0.45
0.40
1.00
Results from [Sutter
2022
Sotttr et al.
2020
and Hwang et al.
2021
MVARE
0.43
0.30
MMVAE
NA
0.75
0.99
0.31
NA
0.30
0.76
NA
MoPoE
NA
0.74
0.99
0.36
NA
0.34
0.96
0.76
NA
MMJSSD
NA
0.82
0.99
0.37
NA
0.36
0.97
0.83
NA
MVTCAE (w/o
NA
NA
NA
Nassed Misiow
0000
-0000
Full
-5000
-2020
-25000
-3000
16.6
Palz
Maxt
Puut
Betttention
BansFraling
Assmaation
Apgreaston
(a) Full Reconstruction DM
(b) Cross Prediction D
Masked
Macked Makure
Malure
10
Raise
Cross
In
MdEf
Pddx
Selttanion
Buntteding
Mdll
Podr
Betktteiion
Banthoudng
Acgrepation
Acg Agrogation
(c) Full Rates RM
(d) Cross Rates Ris
Figure 5: Rate and distortion terms for MNIST-SVHNN-Te with shared latent variables (3 1) for proposed objective ("Masked') and the 'Mixture' based bound.
6 Conclusion
Limitations. A drawback of our bound that computing a gradient step more expensive it requires drawing samples from two encoding distributions. Similarly, learning aggregation functions compu-
23

---

Published in Transactions on Machine Learning Research (09/2024)
tationally expensive compared to fixed schemes. Mixture-based bounds might be preferred if one is interested primarily in cross-modal reconstructions.
Outlook. Using mopality-specifie encoders to learn features and aggregating them with PI function is clearly not the only choice for building multi-modal encoding distributions. However, it allows us to utilize modality-specific architectures for the encoding functions. Alternatively, our bounds could also be used, c.g., when multi-modal transformer architectures (Xu et al., 2022 encode multiple modalities with modality- specific tokenization and embeddings onto a shared latent space. Our approach applies to general prior densities if can compute its cross-entropy relative to the multi-modal encoding distributions. An example would be to apply it with more flexible prior distributions, e.g., specified via score-based diffusion models (Vahdat et al. 2021 Likewise, diffusion models could be utilized to specify PT conditional prior distribution
in the conditional bound by utilizing permutation-cqqiiariinn score models (Dutordoir et al. 2023 Yim
et al. 2023
Mathieu et al. 2023
Acknowledgments
This work is supported by funding from the Wellcome Leap 1kD Program and by the RIE2025 Human Po- tential Programme Prenatal/Eari Childhood Grant (H22POM0002), administered by A*STAR. The com- putational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (https://www nwc.c sg).
References
A. F. Abbas. A Kullback-Leibler view of linear and log-linear pools. Decision Analysis, 6(1):25 000225-7777 2009.
S. Akaho. A kernel method for Canonical Correlation Analysis. In International Meeting of Psychometric Society, 2001, 2001.
A. Alemi, B. Poolc, I. Fischer, J. Dillon, R. A. Saurous, and K. Murphy. Fixing a broken ELBO. International conference on machine learning, pages 159 168. PMLR, 2018.
A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep Variational Information Bottleneck. arXio preprint arXic:1612.00.4 00411, 2016.
E. S. Allman, C. Matias, and J. A. Rhodes. Identifiability of parameters in latent structure models with many observed variables. The Annals of Statistics, 37(6A):3009 3132, 2009.
C. Archambeau and F. Bach. Sparse probabilistic projections. Advances in neural information processing systems, 21, 2008.
R. Argelaguet, B. Velten, D. Arnol, S. Dietrich, T. Zenz, J. C. Marioni, F. Buettner, W. Huber, and O. Stegle. Multi-Omics Factor Analysis framework for unsupervised integration of multi-omics data sets. Molecular systems biology, 14(6):88124, 2018.
J. L. Ba, R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.064500 2016.
F. Bach and M. Jordan. A Probabilistic Interpretation of Canonical Correlation Analysis. 2005.
D. Bahdanau, K. Cho, and Y. Bengio. Neural machine (ranslation by jointly learning align and translate. arXiv preprint LeXii=1409. 0473, 2014.
F. Bao, S. Nie, K. Xue, C. Li, S. Pu, Y. Wang, G. Yue, Y. Cao, H. Su, and J. Zhu. One transformer fits all distributions in multi-modal diffusion at scale. In International Conference on Machine Learning, pages 1692 1717. PMLR, 2023.
Barber and Agakov. The IM Algorithm: a variational approach Information Maximization. Advances in neural information processing systems, 16(320):201, 2004.
24

---

Published in Transactions on Machine Learning Research (09/2024)
S. Bartunov, F. B. Fuchs, and T. P. Lillicrap. Equilibrium aggregation: Encoding sets via optimization. In Uncertainly in Arlificial Intelligence pages 139-149. PMLR, 2022.
M. Biloš and S. Günnemann. Scalable ecalable normalizing flows for permutation invariant densities. In International Conference Machine Learning, pages 957-967. PMLR, 2021.
D. M. Blei, A. Kucukelbir, and D. McAuliffe. Variational inference: review for statisticians. Journal of the American Statistical Association, 112(518):859 877, 2017.
B. Bloem- Bloem-Reddy and Y. W. Teh. Probabilistic symmetries and invariant neural networks. J. Mach. Learn. Res., 21:90-1, 2020.
M. Bounoua, G. Franzese, and P. Michiardi. Multi-modal latent diffusion. ar preprint 23w::3300004445 2023.
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. Van- derPlas, S. Handerman-Milng and Q. Zhang. JAX: composable transformations of Python-Numly programs, 2018. URL http://githubtl com goo///go
M. Browne. Factor analysis of multiple batterics by maximum likelihood. British Journal of Mathematical and Stalistical Psychology, 1980.
A. Bruno, J. Willette, J. Lee, and S. J. Hwang. Mini-batch consistent slot set encoder for scalable set encoding. Advances in Newral Information Processing Systems,
2021.
S. Cao. Choose transformer: Fourier or Galerkin. Advances in neural information processing systems, 34: 24924-2444 24940, 2021.
S. Chatterjee, P. Diaconis, al. The sample size required in importance sampling. The Annals of Applied Probability, 28(2):1099 1135, 2018
Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing deep multitask networks. International conference on machine learning, pages 794 -03. PMLR, 2018.
J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pages 2980 2988, 2015.
B. Dai and D. Wipf. Diagnosing and enhancing vae models. In International Conference on Learning Representations, 2018
B. Dai, Y. Wang, J. Aston, G. Hua, and D. Wipf. Connections with robust PCA and the rolc of emergent sparsity in variational autoencoder models. The Journal of Machine Learning Research, 19(1):1573 1614, 2018.
I. Daunhawer, T. M. Sutter, K. Chin-Cheong, E. Palumbo, and J. E. Vogt. On the Limitations of Multimodal VAEs. In International Conference on Learning Representations, 2022.
I. Daunhawer, A. Bizeul, E. Palumbo, A. Marx, and J. E. Vogt. Identifiability results for multimodal contrastive learning. arXiv preprint arXim:230800916 2023.
P. Diaconis and D. Freedman. Finite exchangeable sequences. The Annals of Probubilily, pages 745 764, 1980.
A. B. Dieng, Y. Kim, M. Rush, and D. M. Blei. Avoiding latent variable collapse with generative skip models. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2397 2405. PMLR, 2019.
N. Dilokthanakul, P. A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and M. Shana- han. Deep unsupervised clustering with Gaussian Mixture Variational Autoencoders. arXio preprint arXiv:161 16.64,26488 2016.
25

---

Published in Transactions on Machine Learning Research (09/2024)
V. Dutordoir, A. Saul, Z. Ghahramani, and F. Simpson. Neural diffusion processes. In International Con- ference on Machine Learning, pages -090-9012. PMLR, 2023.
H. Edwards and A. Storkey. Towards a neural statistician. arXiv preprint arXiv:1006.0211 2016.
F. Falck, H. Zhang, M. Willetts, G. Nicholson, C. Yau, and C. C. Holmes. Multi-facet clustering Variational Autoencoders. Advances in Neural Information Processing Systems, 34:8676 8690, 2021.
M. Figurnov, S. Mohamed, and A. Mnih. Implicit reparameterization gradients. In Advances in Neural Information Processing Systems, pages 441-452, 2018.
J. Fliege and B. F. Svaiter. Steepest descent methods for multicriteria optimization. Mathematical methods of operalions research, 51:479-499 -444, 2000.
A. Foong, W. Bruinsma, J. Gordon, Y. Dubois, J. Requeima, and R. Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Pro- cessing Systems, 33:8284-8295, 2020.
S. Gao, R. Brekelmans, G. Ver Steeg, and A. Galstyan. Auto-encoding total correlation explanation. In The 22nd International Conference on Artificial Intelligence and Statislics, pages 1 1157- 1167-1166. PMLR, 2019.
M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. Rezende, and S. A. Eslami. Conditional neural processes. In International conference machine learning, pages 1704 1713. PMLR, 2018a.
M. Garnelo, Schwarz, D. Rosenbaum, F. Viola, D. Rezende, S. Eslami, and Y. W. Tch. Neural processes. arXiv preprint arXim:1807.01222 2018b.
C. Genest and J. V. Zidek. Combining probability distribution:: A critique and an annotated bibliography. Statistical Science, (1):1144-33, 1986.
C. Genest, K. J. McConway, and M. J. Schervish. Charactcrization of externally Bayesian pooling operators. The Annals of Slabistics, pages 487 501, 1986.
S. Ghalebikesabi, R. Cornish, L. J. Kelly, and C. Holmes. Deep generative pattern-sel mixture models for nonignorable missingness. arXiv preprint orXiv:2103.03522 2021.
G. Giannone and O. Winther. Scha-vae: Hierarchical context aggregation for few-shot generation. In International Conference on Machine Learning, pages 7550-7569. PMLR, 2022.
Y. Gong, H. Hajimirsadoghi, J. Hc, T. Durand, and G. Mori. Variational selective autoencoder: Learning from partially-observed heterogeneous data. In International Conference on Artificial Intelligence and Statisties, pages 2377-2383 PMLR, 2021.
I. Goodfellow, J. Bouget-Abadit M. Mirza, B. Xu, D. Warde-Farles S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A Kernel Method for the Two-Sample- Problem. Advances in neural information processing systems, 19, 2006.
H. Hälvä and A. Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncerlainty in Artificial Intelligence, pages 939 1999488.7 PMT.R, 2020.
H. Hälvä, S. Le Corff, L. Lehéricy, J. So, Y. Zhu, E. Gassiat, and A. Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ICA. Advances in Neural Information Processing Systems, 34:1624-1633, 2021.
D. R. Hardoon, S. Szedmak, and Shawe- Taylor. Canonical correlation analysis: An overview with appli- cation to learning methods. Neural compulation, 16(12):2639 2664,, 2004.
26

---

Published in Transactions on Machine Learning Research (09/2024)
K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In Computer Vision- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Oclober 11-14, 2016, Proceedings, Part IV 14, pages 630 645. Springer, 2016.
K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked Autoencoders are Scalable Vision Learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognation, pages 16000 16009, 2022.
J. Heek, Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL
L. B. Hewitt, M. I. Nye, A. Gane, T. Jaakkola, and J. B. Tenenbaum. The variational homoencoder: Learning to learn high capacity generative models from few examples. arXiv preprint arXiv:1877 08919, 2018.
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. 8-VAE: Learning basic visual concepts with constrained variational framework. In International conference learning representations, 2017.
M. Hoffman and M. Johnson. ELBO surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approzimate Bayesion Inference, NIPS, 2016.
P. Holderrieth, M. J. Hutchinson, and Y. W. Teh. Equivariant learning of stochastic fields: Gaussian processes and steerable conditional neural processes. In International Conference on Machine Learning, pages 4297-4307. PMLR, 2021.
H. Hotelling. Relations between two sets of variates. Biometrika, 28(3/4):321-377 1936.
S. Huang, A. Makhzani, Y. Cao, and R. Grosse. Evaluating lossy compression rates of deep generative models. Xixio preprint arXiv:2008.0660 06658,, 2020.
Y. Huang, J. Lin, C. Zhou, H. Yang, and L. Huang. Modality competition: What makes joint training of multi-modal network fail in deep cearning((provabll) arXiv preprinl arxiv:2003. 12221, 2022.
Hwang, G.-H. Kim, S. Hong, and K.-E. Kim. Multi-view representation learning via total correlation objective. Advances Neural Information Processing Systems, 34:12194-12201 12207,2277772 2021.
A. Hyvarinen and H. Morioka, Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in neural information processing systems, 29, 2016.
A. Hyvärinen and P. Pajunen. Nonlinear Independent Component Analysis: Existence and uniqueness results. Newral networks, 12(3):429 439, 1999.
B. Ipsen, P.A. Mattei, and Frellsen. not-MIWAE: Deep Generative Modelling with Missing not at Random Data. In ICLR Internaatra Conference on Learning Representations, 2021.
E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. arXio preprint arXiv:1611.01144 2016.
A. Javaloy, M. Meghdadi, and I. Valera. Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization. arXiv preprint ar 2rxi::22060044 04496, 2022.
Z. Jiang, Y. Zheng, H. Tan, B. Tang, and H. Zhou. Variational deep embedding: an unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 1965-1972, 2017.
M. J. Johnson, D. Duvenaud, A. B. Wiltschko, S. R. Datta, and R. P. Adams. Structured vacs: Composing probabilistic graphical models and variational autoencoders. arXiv preprint 1rXx:::603.0277, 2016.
M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction variational methods for graphical models. Machine learning, 37(2):183 233, 1999.
27

---

Published in Transactions on Machine Learning Research (09/2024)
Т. Joy, Y. Shi, P. H. Torr, T. Rainforth, S. M. Schmon, and N. Siddharth. Learning multimodal VAEs through mutual supervision. arviv preprint or arXin:2106.12 10061.257, 2021.
M. Karami and D. Schuurmans. Deep probabilistic canonical correlation analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5055-8063, 2021.
Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational Autocncoders and nonlinear ICA: A unifying framework. International Conference on Artificial Intelligence and Statistics, pages 2207 2217. PMLR, 2020a.
I. Khemakhem, R. Monti, D. Kingma, and A. Hyvarinen. ICE-BeeM: Identifiable Conditional Energy- Based Deep Models Based on Nonlinear ICA. Advances in Neural Information Processing Systems, 33: 12768 12778, 2020b.
H. Kim, A. Mnih, J. Schwarz, M. Garnelo, A. Eslami, D. Rosenbaum, O. Vinyals, and Y. W. Teh. Attentive neural processes. International Conference on Learning Representations, 2018.
J. Kim, J. Yoo, J. Lec, and S. Hong. Setvac: Learning hierarchical composition for generative modcling of sel-structured data. In Proceedings of the JEEE/CVP Conference on Computer Vision and Paltern Recognition, pages 15059 15068,, 2021.
Y.-g. Kim, Y. Liu, and X.-X. Wei. Covariate-infommed Representation Learning to Prevent Posterior Collapse of iVAE. In International Conference Artificial Intelligence and Statistics, pages 2641-2660. PMLR, 2023.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.600 2014.
B. Kivva, G. Rajendran, P. K. Ravikumar, and B. Aragam. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 2022.
Klami, S. Virtanen, and Kaski. Bayesian canonical correlation analysis. Journal of Machine Learning Research, 14(4), 2013.
L. Kong, M. Q. Ma, G. Chen, E. P. Xing, Chi, L.-P. Morency, and K. Zhang. Understanding masked autocncoders via hicrarchical latent variable models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7918 7928, 2023.
X. Kong and X. Zhang. Understanding masked image modeling via learning occlusion invariant feature. In Proceedings of the IEFE/CVP Conference on Computer Vision and Pattern Recognition, pages 2023.
W. Kool, H. van Hoof, and M. Welling, Buy reinforce samples, get baseline for free! 2019.
T. C. Koopmans and O. Reiersol. The identification of structural characteristics. The Annols of Mathematical Statisties, 21(2):165 11111111-1111---1111 1950.
D. Kramer, P. T.. Bommer, D. Durstewitz, C. Tombolini, and G. Koppe. Reconstructing nonlinear dynamical systems from multi-modal time series. In International Conference on Machine Learning, pages 11613 11633. PMLR, 2022.
J. B. Kruskal. More factors than subjects, tests and treatments: An indeterminacy theorem for canonical decompasition and individual differences scaling. Psychometrika, 41 (1):22811221 1976.
T. A. Le, H. Kim, M. Garnelo, D. Rosenbaum, Schwarz, and Y. W. Teh. Empirical evaluation of neural process objectives. In NeurIPS workshop on Bayesian Deep Learning, volume 4, 2018.
C. Loe and M. van der Schaar. A variational information bottleneck approach multi-omics data integration. In International Conference on Artificial Intelligence and Slatistics, pages 1513 1521. PMI.R, 2021.
28

---

Published in Transactions on Machine Learning Research (09/2024)
J. Lee, Y. Lee, J. Kim, A. Kosiorck, S. Choi, and Y. W. Teh. Set Transformer: A framework for attention- based permutation-invariant neural networks. In International conference on machine learning, pages 3744 3753. PMLR, 2019.
M. Lee and V. Pavlovic. Private-shared disentangled multimodal vae for learning of latent representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1692 1700, 2021.
C.-L. Li, M. Zaheer, Y. Zhang, B. Poczos, and R. Salakhutdinov. Point cloud GAN. ar preprinl arXiv: 05795, 2018.
Q. Li, T. Lin, and Z. Shen. Deep neural network approximation of invariant functions through dynamical systems. arXiv preprint ar 228::2208000007,, 2022.
Y. Li and Oliva. Partially observed exchangeable modeling. International Conference on Machine Learning, pages 6460-6470. PMLR, 2021.
Y. Li, H. Yi, C. Bender, S. Shan, and J. B. Oliva. Exchangeable neural ode for set modeling. Advances in 2020.
Newral Information Processing Systems,
Linsker. Self-organization a perceptual network. Computer, (1):111 117, 1988.
G. Loaiza-Ganem B. L. Ross, J. C. Cresswell, and A. L. Caterini. Diagnosing and Fixing Manifold Overfitting in Deep Generative Models. Transactions on Machine Learning Research, 2022.
C. Lu, Y. WU, J. M. Hernández-Lobaile and B. Schölkopf. Invariant causal representation learning for distr--disti generalization. In International Conference on Learning Representations, 2022.
J. Lucas, G. Tucker, R. B. Grosse, and M. Norouzi. Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse, In Advances in Newral Information Processing Systems, pages 9408 9418, 2019.
Q Lyu and X. Fu. Finito-sample analysis of deep CCA-based unsupervised post-nonlincar multimodal learning. IEEE Transactions on Neural Nelworks and Learning Systems, 2022.
Q Lyu, X. Fu, W. Wang, and S. Lu. Understanding latent correlation-based multiview learning and self- supervision: An identifiability perspective. Xiv preprint 1=Xiu:2106.011 2021.
C. Ma, S. Tschiatschek, K. Palla, J. M. Hernandez-Lobato, S. Nowozin, and C. Zhang. EDDI: Efficient Dynamic Discovery of High- Value Information with Partial VAE. In International Conference Machine Learning, pages 4234 4243. PMLR, 2019.
C. Maddison, Mnih, and Y. W. Teh. The concrete distribution: continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00719 2016.
A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey. Adversarial Autoencoders. In ICLR, 2016.
H. Maron, E. Fetaya, N. Segol, and Y. Lipman. On the universality of invariant networks. In International conference on machine learning, pages 4363 4371. PMIR, 2019.
E. Mathieu, T. Rainforth, N. Siddharth, and Y. W. Teh. Disentangling disentanglement in Variational Autocncoders. In International Conference on Machine Learning, pages 4402-4412. PMLR, 2019.
E. Mathieu, V. Dutordoir, M. Hutchinson, V. De Bortoli, Y. W. Teh, and R. Turner. Geometric Neural Diffusion Processes. Advances in Neural Information Processing Systems, 37, 2023.
K. Minoura, K. Abe, H. Nam, H. Nishikawa, and T. Shimamura. A mixture--f-cxxeets deep generative model for integrated analysis of single-cell multionics data. Cell reports methods, 1(5):10007,, 2021.
G. Mita, M. Filippone, and P. Michiardi. An identifiable double VAE for disentangled representations. International Conference on Machine Learning, pages 7769-77 -779. PMT.R, 2021.
29

---

Published in Transactions on Machine Learning Research (09/2024)
G. E. Moran, D. Sridhar, Y. Wang, and D. M. Blei. Identifiable deep generative models via sparse decoding. arXiv preprint or Xr::21110. 10804, 2021.
W. Morningstar, S. Vikram, C. Ham, A. Gallagher, and J. Dillon. Automatic differentiation variational inference with mixtures. In International Conference on Artificial Intelligence and Statistics, pages 3250- 3258. PMLR, 2021.
R. Murphy, B. Srinivasan, V. Rao, and B. Riberio. Janossy pooling: Learning deep permutation-invariie functions for variable-size inputs. In International Conference Learning Representations (ICLR 2019). 2019.
A. Nazabal, P. M. Olmos, Z. Ghahramani, and I. Valera. Handling incomplete heterogeneous data using VAEs. Pattern Recognition, 107:107501, 2020.
v. d. Oord, Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint trXi::1807.003 08748, 2018.
E. Palumbo, I. Daunhawer, and J. E. Vogt. Mmvae+: Enhancing the generative quality of multimodal vaes without compromises. Tn The Fleventh International Conference on Learning Representations, 2023.
E. Palumbo, L. Manduchi, S. Laguna, D. Chopard, and J. E. Vogt. Deep Generative Clustering with Multimodal Diffusion Variational Autoencoders. In The Twelfth International Conference on Learning Representations, 2024.
K. Pandey, A. Mukherjee, P. Rai, and A. Kumar. Diffusevae: Efficient, controllable and high-fidelity gener- ation from low-dimensional latents. Transactions on Machine Learning Research, 2022.
B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker. On variational bounds of mutual information.
In International Conference on Machine Learning, pages 5171-5180. PMLR, 2019.
C. R. Qi, H. Su, K. Mo, and L. J. Guíbas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on compuler vision and pallern recognition, pages 552-660,220 2017.
R. Ranganath, S. Gerrish, and D. M. Blei. Black box variational inference. In AISTATS, pages 814-822, 2014.
J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference Machine Learning (ICMT--14). pages 1278 1286, 2014.
H. E. Robbins. An empirical Bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pages 388 394. Springer, 1992.
G. Roeder, Y. Wu, and D. Duvenaud. Sticking the landing: An asymptotically zero-variance gradient estimator for variational inference. arXiv preprint arXiv:1703 191,099999, 2017.
M. Rolinck, D. Zietlow, and G. Martius. Variational Autoencoders pursue PCA directions (by accident). Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12406 12415, 2019.
M. Rosca, B. Lakshminarayanan, and S. Mohamed. Distribution matching in variational inference. arXiv preprint Xrx: 06847, 2018
S. T. Roweis and L. K. Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 290 (5500):2323- 2226, 2000.
D. B. Rubin. Inference and missing data. Biomelrika, 63(3):581 -924 1976.
A. Sannai, Y. Takai, and M. Cordonnier. Universal approximations of permutation invariant voniiann//qquvariintt functions by deep neural networks. arXiv preprint 19X0011903.01939, 2019.
30

---

Published in Transactions on Machine Learning Research (09/2024)
A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap. A simple neural network module for relational reasoning. Aduances in neural informalion processing systems, 30, 2017.
S. Schneider, J. H. Lee, and M. W. Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, pages 9, 2023.
N. Segol and Y. Lipman. On universal equivariant set networks. In International Conference on Learning Representations, 2019.
O. Senor and V. Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
Y. Shi, B. Paige, P. Torr, et al. Variational Nixture-of- Autoencoders for Multi-Mooal Deep Gener- alive Models. Advances in Newral Information Processing Systems, 32, 2019.
Y. Shi, B. Paige, P. Torr, and N. Siddharth. Relating by Contrasting: A Data-cfficient Framework for Multimodal Generative Models. In International Conference on Learning Represenlations, 2020.
P. Sorrenson, C. Rother, and U. Köthe. Disentanglement by nonlinear ICA with general incompressible-for networks (GIN). arXiu preprint xxXi::201.04872, 2020.
J. H. Stock and M. W. Watson. Forecasting using principal components from a large number of predictors. Journal of the Americon statistical association, 97(460):1677-1179 167-1179, 2002.
T. Sutter, I. Daunhawer, and J. Vogt. Multimodal generative learning utilizing Jensen Samsen--hhamon-diveregeee Advances in Newral Information Processing Systems, 33:6100-6111 6110, 2020.
T. M. Sutter, 1. Daunhawer, and J. E. Vogt. Generalized multimodal elbo. In 9th International Conference on Learning Representations (ICLR 2021), 2021.
M. Suzuki and Matsuo. Mitigating the Limitations of Multimodal VAEs with Coordination-base Ap- proach. 2022.
M. Suzuki, K. Nakayama, and Y. Matsuo. Joint multimodal learning with deep generative models. arXiv preprint Xrxi::6111.01 0119919,,, 2016.
Y. Tang and D. Ha. The sensory neuron as a transformer: Permutationn-inaaraan neural networks for reinforcement learning. Advances in Neural Information Processing Systems, 01222744-22887 2021.
A. Tenenhaus and M. Tenenhaus. Regularized generalized Canonical Correlation Analysis. Psychometrika, 76:257-288 284, 2011.
Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding, In Computer Vision ECCV 2020: 16th European Conference, Glosgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 776-794. Springer, 2020.
M. E. Tipping and M. Bishop. Probabilistic Principal Component Analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 61(3):611 622, 1999.
M. Titsias and M. Lázaro-Gredilla. Doubly stochastic variational bayes for non-conjugate inference. Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 1971 1979, 2014.
M. K. Titsias, F. J. Ruiz, S. Nikoloutsopoulos, and A. Galashov. Information thcoretic meta learning with gaussian processes. In Uncerlainly in Arlificial Intelligence, pages 1606. PMLR, 2021.
J. M. Tomezak and M. Welling. Vae with vampprior. orXiv preprint arXim:1705.071100 2017.
Y.HH. H. Tsai, Bai, Liang, Kolter, L.-P. Morency, and R. Salakhutdinov. Multimodal trans- former for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, page 6558. NTH Public Access, 2019a.
31

---

Published in Transactions on Machine Learning Research (09/2024)
Y.-H. H. Tsai, P. Liang, A. Zadch, L.-P. Morency, and R. Salakhutdinov. Learning factorized multimodal representations. In International Conference on Representation Learning, 2019b.
Vahdat, K. Kreis, and Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34, 2021.
M. Vasco, H. Yin, F. S. Melo, and A. Paiva. Leveraging hierarchy in multimodal generative models for effective modastmy inference. Neural Networks, 146:238 255, 2022.
A. Vaswani, N. Shazcer, N. Parmar, J. Uszkorcit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Atlention is all you need. Advonces in neural information processing systems, 30, 2017.
R. Vedantam, I. Fischer, J. Huang, and K. Murphy. Generative models of visually grounded imagination.
In International Conference on Learning Represenlations, 2018.
G. Ver Steeg and A. Galstyan. Maximally informative hierarchical representations of high-dimensional data. In Artificial Intelligence and Statistics, pages 1004-1012. PMLR, 2015.
S. Virtanen, A. Klami, S. Khan, and S. Kaski. Bayesian group factor analysis. In Artificial Intelligence and Statistics, pages 1269 1277. PMLR, 2012.
E. Wagstaff, F. B. Fuchs, M. Engeleke, M. A. Osborne, and I. Posner. Universal approximation of functions sets. Journal of Machine Learning Research, 23(151):1 56, 2022.
Q. Wang and H. Van Hoof. Doubly stochastic variational inference for neural processes with hicrarchical latent variables. In International Conference on Machine Learning, pages 10018-10028.. PMLR, 2020.
Q Wang, B. Li, T. Xiao, J. Zhu, C. Li, D. F. Wong, and L. S. Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810 1822, 2019a.
T. Wang and Isola. Understanding contrastive reprosentation learning through alignment and uniformity on the hypersphere. In International Conference Machviee Learning, pages 9929 9939. PMLR, 2020.
W. Wang, R. Arora, K. Livescu, and J. Bilmes. On deep multi-view representation learning. In International conference on machine learning, pages 1092-1092. PMLR, 2015.
W. Wang, X. Yan, H. Lee, and K. Livescu. Deep Variational Canonical Correlation Analysis. arXiv preprinl arXiv:1610. 03454, 2016.
W. Wang, D. Tran, and M. Feiszli. What makes training multi-modal classification networks hard? Tn Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pages 12695 12705, 2020.
Y. Wang, A. C. Miller, and D. M. Blei. Comment: Variational Autoencoders as Empirical Bayes. 2019b.
Y. Wang, D. Blei, and J. P. Cunningham. Posterior collapse and latent variable non-identifiibiiitty. Advances Neural Information Processing Systems, 34:5443-544 5455, 2021.
S. Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of research and development, 4(1):66-82, 1960.
M. A. Wright and J. E. Gonzalez. Transformers are deep infinite-dimensionall non-mercer binary kernel machines. Xivi preprint #xXiv:2106.01500 2021.
M. Wu and N. Goodman. Multimodal generative models for scalable weakly-supervised lcarning. Advances in Neural Information Processing Systems, 31, 2018.
M. Wu and N. Goodman. Multimodal generative models for compositional representation learning. arXiv preprint Xiv:191 15750057005075, 2019.
32

---

Published in Transactions on Machine Learning Research (09/2024)
Q. Xi and B. Bloem-Reddy. Indeterminacy in latent variable models: Characterization and strong identifi- ability. arXiv preprinl ar Xrx::22006. 00801, 2022.
Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu. On layer normalization in the transformer architecture. In International Conference Machine Learning, pages 10524-10533. PMLR, 2020.
J. Xu, E. Dupont, K. Märtens, T. Rainforth, and Y. W. Teh. Deep Stochastic Processes via Functional Markov Transition Operators. Advonces in Neural Information Processing Systems, 37, 2023.
P. Xu, X. Zhu, and D. A. Clifton. Multimodal learning with transformers: A survey. ar preprint 2rXiv:2206.064 2022.
J. Yim, B. L. Trippe, V. De Bortoli, E. Mathicu, A. Doucet, R. Barzilay, and T. Jaakkola. SE (3) diffusion model with application to protein backbone generation. In International Conference Machine Learning, pages 40001 40039. PMLR, 2023.
T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi-task learning. Aduances in Newral Information Processing Systems, 33:5824-5836, 2020.
C. Yun, S. Bhojanapalli, A. Rawat, S. Reddi, and S. Kumar. Are transformers universal approximators of toeueenceto-seeqencce functions? In International Conference on Learning Represenlations, 2019.
M. Zahoer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola. Deep Sets. Advances in neural informalion processing systems, 30, 2017.
C. Zhang, Z. Han, H. Fu, J. T. Zhou, Q. Hu, et al. CPM-Nets: Cross partial multi-view networks. Advances in Neural Information Processing Systems, 32, 2019.
F. Zhang, B. Liu, K. Wang, Y. Tan, Yang, and Z. Wang. Relational Reasoning via Set Transformers: Provable Efficiency and Applications MARL. ar preprint arXiv:2209.0999 09845, 2022a.
L. Zhang, V. Tozzo, J. Higgins, and R. Ranganath. Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Sets. In International Conference on Machine Learning, pages 08559-26574. PMLR, 2022b.
Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pages 2-25. PMLR, 2022c.
S. Zhao, C. Gao, S. Mukherjee, and B. E. Engelhardt. Bayesian group factor analysis with structured sparsity. The Journal of Machine Learning Research, 2016.
S. Zhao, Song, and S. Ermon. VAE: Balancing Learning and Inference Variational Autoencoders. In Proceedings of the asai conference on artificial intelligence, volume 33, pages 5885 5892, 2019.
Y. Zheng, T. He, Y. Qiu, and D. P. Wipf. Learning manifold dimensions with conditional Variational Autoencoders. Advances in Neural Information Processing Systems, 35:34709 34721, 2022.
D. Zhou and X.-X. Wei. Learning identifiable and interpretable latent models of high-dimensional neural activity using pi- VAE. Advances in Neural Information Processing Systems, 33:7234-7247, 2020.
33

---

Published in Transactions on Machine Learning Research (09/2024)
A Multi-modal distribution matching
The equations for (s(s)) are well known for uni-modal VAEs, see for example Zhao et al. (2019) To derive similar representations for the conditional bound, note that the first equation ZX eXconditional for matching the joint distribution of the latent and the missing modalities conditional on a modality subset follows from the definition of Cis,
Proof [Proof of Proposition
f s(s d)dr,
/ palzisles) / (0(2|2) [logr((m(s22) - log (z(x)) log (2(22/)))) dzdays
= f pa(m)sles) log + / pa ((nnsssssss / (=(()) log p=(assl))ga((lss)) dzdmy
H(ps())slss)) KL (co((ll)ppa(mnssssss)) (aass)
To obtain the second representation (xcoditional for matching the conditional distributions in the data space, observe that pa so z) = ( ss 22) and consequently,
f pule 0, (opdoo/--dodd-/-----)ss))))))))) H(pa(z)ss2z))
/ p pallosslas)ngltlllnaaaa dzdm, (
/ p palengglsssoo((tttooc((sssssssss dzday dzdays
f pa(msslos) log
dædays
po (w)
/ lpss + / pa(s)slos)) / (0(2))) log
pa (
log
pe pa Po as s (=(=s) dzdrys
go [z(2)x) + log p=(2 (
dzdæys.
(0(2(z)
Lastly, the representation Zoonditional for matching the distributions in the latent space given modality subset follows by recalling that
= = aud (al
and consequently,
f palzislas) 0)dr H(pa(e(slas)
f pal log pa(mnslas) dzdrys po(o
- | - (s)sla,as) log La (2 (ss (2 s1..
dzdays
po (o
Ko ( (#(0s) (=(0s)) 2 (2 (KL(g" (s,s (x0 dz.
34

---

Published in Transactions on Machine Learning Research (09/2024)
B Meta-learning and Neural processes
Meta-lcarning. We consider a standard meta-lcarning setup but use slightly non-standard notations to remain consistent with notations used in other parts of this work. We consider a compact input or covariate space A and output space L. Let D = (=A x X)M be the collection of all input-output pairs. meta-learning, we are given a meta-dataset, ie., a collection of elements from D. Each individual data set (a.z) = De D, D is called task and split into a context set D. = (ae:Te), and target set D, (z). We aim to predict the larget set from the context set. Consider, therefore, the prediction map
De (Ac: p( dz, Da) p( selar, ac)/ p
mapping each context data set to the predictive stochastic process conditioned De-
Variational lower bounds for Neural processes. Latent Neural processes (Garnelo et al. 2018b Foong
ct al. 2020) approximate this prodiction map by using a latent variable model with parameters 0 in the form of
~ po. po = II
Pe(x fo(a,
(a,x)
for prior pa, decoder fo and parameter free density Pe- The model is then trained by (approximately) maximizing a lower bound on log po (xx ats Te). Note that for an encoding density dø, we have that
/ (0(2|or, a) log aq, z)d2 - (ppoo= ae)
log (Te at, Ocs xzz =
x)
Since the posterior distribution pe p= xe) is generally intractable, one instead replaces it with a variational approximation learned conditional prior (2(2)ae, Ø2), and optimizes the following objective
CLNP P (x, = f (2(2, a) log (e a 2)d2 - (L(q xc)).
Note that this objective coincides with Lie conditioned the indices of the data points that are part of the context set. Using the variational lower bound LLNP can yield subpar performance compared to another biased log- likelihood objective (Kim et al. 2018 Foong et al. 2020),
the covariate values a and where C comprises
log po (F Ger xc)= =log
exp
Σ
logpo(zelae 2/)
for L importance samples 2 (0(=el=ee ae) drawn from the conditional prior as the proposal distribution. The required number of importance samples for accurate estimation scales exponentially in the forward KL(gs(zlzz (>)g=(2/<z, ac)), see Chatterjee et al. (2018). Unlike a variational approach, such estimator does not enforce Bayes-consistence term for the encoders and may be beneficial in the setting of finite data and model capacity. Note that the Bayes consistency term for including the target set (Ix, az) into the context set (Iz: 0z) writes as
ae)),
R88 (L ar) (c)laa az)) KL (f Re ac) a)dr, go 20(e)1es ae))
Moreover, if one wants to optimize not only the conditional but also the marginal distributions, may additionally optimize the variational objective corresponding to Le, i.e.,
CLNP ar) = go (2|ore a.) log po 2)dz KL(gs(2la., xc)loo
as we do in this work for multi-modal generative models. Note that the objective CP alone can be seen form of Neural Statistician model (Fdwards and Storkey 2016) where C coincides with the indices of
35

---

Published in Transactions on Machine Learning Research (09/2024)
the target set, while form of the mixture-based bound corresponds to Neural process bound similar to
variational Homoeneoders (Hewitt et al. 2018), see also the discussion in Te et al. (2018). The multi-view
variational information bottleneck approach developed in Lee and van der Schaar 2021) for predicting Xis given Xs involves the joint variational objective
(r. - f
- a (z|2s)) f
log (= - (KL(gg())
which can be interpreted as maximizing qo (X s. Zs) information bottleneck for meta-learning in Titsias et al. (0211)
-stub (Xs, Zs) and corresponds to the variational
96
C Information-theoreti perspective
We recall first that the mutual information given by
In (Xs, Zs) - - / 28) log
go =s)
dasdrs,
"82
(2 is the aggregated prior Makhzani et al. 2016 It can be bounded by where 4 (25(()) / pa(xs)9g(())) standard (Brbbr and Agakov 2004 Alemi el al. 2016 2018 lower and upper bounds using the rate and distortion:
Hss Ds S Hs Ds + A Ino (Xs, Zs) = Rs 2 S Rs,
(15)
with A1 = l (=) (rsl=))po(zsss))) (o(oss 2)/40 (2).
0, A2
> 0 and (~ss)>)
Morcover, if the bounds in 7 become tight with A A2 = 0 in the hypothetical scenario of infinite- capacity decoders and encoders, one obtains S PaCs = (1-5) Iga (Xs. Zs) + Hs. For 3 > maximizing Ls yields an auto-decoding limit that minimnizes I,p (xs, =) for which the latent representations do not encode any information about the data, whilst B 1 yields an auto-encoding limit that maximizes Lao (Xs, and for which the data is perfectly encoded and decoded.
The mixture-basen bound can be interpreted the maximization of variational lower bound of L, (XM; Zs) and the minimization of a variational upper bound of Lus (Xs. Zs). Indeed, see also Daunhawer al. (2022
HM Ds - Dg S HM Ds D3 + A Ilc (XM.
where A = 48 (=)3 > 0, due to
Ino (XM, 2s) = Hm - Hap (X|Zs) = HM + / pa(=)) [log (x[=)] dædæ
= =HM + pa(x))a(=((ss) +Hm+fnalwoooteel
[oggo(rs)>) + logpo(s,ssl + log (x|x) dzdr.
Recalling that
f pa(ds) (o(dx)CC) C =- Di -3Rs,
one can see that maximizing the first part of the mixture-based variational bound corresponds to maximizing - Ds D as a variational lower bound of Las (XM, Es), when ignoring the fixed entropy the multi-modal data. Maximizing the sccond part of the masodd variational bound corresponds minimizing Rg as variational upper bound of Typ (Xs: Zs); see 115
Proof Proof of Lemma 7 The proof follows by adapting the arguments in Alemi et al. 2018 The law of
X,s and Z conditional on Xs on the encoder path can be written as
4 (( pe(c),sass = (alas)))) (21s),,
36

---

Published in Transactions on Machine Learning Research (09/2024)
with 9" (2)s12, g0l2: To prove lower bound the conditional mutual informa- tion, note that
1 Zu|xs)
=] |Jaales) /
aBs (2les()) / q- (1)s)2, log Lo (2 Ladd,gdas 1
nsse =fnalss) palus)) / o (2k2)
If (2))) (o + KL ( assllo dzdes - - | / paleysles)) p las)dr
- pala)) / (oge))
[ palos) /
=
(og las)dr
- H(Xxol1s))
pa(as) / 2eE (ongs poll,s)
=Ai + Dist H.ss
The upper bound follows by observing that
Tas (X Zm)Xs)
= |Pales) / pulosslas) log
g0(2 x (p(pss)) dædæ (s (1n (as)
= -f /ate:/kicakkklllllaltssssss pu(xs))LL
aRR (z
18,220
= R s.2.-
Remark 17 (Total correlation based objectives) The objective suggested in Hwang et al. (2021) is motivated by conditional variational bottleneck perspective that aims to maximize the reduction of total correlation when conditioned on Z, as measured by the conditional total correlation, see Watanabe (9600)
Ver Steeg and Galstyan (2015); Gao et al. (2019) i.e.,
TC(X) minimizing [TC(X]Z)= =CC))) -C(XX 2) TC(X) + I (X,
Σ Ta (X,, 2)
(16)
where TC(X) (L(P()) p(x))) for d-dimensional X. Resorting to variational lower bounds and using constant B > 0 that weights the contributions of the mutual information terms, approximations of (16
can be optimized by maximizing
(0,60,))) - p(S)
[log po (x]=)] dz (KL((g((l<<) ds,
where p is concentrated the uni-modal subsets of M.
Remark 18 (Entropy regularised optimization) Let q be density over C, exp(g) be integrable with respect to and 0. The maximum of
f(g) / g(c) (ø(c) - log ((c) de
37

---

Published in Transactions on Machine Learning Research (09/2024)
that is attained at (c) c with normalizing constant Z co de
= =olog eole)/T de.
Remark 19 (Optimal variational distribution) The optimal variational density for the mixture-basee multi-modal objective,
f (5 (x) = 1f pa(us) / (o(2)x)) /
palayslas)
(oog)( log (o)(sss))))) + (log 3 log (=(()=))) day dd
using Remark
18
attained at
q (2(as) exp g puloyssass) logpo(zs))) + log (e( + 3 log po() dryss)
(3,0(21a5) (f pa(essless) log (og
Permutation-invariant architectures
Multi- Multi-head attention and masking. We introduce here a standard multi-head attention
Bahdanan
ct al.
2014 Vaswani et al. 2017) mapping MHAa: RIxDx x RSDDY Rx given by
MHAa(X, Y) = wo [Heead(XX,,,)) YX, Head # (x, YX),,)) 0 (Wq, WK, Wv.
with output matrix Wo DADy projection matrices Wq E R DxxDa WK, Wy E DyxDa and
Head (Q, (Q, V) Att(Qw)) KW VW)) IID
(17)
where we assume that D = DA/H N is the head size. Here, the dot dot-product attention function is K,) a(QK
where a the softmax function applied to each column of QKT.
Masked multi-head attention. In practice is convenient to consider masked multi-head attention models MMHA,,M: Rxxxx Ty Rx for mask matrix M {0, 1) that operate on key value sequences of fixed length T' where the h-th head (17 is given by
Head" (Q, (, = [M ((W J)] V. RTXD
Using the softmax kernel function SMp(q, = exp(a TopplT////l k/vD),
we set
T
Σ X X,, MX Y.W
"
MMHA (X,Y),
kWW.w
(18)
ErS1
which does not depend on Y, if M. =
Masked self-attention. For mask matrix M = mm T with m (1(sES))sEM we write
MHAa Ys) Mi)s)
where MMHAgAM A operates sequences with fixed length and i(Ys)); Y, if E S and otherwise.
38

---

Published in Transactions on Machine Learning Research (09/2024)
LayerNorm and SetNorm. Let h RTxD and consider the normalization
N(h) h h- (h) 07+3
where i and standardize the input h by computing the mean, and the variance, respectively, over some axis of h, whilst y and define transformation. LayerNorm (Ba et al. 2016) standardises inputs over the last axis, e.g., p(h) D Ed= pr,d, i.., separately for each element. In contrast, SetNorm (Zhang et al, 2022b) standardises imputs over both axes, e-g., u(h) TO E Σd=1 Me,d, thereby losing the global mean and variance only. In both cases, y and B share their values across the first axis. Both normalizations are permutation-qquivaiann
Transformer. We consider a masked pre-layer-norm Wang et al. 2019a Xiong el al. transformer block
2020 multi-head
(MMTB,,.((s((ss))s (Z ReLU (LN(2))s
with ReLU being a ReLU non-linearity and
Z = is(ss + (ANH
where M = mmm for m = (1 (1sESS)sEM-
Set-Attention Encoders.
Set 9o = is(xo(hss) and for (l,..., L}, let
g² MMS
Then, we can express the self-attention multi-modal aggregation mapping via fo(hs) = po (Eves gh))
Remark 20 (Context-aware pooling) Assuming a single head for the transformer encoder in Example with head size and projection matrices Wo, WK, Wv RDxxD, the atlention scores for the initial input sequence Is = d = (hs) ) E RRS|x Dr are a(gs, It) = (W Is, W The attention outputs o, E R for can then be written as
02 Σ M(gr.g))u(
tES
where Eies N()..9)) > 0, v(g.) WT gt and
k(gs, 9i) = exp(alge: 9:)) = exp (1
WKg)\/DD
can be seen as Iearnable non-symmetric kernel WWigght and Gonzalez 2021
Cao 2021
Conceptually, the
atlention encoder pools learnable D-dimensional Tunction using learnable context-dependent weighting function. While such attention models directly account for the interaction between the different encodings. DeepSet aggregation approach may require sufficiently high-dimensional latent space Dp to achieve universal approximation properties /Wagstaff et al. 2022)
Remark 21 (Multi-modal time series models) We have introduced multi-modal generative model in general form that also applies to the time-series setup, such when latent Markov process drives multiple time series. For example, consider a latent Markov process Z = (Zi)eN with prior dy- namics po(21,...,,2y 2r) po(2)) (2 for initial density po(21) and homogeneous Markov kernels po(=)(=11). Conditional on Z, suppose that the time-series (X,,)teN follows the dynamics Po 1............ F.. 2T) -Ir (o for decoding densities po(#s,e|24). common choice (Chung et al. 2015) for modeling the encoding distribution for such sequential (uni-modal) VAEs is to assune the factorization Qo(2).... 2T
ØT) = ( (x1) IT (o (2el22-11 21) for I; = (#.,))EM.: with initial encod- ing densitics (z))) and encoding Markov kernels qo (z1/2-11, 2x). One can again consider modality-spooiffi encodings h2 (ha,1....,hb hrr), he (x,,() now applied separately at each time step that are then used to construct Markov kernels that are permutation-invariand in the form of (2(2222 (h)(s))))
12....................... 2t......2/1.............
39

---

Published in Transactions on Machine Learning Research (09/2024)
(e(= (F(Lss) for permutations Sg. Alternatively, in the absence of the auto-regressive encoding structure with Markov kernels, one could also use transformer models that use absolute or relative positional embeddings across the last temporal axis but no positional embeddings across the first modality axis, fol- lowed by sum-pooling operation across the modality axis. Note that previous works using multi-modal time series such as Kramer et al. (2022) use non-amortized encoding distribution for the full multi-modal posterior only. numerical evaluation of permutation-invariant schemes for time series models is, however, outside the scope of this work.
Remark 22 (Alternative multi-modal encoding models) Learning different encoders for each modal- ity subset can be an alternative when the number modalities is small. For example, one could learn different MLP heads for each modality subset S that aggregates the encoded features hş from modality-specift en- coders that are shared for any modality mask. Our initial experiments with such encoders for the simulated linear modalities in Section 5.1 for M = modalities did not improve on the permutation-invarinn models, while also being more computationally demanding.
E Permutation--qquiariancc and private latent variables
Remark 23 (Variational bounds with private latent variables) To compute the multi-modal varia- tional bounds, notice that the required KL divergences can be written as follows:
+ (aseelssssKKoolsss........ (2) KL(as(z', (ps = (L(ga(>/las)loo
and
{lts))
+ f (( (os
where Ps: (ž 2as) + (Es) projects all private latent variables to those contained S.
These expressions can be used to compute our overall variational bound Ls + Lis via
f a0(2' ( as)] logpo 2s) dēs
kK (K (s)
g0(2 co(2'|
(xsk a ss)d''
KL (9ols',
(a0(2',,05 Es. S (m) qo(2z As. {ss
Remark 24 (Comparison with MMVAE+ variational bound) It is instructive to compare bound with the MMVAE+ approach suggested in Palumbo et al. (2023). Assuming uniform masking distribution restricted to uni-modal sets so that S {8) for some M, we can write the bound from
Palumbo et al. (2233 as M7
MM MMMAAE+ (x) with (0)
MMMVAE+ (x)
/ ao(=') )as (eoe) (1 (1,)) d2' d='dā())
logmo(t(e)le''
f (0(2/ logpo(T A dz dz)
-KL(10" (O (M ( ( po(c)))o((()))
40

---

Published in Transactions on Machine Learning Research (09/2024)
Here, it is assumed that the multi-modal encoding distribution for computing the KL-divergence is of the form
ds (MM - Σ
(uol2|
AEM
and ro(2a) ITea (Zs) are additional trainable prior distributions.
Remark 25 (Cross-modal context variables and permutation-equiivriiat models) Tn contrast. to the PoE model, where the private encodings are independent, the private encodings are dependent in the Sum- Pooling model by conditioning a sample from the shared latent space. The shared latent variable Z' can be seen as a shared cross-modal context variable, and similar probabilistic constructions to encode such context variables via permutation-invarian modols have been suggosted in few-shot lcarning algorithms (Edwards and Storkey 2016 Giannone and Winther 2022) or, particularly, for neural process models (Garnelo et al. 2018b a Kim al. 2018). Permutation fermutaatoon-equivvriaat models have been studied for stochastic processes where invariant priors correspond to equivariant posteriors (Holderrieth et al.] 2021), such Gaussian processes Neural processes with private latent variables, wherein dependencies in the private latent variables
be constructed hierarchically (Wang and Van Hool 2020 Xu el. al. 2023
F Multi-modal posterior in exponential family models
Consider the setting where the decoding and encoding distributions are of the exponential family form, that is
pa(#,|z) = ps exp (TT(()))))) - log Zz(Ja
for all 8 E M, while for all S M,
(0(2|=)) plz) exp (o,
log T (A))))((E)))
where Ms and are base measures, (x,) and V(2) are sufficient statistics, while the natural parameters (x.s( and [=,0(2) are parameterized by the decoder or encoder networks, respectively, with Z, and I's being normalizing functions. Note that we made a standard assumption that the multi-modal encoding distribution has a fixed base measure and sufficient statisties for any modality subset. For fixed generative parameters 0, we want to learn multi-modal encoding distribution that minimizes over Es Pd:
= (z(2)))) (2(g) - log (ol))(2) -
Σ log po (2.l=)) de - log p)(ss)
SES
/ (o(2l1))) (V(2)) Ao.s((s)) log (ogTs(Ao.s(ss))
Σ (xo))(rz))
xES
(T,oo((ss) log Σ
Zz (T(oo((3))3 d log (o(ss)
SES
FES
[
[(2)
[ -oooogTT(
I (Tx))
[2) be,
|dz,
1
with by (2) Po(2) log Z, (f
G Mixture model extensions for different variational bounds
We consider the optimization of augmented variational bound
Clx, = p(8)
I/ golc, 2|xs) [[s]) dzde (|ss))cc
+ / go(c, 2|25) (Io dzde Kl
41

---

Published in Transactions on Machine Learning Research (09/2024)
We will pursue here encoding approach that does not require modeling the encoding distribution over the discrete latent variables explicitly, thus avoiding large variances in score-based Monte Carlo estimators [Ranganath et al, 2014) or resorting to advanced variance reduction techniques (Kool et al.) 2019) or alternatives such as continuous relaxation approaches (Jang et al. 2016 Maddison et al. 2016).
Assuming structured variational density of the form
go(c, =|0s) = =
can express the augmented version of 3 via
/ g 2|ms) [ogro(c, (xL(go((;
Ls(rs, ()
/ (uxt)ass + fo(z,
where (z) = -Isgp((ss(z) 3 log (=((s))) and
fe(,,0s)= = / g0((c22, 1- log (o(c)2,0)) + 8 Blogpo(c 2) de. de.
(19)
We can also write the angmented version of 5 in the form of
90 2| [logpo(<)ssl)) da (lol)aal)
/ (2)
where
2n (2, log (oo::)
log (=(x) + 3 log (=(xs)
which does not depend on the encoding density of the cluster variable. To optimize the variational bound with respect the cluster density, we can thus optimize 19), which attains its maximum value of
= Blog / po(c)po(zle)dde = 1 Blogpo(z))
=2)
at go(c) (c)2, = po (c]2) due to Remark 18 below with g(c) = 3 log lo
We derive analogous optimal structured variational density for the mixture-basod and total- cerelation-basede variational bounds. First, we write the mixture-based bound as
(x) I (o(2((s)) [log po po (c, de [KL(golc.
/ (x(()))) (M + felz, x)] dz,
where (2, = logre(<)2) 3 log (2(os) and fc(2,4) has a maximum value of (2, B Blggpo2z). Second, we can express the corresponding terms from the lotal-correlation--aaad bound as
(CC f a (o(2))) (o -C,LLKaLLL, (x)))
/ ()(2) ( (z, de,
where (2, -og B log (olg)(>)))) log (=(
42

---

Published in Transactions on Machine Learning Research (09/2024)
H Algorithm and ST estimators
We consider a multi-modal extension of the bhicking-th--ladinng (STL.) gradient estimator (Roeder et al. 2017) that has also been used in previous multi-modal bounds SSh et al. 2019). The gradient estima- tor ignores the score function terms when sampling go(2(rs)) for variance reduction purposes because it has a zero expectation. For the bounds thus ignore the score terms for both integrals. Consider the reparameterization with noise variables cs: EM ~ p and transformations zs Es, MM tm M. Inwariintt-ggg/ (0, MM Sinvariant-agx(U, es, Ess,, S, S,,s), for hs hp.s and for hM h2. single aggregation function that applies and masks the modalities appropriately. Pseudo-code for computing the gradients are given in Algorithm D If the encoding distribution is mixture distribution, we apply the stop-gradient operalion also to the mixture weights. Notice that. in the case of a mixture prior and an encoding distribution that includes the mixture component, the optimal encoding density over the mixture variable has no variational paramcters and given as the posterior density of the mixture component under the generative parameters of the prior.
(2) that involves sampling from (o(z(rs) and go(2|zm), we
We need to learn only
Algorithm/ 1 Single training step for computing unbiased gradients of C(x).
Input: Multi-modal data point 2, generative parameter 8, variational parameters (p,, f).
Sample S S~pp p.
Sample es. EM~
Set 2s = ( <M) and ZM = (.(
Stop gradients of variational parameters o' = stop stop_grad(o).
Set Ls(0, 6) = log (s)slss) 3 Blogpo(zs) & log go (eslas).
Set (0, Φ) log ( slz)) + loggg(zmlls) - Blog au (ZMlaM).
Output: Vo. Es(0, 6) + (0. o))
In the case of private latent variables, we proceed analogously and rely on reparameterizations zss (ol,,ss.z)) zs) for the shared latent variable z's (0(2([z)) as above and ăs = fs 2', rs) Jequivariant-agg (1) &s,,', S', hs) for the private latent variables És ~ "o(ās|=', æs). Moreover, we write Ps for a projection on the S-coordinates. Pseudo-code for computing unbiased gradient estimates for our bound is given in Algorithm 2
Algorithm 2 Single training step for computing unbiased gradients of C(x) with private latent variables.
Input: Multi-modal data point æ, generative parameter 0, variational parameters 6 = (4p, 0).
Sample S~ p.
Sample e's, es, 6,s; eM, 6,M p.
Set z's =sg( äs = (s,
Set z'm = tt IM), ΣM =ill Z'y SM,
Stop gradients of variational parameters ø' = stop_ (g))_)))
Set Es(0, o) log Æs) 3 (oggpl='ss) Blog go (2/) (r) + 3 Blogpo(2s(z3s) 3
Set (0s Φ) = log log + Blog go (=(ulss)) - Blog gw = + B (Ps( +
Blog (Ssp)PPs EM) - Bgg to (EMM=MM
Output: Vo. Es(0, o) + ())
Evaluation of multi-modal generative models
We evaluate models using different metrics suggested previously for multi-modal learning, see for example
Shi et al. (2019); Wu and Goodman 2019); Sutter al. (2021)
43

---

Published in Transactions on Machine Learning Research (09/2024)
Marginal, conditional and joint log-likelihoods. classic importance sampling
We can estimate the marginal log- oggllikelhhod using
po(=", ,.s)
Pogs) = log
K
Is ((s)
k=1
for 2 qo(-1z)) This also allows to approximate the joint log- oog-likelhhod log po(),, and consequently also the conditional log ( ( (t)) = (og) log po
Generative coherence with joint auxiliary labels. Following previous work (Shi el al. 2019 Sutter et al. 2021 Daunhawer et al. 2022 Javaloy et al.) 2022), we assess whether the generated data share the same information in the form of the class labels across different modalitics. To do so, we use pre-trained classifiers clfg: X + [K] that classify values from modality to possible classes. More precisely, for S C M and m E M, we compute the self- (m E S) or cross- (m & S) coherence Cs-m as the empirical average of
felfm (#m)=y})
over test samples with label where is (2(2))))) and 8. po(smlls)- The case S = M \ {m} corresponds to leave-one-out conditional coherence.
Linear classification accuracy of latent reprosentations. To evaluate how the latent representation can be used to predict the shared information contained in the modality subset S based on a linear model, we consider the accuracy Aces of a linear classifier clf,: Z - [K] that is trained to predict the label based on latent samples 2s ~ (0(ssla))in) from the training values x and evaluated on latent samples 2g (0 ((ttst) from the test values tesst.
J Linear models
Data generation. We generate 5 data sets of N = 5000 samples, each with M = modalities. We set the latent dimension D 30, while the dimension D, of modality drawn from U(30, 60). We set the shared across all modalities,
observation noise to components of b, independently from (V 1). For the setting without modality-speciffc latent variables, W. is the orthonormal matrix from QR algorithm applied to matrix with elements sampled iid from (1(-1,1). The bias coefficients W, are sampled independently from AO0 1/d). Conversoly, the setting with private latent variables in the ground truth model allows us to describe modality-specific variation by considering the sparse loading matrix
standard for PCA model. We sample the
W
W
0
0
W
0
W2
0
WM
W
0
0
WMJ
Here, W W. E DDXDD with D' = D/MM +) 5, Furthermore, the latent variable Z can be written as Z (Z'. Zi...., 2n) for private and shared latent variables 2, resp. Z'. We similarly generate orthonormal [ W. from QR decomposition. Observe that the general generative model with latent variable Z corresponds to the generative model 9 with shared Z' and private latent variables Ż with straightforward adjustments for the decoding functions. Similar models have been considered previously, particularly from Bayesian standpoint with different sparsity assumptions on the generative parameters (Archambeau and Virtanen et al. 2012 Zhao al. 2016).
Bach 2008
Maximum likelihood estimation. Assume now that observe N data points {I consisting of stacking the views xn = (x.,n) for each modality in S and let S = N Em (r, ba(,,. - b)T E P. DD,, Dj -M D., be the sample covariance matrix across all modalities. Let Ua E D. be the matrix the first D eigenvectors of S with corresponding eigenvalues X1,... Ap stored in the diagonal matrix Ap DDD
44

---

Published in Transactions on Machine Learning Research (09/2024)
The maximum likelihood estimates are then given by bML N En= Zn, M.. N-D Σj= ny and WML = UD(AD Bu 1) with the loading matrix identifiable up to rotations.
Model architectures. We estimate the observation noise scale a based on the maximum likelihood esti- mate ML.- We assume linear decoder functions po(x, [2)==] NWW" 2 16, AiL). fixed standard Gaussian prior p(=) I), and generalive parameters 0 (WT, fp. ..... W Mr br.) Details about the various encoding architectures are given in Table 15 The mopelity-specificc encoding functions for the PoE and MoE schemes have a hidden size of 512, whilst they are of size 256 for the learnable aggregation schemes having additional aggregation parameters 4.
K Non-linear identifiable models
We also show in Figure Ö the reconstructed modality values and inferred latent variables for one realization with our bound, with the corresponding results for a mixture-based bound in Figure
-2
(a) Observed data
(b) True latents
(c) PoE (z)
(d) PoE (+)
2
MoB (x)
(f) MoE (z)
(g) SumP, K 1 (x)
(h) SumP, (2)
20
-25
-50
-75
-100
(1) SumP, K 5 (x)
(j) SumP, (2)
(k) SumPM, = (2)
(1 SumPM, (2)
Figure 6: Bi-modal non-linear model with label and continuous modality based on our proposed objective.
SumP: SumPooling, SumPM: SumpPollnggMixtuure.
45

---

Published in Transactions on Machine Learning Research (09/2024)
00-
-15
-10-
-10
(a) Observed data
(b) True latents
(c) PoE (x)
(d) PoF (z)
10
2-
-4-
(e) MoE (2)
(I) MoE (=)
(g) SumP, K (x)
(h) SumP (2)
(i) SumP, K 5 (x)
(i) SumP, K (x)
(k) SumPM, = (x)
(1) SumPM, (x)
Figure 7: Bi-modal non-linear model with label and continuous modality based on mixture bound. SumP: SumPooling, SumPM: SumPoo SumpPolingMixture
46

---

Published in Transactions on Machine Learning Research (09/2024)
L MNIST-SVHN-Terl
L.1 Training hyperparamters
The SNIST-SVHN-Teo. data set is taken from the code accompanying Sutter et al. (2021) with around 1.1 million train and 200k lest samples. All models are trained for 100 epochs with a batch size of 250 using Adam Kingma and Ba, 2014) and a cosine decay schedule from 0.0005 to 0.0001.
L.2 Multi-modal rates and distortions
10000
Masked
13000
4000
-400
-000
-2000
1000
-4000
Made
Palie
Settalenton
SaePPetinng
Reomepation
(a) Full Reconstruction DDM
(b) Cross Prediction D
Nocked Makure
Mosted Meture
2x100
Cross
PdR
BuesPading
Appropation
Agp Aoarogatio
(c) Full Rates RM
(d) Cross Rates R
Figure 8: Rate and distortion terms for MNIST-SVHN-Tex with shared and private latent variables.
47

---

Published in Transactions on Machine Learning Research (09/2024)
19000
8000
ilsoomttrudts
-10000
22000
4000
2000
@ Pbtt)
IB Romioning?
Pall+
ee pimm salngs
e Popto
Bemfooling
at PME-))
fl Patr?
(1.8 Blnd Rudieg)
4 Putt-g
iBela Agrigilon
Clate Appnepatooy
(a) Full Reconstruction D
(b) Cross Prediction D
10'
10
Ru Roks
10'
1B Patr)
et #aastbeing)
(Fa Poifr)
(1. RannPhding
Patk)
HandPaaingg
BB Potr)
MessPaeing
(1a. Petio
(1d Sannbeiegg
(aanthuudin
[Bota, Aggrogatinn
Bott, Approgation
(c) Full Rates RM
(d) Cross Rates R
Figure 9: Rate and distortion terms for MNIST-SVHIN-Tex with shared latent variables and different 3.
48

---

Published in Transactions on Machine Learning Research (09/2024)
L.3 Log-liken log-likelhhod estimates
Table 7: Test log-likelihood estimates for varying choices for the joint data (M+S+T) as well as for the marginal data of each modality based importance sampling (512 particles). Multi-modal generative model with 40-dimensional shared latent variable. The second part of the Table contains reported log- likelihood values from baseline methods that, however, impose more restrictive assumptions on the decoder variances, which likely contributes to much lower log- ligliiklihood values reported in previous works, irrespective of variational objectives and aggregation schemes.
Proposed objuctive
Mixture bound
(8, Aggregation)
M+S+T
M
T
M+S+T
T
10.1, PoF+))
5433 (24.5)
1786 (41.6)
3578 (63.5)
-29 (2.4)
5481 (18.4)
2207 (19.8)
3180 (33.7)
-39 (1.0)
[0.1, SumPooling)
7067 (78.0)
2455 (3.3)
4701 (83.5)
(0.4)
(15.7)
2308 (9.3)
3552 (7.4)
-50 (1.9)
(1.0, PoE+)
6872 (9.6)
2599 (5.6)
4317 (1.1)
-9 (0.2)
5900 (10.0)
2449 (10.4)
3443 (11.7)
-19 (0.4)
(1.0, SumPooling)
7056 (124.4)
2478 (9.3)
4640 (113.9)
-6 (0.0)
6130 (4.4)
2470 (10.3)
3660 (1.5)
-16 (1.6)
(4.0, PoE+)
7021 (13.3)
2673 (13.2)
4413 (30.5)
-5 (0.1)
5895 (6.2)
2484 (5.5)
3434 (2.2)
-13 (0.4)
(4.0, SumPooling))
6690 (113.4)
2483 (9.9)
4259 (117.2)
-5 (0.0)
5659 (48.3)
2448 (10.5)
3233 (27.7)
-10 (0.2)
Results from
Sutter 20021
and Sutter
2020
MVAE
-1790 (3.3)
NA
NA
NA
MMVAR
-191 (5.7)
NA
NA
NA
MoPoE
-1819 (5.7)
NA
NA
NA
MMJSI)
-1961 (NA)
NA
NA
NA
L.4
Generated modalities
/
/
/
7
/
7
2
7
POM
2
7
7
6
6
6
6
6
BIX
6
6
IF
7
2
7
2
4
4
4
4
4
FOFT
8
I
3
7
3
ThE
3
8
$
5
3
2
>
2
7
2
2(
2
2
2
6
4
7
$
J
ewve 7
S
DEVET
7
7
7
6
(a) Proposed objective, =(b) Proposed objective, G =(c) Mixture-based bound, (d)
Mixture-based bound,
0.1
0.1
Figure 10: Conditional generation for different parameters. The first column is the conditioned modality. The next three columns are the generated modalities using a SumPooling aggregation, followed by the three columns for a PoE+ scheme.
49

---

Published in Transactions on Machine Learning Research (09/202))
/
7
/
7
7
7
7
7
7
6
6
6
6
6
4
4
2
4
2
6
(a) Our bound
(b) Mixture-based bound
Conditional generation for wtwpuuttatiio-eqvvviviiittt schemes and private latent variable con- straints. The first column is the conditioned modality. The next three columns are the generated modalities using a HamPooling aggregation, followed by the three columns for a SelfAtiention scheme and a PoE model.
Figure 11:
L.5
Conditional coherence
Table 8: Conditional coherence for models with shared latent variables and bi-modal conditionals. The letters the second line represent the modality which is generated based the sets of modalities the line below it.
Proposed objective
Mixture bound
M
T
M
T
Aggregation
M+T
M+T
S+T
PoE
0.98
0.98
0.60
0.75
0.58
0.77
0.82
1.00
1.00
0.96
0.97
0.95
0.61
0.11
0.61
0.45
0.99
0.98
PoE-
0.97
0.98
0.35
0.73
0.52
0.75
0.83
1.00
0.99
0.97
0.97
0.96
0.61
0.11
0.63
0.15
0.99
0.97
MoB
0.88
0.97
0.90
0.35
0.11
0.35
0.11
0.72
0.69
0.88
0.96
0.89
0.32
0.10
0.33
0:42
0.72
0.69
MoE
0.85
0.94
0.86
0.32
0.10
0.32
0.40
0.71
0.67
0.87
0.96
0.80
0.32
0.10
0.32
0.42
0.72
0.69
SumPooling
1.00
0.97
0.99
0.97
SellAtiention
0.97
0.97
0.82
0.76
0.30
0.78
0.69
1.00
1.00
0.97
0.97
0.99
0.66
0.10
0.63
0.15
0.99
1.00
Results from Sutter et al.
20121
Sutter ml.
21200
and
Hwwnn ml.
2011
MVAE
NA
0.32
NA
0.13
NA
0.29
NA
NA
MMVAE
NA
NA
0.87
NA
0.31
NA
0.84
NA
NA
MoPoE
NA
NA
0.94
NA
0.36
NA
0.93
NA
NA
MMISD
NA
NA
NA
NA
NA
MVTCAE (w/o T)
NA
NA
NA
NA
NA
NA
NA
NA
NA
Table 9: Conditional coherence for models with private latent variables and uni-modiee conditionals. The letters the sccond line represent the modality which is generated based on the sets of modalitics the line below it.
Proposed objective
Mixture bound
M
S
T
Aggregation
S
M
M
$
S
T
PoE+
0.97
0.12
0.13
0.20
0.62
0.24
0.16
0.15
1.00
0.96
0.83
0.99
0.11
0.58
0.11
0.44
0.39
1.00
Sum Pooling
0.97
0.42
0.59
0.44
0.67
0.40
0.65
0.45
1.00
0.97
0.86
0.99
0.11
0.62
0.11
0.45
0.40
1.00
SelfAttrntion
0.97
0.12
0.12
0.27
0.71
0.28
0.46
0.40
1.00
0.96
0.09
0.08
0.12
0.67
0.12
0.15
0.17
1.00
50

---

Published in Transactions on Machine Learning Research (09/2024)
Table 10: Conditional coherence for models with private latent variables and bi-modal conditionals. The letters the second line represent the modality, which is generated based on the sets of modalities the line below it.
Proposed objective
Mixcure bound
M
S
T
M
$
Aggregation
M+S
M-T
S+T
M+S
M++1'
S--1
M++S
M+1
S+'1'
M+S
M++1
S+'
M+S
M+'1
S+11
M++5
M++I
S+11
0.63
Sum Poolinn
0.97
0.97
0.54
0.79
0.43
0.80
0.57
1.00
1.00
0.97
0.97
0.93
0.64
0.11
0.63
0.45
0.90
0.97
SelFArtention
0.97
0.97
0.12
0.80
0.29
0.81
0.49
1.00
1.00
0.96
0.96
0.08
0.70
0.12
0.70
0.15
1.00
1.00
Table 11: Conditional coherence for models with shared latent variables for different As and uni-modal conditionals. The letters the second line represent the modality which is generated based the sets of modalities on the line below it.
Progosed abjective
Mixture bound
M
T
T
(8, Aggregation)
M
8
T
S
M
S
M
T
T
(0.1, PoE-+)
0.98
0.12
0.12
0.62
0.14
0.61
0.25
0.96
0.83
0.11
0.58
0.11
0.45
0.39
1.00
(0.1, SumPoollngg)
0.97
0.18
0.81
0.30
0.72
0.33
0.86
0.55
1.00
0.97
0.86
0.99
0.11
0.61
0.11
0.45
0.40
1.00
(1.0, PoE+))
0.97
0.15
0.63
0.24
0.63
0.42
0.79
0.35
1.00
0.96
0.83
0.99
0.11
0.59
0.11
0.45
0.39
1.00
(1.0, SumPooling)
0.97
0.48
0.87
0.25
0.72
0.36
0.73
0.48
1.00
0.97
0.86
0.99
0.10
0.63
0.10
0.45
0.40
1.00
(4.0, PoE++)
0.38
0.82
0.10
1.00
(4.0, SumPooling]
0.97
0.48
0.85
0.35
0.66
0.44
0.83
0.53
1.00
0.96
0.85
0.99
0.11
0.57
0.10
0.45
0.39
1.00
Table 12: Conditional coherence for models with shared latent variables for different Bs and bi-modal con- ditionals. The letters on the second line represent the modality, which is generated based on the sets of modalities on the line below it.
Proposed objective
Misture bound
(8. Aggregation)
M+S
M-T
S-T
M+S
M+T
S+T
M+S
M+-T
S+:T
M+S
M++T
S+T
M+S
M+T
S+T
M-S
M+T
S+T
(0.1. PoE
0.98
0.98
0.15
0.70
0.14
0.72
0.66
1.00
1.00
0.96
0.96
0.93
0.62
0.11
0.62
0.45
0.99
0.95
SumPooling)
0:97
0.31
0.97
0.94
0.66
0.65
01.45
(196
(1.0, PoE+)
0.97
0.98
0.55
0.73
0.52
0.75
0.83
1.00
0.99
0.97
0.97
0.96
0.64
0.11
0.63
0.15
0.99
0.97
SumPooling)
(1.0, PoE+)
0.97
0.98
0.84
0.76
0.66
0.78
0.82
1.00
1.00
0.97
0.97
0.96
0.62
0.10
0.62
0.45
0.93
0.98
(1.0 SumPooling)
0.97
0.97
0.89
0.77
0.40
0.78
0.80
0.99
1.00
0.97
0.97
0.96
0.61
0.10
0.60
0.15
0.99
0.97
51

---

Published in Transactions on Machine Learning Research (09/2024)
L.6 Latent classification accuracy
Unsupervised latent classification for 3 1 and models with shared latent variables only (top half) and shared plus private latent variables (bottom half). Accuracy computed with a linear classifier (logistic regression) trained on multi-modal imputs (M+S+T) or uni-modal inputs (M, S or T).
Table 13:
Proposed objective
Mixture bound
Aggregation
M+S+T
M
T
M+S-T
M
5
T
Pok
0,9888 (0.0000
0.940 (0.009)
0.649 (0.039)
0,9598 (0.0001
0.991 (0.004)
0.977 (0.012)
0.845 (0.000) 1.000 (0.000)
PoE+
0.978 (0.002)
0.934 (0.001)
0.624 (0.040)
0.999 (0.001)
0.998 (0.000)
0.981 (0.000)
0.851 (0.000)
1.000 (0.000)
MoF
0.841 (0.008)
0.974 (0.000)
0.609 (0.032)
1.000 (0.000)
0.940 (0.001)
0.980 (0.001)
0.843 (0:001)
1,000 (0.000)
MoB-+
0.850 (0.039)
0.967 (0.014)
0.708 (0.167)
0,983 (0.023)
0.928 (0.017)
0.983 (0.002)
0.836 (0.001)
1.000 (0.000)
SellAttention
0.985 (0.001)
0.951 (0.002)
0.693 (0.037)
0.986 (0.006)
0.991 (0.000)
0.981 (0.001)
0.861 (0.003)
1.000 (0.000)
SumPoollng
0.981 (0.0000
0.962 (0.030)
0.704 (0.014)
0.992 (0.008)
0.994 (0.000)
0.983 (0.000)
0 (0.0002)
1.000 (0.000)
PoE+
0.979 (0.009)
0.944 (0.000)
0.538 (0.032)
0.887 (0.07)
0.995 (0.002)
0.980 (0.002)
0.848 (0.006)
1.000 (0.000)
SumPPooling
0.987 (0.0044
0.968 (0.004)
0.370 (0.345)
0,992 (0.000)
0.994 (0.001)
0.982 (0.000)
0.870 (0.001)
1.000 (0.000)
SelCArtention
0.990 (0.003)
0.968 (0.002)
0.714 (0.008)
0.985 (0.000)
0.997 (0.001)
0.974 (0.000)
0.681 (0.031)
1.000 (0.000)
Results from
Sutter et ad.
2021
Sutter et al.
2020 and Hwanng et al.
2021
MVAE
0.96 (0:02)
0.90 (0.01)
(0.01)
(0.10)
MMVAE
0.86 (0.03)
0.95 (0.01)
0.79 (0.05)
0.99 (0:01)
MoPoE
0.98 (0.01)
0.95 (0.01)
0.50 (0.03)
0.99 (0.01)
MAIID
0.98 (NA)
(NA)
(NA)
(N.A)
MVTCAE (w/o T)
NA
0.93 (NA) 0.78 (NA)
NA
Table 14: Unsupervised latent classification for different 3s and models with shared latent variables only.
Accuracy is computed with a linear classifier (logistic regression) trained on multi-mode inputs (M+S+T) or uni-modal inputs (M, S or T).
Proposed objective
Mixture bound
(9, Aggregation)
M+S+T
M
T
M+S+T
M
T
10.1, PoE+)
0.983 (0.006)
0.919 (0.001)
0.561 (0.048)
0.988 (0.014)
0.992 (0.002)
0.979 (0.002)
0.846 (0.004)
1.000 (0.000)
(0.1, SumPoooing)
0,982 (0.004)
0.965 (0.002)
0,692 (0.017)
0.999 (0.001)
0.994 (0.000)
0.981 (0.002)
0.863 (0.005)
1.000 (0.000)
(1.0, PoFF+]
0.978 (0.002)
0.934 (0.001)
0.624 (0.040)
0.989 (0.001)
0.998 (0.000)
0.981 (0.000)
0.851 (0.000)
1.000 (0.000)
(1.0, SunPooling))
(0.000)
0.962 (0.000)
0.704 (0.014)
0.982 (0.008)
0.994 (0.000)
0.983 (0.000)
0.866 (0.002)
(0.000)
(4.0. PoE+)
0.981 (0.006)
0.943 (0.007)
0.630 (0.008)
0.983 (0.001)
0.998 (0.000)
0.981 (0.000)
0.846 (0.001)
1.000 (0.000)
(4.0, SunPoollng)
0 (0.0014)
0.963 (0.001)
0.681 (0.009)
0.995 (0.000)
0.992 (0.002)
0.980 (0.0011)
0.856 (0.001)
1.000 (0.000)
52

---

Published in Transactions on Machine Learning Research (09/2024)
M Encoder Model architectures
M.1 Linear models
Table 15: Encoder architectures for Gaussian models.
(a) -peelity- encoding functions h.()))- Latent dimen-
(b) Model for outer aggregation function Pa for
sion D 30, modality dimension D. (30, 60).
SumPooling and SelfAttention schemes.
MoE/PoE
SumpPoolig/SeffAtteniii
Outer Aggregation
Input: D,
Input: D
Input: 256
Dense D& 512, ReLU
Dense D, 256, ReLU
Dense 256 256, ReLU
Dense 512 512, ReLU
Dense 256 x 256, ReLU
Dense 256 256, ReLU
Dense 512 x 60
Dense 256 x 60
Dense 256 x 60
(c) Inner aggregation function
(d) Transformer parameters.
SumPooling
SelfAttention
SelfAttention (1 Layer)
Input: 256
Inpul: 256
Input: 256
Dense 256 256, ReLU
Dense 256 x 256, ReLU
Heads: 4
Dense 256 x 256, ReLU
Dense 256 x 256
Attention size: 256
Dense 256 256
Hidden size FFN: 256
M.2 Linear models with private latent variables
Table 16: Encoder architectures for Gaussian models with private latent variables.
(a) Modelity-speeffic encoding functions hz(r)). All private and shared latent variables are of dimension 10. Modality dimension D. U(30, 60).
(b) Model for outer aggregation function Po for SumPooling scheme.
Outer Aggrogation (ps)
PoE (hghared and Aprivate)
SumPPoliggSSelAttention
Input: 128
Input: D,
Input: D
Dense 128 x 128, ReLU
Dense Da 512, ReLU
Dense D₂ 128, ReLU
Dense 128 128, ReLU
Dense 512 512, ReLU
Dense 128 128, ReLU
Dense 128 x 10
Dense 512 10
Dense 128 x 10
(c) Inner aggregation functions.
(d) Transformer parameters.
SumPooling (Xo,0, X1,0, X2.0)
SelfAttention (X1,0, X2,0)
SelfAttention (1 Layer)
Input: 128
Input: 128 Dense 128 x 128, ReLU
Input: 128
Dense 128 x 128, ReLU
Heads: 4
Dense 128 x 128, ReLU
Dense 128 x 128
Attention size: 128
Dense 128 x 128
Hidden size FFN: 128
53

---

Published in Transactions on Machine Learning Research (09/2024)
M.3 Nonlinear model with auxiliary label
Table 17: Encoder architectures for nonlinear model with auxiliary label.
(a) Mpdelity-speciffe encoding functions ha (x.)). Modality di- mension D1 (continuous moddaityy) and D2 (label). Embedding dimension DE = 4 for PoE and MoE and De 128 otherwise.
(b) Model for outer aggregation function for SumPooling and SelfAttention schemes and mixtures thereof. Output dimension is Do
25 for mixture densities and Do otherwise.
Modality-specific encoders
Outer Aggregation
Input: D=
Input: 128
Dense D, x 128, ReLU
Dense 128 128, ReLU
Dense 128 x 128, ReLU
Dense 128 128, ReLU
Dense 128 DE
Dense 128 Do
(c) Inner aggregation function Xo-
(d) Transformer parameters.
SumPooling
SelfAttention
SelfAttention
Input: 128
Input: 128
Input: 128
Dense 128 128, ReLU
Dense 128 128, ReLU
Heads: 4
Dense 128 128, ReLU
Dense 128 128
Attention size: 128
Dense 128 128
Hidden size FFN: 128
M.4
Nonlinear model with five modalities
Table 18: Encoder architectures for nonlinear model with five modalities.
(b) Model for outer aggregation function Pe for SumPooling and SelfAttention schemes and
(a) Modality-speciffi encoding functions (xx). Modality di- mensions D. = 25. Latent dimension D = 25
mixtures thereof. Output dimension is Do 50 for mixture densities and Do = other- wise.
MoE/PoE
SummPoolng/SSlfAAttente
Input: D.
Input: D.
Outer Aggregation
Dense D, x 512, ReLU
Dense D, x 256, ReLU
Input: 256
Dense 512 x 512, ReLU
Dense 256 x 256, ReLU
Dense 256 256, ReLU
Dense 512 x 50
Dense 256 x 256
Dense 256 256, ReLU
Dense 256 x Do
(c) Inner aggregation function
(d) Transformer parameters.
SumPooling
SelfAttention
SelfAttention
Input: 256
Input: 256
Input: 256
Dense 256 x 256, ReLU
Dense 256 x 256, ReLU
Heads:
Dense 256 x 256, ReLU
Dense x 256
Attention size: 256
Dense 256 x 256
Hidden size FFN: 256
M.5 MNIST-SVHN-Text
For SVHN and Text, we use 2d- or Id-convolutional layers, respectively, denoted as Conv(f, k. s) for feature dimension f. kernel-size k, and stride We denote transposed convolutions as tConv. We use the neural network architectures implemented in Flax Heek al. 2023
54

---

Published in Transactions on Machine Learning Research (09/2024)
Table 19: Encoder architectures for MNIST-SVHNNTeett
(b) SVHN-specific encoding functions h, (xx). Modality dimensions D. = 3 32 32. The embedding dimension is DE = 2D for PoB/MoE and DE 256 for Sum Soaaloongg//SfAttentiiiii PoE+ /MoEE,, we add four times a Dense layer of size 256 with ReLU layer before the last linear layer.
(a) MNIST-specific encoding functions (z.). Modality dimensions D. 28 28. The embedding dimension is Dæ 2D for PoE/MoE and Dx 256 for SumPooling/ SelFAttention. For PoE+/MoF++ we add four times Dense layer size 256 with ReLU layer before the last linear layer.
For
MoF//PoF//ae MoF//PF./SSum Ptt.:///iiimoooonggefffe....etee
MoB/PoE/SumPone /St//Sooliin/SSlllttteeeeeee
Input: D,
Input: D.,
Conv(32, 2), ReLU
Dense D₂ 400; ReLU
Conv(64, 2), ReLU
Dense 400 x 400, ReLU
Conv(64, 4, 2), ReLU
Dense 400 x DE
Conv (1nv( 4, 2), ReLU, Flatten
Dense 2048 x DE
(c) Text-specific encoding functions ha (x,)). Modal- dimensions D. mension is DE = 2D for PoE/MoE and DE = 256 for permutation-invariant models (SumPool- ing/SelfAttention) and DE = 128 for permutation- equivariant models CaamPoooing/SllfAtteetin PoE+/MoE- we add four times a Dense layer of size 256 with ReLU layer before the last linear layer.
71. Embedding di-
(d) Model for outer aggregation function po for SumPooling and SelfAttention schemes. Output di- for models with shared latent variables only and Do = 10 + 10 for mod- with private and shared latent variables. DE = 256 for permatatioon-invarian and D, = 128 for models.
mension is Do 2D
For
Mot//Pod/SimPoolnn//bllAttttn
Outer Aggregation
Input: D.
Input: DE
Conv(128, 1, 1), ReLU
Dense DE x DE, LReLU
Conv( 128, 4, 2), ReLU
Dense DE x DE, LReLU
Conv 4, 2), ReL.U, Flatten
Dense D Do
Dense 128 x DE
(f) Transformer parameters for permutation- invariant models. DE 256 for permutation- invariant and D, 128 for permutationn-nvariint models.
(e) Inner aggregation function Xa for permutation- invariant models (Dg 256) and permutaion- equivariant models (Dr = 128).
SumPooling
SelfAttention
SelfAttention (2 Layers)
Imput: DE
Input: DE
Input: DE
Dense DE x DE, LReLU
Dense DE x DE, LReLU
Heads:
Dense DE x DE, LReLU
Dense x DE
Attention size: DE
Dense DE x Dg
Hidden size FFN: D
N MNIST-SVHN-Text Decoder Model architectures
For models with private latent variables, we concatenate the shared and private latent variables. We use a Laplace likelihood as the decoding distribution for MNIST and SVHN, where the decoder function learns both its mean as a function of the latent and a constant log-standard deviation each pixel. Following previous works (Shi al. 2019 Sutter et al. 2021), we re-weight the liceliiilsooods for different modalities relative their dimensions.
55

---

Published in Transactions on Machine Learning Research (09/2024)
Table 20: Decoder architectures for MNIST-SVHN-Tee.
(b) SVHN decoder. D, = for models with shared
(a) MNIST decoder. Dj 40 for models with shared latent variables only, and D, 10 + 10 otherwise.
latent variables only, and D, 10 + 10 otherwise.
SVHN
MNIST
Input: Dr
Input: D₁
Dense Dr 128, ReLU
Dense 40 x 400, ReLU
(64, 4, 3), ReLU
Dense 400 x 400, ReLU
tConv (4,, 4, 2), ReLU
Dense 400 x D,, Sigmoid
tConv (32, 4, 2), ReLU
tConv(3, 4, 2)
(c) Text decoder. D, for models with shared latent variables only, and D, 10 + 10 otherwise.
Text
Input: Dj
Dense Dj x 128, ReLU
(Conv(128, 4, 3), ReLU
tConv( 128, 4, 2), ReLU tConv(71, 1)
O Compute resources and existing assets
A reference implementation is available computations were performed on shared HPC systems. All experiments except Section
Our
5.3 were run
CPU server using one or two CPU cores. The experiments in Section 5.3 were run on a GPU server using one NVIDIA A100.
Our implementation based on JAX (Bradbury et al., 2018 and Flax (Heek et al. 2023 We com- pute the mcan correlation cocfficient (MCC) between true and inferred latent variables following Kho- and follow the data and model
makhem et al. (2020b), in httpss//githhb. httpa//..///milee//eeeeeeeeeee generation from Khemakhem et al. (2020a), https:///tttt:ii htwttau....mm/lkheeeeee..e com/1lkhenmiVAB in Section 5.2 well https:// / thub. wwt..///o///..aaaaa.eeeee CCM _Nta from Zhang et al. (2019) for generating the missingness mechanism. In our MNIST-SVHN Text experiments, we use code from Sutter et al. (2021). https:
//ithub. htt.///ooomaaattttee//ooodo
56


Reviews:
[{"review_id": "PcJA62OYO0", "reviewer": "Reviewer_tUXM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- *Clarity*: The paper reads well overall. I really appreciated the Remarks[4-6] providing different perspectives on the objective function. The paper is also very thorough and well-motivated. Furthermore, I think the authors do relatively a good job of explaining the limitations of their approach. \n\n- *Soundness*: The new objective makes sense (Though I do think the paper can be improved by providing more details and intuitions for why the bound is tighter than the mixture-based). The fact that the objective $L(X_{\\S})$ ends up being an approximate lower bound on $\\log p(X_{\\S}|X_S)$ is not super obvious mathematically at first so it's quite interesting! The use of permutation invariant architecture and distributions also makes complete sense.    \n\n- *Experiments*: The authors did a nice set of experiments to compare the objectives performance against the aother bounds. The datasets are ofcourse very simple but they are enough I think to show the potential of the new bound. There could be more highlight and discussion on the individual effect of the new bound against the choice of $q$ and the use of sum-pooling encoder. Figure 1 is also quite interesting!  \n\n\nOverall, I think the TMLR would find the paper interesting and I'm more than happy to recommend acceptance. \n\n**Weaknesses**\n\n- *Novelty*: I am not very familiar with multimodal learning literature so my apologies if this is not the case, but I am very surprised to see that the permutation invariant architecture and distributions are not been used previously before in multi-modal settings, especially given that I know these type of distributions and architectures have been used lot in Amortized VI settings in a variety of applications such as meta-learning etc. Therefore, The use of PE and MoE and the sum-pooling encoders by themselves I do not find very novel. The new bound however I think is a very valuable contribution and can be highlighted more in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a tighter bound a new aggregation scheme for VAE-based models that are infused with Mixture of Experts (MoE) or Product of Experts (PoE). The paper starts by proposing a new variational abound very similar to previous work by  Daunhawer et al., 2022 and Shi et al., 2019. From my understanding, it involves dividing the modalities two 2 subsets $X_S$ and $X_{/S}$ and optimizing the sum of two different ELBO objectives to target $p_d(X_S)$ and $p_d(X_{/S}|X_S)$ respectively (weighted by the prior $p(S)$). The second objective is different than the standard ELBO objective as it uses a the inference distribution $q(z|X_{S})$ rather than $q(z|X_{\\S})$. Intuitively, this makes sense given the latent variable for some modes should contain information to roughly generate the $x$ for other modalities. The authors then provide provide various interpretations of the given bound. The paper then proposes two different variational inference distribution to use: MoE and PoE. The authors additionally propose to use the encoder architecture that is permutation invariant (e.g. sum-pooling or transformer-based encoder) which makes sense given that $x$ modalities are permutation invariant. The proposed model is weakly identifiable given some conditions. Finally the authors do a series of experiments where the they compare the propose bound to the mixture-based bound proposed by  Daunhawer et al., 2022 and Shi et al., 2019. The results do indeed look promising.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Sorry if I missed this but it seems like the paper assumes you have access to the ground truth conditional distribution $p(x_{/S}|x_S)$. Is this is a reasonable data assumption in multi-modal settings?", "broader_impact_concerns": "This is mainly a theoretical paper and to the best of my understanding raises no major social impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5GCl6WfyLG", "reviewer": "Reviewer_y99S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThe paper is solidly written and covers various related work and ideas.\n\nThe theoretical derivations are serious.\n\nThe connection to identifiable VAEs is interesting.\n\n### Weaknesses\n\nThe paper is very dense and could use a gentler introduction.\n\nMany of the Remarks and Examples are related work and ideas, which, while interesting, could be distractive.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper derived a new and tighter variational lower bound for multi-modal VAEs (MMVAEs). It also proposed new and more flexible multi-modal aggregation schemes based on permutation-invariant NNs. A connection to identifiable VAEs is given, by seeing modality as an auxiliary variable. Both theoretical and experimental analyses are presented.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Disclaimer: While I know well about several types of VAEs, I am unfamiliar with MMVAEs. And, as indicated above, due to the density and broadness of the paper, I failed to check many details in the paper. Thus, many of the following could be seen as suggestions to make the paper more accessible.\n\nAs indicated in the Weaknesses, *the authors could move to Appendix some of the Remarks and Examples which are noncritical for understanding the main contributions, and the Introduction could be expanded*. And I hope the following will be useful in the latter regard.\n\nThe conditional independence involved in the generative model of MMVAEs is satisfied when the latent variable Z captures all the unobserved factors shared by the modalities. I think it would be beneficial to mention this intuition for newcomers.\n\nI am not sure why in the first place the previous work would want to consider the bound (1) rather than your (3)? I mean, your (3), which bounds each modality, is much more intuitive to me than (1). Is the scalability problem you mentioned at the beginning of p4 the main reason? Anyway, it would be helpful to touch this question earlier.\n\nI am not sure if it is convincing enough to introduce permutation-invariance simply as a generalization of MoE/PoE. On the one hand, I am not sure if it is still a generalization in multi-modal cases, which is what we really consider. On the other hand, I think permutation-invariance and MoE/PoE contain very different information about the underlying mechanism, i.e., how the modalities work together to generate the observed data.\n\nCould you expand on the reason (a) on p14 why your bound is beneficial for learning an identifiable VAE? A related note is that it would be useful to make it clear that identifiability is a preliminary to estimation, thus your bound or aggregation schemes do not aid the identifiability, rather, given the model which is already identifiable, the aggregation schemes give more flexible parametrization, and the bound gives a better approximation. I know the authors understand this but I think it would be better to spell this out for readers.\n\nI believe it is unnecessary to introduce $\\rho$ as a pdf, because we have finite modalities. It is simply a pmf, and this is much easier for readers.\n\nIt seems that the $Z_S$ in (2) is not only “to emphasize that Z is conditional on $X_S$”. For example, if we just see $Z_S$ as another symbol of Z, then $H(X|Z_S)$ in (2) cannot be reduced back to the 1st term of (1).\n\n### Minor\nAt least in one place, you wrote “multi-modal” as “multi-model”.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3ZVW3rNXuW", "reviewer": "Reviewer_S2Lb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* The paper is on an important task of fitting VAEs on data from multiple modalities (e.g. text/images). The authors address a previously-identified gap in the mixture-based variational objectives for fitting multimodal VAEs, by proposing a new objective that is motivated by its ability to reduce this gap. (Although this motivation is slightly burried in the presentation and needs a bit more work to make it concrete, see weaknesses.)\n* Noting that some previous approaches used permutation-invariant variational distributions, the authors proposed a few more approaches that aim to induce this invariance. The approaches themselves are not particularly new, but seem new in this context. \n* The evaluation seems to support the authors' hypothesis that the proposed approach (using the proposed objective and variational distributions) achieves higher log-likelihood than some existing approaches. \n\n## Weaknesses\n\n### Main comments/questions\n\n* The proposed objective $\\mathcal{L}(x, \\theta, \\phi, \\beta)$ in Section 2 is introduced without derivation. Although the derivation is straightforward, including the derivation where the objective is introduced would clarify that the objective is not truly a lower-bound on the log-likelihood. Consequently, I think calling the objective throughout the paper (and in the title) an \"approximate bound\" can be misleading as it invites potentially erroneous mis-uses of it.\n* A key contribution of the paper, as stated by the authors, is a _tighter_ variational bound/objective. But, until Remark 5 (end of Section 2.1), it is not clearly explained which existing objective is the proposed objective tighter than. (Except for lightly touching upon it with one sentence in the Contributions paragraph in Page 2.) This leaves the reader wondering what is it that the paper solves until the end of Section 2.1. The background information provided in Remark 5 should be discussed with the rest of the background in Section 1, and the advantages of the objective should be discussed in more detail.\n* Assuming that the variational distribution exactly approximates the model's posterior, Remark 5 explains that their objective is tighter than the existing Multimodal VAE objectives, where the gap arises due to sub-sampling of the modalities. However, the case where the variational approximation is not accurate is not discussed. Moreover, the Jensen's gap, alluded to at the end of Remark 5, is also not worked out. Explaining these two points would greatly help their contribution regarding the tightness/advantages of their objective.\n* Remark 3 states that maximising $\\mathcal{L}_{\\setminus \\mathcal{S}}$ yields $q\\_\\phi(z \\mid x\\_{\\mathcal{S}}) = q^{agg}\\_{\\phi, \\setminus \\mathcal{S}}(z \\mid x\\_\\mathcal{S})$ because of equation $\\texttt{Z}\\_{\\text{conditional}}$. Firstly, the text should describe what quantity/parameters are you maximising $\\mathcal{L}\\_{\\setminus \\mathcal{S}}$ _with respect to_. (Is it $\\phi$?) Secondly, if you maximise the expected $\\mathcal{L}\\_{\\setminus \\mathcal{S}}$ in equation $\\texttt{Z}\\_{\\text{conditional}}$ w.r.t. the parameters of the variational distribution $\\phi$, the optimal $q\\_\\phi(z \\mid x\\_{\\mathcal{S}})$ may not be equal to $q^{agg}\\_{\\phi, \\setminus \\mathcal{S}}(z \\mid x\\_\\mathcal{S})$. This is because the parameters $\\phi$ appear in both terms of equation $\\texttt{Z}\\_{\\text{conditional}}$, and hence maximisation of the equation w.r.t. $\\phi$ will likely strike some balance between the two terms in the equation, i.e. $q\\_\\phi(z \\mid x\\_{\\mathcal{S}}) \\not= q^{agg}\\_{\\phi, \\setminus \\mathcal{S}}(z \\mid x\\_\\mathcal{S})$.\n* The discussion below equation 10 in Section 3.2, suggests that due to factorisation of the model $p\\_\\theta(z', \\tilde z\\_{\\mathcal{S}} \\mid x\\_\\mathcal{S}) p\\_\\theta(\\tilde z\\_{\\setminus \\mathcal{S}} \\mid z',  \\tilde z\\_{\\mathcal{S}})$, the variational distribution $q\\_\\phi(z', \\tilde z\\_{\\mathcal{S}}, \\tilde z\\_{\\mathcal{\\setminus S}} \\mid x\\_{\\mathcal{S}})$ should also similarly factorise as $q\\_\\phi(z', \\tilde z\\_{\\mathcal{S}} \\mid x\\_{\\mathcal{S}}) q\\_\\phi(\\tilde z\\_{\\setminus \\mathcal{S}} \\mid z', \\tilde z\\_{\\mathcal{S}})$. But, although it may be a useful inductive bias and/or simplification, this may not necessarily be true if we aim to accurately approximate the posterior $p\\_\\theta(z', \\tilde z\\_{\\mathcal{S}}, \\tilde z\\_{\\mathcal{\\setminus S}} \\mid x\\_{\\mathcal{S}})$. For example, in case $q\\_\\phi(z', \\tilde z\\_{\\mathcal{S}} \\mid x\\_{\\mathcal{S}}) \\not= p\\_\\theta(z', \\tilde z\\_{\\mathcal{S}} \\mid x\\_\\mathcal{S})$, the factorisation $q\\_\\phi(z', \\tilde z\\_{\\mathcal{S}} \\mid x\\_{\\mathcal{S}}) q\\_\\phi(\\tilde z\\_{\\setminus \\mathcal{S}} \\mid z', \\tilde z\\_{\\mathcal{S}}, x\\_{\\mathcal{S}})$ may be able to better approximate the model posterior, because conditioning $q\\_\\phi(\\tilde z\\_{\\setminus \\mathcal{S}} \\mid z', \\tilde z\\_{\\mathcal{S}}, x\\_{\\mathcal{S}})$ on $x\\_{\\mathcal{S}}$ allows the distribution to some extent compensate for the inaccuracies (or loss of information) in $q\\_\\phi(z', \\tilde z\\_{\\mathcal{S}} \\mid x\\_{\\mathcal{S}})$.\n* As explained in the paragraph preceding Examples 3-5 (after equation 11), the distribution $p\\_\\theta(\\tilde z\\_{\\setminus \\mathcal{S}} \\mid z',  \\tilde z\\_{\\mathcal{S}})$ is often analytically tractable. So why do Examples 3-5 not use it? Specifically, Example 3 instead uses $\\prod\\_{s \\in \\mathcal{M}\\setminus\\mathcal{S}} p\\_{\\theta}(\\tilde z\\_s)$ and Examples 4-5 instead use $\\prod\\_{s \\in \\mathcal{M}\\setminus\\mathcal{S}} p\\_{\\theta}(\\tilde z\\_s \\mid z')$. Is this a typo?\n* The authors provided permutation-equivariant versions of PoE, SumPooling, and Self-Attention, but did not discuss permutation-equivariant versions of MoE or MoPoE. Is there a reason why this may not be possible with these aggregation schemes, are they incompatible with permutation-equivariance? Would be good to discuss this.\n* The evaluations use quantities such as \"Full reconstruction\", \"Full rates\", \"Cross reconstruction\", and \"Cross rates\", but the effect of these metrics going up/down is not clearly motivated or explained in the discussion of the results, thus it is unclear how important are they for multimodal VAEs and their contribution. Some interpretation of these metrics and their importance to multimodal VAEs would be helpful.\n* In the MNIST-SVHN-Text experiments, the total number of modalities is 3. You could thus parametrise all possible permutations of modalities (8 of them in total) using a separate encoder (one per permutation). This would serve as a strong baseline to verify how good the permutation-invariant parametrisations really are. \n* In Section 5.3, it seems that using the proposed objective improves cross-generation of the SVHN modality (Figure 2) but decreases cross-reconstruction (Figure 3). Why is that?\n\n### Presentation clarity\n\nI have also found the presentation of the paper slightly lacking in clarity and coherence (in addition to some of the points discussed above). Below are some examples:\n\n* Relevance of the discussion about VampPrior in Remark 3 is not clear. VampPrior is not used in the paper and its use in this setting may be tricky. This is because the goal is to approximate $p\\_\\theta(z \\mid x\\_{\\mathcal{S}})$. Meaning if we wanted to use VampPrior $p^{\\text{Vamp}}(z \\mid \\mathcal{D})$ of the form $\\frac{1}{K}\\sum\\_{k=1}^K q\\_\\phi(z \\mid u\\_k)$ to approximate $p\\_\\theta(z \\mid x\\_{\\mathcal{S}})$ we would have to learn $K$ pseudo-inputs $u\\_k$ for each $x\\_{\\mathcal{S}}$ for all $\\mathcal{S} \\subset \\mathcal{M}$ and $\\mathcal{x}$, which would scale exponentially in the number of modalities $\\mathcal{M}$. So it is unclear how the discussion helps the paper.\n* Remark 4 presents some hypotheses that are not experimentally validated or reflected upon in the rest of the paper, so it is unclear how relevant it is.\n* Remarks 9, 10, 11, and 13 present some results/discussion that seem out-of-context. Why are they relevant to the paper?\n* In Table 2, the numbers in parentheses are not explained. Are they standard deviations/errors, how many independent runs?\n* The tables in the evaluations were a little hard to decipher, this is because \"ours\" is used to denote the proposed bound, but the proposed aggregation schemes (SumPooling/SelfAttention) weren't highlighted. Denoting the proposed bound as \"our bound\" or \"proposed bound\" and appending the proposed scheme names with \"(proposed)\" may help.\n\n### Typos\n\nThere is a bunch of (minor) typos in the paper. Here's a list of some:\n\n* Below Eq 2., the mutual information equation is missing $\\mathrm{d}x\\mathrm{d}y$. \n* Equation $\\texttt{X}\\_{\\text{conditional}}$ should have a minus before the term $\\int q\\_\\phi(z \\mid x) \\log \\frac{q\\_\\phi(z \\mid x\\_{\\mathcal{S}})}{p\\_\\theta(z \\mid x\\_{\\mathcal{S}})}\\mathrm{d}z$ instead of a plus. \n* The text after Corollary 2 should be $\\mathcal{L}\\_{\\setminus \\mathcal{S}}$ instead of $\\mathcal{L}\\_{\\setminus \\mathcal{L}}$.\n* Reference links to Corollary 19 (Appendix) should be links to Corollary 2 (main text). \n* The VampPrior in equation 5 is missing $1/K$.\n* Page 7, missing $\\mathcal{S}$ in the text after the fifth equation on the page, as in \"... for any $\\mathcal{S}$ ...\".\n* Corollary 8, \"minimizing $\\mathcal{L}$\" should be \"maximizing ...\".\n* Remark 9, \"minimizing $\\mathcal{L}_{\\mathcal{S}}^{\\text{Mix}}$\" should be \"maximizing ...\".\n* Page 9, The equation for MoPoE variational distribution should have $q\\_\\phi^{\\text{PoE}}$ inside the summation instead of $q\\_\\phi^{\\text{MoE}}$.\n* Start of Section 5 states that the proposed bound will be called \"masked\" in the following figures, but all of the figures call it \"ours\", with the exception of Figure 3. \n* In Table 1 the wrong results are highlighted as best in the Full-rates column. It should be the cell with 1.02.\n* Page 33, the second equality following lower-bound derivation for $I\\_{q\\_\\phi}(X\\_{\\setminus \\mathcal{S}}, Z\\_{\\mathcal{M}} \\mid X\\_{\\mathcal{S}})$ is missing a $\\mathrm{d}x\\_{\\setminus \\mathcal{S}}$ after the $\\log p\\_\\theta(x\\_{\\setminus \\mathcal{S}} \\mid z)$ term.\n* Page 33, the first equality following the upper-bound derivation for $I\\_{q\\_\\phi}(X\\_{\\setminus \\mathcal{S}}, Z\\_{\\mathcal{M}} \\mid X\\_{\\mathcal{S}})$ is missing a $q\\_\\phi(z \\mid x)$ after the $p_d(x\\_{\\setminus \\mathcal{S}} \\mid x\\_\\mathcal{S})$ term. (Also, the term $p_d(x\\_{\\setminus \\mathcal{S}} \\mid x\\_\\setminus)$ should be $p\\_d(x\\_{\\setminus \\mathcal{S}} \\mid x\\_\\mathcal{S})$.)\n* In Remark 22, the sign before $\\beta \\log p\\_\\theta(z)$ should be a plus (in two places).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new variational objective for fitting Multimodal VAEs. In contrast to existing mixture-based bounds, the proposed objective addresses a gap that arises in the mixture-based bounds due to modality subsampling. This is tackled by deriving a variational objective that resembles a lower-bound to the marginal log-likelihood and that, subject to some conditions, can be shown to be equal to the marginal log-likelihood. Moreover, the authors propose learnable permutation-invariant approaches to parametrise the variational distribution, which mitigates the exponential growth of variational distributions needed due to the subsampling of the modalities. The method is evaluated on some synthetic and realistic multimodal datasets and is shown to achieve better log-likelihood and other related quantities.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses above. The important changes/questions are under \"main comments/questions\". But, I hope the other comments may help improve the overall clarity/correctess of the paper and should be easy to correct.", "broader_impact_concerns": "None. The method presents a technical contribution for multimodal VAE training and the broader impacts are no different from a standard VAE model.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall I see this as solid technical work in a research area that is relevant to the TMLR community.  There are some weaknesses in terms of clarity, but these are not sufficiently serious to warrant rejection of the paper.  The reviewers were unanimous in recommending the paper be accepted, and I see no reason to overwrite their decision.

As all the reviewers were already satisfied with the updates made during the rebuttals/discussions, I do not see the paper as having any specific issues that need to be corrected through minor revisions.  I, therefore, recommend that the paper is accepted "as is".  That said, I think their is scope to improve the clarity of the paper further and encourage the authors to work on this for the final camera-ready version.  To this end, I include some comments made by Reviewer SL2b below from their final recommendations which are not currently visible to the authors, but may be helpful in making such improvements.

> I still believe that the clarity of the paper could be better if the authors moved some discussion points that are less relevant to the main narrative into the appendix, further improved the arguments about the proposed bound, explained the evaluation results more clearly and their relevance to multi-modal VAEs, and analysed the properties of the new objective in more detail.
Decision: Accept (poster)