On the Limitations of Multimodal VAEs

1 INTRODUCTION . In recent years , multimodal VAEs have shown great potential as efficient generative models for weakly-supervised data , such as pairs of images or paired images and captions . Previous works ( Wu and Goodman , 2018 ; Shi et al. , 2019 ; Sutter et al. , 2020 ) demonstrate that multimodal VAEs leverage weak supervision to learn generalizable representations , useful for downstream tasks ( Dorent et al. , 2019 ; Minoura et al. , 2021 ) and for the conditional generation of missing modalities ( Lee and van der Schaar , 2021 ) . However , despite the advantage of weak supervision , state-of-the-art multimodal VAEs consistently underperform when compared to simple unimodal VAEs in terms of generative quality.1 This paradox serves as a starting point for our work , which aims to explain the observed lack of generative quality in terms of a fundamental limitation that underlies existing multimodal VAEs . What is limiting the generative quality of multimodal VAEs ? We find that the sub-sampling of modalities during training leads to a problem that affects all mixture-based multimodal VAEs—a family of models that subsumes the MMVAE ( Shi et al. , 2019 ) , MoPoE-VAE ( Sutter et al. , 2021 ) , and a special case of the MVAE ( Wu and Goodman , 2018 ) . We prove that modality sub-sampling enforces an undesirable upper bound on the multimodal ELBO and thus prevents a tight approximation of the joint distribution when there is modality-specific variation in the data . Our experiments demonstrate that modality sub-sampling can explain the gap in generative quality compared to unimodal VAEs and that the gap typically increases with each additional modality . Through extensive ablations on three different datasets , we validate the generative quality gap between unimodal and multimodal VAEs and present the tradeoffs between different approaches . Our results raise serious concerns about the utility of multimodal VAEs for real-world applications . We show that none of the existing approaches fulfills all desired criteria ( Shi et al. , 2019 ; Sutter et al. , 2020 ) of an effective multimodal generative model when applied to slightly more complex datasets than used in previous benchmarks . In particular , we demonstrate that generative coherence ( Shi et al. , 2019 ) can not be guaranteed for any of the existing approaches , if the information shared between modalities can not be predicted in expectation across modalities . Our findings are particularly relevant for applications on datasets with a relatively high degree of modality-specific variation , which is a typical characteristic of many real-world datasets ( Baltrušaitis et al. , 2019 ) . 1The lack of generative quality can even be recognized by visual inspection of the qualitative results from previous works ; for instance , see the supplementaries of Sutter et al . ( 2021 ) or Shi et al . ( 2021 ) . 2 RELATED WORK . First , to put multimodal VAEs into context , let us point out that there is a long line of research focused on learning multimodal generative models based on a wide variety of methods . There are several notable generative models with applications on pairs of modalities ( e.g. , Ngiam et al. , 2011 ; Srivastava and Salakhutdinov , 2014 ; Wu and Goodman , 2019 ; Lin et al. , 2021 ; Ramesh et al. , 2021 ) , as well as for the specialized task of image-to-image translation ( e.g. , Huang et al. , 2018 ; Choi et al. , 2018 ; Liu et al. , 2019 ) . Moreover , generative models can use labels as side information ( Ilse et al. , 2019 ; Tsai et al. , 2019 ; Wieser et al. , 2020 ) ; for example , to guide the disentanglement of shared and modality-specific information ( Tsai et al. , 2019 ) . In contrast , multimodal VAEs do not require strong supervision and can handle a large and variable number of modalities efficiently . They learn a joint distribution over multiple modalities , but also enable the inference of latent representations , as well as the conditional generation of missing modalities , given any subset of modalities ( Wu and Goodman , 2018 ; Shi et al. , 2019 ; Sutter et al. , 2021 ) . Multimodal VAEs are an extension of VAEs ( Kingma and Welling , 2014 ) and they belong to the class of multimodal generative models with encoder-decoder architectures ( Baltrušaitis et al. , 2019 ) . The first multimodal extensions of VAEs ( Suzuki et al. , 2016 ; Hsu and Glass , 2018 ; Vedantam et al. , 2018 ) use separate inference networks for every subset of modalities , which quickly becomes intractable as the number of inference networks required grows exponentially with the number of modalities . Starting with the seminal work of Wu and Goodman ( 2018 ) , multimodal VAEs were developed as an efficient method for multimodal learning . In particular , multimodal VAEs enable the inference of latent representations , as well as the conditional generation of missing modalities , given any subset of input modalities . Different types of multimodal VAEs were devised by decomposing the joint encoder as a product ( Wu and Goodman , 2018 ) , mixture ( Shi et al. , 2019 ) , or mixture of products ( Sutter et al. , 2021 ) of unimodal encoders respectively . A commonality between these approaches is the sub-sampling of modalities during training—a property we will use to define the family of mixture-based multimodal VAEs . For the MMVAE and MoPoE-VAE , the sub-sampling is a direct consequence of defining the joint encoder as a mixture distribution over different subsets of modalities . Further , our analysis includes a special case of the MVAE without ELBO sub-sampling , which can be seen as another member of the family of mixture-based multimodal VAEs ( Sutter et al. , 2021 ) . The MVAE was originally proposed with “ ELBO sub-sampling ” , an additional training paradigm that was later found to result in an incorrect bound on the joint distribution ( Wu and Goodman , 2019 ) . While this training paradigm is also based on the sub-sampling of modalities , the objective differs from mixture-based multimodal VAEs in that the MVAE does not reconstruct the missing modalities from the set of sub-sampled modalities.2 Table 1 provides an overview of the different variants of mixture-based multimodal VAEs and the properties that one can infer from empirical results in previous works ( Shi et al. , 2019 ; 2021 ; Sutter et al. , 2021 ) . Most importantly , there appears to be a tradeoff between generative quality and generative coherence ( i.e. , the ability to generate semantically related samples across modalities ) . Our work explains why the generative quality is worse for models that sub-sample modalities ( Section 4 ) and shows that a tighter approximation of the joint distribution can be achieved without sub-sampling ( Section 4.3 ) . Through systematic ablations , we validate the proposed theoretical limitations and showcase the tradeoff between generative quality and generative coherence ( Section 5.1 ) . Our experiments also reveal that generative coherence can not be guaranteed for more complex datasets than those used in previous benchmarks ( Section 5.2 ) . 3 MULTIMODAL VAES , IN DIFFERENT FLAVORS . Let X : = { X1 , . . . , XM } be a set of random vectors describing M modalities and let x : = { x1 , . . . , xM } be a sample from the joint distribution p ( x1 , . . . , xM ) . For conciseness , denote subsets of modalities by subscripts ; for example , X { 1,3 } or x { 1,3 } respectively for modalities 1 and 3 . Throughout this work , we assume that all modalities are described by discrete random vectors ( e.g. , pixel values ) , so that we can assume non-negative entropy and conditional entropy terms . Definitions for all required information-theoretic quantities are provided in Appendix A . 2For completeness , in Appendix C , we also analyse the effect of ELBO sub-sampling . 3.1 THE MULTIMODAL ELBO . Definition 1 . Let pθ ( z |x ) be a stochastic encoder , parameterized by θ , that takes multiple modalities as input . Let qφ ( x | z ) be a variational decoder ( for all modalities ) , parameterized by φ , and let q ( z ) be a prior . The multimodal evidence lower bound ( ELBO ) on Ep ( x ) [ log p ( x ) ] is defined as L ( x ; θ , φ ) : = Ep ( x ) pθ ( z | x ) [ log qφ ( x | z ) ] − Ep ( x ) [ DKL ( pθ ( z | x ) || q ( z ) ) ] . ( 1 ) The multimodal ELBO ( Definition 1 ) , first introduced by Wu and Goodman ( 2018 ) , is the objective maximized by all multimodal VAEs and it forms a variational lower bound on the expected logevidence.3 The first term denotes the estimated log-likelihood of all modalities and the second term is the KL-divergence between the stochastic encoder and the prior . We take an information-theoretic perspective using the variational information bottleneck ( VIB ) from Alemi et al . ( 2017 ) and employ the standard notation used in multiple previous works ( Alemi et al. , 2017 ; Poole et al. , 2019 ) . Similar to the latent variable model approach , the VIB derives the ELBO as a variational lower bound on the expected log-evidence , but , in addition , the VIB is a more general framework for optimization that allows us to reason about the underlying information-theoretic quantities of interest ( for details on the VIB and its notation , please see Appendix B.1 ) . Note that above definition of the multimodal ELBO requires that the complete set of modalities is available . To overcome this limitation and to learn the inference networks for different subsets of modalities , existing models use different decompositions of the joint encoder , as summarized in Table 1 . Recent work shows that existing models can be generalized by formulating the joint encoder as a mixture of product of experts ( Sutter et al. , 2021 ) . Analogously , in the following we generalize existing models to define the family of mixture-based multimodal VAEs . 3.2 THE FAMILY OF MIXTURE-BASED MULTIMODAL VAES . Now we introduce the family of mixture-based multimodal VAEs , which subsumes the MMVAE , MoPoE-VAE , and a special case of the MVAE without ELBO sub-sampling . We first define an encoder that generalizes the decompositions used by existing models : Definition 2 . Let S = { ( A , ωA ) |A ⊆ { 1 , . . . , M } , A 6= ∅ , ωA ∈ [ 0 , 1 ] } be an arbitrary set of nonempty subsets A of modalities and corresponding mixture coefficients ωA , such that ∑ A∈S ωA = 1 . Define the stochastic encoder to be a mixture distribution : pSθ ( z | x ) : = ∑ A∈S ωA pθ ( z | xA ) . In the above definition and throughout this work , we write A ∈ S to abbreviate ( A , ωA ) ∈ S. To define the family of mixture-based multimodal VAEs , we restrict the family of models optimizing the multimodal ELBO to the subfamily of models that use a mixture-based stochastic encoder . Definition 3 . The family of mixture-based multimodal VAEs is comprised of all models that maximize the multimodal ELBO using a stochastic encoder pSθ ( z | x ) that is consistent with Definition 2 . In particular , we define the family in terms of all models that maximize the following objective : LS ( x ; θ , φ ) = ∑ A∈S ωA { Ep ( x ) pθ ( z | xA ) [ log qφ ( x | z ) ] − Ep ( x ) [ DKL ( pθ ( z | xA ) || q ( z ) ) ] } . ( 2 ) 3Even though we write the expectation over p ( x ) , for the estimation of the ELBO we still assume that we only have access to a finite sample from the training distribution p ( x ) . The notation is used for consistency with the well established information-theoretic perspective on VAEs ( Alemi et al. , 2017 ; Poole et al. , 2019 ) . In Appendix B.2 , we show that the objectiveLS ( x ; θ , φ ) is a lower bound onL ( x ; θ , φ ) ( which makes it an ELBO ) and explain how , for different choices of the set of subsets S , the objective LS ( x ; θ , φ ) relates to the objectives of the MMVAE , MoPoE-VAE , and MVAE without ELBO sub-sampling . From a computational perspective , a characteristic of mixture-based multimodal VAEs is the subsampling of modalities during training , which is a direct consequence of defining the encoder as a mixture distribution over subsets of modalities . The sub-sampling of modalities can be viewed as the extraction of a subset xA ∈ x , where A indexes one subset of modalities that is drawn from the model-specific set of subsets S. The only member of the family of mixture-based multimodal VAEs that forgoes sub-sampling , defines a trivial mixture over a single subset , the complete set of modalities ( Sutter et al. , 2021 ) .

Summary: The paper discusses a surprising failure mode of multi-modal VAEs, in which their generation quality lags behind that of unimodal VAE. A theoretically grounded explanation based on the ELBO of the multi-modal VAE is proposed to explain this gap, and extensive ablation experiments validating implications of the proposed explanation are carried out. Finally, the theory and experiments proposed in the paper have implications for a large body of research into multi-modal VAEs.