Hierarchical Multimodal Variational Autoencoders

1 INTRODUCTION . Data modalities represent different perspectives of the same concept . Generative models can learn from such data by reproducing it , which can be useful for tasks such as image caption generation ( Vinyals et al. , 2015 ) . This model family can also reproduce feature vector representations of the data , which can be helpful for tasks such as zero-shot classification ( Xian et al. , 2018b ) or reinforcement learning ( Bruce et al. , 2017 ) . One difficulty in multimodal learning lies in the differing probabilistic structures across modalities ( Fig . 1 ) . Our goal is to develop generative models that capture unimodal variations in both modalities . One line of previous works ( Zhu et al. , 2017 ; Zhang et al. , 2017 ; Reed et al. , 2016 ) tackled this challenge with conditional generative adversarial networks ( GANs ) ( Mirza and Osindero , 2014 ) . These models generate samples for one modality conditioned on another modality and are optimized via competition between a generator and a discriminator . In contrast , Suzuki et al . ( 2016 ) used variational autoencoders ( VAEs ) to jointly generate M modalities from a learned latent representation . We focus on VAEs , which are explicit density estimators and can maximize the likelihood of all data variations . GANs are implicit estimators , which can cause the generator to disproportionally favor specific variations ( Razavi et al. , 2019 ) . Some multimodal VAEs ( Wu and Goodman , 2018 ; Shi et al. , 2019 ) incorporate a single latent variable g that captures all relevant information ( Fig . 2a ) . This formulation may disregard modalityexclusive variations . Other works ( Huang et al. , 2018 ; Hsu and Glass , 2018 ; Mahajan et al. , 2020 ; Sutter et al. , 2020 ; Daunhawer et al. , 2021b ; Lee and Pavlovic , 2021 ) have introduced disentangled latent variables z1 : M , which are marginally independent of a shared latent variable g and represent structure specific to modality i ( Fig . 2b ) . We argue that a disentangled latent representation may neglect the dependencies between shared and modality-exclusive variations . As a specific example , consider captioning ( modality 2 ) pictures of birds ( modality 1 ) as in Fig . 1a . The images of seabirds can have sky or water in the background , while those of songbirds often display forest backgrounds . The captions focus on the bird and thereby easily ignore such variations . Therefore , the shared pattern across modalities ( bird species ) dictates these modality-exclusive variations . Consider a generative model where g represents shared structure and z1 image variations . When the two latent variables g and z1 are independent ( Fig . 2b ) , the decoder that maps from latent variables onto images has to learn two distinct functions - one for seabirds ( where the independent variations determine the sky or water background ) and another for songbirds ( where the independent variations determine the forest background ) . We argue that this aspiration is theoretically learnable but practically challenging because it may require a large model with disproportional capacity ( that could generalize poorly , is challenging to train , or requires abundant data ) . In contrast , we suggest that a hierarchical latent representation is an inductive bias that captures realistic data variations and guides learning . The edges between g and z1 : M give the model the flexibility to decide which unimodal variations to capture given a shared latent concept . For example , a hierarchical model could adaptively learn that z1 for seabirds solely captures sky or water background variations while z1 for songbirds captures forest background variations . This can help the decoder to share features between seabirds and songbirds . Contributions ( i ) We propose a hierarchical multimodal VAE ( HMVAE ) that incorporates a latent hierarchy , where the shared variable g resides at the top and lower variables z complement unimodal variations . ( ii ) We compare the HMVAE to several state-of-the-art baselines on the CUB and the Oxford Flower datasets . We report improved quantitative and qualitative measures in terms of semantic coherence and heterogeneity . 2 BACKGROUND AND RELATED WORK . Variational Autoencoders ( VAEs ) ( Kingma and Welling , 2013 ; Rezende et al. , 2014 ) are deep generative models that represent the joint distribution pθ ( x , z ) using neural networks with parameters θ , where x ∈ RD is the observed vector and z ∈ RD′ is the latent vector . As the true posterior pθ ( z|x ) is intractable , an approximate posterior qφ ( z|x ) with parameters φ is used for inference . The parameters θ , φ are usually trained by maximizing the evidence lower bound ( ELBO ) for the marginal likelihood : Eqφ ( z|x ) [ log pθ ( x , z ) qφ ( z|x ) ] ≤ log pθ ( x ) . ( 1 ) Many efforts have been made to increase the expressivity of VAEs , e.g. , by improving the prior of z ( Chen et al. , 2016 ; Tomczak and Welling , 2018 ) , and by introducing auxiliary latent variables ( Maaløe et al. , 2016 ) . Our approach lies in the latter paradigm . Hierarchical VAEs ( HVAEs ) ( Rezende et al. , 2014 ) have a hierarchical latent structure , where the topmost latent variable , drawn from an unconditional prior pθ ( zL ) , represents global features . The lower variables , drawn from conditional priors pθ ( zi|zi+1 ) , complement local characteristics in order to reconstruct the observed data via pθ ( x|z1 ) . Sønderby et al . ( 2016 ) found the tendency for HVAEs to not effectively use higher-level latent variables when they are trained using inference networks of the form qφ ( zi+1|zi ) . They proposed to first infer the top-level variable zL directly with qφ ( zL|x ) and then infer the intermediate variables { zi } with both bottom-up and top-down information through qφ , θ ( zi|zi+1 , x ) for hierarchical level i ∈ { 0 , ... , L− 1 } . The bottom-up and top-down information for hierarchical level i are encoded as hidden variables through neural networks : bi = fφ , i ( x ) and ti = fφ , θ , i ( zi+1 ) . We make use of this idea in our work , concatenate both hidden states and pass the result to an MLP that parameterizes the respective posterior ( see App . C.1 for further details ) . This inference procedure can result in im- proved density estimation and sample generation performance ( Sønderby et al. , 2016 ; Maaløe et al. , 2019 ; Vahdat and Kautz , 2020 ; Child , 2021 ) . Multimodal VAEs represent M modalities x1 : M = { x1 , ... , xM } that are assumed to be conditionally independent given a shared representation g : pθ ( x1 : M |g ) = ∏M m=1 pθ ( xi|g ) . Motivated by the factorization of the true posterior , Wu and Goodman ( 2018 ) used a product of experts ( PoE ) formulation to parameterize the approximate posterior distribution over the shared latent variable , where each expert characterizes information within a modality : qφ ( g|x1 : M ) ∝ pθ ( g ) ∏M m=1 qφ ( g|xm ) . ( 2 ) When all experts are Gaussian , the product posterior of any combination of experts is easily computed . Shi et al . ( 2019 ) showed that an inference model with a Gaussian PoE formulation can be miscalibrated , i.e. , overconfident experts qφ ( g|xi ) with low densities dominate the product qφ ( g|x1 : M ) . They instead used a mixture of experts ( MoE ) formulation : qφ ( g|x1 : M ) = 1M ∑M m=1 qφ ( g|xm ) . ( 3 ) With the MoE formulation , optimization is analogous to a vote , which can avoid unreasonable dominance by a single modality . Mahajan et al . ( 2020 ) applied normalizing flows ( Rezende and Mohamed , 2015 ) to map between the latent spaces of g1 and g2 in a VAE , where qφ ( gi|xi ) constitutes the posterior over gi . Normalizing flows can be more expressive than other distribution choices , such as Gaussians , which can help to represent complex multimodal relationships . However , optimizing the ELBO requires tractable sampling and density estimation in either direction , for example , by using coupling layers ( Dinh et al. , 2014 ; 2017 ) . This can be a challenging constraint in practice . Vasco et al . ( 2020 ) proposed the MHVAE , which also incorporates a hierarchical generative model . Note that both works were developed independently and offer complementary perspectives . We summarize the differences in the following : first , the MHVAE ’ s generative model is limited to two hierarchical levels ( like Fig . 3a , but for two levels ) . Our proposed generative model supports arbitrary hierarchical depth . Second , the MHVAE ’ s inference model is non-hierarchical ( Fig . 2c , Eq . 9 from the original paper ) . Our proposed inference model is hierarchical ( Fig . 3b ) Third , the MHVAE ’ s posteriors over z are unimodal . Our proposed “ top-down ” inference model computes the latent variables in the same order as the generative model . Therefore , the respective posteriors depend on all modalities . Fourth , the MHVAE ’ s inference network drops modality-specific hidden states at random during training . We use a mixture of experts posterior . Fifth , Vasco et al . ( 2020 ) evaluate their model on surjective data where single labels or attributes describe classes of images , i.e. , there is not much variation in a single modality but lots in the other . We focus on a different data scenario where each modality has a large degree of variation . 3 MULTIMODAL LATENT HIERARCHIES . We propose a hierarchical multimodal VAE ( HMVAE ) that captures the generative process of multiple modalities . The hyperparameter L ( m ) ≥ 2 defines the number of hierarchical levels for modality m. If L ( m ) = 2 , the hierarchy solely expands between the shared and unimodal variables . If L ( m ) > 2 , the model incorporates additional unimodal hierarchies . Generative model We assume conditional independence of modalities given a shared variable g ( see Fig . 3a portraying the case of two modalities and three hierarchical levels ) : pφ , θ ( x , g , z ) = ∏M m=1 pθ ( xm|zm,1 ) · ( ∏L ( m ) −2 i=1 pφ , θ ( zm , i|zm , i+1 ) ) pφ , θ ( zm , L ( m ) −1|g ) pθ ( g ) . ( 4 ) The prior for the shared variable g is isotropic Gaussian . All conditional distributions for the intermediate variables { zm,1 : L ( m ) −1 } for m ∈ { 1 , . . .M } are also isotropic Gaussian , where mean and variance are parameterized using neural networks . We use the hierarchical formulation introduced in § 2 where some parameters are shared across the generative and inference networks . The conditional distributions for the observed variables { xm } can , for example , be parameterized using isotropic Gaussian distributions ( for continuous-valued data ) or Bernoulli distributions ( for binary data ) . Inference model We extend Sønderby et al . ( 2016 ) ’ s hierarchical approach described in § 2 to the multimodal case ( see Fig . 3a depicting the case of two modalities and three hierarchical levels ) : qφ , θ ( g , z|x1 : M ) = qφ ( g|x1 : M ) · M∏ m=1 qφ , θ ( zm , L ( m ) −1|g , xm ) L ( m ) −2∏ i=1 qφ , θ ( zm , i|zm , i+1 , xm ) . ( 5 ) All distributions except the first factor qφ ( g|x1 : M ) are isotropic Gaussian with mean and variance inferred through neural networks from the conditional variables . The network employs skip connections from x1 : M to g. The inference and generative networks share some parameters in the topdown networks that point from g to the lowest unimodal variable zm , i=0 . This architecture can reinforce hierarchical decomposition ( Sønderby et al. , 2016 ) . Furthermore , it ensures that the unimodal latent variables are conditioned on all modalities ( as indicated by the red edges in Fig . 3b ) which helps learn crossmodal relationships . Section 2 and App . C.1 provide further details on the hierarchical architecture . We parameterize the posterior distribution over the shared latent variable qφ ( g|x1 : M ) using a mixture of experts formulation ( Eq . 3 ) . We follow Shi et al . ( 2019 ) and assume that several modalities can entail modality-exclusive variations , i.e. , some variations for modality i do not correlate with variations in modality j 6= i. Optimization We train the HMVAE by maximizing the ELBO with stochastic backpropagation : ELBO : = Eqφ , θ ( g , z|x1 : M ) [ log pφ , θ ( x1 : M , g , z ) qφ , θ ( g , z|x1 : M ) ] ≤ log pθ ( x1 : M ) . ( 6 ) Further motivation Figure 4 visualizes how multimodal VAEs could capture the data from the example in the Introduction ( Fig . 1 ) . The challenge lies in representing crossmodal dependencies – not the joint distribution . That is because one modality can never inform about the modality-exclusive variations of another modality . Daunhawer et al . ( 2021a ) demonstrated that this problem constitutes a core limitation across the multimodal VAE literature . In a non-hierarchical VAE , crossmodal generation is indeed challenging because g must capture all variations . This becomes problematic when generating pθ ( xi|g ) from qφ ( g|xj 6=i ) because the latter misses information about xi . In contrast , in a two-level hierarchy , z could theoretically capture the entirety of modality-exclusive variations . However , the model may choose to represent some of these variations in g to capture the hierarchical dependencies within the modality-exclusive variations . A deep hierarchy could circumvent this Problem : crossmodal generation Problem : conditional independent variations within z problem by adding infinitesimal small modality-exclusive variations per hierarchical level ( in the limit ) . Note that Fig . 4 demonstrates one set of possible hierarchical representations . A model could also utilize its degrees of freedom differently . For example , the deep model may incorporate shared structure and modality-exclusive variations for the more complex modality in g. In general , we expect a ( deep ) hierarchy to be beneficial for representing complex data such as image or text modalities .

Summary: The authors propose a Hierarchical framework for multimodal learning HMVAE. They define modality specific latent factor as well as the shared latent factor across multiple modalities. They represent modality-specific variations using latent variables dependent on the shared top-level variable. They parameterize the posterior distribution over the shared latent variable using a mixture of experts. The modality specific latent factors are adaptive inferred with both bottom-up and top-down information. They evaluate the proposed method on the Oxford Flower and the CUB datasets with various cross-modal experiments.