Year: 2021

Abstract:
Humans find structure in natural phenomena by absorbing stimuli from multiple input sources such as vision, text, and speech. We study the use of deep generative models that generate multimodal data from latent representations. Existing approaches generate samples using a single shared latent variable, sometimes with marginally independent latent variables, to capture modality-specific variations. However, there are cases where modality-specific variations depend on the kind of structure shared across modalities. To capture such heterogeneity, we propose a hierarchical multimodal VAE (HMVAE) that represents modality-specific variations using latent variables dependent on a shared top-level variable. Our experiments on the CUB and the Oxford Flower datasets show that the HMVAE can outperform existing methods in terms of generative heterogeneity and coherence across several quantitative and qualitative measures. We provide the code to reproduce the results in the supplementary material.

heading: 1 INTRODUCTION
text: Data modalities represent different perspectives of the same concept. Generative models can learn from such data by reproducing it, which can be useful for tasks such as image caption generation (Vinyals et al., 2015). This model family can also reproduce feature vector representations of the data, which can be helpful for tasks such as zero-shot classification (Xian et al., 2018b) or reinforcement learning (Bruce et al., 2017). One difficulty in multimodal learning lies in the differing probabilistic structures across modalities (Fig. 1). Our goal is to develop generative models that capture unimodal variations in both modalities.
One line of previous works (Zhu et al., 2017; Zhang et al., 2017; Reed et al., 2016) tackled this challenge with conditional generative adversarial networks (GANs) (Mirza and Osindero, 2014). These models generate samples for one modality conditioned on another modality and are optimized via competition between a generator and a discriminator. In contrast, Suzuki et al. (2016) used variational autoencoders (VAEs) to jointly generate M modalities from a learned latent representation. We focus on VAEs, which are explicit density estimators and can maximize the likelihood of all data variations. GANs are implicit estimators, which can cause the generator to disproportionally favor specific variations (Razavi et al., 2019).
Some multimodal VAEs (Wu and Goodman, 2018; Shi et al., 2019) incorporate a single latent variable g that captures all relevant information (Fig. 2a). This formulation may disregard modalityexclusive variations. Other works (Huang et al., 2018; Hsu and Glass, 2018; Mahajan et al., 2020; Sutter et al., 2020; Daunhawer et al., 2021b; Lee and Pavlovic, 2021) have introduced disentangled latent variables z1:M , which are marginally independent of a shared latent variable g and represent structure specific to modality i (Fig. 2b). We argue that a disentangled latent representation may neglect the dependencies between shared and modality-exclusive variations.
As a specific example, consider captioning (modality 2) pictures of birds (modality 1) as in Fig. 1a. The images of seabirds can have sky or water in the background, while those of songbirds often display forest backgrounds. The captions focus on the bird and thereby easily ignore such variations. Therefore, the shared pattern across modalities (bird species) dictates these modality-exclusive variations. Consider a generative model where g represents shared structure and z1 image variations. When the two latent variables g and z1 are independent (Fig. 2b), the decoder that maps from latent variables onto images has to learn two distinct functions - one for seabirds (where the independent variations determine the sky or water background) and another for songbirds (where the independent variations determine the forest background). We argue that this aspiration is theoretically learnable but practically challenging because it may require a large model with disproportional capacity (that could generalize poorly, is challenging to train, or requires abundant data). In contrast, we suggest that a hierarchical latent representation is an inductive bias that captures realistic data variations and guides learning. The edges between g and z1:M give the model the flexibility to decide which unimodal variations to capture given a shared latent concept. For example, a hierarchical model could adaptively learn that z1 for seabirds solely captures sky or water background variations while z1 for songbirds captures forest background variations. This can help the decoder to share features between seabirds and songbirds.
Contributions (i) We propose a hierarchical multimodal VAE (HMVAE) that incorporates a latent hierarchy, where the shared variable g resides at the top and lower variables z complement unimodal variations. (ii) We compare the HMVAE to several state-of-the-art baselines on the CUB and the Oxford Flower datasets. We report improved quantitative and qualitative measures in terms of semantic coherence and heterogeneity.

heading: 2 BACKGROUND AND RELATED WORK
text: Variational Autoencoders (VAEs) (Kingma and Welling, 2013; Rezende et al., 2014) are deep generative models that represent the joint distribution pθ(x, z) using neural networks with parameters θ, where x ∈ RD is the observed vector and z ∈ RD′ is the latent vector. As the true posterior pθ(z|x) is intractable, an approximate posterior qφ(z|x) with parameters φ is used for inference. The parameters θ, φ are usually trained by maximizing the evidence lower bound (ELBO) for the marginal likelihood:
Eqφ(z|x) [ log pθ(x,z)qφ(z|x) ] ≤ log pθ(x). (1)
Many efforts have been made to increase the expressivity of VAEs, e.g., by improving the prior of z (Chen et al., 2016; Tomczak and Welling, 2018), and by introducing auxiliary latent variables (Maaløe et al., 2016). Our approach lies in the latter paradigm.
Hierarchical VAEs (HVAEs) (Rezende et al., 2014) have a hierarchical latent structure, where the topmost latent variable, drawn from an unconditional prior pθ(zL), represents global features. The lower variables, drawn from conditional priors pθ(zi|zi+1), complement local characteristics in order to reconstruct the observed data via pθ(x|z1). Sønderby et al. (2016) found the tendency for HVAEs to not effectively use higher-level latent variables when they are trained using inference networks of the form qφ(zi+1|zi). They proposed to first infer the top-level variable zL directly with qφ(zL|x) and then infer the intermediate variables {zi} with both bottom-up and top-down information through qφ,θ(zi|zi+1,x) for hierarchical level i ∈ {0, ..., L− 1}. The bottom-up and top-down information for hierarchical level i are encoded as hidden variables through neural networks: bi = fφ,i(x) and ti = fφ,θ,i(zi+1). We make use of this idea in our work, concatenate both hidden states and pass the result to an MLP that parameterizes the respective posterior (see App. C.1 for further details). This inference procedure can result in im-
proved density estimation and sample generation performance (Sønderby et al., 2016; Maaløe et al., 2019; Vahdat and Kautz, 2020; Child, 2021).
Multimodal VAEs represent M modalities x1:M = {x1, ...,xM} that are assumed to be conditionally independent given a shared representation g: pθ(x1:M |g) = ∏M m=1 pθ(xi|g). Motivated by the factorization of the true posterior, Wu and Goodman (2018) used a product of experts (PoE) formulation to parameterize the approximate posterior distribution over the shared latent variable, where each expert characterizes information within a modality:
qφ(g|x1:M ) ∝ pθ(g) ∏M m=1 qφ(g|xm). (2)
When all experts are Gaussian, the product posterior of any combination of experts is easily computed.
Shi et al. (2019) showed that an inference model with a Gaussian PoE formulation can be miscalibrated, i.e., overconfident experts qφ(g|xi) with low densities dominate the product qφ(g|x1:M ). They instead used a mixture of experts (MoE) formulation:
qφ(g|x1:M ) = 1M ∑M m=1 qφ(g|xm). (3)
With the MoE formulation, optimization is analogous to a vote, which can avoid unreasonable dominance by a single modality.
Mahajan et al. (2020) applied normalizing flows (Rezende and Mohamed, 2015) to map between the latent spaces of g1 and g2 in a VAE, where qφ(gi|xi) constitutes the posterior over gi. Normalizing flows can be more expressive than other distribution choices, such as Gaussians, which can help to represent complex multimodal relationships. However, optimizing the ELBO requires tractable sampling and density estimation in either direction, for example, by using coupling layers (Dinh et al., 2014; 2017). This can be a challenging constraint in practice.
Vasco et al. (2020) proposed the MHVAE, which also incorporates a hierarchical generative model. Note that both works were developed independently and offer complementary perspectives. We summarize the differences in the following: first, the MHVAE’s generative model is limited to two hierarchical levels (like Fig. 3a, but for two levels). Our proposed generative model supports arbitrary hierarchical depth. Second, the MHVAE’s inference model is non-hierarchical (Fig. 2c, Eq. 9 from the original paper). Our proposed inference model is hierarchical (Fig. 3b) Third, the MHVAE’s posteriors over z are unimodal. Our proposed “top-down” inference model computes the latent variables in the same order as the generative model. Therefore, the respective posteriors depend on all modalities. Fourth, the MHVAE’s inference network drops modality-specific hidden states at random during training. We use a mixture of experts posterior. Fifth, Vasco et al. (2020) evaluate their model on surjective data where single labels or attributes describe classes of images, i.e., there is not much variation in a single modality but lots in the other. We focus on a different data scenario where each modality has a large degree of variation.

heading: 3 MULTIMODAL LATENT HIERARCHIES
text: We propose a hierarchical multimodal VAE (HMVAE) that captures the generative process of multiple modalities. The hyperparameter L(m) ≥ 2 defines the number of hierarchical levels for modality m. If L(m) = 2, the hierarchy solely expands between the shared and unimodal variables. If L(m) > 2, the model incorporates additional unimodal hierarchies.
Generative model We assume conditional independence of modalities given a shared variable g (see Fig. 3a portraying the case of two modalities and three hierarchical levels):
pφ,θ(x, g, z) = ∏M m=1 pθ(xm|zm,1)
· (∏L(m)−2 i=1 pφ,θ(zm,i|zm,i+1) )
pφ,θ(zm,L(m)−1|g) pθ(g). (4)
The prior for the shared variable g is isotropic Gaussian. All conditional distributions for the intermediate variables {zm,1:L(m)−1} for m ∈ {1, . . .M} are also isotropic Gaussian, where mean and variance are parameterized using neural networks. We use the hierarchical formulation introduced in § 2 where some parameters are shared across the generative and inference networks. The conditional distributions for the observed variables {xm} can, for example, be parameterized using isotropic Gaussian distributions (for continuous-valued data) or Bernoulli distributions (for binary data).
Inference model We extend Sønderby et al. (2016)’s hierarchical approach described in § 2 to the multimodal case (see Fig. 3a depicting
the case of two modalities and three hierarchical levels):
qφ,θ(g, z|x1:M ) = qφ(g|x1:M )· M∏ m=1 qφ,θ(zm,L(m)−1|g,xm) L(m)−2∏ i=1 qφ,θ(zm,i|zm,i+1,xm). (5)
All distributions except the first factor qφ(g|x1:M ) are isotropic Gaussian with mean and variance inferred through neural networks from the conditional variables. The network employs skip connections from x1:M to g. The inference and generative networks share some parameters in the topdown networks that point from g to the lowest unimodal variable zm,i=0. This architecture can reinforce hierarchical decomposition (Sønderby et al., 2016). Furthermore, it ensures that the unimodal latent variables are conditioned on all modalities (as indicated by the red edges in Fig. 3b) which helps learn crossmodal relationships. Section 2 and App. C.1 provide further details on the hierarchical architecture.
We parameterize the posterior distribution over the shared latent variable qφ(g|x1:M ) using a mixture of experts formulation (Eq. 3). We follow Shi et al. (2019) and assume that several modalities can entail modality-exclusive variations, i.e., some variations for modality i do not correlate with variations in modality j 6= i. Optimization We train the HMVAE by maximizing the ELBO with stochastic backpropagation:
ELBO := Eqφ,θ(g,z|x1:M ) [ log pφ,θ(x1:M ,g,z) qφ,θ(g,z|x1:M ) ] ≤ log pθ(x1:M ). (6)
Further motivation Figure 4 visualizes how multimodal VAEs could capture the data from the example in the Introduction (Fig. 1). The challenge lies in representing crossmodal dependencies – not the joint distribution. That is because one modality can never inform about the modality-exclusive variations of another modality. Daunhawer et al. (2021a) demonstrated that this problem constitutes a core limitation across the multimodal VAE literature. In a non-hierarchical VAE, crossmodal generation is indeed challenging because g must capture all variations. This becomes problematic when generating pθ(xi|g) from qφ(g|xj 6=i) because the latter misses information about xi. In contrast, in a two-level hierarchy, z could theoretically capture the entirety of modality-exclusive variations. However, the model may choose to represent some of these variations in g to capture the hierarchical dependencies within the modality-exclusive variations. A deep hierarchy could circumvent this
Problem: crossmodal generation
Problem: conditional independent variations within z
problem by adding infinitesimal small modality-exclusive variations per hierarchical level (in the limit). Note that Fig. 4 demonstrates one set of possible hierarchical representations. A model could also utilize its degrees of freedom differently. For example, the deep model may incorporate shared structure and modality-exclusive variations for the more complex modality in g. In general, we expect a (deep) hierarchy to be beneficial for representing complex data such as image or text modalities.

heading: 4 EXPERIMENTS
text: We evaluate on the CUB (Wah et al., 2011) and Oxford Flower (Nilsback and Zisserman, 2008) datasets. The modalities in both datasets are images and captions (text). We will focus on crossmodal generation, i.e., caption-to-image generation and the reverse. We also consider unconditional generation. We evaluate generated samples mainly in terms of heterogeneity and coherence. Heterogeneity describes diversity across samples for a single setting of the shared latent variable g and is essential to ensure that the generated samples capture the diversity of the data. Coherence indicates whether a crossmodal sample lies on the manifold of the true data. Crucially, heterogeneity and coherence are often both necessary. Imagine a model that produces class-agnostic generations (high heterogeneity, low coherence for classification tasks). As for the HMVAE, we will validate that the model hierarchically decomposes the data and outperforms state-of-the-art competitors.

heading: 4.1 EXPERIMENTAL DESIGN
text: Models We are not aware of any standard benchmark dataset in the literature with reported performance for all considered baselines and sufficient hierarchical structure in the data. Therefore, we reimplement all considered baselines: the MVAE (Wu and Goodman, 2018), the MMVAE (Shi et al., 2019), the MDVAE (Huang et al., 2018; Hsu and Glass, 2018; Mahajan et al., 2020; Sutter et al., 2020; Daunhawer et al., 2021b; Lee and Pavlovic, 2021), and the HMVAE (Vasco et al., 2020). Related works suggested many different MDVAEs variants. Our reimplementation uses a posterior choice suitable to the given data regime and stays close to the HMVAE– except that z are marginally independent of g. In general, we keep the architecture and optimization hyperparameters similar across all models. App. C describes our implementation in detail. The use of larger models may produce photorealistic images with a high computational cost. For example, Child (2021) train a unimodal hierarchical VAE for around 2.5 weeks using 32 V100 GPUs on datasets such as FFHQ. However, our goal is not photorealism but to validate that the proposed HMVAE represents multimodal data differently than the baselines.
Data In some of our experiments, we follow the common practice in the multimodal learning literature and preprocess the data by extracting feature vector representations (Xian et al., 2018b; Sariyildiz and Cinbis, 2019; Schonfeld et al., 2019; Shi et al., 2019; 2020). Across all experiments, we use the caption feature vectors provided by Zhang et al. (2017). These feature vectors x2 ∈ R1024 were extracted using a CNN-RNN (Reed et al., 2016). Each image is paired with ten captions. We average the caption features for each image (except for the qualitative experiments which visualize captions explicitly). We find that this procedure can improve training and reduce computational requirements. In § 4.2, we extract features x1 ∈ R2048 from a ResNet-101 trained on ImageNet. In § 4.3, we use the images x1 ∈ R3×64×64 directly. We refer to App. C for details. To visualize image features and caption features, we follow Shi et al. (2019; 2020), search for the nearest neighbor feature vector in the test set using the mean from p(xi|·), and visualize the respective image or caption. Using this practice, we can evaluate coherence (relationship between generation and condition) and heterogeneity (sample diversity). To improve readability, we trim exceptionally long captions in the qualitative figures.

heading: 4.2 FEATURE GENERATION
text: In this section, we maximize the likelihood of image features x1 ∈ R2048 and caption features x2 ∈ R1024 on the CUB and the Oxford Flower datasets. We will validate the general utility of latent hierarchies relative to single-variable models. For the HMVAE, we choose L(1) = L(2) = 2.
Figures 5 and 6 display p(x1|x2) and p(x2|x1), respectively. For the baselines, we vary g to generate xi, where g1:K ∼ q(g|xj 6=i) and p(xi|gk). For the proposed HMVAE, we utilize the hierarchy by varying zi to generate xi, where g ∼ q(g|xj 6=i), z1:Ki ∼ p(zi|g), and p(xi|zki ). All models generate features that are mainly semantically coherent with the condition, indicating high coherence. However, the proposed HMVAE generates samples of higher diversity than the baseline, indicating high heterogeneity. This diversity must be represented in z1, because we fix g and vary z1. Table 1 demonstrates that the HMVAE achieves state-of-the-art performance across most evaluated likelihoods. Likelihood estimates quantify both coherence and heterogeneity because they measure the divergence between the true and approximated distribution.

heading: 4.3 IMAGE GENERATION
text: In this section, we maximize the likelihood of images x1 ∈ R3×64×64 and caption feature vectors x2 ∈ R1024. Because the data is more complex than in the previous section, we now investigate deeper hierarchies for the HMVAE1 and incorporate additional baselines with multiple latent variables (MDVAE and MHVAE). Note that the deep HMVAE employs the fewest parameters (see Table 4 for an overview) and is the only model that does not use a warmup scheme for the KLdivergence loss (Sønderby et al., 2016) .
Quantitative results The Fréchet Inception Distance (FID) compares the means and covariances between the true and approximate distributions in the feature space of an Inception network (Heusel
1The shallow variant incorporates L(1) = L(2) = 2, the deeper variant incorporates L(1) = 5 and L(2) = 2.
et al., 2017; Seitzer, 2020). For the remaining measures, we preprocess images by extracting feature representations from a ResNet-101 pre-trained on ImageNet. For example, we generate images x1 ∈ R3×64×64 and then project these samples into the mentioned feature space. We do not use any preprocessing for the caption features because of their lower dimensionality (1024). Precision measures the relative amount of generated samples that lie on the true manifold. Recall measures the opposite. We follow (Kynkäänniemi et al., 2019) and approximate the target manifold using hyperspheres with radii equivalent to the 3-nearest-neighbors of the respective samples. We also compute the harmonic mean across precision and recall (F1). Finally, we compute the average variance of image representations: for each conditioning caption, we sample g1:100 ∼ q(g|x2) to generate p(x1|g) and project these samples into the ResNet feature space f(x1|g). We then compute the variance of these samples to a prototype representation f(x̄2|g) = 1100 ∑100 i=1 f(x i 2|g).
Table 2a presents quantitative results.2 The MHVAE and the HMVAE are the only models that perform reasonably well across all evaluated measures. For example, even though the MVAE achieves the best FID, the model scores poorly in the precision measures. Furthermore, the model’s qualitative samples display a lack of fidelity (Fig. 7). In contrast, the MMVAE generates coherent samples (high precision) but lacks heterogeneity (low variance and recall). The HMVAE also uses a mixture
2Note that we report the image recall and F1 values for the sake of completeness. These values are low across all models due to the complexity of the image space given a relatively small dataset.
of experts posterior as the MMVAE, but adds hierarchical depth which correlates positively with variance and recall. Finally, we observe that the HMVAE improves the MHVAE across measures related to coherence (FID, image precision) and heterogeneity (FID, caption recall). Table 2b examines hierarchical decomposition in the HMVAE. We first sample conventionally from q(g|xj 6=i). We then generate p(xi|g) by using the conditional priors in the latent hierarchy for xi. Normally, we would sample conventionally from each of these latent distributions (rows 2 and 7). However, we
also evaluate models where we propagate the latent distribution means corresponding to the latent variables from the left column. In other words, we omit variations in some hierarchical levels. For the experiment on the image hierarchy, all measures improve with increased hierarchical expressivity. For the experiment on the caption hierarchy, greater hierarchical capacity is beneficial (F1) by increasing recall and decreasing precision. Increased recall is expected to correlate negatively with precision: generating a larger manifold typically increases the likelihood of generated samples beyond the target manifold.
Qualitative results Figure 7 focuses on p(x1|x2). We generate p(x1|gk) via g1:K ∼ q(g|x2). For the MDVAE, we must additionally sample from the disentangled variable and hence generate images p(x1|z1, g) from z1 ∼ p(z1) and g ∼ q(g|x2). Note that we always sample from the (conditional) priors over z across all multi-latent-variable models3. The MVAE’s samples display a noticeable lack of coherence and fidelity which is in line with the results from Shi et al. (2019). The MMVAE’ samples have high coherence and low heterogeneity. The MDVAE’s samples display precision but misalign some unimodal variations with the shared concept. The MHVAE’s samples sometimes lack semantic coherence. The shallow HMVAE generates samples of good coherence and low heterogeneity. The deep HMVAE generates the most compelling samples relative to the baselines (as also indicated by the FID from Table 2a). Figure 7 presents the counterpart for the quantitative hierarchical ablation experiment from Table 2b. This figure demonstrates that the HMVAE decomposes image variation along its hierarchy.

heading: 5 CONCLUSION
text: We have proposed a hierarchical multimodal VAE (HMVAE) where unimodal latent hierarchies depend on a shared latent variable. We have demonstrated that the model improves generative modeling performance on multimodal data. Future work may further improve sample quality, for example, by incorporating different ways to parameterize the conditional densities: normalizing flows could improve the posterior’s flexibility (Kingma et al., 2016), autoregressive decoders could explicitly represent the conditional dependencies across observed dimensions (Chen et al., 2018), and the transformer architecture may be a good candidate to replace the convolutional neural networks (Dosovitskiy et al., 2021). Note that we discuss ethical implications in App. E.
Limitations Our goal is the the representation of heterogeneity within modalities. Although the proposed HMVAE outperforms our baselines, the model is not perfect. For example, when generating images by sampling from the conditional distribution p(x1|z1), where z1 ∼ p(z1|g) and g ∼ q(g|x2), the HMVAE must hallucinate a coherent set of visual representations (possibly conditional on a given caption). This can be challenging when the caption is vague and does not contain sufficient information for the inference network to infer the shared underlying structure correctly. In other words, the additional expressivity that the HMVAE obtains through the conditional latent variables also makes learning harder since p(x1|·) must represent a larger manifold for the proposed HMVAE than it does for the baselines.

heading: APPENDIX
text: We partition the Appendix into five sections:
A. Additional experiments: For the experiments from § 4.2, we provide further results on crossmodal generation. For the experiments from § 4.3, we provide results on unconditional generation.
B. Likelihood estimation: This section presents the estimators used to compute the likelihoods reported in Table 1.
C. Implementation: This section describes our implementation in detail. App. C.1 complements the background section § 2 by describing details of the employed hierarchical VAE architecture. This section also presents dataset-agnostic model specifications for the baselines. We then focus on dataset-specific implementation specifications, where App. C.2 covers feature generation on the CUB dataset, App. C.3 feature generation on the Oxford Flower dataset, and App. C.4 image generation on the Oxford Flower dataset.
D. MDVAE: inference networks: This section discusses an alternative inference network choice for multimodal disentanglement VAEs (MDVAEs).
E. Ethical statement: This section discusses ethical implications of the proposed model, e.g., its societal impact.

heading: A ADDITIONAL EXPERIMENTS
text: This section presents additional experimental results that complement § 4 from the main paper.

heading: A.1 FEATURE GENERATION
text: In this section, we maximize the likelihood of image features x1 ∈ R2048 and caption features x2 ∈ R1024 on the CUB dataset and the Oxford Flower dataset. We use the same experimental setup as in § 4. Image feature generation Figure 8a presents generated samples where the captions are from the training set. We observe that the MMVAE still lacks variety in its generations, which indicates that this tendency is not caused by poor generalization. Figure 8b presents generated samples where the captions are from the test set. The HMVAE can represent different backgrounds (e.g., various sea or forest settings), even though this information is missing in the conditioning captions.
Caption feature generation Figure 9 presents additional results for caption feature generation which confirm our findings from § 4.
A.2 IMAGE GENERATION
In this section, we report additional results for § 4.3. Figure 10 visualizes unconditional samples. We observe that the HMVAE can generate diverse samples without relying on any conditioning.

heading: B LIKELIHOOD ESTIMATION
text: In this section, we formalize the estimators that we use to evaluate the HMVAE, the MVAE (Wu and Goodman, 2018), and the MMVAE (Shi et al., 2019). We report all likelihoods in nats.

heading: B.1 JOINT LIKELIHOOD
text: We now focus on the joint likelihood over M modalities. The special case with M = 1 describes the unimodal likelihoods p(xi).
Single latent variable We now discuss models that employ a single shared variable g. We approximate the likelihood using K importance samples:
log p(x1:M ) ≈ log 1
K K∑ k=1 p(x1:M , g k) q(gk|x1:M ) where g1:K ∼ q(g|x1:M ) (7)
We follow Shi et al. (2019), evaluate the mixture of experts VAE using stratified sampling, and estimate a tighter estimate, which averages over modalities inside the logarithm. This objective should only be used during the evaluation because it may lead to an outsized impact of more informative modalities during optimization. Notably, K samples of the joint posterior are equivalent to T = K/M samples from each modality-specific posterior.
log p(x1:M ) ≈ log 1
M M∑ m=1 1 T T∑ t=1 p(gtm,x1:M ) q(gtm|xm) where g1:Tm ∼ q(g|xm) (8)
Latent hierarchy We now discuss models that employ both a shared variable g and conditionally independent, modality-specific variables zm,i. We approximate the likelihood using K importance samples and omit the super- and subscripts in the sampling definition for improved readability:
log p(x1:M ) ≈ log 1
K K∑ k=1 p(x1:M , g k, z)
q(gk, z|x1:M ) where g, z ∼ q(g, z|x1:M )
(9)
The factorized generative and inference networks for the proposed latent hierarchy of the HMVAE are formalized in Eqs. 4 and 5 of the main paper. Note that both product and mixture of experts formulations of the shared posterior can be used within these hierarchical formulations.

heading: B.2 CROSSMODAL LIKELIHOOD
text: In this section, we discuss estimators for the crossmodal likelihood p(xt|xc) given a target modality t and a conditioning modality c. Analogously to Eq. 7, we evaluate the likelihoods using K importance samples.

heading: Single latent variable
text: We first formalize the conditional likelihood similarly to Suzuki et al. (2016) and Wu and Goodman (2018):
log p(xt|xc) = log ∫ g p(xt, g|xc)dg = log ∫ g p(xt,xc, g) p(xc) dg
= logEq(g|xc) [ p(xt,xc, g)
q(g|xc) ] ︸ ︷︷ ︸
1
− logEq(g|xc) [ p(xc, g)
q(g|xc) ] ︸ ︷︷ ︸
2
(10)
In Eq. 10, term 1 constitutes an estimation of the joint likelihood p(x1:M ) using an importance distribution that is conditioned on modality c. Term 2 estimates the unimodal likelihood p(xc).
Latent hierarchy We now generalize the estimator from Eq. 10 to the proposed latent hierarchy of the HMVAE. We first formalize the conditional likelihood:
p(xt|xc) = ∫ g,z p(xt, g, z|xc)dzdg = ∫ g,z p(xt, g, z,xc) p(xc) dzdg (11)
We assume that the modality-specific generative networks are conditionally independent given the shared representation (§ 3):
p(xt|xc) = ∫ g,z p(xt, zt|g)p(xc, zc|g)p(g) p(xc) dzdg (12)
We then define the importance distributions as the posterior over g and the conditional prior over z. The method must learn p(xt|zt) with zt being solely conditioned on the other modality c. This formulation allows a fair comparison to the non-hierarchical baseline, which must do the same. In contrast, an importance distribution q(zt|xt, g) would make the conditioning of p(xt|zt) multimodal. log p(xt|xc) = logEq(g,zc|xc),p(zt|g) [ p(xt|zt, g)︸ ︷︷ ︸
1
p(xc, g, zc) q(g, zc|xc)
] − logEq(g,zc|xc) [ p(xc, g, zc)
q(g, zc|xc) ] (13)
In Eq. 13, we can amortize computation for the unimodal likelihood p(xc) which was defined in App. B.1. Term 1 is the only one that requires additional forward passes through the network. Note that the remaining terms do not depend on the adjusted importance distribution p(zt|g).
C IMPLEMENTATION
C.1 GENERAL SPECIFICATIONS
Hierarchical models The following formalizes latent distributions assuming the unimodal hierarchical VAE inspired by Sønderby et al. (2016) from § 2. We start with the conditional isotropic Gaussian prior
pφ,θ(zi|zi+1) = N (zi|µθ(dφ,θ,i(zi+1)), σθ(dφ,θ,i(zi+1))), (14)
where i ∈ {1, ..., L} refers to the hierarchical level. The MLP dφ,θ,i represents the top-down network, which is partially shared across the inference and generative networks. Its output is passed to the stochastic layers µθ and σθ that parameterize the prior distribution.
We then focus on the more complicated case of inferring the isotropic Gaussian posterior qφ,θ(zi|zi+1,x) where i < L for L latent variables (Fig. 11). We first infer the hidden states for zi+1, x and for hierarchical level i ∈ {1, ..., L}, where dθ,φ is the top-down MLP and dφ the bottom-up MLP:
hi,t = dφ,θ,i(zi+1), hi,b = dφ,i(x). (15)
We concatenate both hidden states along the first dimension and pass the result through another neural networkmφ to create the joint representation. We denote the concatenation operation as 〈·, ·〉:
hi,j = mφ(〈hi,t,hi,b〉). (16)
This joint representation is input to the stochastic layers µφ and σφ that parameterize the posterior distribution:
qφ,θ(zi|zi+1,x) = N (zi|µφ(hi,j), σφ(hi,j)). (17)
The remaining priors and posteriors are parameterized similarly. Appendix A from Maaløe et al. (2019) provides further background on unimodal hierarchical VAEs that inspire this work.
MVAE and MMVAE We follow Shi et al. (2019), train the MMVAE by maximizing a loose bound and evaluate using a tight estimate as described in App. B.1. The following describes exemplary differences in implementation compared to the original formulations:
• We employ a slightly different network architecture for the MVAE and the MMVAE (Wu and Goodman, 2018) relative to their original formulation, respectively. For example, we do not include dropout regularization across all implementations. • For the MMVAE, we neither employ importance weighted autoencoders (Burda et al.,
2015) nor use a doubly reparameterized gradient estimator (Tucker et al., 2018). Note that these techniques increase computational requirements. Neither the MVAE nor the HMVAE relies on such measures. • We do not use Laplace latent distributions in the MMVAE, but Gaussian latent distributions
as in all implemented methods.

heading: C.2 FEATURE GENERATION: CUB DATASET
text: Data The CUB dataset includes an image and a caption modality. For the images, we use the features vectors x1 ∈ R2048 provided by Xian et al. (2018a), who employed a ResNet-101 (He et al., 2016) trained on ImageNet (Deng et al., 2009). As described in § 4, we use caption features x2 ∈ R1024. We divide the dataset into three splits that do not share classes. The training set consists of 111 classes. The validation split consists of 39 classes and the test split contains 50 classes. We train the model on the training set and tune hyperparameters on the validation set. We then evaluate the trained model on the test set, i.e., we do not use the validation set for training as in some previous works. We use the same test split as Reed et al. (2016) who provided the CNN-RNN feature extractor for the captions, which is essential to avoid test data leakage. We standardize the features for both modalities, i.e., we subtract the mean µ and divide by the standard deviation σ. We compute both parameters on the training set. We assume that such preprocessing can be helpful, because the models generate a Gaussian likelihood p(xi|·). All models The generative models maximize the likelihood of the data under learned Gaussian distributions. We train the methods for 100 epochs with a learning rate of 1e-4 and a batch size of 256. For each datapoint in a batch, we use ten samples from the respective shared posterior (i.e., g1:10 ∼ q(g|x1:M )) during training. We gradually increase the KL-regularization weighting factor from zero to one for a warm-up period of 25 (Sønderby et al., 2016).
Proposed HMVAE The hierarchical VAEs are visualized in Fig. 12a and Fig. 12b. The shared latent space g contains 16 dimensions.
MVAE and MMVAE The models are visualized in Table 3 and Table 3. We share the architecture across baselines. Those models are similar in their architecture to the hierarchical method, but lack the hierarchical components. We tested a version with additional layers to compensate for this reduced capacity. However, models with more capacity (i.e., more weights) did not improve the performance of the flat models. The shared latent space g contains eight dimensions.

heading: C.3 FEATURE GENERATION: OXFORD FLOWER DATASET
text: Data The Oxford Flower dataset includes an image and a caption modality. We use feature vector representations of the images, where x1 ∈ R2048, and of the captions, where x2 ∈ R1024. We standardize the features for both modalities, i.e., we subtract the meanµ and divide by the standard deviation σ. We compute both parameters on the training set. We assume that such preprocessing can be helpful, because the respective models generate a Gaussian likelihood p(xi|·). We divide the dataset into three splits that do not share classes. The training set consists of 62 classes. Both validation and test split contain 20 classes. We train the model on the training set and tune hyperparameters on the validation set. We then evaluate the trained model on the test set, i.e., we do not use the validation set for training as in some previous works. We use the same test split as Reed et al. (2016) who provided the CNN-RNN feature extractor for the captions, which is essential to avoid test data leakage.
All models The VAEs maximize the likelihood of each modality under a learned Gaussian distribution. All models are trained for 100 epochs. We gradually increase the KL-regularization weighting factor from zero to one for a warm-up period of 25 (Sønderby et al., 2016). We train the models with a learning rate of 1e-4 and a batch size of 256. For each datapoint in a batch, we use ten samples from the respective shared posterior (i.e., g1:10 ∼ q(g|x1:M )) during training. Proposed HMVAE Figure 12a displays the image VAE, Fig. 12b shows the caption feature vector VAE.
MVAE and MMVAE Table 3 displays the image VAE, Table 3 shows the caption feature vector VAE.

heading: Model Name # LV # Params
text: C.4 IMAGE GENERATION: OXFORD FLOWER DATASET
Data We preprocess the images by first making them quadratic (using CenterCrop in PyTorch) and then resizing them to x1 ∈ R3×64×64. Note that we described general specifications in App. C.3. All models We train the models with a learning rate of 1e-4 and a batch size of 64.
Proposed HMVAE (shallow hierarchy) Figure 13a portrays the image VAE. Figure 13b presents the hierarchical VAE for the other modality. We gradually increase the KL-regularization weighting factor from zero to one for a warm-up period of 20 (Sønderby et al., 2016).
Proposed HMVAE (deep hierarchy) To insert further hierarchical levels, we “slice” deterministic blocks. We then add stochastic layers and respective pre- and postprocessing layers for each hierar-
chical level. Note that these layers often add just a few trainable parameters because of 1x1 convolutions or small channel sizes for the stochastic variables. Note that the deeper model uses convolutions blocks of three layers instead of single convolutional layers. Input and output channel dimension are identical on the outside and reduced within the block. We fix the channel size across the deterministic parts of the model and gradually down- or upsample the spatial dimension. We downsample with average pooling and upsample using nearest-neighbor interpolation (Child, 2021). We fix the channels for the latent variables along the image hierarchy to 16 and solely adjust the spatial dimensions. We do not use any warm-up scheme for the KL-divergence term in the loss, i.e., we use the natural ELBO.
MVAE and MMVAE The MVAE (Wu and Goodman, 2018) and MMVAE (Shi et al., 2019) share their basic architecture. Table 5 displays the image VAE, Table 5 shows the caption feature VAE. We add extra capacity to the decoder to compensate for the missing parameters from hierarchical components. We gradually increase the KL-regularization weighting factor from zero to one for a warm-up period of 20 (Sønderby et al., 2016).
MDVAE There are many implementation choices for MDVAEs (Huang et al., 2018; Hsu and Glass, 2018; Mahajan et al., 2020; Sutter et al., 2020; Daunhawer et al., 2021b; Lee and Pavlovic, 2021). However, many related works employ a product of experts posterior (Eq. 2). This posterior choice can produce poor results given the type of data considered in this work (where there is significant variation in both modalities) (Shi et al., 2019). Therefore, we use a mixture of experts posterior (Eq. 3) as in the HMVAE. The implemented MDVAE is almost identical to the HMVAE– except that the unimodal variables z are marginally independent from g (Figs. 2b and 2c). We train the model by maximizing the ELBO:
log pθ(x1:M ) ≥ Eqφ [ log
pθ(g) ∏M i=1 pθ(zi)
qφ(g|x1:M ) ∏M i=1 qφ(zi|xi) + log ∏M i=1 pθ(xi|g, zi) ] , (18)
where g ∼ qφ(g|x1:M ) and zi ∼ qφ(zi|xi) with i ∈ {1, ...,M}. We use z1 ∈ R8, g ∈ R100, z2 ∈ R8 and find that increasing the unimodal latent sizes tends to decrease crossmodal coherence. We gradually increase the KL-regularization weighting factor from zero to one for a warm-up period of 20 (Sønderby et al., 2016). Table 6 shows the implementation.
MHVAE We follow Vasco et al. (2020) (the MHVAE authors) and implement the generative network from Fig. 3a for two hierarchical levels and the inference network from Fig. 2c. As done by
the authors, we employ domain dropout to learn representations over multiple modalities. We use a uniform domain dropout distribution, i.e., maximize three ELBOs given importance samples from q(g|x1), q(g|x2), and q(g|x1:M ), respectively (Eq. 19, similar to Wu and Goodman (2018)). We follow Vasco et al. (2020) and maximize the sum over the multi- and unimodal ELBOs:
L = Ljoint + Limages + Lcaptions
Ljoint := M∑ i=1 Eqφ(zi|xi) [log pθ(xi|zi)]− M∑ i=1 Eqφ(zi|xi) [ log pθ(zi|g) qφ(zi|xi) ] − Eqφ(g|x1:M ) [ log pθ(g)
qφ(g|x1:M ) ] Limages :=
M∑ i=1 Eqφ(zi|xi) [log pθ(xi|zi)]− M∑ i=1 Eqφ(zi|xi) [ log pθ(zi|g) qφ(zi|xi) ] − Eqφ(g|x1) [ log pθ(g)
qφ(g|x1) ] Lcaptions :=
M∑ i=1 Eqφ(zi|xi) [log pθ(xi|zi)]− M∑ i=1 Eqφ(zi|xi) [ log pθ(zi|g) qφ(zi|xi) ] − Eqφ(g|x2) [ log pθ(g)
qφ(g|x2)
]
(19)
We gradually increase a factor before the KL-divergence term from zero to one for five epochs (unimodal variables) or ten epochs (shared variable) as done by Vasco et al. (2020). We use z1 ∈ R256, g ∈ R100, z2 ∈ R8. Tables 7 and 8 present the architecture.

heading: D MDVAE: ALTERNATIVE INFERENCE NETWORK
text: There are several inference network choices for multimodal disentanglement VAEs (MDVAEs): Fig. 2c depicts the natural inference network. Figure 14 modifies this network to include dependencies between g and z1:M . This modified architecture recovers our proposed hierarchical inference network for the special case of two hierarchical layers (Fig. 3b). Note that the edges between g and z1:2 are still absent in the generative model (Fig. 2b): the model must learn to represent independent variation in z1. For example, the model must generate pθ(xi|zi, g) from zi ∼ pθ(z1) (not necessarily zi ∼ qφ(z1|xi, g)) and g ∼ qφ(g|xj 6=i).

heading: E ETHICS STATEMENT
text: Generative models can have pernicious effects on society via creating and propagating synthetic data that mimics reality (e.g., DeepFakes). Extending these models to multiple modalities can strengthen their performance and hence amplify these challenges. Therefore, annotating and marking data as synthetically generated is vital to ensure that people can spot and identify synthetic data outside of contexts where their synthetic nature is clear. Furthermore, the training data for deep generative models rarely represents the rich diversity of people, images, and objects in the world. Biases towards overrepresented groups will inevitably creep into generative models, and if the model is used for solving classification tasks on underrepresented groups, it is unlikely to prove useful. Extending the training data to multiple modalities can mitigate these effects if the additional modalities are less biased. In general, building generative models that can better cope with heterogeneity, e.g., by improving models or collecting more realistic data, is a good step towards alleviating harmful biases.

Review summaries:
The authors propose a Hierarchical framework for multimodal learning HMVAE. They define modality specific latent factor as well as the shared latent factor across multiple modalities. They represent modality-specific variations using latent variables dependent on the shared top-level variable. They parameterize the posterior distribution over the shared latent variable using a mixture of experts. The modality specific latent factors are adaptive inferred with both bottom-up and top-down information. They evaluate the proposed method on the Oxford Flower and the CUB datasets with various cross-modal experiments. 

This paper proposes a new type of model called a hierarchical multimodal VAE (HMVAE) that captures modality-specific variations using latent variables dependent on a shared top-level variable, in a manner similar to a multi-layer hierarchy. Their assumption is that modality-specific variations can sometimes depend on the structure shared across modalities which motivates their design decision to have modality-specific variables dependent on a shared top-level multimodal variable, which is in contrast to existing works on multimodal generative models that factorize into marginally independent latent variables to capture modality-specific variations (in other words not depending on a shared top-level multimodal variable).  Experimental results show promising performance on the CUB and the Oxford Flower datasets and outperform existing methods in sample generation quality and quantitative measures as the held-out log-likelihood.