Hierarchical Multimodal Variational Autoencoders
Authors: Jannik Wolff
Rahul G Krishnan
Lukas Ruff
Jan Nikolas Morshuis
Tassilo Klein
Shinichi Nakajima
Moin Nabi
Venue: ICLR 2022 Submitted (2022)
Keywords: hierarchical vae
variational inference
multimodal learning

Abstract: Humans find structure in natural phenomena by absorbing stimuli from multiple input sources such as vision, text, and speech. We study the use of deep generative models that generate multimodal data from latent representations. Existing approaches generate samples using a single shared latent variable, sometimes with marginally independent latent variables to capture modality-specific variations. However, there are cases where modality-specific variations depend on the kind of structure shared across modalities. To capture such heterogeneity, we propose a hierarchical multimodal VAE (HMVAE) that represents modality-specific variations using latent variables dependent on a shared top-level variable. Our experiments on the CUB and the Oxford Flower datasets show that the HMVAE can represent multimodal heterogeneity and outperform existing methods in sample generation quality and quantitative measures as the held-out log-likelihood.

Under review as a conference paper at ICLR 2022
HIERARCHICAL MULTIMODAL VARIATIONAL AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
Humans find structure in natural phenomena by absorbing stimuli from multiple input sources such as vision, text, and speech. We study the use of deep gen- erative models that generate multimodal data from latent representations. Exist- ing approaches generate samples using single shared latent variable, sometimes with marginally independent latent variables, to capture modality varia- tions. However, there cases where modality-specific variations depend the kind of structure shared across modalities. To capture such heterogeneity, we pro- pose hierarchical multimodal VAE (HMVAE) that represents modality-specifie variations using latent variables dependent on a shared top-level variable. Our ex- periments on the CUB and the Oxford Flower datasets show that the HMVAE can outperform existing methods in terms of generative heterogeneity and coherence across several quantitative and qualitative measures. We provide the code to re- produce the results in the supplementary material.
INTRODUCTION
Modality variations
eyering
This lange sind has with lange wing
shared
white ored head.
structure
Xy
image variations
caption variations
xM\{2 nizzh
(a)
(b)
(c)
Figure 1: Hierarchical decomposition. (a) The image background in the two pictures depends on the bird species. (b) We propose to capture such dependencies via hierarchies. (c) A closer look at the image background reveals that it is modality-exclusive, i.e., described by the caption.
Data modalities represent different perspectives of the same concept. Generative models can learn from such data by reproducing it, which can be useful for tasks such as image caption generation (Vinyals et al., 2015). This model family can also reproduce feature vector representations of the data, which can be helpful for tasks such zero-shot classification (Xian et al., 2018b) reinforce- ment learning (Bruce et al., 2017). One difficulty in multimodal learning lies in the differing prob- abilistic structures across modalities (Fig. 1). Our goal is to develop generative models that capture unimodal variations in both modalities.
One line of previous works (Zhu et al., 2017; Zhang al., 2017: Reed et al., 2016) tackled this chal- lenge with conditional generative adversarial networks (GANs) (Mirza and Osindero, 2014). These models generate samples for one modality conditioned on another modality and are optimized via competition between generator and discriminator. In contrast, Suzuki et al. (2016) used varia- tional autoencoders (VAEs) to jointly generate M modalities from a learned latent representation. We focus VAEs, which explicit density estimators and can maximize the likelihood of all data variations. GANs are implicit estimators, which can cause the generator to disproportionally favor specific variations (Razavi al., 2019).

---

Under review as a conference paper at ICLR 2022
Some multimodal VAEs (Wu and Goodman, 2018; Shi et al., 2019) incorporate single latent vari- able g that captures all relevant information (Fig. 2a). This formulation may disregard modality- exclusive variations. Other works (Huang et al., 2018: Hsu and Glass, 2018; Mahajan et al., 2020; Sutter et al., 2020; Daunhawer et al., 2021b; Lee and Pavlovic, 2021) have introduced disentangled latent variables 21:M, which are marginally independent of shared latent variable g and represent structure specific to modality i (Fig. 2b). We argue that disentangled latent representation may ne- glect the dependencies between shared and modality-ecclusie variations.
As a specific example, consider captioning (modality 2) pictures of birds (modality 1) as in Fig. la. The images of seabirds can have sky or water the background, while those of songbirds often dis- play forest backgrounds. The captions focus on the bird and thereby easily ignore such variations. Therefore, the shared pattern across modalities (bird species) dictates these modality-exclusive vari- ations. Consider a generative model where represents shared structure and image variations. When the two latent variables g and 2 are independent (Fig. 2b), the decoder that maps from latent variables onto images has to learn two distinct functions one for seabirds (where the independent variations determine the sky or water background) and another for songbirds (where the independent variations determine the forest background). We argue that this aspiration is theoretically learnable but practically challenging because it may require a large model with disproportional capacity (that could generalize poorly, is challenging to train, or requires abundant data). In contrast, we suggest that a hierarchical latent representation is an inductive bias that captures realistic data variations and guides learning. The edges between g and 21:M give the model the flexibility to decide which uni- modal variations to capture given a shared latent concept. For example. a hicrarchical model could adaptively learn that ø for seabirds solely captures sky or water background variations while ø1 for songbirds captures forest background variations. This can help the decoder to share features be- tween seabirds and songbirds.
Contributions (i) We propose a hierarchical multimodal VAE (HMVAE) that incorporates a latent hierarchy, where the shared variable g resides at the top and lower variables = complement uni- modal variations. (ii) We compare the HMVAR to several state-of-the-art baselines on the CUB and the Oxford Flower datasets. We report improved quantitative and qualitative measures in terms of semantic coherence and heterogeneity.
2 BACKGROUND AND RELATED WORK
Variational Autoencoders (VAEs) (Kingma and Welling, 2013; Rezende al., 2014) are deep generative models that represent the joint distribution po z) using neural networks with parame- ters 6, where x E RR is the observed vector and x E RR' is the latent vector. As the true posterior po(z|z) is intractable, an approximate posterior q (z) with parameters ø is used for inference. The parameters 0, Φ are usually trained by maximizing the evidence lower bound (ELBO) for the marginal likelihood:
(x((a) log po(a,x) (=(
S log Por))
Many efforts have been made to increase the expressivity of VAEs, e.g., by improving the prior of (Chen et al., 2016; Tomczak and Welling, 2018), and by introducing auxiliary latent variables (Maaløe et al., 2016). Our approach lies in the latter paradigm.
Hierarchical VAES (HVAEs) (Rezende al., 2014) have a hierarchical latent structure, where the topmost latent variable. drawn from an unconditional prior (=(z)), represents global features. The lower variables. drawn from conditional priors pe (2] complement local characteristics in order to reconstruct the observed data via pa (zz|21).
Sønderby et al. (2016) found the tendency for HVAES to not effectively use higher-level latent vari- ables when they are trained using inference networks of the form No(2 (z.). They proposed to first infer the top-level variable y directly with ( f.. (x) and then infer the intermediate variables { (z:} with both bottom-up and top-down information through g a (z) 2)) for hierarchical level d E
L - 1). The bottom-up and top-down information for hierarchical level i are encoded as hidden variables through neural networks: b. (x) and t, = (z(41). We make of this idea in our work, concatenate both hidden states and pass the result to an MLP that parameterizes the respective posterior (see App. C.1 for further details). This inference procedure can result in im-
2

---

Under review as a conference paper at ICLR 2022
g
11
9
22
21
9
22
21
22
I
22
I22
I1
2/2
(a)
(b)
(c)
Related work. (a) MVAE, MMVAE, (b) MDVAE, (c) MDVAE, MHVAE. Note that discuss an alternative inference network for the MDVAE in App. D.
Figure 2:
proved density estimation and sample generation performance (Sonderby al., 2016; Maalge al., 2019; Vahdat and Kautz, 2020; Child, 2021).
Multimodal VAEs represent M modalities 21:35 {=1,..., FM3 that are assumed to be condi- tionally independent given a shared representation g: pa (1 (g) I Po (a: ()) Motivated by the factorization of the true posterior, Wu and Goodman (2018) used a product of experts (PoE) formulation to parameterize the approximate posterior distribution over the shared latent variable, where each expert characterizes information within a modality:
( (.ar)) Po(g) I (g(am))
When all experts are Gaussian, the product posterior any combination of experts is easily com- puted.
Shi et (2019) showed that an inference model with Gaussian PoE formulation can be miscal- ibrated, i.e., overconfident experts 4o (glæ;) with low densities dominate the product (g (.M) They instead used a mixture of experts (MoE) formulation:
(g .1 = M Em (o(g)a (.))
With the MoE formulation, optimization is analogous to vote, which can avoid unreasonable dom- inance by single modality.
Mahajan al. (2020) applied normalizing flows (Rezende and Mohamed, 2015) to map between the latent spaces of 91 and 92 in VAE, where 46 (g, a:;) constitutes the posterior over 9;- Normalizing flows can more expressive than other distribution choices, such as Gaussians, which can help to represent complex multimodal relationships. However, optimizing the ELBO requires tractable sampling and density estimation in either direction, for example, by using coupling layers (Dinh et al., 2014 4; 2017). This can be a challenging constraint in practice.
Vasco et (2020) proposed the MHVAE, which also incorporates a hierarchical generative model. Note that both works were developed independently and offer complementary perspectives. We summarize the differences in following: first, the MHVAE's generative model is limited to two hierarchical levels (like Fig. 3a, but for two levels). Our proposed generative model supports arbitrary hicrarchical depth. Sccond, the MHVAE's s inference model is non-hicrarchical (Fig. 2c, Eq. 9 from the original paper). Our proposed inference model is hierarchical (Fig. 3b) Third, the MHVAE's posteriors over z are unimodal. Our proposed "top-down" " inference model computes the latent variables in the same order as the generative model. Therefore, the respective posteriors depend on all modalities. Fourth, the MHVAE's inference network drops modality-specifie hidden states at random during training. We evaluate their model on surjective data where single labels or attributes describe classes of images, i... there is not much variation single modality but lots the other. We focus on different data scenario where each modality has large degree of variation.
mixture of experts posterior. Fifth, Vasco al. (2020)
3 MULTIMODAL LATENT HIERARCHIES
We propose a hierarchical multimodal VAE (HMVAE) that captures the generative process of multi- ple modalities. The heprrarrameter L(m) > 2 defines the number of hierarchical levels for modal- ity m. If L(m) = 2, the hierarchy solely expands between the shared and unimodal variables. If [mm) > the model incorporates additional unimodal hierarchies.
3

---

Under review as a conference paper at ICLR 2022
Generative model We assume conditional in- dependence of modalities given a shared vari- able g (see Fig. 3a portraying the case of two modalities and three hierarchical levels):
(0, 9: III
Amm (# (m,11)
21,2
222
21.2
22222
[II Po.0 ( (0.0(2m,lRm.tt1)) (mm,+1))))
(4)
Po.0 Pe.0(2m,6(m)-11)) (g)
21,1
22,1
F1,1
22,1
The prior for the shared variable g is isotropic Gaussian. All conditional distributions for the intermediate variables (#m,1:((m)-1) for m E (I....M} are also isotropic Gaussian, where mean and variance are parameterized using neural networks. We use the hierarchical for- mulation introduced in 2 where some param- eters are shared across the generative and in- ference networks. The conditional distributions for the observed variables { can, for ex- ample, be parameterized using isotropic Gaus- sian distributions (for continuous-valle data) or Bernoulli distributions (for binary data).
11
22
22
(a)
(b)
Figure 3: Proposed generation and inference. The HMVAE represents unimodal variations in conditional variables z. We generalize the infer- network from Sønderby et al. (2016) to the multimodal case, where all posteriors are multi- modal. For example, the red edges visualize how an observed modality D2 can affect 21,1 in the other modality. The figure visualizes the special case of three hierarchical levels.
Inference model We extend Sønderby et al. (2016)'s hierarchical approach described in $ to the multimodal case (see Fig. 3a depicting
the case of two modalities and three hierarchical levels):
M
L(m)-2
gg.0(g. x a/ (g(
II m==1
9s.0 (0. Fm)) 9 (2m. Tm)) (5)
All distributions except the first factor do (gggat):)) are isotropic Gaussian with mean and variance inferred through neural networks from the conditional variables. The network employs skip nections from #1:M to The inference and generative networks share some parameters in the top- down networks that point from g the lowest unimodal variable mm,i=0- This architecture can rein- force hierarchical decomposition (Sønderby et al., 2016). Furthermore, it ensures that the unimodal latent variables are conditioned on all modalities (as indicated by the red edges in Fig. 3b) which helps learn crossmodal relationships. Section 2 and App. C.I provide further details on the hierar- chical architecture.
We parameterize the posterior distribution over the shared latent variable ( æ1:as) using mix- ture of experts formulation (Eq. 3). We follow Shi et (2019) and assume that several modalities can entail modality-exclusive variations, i.e., some variations for modality do not correlate with variations in modality j &
Optimization We train the HMVAE by maximizing the ELBO with stochastic backpropagation:
ELBO = Ego.0 [lg pe. 4o gg.0gg.z|mmmm)
log Pa (T
(6)
Further motivation Figure visualizes how multimodal VAEs could capture the data from the ex- ample in the Introduction (Fig. 1). The challenge lies in representing crossmodal dependencies not the joint distribution. That is because one modality can never inform about the modality-exclusive variations of another modality. Daunhawer et al. (2021a) demonstrated that this problem constitutes a core limitation across the multimodal VAE literature. In a non-hion -in-heearchhaal VAE, crossmodal gen- eration is indeed challenging because g must capture all variations. This becomes problematic when generating po (x; from (g)g Æ ( because the latter misses information about I;- In contrast, in two-level hierarchy, could theoretically capture the entirety of modality excluusvv variations.
However, the model may choose to represent some of these variations in to capture the hierarchi- cal dependencies within the modality variations. A deep hierarchy could cireumvent this
4

---

Under review as a conference paper at ICLR 2022
No hierarchy
Two-level hierarchy
Deep hierarchy
9
g
g
2
Z
2
2
a
L
Figure 4: Hierarchical models factorize unimodal variations along the hierarchy This figure visualizes possible latent representations for the multimodal data from Fig. I: two modalities æ1 and 22. where the brown color indicates shared structure and the yellow/green colors modality-excluss variations.
problem by adding infinitesimal small modality-exclusiig variations per hierarchical level (in the limit). Note that Fig. demonstrates one set of possible hierarchical representations. model could also utilize its degrees of freedom differently. For example, the deep model may incorporate shared structure and moxality-exclusive variations for the more complex modality in In general, we ex- pect (deep) hierarchy to be beneficial for representing complex data such image text modali- ties.
4 EXPERIMENTS
We evaluate on the CUB (Wah et al., 2011) and Oxford Flower (Nilsback and Zisserman, 2008) datasets. The modalities in both datasets are images and captions (text). We will focus cross- modal generation, i.e., caption-to-image generation and the reverse. We also consider unconditional generation. We evaluate generated samples mainly in terms of heterogeneity and coherence. Het- erogeneity describes diversity across samples for a single setting of the shared latent variable g and essential to ensure that the generated samples capture the diversity of the data. Coherence indi- cates whether a crossmodal sample lies the manifold of the true data. Crucially, heterogeneity and coherence are often both necessary. Imagine model that produces class-agnostic generations (high heterogeneity, low coherence for classification tasks). As for the HMVAE, we will validate that the model hierarchically decomposes the data and outperforms state-of-the-art competitors.
4.1 EXPERIMENTAL DESIGN
Models We are not aware of any standard benchmark dataset in the literature with reported per- formance for all considered baselines and sufficient hierarchical structure in the data. Therefore, we reimplement all considered baselines: the MVAE (Wu and Goodman, 2018), the MMVAE (Shi al., 2019), the MDVAE (Huang et al., 2018; Hsu and Glass, 2018; Mahajan et al., 2020; Sut- ter et al., 2020: Daunhawer et al., 2021b; Lee and Pavlovic, 2021), and the HMVAE (Vasco et al., 2020). Related works suggested many different MDVAES variants. Our reimplementation uses posterior choice suitable to the given data regime and stays close to the HMVAE- except that are marginally independent of g. In general, we keep the architecture and optimization hyperparameters similar across all models. App. C describes our implementation in detail. The use of larger models may produce photorealistic images with a high computational cost. For example, Child (2021) train unimodal hierarchical VAE for around 2.5 weeks using 32 VI0O GPUs datasets such as FFHQ. However, our goal is not photorealism but to validate that the proposed HMVAE represents multi- modal data differently than the baselines.
5

---

Under review as a conference paper at ICLR 2022
Data In some of our experiments, follow the common practice in the multimodal learning liter- ature and preprocess the data by extracting feature vector representations (Xian et al., 2018b; Sariy- ildiz and Cinbis, 2019; Schonfeld et al., 2019; Shi et al., 2019; 2020). Across all experiments, use the caption feature vectors provided by Zhang et al. (2017). These feature vectors #2 E IR 1R:0221 were extracted using CNN-RNN (Reed et al., 2016). Each image is paired with captions. We average the caption features for each image (except for the qualitative experiments which visualize captions explicitly), We find that this procedure can improve training and reduce computational re- quirements. In 3 4.2, we extract features 21 E IR 2R2048 from a ResNet- 101 trained on ImageNet. In $ 4.3, we use the images æ1 E R directly. We refer to App. C for details. To visualize image features and caption features, we follow Shi et al. (2019; 2020), search for the nearest neighbor fea- ture vector in the test set using the mean from p(ar I). and visualize the respective image or caption. Using this practice, we can evaluate coherence (relationship between generation and condition) and heterogeneity (sample diversity). To improve readability, we trim exceptionally long captions in the qualitative figures.
4.2 FEATURE GENERATION
In this section, we maximize the likelihood of image features E R2048 and caption features I2 10 on the CUB and the Oxford Flower datasets. We will validate the general utility of latent hierarchies relative to single-variable models. For the HMVAE, we choose L(1) I(2) = 2.
Figures 5 and 6 display p(a1 (22) and p(12) respectively. For the baselines, we vary g to generate 2,, where g I: ~ [(g]a and p(z,lg"). For the proposed HMVAE, we utilize the hierarchy by varying to generate 2; where g ((gla)zz)). z!!KK p(zilg), and p(æ,/=f). All models generate features that are mainly semantically coherent with the condition, indicating high coherence. However, the proposed HMVAE generates samples of higher diversity than the baseline, indicating high heterogeneity. This diversity must represented in 21, because we fix g and vary 1 Table demonstrates that the HMVAE achieves offee-of-theaat performance across most evaluated likelihoods. Likelihood estimates quantify both coherence and heterogeneity because they measure the divergence between the true and approximated distribution.
Table I: Likelihood estimates. We approximate the joint, marginal, and crossmodal likelihoods (defined in App. B) of test samples using 500 importance weighted samples. All models maximize the likelihood of image feature vectors and caption feature vectors. We report average and standard deviation over five runs per model. (higher is better)
Dataset and Model
logp(221, mb)
log
logp(2g)
logp(2t [2g)
log 1ogp(22)(12)
CUB
MVAE (Wu and Goodman, 2018)
-2046.0 + 6.6
-1902.3 + 8.8
142.8 + 16.1
-3594.0 + 337.0
-3024.9 = 297.4
MMVAE (Shi et al., 2019)
-2633.5 = 47.7
-2075.5 8.2
180.9 : 10.5
-3227.2 +: 145.1
-0022.2 +: 77.5
HMVAF (this work)
-1424.8 14.8
-1854.8
14.2
439.6
2.0
-2382.9 + 19.9
.879.7 31.8
Oxford Flower
MVAE (Wu and Goodman, 2018)
-2471.3 = 59.4
-2433.4 = 22,5
215.4 + 21.9
-3067.7 + 67.5
-5282.8 = 835.1
MMVAE (Shi et al., 2019)
-3029.9 : 50.5
-2575.5 1 8.6
141.2 + 26.0
-3179.3 1 44.2
-3454.0 = 218.4
HMVAE (this work)
-2069.1 40.0
-2497.6 = 21.7
442.2 :: 27.7
-3009.3 4 38.0
.1400.0 77.1
4.3 IMAGE GENERATION
In this section, we maximize the likelihood of images T₁ E 3x3644644 and caption feature vec- tors I2 E 1024 Because the data is more complex than in the previous section, we now investi- gate deeper hierarchies for the HMVAE' and incorporate additional baselines with multiple latent variables (MDVAE and MHVAE). Note that the deep HMVAE employs the fewest parameters (see Table 4 for overview) and is the only model that does not use warmup scheme for the KL- divergence loss (Sønderby et al., 2016).
Quantitative results The Fréchet Inception Distance (FID) compares the means and covariances between the true and approximate distributions in the feature space of an Inception network (Heusel
'The shallow variant incorporates T(1) T.(2) 2, the deeper variant incorporates I(1) 5 and
T.(2) 2.

---

Under review as a conference paper at ICLR 2022
flying bird long brown wings black bill white stripe
is small bird with green back head prom eye stripad bally.
MVAE
MMMAE
HMVAE (propoeed)
MMVAE
HMMAE (proposnd)
this Howeer has petals dilferent shapes and patterns accompan ed big, pistil.
this flower purple petals the shape of buil, with of the
MVAE
MMVAE
HMVAE (propcaed)
MVAE
MMVAE
HMMAE (proposedd)
Figure 5: CUB/Oxlord Flower: Generating image feature vectors from caption feature vectors. We generate image features, look up the nearest-ncighbor feature in the test set, and visualize the associated image.
for
Condition
MVAE
MMVAE
HMVAE (proposed)
Condition
for
MVAE
MMVAE
HMVAE (proposed)
Figure 6: CUB/Oxford Flower: Generating caption feature vectors from image feature vectors. We generate caption features, look up the nearest-neighhor feature from the test set, and visualize the respective caption.
et al., 2017; Seitzer, 2020). For the remaining measures, we preprocess images by extracting feature representations from ResNet-101 pre-trained on ImageNet. For example, we generate images æ1 64364 and then project these samples into the mentioned feature space. We do not use any preprocessing for the caption features because of their lower dimensionality (1024). Precision measures the relative amount of generated samples that lie on the true manifold. Recall measures the opposite. We follow (Kynkäänniemi et al., 2019) and approximate the target manifold using hyperspheres with radii equivalent to the 3-nearest-neighhorss of the respective samples. We also compute the harmonic mean across precision and recall (F1). Finally, we compute the average variance of image representations: for each conditioning caption, we sample g gl:100 ~ ((g[z2) to generate p lg) and project these samples into the ResNet feature space f( gg) We then compute the variance of these samples to a prototype representation f( TOo 200 s(øigg).
Table 2a presents quantitative results.2 2 The MHVAE and the HMVAE are the only models that per- form reasonably well across all evaluated measures. For example, even though the MVAE achieves the best FID, the model scores poorly in the precision measures. Furthermore, the model's qualita- tive samples display a lack of fidelity (Fig. 7). In contrast, the MMVAE generates coherent samples (high precision) but lacks heterogeneity (low variance and recall). The HMVAE also uses a mixture
Note that report the image recall and Fl values for the sake of completeness. These values are low across all models due to the complexity of the image space given relatively small dataset.
7

---

Under review as a conference paper at ICLR 2022
Table Oxford Flower All models maximize the likelihood of images and caption features vectors. We report average and standard deviation over three runs per model. (a) The second column high- lights the number of latent variables in the model. (b) Hierarchical decomposition the HMVAE.
p(R)) vs. p(an (tt)
p(zs] vs. pluta Zi)
Precision
Precision
MVAE (Wu and Goodman, 2018)
142.1
758
=
5,5 =.9
0.51 0.02
7.1
928 =
12.7
MMVAE (Shi ol., 2019)
161.1
+ 4.5
915 23
131
2.30..
0.04 + 0.0
74.3
26.1 1.8
38.6 6 1.8
MIVAA (Mahajan
2020),
167.8
914
0.4
1.2-0.8
0.22
6.1
53.7
(ablation)
MHVAE (Visen et al., 2020)
159.9 3.2
8688 12
16+022
32 04
0.29 0:00
5344 30
630 +.0+422
57.7 2.6
HMVAF (thés work))
147.9
+ 33
94.6
+ 1.0
18+0.
3.6 0.6
0.26 + 0.0
48.9 + 0.5
748 + 3.0
59.1 + 1.2
(b)
Effective
p(a,,) vs. pllp((ligg
platg) p(asgans)
Mean over
Hierarchical Layers
FID
Precision
Recall
Variance
Precision
Recall
147.9 3.3
94.6
0.3
3.6 0.6
0.26 = 0.0
[Ria] (Ru (a
273.0 6.8
62.7 2,0
+.0000
0. 0.0
0.29 0.00
297.3 1.6
4,3
000:0.
0:0:00
0.26 0.001
0.7
38.8
48.9 0.5
74,8 3.0
59.1 122
84.0 2.1
7.2 1
13.3 a
"this flower has purple petals in the shape of bell, with star of yellow stamen in the center."
"flowers alternately arranged, they red color"
HMVAE (ablation)
HMVAE
MVAE
MMVAE
MDVAE
MHVAE
(proposed)
"this flower has purple petals in the shape of bell, with star of [...]'
" fflowers are alternately arranged, they are red in color"
Mean from 21,4 Mean from 21,3 Mean from A1,2 Mean from $1,1
Figure 7: Oxford Flower: generating images from captions. Top: The penultimate column rep- resents the HMVAE with two hierarchical levels for the images (L(1) 2). The last column corre- sponds the HMVAE with L(1)
which the best model considering all measures across all modalities. Bottom: in the plot above, generate p(x:lg) from ((g) (z))) However, for the la- tent distributions starting the latent variables indicated below the plot, we use the mean and not conventional sample. We can thereby evaluate generated samples given varying degrees of hierar- chical expressivity.
of experts posterior as the MMVAE, but adds hierarchical depth which correlates positively with variance and recall. Finally, we observe that the HMVAE improves the MHVAE across measures related to coherence (FID, image precision) and heterogeneity (FID, caption recall). Table 2b ex- amines hierarchical decomposition in the HMVAE. We first sample conventionally from 9(g z,zii) We then generate p(æ,gg) gg) by using the conditional priors in the latent hierarchy for æ, Normally, would sample conventionally from each of these latent distributions (rows 2 and 7). However,
8

---

Under review as a conference paper at ICLR 2022
also evaluate models where we propagate the latent distribution means corresponding to the latent variables from the left column. In other words, we omit variations in some hierarchical levels. For the experiment on the image hierarchy, all measures improve with increased hierarchical expressiv- ity. For the experiment on the caption hierarchy, greater hierarchical capacity is beneficial (F1) by increasing recall and decreasing precision. Increased recall is expected to correlate negatively with precision: generating a larger manifold typically increases the likelihood of generated samples be- yond the target manifold.
Qualitative results Figure 7 focuses on p(æ) (#2). We generate p(æ via g ~ (09 (2).. For the MDVAE, we must additionally sample from the disentangled variable and hence generate im- ages p[z1|zz1 g) from 21 ~ p((1) and ~ ((g[22). Note that we always sample from the (condi- tional) priors over z across all multi-latten-variihbe models². The MVAE's samples display a notice- able lack of coherence and fidelity which is in line with the results from Shi et al. (2019). The MM- VAE' samples have high coherence and low heterogeneity. The MDVAE's samples display preci- sion but misalign some unimodal variations with the shared concept. The MHVAE's samples some- times lack semantic coherence. The shallow HMVAE generates samples of good coherence and low heterogeneity. The deep HMVAE generates the most compelling samples relative the baselines (as also indicated by the FID from Table 2a). Figure 7 presents the counterpart for the quantitative hierarchical ablation experiment from Table 2b. This figure demonstrates that the HMVAE decom- poses image variation along its hierarchy.
CONCLUSION
We have proposed hierarchical multimodal VAE (HMVAE) where unimodal latent hierarchies de- pend on shared latent variable. We have demonstrated that the model improves generative mod- cling performance multimodal data. Future work may further improve sample quality, for ex- ample, by incorporating different ways to parameterize the conditional densities: normalizing flows could improve the posterior's 's flexibility (Kingma al., 2016), autoregressive decoders could ex- plicitly represent the conditional dependencies across observed dimensions (Chen et al., 2018), and the transformer architecture may be good candidate to replace the convolutional neural networks (Dosovitskiy et al., 2021). Note that we discuss ethical implications in App. E.
Limitations Our goal is the the representation of heterogeneity within modalities. Although the proposed HMVAE outperforms our baselines, the model is not perfect. For example, when gen- erating images by sampling from the conditional distribution |=)) where æ1 ~ p(z11g) and {(g 22). the HMVAE must hallucinate a coherent set of visual representations (possibly condi- tional on given caption). This can be challenging when the caption is vague and does not contain sufficient information for the inference network to infer the shared underlying structure correctly. In other words, the additional expressivity that the HMVAE obtains through the conditional latent variables also makes learning harder since p(æ) |-) must represent larger manifold for the proposed HMVAE than it does for the baselines.
REFERENCES
J. Bruce, N. Sünderhauf, P. Mirowski, R. Hadsell, and M. Milford. One-shot reinforcement learning for robot navigation with interactive replay, arXiv preprint arXiv:1711.10133 2017.
Y. Burda, R. Grosse, and R. Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv: mtt.1109900511.. 2015.
X. Chen, D. P. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, I. Sutskever, and P. Abbeel. Variational lossy autoencoder. arXiv preprint 1r 1,000031,, 2016.
X. Chen, N. Mishra, M. Rohaninejad, and P. Abbeel. Pixelsnail: An improved autoregressive gen- erative model. International Conference on Machine Learning, pages 864-872. PMLR, 2018.
A core difference between the MDVAR and the HMVAE lies in their priors. In the former, the priors over unconditional. In the latter, the priors over z are conditional.

---

Under review as a conference paper at ICLR 2022
R. Child. Very deep vaes generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021. URL httttps:://pprrrrr///e net/Eorunt RWR.XCCXCDD.......
Daunhawer, T. Sutter, K. Chin-Cheong, E. Palumbo, and E. Vogt. On the limitations of multimodal vaes. arXiv preprini urXiv:2110.0412// 2021a.
I. Daunhawer, T. M. Sutter, R. Marcinkevičs, and J. E. Vogt. Self-supervised disentanglement of modality-speciffe and shared factors improves multimodal generative models. Pattern Recogni- tion, 12544:459, 2021b.
J. Deng, W. Dong, R. Socher, L.-J. K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and paltern recognition, pages 255. leee, 2009.
L. Dinh, D. Krueger, and Y. Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1400.8544 2014.
L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using real nvp. International Confer- ence on Learning Representations, 2017.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, In International Conference on Learning Representations, 2021. URL httpstttsse..... http///pppreeiieeeeeee net wtt.///wwweeeeeeummeeeeeeeeeeeee Beeet/oooomeiGGecccaPTTTT
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint /606 004108 2016.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by time- scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 2017.
W.N. Hsu and J. Glass. Disentangling by partitioning: A representation learning framework for multimodal sensory data. arXiv preprint 1805: /264, 2018.
X. Huang, M.-Y. Liu, S. Belongie, and Kautz. Multimodal unsupervised image-to-image trans- lation. In Proceedings of the European conference computer vision (ECCV), pages 172-189, 2018.
D. P. Kingma and M. Welling, Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114, 2013.
D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, Sutskever, and M. Welling. Improved varia- tional inference with inverse autoregressive flow. In Advances in neural information processing systems, pages 4743-4751, 2016.
T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. In NeurIPS, 2019.
M. Lee and V. Pavlovic. Private-shared disentangled multimodal vae for learning of latent represen- tations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recogni- fion, pages 1692-1700, 2021.
L. Maaløe, C. K. Sønderby, S. K. Sønderby, and O. Winther. Auxiliary deep generative models. In International conference on machine learning, pages 1445-1453. PMLR, 2016.
L. Maalóe, M. Fraccaro, V. Liévin, and O. Winther. Biva: A very deep hierarchy of latent variables for generative modeling. Advances in neural information processing systems, pages 6548- 6558, 2019.
10

---

Under review as a conference paper at ICLR 2022
S. Mahajan, I. Gurevych, and S. Roth. Latent normalizing flows for many-to many cross-domain mappings. In International Conference on Learning Representations, 2020. URL https:// openreview net
M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprini arXiv: 2014.
M.-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pages 722- 729. IEEE, 2008.
A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 2015.
P. Ramachandran, B. Zoph, and Q. V. Le. Searching for activation functions. arXiv preprint 1rXiv: 2017.
A. Razavi, A. van den Oord, and O. Vinyals. Generating diverse fighlfidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pages 14837-14847, 2019. 2019.
S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text image synthesis. arXiv preprint
2016.
D. J. Rezende and S. Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv: 2015.
D. J. Rezende, Mohamed, and D. Wierstra. Stochastic backpropagation and approximate in- ference in deep generative models. In E. P. Xing and T. Jebara, editors, Proceedings of the 31st International Conference on Machine Learning. volume 32 of Proceedings of Machine Learning Research, pages 1278-1286, Bejing, China, 22-24 Jun 2014. PMLR. URL http: / //proceedings mlr.. press /vees///////// htte//ezendelle 4. html.
M. B. Sariyildiz and R. G. Cinbis. Gradient matching generative networks for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2168- 2178, 2019.
E. Schonfeld, S. Ebrahimi, S. Sinha, T. Darrell, and Z. Akata. Generalized zero-and few-shot learn- ing via aligned variational autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8247-8255, 2019.
M. Seitzer. pytorch-fid: FID Score for PyTorch. https://ggttub..comseeeere pytorch-fid, August 2020. Version 0.1.1.
Y. Shi, N. Siddharth, B. Paige, and P. Torr Variational mixture-of-experts autoencoders for multi- modal deep generative models. In Advances in Neural Information Processing Systems, pages 15692-15703, 2019.
Y. Shi, B. Paige, P. H. Torr, and N. Siddharth. Relating by contrasting: data-efficient framework for multimodal generative models. arXiv preprint arXiv:2007.01179, 2020.
C. K. Sønderby, T. Raiko, L. Maalge, S. K. Sønderby, and O. Winther. Ladder variational autoen- coders. Advances in neural information processing systems, pages 3738-3746, 2016.
T. M. Sutter, I. Daunhawer, and J. E. Vogt. Multimodal generative learning utilizing jensen- shannon- divergence. arXiv preprint arXiv:2006. 08242,, 2020.
M. Suzuki, K. Nakayama, and Y. Matsuo. Joint multimodal learning with deep generative models. arXiv preprint arXiv: 161101891 2016.
J. Tomezak and M. Welling. Vae with vampprior. In International Conference Artificial Intel- ligence and Statistics, pages 1214-1223. PMLR, 2018.
G. Tucker, D. Lawson, S. Gu, and C. J. Maddison. Doubly reparameterized gradient estimators for monte carlo objectives. arXiv preprint arXiv: 1510.0412, 2018.

---

Under review as a conference paper at ICLR 2022
A. Vahdat and J. Kautz. Nvac: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33, 2020.
M. Vasco, F. S. Melo, and A. Paiva. Mhvae: a human-inspired deep hierarchical generative model for multimodal representation learning. arXiv preprint arXiv:2006. 02991, 2020.
O. Vinyals, Toshev, S. Bengio, and D. Erhan. Show and tell: neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156- 3164, 2015.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongic. The caltech-uesd birds-200-2011 dataset. 2011.
M. Wu and N. Goodman. Multimodal generative models for scalable weakly-supervised learning. In Advances in Neural Information Processing Systems, pages 5575-5585, 2018.
Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata, Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE Iransactions on puttern analyvis and machine intelligence, (19))2251-2265 201 aa.
Y. Xian, T. Lorenz, B. Schiele, and Z. Akata. Feature generating networks for zero-shot learning- In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5542- 5551, 2018b.
H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 5907-5915, 2017.
J.-Y. Zhu, T. Park, Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- the IEEE international conference on computer
consistent adversarial networks. In Proceedings the vision, pages 2223-2232, 2017.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
We partition the Appendix into five sections:
A. Additional experiments: For the experiments from 4.2, we provide further results on crossmodal generation. For the experiments from 44.3, we provide results on unconditional generation.
B. Likelihood estimation:
This section presents the estimators used to compute the likeli-
hoods reported in Table
C. Implementation: This section describes our implementation in detail. App. C.I comple- ments the background section 3 2 by describing details the employed hierarchical VAE architecture. This section also presents dataset-agnostic model specifications for the base- lines. We then focus on dataset-specific implementation specifications, where App. C.2 covers feature generation the CUB dataset, App. C.3 feature generation the Oxford Flower dataset, and App. C.4 image generation on the Oxford Flower dataset.
D. MDVAE: inference networks: This section discusses an alternative inference network choice for multimodal disentanglement VAEs (MDVAEs).
E. Ethical statement: This section discusses ethical implications of the proposed model, e.g.,
its societal impact.
13

---

Under review as a conference paper at ICLR 2022
A ADDITIONAL EXPERIMENTS
This section presents additional experimental results that complement 9 from the main paper.
A.1 FEATURE GENERATION
In this section, we maximize the likelihood of image features æ1 E 2004 and caption features 22 E R R on the CUB dataset and the Oxford Flower dataset. We use the same experimental setup as in $ 4.
Image feature generation Figure 8a presents generated samples where the captions are from the training set. We observe that the MMVAE still lacks variety in its generations, which indicates that this tendency not caused by poor generalization. Figure 8b presents generated samples where the captions are from the test set. The HMVAE can represent different backgrounds (e.g., various sea or forest settings), even though this information is missing in the conditioning captions.
this little bird has yellow belly and breast with dark superciliary and wingbars.
small sized bird that has tones of blue and green all over
MVAE
MMVAE
HMVAE (proposnd)
MVAE
MMVAE
HMVAE (proposed)
the bird has green breas: and abdomen as well as a tiny bill.
this a small raddish zrown bird with bright grean wing and brown beak.
MVAE
MMVAE
HMVAE (propeand)
MVAE
MMVAE
HMVAE (proposnd)
large
long grey
eye.
with black
MVAE
MMVAE
(proposed)
MMVVEE
(proposed]
the petals of this flover orange with short stigma
this flower has petals that are purple and very thin
MVAE
MMVAE
HMVAE (proposed)
MVAE
MMVAE
HMVAE (proposod)
(b)
Figure 8: CUB/Oxlord Flower: Generating image feature vectors from caption feature vectors. Plot (a) represents the training set, plot (b) the test set. We generate image features, look up the neirest-neighhor feature in the test set, and visualize the associated image.
Caption feature generation Figure presents additional results for caption feature generation which confirm our findings from $
14

---

Under review as a conference paper at ICLR 2022
Condition
wnth
MVAE
MMVAE
HMVAE (proposed)
Condition
gre mas can
cat
MVAE
MMVAE
HMVAE (praposed)
pouk
dights
Condition
MVAE
MMVAE
HMVAE (propased)
Figure 9: CUB/Oxford Flower: Generating caption feature vectors from image feature vectors. We generate caption features, look up the nearest-neighhor feature from the test set, and visualize the respective caption.
has toune aad tost rrand douniaated witters
vodue daman
piill.
havr mam of deeggle feem Restie mi-r
pata weeh petaiition
puppi ooger.
two fow hes hes out wide accemood oala sumoor dener
da Ioarthe
fooe with Large inre cuamen and
0200
pupee past
seaa to a rumers bories
his wo parue and creen berry comen tha doot
Tooes the peotle that 1igt red
demen
but muspi tit wi orarge 'tring shape
(a)
(b)
Figure 10: Oxford Flower: Unconditional generation. We generate p(z;1g) from 9 ~ p(g) for i E (1, In (b), we generate caption features, look up the nearest-neighhor feature from the test set, and visualize the respective caption.
A.2 IMAGE GENERATION
In this section, report additional results for & 4.3. Figure 10 visualizes unconditional samples. We observe that the HMVAE can generate diverse samples without relying on any conditioning.
15

---

Under review as a conference paper at ICLR 2022
B LIKELIHOOD ESTIMATION
In this section, we formalize the estimators that we use to evaluate the HMVAE, the MVAE (Wu and Goodman, 2018), and the MMVAE (Shi et al., 2019). We report all likelihoods in nats.
B.1 JOINT LIKELIHOOD
We now focus on the joint likelihood over M modalities. The special case with M describes the unimodal likelihoods p(æ:)-
Single latent variable We now discuss models that employ single shared variable g. We approx-
imate the likelihood using K importance samples:
K p(a1.M.9))
log p(1):M) = log
Σ (()M)
where g I:K ~ ((g/)1:))
K
We follow Shi et al. (2019), evaluate the mixture of experts VAE using stratified sampling, and esti- tighter estimate, which averages over modalities inside the logarithm. This objective should only be used during the evaluation because it may lead to an outsized impact of more informa- tive modalities during optimization. Notably, K samples of the joint posterior are equivalent to T = K/M samples from each modality-specifie posterior.
M
p plgm x1.M)
logp(2)r)) Ro log
where
Im I:
g(glam))
(8)
M
Latent hierarchy We now discuss models that employ both a shared variable g and conditionally independent, modality-speciff variables m,,t- We approximate the likelihood using importance samples and omit the super- and subscripts in the sampling definition for improved readability:
1
K
p p( g" x) 4(gh 2 2
log p(#).)) Ro log
K
where 9, ~ (g. x
The factorized generative and inference networks for the proposed latent hierarchy of the HMVAE are formalized in Eqs. and 5 of the main paper. Note that both product and mixture of experts formulations of the shared posterior be used within these hierarchical formulations.
B.2 CROSSMODAL LIKELIHOOD
In this section, we discuss estimators for the crossmodal likelihood p(æ, (x2) given target modality I and conditioning modality Analogously Eq. 7, we evaluate the likelihoods using K impor- tance samples.
Single latent variable
We first formalize the conditional likelihood similarly to Suzuki et al. (2016) and Wu and Goodman (2018):
logp(x;;(ee) log / p( g glx = log
/
p( Be. g)
dg
p(xx))
9
p(a,, xc:g)
p(xe: g) 9(g) xc,)
(10)
log Ee(ol(e))
log Eggg)))))
9(g xc)
In Eq. 10, term 1 constitutes estimation of the joint likelihood p(#1.M) using importance distribution that is conditioned modality c. Term 2 estimates the unimodal likelihood p(x..).
16

---

Under review as a conference paper at ICLR 2022
Latent hierarchy We now generalize the estimator from Eq. 10 the proposed latent hierarchy of the HMVAE. We first formalize the conditional likelihood:
p(0z/xe)) = / p(ær, 9, 11, =/zo)dzdg-
112,,9.2,23 dzdg
(11)
p(æe)
We assume that the modality-specific generative networks are conditionally independent given the shared representation (5 3):
p(x) = { p( 2: 2 zclg)((g) dzdg
-f
(12)
p(xc)
9.
We then define the importance distributions as the posterior over g and the conditional prior over z. The method must learn p(æz/zz) with z, being solely conditioned on the other modality c. This for- mulation allows fair comparison to the non-hierarchical baseline, which must do the same. con- trast, an importance distribution [(z//2; g) would make the conditioning of p(æe|zz) multimodal.
Zr. g) pl 9: %o) p( p(zr|Zz.1))
plaes g. 2c)
log p(x)) log Eggg. m.)lp(=)1))
log (c((.==/1.))
g(g, 2 2x(220)
((g, Z (T.)
(13)
In Eq. 13, can amortize computation for the unimodal likelihood p(æ.) which was defined in App. B.1. Term ! is the only one that requires additional forward passes through the network. Note that the remaining terms do not depend on the adjusted importance distribution p(z: g))
17

---

Under review as a conference paper at ICLR 2022
C IMPLEMENTATION
C.I GENERAL SPECIFICATIONS
Hierarchical models The following formalizes latent distributions assuming the unimodal hierarchical VAE in- spired by Sønderby et al. (2016) from ğ 2. We start with the conditional isotropic Gaussian prior
h2
22
Po.0 = =N(zslpo(de.0002 1)), o0 (0(d0,,,(Zz511))
(14)
h1
21
where i E {I,..., refers to the hierarchical level. The MLP dg.0, represents the top-down network, which is partially shared across the inference and generative net- works. Its output is passed the stochastic layers μ4 and oy that parameterize the prior distribution.
x
(a)
(b)
We then focus on the more complicated case of inferring the isotropic Gaussian posterior 2o.0 (g.0(2x/z;+1:- .) where i < for latent variables (Fig. 11). We first infer the hidden states for Zi+1. and for hierarchical level i e {1.... L}, where dg.s the top-down MLP and dø the bottom-up MLP:
Figure 11: Unimodal hierarchical
VA (Sønderby et al., 2016) (a) Inference network. The red edges visualize the merging procedure of bottom-up and top-down information. (b) Generative network.
h (z
(15)
h4 (2).
We concatenate both hidden states along the first dimension and pass the result through another neural network my to create the joint representation. We denote the concatenation operation (::
hi.j = mo((hs, h.
(16)
This joint representation is input to the stochastic layers Ps and 0g that parameterize the posterior distribution:
g0.02zi22411 = (( as (ha.)))
(17)
The remaining priors and posteriors are parameterized similarly. Appendix from Maalge et al. (2019) provides further background on unimodal hicrarchical VAEs that inspire this work.
MVAE and MMVAE We follow Shi al. (2019), train the MMVAE by maximizing a loose bound and evaluate using tight estimate described in App. B.I. The following describes exemplary differences in implementation compared to the original formulations:
We employ slightly different network architecture for the MVAE and the MMVAE (Wu and Goodman, 2018) relative to their original formulation, respectively. For example, we do not include dropout regularization across all implementations.
For the MMVAE, we neither employ importance weighted autoencoders (Burda et al., 2015) nor use a doubly reparameterized gradient estimator (Tucker et al., 2018). Note that these techniques increase computational requirements. Neither the MVAE nor the HMVAE relies on such measures.
We do not use Laplace latent distributions in the MMVAE, but Gaussian latent distributions in all implemented methods.
18

---

Under review as a conference paper at ICLR 2022
Bottom-Up
Top Dop-Down
Bottom-UU
Top-Down
Datet:
Disage 16
Damas: 733
763
Denon: 700
Dantr: 72B
Cenoe 700
Danse: 728
[196]
(
1788)
Deree
Deraa:
(P88)
Donee::
Saree:
(768)
[res1
[768]
(resl
of p(2x
of
Lesampinng Ceras: 788
Derae: 788 [Hes]
(Pt))
Ders:: 208
Gentr 1034
Derse: 705
Dentr 1034
Denin 1004
Dester 1004
(M006)
Uctuet 4:28
Datust 2048
(2048.
(M424)
(a)
(b)
Figure 12: Feature genereation on CUB/Oxford Flower HMVAE. (a) Image ft. VAE. (b) Cap- tion ft. VAE. Both decoders parameterizes normal distributions. We do not display the LeakyReL.U activation functions.
C.2 FEATURE GENERATION: CUB DATASET
Data The CUB dataset includes an image and caption modality. For the images, we use the features vectors 1 2R2018 provided by Xian et al. (2018a), who employed a ResNet-101 (He et al., 2016) trained ImageNet (Deng al., 2009). As described in $ 4, we caption features 22 E 1024 We divide the dataset into three splits that do not share classes. The training set consists of 111 classes. The validation split consists of 39 classes and the test split contains 50 classes. We train the model on the training set and tune hyperparameters on the validation set. We then evaluate the trained model on the test set, i.e., we do some previous works. We use the same test split as Reed et al. (2016) who provided the CNN-RNN feature extractor for the captions, which essential avoid test data leakage. We standardize the features for both modalities, i.c., we subtract the mean and divide by the standard deviation o. We compute both parameters on the training set. We assume that such preprocessing can be helpful, because the models generate a Gaussian likelihood p(æ,,.).
the validation
for training in
All models The generative models maximize the likelihood of the data under learned Gaussian distributions. We train the methods for 100 epochs with a learning rate of le-4 and batch size of 256. For each datapoint 1:10 u(g a during training. We gradually increase the KL-regularization weighting factor from zero to one for a warm-up period of 25 (Sønderby et al., 2016).
batch, we use ten samples from the respective shared posterior (i.e..
Proposed HMVAE The hierarchical VAEs visualized in Fig. 12a and Fig. 12b. The shared latent space g contains 16 dimensions.
MVAE and MMVAE The models are visualized in Table 3 and Table 3. We share the architecture across baselines. Those models are similar in their architecture to the hierarchical method, but lack the hierarchical components. We tested version with additional layers to compensate for this reduced capacity. However, models with more capacity (i.c., more weights) did not improve the performance of the flat models. The shared latent space g contains eight dimensions.
19

---

Under review as a conference paper at ICLR 2022
Table 3: Feature generation on CUB/Oxford Flower MVAE and MMVAE. Both decoders
parameterize normal distributions.
Image ft, encoder
Image ft. decoder
Caption ft. encoder
Caption ft. decoder
Input E 222048
Input E IR
Input E |024
Input E R
(2048 1024)
Dense 16, 768)
Dense(1024, 1024)
Dense(16, 16, 768)
LeakyReLU
LeakyReLU
LeakyReLU
LeakyReLUU
Dense(1024, 768)
Dense(768, 768)
Dense 768)
Dense(768, 768)
LeakyReL.U
LeakyReLUU
LeakyReL.U
LeakyReL.U
D768 768)
Dense(768, 1024)
Dense(768, 768)
Dense(768, 1024)
LeakyReLU
LeakyReLU
LeakyReLU
LeakyReLU
Dense(768, 768)
Dense( 1024, 1024)
Dense 768)
Dense( 1024, 1024)
LeakyReL.U Dense(768, 32)
LeakyRel.U Dense(1024, 4096)
LeakyReL.U
LeakyReL.U
Dense(768, 32)
Dense (10se 2048)
C.3 FEATURE GENERATION: OXFORD FLOWER DATASET
Data The Oxford Flower dataset includes image and caption modality. We use feature vector representations of the images, where 21 E IR 2048 and of the captions, where ø2 E 1024, We stan- dardize the features for both modalities, i.e., subtract the mean μ and divide by the standard devi- ation a. We compute both parameters on the training set. We assume that such preprocessing can be helpful, because the respective models generate Gaussian likelihood p(ar;-)). We divide the dataset into three splits that do not share classes. The training set consists of 62 classes. Both validation and test split contain 20 classes. We train the model on the training set and tune hyperparameters on the validation set. We then evaluate the trained model on the test set, i.c., we do not use the validation set for training as in some previous works. We use the same test split as Reed al. (2016) who pro- vided the CNN-RNN feature extractor for the captions, which is essential to avoid test data leakage.
All models The VAEs maximize the likelihood of each modality under a learned Gaussian distri- bution. All models are trained for 100 epochs. We gradually increase the KL-regularization weight- ing factor from zero to one for a warm-up period of 25 (Sønderby et al., 2016). We train the models with learning rate of le-4 and batch size of 256. For each datapoint in batch, use ten sam- plcs from the respective shared posterior (i.e., g 1:10 ~ q(g (1:M) )) during training.
Proposed HMVAE Figure 12a displays the image VAE, Fig. 12b shows the caption feature vector VAE.
MVAE and MMVAE Table 3 displays the image VAE, Table 3 shows the caption feature vector VAE.
20

---

Under review as a conference paper at ICLR 2022
Battom--p
Tep-Down
Battom-Ui
Top-Dows
Derner
Denee 330
Domer 132
Donce 785
Deass
DRAL 8.41
[NH
Verp
Verge
Oren 1.1.
Duine
Temer
BALAN
[V0]
(he)
Earw 23 1:1.0
04.
COE Boooe
Dreas 22
24g
Biter
p(nioo
C211
6211
Dera: HES2
14
(008A)
(b)
Figure 13: Image generation Oxford Boower-HMVAL (a) Image VAE. The decoder generates scalars, which are used to compute the binary cross-entropy with the ground truth. We do not display the Gelu activation functions (Hendrycks and Gimpel, 2016). (b) Caption ft. VAE. The decoder parameterizes a normal distribution. We do not display the LeakyReLU activation functions.
Table 4: Image generation on Oxford Flower number of trainable parameters. The second column represents the number of latent variables. One of the reason the deeper HMVAE has fewer parameters than the shallow HMVAE lies in the dense layer that maps between spatial (unimodal) and vector (shared) representations. The deeper HMVAE achieves a lower spatial resolution and which minimizes the parameters of this layer. Note that this single layer can easily reach more than three million parameters, see the official MVAE implementation at www tit hub. com/mhw'3 mult t iiiiooilllooe seall-vae-paee pule-puulli cc/bllb mww.ta/s/e///////maam//ao ce seeeeba/non /oodel py. In general, want to primarily ensure that the HMVAE does not improve performance due to excessive capacity.
Model Name
LV
# Params
MVAE (Wu and Goodman, 2018)
1
19.2M
MMVAE (Shi et al., 2019)
19.2M
MDVAE (Mahajan et al., 2020), i.a.
3
13.3M
MHVAE (Vasco al., 2020)
3
16.8M
HMVAE (shallow)
3
15.9M
HMVAE (this work)
6
11.6M
C.4 IMAGE GENERATION: OXFORD FLOWER DATASET
Data We preprocess the images by first making them quadratic (using CenterCrop in PyTorch) and then resizing them to i E R 33x64264 Note that we described general specifications in App. C.3.
All models We train the models with learning rate of lc-4 and a batch size 64.
Proposed HMVAE (shallow hierarchy) Figure 13a portrays the image VAE. Figure 13b presents the hierarchical VAE for the other modality. We gradually increase the KL-regularization weighting factor from one for warm-up period of 20 (Sønderby et al., 2016).
Proposed HMVAE (deep hierarchy) To insert further hierarchical levels, we "slice" deterministic blocks. We then add stochastic layers and respective pre- and postprocessing layers for each hierar-
21

---

Under review as a conference paper at ICLR 2022
chical level. Note that these layers often add just few trainable parameters because of 1x1 convo- lutions or small channel sizes for the stochastic variables. Note that the deeper model uses convolu- tions blocks of three layers instead of single convolutional layers. Input and output channel dimen- sion are identical on the outside and reduced within the block. We fix the channel size across the deterministic parts of the model and gradually down- or upsample the spatial dimension. We down- sample with average pooling and upsample using nearest-neighhor interpolation (Child, 2021). We fix the channels for the latent variables along the image hierarchy 16 and solely adjust the spatial dimensions. We do not use any warm-up scheme for the KL-divergence term in the loss, i.e. we use the natural ELBO.
MVAE and MMVAE their basic architecture. Table displays the image VAE, Table shows the caption feature VAE. We add extra capacity to the decoder to compensate for the missing parameters from hierarchical components. We gradually increase the Kl-regglariiztton weighting factor from zero to one for warm-up period of 20 (Sønderby et al., 2016).
The MVAE (Wu and Goodman, 2018) and MMVAE (Shi et al., 2019) share
Table 5: Image generation on Oxford Flower - MVAE and MMVAE. The image decoder gener- ates scalars, which are used to compute the binary cross-entropy with the ground truth. The caption decoder parameterizes a normal distribution. We use the Swish activation function (Ramachandran al., 2017).
Image ft. encoder
Image ft. decoder
Caption ft. encoder
Caption ft. decoder
Input C |33664644
Input C li 100
Input C I2 1024
Input C IR
Conv(3, 32, 4, 2, 1)
Dense(100, 512)
Densc(1024, 1024)
Dense(16, 768)
Swish
Swish Dense(512, 4096)
LeakyReL.U
LeakyReL.U
Conv(32, 64, 4, 1)
Dense(1024, 768)
Dense(768, 768)
Swish
Swish
LeakyReL.U
LeakyReLUU
Conv(64, 128, 4, 2, I)
Conv T(26, 128, 4, 2, 1, 0)
Dense(768, 1024)
Dense(768, 768)
Swish
Swish Con2T(128, 64, I, 0, 0)
LeakyReLU
LeakyReLU
Conv(64, 128, 1, 0)
Dense(1024 768)
Dense(1024, 1024)
Swish
Swish ConvT(64, 64, 1, 0)
LeakyReLU
LeakyReLU
Conv(128, 256, 4, 2,
Dense(768, 768)
Dense( (002,, 1024)
Swish
Swish
LeakyReLU
LeakyReLU
Dense(4096, 512) Swish Dense(512, 200)
ConvT((6,, 64, 4, 2, 1, 0)
Dense(768, 768)
Dense(1024, 1024)
Swish
LeakyReLU
LeakyReLU
ConyT((6,, 32, 4, 2, 0, 0)
Dense(768, 32)
Dense(1024, 1024)
Swish
LeakyReLU
ConvT(32, 3, 4, 2, 0, 0)
Dense (1024, 1024) LeakyReLU
Swish
Conv(3, 3, 4, 2, 1)
Densc(1024, 1024) LeakyReLU Dense 2048)
Sigmoid
MDVAE There are many implementation choices for MDVAEs (Huang et al., 2018; Hsu and Glass, 2018; Mahajan et al., 2020; Sutter et al., 2020; Daunhawer et al., 2021b; Lee and Pavlovic, 2021), However, many related works employ a product of experts posterior (Eq. 2). This posterior choice can produce poor results given the type of data considered in this work (where there is significant variation in both modalities) (Shi et al., 2019). Therefore, we use a mixture of experts posterior (Eq. 3) as the HMVAE. The implemented MDVAE is almost identical to the HMVAE- except that the unimodal variables are marginally independent from g (Figs. 2b and 2c). We train the model by maximizing the ELBO:
log (x 2 Egg log polg) pola.)
(o(g(T IR (=(=)(m;) + log IIM Pa (0 2:)
(18)
where g (g(g)=11M) and z; (2, with (1,...,M) We 21 E RR g E R100 22 RS and find that increasing the unimodal latent sizes tends to decrease crossmodal coherence. We gradually increase the KL-regularization weighting factor from zero to one for a warm-up period of 20 (Sønderby et al., 2016). Table 6 shows the implementation.
MHVAE We follow Vasco et al. (2020) (the MHVAE authors) and implement the generative net- work from Fig. 3a for two hierarchical levels and the inference network from Fig. 2c. As done by
22

---

Under review as a conference paper at ICLR 2022
Table Image generation on Oxford Flower MDVAE. For both models, the encoder produces single tensor of size D. We split this tensor into two parts that represent g E R 100 and 2; E R for {1,2), respectively. The decoder for z; uses concatenation of 2; and g as input, where again i (1, For the image VAE, the decoder generates scalars, which are used to compute the binary cross-entropy with the ground truth. For the caption VAE, the decoder parameterize normal distributions.
Image encoder
Image decoder
Caption encoder
Caption decoder
Input R3x61x61
Input IR
Input R1024
Input R1000
Conv(3, 32, 4, 2, 1) Swish
Dense( 100, 256 5 5)
Dense(1024, 1024)
Dense 16, 768)
Swish
LeakyReI.U
LeakyRel.U
Conv(32, 64, 4, 2, 1)
ConvT((256, 128, 4, 0)
Dense(1024, 768)
Dense(768, 768)
BatchNorm
BatchNorm Swish
LeakyReLU
LeakyReLU
Swish
Dense(768, 768)
Dense(768, 1024)
Conv(64, 128, 4, 2, 1)
ConvT(122, 64, 4, 2, 1)
LeakyReLU
LeakyReLU
BatchNorm
BatchNNrm
Dense(768, 768)
Dense 1024)
Swish
Swish
LeakyReLU
LeakyReLU
Conv(128, 256, 1, 0)
ConvT(64, 32. 2, 1)
Dense(768, 200)
Dense(1024 2048)
BatchNorm Swish
BatchNorm Swish
55
ConvT(32,, 3, 4, 1)
LeakyReLU Dense(512, 200)
the authors, we employ domain dropout to learn representations over multiple modalities. We use uniform domain dropout distribution, i.e., maximize three ELBOs given importance samples from ( ()). (g (2), and ((g ( respectively (Eq. 19, similar to Wu and Goodman (2018)).
We follow Vasco al. (2020) and maximize the
the multi- and unimodal ELBOs:
C = Ejoimi + Linages + Leaptions
Ad
M E go (z, (x,)))
po(zi1g7 log
Ljuint
E (x. (xe)) po
qo (z ax;))
==1
log (g(g)
Bgo msaar)
(o(g a II.M)
log (z
Cimagos
Σ (x) [log po (1;
Σ I8 go (z)lz))
U (z) (x:)
(19)
log (g(g)
Eg m))
q0 x1))
M
w Σ
Pe(2.19) log
Ceaptions
Σ i=1
E (ms()
[log po (2:
E Egs (2
q0 (2)))
2m1
pe (g(g
Egg (glxx)) xx))
log
To (g 222)
We gradually increase a factor before the KL-divergence term from zero to one for five epochs (unimodal variables) ten epochs (shared variable) done by Vasco et al. (2020). We use 21 255 g E IR 1000
x2 E R Tables and present the architecture.
23

---

Under review as a conference paper at ICLR 2022
Table 7: Image generation on Oxford Flower - MHVAE (1)
Networks for 2: We follow Vasco al. (2020) and the miniature DCGAN architecture (Radford et al., 2015). The decoder generates scalars, which are used to compute the binary cross-entropy with the ground truth. Networks for 22: The decoder for the second modality parameterizes normal distribution.
Lower encoder:
Lower decoder: ø1 - 21
Lower encoder:
Lower decoder: 22 + 22
I1 - h1
22 -> h2
Input E R33664664
Input 128 Dense(128, 256*5*5)
Input IR 1024
Input I248
C33,(3,32,4,2,1)
Dense(1024, 1024)
Dense(48, 768)
Swish
ConvT(256, 128, 4, 0)
LeakyReLU
LeakyReLU
Conv(32, 64, 4, 1)
BatchNorm Swish
Dense(1024, 768)
Dense(768, 1024)
Swish
LeakyReLU
LeakyReLU
Conv(64, 128, 4, I)
ConvT(128, 64, 1)
Dense(768, 768)
Dense(1024, 1024)
BatchNorm
BatchNorm
LeakyReLU
LeakyReLU
Swish
Swish
Dense(768, 512)
Dense(1024, 1024*2)
Conv(128, 256, 4, 1, 0)
ConvT(64, 32, 4, 2, 1)
LeakyReLU
BatchNorm Swish
BatchNorm
Swish
Dense(256*5 512)
ConvT(32, 4, 4, 1)
Swish
Table Image generation on Oxford Flower MHVAE (2)
The networks for g - and h are identical across modalities except for the final output dimension which depends on the latent size, where 21 e R and 22 E R48
Upper encoder:
Upper decoder:
Encoder: h - z
h
+
Input 522+2
Input E IR
Input IR 100 Dense(512,
Dense(51 12*2, 768)
Dense(-, 500)
LeakyReLU
LeakyReLU
Dense(768, 768)
Dense(500, 500)
LeakyRell ReLU Dense(768, 768)
LeakyReLU
Dense(500, 500)
LeakyReLU
LeakyReLU
Dense(768, 100*2)
Dense(500, -)
24

---

Under review as a conference paper at ICLR 2022
22
222
Figure 14: MDVAE: Alternative inference network
D MDVAE: ALTERNATIVE INFERENCE NETWORK
There several inference network choices for multimodal disentanglement VAES (MDVAEs): Fig. 2c depicts the natural inference network. Figure 14 modifies this network to include dependen- cies between g and II:M. This modified architecture recovers network for the special case of two hierarchical layers (Fig. 3b). Note that the edges between g and 1:2 are still absent in the generative model (Fig. 2b): the model must learn to represent independent variation in 21. For example, the model must generate (a from z; ~ Po(21) (not neces- sarily 2; qo (211 g)) and g (g(g xz
proposed hierarchical inference
ETHICS STATEMENT
Generative models can have pernicious effects on society via creating and propagating synthetic data that mimics reality (e.g.. DeepFakes). Extending these models to multiple modalities can strengthen their performance and hence amplify these challenges. Therefore, annotating and marking data as synthetically generated is vital to ensure that people can spot and identify synthetic data outside of contexts where their synthetic nature is clear. Furthermore, the training data for deep generative models rarely represents the rich diversity of people, images, and objects in the world. Biases towards overrepresented groups will inevitably creep into generative models, and if the model is used for solving classification tasks underrepresented groups, it is unlikely prove useful. Extending the training data to multiple modalities can mitigate these effects if the additional modalities are less biased. In general, building generative models that can better cope with heterogeneity, e.g., by improving models or collecting more realistic data, is good step towards alleviating harmful biases.
25


Reviews:
[{"review_id": "IiKD7WepGbz", "reviewer": "Reviewer_j3q4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed method is interesting and well-motivated.\n2. Experimental results are quite comprehensive, and results are promising.\n3. The paper is generally well-written and clear.\n\nWeaknesses:\n1. The model hinges on the motivation that 'modality-specific variations can sometimes depend on the structure shared across modalities'. Intuitively this seems contradictory since 'modality-specific variations' in one modality are by definition independent of those in another modality, so I am not sure what it means to 'depend on the structure shared across modalities'. It would be good to formally state this assumption and test it on real-world datasets, perhaps using some information-theoretic/dependency-measure metric?\n2. The results are better but the improvement could be confounded by the increased number of parameters (I believe the hierarchical model does have more params + potentially more flexibility in modeling the multimodal data). It would be important to control this more carefully when reporting results.\n3. Could have some qualitative results + human evaluation results for evaluating caption generation (figure 5).\n4. Would also be good to have some human evaluation results for text to image generation (figure 6).\n5. The paper tries very hard to distinguish itself from other factorized or hierarchical multimodal generative models (figure 7). While I believe that there is merit to their approach, there are still many possible confounding factors in my opinion, especially regarding the issue of the number of parameters. Also, since each directional arrow in the graphical model is parametrized by multi-layer neural nets, it is not clear the exact difference between explicitly defining a hierarchy of latent variables versus defining a shallow LVM (figure 7c) but having deeper layers in the model. \n6. I still have concerns over the novelty of the approach since the main contribution is to define a hierarchy of latent variables. It is not clear when and why this works (related to weakness point 1) so the paper can be improved if there were deeper insights in this part.\n7. Quality of figure 8 text can be improved.", "summary_of_the_paper": "This paper proposes a new type of model called a hierarchical multimodal VAE (HMVAE) that captures modality-specific variations using latent variables dependent on a shared top-level variable, in a manner similar to a multi-layer hierarchy. Their assumption is that modality-specific variations can sometimes depend on the structure shared across modalities which motivates their design decision to have modality-specific variables dependent on a shared top-level multimodal variable, which is in contrast to existing works on multimodal generative models that factorize into marginally independent latent variables to capture modality-specific variations (in other words not depending on a shared top-level multimodal variable).\n\nExperimental results show promising performance on the CUB and the Oxford Flower datasets and outperform existing methods in sample generation quality and quantitative measures as the held-out log-likelihood.", "summary_of_the_review": "The paper is well-written and the results are promising, but there are concerns over the motivation and novelty as compared to existing work, as well as possible confounding factors in the experimental setup.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H-FnGY0b0of", "reviewer": "Reviewer_XV1Z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is a very interesting paper. The hierarchical multimodal VAE can capture the realistic variations and help the decoder to share features between different modalities. \n\nQuestions:\n\n1. How does the number of hierarchical levels pre-defined in real scenarios? The two experiments used L(m) = 2, will that be better to set it as a tuned parameter? \n\n2. The evaluation metrics used in the paper (Fig 4, 5, 6) seem difficult to conclude which method performs better.  Are there any other quantitative metrics other than likelihood can be used?\n\n3. This method is limited to images and text data. Might be an application limitation for other multimodal data. \n\nMinor: In figure 8, all the captions for the subfigure are the same.\n", "summary_of_the_paper": "The authors propose a hierarchical multimodal VAE to capture the heterogeneity through latent variables dependent on a shared top-level variable. CUB and Oxford flower datasets were used for performance evaluation. \n", "summary_of_the_review": "Overall, I think the idea to consider the hierarchy for the latent variables between shared and modal-specific variations to learn the representation is novel and interesting. But, I do have concerns about this method’s application in real situations of multimodal learning. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "CBc4TqMMjx9", "reviewer": "Reviewer_Gac5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n* a new approach incorporates a hierarchy between shared and modality-specific latent variables. \n* Paper is generally well-written.\n\nWeaknesses\n* evaluation is restricted to dense modalities such as images and text. \n* The paper can be improved if the author can provide further insight into why defining a hierarchy of latent variables provides a better result. \n* The authors claim that a hierarchical latent representation is an inductive bias that captures realistic data variations and guides learning. It would be helpful if they demonstrate this by showing the representation of modality-specific variations with quantification.\n* The authors mentioned that they found that increasing the unimodal latent sizes tends to decrease crossmodal coherence. It would be interesting to show this result.\n* It would be helpful if the author can add a heterogeneity metric as their goal was identifying the representation of heterogeneity within modalities. The current evaluation metrics used in this paper are somewhat subjective and difficult to claim which method performs better. \n\n", "summary_of_the_paper": "The authors proposed a hierarchical multimodal VAE (HMVAE) that represents modality-specific variations using latent variables dependent on a shared top-level variable. They demonstrate that the proposed approach can represent multimodal heterogeneity and outperform existing methods in sample generation quality and quantitative measures.", "summary_of_the_review": "The authors proposed an HMVAE where unimodal latent variables depend on a shared latent variable. \nThey demonstrated that the model improves generative modeling performance on multimodal data but is restricted to dense modalities such as images and text. \nThe proposed method is well-motivated but the paper can be improved if the author clearly demonstrated how a hierarchy of latent variables improves identifing modality-specific latent variables. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B8FDWGrIot9", "reviewer": "Reviewer_vGFa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of the paper compared to the most similar baseline model MHVAE is in that they utilize the mixture of experts and can, potentially, build a deeper hierarchy.\n\nHowever, there are a few concerns about the paper, the approach proposed, and the evaluation:\n\n* On page 3, authors clarify the differences between MHAVE and HMAVE(the proposed model). However, MHVAE is also a hierarchical model, and thus, Fig.17a seems not a proper graph for MHVAE. It is not clear why the authors argue that MHVAE is “non” hierarchical model.\n\n* In section 4, the baseline models are not consistent throughout the experiments. Though section 4.4 shows comparison to SOTA methods that have modality specific latent factors, the authors use MVAE and MMVAE in section 4.2 and 4.3, both of which have only the shared latent space without the modality specific space. I find this unfair.\n\n* In section 4.4, the authors implement the MDVAE model by themselves rather than use one of the mentioned MDVAE model implementations. For fair comparison, it would be better to use the baseline network models or justify why that was not done.\n\n* The top row of page 8, they claim “when using deeper hierarchies because a hierarchical inference network can explicitly capture dependencies between different levels of abstraction”. This statement is not substantiated and needs to be grounded in experiments such as the ablation study wrt the hierarchy depth.\n\n* In Figure 6, unconditioned generation results in the first row are provided with the captions above them, inducing confusion (ie, imply conditioning, not unconditional generation). In the unconditioned case, shouldn’t g follow the prior distribution, and not be conditioned on the input, as stated in the text?  This needs clarification.\n\n* In Figure 8 caption, all descriptions indicate (a).\n", "summary_of_the_paper": "The authors propose a Hierarchical framework for multimodal learning HMVAE. They define modality specific latent factor as well as the shared latent factor across multiple modalities. They represent modality-specific variations using latent variables dependent on the shared top-level variable. They parameterize the posterior distribution over the shared latent variable using a mixture of experts. The modality specific latent factors are adaptive inferred with both bottom-up and top-down information. They evaluate the proposed method on the Oxford Flower and the CUB datasets with various cross-modal experiments.\n", "summary_of_the_review": "Two primary element should be resolved to provide convincing arguments about the utility of the proposed approach. (1) Consistent baseline model comparisons throughout section 4 experiments.  (2) Provide clear comparative analysis with MHVAE and show what is specifically novel (or identical to) this close competitor and substantiate improved performance in clear experiments.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: PAPER: This paper presents a multimodal auto-encoder architecture built on the premise that unimodal variations can be best generated when taking advantage of a shared latent space. This is operationalized by defining a hierarchical model with two primary levels: a shared structure space and unimodal variations (which could be multi-layer). 
DISCUSSION: The reviewers and follow-up discussion brought many questions and issues. The authors submitted a significantly revised version of their paper which clarified many issues and added a few extra results. While many of the reviewers’ questions were addressed by the authors, it seems that reviewers ended up not changing significantly their review scores. One fundamental concern is if the basic assumption about the shared structure is effectively the proper way to approach such generative modeling task. The experimental for image generation did not seem to support this hypothesis.
SUMMARY: While the revised version was an improvement over the original submission, improving clarity and adding some experimental measures, the experimental results did not seem to always support the main hypothesis. Human evaluation results may help in this direction.
Decision: Reject