Year: 2018

Abstract:
The application of multi-modal generative models by means of a Variational Auto Encoder (VAE) is an upcoming research topic for sensor fusion and bi-directional modality exchange. This contribution gives insights into the learned joint latent representation and shows that expressiveness and coherence are decisive properties for multi-modal datasets. Furthermore, we propose a multi-modal VAE derived from the full joint marginal log-likelihood that is able to learn the most meaningful representation for ambiguous observations. Since the properties of multi-modal sensor setups are essential for our approach but hardly available, we also propose a technique to generate correlated datasets from uni-modal ones.

heading: 1 INTRODUCTION
text: Auto Encoder (AE), Variational Auto Encoder (VAE), and more recently Disentangled Variational Auto Encoder (β-VAE) have a considerable impact on the field of data-driven leaning of generative models. Furthermore, recent investigations have shown the fruitful applicability to deep reinforcement learning (DRL) as well as bi-directionally exchange of multi-modal data. VAEs tend to encode the data into latent space features that are (ideally) linearly separable as shown by Higgins et al. (2017a). They also allow the discovery of generative joint models (e.g. Suzuki et al. (2017)), as well as zero-shot domain transfer in DRL as shown by Higgins et al. (2017b).
However, a good generative model should not just generate good data and achieve a good quantitative score, but also gives a coherent and expressive latent space representation. This property is decisive for multi-modal approaches if the data shows correlation, as it is the case for every sensor setup designed for sensor fusion. With this contribution, we investigate the characteristic of the latent space as well as the quantitative features for existing multi-modal VAEs. Furthermore, we propose a novel approach to build and train a novel multi-modal VAE (M2VAE) which comprises the complete marginal joint log-likelihood without simplifying assumptions. As our objective is the consideration of raw multi-modal sensor data, we also propose an approach to generate correlated multi-modal datasets from available uni-modal ones. Lastly, we draw connections to in-place sensor fusion and epistemic (ambiguity-resolving) active-sensing.
Section 2 comprises the related work on multi-modal VAEs. Our comprehensive approach (i.e. M2VAE) is given in Sec. 3. Furthermore, we describe multi-modal datasets as well as the generation of correlated sets in Sec. 4 which are evaluated in Sec. 5. Finally, we conclude our work in Sec. 6.

heading: 2 RELATED WORK
text: Variational auto encoder (VAE) combine neural networks with variational inference to allow unsupervised learning of complicated distributions according to the graphical model shown in Figure 1 (left). A Da-dimensional observation a is modeled in terms of a Dz-dimensional latent vector z using a probabilistic decoder pθa(z) with parameters θ. To generate the corresponding embedding z from observation a, a probabilistic encoder network with qφa(z) is being provided which parametrizes the posterior distribution from which z is sampled. The encoder and decoder, given by neural networks, are trained jointly to bring a close to an a′ under the constraint that an approximate distribution needs to be close to a prior p(z) and hence inference is basically learned during training.
The specific objective of VAEs is the maximization of the marginal distribution p(a) =∫ pθ(a|z)p(z) da. Because this distribution is intractable, the model is instead trained via stochastic gradient variational Bayes (SGVB) by maximizing the evidence lower bound (ELBO) L of the marginal log-likelihood log p(a) := La as
La≥L=−DKL(qφ(z|a)‖p(z))︸ ︷︷ ︸ Regularization +Eqφ(z|a) log(pθ(a|z))︸ ︷︷ ︸ Reconstruction . (1)
This approach proposed by Kingma & Welling (2013) is used in settings where only a single modality a is present in order to find a latent encoding z (c.f. Figure 1 (left)).
In the following chapters, we give a briefly comprise related work by means of multi-modal VAEs. Further, we stress the concept of two joint multi-modal approaches to derive the later proposed variational Auto Encoder (VAE).

heading: 2.1 MULTI-MODAL AUTO ENCODER
text: Given a set of modalities M = {a,b,c, . . .}, multi-modal variants of Variational Auto Encoders (VAE) have been applied to train generative models for multi-directional reconstruction (i.e. generation of missing data) or feature extraction. Variants are conditional VAEs (CVAE) and conditional multi-modal autoencoders (CMMA), with the lack in bi-directional reconstruction (Sohn et al. (2015); Pandey & Dukkipati (2017)). BiVCCA by Wang et al. (2016) trains two VAEs together with interacting inference networks to facilitate two-way reconstruction with the lack of directly modeling the joint distribution. Models, that are derived from the variation of information (VI) with the objective to estimate the joint distribution with the capabilities of multi-directional reconstruction were recently introduced by Suzuki et al. (2017). Vedantam et al. (2017) introduce another objective for the bi-modal VAE, which they call the triplet ELBO (tVAE). Furthermore, multi-modal stacked Auto Encoders (AE) are a variant of combining the latent spaces of various AEs ( Larochelle et al. (2007); Ranzato et al. (2006)) which can also be applied to the reconstruction of missing modalities (Ngiam et al. (2011); Cadena et al. (2016)). However, while Suzuki et al. (2017) and Vedantam et al. (2017) argue that training of the full multi-modal VAE is intractable, because of the 2|M|−1 modality subsets of inference networks, we show that training the full joint model estimates the most expressive latent embeddings.

heading: 2.1.1 JOINT MULTI-MODAL VARIATIONAL AUTO ENCODER
text: When more than one modality is available, e.g. a and b as shown in Figure 1 (mid.), the derivation of the ELBO LJ for a marginal joint log-likelihood log p(a) := LJ is straight forward:
LJ ≥ LJ = −DKL(qφab(z|a,b)‖p(z))︸ ︷︷ ︸ Regularization +Eqφab (z|a,b) log(pθa(a|z))︸ ︷︷ ︸ Reconstruction wrt. a +Eqφab (z|a,b) log(pθb(b|z))︸ ︷︷ ︸ Reconstruction wrt. b (2)
However, it is not clear how to perform inference if the dataset consists of samples lacking from modalities (e.g. for samples i and k: (ai,∅) and (∅,bk)). Ngiam et al. (2011) propose training of a bimodal deep auto encoder using an augmented dataset with additional examples that have only a single-modality as input. We, therefore, name the resulting model of Eq. 2 joint multi-modal VAE-Zero (JMVAE-Zero).

heading: 2.1.2 JOINT MULTI-MODAL VARIATIONAL AUTO ENCODER FROM VARIATION OF INFORMATION
text: While the former approach cannot directly be applied to missing modalities, Suzuki et al. (2017) propose a joint multi-modal VAE (JMVAE) that is trained via two uni-modal encoders and a bimodal en-/decoder which share one objective function derived from the variation of information (VI) of the marginal conditional log-likelihoods log p(a|b)p(b|a) =: LM by optimizing the ELBO LM:
LM ≥ LM ≥ LJ −DKL(qφab(z|a,b)‖qφb(z|b))︸ ︷︷ ︸ Unimodal PDF fitting of encoder b −DKL(qφab(z|a,b)‖qφa(z|a))︸ ︷︷ ︸ Unimodal PDF fitting of encoder a
(3)
Therefore, uni-modal encoders are trained, so that their distributions qφa and qφb are close to a multimodal encoder qφab in order to build a coherent posterior distribution. The introduced regularization by Suzuki et al. (2017) puts learning pressure on the uni-modal encoders just by the distributions’ shape, disregarding reconstruction capabilities and the prior p(z). Furthermore, one can show that deriving the ELBO from the VI for a set of M observable modalities, always leads to an expression
of the ELBO that allows only training of M̃ = {m|m ∈ P(M), |m| = |M| − 1} modality combinations. This leads to the fact that for instance in a tri-modal setup, as shown in Fig. 1 (right), one can derive three bi-modal encoders from the VI, but no uni-modal ones.

heading: 3 MULTI-MODAL VARIATIONAL AUTO ENCODER APPROACH
text: While the objective of Wang et al. (2016), Ngiam et al. (2011), Suzuki et al. (2017), and Vedantam et al. (2017) is to exchange modalities bi-directionally (e.g. a → b′), our primary concern is twofold: First, find a meaningful posterior distribution where the sampled statistics of an encoder network allows inference about further actions. Second, find an expression to jointly train all 2|M|−1 permutations of modality encoders.
By successively applying logarithm and Bayes rules, we derive the ELBO for the multi-modal VAE (M2VAE) as follows: First, given the independent set of observable modalities M = {a,b,c, . . .}, its marginal log-likelihood log p(M) =: LM2 is multiplied by the cardinality of the set as the neutral element 1 = |M|/|M|. Second, applying logarithm multiplication rule, the nominator is written as the argument’s exponent. Third, Bayes rule is applied to each term wrt. the remaining observable modalities to derive their conditionals. Further, we bootstrap the derivation technique in a bi- and tri-modal (c.f. tri-modal case in Sec. 6.1) case to illustrate the advantages. By excessively applying the scheme until convergence of the mathematical expression, it leads for a bi-modal set M = {a,b} to the following result:
LM2 = 2/2 log p(a,b) = 1/2 log p(a,b)2 = 1/2 log p(a,b)p(a,b) = 1/2 log p(b)p(a|b)p(b|a)p(a) (4)
= 1/2(log p(a) + log p(b|a) + log p(a|b) + log p(b)) = 1/2(La + LM + Lb) (5)
This term can be written as inequality wrt. each ELBO of the marginals La, Lb and conditionals LM:
2LM2 ≥ 2LM2 = La + Lb + LM = (6)
− βa DKL(qφa(z|a)‖p(z)) + Eqφa (z|a) log(pθa(a|z)) (7) − βb DKL(qφb(z|b)‖p(z)) + Eqφb (z|b) log(pθb(b|z)) (8) + Eqφab (z|a,b) log(pθa(a|z)) + Eqφab (z|a,b) log(pθb(b|z))− βab DKL(qφab(z|a,b)‖p(z)) (9) − αDKL(qφab(z|a,b)‖qφa(z|a))− αDKL(qφab(z|a,b)‖qφb(z|b)). (10)
Equation 6 is substituted by all formerly derived ELBO expressions lead to the combination of the uni-modal VAEs wrt. a and b (c.f. Eq. 7 to 8) and the JMVAE comprising the VAE wrt. the joint modality ab (c.f. Eq. 9) and mutual latent space (c.f. Eq. 10). Equation 7 and 8 have the effect that their regularizers care about the uni-modal distribution to deviate not too much from the common prior while their reconstruction term shapes the underlying embedding of the mutual latent space. We further apply the concept of β-VAE (Higgins et al. (2016; 2017a); Burgess et al. (2018)) to the regularizers via β∗ and adopt the factor α from Suzuki et al. (2017) for the mutual regularizer.
However, while β-VAE have the property to disentangle the latent space, our main concern is the balance between the input and the latent space using a constant normalized factor βnorm = β∗D∗/Dz.
If the derivation, which we leave out for the sake of brevity, is applied to the log-likelihood LM2M of a set M, one can show that it results into a recursive form consisting of JMVAEs’ and M2VAEs’ log-likelihood terms
LM2M = 1
|M|
 LMM + ∑
m̃∈M̃
LM2m̃
  ≥ 1
|M|
 LMM + ∑
m̃∈M̃
LM2m̃   =: LM2M . (11)
While the derivation of Eq. 11 is given in Sec 6.1.3, the properties are as follows:
• the M2VAE consist out of 2|M|−1 encoders and |M| decoders comprising all modality combinations
• while it also allows the bi-directional exchange of modalities, it further allows the setup of arbitrary modality combinations having 1 to |M| modalities
• subsets of minor cardinality are weighted less and have a therefore minor impact in shaping the overall posterior distribution (vice versa, the major subsets dominate the shaping and the minor sets adapt to it)
• all encoder/decoder networks can jointly be trained using SGVB

heading: 4 DATA SETS
text: It is quite common in the multi-modal VAE community to model a bi-modal dataset as follows (Wang et al. (2016); Ngiam et al. (2011); Suzuki et al. (2017); Vedantam et al. (2017)): The first modality a denotes the raw data and b denotes the label (e.g. the digits’ images and labels as onehot vector wrt. the MNIST dataset). This is a rather artificial assumption and only sufficient when the objective is within a semi-supervised training framework. Real multi-modal data does not show this behavior as there are commonly multiple raw data inputs. Unfortunately, only complex multimodal datasets of heterogeneous sensor setups exist (Ofli et al. (2013); Udacity (2016); Kragh et al. (2017)), which makes a comprehensive evaluation for VAEs futile. On the other hand, creating own multi-modal datasets is exhaustive since training generative models either demand dense sampling or supervised signals to form a consistent latent manifold (Bengio et al. (2012)).
While naı̈ve consolidation of non-coherently datasets does not meet the conditions of data continuity, as discussed later, we propose a consolidation technique by sampling from superimposed latent spaces of various uni-modal trained CVAEs in Sec. 4.1. This approach allows the generation of multi-modal datasets from distinct and disconnected uni-modal sets. Second, we propose and bi-modal mixture of Gaussians (MoG) dataset to show particular behaviors of the various VAE approaches in Sec. 4.2.

heading: 4.1 MULTI-MODAL DATA GENERATION
text: Perry et al. (2010) state that Hebbian learning relies on the fact that the same objects are continuously transformed to their nearest neighbor in the observable space. Higgins et al. (2016) adopted this approach to their assumptions, that this notion can be generalized within the latent manifold learning. Further, neither a coherent manifold nor a proper factorization of the latent space can be trained if these assumptions are not fulfilled by the dataset. In summary, this means that observed data has to have the property of continues transformation wrt. to their properties (e.g. position and shape of an object), such that a small deviation of the observations results in proportional deviations in the latent space. We adopt this assumption for multi-modal datasets where observations should correlate if the same quantity is observed, such that a small deviation in the common latent representation between all modalities conducts a proportional impact in all observations. This becomes an actual fundamental requirement for any multi-modal dataset, as correlation and coherence are within the objective of multi-modal sensor fusion. In the following, we propose a technique to generate new multi-modal datasets, given different uni-modal enclosed sets which meet the former conditions.
A valuable property of the VAE’s learned posterior distribution is, that it matches the desired prior quite sufficiently if only a single class is observed. This characteristic can be found again in the conditional VAE (CVAE) Kingma et al. (2014); Sohn et al. (2015) as it’s training is supported by the ground truth labels of the observations. Thus, it actually builds non-related posterior distribution for each class label, where every distribution matches a given prior. Furthermore, we adopt the idea of β-VAE Higgins et al. (2017b) which learns disentangled and factorized latent representations. Combining the properties of both advantages allows the superimposing of latent manifolds from various uni-modal encoders as shown in Fig. 2 (Top-Right). Now, latent samples can be drawn from the posterior to operate all CVAE encoders, with the desired label, to generate continues multi-modal data.
To test the approach we consolidate MNIST (LeCun Yann et al. (1998)) and fashion-MNIST (Xiao et al. (2017)) to an entangled-MNIST (e-MNIST) set by sampling from the prior (i.e. z ∼ N (0,I)) to generate observation tuples from the corresponding encoder networks pθa(a|z,C) and pθb(b|z,C) with class label C. The network architecture is explained in Sec. 3. To avoid artifacts, only samples from within 2σ of the prior are obtained.
Furthermore, we train a bi-modal JMVAE on the newly generated data to depict properties of the different datasets. We are aware of the fact that consolidation of uni-modal datasets cannot be achieved easily since continuity is hardly measurable. Therefore, naı̈ve consolidation results in a mixed dataset (i.e. mixed-MNIST) as shown in Fig. 2. To mimic this behavior and to achieve a fair comparison of the ELBO, we shuffle the generated fashion-MNIST per class label of e-MNIST to generate an equivalent mixed-e-MNIST (me-MNIST) set.
As shown in Fig. 2 (bottom), the JMVAE’s latent space reveals that for m-MNIST single clusters share the same mean as the best representative of a single label, but the variance of any uni-modal trained encoder remains orthogonal. Thus, the continuity in the observations does not correlate with each other by any means. On the other hand, the e-MNIST set with continues samples shows the desired behavior of multi-modal datasets as the JMVAE trains a coherent distribution for all uniand multi-modal encoders. These observations show that our proposed approach for generating new entangled datasets meet the formulated requirements of multi-modal datasets.

heading: 4.2 MOG-EXAMPLE
text: We investigate a Mixture-of-Gaussians (MoG) distribution, as depicted in Fig. 3, as bi-modal observations to mimic the output of e.g. feature extractors or classifiers. While they commonly already provide linear separable observations, we focus on ambiguity resolving properties of the VAE in particular.
The bi-modal (a,b) observations of Mixture-of-Gaussians have ten classes (0, . . . , 9) each. a’s observations are organized on a grid where (5,6,7) and (0,8) result in ambiguous observations by sharing the same mean. b’s observations are organized on a circle where (0,9) have ambiguous mean values.
This rather artificial experiment has the purpose to depict and evaluate ambiguous resolving properties of the VAEs. However, data of multi-modal sensor setups for complementary fusion show similar behavior, as various modalities are rectified to achieve a complete view of the scene (e.g. vision and grope to rectify objects). In that case, various dependencies of the generative process, given the class labels as factorized latent state representation z = (z0, . . . ,z9), are possible. This is mimicked by the MoG-Experiment, in a simplified assumption, as: p(a,b|z), p(a|z1, . . . , z4,z9), and p(b|z1, . . . , z8).

heading: 5 EXPERIMENTS
text: We apply the datasets explained in Sec. 4 to test and depict the capabilities of the M2VAE. First, we investigate the MoG data comprehensively. Second, we evaluate the ELBO of various approaches to the e-MNIST dataset. The VAEs are compared qualitatively, by visualizing the latent space, and
quantitatively by performing lower bound tests L M̃ for every subset M̃⊆M wrt. to the decoding
of all modalities pθM :
L M̃ = Eq φ M̃ (z|M̃) log
pθM(M|z)p(z)
qφ M̃
( z|M̃ ) (12)
with p(z) = N (z;0,I). All VAE architectures can be found in Sec. 6.1.4.

heading: 5.1 MOG-EXPERIMENT
text: We evaluate the latent space with the premise in mind, that a good generative model should not just generate good data but also gives a good latent representation z.

heading: 5.1.1 PARAMETRIZATION
text: We first investigate the impact of the parameter set (β∗, α) on the M 2VAE to find a latent space representation, which suits our needs to learn actions from it.
As the α parameter controls the mutual connection of all encoders in latent space, we found that a direct connection (i.e. α = 1.) puts too much learning pressure on matching the mutual latent distributions between uni- and multi-modal encoders. Thus, classes which should be separated in the multi-modal latent space collapse to the mean distributions of the uni-modal encoders. For α/10−2, the encoders are able to find an expressive latent space distribution by means of separable collapsed classes of uni-modal encoders, and expanded classes of multi-modal around it (c.f. Fig. 4 top/left).
By the findings of Higgins et al. (2017b), high β values result in highly entangled factors in latent space whereas small normalized βnorm / 10−2 show pretty robust disentanglement in all their test cases. The impact of β shows similar behavior on the M2VAE and thus, we chose small β values of βnorm =10
−2 to relax the learning pressure caused by the prior. While the over optimization wrt. to the prior leads to a fuzzy generation of data p(M|z) and collapse in latent space, high relaxation (βnorm ≪ 10
−3) causes loss of expressiveness between uni- and multi-modal encoding of a single class by means of the difference in the ELBO.
It is worth noticing, that diverging β parameters between multi- and uni-modal regularization (e.g. βab ≪ βa or vice versa) results in lower ELBOs, but for the sake of expressiveness of latent embedding and the ELBO between encoders’ embeddings. We argue that learning pressure should be applied equally to all encoders so that they experience a similar learning impact.
Another observation results from the fact, that the reconstruction loss of the M2VAE’s objective causes learning of mean representatives of classes in the observation space. This causes the artifact, that if for instance three classes exist in the output space, where one represents the overall mean, and an uni-modal encoder only sees the collapse of classes to that particular mean value, the latent encoding of this uni-modal encoder will collapse to the same mean as well. However, while it is not longer separable (not even non-linearly) in latent space by its mean value, the ELBO for the observation drives up and gives, therefore, evidence about the embedding quality. This insight might be fruitful in terms of epistemic (ambiguity-resolving) tasks, where for instance an unsupervised reinforcement learning approach could use the ELBO as a signal to learn epistemic exploration.
5.1.2 COMPARISION
Comparing the three approaches to estimate the multi-modal marginal log-likelihood by maximizing the ELBO, one can see from Fig. 5 that the most coherent latent space distribution was learned by the proposed M2VAE.
While the JMVAE-Zero learned similarities between qφab and qφa , it learned a complete new embedding for the classes (1,2) with qφb (denoted by ($)). Furthermore, the ELBO per embedding allows no conclusion between the embeddings of the various encoders.
The tVAE founds a much more coherent embedding between the encoders. This was achieved by the fact, that first the full multi-modal VAE, consisting out of the encoder qφab and two decoder pθa and pθb , was trained. Second, the decoder weights are pinned to train the remaining uni-modal networks which enforces coherence. However, the ELBO per embedding also does not allow any direct conclusion between the embeddings of the various encoders. This is depicted by (∼), where the multi-modal encoder qφab produces embeddings of higher energy than these of the uni-modal ones. This can happen as there is no regularizer which enforces the variational distribution of the encoders to match each other and thus, the KL-divergence may differ between the models for similar encodings.
The M2VAE, on the other hand, enforces the encoders inherently to approximate the same posterior distribution which can be seen by the strong coherence between all embeddings. Furthermore, classes which are separated in the multi-modal latent embedding collapse to the mean values in
the uni-modal ones as denoted by (+) and (−). This behavior is also rendered by the ELBO. As the M2VAE makes ambiguous embeddings, the reconstruction loss drives up (c.f. (∗) and (/)).
The embeddings also show an interesting fact about the class (0): As this class is only ambiguously detectable in the uni-modal case, all VAEs learn a linear separable and therefore unambiguous embedding if both modalities make an observation of this class (denoted by (−) for the M2VAE).

heading: 5.2 IN-PLACE SENSOR FUSION
text: Further, we introduce the concept of in-place sensor fusion using multi-modal VAEs. This approach is applicable in distributed active-sensing tasks where the latent space representation z of observations M′ (i.e. an object or point of interest was observed by a set of modalities) can be interpreted as inverse sensor model (c.f. Thrun et al. (2005)). This compressed information can be efficiently transmitted between all sensing agents and also be updated as follows: z can be unfolded to the original observation using the VAE’s decoder networks and combined with any new observation m to update the information in-place z → z∗ via
qφm∪M′ (z ∗|m,M′) with M′ =
⋃
m′inM′
pθm′ (m ′|z). (13)
However, a necessary requirement of Eq. 13 is that auto re-encoding (i.e. z → z via qφM′ (z|M ′)) does not manipulate the information comprised by z in an unrecoverable way (e.g. label-switching). Thus, we assume that VAEs tend to have a natural denoising characteristic (despite the explicit denoising Auto Encoders) which should re-encode any z in a better version of its own by means of the reconstruction loss wrt. z. This behavior is shown in Fig. 6 where we underlay the latent representation with the reconstruction loss of every particular z. One can see the learned discrimination of the latent space by means of high entropy separating the clusters vicinity. Furthermore, initial z values are auto re-encoded which draw the trajectories along their path in latent space. The observable properties of the VAE are that every seed converges to a fixed-point while performing descending steps on the latent space manifold. However, this statement is only valid in general for the proposed M2VAE, as the JMVAE-Zero and tVAE learn no or only similar coherent latent spaces between the encoder networks. Thus, seeds may be attracted by wrong attractors which makes these approach not sufficient for in-place sensor fusion.

heading: 5.3 E-MNIST EVALUATION
text: For this experiment, we estimated the ELBO by Eq. 12 to evaluate the performance of models JMVAE-Zero, tVAE, and M2VAE. We chose the model wrt. to the evaluation in Fig. 4 with βnorm = 0.01 which is β∗ ≈ 4 for the given MNIST image dimension of Da = ||(28,28,1)|| and Dz = 2. However, Tbl. 1 shows quantitatively and Fig. 7 depicts qualitatively that the pro-
Table 1: Evidence lower bound test for uni- and multi-modal setups of the VAEs (higher is better). M2VAE tVAE JMVAE-Zero
La,b La Lb La,b La Lb La,b La Lb −10.75 −10.91 −16.01 −23.6 −101.28 −88.75 −24.19 −131.05 −99.71
qφb qφaqφa,b
pθb pθa
Figure 7: From top to bottom: JMVAE-Zero, tVAE, and M2VAE. Left: Latent space representation with class colorization. Right: Reconstruction from latent space by applying the corresponding decoder networks. z is sampled linearly within 2σ of the prior for all figures.
posed M2VAE reaches the highest ELBO value, as well as it learns the most expressive latent space distribution. Furthermore, by sampling from the latent space for data generation, the M2VAE reveals crisp reconstructions in comparison to the other approaches.

heading: 6 CONCLUSION
text: This work presents a novel multi-modal Variational Auto Encoder which is derived from the complete marginal joint log-likelihood. We showed that this expression can jointly be trained on an Mixture-of-Gaussian dataset with ambiguous observations, as well as on a complex dataset derived from MNIST and fashion-MNIST. Furthermore, we formulated requirements and characteristics for multi-modal data for sensor fusion and derived a technique to learn new datasets, namely the proposed entangled-MNIST, which suffice these requirements. Lastly, we developed the idea of in-place sensor fusion in distributed, active sensing scenarios and formulated the requirements, by means of auto re-encoding, to VAEs. This revealed the properties of VAEs, that they tend to denoise the observable data which leads to an attractor behavior in latent space. However, we performed all qualitative evaluations of the latent space with the premise in mind, that a good generative model
should not just generate good data but also gives a good latent representation. This does also correlate with the quantitative behaviors, as our proposed model achieved the highest ELBO values. Future work will concentrate on the integration of the ambiguous resolving characteristics to an epistemic-exploration scenario.

heading: ACKNOWLEDGMENTS
text: This research was supported by ’CITEC’ (EXC 277) at Bielefeld University and the Federal Ministry of Education and Research (57388272). The responsibility for the content of this publication lies with the author.

heading: 6.1 EXTENSION TO THREE MODALITIES
text: The proposed, as well as approach by Suzuki et al. (2017), can be extended to multiple modalities M={a,b,c}. The conditional marginal log-likelihood of a can be written as
log p(a|b,c)=L M̃a +DKL(q(z|M)‖p(z|M))≥LM̃a . (14)

heading: 6.1.1 JMVAE FOR THREE MODALITIES
text: The VI between a set of distributions M can be written as −Ep(M) ∑
m∈M log p(m|M \m), which leads to an expression of maximizing the ELBO of negative VI (c.f. Suzuki et al. (2017)). Following this approach, the log-likelihood L3M can be expressed by the ELBOs, by utilizing Eq. 14, of their conditionals and KL divergence:
L3M = log p(a|b,c) + log(p(b|a,c)) + log(p(c|b,c)) (15)
≥ L M̃a + L M̃b + L M̃c
(16)
≥ L J̃ −DKL(q(z|a,b,c)‖p(z|b,c)) (17) −DKL(q(z|a,b,c)‖p(z|a,c))−DKL(q(z|a,b,c)‖p(z|b,c)) (18)
with L J̃ being the joint ELBO of a joint probability p(M) which expression is analog to Eq. 2.

heading: 6.1.2 M2VAE FOR THREE MODALITIES
text: Applying the proposed scheme to the joint log-likelihood of three modalities results in the following expression:
L3M2 (19)
=3/3 log p(a,b,c) = 1/3 log p(a,b,c)3 (20)
=1/3 log p(a,b,c)p(a,b,c)p(a,b,c) (21)
=1/3 log p(a,b)p(b,c)p(a,c)p(a|b,c)p(b|a,c)p(c|a,b) (22)
=1/3(log(p(a,b)) + log(p(b,c)) + log(p(a,c)) (23)
+ log p(a|b,c) + log p(b|a,c) + log p(c|a,b)) (24)
=1/3(2/2(log p(a,b) + log p(b,c) + log p(a,c))+L3M) (25) =1/6 ( log p(a,b)2 + log p(b,c)2 + log p(a,c)2 ) +L3M/3 (26) =1/6 ( LM2ab + LM2bc + LM2ac ) + 1/3L3M (27)
From here on, one can substitute all log-likelihoods given the expressions in Sec. 3 and ??, to derive the ELBO L3M2 .

heading: 6.1.3 M2VAE DERIVATION
text: LM2M = log p(M) mul. 1 = |M|/|M| log p(M) log. mul. = 1/|M| log p(M)|M| (28)
Bayes = 1/|M|
∑
m∈M
log p(M\m)p(m|M \m) (29)
log. add = 1/|M|
∑
m∈M
log p(M\m) + log p(m|M \m) (30)
The expression ∑
m∈M log p(m|M \m) is the general form of the marginal log-likelihood for the variation of information (VI), as introduced by Suzuki et al. (2017) for the JMVAE, for any set M. Thus, it can be directly substituted with LMM . The expression ∑ m∈M log p(M\m) is the combination of all joint log-likelihoods of the subsets of M which have one less element. Therefore, this term can be rewritten as
∑
m∈M
log p(M\m) = ∑
m̃∈M̃
log p(m̃) (31)
with M̃ = {m|m ∈ P(M), |m| = |M| − 1} Finally, log p(m̃) can be substituted by LM2m̃ without loss of generality. However, it is worth noticing that substitution stops at the end of recursion and therefore, all final expressions log p(m̃) ∀ |m̃| ≡ 1 remain.

heading: 6.1.4 NETWORK ARCHITECTURE
text: We designed all VAEs such that the latent space prior is given by a Gaussian with unit variance. Furthermore, all VAEs sample from a Gaussian variational distribution that is parametrized by the encoder networks. A summary of all architectures used in this paper can be seen in Tbl. 2. The reconstruction loss for calculating the evidence lower bound was performed by binary cross-entropy (BCE) for the e-MNIST and root-mean-squared error (RMS) for the MoG experiment.
Furthermore, the CVAE for training the e-MNIST dataset is designed as depicted in Tbl. 3.

Review summaries:
This paper proposes an objective, M^2VAE, for multi-modal VAEs, which is supposed to learn a more meaningful latent space representation. To summarize my understanding of the proposed objective, in the bi-modal case, it combines both objectives of TELBO [1] and JMVAE-kl [2] with some hyperparameters to learn the uni-modal encoders. The terms of Eqns 7,8, and 9 are equivalent to TELBO and Eqns 9 and 10 are JMVAE-kl. It would be very beneficial for the readers if you could more clearly contrast your objective with the related work given how similar they are.