Generaling Multimodal Variational Methods to Sets
Authors: Jinzhao Zhou
Yiqun Duan
Zhihong Chen
Yu-Cheng Chang
Chin-teng Lin
Venue: Submitted to ICLR 2023 (2023)
Keywords: multimodal variational auto encoder
self attention
unupervised learning
set representation learning

Abstract: Making sense of multiple modalities can yield a more comprehensive description of real-world phenomena. However, learning the co-representation of diverse modalities is still a long-standing endeavor in emerging machine learning applications and research. Previous generative approaches for multimodal input approximate a joint-modality posterior by uni-modality posteriors as product-of-experts (PoE) or mixture-of-experts (MoE). We argue that these approximations lead to a defective bound for the optimization process and loss of semantic connection among modalities. This paper presents a novel variational method on sets called the Set Multimodal VAE (SMVAE) for learning a multimodal latent space while handling the missing modality problem. By modeling the joint-modality posterior distribution directly, the proposed SMVAE learns to exchange information between multiple modalities and compensate for the drawbacks caused by factorization. In public datasets of various domains, the experimental results demonstrate that the proposed method is applicable to order-agnostic cross-modal generation while achieving outstanding performance compared to the state-of-the-art multimodal methods. The source code for our method is available online https://anonymous.4open.science/r/SMVAE-9B3C/.

Under review as a conference paper at ICLR 2023
GENERALING MULTIMODAL VARIATIONAL METHODS TO SETS
A Anonymou authors
Paper under double-blind review
ABSTRACT
Making sense of multiple modalities can yield more comprehensive description of real-world phenomena. However, learning the co-representation of diverse modalities is still a long-standing endeavor emerging machine learning ap- plications and research. Previous generative approaches for multimodal input approximate a joint-modality posterior by uni-modality posteriors as product-of- experts (PoE) or mixture-ofeexpert (MoE). We argue that these approximations lead to defective bound for the optimization process and loss of semantic con- nection among modalities. This paper presents novel variational method sets called the Set Multimodal VAE (SMVAE) for learning a multimodal latent space while handling the missing modality problem. By modeling the joint-modality posterior distribution directly, the proposed SMVAE learns to exchange infor- mation between multiple modalities and compensate for the drawbacks caused by factorization. In public datasets of various domains, the experimental results demonstrate that the proposed method is applicable to order-agnostic cross-modal generation while achieving outstanding performance compared to the state-or of- the-art multimodal methods. The source code for our method is available online
INTRODUCTION
Most real-life applications such robotic systems, social media mining, and recommendation systems naturally contain multiple data sources, which raise the need for learning co-representation among diverse modalities [ee et al. (2020). Making use of additional modalities should improve the general performance of downstream tasks as it can provide more information from another perspective. In literatures, substantial improvements can be achieved by utilizing another modality as supplementary information Asano et al. (2020); Nagrani et al. (2020) or by multimodal fusion Atrey et al. (2010); Hori et al. (2017); Zhang et al. (221)) However, current multimodal research suffers severely from the lack of multimodal data with fine-grained labeling and alignment (Snn al. (2017 Beyer et al. (2020); Rahate et al. (2022): Baltrušaitis et al. (2018) and the missing of modalities Ma al. 2021 Chen al." (2021).
In the self-supervised and for multimodal data Kingma & Welling (2013) Wu & Goodman (2018): Shi et al. (2019); Sutter ct al. (202T) have been a dominating branch of development. VAES are generative self-supervised models by definition that capture the dependency between an unobserved latent variable and the input observation. To jointly infer the latent representation and reconstruct the observations properly, the multimodal VAEs are required to extract both -opelic and modality- -nnvaraat features from the multimodal observations. Earlier works mainly suffer from scalability issues as they need to learn separate model for each modal combination Pandey & Dukkipati| (2017): Yan al. (2016). More recent multimodal VAEs handle this issue and achieves scalability by approximating the true joint posterior distribution with the mixture or the product of uni-modality inference models Shi et al. (2019): Wu & Goodman (2018); Sutter et al. (202I). However, our key insight is that their methods suffer from two critical drawbacks: 1) The implied conditional independence assumption and corresponding factorization deviate their VAEs from modeling inter-mon snter-modality correlations. 2) The aggregation of inference results from uni-modality by no means a co-representation of these modalities.
learning field, the variational autoencoders (VAEs)

---

Under review as a conference paper at ICLR 2023
To overcome these drawbacks of previous VAE methods, this work proposes the Set Multimodal Variational Autoencoder (SMVAE), a novel multimodel generative model eschewing factorization and instead relying solely upon set operation to achieve scalability. The SMVAE allows for better performance compared to the latest multimodal VAE methods and can handle input modalities of variable numbers and permutations. By learning the actual multimodal joint posterior directly, the SMVAE is the first multimodal VAE method that achieves scalable co-representation with missing modalities, A high-level overview of the proposed method illustrated in Fig The SMVAE can handle set of maximally M modalities as well as their subsets and allows cross- modslit generations. E, and D; represent the i-th embedding network and decoder network for the specific modality. M4,, 7, and Mk:; o% represent the parameters for the posterior distribution of the latent variable. By incorporating set operation when learning the joint-modality posterior, we can simply drop the corresponding embedding networks when modality is missing. Comprehensive experiments show the proposed Set Multimodal Variational Autoencoder (SMVAE) outperforms state-of-the-ar multimodal VAE methods and is immediately applicable to real-life multimodality.
Text
E
"one, DNC, three "shoe, shirt, dieest
Image
Ey
Hs
"smilig male
dx
Depth
E
D3
Force
M
class class
:
Text
Eg
Image
Eg
Hg
DM-1
Gk
Depth
RM
D2
Force
EM
Multimodal
Inputs
Embedding
Networks
Set Operation
Joint Posterior
Decoders
Reconstruction
Figure 1: Overview the proposed method for learning multimodal latent space. The SMVAE is able to handle any combination number of input modalities while having discriminative latent space and proper reconstruction.
2 RELATED WORK
2.1 MULTIMODALITY VAES
The core problem of learning a multimodal generative model is to maintain the model's scalability to the exponential number of modal combinations. Existing multimodal generative models such as Conditional VAE (CVAE]Pandey & Dukkipati (2017) and joint-modate VAE (JMVAE) Suzuki et al. (2016) had difficulty scaling since they need to assign separate inference model for each possible input and output combinations. To tackle this issue, follow-up works, such as, TELBO Vedantam et al. (2017). MVAE Wu & Goodman (2018), MMVAE al. (201)), MoPoE Sutter cet al. 2, assume the variational approximation is factorizable. Thus, they focused on factorizing the approximation of the multimodal joint posterior gz]|x|, ..., xM) into set of uni-modality inference encoders (z)x;), such that (z)[x,,.". ..., ,.,,,,))) F( } ). where F(.) a product or mean operation, depending on the chosen aggregation method. As discussed in Sutttr et al. (202I). these scalable multimodal VAE methods differ only in the choice of aggregation method. Different from those mentioned above multimodal VAE methods, we attain the joint posterior in its original form without introducing additional assumptions on the form of the joint posterior. To handle the issue of scalability, we exploit the deterministic set operation function in the noise-outsoureing process. While existing multimodal VAE methods can be viewed as typical late fusion method that combines decisions about the latent variables Khaleghi et al. (2013), the proposed SMVAE method corresponds to the early fusion method at the representation level, allowing for the learning of correlation and co-representation from multimodal data.
2

---

Under review as a conference paper at ICLR 2023
2.2 METHODS FOR SET-INPUT PROBLEMS
Multiple instance learning (MIL) Carbonneau et al. 2018) and 3D shape recognition Su et al. (2015); Hofer al. (2005); Wu et al.
2015), well-known examples of weakly-supervised learning problems that deal with set-input. MIL handles training data as numerous sets of instances with only set-level labels. A typical way to solve set-level classification problems is to use pooling methods for information aggregation Shao et al. (02TT) Recently, [ec et al. (2019) observed that classical feed-forward neural networks like the multi-layer perception (MI.P) (MI.P) Murtagh 1991) cannot guarantee invariance under the permutation of the elements in the input as well as the input of arbitrary sizes. Furthermore, recursive neural networks such as RNN and LSTM Hochreiter & Schmidhuber (1997) are sensitive to the order of the input sequences, and cannot fit the multimodal case since there is no natural order for modalities. Recently, Deep Sets Zaheer et al. for a permutation invariant function in set-input problems and proposed universal approximator for arbitrary set functions. Later on, Set Transformer Lee et al. (2019) further extends this idea by using the self-attention mechanism to provide interactions well as information aggregation among elements from an input set. However, their method only models a set of outputs as a deterministic function. Our work fills the gap between a deterministic set function to a probabilistic distribution and applies it to multimodal unsupervised learning.
(2017) provided a formal definition
PROPOSED METHOD
3.1 PRELIMINARIES
This work considers the multimodal learning problem set modeling problem and presents scalable method for learning multimodal latent variables and cross-modality generation. Given a dataset (x ) of N i..dd multimodal samples, we consider each of the sample as a set of M
modalities observations ((1) {x] (0) )-= The multimodal data assumed to be generated following the successive random process p(X, z) = (x|/z)((( which involves an unobserved latent variable The prior distribution of the latent variable is assumed to be po (z), with @ denoting its parameters.
The marginal log-likelln likelihhood of this dataset of multimodal sets can be expressed as a summation of = Σ log l)) Since the marginal likelihood of the dataset is intractable, we cannot optimize (((x() Fi-1) with regards to 0 directly. We instead introduce the variational approximation zo (z|X) from parametric family, parameterized by o, as an importance distribution. g (z|x)) is often parameterized by a neural network with o as its trainable parameters. Together, we can express the marginal likeelildood of a single multimodal set as:
marginal log- likelloooood of individual sets as log l(g)(x(()) as log [T,x1((X''
log (xg((x)) C(0,0:x' (2- [logpo(X' (x ), log q (x)x
L(), 0: ((0,0; x()
=-Dxi(q (1) " (z)) Ex-gg (z|x [og po (1)
[z)
where D-1(--l1-) the Kullback- Leibler (KL) divergence between two distributions. The non- negative property of the KL divergence term between the variational approximation (z)x))())
and the true posterior ()) in the first line makes L(o, 0; x the natural evidence lower bound (ELBO) for the marginal log- ligeliiel..ood. The last line indicates that maximizing the ELBO is equivalent to maximizing the reconstruction performance and regulating the variational approximation using the assumed prior distribution for the latent variable. To avoid confusion, we term neural networks used for mapping the raw input observations into a fixed-sized feature vector as the embedding network while the neural network used parameterize the variational approximation qs (z( (1) ) as the encoder network. A frequently used version of the objective function is written as:
1 (1) )llp())) (xlx)() [ log (xx [2)]
arg min &
where additional annealing coefficients 3 and reweighting coefficient À are used in the ELBO to allow gradients and warm-up training which gradually increases the regularization effect from the prior distribution and avoids reaching local minima in the early training stage Bowman et al. (2015);
Sønderby et al. 2016)). We drop the superscript of x(0 maintain brevity in the following paper.
3

---

Under review as a conference paper at ICLR 2023
3.2 SET MULTIMODAL VARIATIONAL AUTOENCODER
In multimodal scenarios with missing modalities, we consider each sample X (x,1i"h modaltiy present) subset of X and the powerset P(X) denoting all the 2d combinations, such that X e P(X). Our goal is to perform inference and generation from any number and permutation of available modalities, which requires an inference process is invariant to permutations and input of variable size. Following Definition] | we denotes the invariant inference process as p(z(X,)) p(2|xTX.). The ELBO for subset X, can be written as Eq
C. = (2) ())lo())) + Ez-zo (z) Xa) (z|Xaa [log po (Xxl2)))
(3)
Definition
Let Sn be a set of all permutations of indices 1,.", N, X (x1, (xx,))) denotes
random variables. A probabilistic distribution X)) is permutation inariant if and only if for any permutation π E Sn,, x) pXyl where the group action.
The difference between ( 0; X) Eq and C, 0; Xx) in Eq iis that the ELBO for a subset X is not yet valid bound for log p(X) by itself. Additional sampling from P(X) in the optimization objective as Eq ii is needed for theoretical completeness.
arg min
Σ Xx-P(X)
C,
TCSn
where is randomly generated permutation to the input subset X.. However, this sampling process can be trivial if we combine the sampling of the subsets with the sampling of mini-batch during training. By assuming the Gaussian form of the latent variable z and applying the reparameterization technique, the inference process of SMVAE can be written as:
p(z(x,) N ( ) N(O,1)
(5)
== Oe
(6)
Mz, log - == go (E) (x1), ... Em(m))
where E, are embedding network for the thh modality, (-(-) is neural network with trainable parameters & that provide the parameter for the latent's posterior distribution (i.e., f and σ) o denotes the element- wise multiplication. For the generation process, it is desired to models the joint likelihood of modalities conditioned on the latent variables Po (x,, 2) = p( ((2)p=(( (x,)=xx,))) (2) so that the model can utilize information from other available modalities more easier when generating complex modality. However, for the sake of easy implementation, we assign separate decoders D,, DM for all possible modalities as Po (x. Iz) Do, (z), ..., Do,, (=) We find empirically that, without loss of generality, using L normmalzzzaionn as additional regularization to regulate the parameter oμ and a of the inference network and respectively could facilitate the learning efficiency because the gradient from the ELBO often favors the reconstruction term over the regularization term.
3.3 SET REPRESENTATION FOR JOINT DISTRIBUTION
The scalability issue comes from the requirement for an inference process for the powerset P(X). We achieve scalability by using the noise-outsourced functional representation, i.e. z g(e, X.), to bridge the gap between the deterministic set functions to a stochastic function. The properties of the deterministic function thus can be passed to the stochastic distribution under minor conditions Blocm- Reddy & Tch (202)). With such foundation, the problem of modeling the posterior for a superset immediately reduces to designing a differentiable deterministic function that has the desired invariant elastic properties. Specifically, identify four critical requirements for weakly- supervised multimodal learning. Being that the model should 1) be scalable in the number of observable modalities: 2) be able to process input modalities sets of arbitrary size and permutation; 3) satisfy Theorem [ and 4) be able to learn the co-representation among all modalities.
Theorem A valid set function f(x) is invariant to the permutation of instances, iif can be decomposed in the form &{Σ (x))), for any suitable transformations & and V.
4

---

Under review as a conference paper at ICLR 2023
An oversimplified example of set function can be summation or product done in MVAE Wu & Goodman (2018) and MMVAE Shi et al. (2019). Pooling operations such as average pooling or max pooling also fit the definition. However, these set aggregation operations will require additional factorization assumptions to the joint posterior and ultimately forbid the VAE to learn co- representation of the input modalities aggregation is only applied at the decision level. To establish the inductive bias of inter-modality correlation, the self-attention mechanism without positional embeddings is reasonable choice Edelman et al. (2022); Shvetsova al. (2022).
Therefore, the proposed SMVAE leverages self-attention as the deterministic set function to aggregate embeddings of multimodal inputs. Given the query Q, key K and value V, an attention function denoted as Aul(Q, K, V) w( QK" ) where K I mxdx and V IR" mxnd are m vectors of vdd
dimension dg and d,. Q E TR. axxd, are vectors of dimension dy. w is the softmax activation function. In our case, the key-value pairs represent the m available embeddings of input modalities, m M. Each embedding is mapped to d- dd-iiensional embedding space by modality-speciff embedding network. By measuring the compatibility of the corresponding key and the query Q, information that is shared among modalities is aggregated co-representation.
In practice, we utilize the multi-head extension of self-attention denoted as MultiHead(Q,K,, V.h) Concat (A1, " A,) where A, Att, KW,", VW,") is obtained from the i" attention function with projection parameters W.º (d//))xd .w E TR. (d//))xxd W.Y [d(d//)xde and 8o E I dd h denotes the total number of attention heads and d denotes the dimension of the projections for keys, values and queries. Inspired by Lee et al. (2019), we design our deterministic set representation function (%,)) as follows:
96 (x.) : + ((((H) = MultiHead Xg, %g.h)
where I R. is an d,-dimensional trainable vector as the query vector for multimodal embeddings. Je is fully-connected layer. By calculating attention weights using / and each embedding. Not only does work as an aggregation vector that regulates the number of output vectors from 90 (X.)) to be constant regardless of the number of input embeddings, but also it selects relevant information from each embedding base on similarity measurement. The former justifies (X,)) as suitable permutation invariant set-processing function while the latter yields the desired co co-representation among modalities. Finally, Since the set representation function g (X.) is invariant the input permutations of different input sizes, achieved an invariant inference probabilistic function that satisfies Definition D through the noise-outsourced process as shown Eq. 6 Thus, by introducing the set representation function in the noise-outsourced process, the SMVAE readily scalable multimodal model for any subsets of modalities.
3.4 TOTAL CORRELATION OPTIMIZATION WITHOUT CONDITION INDEPENDENCE
The lower bound of the multimodal data without factorizing the joint posterior (i.e., Eq. provides additional information about the correlations of modalities during the optimization process compared to factorized methods. It is noteworthy that both MVAE and MMVAE depend on the assumption of conditional independence between modalities in factorization. Without loss of generality, the relation between L( 0; X) and the factorized case Lai can be shown in Eq.
(2(2)
M i=1 Pa(a;; 12)
Pa XX Z)
L(,, (:) = Bx(((1x)) log 4a (z X)
+ log
(2(2) MI p(x;
P z.)
= Ler + Bxo(e(x)
log
T. Pe
x
where X = (x), ".., x,r) and Lor is the lower bound for factorizable generative process as MVAE or MMVAE. Specifically, let ((%) denotes the empirical distribution for the multimodal dataset,
5

---

Under review as a conference paper at ICLR 2023
have:
po X
z)
Byx()) [C(0) [;)]] (x)()) [Coi] [
Eg po (x|z)
log
(10)
M -i= (o 2)
conditional total corretation
which reveals that without assuming factorizable generative process and enforcing conditional independence among modalities, our optimization objective naturally models the conditional total correlation which provides information of dependency among multiple input modalities [Watanabl (1960); Studeny Vejnarová 1998). Therefore, the SMVAE has the additional advantage of learning correlations among different modalities of the same event, which is also what we desired for good co-representation.
4 EXPERIMENTS
4.1 EXPERIMENT SETTINGS
We make use of uni-modal datasets including MNIST LeCun al. (1998), FashionMNIST Xiao et al. (0177) and CelebA Liu et al. (2015) to evaluate the performance of the proposed SMVAE and compare with other state-of oh-the- methods. We convert these uni-modal datasets into bi-modal dataset by transforming the labels to one-hot vectors as the second modality as in Wu & Goodman (2018); Suzuki al. (2016). For quatitative evaluation,
denote X and X2 the image and text modality
and measure the marginal log- -igllike,hhood, log log)(x) = log E [(x(e)p(()
((xx-) ]. the joint likelihood log p(x, y) log [ (( p()))(x(z))((xlz)) ]. and the marginal conditional probability, log p(xly) <(2)-)
log F p(a)_(xll)p((xl)) J- [ppts[[(y|2)]] using data samples from the test set. ((2l.) denotes the importance distribution. For all the multimodal VAE methods, we keep the architecture of encoders and decoders consistent for a fair comparison. Detailed training configurations and settings of the networks are listed in Appendix. B The marginal probabilities measure the model's ability to capture data distributions while the conditional log probability measures classification performance. Higher scoring of these matries means the better model is able to generate proper samples and convert between modalities. These are the desirable properties for learning a generative model.
((x|-)
Joint
Joint
Joint
Image
Label
Image
Text
Image
Text
(a) MVAE
(b) SMVAE
(c) MMVAR
Figure 2: Visualization of 2-D latent representation learned by our model compared MVAE and MMVAE from bi- modality MNIST dataset. Our model can learn natrual clusters while maintaining good structure for their joint distribution. Left: MVAE. Middle: SMVAE. Right: MMVAE.
4.2 GENERATION QUALITY AND QUANTITATIVE EVALUATION
We obtain 1000 important samples to estimate the probability metrics. Table shows the quatitative results of the proposed SMVAE for each dataset. We can see that the SMVAE outperforms other methods in almost all metrics. The outstanding of SMVAE mainly contributes to the direct modeling of the joint posterior distribution and optimization on a more informative objective. Fig. 3 Fig. 4 and Fig. show mross-modality generation of image samples for each domain generated by the SMVAE

---

Under review as a conference paper at ICLR 2023
Table 1: Statistical results
DataSet
Method
sampled from p
sampled from p x
log y)
log p(x)
log p(pp(r
pog y)
log.((
log
SMVAE
-225.10
-230.81
-232.85
-225.14
-232.36
-232.84
JMVAE
232.70
-232.40
-230.65
-232.94
-232.63
-230.39
FASHION
MVAE
233.01
-232.54
-230.69
-233.01
-232.53
-230.69
MMVAE
-235.37
-233.16
-231.32
-231.25
-235.18
-237.64
SMVAE
-83.41
-89.38
-86.45
-83.46
-89.09
-86.47
JMVAE
991.33
90.962
-88.43
90.76
-90.69
-88.69
MNIST
MVAE
-0.85
90.616
-88.55
90.85
-9.61
-88.56
MMVAR
-91.53
91.23
-90.64
99.77
-91.11
-90.01
PoMoE
-90.16
90.29
-91.36
-91.77
SMVAE
-6000.95
6087.91
-6085.31
-6025.38
-6000.66
-6084.94
CelebA
JMVAE
6238.28
-6234.54
-6235.33
-6242.19
66377.97
-6211.46
MVAE
6241.62
-627.10
-6235.36
-6242.03
-6236.92
-6234.95
MMVAE
6513.44
6560.57
-6516.14
-6255.45
6284.28
-6290.74
9
2
3
7
4
3 2
2
3
3
4
4
2
2
A
2
2
2
J
3
4
4
4
4
J
2
9
2
à
3
3
7
J
3
2
2
2
2
2
3
3
3
3
4
4
3
3
3
4
(a) digit 0
(b) digit I
(c)
digit
(d)
digit
(c)
digit 4
5
5
5
5
5
6
6
7
9
9
0
9
6
S
5
5
7
8 8
8
8
8
8
9
3
9
9
9
A
5
6
5
9
5
5
6 6
6
A
5
5
9
5
5555
6 4
6
6666666
6
8
8
8
6
9
9
5
8
5
6
6
7
7
8
8
8
9
9
9
4
2
9
5
5
8 5
6
6
8
1
3
9
(f) digit
(g) digit 6
(h) digit 7
(i) digit 8
(i) digit 9
Figure 3: Conditional image reconstruction of digits generated by SMVAE given text input.
(a) Ankle hoot
(b) Bag
(c) Coat
(d) Dress
Pullover
0
(f) Sandal
(g) Shirt
(h) Sneaker
(i) Trouser
(j) T-shirt
Figure 4: Conditional image reconstruction of fashionMNIST generated by SMVAE given text input.
model. We can see that given the text modality only, the SMVAE can generate corresponding images of good quality.
We further visualize the learned latent representation using ISNE Hinton Hinton Roweis 2002). As shown
in Fig latent space learned by MVAE method can only produce cohesive latent representation when
7

---

Under review as a conference paper at ICLR 2023
(a) Black Hair
(b) Malc
(c) Smiling
Figure 5: Generated face images by SMVAE given facial attributes.
both modalities are presented. When one modality is missing, representations from their method are distributed irrespective of the semantic category of the data. On the other hand, although the MMVAE method achieves cohesive representation for single-modality posterior, their joint representation is less discriminative. Indicating that using only the combination of uni-modal inference networks is insufficient capture intermodality co-representation. Nonetheless, our SMVAE method can achieve discriminative latent space for both single- and joint-modality inputs thanks to ability to exploit shared information from different modalities.
Edge and Facial Landmark Detection
(b) Image Recovery from Occlusion
(c) Image Painting
(d) Watermarks Removal
Figure 6: Learning computer vision transformations: (a) Ground truth images are randomly selecetd from CelebA dataset and we present the generated recontructed images, detected edges and facial landmarks. (b) Recontructed color images based on grayscale images. (c) Image recovery from obscured image. (d) Image recovery from watermark images.
4.3 CASE STUDY: COMPUTER VISION APPLICATION
We demonstrate that SMVAE is able to learn image transformations including colorization, edge detection, facial landmark segmentation, image completion, and watermark removal. With original image and each transformation as different modalities, we obtain 6 modalities in total by applying different transformations to the ground-truth images for this multimodal setting. This case study demonstrates the SMVAE's ability to generate in multiple directions and combinations.
Similar to Wu & Goodman (2018), for edge detection, we use Canny detector detector(Canny 1986) from Scikit- Image module Van der Walt et al. (2014) to extract edges of the facial image. For facial landmark segmentation, we use Dlib tool King (2009) and OpenCV Bradski & Kachler 2000). For colorization, we simply convert RGB colors to grayscale. For watermark removal, we add watermark overlay to the original image. For image completion, we replace half of the image with black pixels. Fig shows the samples generated from a trained SMVAE model. As can be seen in Fig 6(a) the SMVAE generates good reconstruction of the facial landmark segmentation and extracted edges. In Fig 6(b) we can see that the SMVAE is able to put reasonable facial color to the input grayscale image. Fig demonstrass that the SMVAE can recover the image from the watermark and complete the image quite well. The reconstructed right half of the image is basically agreed the left half of the original image. In o(d) all traces of the watermark is also removed.
8

---

Under review as a conference paper at ICLR 2023
Although reconstructed images suffer from the same blurriness problem that is shared in VAE methods Zhao et al. 2. the SMVAE is able to perform cross- moddalityy generation thanks to its ability to capture share information among modalities.
Avg. Force Input
(a) Non-image input
(b) Full information
Figure 7: Learning to reconstruct visual event in robotics scenario. (a) image sampled from SMVAE conditioned on force and action inputs. SMVAE able to reconstruct different visual modalities that depicts the relative position of the robotic arms to the target box properly. (b) From top to bottom, sampled reconstructed images from a action sequence
4.4 CASE STUDY: ROBOTICS CONTROL APPLICATION
The second case study shows that our method is readily applicable in robotics control scenarios using Usion&&Touch datase et al. (021). We use the SMVAE to learn cross-modality generation from continuous sensory input images. Emerging numan-in-the-lonn
shared autonomy systems are often equipped with multiple sensors, which pose a high requirement to the model's ability to learn co-representatiorl.cc
al. (2020): Luo et al. 20211: Chen al. (2021) [Selvaggio al. (202T); Newman et al. [2022); Li et al. 2021). The Vision& Touch dataset is a real-world robot manipulation dataset that contains visual, tactile, control action, and robot proprioception data which pocess more diverse modalities. The robotic arm attempts to insert the peg located on its tip into the target object. We use a total of 4 modalities including the depth images, RGB images, the 6-axis force sensor feedbacks, and the control action given to the robotics arm in each time step. Fig. that as the robotic arm is not receiving force signals early steps, reconstruction results of the RGB image show clearly that the arm has no contact with the taget box below. Only when the robotic arm is receiving high force readings, the generated image depicts the contact between the robotics arm and the target box. The quality of the reconstructed rgb and depth images is also differ between partial observation and full observation. While only limited information is observed (i.e., force and action inputs), our method is only able reconstruct rgh and depth images that can properly reflex the relative posistion between the robotic arm and the target object (Fig. [7a)). But when more information is presented, the latent variables can have more comprehensive information about the event and better reconstruction result as we removed the conditional independence assumption (Fig. 7(b).
CONCLUSION
This paper proposes a multimodal generative model by incorporating the set representation learning in the VAE framework. Unlike the previous multimodal VAE methods, the proposed SMVAE method provides a scalable solution for multimodal data of variable size and permutations. Critically, our model learns the joint posterior distribution directly without additional assumptions for factorization, yielding a more informative objective and the ability to achieve co-representation between modalities. Statistical and visualization results demonstrate that our method excels with other state-of of-the-art multimodal VAE methods. Which has high potential in emerging multimodal tasks that need to learn co-representation of diverse data sources while taking missing modality problems or set-input processing problems into consideration. Application dataset further indicates the proposed SMVAE has high potential in emerging multimodal tasks. In the future, we will explore methods that extend the current SMVAE framework to more diverse modalities as well as dynamic multimodal sequences to provide solutions for real-wor world multimodal applications.
cross-modality reconstruction in robotic

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yuki Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi-modal self-supervision. Advances Neural Information Processing Systems, 33:4660-4671, 2020.
Pradeep Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S Kankanhalli. Multimodal fusion for multimedia analysis: survey. Multimedia systems, 16(6):345-379) 2010.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Challenges and applications in multimodal machine learning. The Handbook of Multimodaa--Mlliieessoo Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition-Volume 2, pp. 17-48, 2018.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159 2020.
Benjamin Bloem-Reddy and Yee Whye Teh. Probabilistic symmetries and invariant neural networks. J. Mach. Learn. Res., 21:90-1, 2020.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. arXiv preprint arXiv:1511.0633 2015.
Gary Bradski and Adrian Kaehler. Opencv. Dr. Dobb's 's journal of software tools, 3: 20, 2000.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509.00511 2015.
John Canny. A computational approach edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6):679-698, 1986.
Marc-Andre Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey of problem characteristies and applicatioons. Patter Recognition, 77: 229-353, 2018. ISSN 0031-3203, doi: http:://ouaag////.ooooo//ogoooop 10.009, WRthttss
wwww. www.sccienceiirec..o com/
Kaiqi Chen, Yong Lee, and Harold Soh. Multi-modal mutual information (mummi) training for robust self-supervised deep reinforcement learning. In 202/ IEEE International Conference Robotics and Automation (ICRA), pp. 4274-4280. IEEE, 2021.
Chris Cremer, Quaid Morris, and David Duvenaud. Reinterpreting importance-weightee autoencoders.
arXiv preprint arXiv: 1704.02966 2017.
Benjamin L. Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. Inductive biases and variable creation in self-attention mechanisms. In International Conference Machine Learning, pp. 5793-5831. PMLR, 2022.
Haozhe Feng, Kezhi Kong, Tianye Zhang, Siyue Xue, and Wei Chen. Good semi-supervised vae requires tighter evidence lower bound. 2019.
Geoffrey Hinton and Sam Roweis. Stochastic neighbor embedding. Advances in neural information processing systems, 15, 2002.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()) 1735-1780, 1997.
Michael Hofer, Boris Odehnal, Helmut Pottmann, Tibor Steiner, and Johannes Wallner. 3d shape recognition and reconstruction based on line element geometry. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, volume 2, pp. 1532-1538. IEEE, 2005.
Chiori Hori, Takaaki Hori, Teng- Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi. Attention dateetion-bbsed multimodal fusion for video description. In Proceed- ings of the IEEE international conference computer vision, pp. 4193 4202, 2017.
10

---

Under review as a conference paper at ICLR 2023
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. PMLR. 2015.
Bahador Khaleghi, Alaa Khamis, Fakhreddine O Karray, and Saiedeh N Razavi. Multisensor data fusion: A review the state-of-the-art. Information fusion, 14(1):28-44, 2013.
Davis King. Dlib-ml: machine learning toolkit. The Journal of Machine Learning Research, 10: 1755-1758, 2009.
Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: A framework for attention-based permutation-invariant neural networks. In International conference on machine learning, pp. 3744-3753. PMLR, 2019.
Michelle A Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. IEEE Transactions on Robotics, 36(3):582-56,, 2020.
Shufei Li, Pai Zheng. Junming Fan, and Lihui Wang. Toward proactive
collaborative
assembly: A multimodal transfe-learniig-enabh action prediction approach. IEEE Transactions on Industrial Electronics, 69(8):8579-8588 2021.
Yingzhen Li and Richard E Turner. Rényi divergence variational inference. Advances in neural
information processing systems, 29, 2016.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslic Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. arXiv preprint arXiv:2107.07500 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Shan Luo, Nathan F Lepora, Uriel Martinez-Herrandder Joao Bimbo, and Huaping Liu. Vitae: Integrating vision and touch for multimodal and cross-modal perception. Frontiers in Roboties and AI, pp. 134, 2021.
Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2302-23100 2021.
Fionn Murtagh. Multilayer perceptrons for classification and regression. Neurocomputing, 2(5-6)) 183-197, 1991.
Arsha Nagrani, Chen Sun, David Ross, Rahul Sukthankar, Cordelia Schmid, and Andrew Zisserman. Speech2action: Cross-modal supervision for action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10317-10326. 2020.
Benjamin A Newman, Reuben M Aronson, Siddhartha S Srinivasa, Kris Kitani, and Henny Admoni. Harmonic: A multimodal dataset of assistive human-robot collaboration. The International Journal of Robotics Research, 41(1):3-11, 2022.
Gaurav Pandey and Ambedkar Dukkipati. Variational methods for conditional multimodal deep learning. In 2017 International Joint Conference on Neural Networks (IJCNN), pp. 308-315. IEEF, 2017.
Anil Rahate, Rahee Walambe, Sheela Ramanna, and Ketan Kotecha. Multimodal co-learning: challenges, applications with datasets, recent advances and future directions. Information Fusion, 81:203-239, 2022.
11

---

Under review as a conference paper at ICLR 2023
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Mario Selvaggio, Marco Cognetti, Stefanos Nikolaidis, Serena Ivaldi, and Bruno Siciliano. Autonomy in physical human-robot interaction: A brief survey. IEEE Robotics and Automation Letters, 2021.
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang. Xiangyang Ji. et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems, 34:2136-2147, 2021.
Yuge Shi, Brooks Paige, Philip Torr, et al. Variational mixture- of experts autoencoders for multi- modal deep generative models. Advances in Neural Information Processing Systems. 32, 2019.
Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio Feris, David Harwath, James Glass, and Hilde Kuehne. Everything at mnce-muli-modal fusion transformer for video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20020-20029, 2022.
Casper Kane Sønderby, Tapani Raiko, Lars Maalge, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29, 2016.
Milan Studený and Jirina Vejnarová. The multinformation function as a tool for measuring stochastic dependence. In Learning in graphical models, pp. 261-297. Springer, 1998.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Mill Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pp. 945-945555 2015.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Thomas M. Sutter, Imant Daunhawer, and Julia E Vogt. Generalized multimodal ELBO. In Inter- national Conference on Learning Representations, 2021. URL URU/sttsss//peeeeeeee....e
Forummid:=y21v0RRD
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint arXiv: /61101891 2016.
Stefan Van der Walt, Johannes L Schönberger, Juan Junez-Iglesias François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu. scikit-image: image processing in python. PeerJ, 2ze453, 2014.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. arXiv preprint 17Xiv:1705.10762, 2017.
Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of research and development, 4(1 (11):66-822 -2,6882,1 1960.
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learning. Advances in Neural Information Processing Systems, 31, 2018.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. Attribute2imaee: Conditional image generation from visual attributes. In European conference computer vision, pp. 776-791. Springer, 2016.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
12

---

Under review as a conference paper at ICLR 2023
Yifei Zhang, Désiré Sidibé, Olivier Morel, and Fabrice Mériaudeau. Deep multimodal fusion for semantic image segmentation: A survey. Image and Vision Computing, 105:104042, 2021.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Towards deeper understanding of variational autoencoding models. arXiv preprint arXiv:1702.08663 2017.
13


Reviews:
[{"review_id": "rs_fIDeCd4", "reviewer": "Reviewer_LHYz", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work presented in this paper targets multimodal representation learning, following a line of work that uses a variational autoencoder (VAE) as a basis that is extended to learn a latent representation from multiple modalities, that is observations of the same phenomenon through the lenses of a variety of input sources. The objective is to build a model that scales to a potentially large number of input modalities, and that is capable of reconstructing the inputs, even in the case of missing modalities.\n\nThe training objective is an extension of the traditional VAE ELBO, that considers subsets of input modalities at training time. Instead of using either product or mixture of experts (or combinations thereof) to compute the joint posterior distribution, the authors propose to use a multi-head self attention layer. The claim is that the proposed approach does not require the assumption of conditional independence of a modalities given the latent variable, thus allowing to learn correlations between input modalities, which are encoded in the latent representation.\n\nA series of experiments on synthetic multimodal datasets, that extend classical datasets such as MNIST, FashionMNIST and CelebA to account for a one-hot-encoded of the available labels as additional modalities, the authors compare the proposed method to alternatives from the literature (MVAE, MMVAE and JMVAE), using likelihood and conditional variants.\nFinally, the authors discuss two real-life use cases, providing visual examples of the benefit of the proposed method.\n", "summary_of_the_review": "This is a preliminary work that requires additional work to be understood by a reader, additional mathematical rigor and clarity of exposition. The experimental section is not on par with current literature, and should be extended by considering more realistic and challenging multimodal datasets, as well as appropriate evaluation metrics.\n", "strength_and_weaknesses": "The strength are:\n* This work addresses an important topic in representation learning\n\nThe weaknesses are:\n* The editorial quality must be drastically improved, both on clarity of exposition, and on mathematical rigor. \n\n* Several aspects of the mathematical formulation of the proposed method are elusive.\nThe parametrization of the prior with parameters $\\theta$ is not what you actually do. The prior has no parameters in the remaining of the paper.\nEq.1 has a typo.\nThe distinction between embedding and encoder networks is not clear.\nEq.3 has a typo.\nTraining process is not clear: do you assume to have all input modalities perfectly aligned, but omit some of them at training time?\nThe introduction of function $g(\\cdot)$ in eq.7 is confusing. Are we talking about the encoder network?\nThe “noise outsourced functional representation” is not clear: I assume this is the latent variable.\nThe way the multi-head attention mechanism is used is obscure: keys, values and queries are not clearly specified.\nThe deterministic set representation function is indicated again by the function $g(\\cdot)$, which is confusing.\n\n* The section on total correlation is (to the best of my understanding) not correct. A recent article explicitly tackling multimodal representation learning with a total correlation objective is, for example, [1]\n\n\n[1] @inproceedings{hwang2021-neurips,\n    title={Multi-View Representation Learning via Total Correlation Objective},\n    author={HyeongJoo Hwang and Geon-Hyeong Kim and Seunghoon Hong and Kee-Eung Kim},\n    booktitle={Advances in Neural Information Processing Systems},\n    editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},\n    year={2021},\n    url={https://openreview.net/forum?id=SV4NhqUoO8}\n}\n\n* The experimental section is very weak, especially compared to the literature on the topic, including some of the references cited in the paper, such as Shi et al. ‘19. The datasets are simplistic, compared to the current state of the art, and the metrics used to compare methods are insufficient, see for example Shi et al. ‘19 and subsequent work from the authors, as well as Sutter et al. 20, and subsequent work from the same group.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "F2uwlHc1e-", "reviewer": "Reviewer_y89s", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied multimodal VAE. Compared to existing methods that rely on the conditional independence assumption, this paper relaxes such assumption by introducing Set Multimodal VAE (SMVAE). The author conducted experiments on three multimodal datasets to verify the effectiveness of SMVAE. ", "summary_of_the_review": "My major concern is incremental technical contributions and missing computational cost analysis. ", "strength_and_weaknesses": "Strength:\n\n+ This paper aims to address the scalability issues, which are well-motivated and critical in multimodal representation learning (MRL).\n\n+ The proposed method is technically feasible with intensive theoretical justification.\n\n+ Extensive experiments on multiple datasets empirically validate the performance of proposed techniques, especially by comparing them with existing related works.\n\nWeaknesses:\n- Incremental technical contribution. Leveraging self-attention to model cross-modal interaction is a well-studied idea in MRL empirically and literally. \n\n- Computation overhead. In high-modality cases (e.g., >10 modalities), calculating self-attention is computationally prohibitive. It seems that the proposed method does not take into account this computational issue, which significantly compromises the claimed addressing of the scalability issues.\n\n- Missing details. What is the function of I (the query vector) in Eqn.8? Is it to obtain the mean and variance of the joint distribution?\nIt is a common practice to add modality-type embedding to the input data. Does your method follow the same practice?\nWhat is the purpose of adding noise to variance (Eqn.6)?\nWhat is the function of I (the query vector) in Eqn.8? Is it to obtain the mean and variance of the joint distribution?\nHow many self-attention layers are used in your method?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Et6DhuC4L56", "reviewer": "Reviewer_YcXY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a multimodal VAE, which includes an intermediate set representation. \nThis fixed-size representation of every modality is then used for the mapping to the variational approximation. \nThe authors show the performance of their proposed method on a computer vision study, a label-image dataset, and a robotics application. ", "summary_of_the_review": "To me, this paper is not ready to be published at ICLR. I like the idea of using a set-based approach to multimodal VAEs, but the weaknesses are dominating the good idea. ", "strength_and_weaknesses": "## Strengths \n- **Idea**: I find the idea of a set-based approach to multimodal learning interesting \n- **Motivation**:  the idea is well-motivated including related work. \n\n## Weaknesses \n- **Evaluation**: Previous Work (MMVAE, MoPoE) has used datasets with more than 2 modalities, which is an additional level of difficulty, but highlights the sensitivity of different methods to the number of modalities. MMVAE and MoPoE (and others) have highlighted the trade-off between the quality of generated samples (can be measured using test set log-likelihoods), coherence of samples (could be measured in conditional coherence), and a meaningful latent representation (accuracy of latent representation classification). The evaluation performed in this paper seems limited with only reporting quantitative results for log-likelihoods, but not the other metrics.\n- **Clarity**: Not all terms are well-defined. For instance, I could not find a definition of co-representation. Co-representation seems to be an important term because it is repeatedly used. But without giving a clear definition of it. It is unclear to me where the performance gains come from (Table 1). Is it the set-based formulation? Or the additional conditional total correlation term? In my opinion, it would strengthen the paper if more insights are given into what parts of the proposed method are responsible for the performance improvements, and how sensitive the method to hyperparameters is (see Questions) \n\n## Questions \n- How is $ p(\\mathbb{X}|z)$ calculated? What is the cost of that?\n- Does the proposed framework also work with simpler set functions? The objective in eq 10 does not define the set function. Hence, it should work with any function that fulfills theorem 1. Are there any insights on that? \n- Are there additional hyperparameters needed to tune the optimization of eq. 10? If yes, are there any sensitivity analyses with respect to hyperparameter selection? \n- At the end of section 3.2., the authors mention the use of additional L2-regularization. What is the effect of this regularization? To which term is it exactly applied? What is the regularization weight? It would make the final loss function of the method clear if more details would be provided.\n- What is a co-representation? To me, the term is not clear, and I could not find a definition in this submission. \n- In Section 3.3. the authors say that former multimodal VAEs are not able to learn co-representation. Is there any proof or reference for that? Or at least empirical evidence? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0clt3GEnmC", "reviewer": "Reviewer_PMPV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces the Set Multimodal VAE (SMVAE), a new type of multimodal VAE that uses self-attention to model the joint posterior as a learnable function of an arbitrary subset of input modalities. While previous approaches use basic aggregation functions (e.g., the product of experts) to aggregate the unimodal embeddings across modalities, the SMVAE employs a self-attention module, i.e., a learnable aggregation function. The paper claims that previous approaches are limited by the conditional independence assumption implied by the respective aggregation function that they use and that the self-attention aggregation is better for modeling correlations between modalities. Empirically, the SMVAE shows slight improvements over some of the existing methods as well as promising results in two case studies.", "summary_of_the_review": "The paper proposes a simple yet intuitive and potentially impactful idea. However, there are strong claims that are not sufficiently well supported by the results provided. The paper has the potential to provide a solid contribution to the lineage of multimodal VAEs, but it requires a major revision. In its current form, I tend to reject the paper.", "strength_and_weaknesses": "**Strengths**\n\nThe authors propose a simple yet intuitive and potentially impactful idea: to use a learned aggregation function using an attention mechanism to model the joint posterior in multimodal VAEs.\n\nThe qualitative and quantitative results on bi-modal image/label datasets look promising.\n\nThe two case studies show promising results for real-world applications.\n\n\n**Weaknesses**\n\nThe authors claim that posteriors with basic aggregation functions (e.g., PoE and MoE) lead to a \"defective bound\" for the optimization and a loss of semantic information. However, the paper provides insufficient theoretical or empirical evidence to back up these claims.\n\nThe term \"co-representation\" does not seem to be defined, neither in the manuscript nor in the cited works. The term is used throughout the manuscript, including in central places like the abstract, where it is stated that \"learning the co-representation [...] is a long-standing endeavor\".\n\nEmpirically, the SMVAE shows slight improvements in multimodal generative learning compared to some of the existing methods. However, the quantitative comparison only features very simple multimodal datasets with only two modalities, one of which represents one-hot encoded labels. It would be helpful, if the authors would compare the performance on slightly more realistic multimodal datasets, such as MNIST/SVHN, CUB, and PolyMNIST.\n\nThere are many components and hyperparameters whose effects on the performance are not sufficiently clear and should be studied with additional ablations. For example: L2-normalization, the aggregation vector in Equation (8), the use of multiple attention heads, no use of positional embeddings, etc.\n\nThere are several things unclear about the objective, which does not even seem to be defined explicitly. At training time, does the input to the multi-head attention function (Eq. 8) consists of two times the same subset of modalities? Do these two sets differ in their order of modalities? Is the objective a valid ELBO on the complete set of modalities?\n\nWhere is the proof of Theorem 1? Is it an existing result from previous work?\n\nThe related work section does not make it sufficiently clear how the proposed idea of using attention to model the correlations between modalities relates to existing multimodal models with attention mechanisms. E.g.:\n- Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, Chen Sun: Attention Bottlenecks for Multimodal Fusion. NeurIPS 2021\n- Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, João Carreira: Perceiver: General Perception with Iterative Attention. ICML 2021\n- Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, Pieter Abbeel: Multimodal Masked Autoencoders Learn Transferable Representations. CoRR abs/2205.14204 (2022)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject