Importance-based Multimodal Autoencoder
Authors: Sayan Ghosh
Eugene Laksana
Louis-Philippe Morency
Stefan Scherer
Venue:  (2021)
Keywords: Neural networks
Speech analysis
Multimodal
Autoencoders
Representation Learning

Abstract: Integrating information from multiple modalities (e.g.,  verbal,  acoustic and visual data) into meaningful representations has seen great progress in recent years. However, two challenges are not sufficiently addressed by current approaches: (1) computationally efficient training of multimodal autoencoder networks which are robust in the absence of modalities, and (2) unsupervised learning of important subspaces in each modality which are correlated with other modalities.   In this paper we propose the IMA (Importance-based Multimodal Autoencoder) model, a scalable model that learns modality importances and robust multimodal representations through a novel cross-covariance based loss function.  We conduct experiments on MNIST-TIDIGITS a multimodal dataset of spoken and image digits,and on IEMOCAP, a multimodal emotion corpus. The IMA model is able to distinguish  digits  from  uncorrelated  noise,  and  word-level  importances  learnt  that correspond to the separation between function and emotional words.  The multimodal representations learnt by IMA are also competitive with state-of-the-art baseline approaches on downstream tasks.

Under review conference paper ICLR 2021
IMPORTANCE-BASED MULTIMODAL AUTOENCODER
Anonymous authors Paper under couble-bling review
ABSTRACT
Integrating information from multiple modalities (e-g., verbal, acoustic and vi- sual data) into meaningful representations has seen great progress in recent years. However, two challenges are not sufficiently addressed by current approaches: (1) computationally efficient training of multimodal autoencoder networks which are robust in the absence of modalities, and (2) unsupervised learning of important subspaces in each modality which are correlated with other modalities, In this pa- per we propose the IMA (Importance-basee Multimodal Autoencoder) model, scalable model that learns modality importances and robust multimodal represen- tations through a novel cross-covariance based loss function. We conduct exper- iments on MNIST-TIDIGITS, multimodal dataset of spoken and image digits, and on IEMOCAP, a multimodal cmotion corpus. The IMA model is able to dis- tinguish digits from uncorrelated noise, and word-level importances are learnt that correspond to the separation between function and emotional words. The multi- modal representations learnt by IMA are also competitive with state-of-the-art baseline approaches on downstream tasks.
INTRODUCTION
With the ever-increasing amount of heterogeneous multimedia content present on the internet, ma- chine learning approaches have been applied to automated perception problems such as object recog- nition (Krizhevsky al., 2012). image captioning (Vinyals et al., 2015) and automatic language translation (Choi et al. 2018). An important research direction is the problem of learning repre- sentations from multiple modalities which represent primary channels of communication and sensation, such as vision or touch (Baltrušaitis et al. 2018). With respect to this area of research, there are two major challenges in this research area which our paper addresses. The first is the design of encoder networks to enable learning and inference of multimodal representations in the absence of modalities. This useful for scenarios such as sensor failure or imputation/bddircttoonn gener- ation of missing modalities from any combination of the observed ones. The caveat is that to have this property. recent approaches such as the JMVAE-KL model (Suzuki et al. (2016) and MVAE (Wu) Goodman 2018) have encoders with high complexity for a large number of modalities. When M is the number of modalities, JMVAE-KE KL needs 2 sub-networks for every combination of input modalities, while MVAE requires M sub-networks but additional ((2 subsampled loss terms handle missing modalities.
The second challenge that multimodal data, such as emotional spoken utterances or web ages with captions are often generated not only by an underlying shared latent factor, but also by modality-specific private latent factors. For example in spoken utterances, the verbal modal- ity (words) are generated not only due to emotion but also due to syntax and semanties. Function words such as / and the are mostly syntactic and do not relate to emotion, similarly not all recorded audio frames are indicative of emotion. The inference of how relevant sample in each modality shared latent factor (subsequently referred to as importance) is important for downstream tasks. For the remainder of this paper, non-relevant samples are refered to as uncorrelated noise.
In supervised setting, the latent factors and modality importance weights can be learnt from task labels. When labels are absent the unsupervised scenario, for the purpose of this paper we de- fine the concept of modality importances based on correlations between the latent factor and each modality. In the important subspace of each modality the multimodal and unimodal representations both maximally correlate, indicating that samples in that modality subspace can be attributed to

---

Under review as conference paper ICLR 2021
shared latent factor, and not an independent private one. In contrast, for unimportant samples in modality, the correlation is minimal.
The main contributions of our proposed approach are two-fold. The first is a multimodal autoen- coder framework where training requires additional loss terms which are O(M), ie. linear in the number of modalities, and thus only require M per-modality encoders to handle missing modal- ities. Computationally, this is advantageous compared to JMVAE-KL, and MVAE, which require exponential number of sub-networks and loss terms respectively. Secondly, we define the concept of importance an unsupervised setting, and propose novel cross-correlation based loss terms to learn important regions in each modality's 's representation space. The importances are modeled by separate unimodal networks referred as importance networks. slyper-paraneter P5 for the j-th modality controls the integration of prior domain knowledge about the degree of importance in that modality. While not trained on any supervised labels, the learnt importances from IMA are analyzed quantitatively and found to correspond to the separation between digit vs. noise labels and emotion vs. neutral categories.
2 RELATED WORK
Following the great success of deep neural networks for representation learning, the research area of multimodal machine learning is gaining interest (Baltrušaitis et al., 2018). Our proposed IMA model is relevant to two main research areas in this domain, Inter-modality Correlation Learning and Efficient Multimodal VAEs. The idea of learning acoustic embeddings for words has also been explored in Wang al. 2018) and Jung et al. (2019) however we attempt to map words to their affective rather than phonetic representations. In this section, we describe each area and conclude with the similarities and differences between the IMA model and prior approaches.
Inter-modality Correlation Learning: There have been several approaches which measure corre- lations between modalities/sources of data understand how observed data in each modality can be explained by shared underlying concepts. The IBFA (Inter-Battery Factor Analysis) introduced by [Tucker[[ (oxxer((195)) (1958) and its successor, the MBFA (Multi-battery factor Analysis) (Browne) (980) are among the earliest proposed techniques to study shared factors between score sets from batteries of tests. DeepCCA (Deep Canonical Covariance Analysis) proposed by [Benton et al. (2017) learns a deep projection of each modality in bimodal understanding scenario that the projections are maximally correlated, effectively extending the classical CCA technique (Knapp) 1978) to deep neu- ral networks. Our proposed model extends these approaches to also detect important regions of each modality correlated with the shared latent factor.
Efficient Multimodal VAEs: VAEs (Variational Autoencoders) have been applied to multimodal data for applications such as inference and bidirectional generation of modalities. This poses a ma- jor challenge of constructing encoders to model the latent posterior which are efficient in train- ing/inference under any combination of input modalities. Recent work addresses this by focusing on factorized models for efficient inference. |Vedantam et al. (2017) employs a productt-of-experts decomposition with modality specific inference networks to train image generation models. |Wu & Goodman| (2018) propose MVAE (Multimodal Variational Autoencoders), where the latent poste- rior is modeled with a parameter shared product of experts network. (Shi et al. (2019) proposed a mixture-of-experr multimodal variational autoencoder (MMVAE) where the posterior is a mixture of experts instead. These approaches have been extended more recently, for example in multi-source neural variational inference (Kurle et al. 2019) where the multimodal posterior is constructed using learnt and integrated beliefs from multple posteriors, each being informed by a different source. Sutter et al. (2020) introduce novel Sonsen-Shanoon divergence based objective function which can be used approximate both unimodal and joint multimodal posteriors.
While existing approaches attempt to efficiently learn multimodal representations through posterior modeling, our proposed IMA model aligns modalities during autoencoder training for projection to a common space which facilitates inference even in absence of modalities. Only M encoders and O(M) loss terms are required by the IMA model for inference with M modalities. Prior work has also not focused sufficiently on unsupervised learning of modality importances (through detection of subspaces maximally correlated with shared latent factors) which address in this paper.

---

Under review conference paper ICLR 2021
3 MODEL DESCRIPTION
The IMA model consists of two main components : (I) the multimodal autoencoder and (2) the unimodal importance networks. In Figure]wee have provided an overview diagram of the proposed model, including the loss functions utilized in training.
Multimodal Autoencoder: Assume that the input training examples consist of multimodal data, where each multimodal sample is denoted as x [x1, Xx,..XXx There are M modalities and the training set consists of N multimodal samples. The input data X; in the j-th modality is passed through an encoder for that modality and its output denoted as Uy(x)). The latent multimodal representation z could be modeled with different approaches for example, concatenated fusion with fully connected layers (Suzuki et al. (016). We wish to model the latent multimodal representation ((x1, X2)..XM) with a pooling weighted by distinct modality importances Vj (xj) as given by:
x(11, X2, XM)
Σ (x5)(1((()))
Σ (xy) =1
(1)
The multimodal representation z is passed through the decoder networks to obtain the reconstruction is the reconstruction loss for the j-th modality. Catos is global reg- ularization term which encourages z to be centered with zero mean. This term is for regularization of the multimodal representation and expands to Lylos
in the j-th modality as 8z. c(
||x(x1, X2...xx)
Multimodal alignment in IMA For our proposed model, have M sub-networks for each modality, where each j-th unimodal sub-network is trained using the autoencoder reconstruction losses and an additional alignment loss Culigre By forcing the multimodal representation z and each modality's 's view uz(x)) to be similar, we also enforce each unimodal sub-network to learn its contribution to the latent factor z during autoencoder training. Per-sample, this requires only M sub-networks and M additional Ioss terms instead of random subset of 2 losses for sub-sampled training as in the MVAE model (Wu & Goodman (2018) or 2M subnetworks as in the JMVAE-KL model (Suzuki et al. 2016). Similar to MVAE, this model also can learn and infer in the absence of modalities. can be expressed corporated into the overall loss. If the j-th modality is missing, we can set y, (x)) = in Equation]] [] For the j-th modality, c(u) aliggn is the SSE eтror between the multimodal representation z and the uni- modal encoder outputs uj (x4) given by: c(f) (z. Un) = ||z(X1, X2. xx(x))| When there are N training samples, during multimodal autoeneoder learning the loss below optimized:
linear average of the unimodal representations present and in-
N
N
M
Lauto = Agtoo Σ Egiobbaa)++ + (-1
ΣΣ
(6) (xi), st)
+
ΣΣ
(z), my)
(2)
Aatign: Arec and Aglab are the associated hyper-parameter weights for the alignment, reconstruction and global multimodal regularization terms which appear in Equation (2). Since it is not straight- forward to tune such parameters for an unsupervised model, we start with equal weights for all
losses and observe if that is sufficient to minimize all terms simultaneously during training. For the MNIST-TIDIGITS and IEMOCAP experiments, we have found that equally weighing them works, thus Aalign Xrec
Aytos 1.0.
Importance Network Training: Dection[[explann that there are samples in observed data for each modality which do not correlate with the multimodal representation z. For the scope of this paper, we consider such samples as uncorrelated noise in the data. We have assumed the presence of a subspace Ry inside each j-th modality's representation space so that if xj E R,, uj is minimally correlated with z. We seek to learn weights Miy E [0, 1] for the i-th sample in the j-th modality, so that Biy denotes the importance of each sample Xıj (i.e. the degree to which xı does not belong R,). A unimodal neural network is trained to map from Xij Yij for the j-th modality; this is called the importance network. The multimodal autoencoder solcly being trained to reconstruct all modality inputs would not have learnt Vij corresponding to uncorrelated noise. This motivates the need for explicitly defining an importance-hased loss.
correscorreltionn Losses: To train the importance networks, we have made use of a loss function explicitly capturing cross-correlation between z and uj. weighted by yj- Minimizing this cost is equivalent to enforcing zero-correlation between uy and z based on mini-batch statistics dur- ing training, where size of a mini-batch is B samples. This cost can be derived from the definition of cross-correlation, where each i-th sample in the mini-batch is not weighted equally. Recent work

---

Under review conference paper ICLR 2021
(1) local
Importance
Network
Tec
Modality (1)
allgnn
Latent Representation (z)
Encoder Networks
X1
U1
8
Decoder
+
X2
U2
X2
aligo
Importance Network
(2) Lepee
Y2
Modality (2)
1. (2) Lecurr
L
gglab
(2) local
Figure Overview of the proposed model IMA, including the multimodal autoencoder, the impor- tance networks and the main loss functions to be optimized
also applied other independence criterion, such as MMD (Maximum Mean Discrepancies) which act as auxiliary losses in variational framework (Louizos et al. (013). Assuming the latent variables Gaussians, independence and uncorrelated properties equivalent, and define the following alternative loss term based on the Frobenius norm the cross-covariance between z and
Uj. where 8/1
Mey and w/y n/y
Σ
R L Σ w/y [z (us pa))
Σ 2i
we Usj
(3)
Importance Priors: Learning to predict Vij 1.0 V trivially decreases Leorr down zero, and thus we need to regularize the importance network training, through an additional loss function which we refer as c(3) This loss utilizes the prior about how much of the j-th modality is corrupted by uncorrelated noise, and is defined as the KL-divergence D (Beroouli(g) |[Brrooulli((;)) where V0 E. Mk/ is the average value of Vij as computed over a mini-hatch of size Enforcing the loss Lioeat at a sample level would have forced Vij instead. The importance network for each modality minimizes Limp. which is the weighted sum of
Pj [o, 1] which serves as
for all samples, which addressed by defining the loss on a mini-batch
the cross-covariance based loss hyper-parameter weights for each loss term.
regularization term, as defined below. Local and X are
Limp
( local Local
G
(4)
2-1
It is important to note the difference between Vij and bíjj = Bij in terms of notation. Yij is the importance of the i-th sample the j-th modality and is required in the joint representation, whereas its complement Vly is utilized importance network training. The intuition is that a model minimizing Equation would tend leam high values of V/ for maximally uncorrelated samples, which translates to low importance (yn;) for these same samples.
4 DATASETS
We perform experiments with the proposed IMA model on (1) MNIST-TIDIGITS which a multi- modal digit dataset created by pairing image and spoken samples from common image and speech digit datasets and (2) the IEMOCAP corpus (Busso al. 2008). MNIST-TIDIGITS is synthesized

---

Under review as conference paper ICLR 2021
from three datasets for digit recognition from images and speech MNIST dataset (Salakhutdinov & Hinton 2007): TIDIGITS connected spoken digit sequence corpus (Leonard, 1984) and T146 dig- its (Liberman 1993). We have merged these two spoken digit datasets to create MIIST-TIDIGITS, a dataset as large as the number of samples in MNIST. The motivation underlying this is to pair im- age and spoken digits to create a multimodal dataset. ASSCIIEMOAA is a well-known dataset for speech emotion recognition comprising of acted and spontaneous multimodal interactions of dyadic sessions between human actors. The dataset consists of around 12 hours of speech from 10 human subjects, and is labeled by three annotators for emotions such as Happy, Sad, Angry. Excitement, Neutral and Surprise. We train IMA model on all utterances, but perform classification/retiivaa ex- periments only for the four standard primary emotion categories Neutral, Angry, Sad and Happy. with a total of 5531 utterances belonging to these categories (Kim et al. 2013 Jin et al., 2015).
We construct multimodal dataset by pairing the MNIST digit images with the TIDIGIT spoken digits based on the common digit label. We define parameter R called the noise factor to control the amount by which each digit image or spoken digit co-occurs with synthesized noise samples. For each label category e with N image/spoken digits, we create three sets of multimodal samples: N R samples consisting of images of digit paired with synthetic spoken noise; N R samples consisting of synthetic image noise paired with samples of spoken digit and N * (1 R) samples consisting of images of digit e paired with spoken digit c. For all multimodal samples, the ground truth labels are set to the actual digit labels. disregarding the synthetic noise. For example, for multimodal sample with (image '7', spoken noise), the sample label is set to 7. Note that the sample label is not known at model training time.
MNIST-
Consider each IEMOCAP spoken utterance to have word tokens and word- level acoustic feature vectors (for example, descriptors such as pitch). We pair each token with all N acoustic vectors in the same utterance, creating N paired multimodal samples. Thus if an JEMOCAP utterance initially has N word-acoustic pairs. training. We also remove the MFCC features, and thus most of the local phonetic variability within the utterance is removed resulting in global factors of variation, such emotion. The scope of the non-verbal acoustic features co-occurring with a word is expanded to entire utterance. This is analogous to techniques such as skip-gram word embeddings (Mikolov et al.] 2013) where a word context spans beyond adjacent words, resulting in expansion of word pairs to train on.
IEMOCAP:
expanded to N pairs for model
5 EXPERIMENTAL SETUP
In this section, we describe the experimental setup including downstream experiments evaluate the quality of the multimodal representations learnt by the proposed IMA model. The model train- ing is performed in two stages. In the first stage the importance network weights are kept constant, and the multimodal autoencoder is trained. In the second stage the modality-ppecific importance networks are trained; with multimodal autoencoder weights kept constant. This sequence of alter- nating optimization could be performed for a finite number of iterations, but in practice we have found one stage to be sufficient. The image and speech modality inputs for the MIIITTTIDIGITS datasets have 784 and 696 dimensions respectively. The two-layer encoder and decoder networks for the multimodal autoencoder have 500 neurons/layer for image and 100 neurons/layer for the speech modality. A binary cross-entropy loss used for autoencoder training. For TEMOCAP, the verbal embeddings (GLÖVE) and acoustic features have 200 and 55 dimensions respectively. The two- layer autoencoder has 100 neurons/layer in both modalities, and the SSE (Sum of Squared Error) objective is used for training.
5.1 IMPORTANCE PRIOR
We refer the parameter P5 [0, 1] as the importance prior for the j-th modality. Pj can be adjusted to control the proportion of samples detected as uncorrelated noise and is related to do- main knowledge. For example in IEMOCAP, p for text can be set to a very low value, since we hypothesize that vast majority of vocabulary words (c.g. function words) do not correlate with the latent factor (such as emotion). Similarly for the MNIST-TIDIGIT dataset, the amount of uncorre- lated noise (the aforementioned noise factor in Section 4 in the paired multimodal dataset is 10%, requiring a high value of Pj since 90% of samples in each modality correspond to valid digits.

---

Under review conference paper ICLR 2021
5.2 EVALUATION OF IMPORTANCE NETWORK PERFORMANCE
Since the model is unsupervised where the actual importances are available for training, there is no quantitative way to determine the best value of the hyyper-aaamettr Pj for each problem. How- ever we have utilized the ground truth labels from the AANIT-TIDIGIT'S and IEMOCAP datasets evaluating performance of the modality-specifie importance networks. For the i-th sample in the j-th modality, the importance network maps input Xyj to an importance score Yi E [0, 1J. Given the ground truth assignment of zyy to positive or negative class depending on whether it is uncorrelated noise, can compute F1 scores determine the optimal importance prior parameter, well quantify importance network performance with varying values of p)-
For the IEMOCAP dataset, the important verbal category is defined to be the class of all vocabulary words belonging to the LIWC emotional categories (Pennebaker et al., 2001). namely- sad, angry, anxiety, negative emotion and positive emotion. In the acoustic modality, the important category is defined to be the class of non-verbal acoustic samples belonging to the happy, angry and sad emotions as annotated in the IEMOCAP dataset. For MNIST-TIDIGITS, importance is indicated by the presence of actual images/spoken digits in any of the modalities. Uncorrelated synthetic noise is not important, and expected to be detected by the importance networks.
5.3 MULTIMODAL REPRESENTATIONS
MNIST-TIDIIIS: IMA learns importances yiy the j-th modality (images:j 0/speech:j 1) and for the i-th th multimodal sample, if uy are outputs of unimodal encoders, the multimodal representation is obtained by: z (yoo 3/1 (n)) + 3/1)
IEMOCAP: IMA learns token-level importances from the co-occurrences of words (modal- ity 0) and their paired acoustic feature vectors (modality 1), which we combine to produce utterance representations. Each i-th utterance of length L comprises of bag of words {wa, W/2.- ) and bag of acoustic vectors {Vi1, Vi2. V.L). Their respective unimodal rep- resentations are (uo (W:1), -. (w,,)))) and (ur (Viz), .....
... un (Vnr)). Corresponding importances are (wa),)), yo(2e (w.)) and {un (Vi1), -....- (Vir)). The multimodal representation of the whole utterance 8 is given by:
Mo(wa)uo(wa) + ya (v
(vo( + in (vn))
(5)
We also compare multimodal performance with representations obtained only from the unimodal encoders. In all cases, we also perform an ablation experiment which does not utilize the learnt importances yo or y1 and computes an unweighted average of the unimodal representations.
5.4 RETRIEVAL EXPERIMENTS
We conduct an intrinsic evaluation of the multimodal representation quality through retrieval exper- iments. Even if a modality is corrupted through uncorrelated noise, the multimodal representation should filter it out by weighing less. Ideally, two samples with the same label should have similar representations. We define the concept of relevance as: Two multinodal samples are relevant if they correspond to the same ground truth label, even with noise in any of the modalities. For each of e (1,2, 2, 3...N). }. we obtain the K nearest neighbors of the i-th sample in terms of the Euclidean distance and count the number of relevant neighbors as C,. The Precision @ score computed as Pr@K = E: C/NKK
Results are reported both for two and 50-dimensional representations at K E (10, 50). The ground truth lahel is the digit label. The representation of a (MNIST image digit, speech noise) pair should be similar to the representation of (image noise, spoken TIDIGITS digit) if the underlying digit labels are the same. IEMOCAP: The retrieval experiments on the JEMOCAP dataset are utterance level, and consider the primary emotion categories as ground truth labels. For the retrieval experiments, we define two utterances to be relevant if they occur in the same IEMOCAP primary emotion category, one out of: happy, angry, sad and neutral. The Precision @ K scores for K 10 reported for 100-dimensional representations.
MNIST-TIDIGITS:

---

Under review conference paper ICLR 2021
(a) Image/Truth
(b) Image/Pred
(c) Speech/Truth
(d) Speech/Pred
Figure t-SNE visualizations importance network representations learnt by the model multimodal MNIST-TIDIGITS Sub-figures (a) and shows the digit clusters in the representations. The grey cluster corresponds uncorrelated noise. In sub-figures (b) and uncorrelated noise in red; otherwise in blue.
you are for too tenter mentat try adts central porreot
radks I den't really haw any Smegirattin cally Enet all
boM how to calls <alb
It centeinte i5 we eotls It's endle stiphly eaggeniced aetemttffe fact very will oalle if you Jesist or locing wills teeriss Mokii
they dies's die they cully killed thenation aato
atler oulle
mo. Uhe
snallly oall JAs
Litowe
Figure 3: Word level importances learnt by the IMA model observed on five example IEMOCAP utterances.
EXPERIMENTAL RESULTS
We have compared performances of the following models in our experiments: (1) Joint multimodal representations (unweighted and importance-weigeee weighhedd from proposed IMA model as described Section 5.3 (2) Unimodal representations from IMA model which just considers the output uj of each of the j-th modality-specifie encoders as the final representation.
(3) MVAB model proposed in Wu & Goodman (2018) (4) JMVAE- KL model proposed in (Suzuki et al. (016))
6.1 IMPORTANCE NETWORKS
We evaluate the performance of the importance networks by evaluating the learned importances in each modality against domain knowledge based labels for uncorrelated noise presence.
record precision, recall and F1 scores for values of E [0.05- - 0.95]. The best Fl score occurs at p = 0.8 and 0.3, which are empirically the suitable priors for the importance network predictions to agree with external labels in the image and speech modalities respectively. For these optimal values of in each modality, we visualize the top layer of each importance network which implements Yiy for the j-th modality given Xij- Figure 2 a- shows the t-SNE representations for MNIST-TIDÍGITS, with both the ground truth label clusters and the importance network outputs. We observe that the network correctly learns the regions of the representation which correspond to uncorrelated noise. Around 2% by the importance network, which manifests in the small red region in Figures [2]b, d). The t-SNE visualizations for all the representations in the paper obtained using perplexity of 10.0 and delta 0.5.
MNIIT-TIDIGITS: On MNIST-TIDIGITS
data is misclassified
We recorded F1 scores for the importance network with P varying in [0.0, 1.0) for two (a) vocabulary words and (b) acoustic non-verbal features. The evaluation is performed on the validation set. The F1 scores are computed for both the positive and negative classes in each modality, along with the average scores. The average F1 score for words peaks at & = 0.08 (value 0.595), and for the acoustic modality at p 0.7 (value of 0.508). Since a lot of vocabulary words do not have emotional information, p which the average verbal Fl score reaches maximum value. We compare with a random classifier parameterized on r which labels each sample as True with probability r and False with probability 1 where E [0. 1]. For vocabulary words, the best average Fl score obtained score obtained was 0,500 at = 0.1. In both cases, the best F1 score obtained from the importance network outperforms that of the random baseline. We also qualitatively examine the word-level importances learnt by the IMA model for IEMOCAP. For the best performing value of p. Figure 3 shows five example utterances and for each utterance, the words are shaded from light to dark in order of increasing importance. Emotional words such as temperamental, boorish and exaggerated detected as important by the model while function words (to, the, pause 'sil' are ignored.
IEMOCAP:
modalities:
small
0.552 at 0.2; and in the acoustic modality the best average Fl

---

Under review conference paper ICLR 2021
Table 1: Precision @ scores (P@K) corresponding to K=10 and 50 for the IMA model and base- lines for the task of retrieval of similar multimodal MNIST-TIDIGITS paired data.
Multimodal
Image (MNIST)
Speech (TIDIGITS)
Dim
Precision
Model
Weighted
Unweighted
IMA
0.8042
0.7649
0.7448
0.6180
2D
Pr@10 10
JMVAE-KL
0.7392
0.6080
0.6676
MVAE
0.6116
0.4638
0.5285
IMA 0.7812
0.7354
0.7180
0.5750
2D
Pr@50
JMVAT-KL
0.7013
0.5655
0.6298
MVAE
0,5655
0.4090
0.4656
IMA
0.9489
0.9414
0.8500
0.7788
50D
Pr@ 10
JMVAE-KL
0,90T3
0.8618
0.8436
MVAE
0.9015
0.8260
0.8071
IMA
0.9326
0.8988
0.8240
0.7248
50D
Pr@50
JMVAE-KL
0.7645
0.7923
0.7309
MVAE
0.7666
0.7139
0.6573
Table 2: Precision @ scores at K=10 for the proposed IMA model and other baselines for the task of same-emotion utterance retrieval from IEMOCAP (for happy, angry, sad, neutral emotions).
Model
Verbal
Acoustic
Multimodal
Unwt
Wt
Unwt
Unwt
Wt
0.3735
0.402
0.4634
0.4501
0.4781
0.4585
JMVAE-KL
0.3821
0.4831
0,5003
MVAE
0.3874
0.4759
0,5045
6.2 RETRIEVAL EXPERIMENTS
computed by weighing unimodal im- age/speech digit representations with learned importances as described Section [5.3] IMA model would be expected to filter uncorrelated noise and map paired combination of (noise sample in modality
MNIST-TIDIGITS:
Multimodal representations
cluster corresponding the true digit. This property may not be observed in other approaches to multimodal representations includ- ing basclines (Suzuki et al. 2016;) Wu & Goodman 2018). Representation quality can be evaluated through retrieval experiments (which are
digit in the other modality)
explained in Section 5.. Table [ presents Precision @ scores range [O, I]) at K=10 and 50 the test set MNIST-TIDIGITS.
compute precision results for the IMA model as well as the competing baselines, including a multimodal denoising autoencoder (with denoising rate=0.3). The importance-weighted weighted multimodal represen- tations outperform the JMVAE-KL and MVAE models (and also the unweighted multimodal rep- resentation from IMA) due to the model's ability to filter uncorrelated noise in cach modality. The JMVAE- KL model does not weigh modalities, and while the IMA model does this when comput- ing the product of experts, those weights not take into account regions each modality having uncorrelated noise. The denoising autoencoder obtains multimodal precisions (2D Pr@ 10: 0.5975 Pr@ 50: 0.5488 50D @r@ 10: 0.8968 Pr@ 50: 0.7818) which worse than the IMA model. also report performance when only one modality (either images or speech) is present at infer- ence time. For the image modality the IMA model performs the hest, while for speech IMVAF-KI. provides the highest precision.
IEMOCAP: Section 5.3 describes how the aaporrancce-eiggheee unimodal and multimodal repre- sentations are obtained for each utterance, along with representations which do not consider Iearned importances. Table shows the Precision@ @ K scores on the validation set for the multimodal rep- resentation obtained from the IMA model and the comparison baselines. For all approaches, the verbal representation performance is lower than acoustic non-verbal modality which is due to the GLOVE representations capturing semantic (and not necessarily emotional) information from words. The multimodal representations also have superior Precision @ scores compared the unimodal performances. The improvement from learning importances is observed for the verbal modality, where it leads to a 2.67% improvement in the Precision @ 10 metric. This improvement occurs the weighted utterance representations obtained by filtering low importance (ex. func-

---

Under review conference paper ICLR 2021
(a) Image digits
(b) Spoken digits
(c) Noise(lmage)
(d) MoistmMulli
Figure 4: 1-SNE visualizations of unimedal representations leamt by the IMA autoeneoder on MNIST- TIDIGITS. Colors denote digit labels (0:Red: 1:Green; 2:Blue: 3:Purple; 4:arng S:Cy; 6: 6:Yellow; 7:Ma- genta; 8:Olive; 9:Black; Gray: Uncorrelated Nožse). Sub-figures (c) and (d) show the region of image noise in red, superimposed the image and multimodal representation spaces respectively.
(a) Word Rep.
(b) Word Imp.
(c) Acoustic Rep.
(d) Acoustic Imp.
Figure 5: Visualization of different unimodal representations learnt by the IMA model IEMOCAP. Fig- ure (Ja) shows the unimodal word representations. FigureB]b shows the same representations colored with respective word importances. Figure (Jc) and (d) show the acoustic representations colored by emotion and importances respectively. Note that the colors blue, red and yellow respectively denote the happy, angry and sad emotions. The regions learned by IMA as important are in blue; non-important in red.
tion words) out from the utterance. The weighted verbal representations from the IMA model out- perform the JMVAE-KL KL and MVAE baselines on the Precision @ 10 metric, however corresponding scores are less on the acoustic and multimodal representations.
6.3 VISUALIZATION OF LEARNT REPRESENTATIONS
MNIIT-TIDIGITS In Figure 4 b), the unimodal representations uj for E {1, 21 are presented. along with the joint multimodal representation the paired samples. For MNIST-TIDIGITS. and j = 2 correspond to image and speech respectively. Wc note that the loss term Lalign makes the unimodal representations super-imposable. Figure [c. d) shows the uncorrelated noise region in the image embedding space. They get mapped to their relevant digit locations in the multimodal space due to importance weighting (in this case Wapeoeh >> > Wimage). ). IEMOCAP: Figure 5 shows t-SNE visualizations of representations learnt by the IMA model on IEMOCAP. Figures 5 J1 b) show word representations, and we observe that the emotionally colored words are at the periphery and distinct from neutral words. The IMA model also detects most words as unimportant at the optimal value of These word representations are learnt from GLOVE embedding inputs to the model. The acoustic representations are Figures eic. d). The regions which correspond to distinct emotion clusters such as happy, angry and sad are detected as important by the IMA model, and regions with neutral emotion and/or high confusion between emotions are assigned low importances.
CONCLUSIONS
We have proposed the IMA model for robust learning of multimodal representations. The proposed model performs unimodal inference in absence of modalities and also addresses the problem of de- tecting important subspaces in each modality through weighted cross-covariance loss terms, which are minimized by unimodal importance networks. The IMA model is trained on standard datasets such as MNIST-TIDIGITS performed to evaluate quality of the learned representations for downstream multimodal tasks such as digit recognition and emotion understanding. The IMA model achieves performance competitive with baseline approaches. Accuracy on these tasks is also improved by incorporating learnt uni- modal importances into the multimodal representations. Future work could focus on extension to more than two modalities (such as visual), and sequence models such as RNNs (Recurrent Neural Networks) in encoder-decoder networks for utterance level analysis.
well as conversational datasets such IEMOCAP. Experiments are

---

Under review conference paper ICLR 2021
REFERENCES
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis- Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions pattern analysis and machine intelligence, 41(2): 423-443 2018.
Adrian Benton, Huda Khayrallah, Biman Gujral, Dee Ann Reisinger, Sheng Zhang, and Raman Arora. Deep generalized canonical correlation analysis. arXiv preprint arXiv: (00..0270002211 2017.
Michael W Browne. Factor analysis of multiple batteries by maximum likelihood. British Journal of Mathematical and Statistical Psychology. 33(2):184-19))) 1980.
Carlos Busso, Murtaza Bulut, Chi-Chrr Ai--Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang. Sungbok Lee, and Shrikanth S Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335, 2008.
Heeyoul Choi, Kyunghyun Cho. and Yoshua Bengio. Fine- Fine-grained attention mechanism for neural machine translation. Neurocomputing. 284:171-176, 2018.
Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. Covarep collaborative voice analysis repository for speech technologies. In Acoustics, Speech and Signal Processing (ICASSP). 2014 IEEE International Conference on, pp. 960-964. IEEE, 2014.
Qin Jin, Chengxin Li, Shizhe Chen, and Huimin Wu. Speech emotion recognition with acoustic and lexical features. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pp. 4749-4753. IEER, 2015.
Myunghun Jung, Hyungjun Lim., Jahyun Goo, Youngmoon Jung, and Hoirin Kim. Additional shared decoder on siamese multi-view encoders for learning acoustic word embeddings. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 629-636. IEEE, 2019.
Yelin Kim, Honglak Lee, and Emily Mower Provost. Deep learning for robust fcature generation in audiovisual emotion recognition. In Acoustics. Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp- 3687-3691, IEEE, 2013.
Thomas R Knapp. Canonical correlation analysis: A general parametric significance--estingg system. Psychulogical Bulletin, 85(2):410, 1978.
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances neural information processing systems, pp. 1097-1105, 2012.
Richard Kurle, Stephan Günnemann, and Patrick Van der Smagt. Multi-source neural variational inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 41 14-4112, 2019.
Leonard. A database for speakcr-independent digit recognition. Acoustics, Speech, and Signal Processing, IEEE International Conference on ICASSP 84., volume 9, pp. 328-331. IEEE, 1984,
Liberman. Ti46-word spoken digits corpus.
LDC (Linguistic Data Consortium),
https://catalog.ld... ldc.upenn.ccm wwwpemmwewwddc339... 1993.
Christos Louizos, Kevin Swersky. Yujia Li. Max Welling, and Richard Zemel. The variational fair autoencoder. arXiv preprint 1510:00030000 2015.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 311 111111133 19, 2013.
James W Pennebaker. Martha E Francis, and Roger J Booth. Linguistic inquiry and word count: Liwe 2001. Mahway: Lawrence Erlbaum Associates, 71(2001):2001, 2001.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1533-1553, 2014.
10

---

Under review conference paper ICLR 2021
Ruslan Salakhutdinov and Geoff Hinton. Learning a nonlinear embedding by preserving class neigh- bourhood structure. Artificial Intelligence and Statistics, pp. 412-419, 2007.
Yuge Shi, Brooks Paige, Philip Tomr, al. Variational cexturre-o-experre autoencoders for multi- modal deep generative models. Advances in Neural Information Processing Systems, 32:15718 15729, 2019.
Thomas Sutter, Imant Daunhawer, and Julia Vogt. Multimodal generative learning utilizing jensen- shamnon-iivrrence. Advances in Neural Information Processing Systems, 33, 2020.
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint 16Xiv:1611.01860 2016.
Ledyard R Tucker. An inter-battery method of factor analysis.
2302)111133 1958.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. arXiv preprint arXiv:1705. 2017.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2015.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International Conference Machine Learning, pp. 1058-1066. 2013.
Shaonan Wang, Jiajum Zhang, and Chengqing Zong. Associative multichannel autoencoder for multimodal word representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 115-124, 124, 2018.
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learn- ing. arXiv preprint urXiv:1802.055: 2018.
A APPENDIX
A.I MNIST-TIDIGITS DATASET PROCESSING
The datasets which we have considered for our experiments are further pre-processed to create feature sets amenable to model training, these steps mainly involve:
1. Feature extraction from the MNIST images and the TIDIGITS/TI4 speech corpora.
2. Multimodal pairing for model to learn co-occurrences ie. images of spoken digits paired with their respective spoken utterances to create multimodal paired samples.
3. Addition of samples with uncorrelated noise in each modality which co-occur with all im- age/speech digits in the other modality. The model is expected to learn that the noise samples in each modality are not important.
The composition of each noise image speech sample occurrence pattern with the rest of the modalities which the model learns from during training. The only constraint we impose on the noise samples is the locality of their representations in the modality space enable learnt hyperplane separate it out from the rest the samples. We pre-process each image digit from MNIST and each spoken digit from TIDIGIT//T16 to create 784 and 696- dimensional binary feature vectors respectively. We also construct synthetic image and spoken noise samples for each modality which serves uncorrelated noise.
important here; rather it their

---

Under review as conference paper ICLR 2021
A.1.1 PROCESSING OF MNIST AND NOISE DIGITS
Each MNIST digit is of 28x28 dimensionality, and the pixel intensities are in the range [0,255]. We divide each intensity by 255 to normalize the pixel values to the range [0.1]. We also construct images which the same dimensionality (28*28) as the MNIST digits, but consist of white Gaus- sian noise at a predefined level of sparsity K. In our experiments, we have selected K = 0.8. The sparsity is introduced to make the synthetic noise compatible with the cross-entropy loss function. Synthesis is performed by creating 28*28 grayscale image of white Gaussian noise each pixel with an intensity mean of 0.5 and a standard deviation of 0.12, which ensures that almost all the image pixels are in the range [0.1], and then setting K K-fraction of those pixels to zero. Negative intensity pixels are also set to zero. All images (MNIST and noise) are subsequently flattened, resulting in 784-dimensional feature vectors.
A.1.2 PROCESSING OF TIDIGITS/TI46 SPOKEN DIGITS
The speech waveforms from the TIDIGITS and the TI46 datasets are first resampled to 12.5 KHz, and then the TIDIGITS dataset is further segmented by speaker and digit to generate audio files, each corresponding to a single speaker uttering one digit. Noise speech samples are generated by sampling white Gaussian noise for each time step with a mean of and a standard deviation of which corresponds to the same numeric amplitude ranges as the digit audio files. Each synthesized noise waveform is of one second duration and has sampling rate of 12.5 KHz. Subsequently, MFCC (Mel Frequency Cepstral Coefficient) features are extracted from each audio file (both the synthesized noise and the spoken digits), with 20 ms windows; 10 ms shift and 12 cepstral coeffi- cients not including the energy term. Each digit utterance also consists of recording pauses at the start and end time samples. To filter these pauses, 29 frames from either side (left and right) of the utterance midpoint are selected and concatenated to form a 58 12-696 dimensional feature vector; the remaining frames are discarded.
A.2 IEMOCAP DATASET PROCESSING
We represent vocabulary words in the IEMOCAP dataset with their pre-trained embeddings using GLOVE (Global Vectors for Word Representation) (Pennington et al., 2014). popular technique for word embeddings. For model training we have only considered vocabulary words which appear at least times in the corpus, resulting in a vocabulary size of 1215 words. We use the COVAREP toolbox to extract features from speech waveforms in the IEMOCAP dataset. COVAREP is open-source toolbox (Degottex al.) 2014) commonly used in voice-based analysis applications. Acoustic features are extracted at speech frame level to form a 55-dimensional feature vector, and a spoken word is represented by the mean of co-occurring frame vectors.
Acoustic signals incorporate multiple factors of variation such phonetics (acoustic characteristics pertaining to what is said) as well as extra-linguistic factors (acoustic characteristics pertaining to how or by whom it is said) such as emotions (happy vs. angry voice, etc.), loudness, tone of voice, and gender. While all of these latent factors influence the observed acoustic features, they both local as well as global nature. Local factors variation are transient, for example, phonetics (fricative, vowel sounds, etc.) and pause fillers. Global factors of variation influence the whole utterance, for example higher lower average pitch (which can vary with gender, i.c., global attribute).
The MFCCss mostly capture phonetics, and we have removed them as we do not wish to learn how words correlate with their phonetics, but rather to their paadiionlleetrr-linnistte phonetic latent factors would tend to dominate shared latent representation if MFCCs are included during training. We are seeking to learn latent extra- liraa-linuistiic speech factors instead of phonetic components. The dataset also has transcriptions with token-level timestamps. For each spoken token, we average the COVAREP features extracted from the waveform co-occurring with it create a single 55-dimensional acoustic feature vector. Each acoustic vector consists of features useful voice analysis such as NAQ (Normalized Amplitude Quotient), QOQ (Quasi-Opan Quotient) and FO (Fundamental Frequency) with sliding window of 20 ms. duration and 10 ms. shift.
attributes. The
12

---

Under review conference paper ICLR 2021
(0) MMIST/mmage
(b) TIDIGITS/Specch
(c) IEMOCAP/Vebbal (d) IEMOCAP/Speech
Figure 6: Precision. Recall and TIDIGITS dataset. We have selected only the Positive category (positive indicates presence of noise) for porting metrics.
for the importance- autoencoder trained the MNIST-
A.3 IMPORTANCE NETWORK PERFORMANCE
In F present the variation of the precision, recall and Fl scores for the importance network performance on the MNIST-TIDIGIT and IEMOCAP datasets with varying values of the importance prior P in the range [0, 1]. The scores explained in Section 6.1 correspond to these figures, and also include the comparison with random baseline importance network for the IEMOCAP experiments.
Table 3: Experimental setup for digits (MNIST-TIDIGITS) and emotion classification (IEMOCAP)
Parameters
Digits
Emotion
Dataset
(MNISTTIDDIGITS)
(IEMOCAP)
Learning Rate
(0.0625 (0.0625,00125.0225,,
(eta) range
0.5.1.0.2.044......) 0.5.1.0.2.004.0088.)
Train set size
50000
1115
Validation set size
15000
526
Test set size
10000
Neurons/layer
50
50
A.4 MODEL SIZES OF IMA AND BASELINES
The IMA model has the advantage of having only M encoders and O(M) loss terms to optimize compared the JMVAE-KL and MVAE models. That in conjunction with stochasticity assump- tion (present in VAEs) results fewer autocncoder parameters to train compared to JMVAE-KL and MVAE, even when the encoder and decoder sizes are the same. Table the model sizes for the IMA model and the baseline models for the experimental settings described in Section 5 The IMA model has slightly higher parameters compared to MVAE, however that increase comes with the ability of the importance networks to learn uncorrelated noise across modalities.
Table 4; Number of parameters of IMA model compared with the JMVAE-KL MVAE baselines
Model Component
IMA
JMVAE-KL
MVAE
Unimodal Encoders
753000
783000
783000
Multimodal Encoders
0
1309300
Decoders
754280
1217360
1217360
Importance Network
723402
O
Total Size
2230682
3309660
2000360

---

Under review conference paper ICLR 2021
Table 5: Test set accuracies obtained by the multimodal embeddings from the proposed IMA model and other baselines on MNIST-TIDIGITS dataset. Accuracies are shown both for 2D and 50D representations. Multimodal embeddings from the proposed model outperform unimodal and fusion approaches without importance-bart based weighting
2D
50D
Model
image
speech noise
image
speech noise
overall
overall
noise
noise
IMA (importance weights)
83.88
69.29
82.20
96.94
86.99
94.46
IMA (no importances)
79.88
59.86
63.13
96.84
86.54
93.61
IMA (unimodal image)
76.90
12.06
83.05
86.92
9.41
95.31
LMA (unimodal speech)
62.26
66.00
10.80
82.88
90.80
11.70
JMVAE-KL (Szzkii et al. (2016)
53.06
23.24
14.40
97.9
93.85
93.22
M Wu & Goodman (2018)
41.32
39.69
10.38
98.18
94.95
92.37
Table 6: Emotion recognition accuracies for the best model on the validation set achieved by IMA model IEMOCAP dataset, along with IMVAE-KI. and MVAE baseline models.
Model
Verbal
Acoustic
Multimodal
Unwt
Wt
Unwt
Wt
Unwt
Wt
IMA
49.47
53.33
58.80
58.32
59.37
60.57
JMVAE-KL
53,43
60.37
61.17
MVAE
54.96
58.03
62.63
A.5 DOWNSTREAM CLASSIFICATION EXPERIMENTS
The quality of the joint multimodal representation z learnt by the model is quantitatively evaluated by downstream digit and emotion classification tasks respectively from the MIIIT-TIDIGITS and IEMOCAP datasets. We use a two-layer MLP (Multi-layer Perceptron) classifier with ReLU (Rectified Linear Unit) activation functions for the supervised tasks of emotion and digit classifi- cations, given the representations learnt by the importance-baser autoencoder and other baseline models. Table 3 contains details of classifier settings for each dataset. For MIIST-TIDIGITS, the classification is at digit level (each sample corresponds to multimodal spoken and image digit pair), and for IEMOCAP is utterance level (where each sample corresponds to spoken emotional utterance).
MINIIITIDIGITS : Table 5 reports the classification accuracies on the digit classification task for both 2-D and 50-D multimodal representations for the importance based autoencoder and other baselines reported in Section[5.4 The accuracy on the overall evaluation set is reported along with that computed on subset of samples with uncorrelated noise (samples with synthesized image noise: and with synthesized speech noise refer Section 4. In all cases the apportance-weightee multimodal representations from IMA outperform those without importances.
In 2D, the importance-weighted representations from the IMA model outperforms the JMVAE-K KL and MVAE baselines, as well the unweighted IMA representations. For higher dimensionality (50D), overall classification accuracy for the weighted IMA model is slightly less than other baseline approaches (including MVAE and JMVAE-KI KL). The unimodal encoders perform well as they have access to the other modality even presence of noise. For example, the image encoder achieves an accuracy of 83.05% by utilizing the image digit even if the corresponding speech consists of noise, and vice-versa. Note that all classification accuracies (199.5%) ) obtained on MNIST in contemporary literature Wan al. (2013) due to no specialized convolutional/poolia layers and training on samples which include uncorrelated noise.
not comparable to state-of-the-art scores
IEMOCAP: In Table emotion recognition results on IEMOCAP are similar to the preci- sion results reported in Section 6.2 After incorporating importances, verbal accuracy improves (absolute) by 3.86%, and multimodal accuracy improves by 1.2%. The IMA model outperforms achieves competitive performance to the baselines JMVAE-KL. and MVAE for this task of emotion recognition.
14


Reviews:
[{"review_id": "mW3JD5Z8eN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors are motivated by two properties that are often missing from multi-view or multimodal algorithms, namely robustness to missing modalities at test time, as well as learning the relevance of modality/samples to the so-called shared latent space in an unsupervised manner.\n\nTo do so, the authors first consider that the shared latent space z results from a weighted combination of the outputs of M encoders corresponding to M different modalities.  This appears to be a convex combination (Eq. 1), although it is not mentioned that each weight should be positive explicitly (authors can clarify this in an updated version).  The shared space is also regularized with a squared l2 norm.\n\nThe authors subsequently force z to be close to the encodings of each modality (u(x)_j) while also reconstructing each observation from the same modality.  The authors introduce two more loss functions as a way to learn importance weights for samples/modalities (_{ij}).  This is done by a cross-correlation loss that learns high importance when there is high correlation between modality/sample encodings and the shared latent space.    This is also influenced by a hyperparameter indicating prior knowledge on data quality per modality.  The experimental results demonstrate that the importance mechanism improves results, at least for the specific datasets and experimental setting.\n\nSome comments follow below:\n\n-> generalization and robustness:  As an importance is assigned to each sample per modality, what happens when outliers are present in the data?  Could this mechanism essentially overfit such noise (perhaps common in all modalities) due to the individual sample weighting, and lead to a lack of generalization on an unseen dataset?  Robust methods (e.g., depending on l1 regularization) might be able to cope with such scenarios better\n\n\n-> missing modalities: the authors highlight that the method can handle missing modalities - by setting the importance weight for the corresponding modality to zero.  However there is not enough experimental evidance (unless I have missed something) in the main paper to demonstrate the performance of the proposed method in this respect. Also, models that impose a shared-space assumption (in the most simple way, shared-siamese style encoders leading to common layers) can also handle missing modalities in the same manner - so I wouldn't say that this is the strongest contribution of this paper.\n\n\n->  The cross-covariance loss function (and in general, soft orth constraints) have been used in the context of private-shared space models (e.g., stemming from the classical Inter-Battery Factor Analysis).  In many ways, it would be interesting to review and link this literature with work dealing with discovering shared-representations (e.g., Deep CCA, Andrew ICML 2013), as this work could potentially be considered an extension of these methods.\n\n-> attention mechanisms are not mentioned in the paper, however Eq. 1 is reminiscent of an attention mechanism.  Could we consider this approach as a form of attention-autoencoder, where the attention weight is used to indicate importance?\n\n-> was the prior hyperparameter that controls the number of corrupted samples tuned in some automatic way? (e.g., grid-search). Of course the features are extracted in an unsupervised manner, however (i) the samples are corrupted by artificial rather than real noise in the experiments and this is known during the experiment, and (ii) in unknown test-sets, the noise distribution might vary and could be unknown.  For a particular choice of hyper-parameter values, the authors state that  this \"indicates that these would be the optimal values for best performance in image and speech modalities respectively\" - but can we really be confident that these values will be consistently better?  Could this just be fitting the particular dataset properties and harming generalization?\n\n-> optimization: why is the particular approach employed with alternating optimization?   It is difficult to claim convergence for the given setting, but have the authors observed good convergence properties in the model?  What values were given to hyperparameters \\lambda for each experiment?  Also, it appears that there are several constraints on z - some countering each other in some way.  \n\n-> continuing from the previous point, for example z has to be equal to each u(x)_j, but also equal to a convex combination of all u(x)_js, while an additional cross-covariance loss is imposed so that z correlates with u(x)_j.  At the same time, u(x)_j must reconstruct each x_j - which essentially means to my understanding that in an optimal setting, the decoder should contract or absorb all information that is not 'shared' amongst the modalities (what the authors call uncorrelated noise).  Does this cause additional instability during training?  Could it be that information inherent only to one modality is still useful, and not uncorrelated noise - which however would be disregarded within this approach?   ", "extra_scores": {}, "extra_text": {}}, {"review_id": "iwxcSqVxCU6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes the IMA model, a scalable model that learns modality importances and robust multimodal representations through a novel cross-covariance based loss function. The proposed model performs unimodal inference in absence of modalities and also addresses the problem of detecting important subspaces in each modality through weighted cross-covariance loss terms, which are minimized by unimodal importance networks. Results are shown that the IMA model is able to distinguish digits from uncorrelated noise, and word-level importances are learned that correspond to the separation between function and emotional words. The multimodal representations learned by IMA are also competitive with state-of-the-art baseline approaches on downstream tasks.\n\n\nPros: \n\n+ Importance-based Multimodal Autoencoder is a very interesting topic, Importance Network Training also would benefit to other fresh ideas and new approaches.\n+ I really like the idea of learning modality importances and robust multimodal representations through a novel cross-covariance based loss function. I agree with the authors that this should help a lot for unimodal inference in absence of modalities and also addresses the problem of detecting important subspaces in each modality.\n\n \nCons: \n\n- While I like the premise of the paper, I feel that it needs more work. My main concern is that seeking to learn weights $y_{ij}$ (the importance of each sample $x_{ij}$) should not be equivalent to the degree to which $x_{ij}$ does not belong to $R_j$. In other words, if the goal of *importance network* is only filtering uncorrelated noise, how is the importance reflected? Perhaps this should be called correlation. My understanding is that the importance network should be quantified to a weight range, similar to the attention mechanism. If not, the author should make a more clear explanation of the *importance network*. \n\n- The text is quite hard to read, there are many places in the paper that are not explained clearly, especially the related work. This section more like a pile of other related works but lacks coherence and puts the proposed method into context. \n\n- The model description section would be easier to understand if the author can add an overall framework figure, especially *Importance Network Training* need a more detailed description.\n\n- At the *Importance Priors* section, what is the $L_{local}$? At *Dataset IEMOCAP* section, the author stated \"We also remove the MFCC features, and thus most of the acoustic variability within the utterance is removed resulting in factors of variation (such as emotion) which are more global in nature.\" What is *global in nature*? What is the difference between defining local and global? \n\n- The experimental part is a bit confusing and needs more analysis, why is the result of Table 1 worse than those two models: JMVAE-KL and MVAE?\n\n- This paper hardly cited references from 2019 and 2020, and it is recommended that the author pay attention to some updated work.\n\n-  The labels of the figures and tables in the paper should preferably appear and quote in order.", "extra_scores": {}, "extra_text": {}}, {"review_id": "h7U0Sk-mlM1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a multimodal Autoencoder framework that learns the multimodal latent representations alongwith the importance of regions in each modality’s representation space in an unsupervised fashion. Multimodal fusion algorithms either use complex architecture representations or use disentangling joint representations for improving generative auto-encoding architectures using VAEs, GANs, WAE and some variants of these. This paper presents an elegant importance based model and architecture that takes into account various local and joint loss functions along with alignment factors to represent the Autoencoder model.\nSome questions/comments that would make the paper more readable:\n- Architecture diagram: Please provide an architecture diagram for the model to help the readers\n- How good are the embeddings for downstream applications: Can we use these representations and compare performance with other learnt embedding models? How would the performance be with missing modalities? \n- Table 1 - It is surprising to see that weighted Precision is less than unweighted Precision - any thoughts?\n- Comparison to other Autoencoders: How does this compare to denoising Autoencoder? How does this compare to Wasserstein autoencoders and the variants such as the multimodal factorization model neural architecture?\nFigure 2: Any relationship or companion to self-attention weights and word level importances \n- Number of network params compared to MVAE model that the authors have compared to? How fast is the model training compared to other similar models?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HtonQQ1koQs", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper tackles the problem of multimodal representation learning by proposing an importance based multimodal autoencoder (IMA). The advantages of their proposed IMA are two folds: it is flexible to remove some modalities, as the loss function is easy to include any combination of observations; it allows to learn modality specific features along with the shared latent representations.\n\nPros:\n1. This paper is well-written and easy to understand.\n2. The idea is novel. They addressed the important problems of multimodal representation learning, which could have wide applications in multimedia, computer vision and speech/emotion recognition context.\n3. The experiments are extensive, and can support their conclusions.\n\nCons:\nSome details are missing: \n1) In Equation (2), what are lamda(align), lamda(rec), lamda(glob)? How do you choose them in your implementations? Are they hyperparameters?\n2) What parameters you choose to visual the t-SNE of representations in Figure 1? As I know, the parameters inside t-SNE determine the entire visualization process.\n3) In Section 5, why do you use two stages process to train your model? in the first stage, when the importance network is kept constant, does the network output a constant number (zero or one or other number)?\n4) I would suggest to have a algorithm figure to demonstrate the relations between different network modules and loss functions. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an auto-encoder framework IMA, a scalable model that learns the importance of modalities along with robust multimodal representations through a novel cross-covariance based loss function, in an unsupervised manner. They have compared their approach to SOTA methods via multiple experiments and shown how IMA gives better performance.

The authors have addressed some of the reviewers' feedback. However, as pointed out by the reviewers, the experimental section needs better analysis of results and comparison to other methods, and the modeling section needs to be better explained and motivated. The authors have made changes in their revision, however the ICLR review process does not allow for checking the camera-ready. Since we cannot accept the paper in its current form (or with small variations) and there have been many competitive submissions, we would encourage the authors to make their revisions and resubmit to other venues.
Decision: Reject