Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a multimodal Autoencoder framework that learns the multimodal latent representations alongwith the importance of regions in each modality’s representation space in an unsupervised fashion. Multimodal fusion algorithms either use complex architecture representations or use disentangling joint representations for improving generative auto-encoding architectures using VAEs, GANs, WAE and some variants of these. This paper presents an elegant importance based model and architecture that takes into account various local and joint loss functions along with alignment factors to represent the Autoencoder model.
Some questions/comments that would make the paper more readable:
- Architecture diagram: Please provide an architecture diagram for the model to help the readers
- How good are the embeddings for downstream applications: Can we use these representations and compare performance with other learnt embedding models? How would the performance be with missing modalities? 
- Table 1 - It is surprising to see that weighted Precision is less than unweighted Precision - any thoughts?
- Comparison to other Autoencoders: How does this compare to denoising Autoencoder? How does this compare to Wasserstein autoencoders and the variants such as the multimodal factorization model neural architecture?
Figure 2: Any relationship or companion to self-attention weights and word level importances 
- Number of network params compared to MVAE model that the authors have compared to? How fast is the model training compared to other similar models?


review_title: Elegant unsupervised importance based auto-encoding model - good ideas and some early results - needs more experiments and analysis

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors are motivated by two properties that are often missing from multi-view or multimodal algorithms, namely robustness to missing modalities at test time, as well as learning the relevance of modality/samples to the so-called shared latent space in an unsupervised manner.

To do so, the authors first consider that the shared latent space z results from a weighted combination of the outputs of M encoders corresponding to M different modalities.  This appears to be a convex combination (Eq. 1), although it is not mentioned that each weight should be positive explicitly (authors can clarify this in an updated version).  The shared space is also regularized with a squared l2 norm.

The authors subsequently force z to be close to the encodings of each modality (u(x)_j) while also reconstructing each observation from the same modality.  The authors introduce two more loss functions as a way to learn importance weights for samples/modalities (_{ij}).  This is done by a cross-correlation loss that learns high importance when there is high correlation between modality/sample encodings and the shared latent space.    This is also influenced by a hyperparameter indicating prior knowledge on data quality per modality.  The experimental results demonstrate that the importance mechanism improves results, at least for the specific datasets and experimental setting.

Some comments follow below:

-> generalization and robustness:  As an importance is assigned to each sample per modality, what happens when outliers are present in the data?  Could this mechanism essentially overfit such noise (perhaps common in all modalities) due to the individual sample weighting, and lead to a lack of generalization on an unseen dataset?  Robust methods (e.g., depending on l1 regularization) might be able to cope with such scenarios better


-> missing modalities: the authors highlight that the method can handle missing modalities - by setting the importance weight for the corresponding modality to zero.  However there is not enough experimental evidance (unless I have missed something) in the main paper to demonstrate the performance of the proposed method in this respect. Also, models that impose a shared-space assumption (in the most simple way, shared-siamese style encoders leading to common layers) can also handle missing modalities in the same manner - so I wouldn't say that this is the strongest contribution of this paper.


->  The cross-covariance loss function (and in general, soft orth constraints) have been used in the context of private-shared space models (e.g., stemming from the classical Inter-Battery Factor Analysis).  In many ways, it would be interesting to review and link this literature with work dealing with discovering shared-representations (e.g., Deep CCA, Andrew ICML 2013), as this work could potentially be considered an extension of these methods.

-> attention mechanisms are not mentioned in the paper, however Eq. 1 is reminiscent of an attention mechanism.  Could we consider this approach as a form of attention-autoencoder, where the attention weight is used to indicate importance?

-> was the prior hyperparameter that controls the number of corrupted samples tuned in some automatic way? (e.g., grid-search). Of course the features are extracted in an unsupervised manner, however (i) the samples are corrupted by artificial rather than real noise in the experiments and this is known during the experiment, and (ii) in unknown test-sets, the noise distribution might vary and could be unknown.  For a particular choice of hyper-parameter values, the authors state that  this "indicates that these would be the optimal values for best performance in image and speech modalities respectively" - but can we really be confident that these values will be consistently better?  Could this just be fitting the particular dataset properties and harming generalization?

-> optimization: why is the particular approach employed with alternating optimization?   It is difficult to claim convergence for the given setting, but have the authors observed good convergence properties in the model?  What values were given to hyperparameters \lambda for each experiment?  Also, it appears that there are several constraints on z - some countering each other in some way.  

-> continuing from the previous point, for example z has to be equal to each u(x)_j, but also equal to a convex combination of all u(x)_js, while an additional cross-covariance loss is imposed so that z correlates with u(x)_j.  At the same time, u(x)_j must reconstruct each x_j - which essentially means to my understanding that in an optimal setting, the decoder should contract or absorb all information that is not 'shared' amongst the modalities (what the authors call uncorrelated noise).  Does this cause additional instability during training?  Could it be that information inherent only to one modality is still useful, and not uncorrelated noise - which however would be disregarded within this approach?   
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes the IMA model, a scalable model that learns modality importances and robust multimodal representations through a novel cross-covariance based loss function. The proposed model performs unimodal inference in absence of modalities and also addresses the problem of detecting important subspaces in each modality through weighted cross-covariance loss terms, which are minimized by unimodal importance networks. Results are shown that the IMA model is able to distinguish digits from uncorrelated noise, and word-level importances are learned that correspond to the separation between function and emotional words. The multimodal representations learned by IMA are also competitive with state-of-the-art baseline approaches on downstream tasks.


Pros: 

+ Importance-based Multimodal Autoencoder is a very interesting topic, Importance Network Training also would benefit to other fresh ideas and new approaches.
+ I really like the idea of learning modality importances and robust multimodal representations through a novel cross-covariance based loss function. I agree with the authors that this should help a lot for unimodal inference in absence of modalities and also addresses the problem of detecting important subspaces in each modality.

 
Cons: 

- While I like the premise of the paper, I feel that it needs more work. My main concern is that seeking to learn weights $y_{ij}$ (the importance of each sample $x_{ij}$) should not be equivalent to the degree to which $x_{ij}$ does not belong to $R_j$. In other words, if the goal of *importance network* is only filtering uncorrelated noise, how is the importance reflected? Perhaps this should be called correlation. My understanding is that the importance network should be quantified to a weight range, similar to the attention mechanism. If not, the author should make a more clear explanation of the *importance network*. 

- The text is quite hard to read, there are many places in the paper that are not explained clearly, especially the related work. This section more like a pile of other related works but lacks coherence and puts the proposed method into context. 

- The model description section would be easier to understand if the author can add an overall framework figure, especially *Importance Network Training* need a more detailed description.

- At the *Importance Priors* section, what is the $L_{local}$? At *Dataset IEMOCAP* section, the author stated "We also remove the MFCC features, and thus most of the acoustic variability within the utterance is removed resulting in factors of variation (such as emotion) which are more global in nature." What is *global in nature*? What is the difference between defining local and global? 

- The experimental part is a bit confusing and needs more analysis, why is the result of Table 1 worse than those two models: JMVAE-KL and MVAE?

- This paper hardly cited references from 2019 and 2020, and it is recommended that the author pay attention to some updated work.

-  The labels of the figures and tables in the paper should preferably appear and quote in order.
review_title: Official Blind Review #1

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper tackles the problem of multimodal representation learning by proposing an importance based multimodal autoencoder (IMA). The advantages of their proposed IMA are two folds: it is flexible to remove some modalities, as the loss function is easy to include any combination of observations; it allows to learn modality specific features along with the shared latent representations.

Pros:
1. This paper is well-written and easy to understand.
2. The idea is novel. They addressed the important problems of multimodal representation learning, which could have wide applications in multimedia, computer vision and speech/emotion recognition context.
3. The experiments are extensive, and can support their conclusions.

Cons:
Some details are missing: 
1) In Equation (2), what are lamda(align), lamda(rec), lamda(glob)? How do you choose them in your implementations? Are they hyperparameters?
2) What parameters you choose to visual the t-SNE of representations in Figure 1? As I know, the parameters inside t-SNE determine the entire visualization process.
3) In Section 5, why do you use two stages process to train your model? in the first stage, when the importance network is kept constant, does the network output a constant number (zero or one or other number)?
4) I would suggest to have a algorithm figure to demonstrate the relations between different network modules and loss functions. 
review_title: A good paper but some details are missing

Meta-review: The paper proposes an auto-encoder framework IMA, a scalable model that learns the importance of modalities along with robust multimodal representations through a novel cross-covariance based loss function, in an unsupervised manner. They have compared their approach to SOTA methods via multiple experiments and shown how IMA gives better performance.

The authors have addressed some of the reviewers' feedback. However, as pointed out by the reviewers, the experimental section needs better analysis of results and comparison to other methods, and the modeling section needs to be better explained and motivated. The authors have made changes in their revision, however the ICLR review process does not allow for checking the camera-ready. Since we cannot accept the paper in its current form (or with small variations) and there have been many competitive submissions, we would encourage the authors to make their revisions and resubmit to other venues.
Decision: Reject