Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors are interested in the multimodal learning problem. They want to focus on cases where incomplete data (not data from all modalities) are present during training. To do that, they assume that the fused vector x (output of LMF) can be given by the task classification (or regression) labels times e_l, which is the complete task prototype for all possible modality combinations L. Since (x,y) are given, the paper's main focus is how to learn the vector e_l efficiently. Finally, they test their trained models on unseen modality combinations.
strength_and_weaknesses: > Strengths
1.	The authors try to develop a method to handle more realistic scenarios where some data modalities are missing.
2.	The authors try to make good use of all available problem information with the regularized losses (disentangled contrastive constraints, ensemble learning). 
3.	They reduce the exponential complexity of the problem, which is dependent on the possible modality combinations (parameter l) through low-rank representations and CP decomposition.

> Weaknesses
1.	For the unseen modality combination: We believe that the current experiments do not test the model's performance on unseen modalities. It would make sense if a completely new modality were present or some modality was abnormal or out-of-distribution during testing. The fact that the trained models were exposed to all possible multimodal combinations except the case where all modalities are present seems not to test the unseen case but rather how much the model's performance can be improved when all modalities are available. 
2.	Based on the previous argument, we would like to ask the authors to compute the performance of their method (and the rest) when all modalities are present to see the impact on the model when we have missing modalities. This will help us assess its performance compared to the other methods. Page 8 "...methods (i.e. MCTN, MMIN) depends on the existence of all the modalities to obtain the supervised information in the training stage; therefore, when the samples for training are imcomplete, there will be a big drop in performance. " these experiments could potentially help support this argument. Since the performance of MMIN is very close to IPD, it even, in some cases, outperforms it.
3.	Enrichment of related work, most of the referenced publications are from 2017,2018, with only one from 2019 (the most recent one). Also, give more details about previous papers, for example, " Liang et al. (2019) employs low-rank fusion for each time step of multi-view sequential input. " and how the proposed method compares to Liang et al. (2019).
4.	Some captions of figures are too short. Specifically, Figure 2. Not all diagram variables are explained. Figure 3: similarly, what is k, and what is R?
5.	Figures 3: We believe the figures would be more readable if there were a title above the bar graphs "With, without contrastive constraints."
6.	Figure 2: Please consider adding more details, like the equation connecting the two decoding parts. How are vectors connected to the architecture, and what are the gray cells? In general, the figure and caption need more work. It is tough to understand what the model is doing and how the different pieces connect to the figure. Even if we read the paper and then look at the figure, it isn't easy to map things from text to pictorial.
7.	Throughout the document, there exist claims/sentences that remain unsupported or not fully explained; we would like the authors to go through these sentences and explain what they hold.
a.	Page 4 " Furthermore, the decomposition of A\beta^T does not consider the fine-grained interaction among different modality combinations. " why is this the case? How does CP decomposition solve this problem? Does it?
b.	Pages 6: "We report the metrics of BA (binary accuracy), F1, Corr (Correlation Coefficient), MA (Multi-class accuracy, higher is better), and MAE (Mean-Absolute Error, lower is better). " "We report the common metrics of R@K and MdR. " Can the authors include the definition of these metrics or provide references?
c.	Page 7: "To be realistic, we randomly generate the ratio of 7 pieces (by giving each piece a number from 0 to 1 and employing normalization) ": not clear what normalization is applied and why.
d.	Page 8: " Even MulT utilizes Transformer Vaswani et al. (2017). " what's the conclusion for this sentence? What are the authors trying to say to the reader?
e.	Page 9: "Note that MMIN, MCTN, and most of the existing methods (i.e. Ji et al. (2022);Lei et al. (2021)) can not be used for 7 modalities, thus, we do not compare IPD with them." Please give more details, why not applicable.

clarity,_quality,_novelty_and_reproducibility: The paper needs to be improved for clarity (unsupported claims, not enough details are given, figure 2 needs to be improved (please see weaknesses).
The method seems to be novel, although each submethod seems to be already proposed in other multimodal papers. For example low-rank decomposition and CP decomposition was used in the original LMF paper as well (https://arxiv.org/pdf/2204.13707.pdf).
We believe that the results are reproducible.

summary_of_the_review: We believe that it is important that the authors tried to develop a method for incomplete modality information during training. Because it is closer to real scenarios. The proposed method managed to improve the classification/regression performance when compared to other methods. But we believe that the paper can be improved and make more clear its advantages by addressing our comments in the previous sections.

------------------------------------
Post rebuttal summary:

We thank the authors for answering all of our questions. 
1) We believe the authors' responses for 2.1, 2.2, and 2.3 should be part of the paper.
2) For 2.3 and the L1 normalization, the question referred to normalized data. It needs to be clarified from the text on what the L1 is applied.
3) Figure 2 is more clear now! But it can be further improved; currently, it mainly lists the definitions of each parameter. The authors could reformulate it, so it has a better flow.

After reading the other reviews and authors' responses, we kept our recommended score to 6.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of robustness in the face of noisy or missing modalities and proposes a method called Interaction Augmented Prototype Decomposition (IPD) for the setting where the number of modalities is arbitrary and there are various incomplete modality conditions happening in both training and inference phases. Their approach jointly learns the common and modality-specific task factors via low-rank decomposition, which seems to promote unseen generalization in empirical experiments. They show strong results on 3 major multimodal datasets.
strength_and_weaknesses: Strengths:
1. The paper is well motivated and described. The ideas are clear and experiments are on several large multimodal datasets.
2. The paper is largely clear with clear figures and exposition.

Weaknesses:
1. There are insufficient comparisons to work in missing modalities, with 'Jinming Zhao, Ruichen Li, and Qin Jin. Missing modality imagination network for emotion recognition with uncertain missing modalities. In ACL, 2021.' being the only one. The authors cite 'Paul Pu Liang, Zhun Liu, Yao-Hung Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, and LouisPhilippe Morency. Learning representations from imperfect time series data via tensor rank regularization. In ACL 2019.' but do not compare to it (they also use low rank decomposition) of multimodal representations, and there are also some simple baselines such as including modality dropout during training that should also be compared to.
2. There should be more comparisons in both performance and complexity - the proposed method does better but certainly uses more computation so the tradeoff should be analyzed.
clarity,_quality,_novelty_and_reproducibility: Paper is largely clear. Some concerns with comparisons to existing work.
summary_of_the_review: Some additional comparisons and tradeoff analysis would be good.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper under review considers the problem of multimodal learning in the case where one or more of the modalities might be unavailable during either training or testing. The proposed solution works by decomposing representation space into a collection of orthogonal vectors, a “common” component to capture high-level semantic information, and modality-specific components to capture low-level information associated with each modality.
strength_and_weaknesses: *Strengths:* Dealing with missing modalities during both training and testing is an important practical problem that deserves study. The proposed method outperforms other recent methods on a benchmark introduced by the authors.

*Weaknesses:* I found this paper very difficult to follow. After several close readings, I am still not confident that I understand the technical details of the proposed method. Specifically, 

(1) Are the pairs (x, y) the training data? If so, how is the input x formed?

(2) Is Equation (1) the abstract model of data generation? If so, why is that a reasonable model? If not, then where is Equation (1) coming from?

(3) How does the “maximum suppression” scheme described in Section 3.5 stop Equation (8) from blowing up exponentially?

(4) Is the ensemble learning scheme of Section 3.6 used during training only or also during testing?

clarity,_quality,_novelty_and_reproducibility: *Clarity:* In addition to the difficulties understanding the technical details described above, I also found the overall exposition quite difficult to follow. There is no dataflow diagram, model description, or algorithm explicitly describing the proposed method. The authors make frequent reference to “LMF” without providing any hint to the reader how that algorithm works. The writing style varies considerably from paragraph to paragraph.

*Quality:* I think the submission is of low quality. In addition to the lack of clarity, it seems that the authors have assembled several existing ideas (low rank decomposition, orthogonal decompositions, etc.) and shown that they work well together. But they give little explanation as to why they work well. They offer some vague generalities (e.g. “Intuitively, the common component contains more high-level semantic information (i.e., the reasoning of performance), and the modality-specific part contains more low-level detailed information (i.e., the loudness of voice, the movement range” (p4)), but no rigorous empirical or theoretical justification to explain the choices that were made (this includes Proposition 2, which offers little in the way of understanding).

*Novelty:* No single part of the pipeline appears to be novel, but the combination may be. 

*Reproducibility:* Several key details are missing that would make it difficult to reproduce the results in this paper. Namely, the precise breakdown of the datasets into training/validation/testing sets, the optimization algorithm used and it’s hyperparameters, and how the data was packaged into (x,y) pairs.

summary_of_the_review: I recommend rejection for this paper. The proposed method may work well, but I think the exposition is too difficult to follow for readers to learn much from this work. I suggest that the authors redraft this work with a focus on clarity of method, clarity of data, and clear understanding of why the proposed method works well.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an Interaction Augmented Prototype Decomposition (IPD) model for missing modality learning.
strength_and_weaknesses: The idea is interesting. However, the motivation is not clearly given. Besides, the writing can be improved.
clarity,_quality,_novelty_and_reproducibility: Some figures are unclear. The authors should do more experiments on different datasets, including XRMB and RGB-D. Besides, the authors can public their code for reproducibility.
summary_of_the_review: This paper proposes an Interaction Augmented Prototype Decomposition (IPD) model for missing modality learning. Some key points are easy to follow. However, the clarity needs to be improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new method for multimodal learning in the event of missing modalities, both during training time and at test time. 
I am basing my meta-review mostly on the assessments by Reviewers SiTt and Reviewer pCS1, who seemed both knowledgeable on the topic and seemed to have taken the time to read the paper. The method appears to be sufficiently novel and it's designed for a relatively under-explored setting wherein a modality is missing (or partially missing).
Some issues regarding clarity were raised, with the authors having handled some of them (Fig 2), through more might be done, such as including the responses for 2.1,2.2 and 2.3 in the papers, as suggested by SiTt. 
The main weakness of the paper is the experimental evaluation, which is tricky to do because the method is designed to work in a setting not addressed by previous methods. However, most prior work is not designed to work under these conditions, so a reasonable effort must be made by the authors to make sensible adaptations of existing methods to this setting, such as fine-tuning the baselines on whatever modalities exist during training, and/or pre-processing the data to perform some imputation of the missing values (even just mean imputation). Unfortunately, even though I read the experimental section, I could not discern whether this has been done, as details are missing.

Moreover, the authors might want to include the following contenders for MOSI/MOSEI in a future version:
MAGBERT/MAGXLNET (Rahman et al 2020)
DynMM (Xue and Marculescu, 2022)

justification_for_why_not_higher_score: I could not ascertain that the authors have done a decent enough job of adapting the baselines - in simple, common-sense ways - to their setting.
Decision: Reject