Title: MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Fusion

Multimodal Learning

Normalizing Flows
Abstract: Multimodal learning has gained much success in recent years. However, current multimodal fusion methods adopt the attention mechanism of Transformers to implicitly learn the underlying correlation of multimodal features. As a result, the multimodal model cannot capture the essential features of each modality, making it difficult to comprehend complex structures and correlations of multimodal inputs. This paper introduces a novel Multimodal Attention-based Normalizing Flow (MANGO) approach to developing explicit, interpretable, and tractable multimodal fusion learning. In particular, we propose a new Invertible Cross-Attention (ICA) layer to develop the Normalizing Flow-based Model for multimodal data. To efficiently capture the complex, underlying correlations in multimodal data in our proposed invertible cross-attention layer, we propose three new cross-attention mechanisms: Modality-to-Modality Cross-Attention (MMCA), Inter-Modality Cross-Attention (IMCA), and Learnable Inter-Modality Cross-Attention (LICA). Finally, we introduce a new Multimodal Attention-based Normalizing Flow to enable the scalability of our proposed method to high-dimensional multimodal data. Our experimental results on three different multimodal learning tasks, i.e., semantic segmentation,  image-to-image translation, and movie genre classification, have illustrated the state-of-the-art (SoTA) performance of the proposed approach.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies a key limitation in existing multimodal fusion approaches: leveraging Transformer-based attention mechanisms for fusion hinders the model's ability to effectively capture the complex structures and correlations inherent in multimodal data. To address this issue, the authors propose a method called Multimodal Attention-based Normalizing Flow (MANGO), which builds a multimodal modeling framework using Invertible Cross-Attention (ICA) layers. By introducing three types of cross-attention mechanisms, MANGO effectively captures the intricate structures and relationships within multimodal inputs. Experiments conducted on three different multimodal tasks validate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n\n1）The methodology and evaluation metrics used in the paper are appropriate.\n\n2）The authors propose a multimodal fusion method from the perspective of explicit modeling, which is relatively novel.\n\nWeaknesses：\n\n1）Although the authors point out in the Introduction that \"implicit modeling fails to fully capture the alignment and structural relationships across modalities,\" the discussion on the underlying causes and theoretical foundations of this phenomenon is relatively weak. The paper leans more toward phenomenological description, lacking rigorous mathematical definitions and theoretical explanations, which makes this concept feel more like an intuition rather than a well-grounded theoretical proposition.\n\n2）In Section 4.2 \"Semantic Segmentation,\" the authors claim that “Our model consistently outperforms the prior methods in all evaluation metrics and datasets.” However, upon reviewing the experimental results in Table 1, it is evident that the performance on the Pixel Accuracy metric for the SUN RGB-D dataset is not the best. Moreover, the visualizations in Figure 5 do not clearly demonstrate the advantages of the proposed method.\n\n3）In the Introduction, the authors mention that existing implicit multimodal fusion methods tend to overly rely on a single modality when it provides more data or stronger signals, leading to suboptimal performance. However, the paper does not include any experiments to verify whether MANGO effectively addresses this issue.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a multimodal learning method using normalizing flows. Three fusion approaches are proposed to control the degree of modality mixing, which are placed in the model to progressively increase the mixing degree. Three applications are tested: semantic segmentation, image-to-image translation, and movie genre classification, and superiority in performance is shown.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths\n\n1. The writing is mostly fine.\n2. It seems that there are not many prior methods using normalizing flows for multimodal learning.\n\n* Weaknesses\n\n1. It is interesting to use invertible operations for multimodal learning, but it is not clear why invertibility is required/beneficial for multimodal learning. It seems the reverse direction is not used at all anyway.\n\n2. It is said that the existing approaches lack explicit modeling of inter-modal correlation. However, the proposed method relies on ICA, which is similar to the popular cross-modal attention except for the assignment of each modality to query, key, and value. So the modeling capability of the proposed method doesn't seem to go beyond the popular cross-modal attention. It is not clearly explained that except for the assignment, how the proposed method is distinguished, and which part of the proposed method allows explicit modeling unlike existing methods. \n\n3. It is expected to justify better the motivation of the operation in the proposed ICA. Basically, the attention scores are calculated from one modality (green in Fig. 3) and are applied to combine tokens of the other modality (orange in Fig. 3). How can this resolve the research question (i.e., how to explicit model correlation better than existing methods)?\n\n4. It seems that the authors intend to emphasize the universality of the proposed method through multiple applications. While the experimental results show better performance of the proposed method, it is not quite clear why the proposed method works well in different applications. For instance, semantic segmentation requires modeling interaction between depth and RGB, whose embeddings would have high spatial correlation, whereas in the movie genre classification, a pair of image and text would have almost no spatial relationship but more complicated semantic relationship. It would be better to both conceptually explain and analyze learned models/representations to understand how the proposed method allows flexible modeling of different inter-modal relationship.\n\n5. Discussion on how the proposed method is distinguished from existing methods is rather insufficient. In the introduction, the limitations of existing approaches are mentioned only briefly and roughly. The argument at the end of Sec. 2.1 about the limitations of existing approaches (unable to model long-range dependency and complex cross-modal correlation) should be supported better by explicitly demonstrating whether the proposed method can overcome those limitations and how. It is not quite clear whether the three applications of the experiments actually require long-range dependency and complex cross-modal correlation and whether the proposed method actually improves the performance specifically due to better modeling of long-range dependency and complex cross-modal correlation. \n\n6. In the three applications, further theoretical/experimental analysis on how the proposed method is distinguished from the compared methods. For instance, how different the proposed method is from the fusion approaches in GeminiFusion or BridgeTower. \n\n7. Overall, the novelty of the proposed method is rather unclear to me.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper introduces MANGO framework which is for multimodal fusion that leverages normalizing flows combined with an invertible cross-attention (ICA) mechanism.  MANGO incorporates three partitioning strategies (MMCA, IMCA, and LICA), with LICA introducing a learnable token permutation via LU decomposition. Experiments across semantic segmentation, image translation, and multimodal classification tasks demonstrate state-of-the-art performance.

STRENGTHS AND WEAKNESSES:
Strengths
- The paper is generally well-written, with a clear structure.
- Experimental results are compelling, showing state-of-the-art performance across multiple tasks including segmentation, image translation, and classification.
- The learnable partitioning strategy (LICA) using LU decomposition is an innovative way to address the rigidity of traditional flow partitions.
    
Weaknesses
- No runtime or efficiency comparison is provided. While the latent model approach is designed to improve efficiency, the overall architecture is likely to be computationally expensive compared to simpler fusion methods. 
- Although the paper introduces a method to improve scalability, the authors themselves concede it remains a limitation. The complexity of the model might hinder its adoption in resource-constrained environments or for applications requiring very low latency inference.

QUESTIONS:
- Could the author provide a comparative analysis of computational cost? Like a table comparing MANGO to key baselines on metrics such as total parameters, training time, and/or inference speed on a standard input size.
- Can MANGO handle generation tasks (e.g., image captioning or text generation), or is it limited to fusion-only applications?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MANGO, a novel multimodal fusion framework that combines normalizing flows with a tailored Invertible Cross-Attention (ICA) mechanism. MANGO explicitly models the joint distribution of multimodal data through invertible transformations, enabling interpretable fusion and improved performance across tasks such as semantic segmentation, image-to-image translation, and movie genre classification. It introduces three novel partitioned attention mechanisms (MMCA, IMCA, LICA) and demonstrates SoTA results on several benchmark datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.\tIntroduces a novel approach for multimodal fusion using Normalizing Flows and ICA, addressing key challenges in cross-modality correlation.\n2.\tStrong performance across multiple tasks (semantic segmentation, image-to-image translation, etc.).\n3.\tClear implementation details and experimental setups allow for easy reproduction of results.\nWeaknesses:\n1.\tNo formal proof regarding the correctness or convergence of ICA, which weakens the paper’s theoretical foundation.\n2.\tClaims of interpretability are not demonstrated with concrete visualizations or quantitative analyses, leaving the interpretability claim unsubstantiated.\n3.\tContradictory claims regarding scalability and lack of quantitative comparisons with other models.\n4.\tVisual aids are unclear, and more effective diagrams or illustrations would improve the understanding of the model.\n\nQUESTIONS:\n1.\tWhy have you not provided attention maps or saliency maps to show how the attention mechanisms in this paper capture cross-modality correlations, validating the interpretability claim?\n2.\tYou should clarify MANGO’s scalability and provide quantitative comparisons of computational costs against other similar and SoTA models.\n3.\tYou should enhance architectural diagrams and include visualizations to show better how different MANGO functions work\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}