MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding
Authors: Zhicheng Zhang
Wuyou Xia
Chenxi Zhao
Zhou Yan
Xiaoqiang Liu
Yongjie Zhu
Wenyu Qin
Pengfei Wan
Di ZHANG
Jufeng Yang
Venue: ICML 2025 spotlightposter (2025)
Keywords: emotion
attention
multimodal large language model
large language model
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: A modular attention-empowered Multimodal LLM that delving deep into the fine-grained cues for emotion understanding and cognition analysis.

Abstract: Multimodal large language models (MLLMs) recently showed strong capacity in integrating data among multiple modalities, empowered by generalizable attention architecture. Advanced methods predominantly focus on language-centric tuning while less exploring multimodal tokens mixed through attention, posing challenges in high-level tasks that require fine-grained cognition and emotion understanding. In this work, we identify the attention deficit disorder problem in multimodal learning, caused by inconsistent cross-modal attention and layer-by-layer decayed attention activation. To address this, we propose a novel attention mechanism, termed MOdular Duplex Attention (MODA), simultaneously conducting the inner-modal refinement and inter-modal interaction. MODA employs a correct-after-align strategy to effectively decouple modality alignment from cross-layer token mixing. In the alignment phase, tokens are mapped to duplex modality spaces based on the basis vectors, enabling the interaction between visual and language modality. Further, the correctness of attention scores is ensured through adaptive masked attention, which enhances the model's flexibility by allowing customizable masking patterns for different modalities. Extensive experiments on 21 benchmark datasets verify the effectiveness of MODA in perception, cognition, and emotion tasks.

MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding
Zhicheng Zhang 2
Wuyou Xia
Chenxi Zhao It
Xiaogiang Liu Yongjie Zhu Di Zhang Jufeng Yang" 12
Yan Zhou?
34 Wenyu Qin
Pengfei Wan
(a)
Abstract
Corleona: A man who doesn't spend time with his family real
SOTA NLLAVA-NGXT): dialguee
seeene from film where two characters
shree),)), are engaged in conversation. The character the left, who appears to be
Multimodal large language models (MLLMs) re- cently showed strong capacity in integrating data among multiple modalities, empowered by a gen- eralizable attention architecture. Advanced meth- ods predominantly focus on language-centric tun- ing while less exploring multimodal tokens mixed through attention, posing challenges in high-level tasks that require fine-grained cognition and emo- tion understanding. In this work, identify the attention deficit disorder problem in multimodal learning, caused by inconsistent cross- modal at- tention and layer-by-layer decayed attention ac- tivation. To address this, we propose a novel attention mechanism, termed MOdular Duplex Attention (MODA), simultancously conducting the inner-modal refinement and inter-modal in- teraction. MODA employs correct-after-align strategy to effectively decouple modality align- ment from cross-layer token mixing. In the alignment phase, tokens are mapped to duplex modality spaces based on the basis vectors, en- abling the interaction between visual and lan- guage modality. Further, the correctness of atten- tion scores is ensured through adaptive masked attention, which enhances the model's flexibility by allowing customizable masking patterns for different modalities. Extensive experiments on benchmark datasets verify the effectiveness of MODA in perception, cognition, and emotion tasks. Source code and demo available in htop://rzcheng..o//OMO0.
and speaking the
on right,
wearing
with red flower the lapel The character
looking at or small his and his eggression peace contemplation gravity).
on the right la looking down (
Johnny), possibly at piaco
(c)
Layer-wise Attention Score
1917
(nV
0008
120
Visual
2010
Figure I. Illustration of deficit disorder attention problem. (a) Given the detailed image and lines from The Godfather, (b) highlight incorrect responses, corresponding hallucinated explana- tions, and attached answers. (c) We visualize attention across layers, highlighting inconsistent attention across modalitics.
1. Introduction
Benefiting from the blossom of large language models (Chi- ang al., 2023; Dubey al., 2024; Teknium et al., 2024), multimodal large language models (MLLMs) have shown strong capacity in integrating multimodal data as human (Tong et al., 2024a; Brown et al., 2020; Bai et al., 2023), which illuminate a promising pathway toward Artifi- cial General Intelligence (AGI). Advanced effort has been devoted constructing MLLM (Achiam et al., 2023). fo- cusing on exploring
insightful data curation, model tuning, and evaluation benchmarks. As the controller of agent, MLLMs provide a natural solution by conducting content perception (Liu et al., 2023), understanding role cognition (Dai et al., 2025), and analyzing human emo- tion (Yang et al., 2024). One more step forward into AGI lies in high-level multimodal understanding like humans, in- cluding cognition and emotion. Cognition, as a higher-level capability, requires the ability model relationships and reasoning across modalities (Dai et al., 2025; Pessoa, 2022). Beyond cognition, emotion understanding is another critical aspect of fine- mee-gaineed multimodal comprehension (Yang et al., 2024; Zhang et al., 2023). These high-level multi- modal tasks pose new challenges for MILMs.
"ork done during the internship at Kuaishou 'Project Leader 'VCIP & TMCC & DISSec, College of Computer Science, Nankai University PPegghheg Laboratory "uuiishuu Technology. Corre- spondence to: Jufeng Yang <yangjufeng @ mdn.aai.cuu.c.>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MODA; MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion
While recent MILMS show promising results in basic perception, they still struggle perceive fine-grained de- tails (Tong et al., 2024b), which is essential for understand- ing cognition and emotion. Public benchmarks reveal that these advanced MI.LMS can underperform relative to ran- dom guessing (Yang et al., 2024), with 3 SOTAs achieving approximately 50:50 accuracy in 2-class sarcasm detection on the HFM dataset. This phenomenon arises from an ex- cessive emphasis the dominant modality data, leading to neglect of fine-grained details in alternative modality.
2. Related Work
Multimodal large language model (MLLM) have gar- nered significant attention recently due to their ability to integrate pre-trained foundational models, especially powerful Large Language Models (LLMs)AAchia et al., 2023; Touvron et al., 2023), alongside multimodal en- coders(Dosovisklyy et al., 2021; Radford et al., 2021). These models enhance the processing of multimodal inputs and outputs, as demonstrated in advanced works (Alayrac et al., 2022; Bai et al., 2023). MILLMs leverage attention mecha- nisms facilitate multimodal token mixing, enabling both inductive and deductive understanding across modalities. However, the vision modality's 's potential remains underuti- lized in many of these models. MMVP (Tong et al., 2024b) identifies a critical issue, highlighting how existing MLLMs fail to fully activate the vision modality due to improper handling of low-level visual attributes. Further, Cambrian- (Tong et al., 2024a) confirms this limitation and introduces spatial vision aggregator to enhance visual feature. In this work, investigate the root cause of these limitations, identifying the bottleneck in the design of the multimodal attention mechanism. To address the issue of imbalanced attention scores, we propose a novel multimodal attention that better balances the contributions of each modality.
We delve deep into the reason and analyze the multimodal tokens mixed by attention in MLLM. As shown in Fig. (a)&(b), we observe that SOTA MLLM struggles to capture fin-ggainee details (e.g., eyesights of character), leading to error in emotion understanding. The reason behind this is inconsistent attention across multiple layers in MLLM (63% disparity in Fig. 1 (c)), which we call deficit disorder attention problem. On the one hand, the attention scores in MLLM exhibit a bias towards the language component. On the other hand, layer-by-layer decay of attention further accentuates this disparity. As a result, the attention score disparity across modalities can reach up to 10 times.
Our intuition is that multimodal attention mechanisms often suffer from imbalances between self-modal and cross-modal interactions, leading to suboptimal feature co- oo-operation across modalities. By explicitly separating and modulat- ing these two components, we can better align multimodal features while preserving the unique characteristies of each modality. To achieve this, we propose MOdular Duplex Attention (MODA), which splits attention into self-modal and cross-modal parts, each with its own modulated atten- tion mask. The self-modal attention component focuses on capturing the intrinsic relationships within individual modal- ities. In contrast, the cross- attention component is responsible for aligning features across different modalities, facilitating effective information exchange. At the core of the MODA model is the Duplex (V/T)-Aligner, which maps the tokens into shared dual-modality representation space defined by two gram matrices. Additionally, the Modular Masked Attention component allows the model to adaptively focus on relevant modalities by applying customized mask- ing patterns, further enhancing its flexibility on multimodal understanding tasks.
Understanding cognition and emotion (Fu et al., 2023; Yang et al., 2024) play an important role in the pathway toward building an intelligent agent, except for content un- derstanding demonstrated by prior MLLMs. As two of high-level understanding, cognition (Wang et al., 2024a; Kong et al., 2024; Salemi et al., 2024) typically refers to the ability to make decisions and judgments similar to charac- ters (Binz & Schulz. 2023; Wang et al., 2024c: Deshpande et al., 2023), such as generating website code (Zhu et al., 2024; Wang et al., 2025), or role playing (Chen et al., 2024; Zhang et al., 2018). Emotion mainly depends on the psy- chology assumptions (Zhao et al., 2021; Zhang et al., 2024), where the categorical one is mostly used due to it being easily understandable (Yang et al., 2018; Mai et al., 2022; Lian et al., 2022; Zhang & Yang. 2022). However, it is less explored due its requirements for fine-grained content understanding, which MLLMs can hardly achieve.
A Atention in MILM plays a critical role in addressing the computational and memory challenges inherent in their design. Significant progress has been made in develop- ing efficient attention mechanisms for Transformer archi- tectures, which include fixed patterns (Child al., 2019), combinations of patterns (Zaheer al., 2020), learnable patterns (Kitaev et al., 2020), neural memory (Beltagy et al., 2020), low-rank methods (Wang et al., 2020), and kerne-- based techniques (Choromanski et al., 2021). For example, the Set Transformer introduces inducing points to handle set-input problems (Wang et al., 2020), while the Axial
Our contributions two-fold follows: (1) From novel perspective of the attention shift mechanism, we indicate the key bottleneck of attention among SOTA MILMS and ana- lyze the core reason in depth. We further propose a modular and duplex attention mechanism based on our observation. (2) We investigate a new MI.LMM for perception, cognition, and emotion, enabling applications in fine-grained under- standing and planning. Extensive experiments 21 bench- marks verify the generalization and effectiveness of MODA.
2

---

MODA; MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion
(a)
(b)
(c)
General
Knowledge
Layer 0
Layer
Text
Visual
led: 51083
lee: Bu0ler
Image 0000
Image 0.0355
238-
22-
Avg. Att. distribution
Layer- wise Att. map
Per Layer Att. score
2.1
708-
Layer
CharC
Vision
Layer 16
Layer 24
23-
Tee: SUu6s 6.009
Ted: Butes
0.000
256
43000
24-
24-
512-
-80
4000
Self
Cross
Self
Cross
4
10
10
10
10
230
512
768
230
512
Aug. Diff.: 56.97%
Avg. Diff.: 62, 44%
Figure 2. Analysis of existing MLLMs on four fine- tna-grained understanding tasks. (a) The distribution of attention activation values among visual and textual tokens. (b) The attention map for multimodal tokens among stages. (c) The self- and cross-modal attention activation scores with their disparity among the attention layers.
Transformer applies attention along individual axes of input tensors, reducing computational overhead (Beltagy et al., 2020). These innovations collectively enhance the scala- bility of Transformer models, enabling their application to tasks with large inputs or long sequences (Choromanski et al., 2021; Han et al., 2024). While previous approaches have focused on improving the efficiency and scalability of attention in single-modal tasks, the multimodal context introduces unique challenges, such as balancing attention scores across heterogeneous modalities (Zhao al., 2021). Our work extends this line of research by specifically ad- dressing the multimodal attention mechanism in MILMs.
sequence, self-modal attention and cross-modal attention. We have (-)lm.m)) which represents the tokens derived from the mh modality and rest. For the self-modal and cross- modal attention, we have
Q
Km
O Oseff Softmax
M)V"
(2)
Q K
O cross
Softmax
M)V"
(3)
3.2. Deficit Disorder Attention Problem
Recently, multimodal attention has played a very important role in multimodal areas, including diffusion models that involve cross-modal generation and MLLM that involves cross-modal understanding. The attention mechanism gov- erns token interactions by computing similarities and ap- plying masks. To further investigate the Attention Deficit Disorder (DDA) phenomenon, we conduct series of analy- ses on four categories of fine-grained understanding tasks.
3. Methodology
3.1. Preliminary
Attention Given the input multimodal tokens, X E N N be the number of tokens and d be the dimen- sionality of the hidden state. Let E NxN denote the attention score matrix computed among N tokens, we have A QK /T, and the output of attention layer as:
As shown in Fig. (a), we observe that the attention devoted to visual content is significantly weaker compared to that for the textual modality. This observation aligns with the challenges faced by MILMs fine-tuned from autoregres- sive models in handling fine-grained visual perception. The inherent design of MLLM, which is primarily optimized for text-based tasks, may lead to an underrepresentation of visual features when extended to multimodal contexts. This imbalance highlights a critical limitation in the cur- rent architecture, where the model's proficiency textual processing does not seamlessly translate an equivalent ca- pability. Further, we conduct experiments on Fig. 2 (b))(c), and we observe a distinct cross-attention bias in the lower layers of the model across its 32 layers. This bias is notably inconsistent with the distribution of attention in the higher layers, which are known for their stronger representational capabilities. Specifically, the lower layers tend to focus
QK + M)V.
O -Softmax
(L)
where Q, K, V dddd represents query, key, and value matrix derived from input tokens. Attention is also practi- cally masked M E NxN to filter out special tokens (Li et al., 2023) or conduct causal sequential modeling (Wang et al., 2024b; Achiam et al., 2023).
Multimodal Attention Formally, consider a multimodal token sequence XM comprising M modalities. The total token length is NM = N where N,, represents the length of the m th modality token sequence Xm- The attention can be split into two parts for each modality token
3

---

MODA; MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion
(a)
(b)
(c)
(Sec. 3.1)
MODA Model
(Sec.
3.3) Modular Attention Mask
OK!
0000
Conditional
Mask
Mask
N x Attention Block
Norm
0000
Visual
Text Coord
Yn Johnny cn
MODA
Coord
Ql
(Soc. 3.3) Duplex Attention Alignment
V-Aligner
T-Aligner
tokens
Context
(background, dialogue)
Norm
Figure 3. MOdular Duplex Attention. (a) MODA takes the image and contextual prompt as input, including the background and history of the conversation. (b) With MODA, the token flows are justified in cach Transformer block of MLLM. MODA modifics the deficient attention scores in a correct-after-align manner via (c) Modular masked attention and (d) Duplex -V/T)-aligner.
modal interactions, potentially at the expense of effectively capturing intra-modal features, leading to suboptimal multimodal integration.
disproportionately
ping in diffusion model (Rombach al., 2022), propose mapping the token into the other modality space according to the embedding space bases of the gram matrix. We ex- tract the basis vector of each modality space according to the gram matrix of tokens (Ryu et al., 2023; Peebles Xie, 2023), thus compressing the semantics of each modality and serving transfer for other modalities. Thus, the duplex attention alignment consists of V-Aligner and T-Aligner responsible for visual and language modality, respectively.
This leads to the formal introduction of the Deficit Disorder Attention (DDA) problem. Given the visual tokens æ/, and text tokens of in the block 1, the multimodal attention builds the link from two parts (i.e., self- modal of 2f+1 xtt xr{t1 and cross-modal =/ + x[+1 xyy - 21+1 where the links are commonly implemented by the pair-wise token similarity and weighted sum. However, the modality gap between tokens decrease the magnitude of links, as we ob- served, the link value of at + I 1+1 and 2/ + x++1 decays exponentially with depth (a (a', x x2 # 1). This mis- alignment propagates layer-wise, causing the cumulative error in cross-modal interaction to grow as
Specifically, for the m modality, the space bases are given according to the normed gram matrix ||GT||| E gdxd where G the inner product between tokens and j:
G
K Th K
K
KT
EDDA
II
(4)
where K are the key states of the mh modality tokens and No, is the number of token belong to modality m. By includ- ing the base vectors the space defined by the Gram matrix, we can effectively capture the relationships among the to- kens within the mht modality. This allows us to construct feature representation that is not only rich in information but also maintains the intrinsic structure of the data.
where 6 denotes the layer-specific alignment error. This phenomenon aligns with the theoretical insights in (Dong et al., 2021), where pure attention mechanisms experience rank collapse, a critical factor that exacerbates the imbalance in attention distribution.
3.3. MOdular Duplex Attention
As a following product, the normed gram matrix serves as a cross-modal token transfer function, enabling an efficient transformation of tokens from other modality m into the modality m as kernelized mapping function f Rd Rd The aligned tokens are computed follows:
When the gap across modalities arises, we propose to align the tokens from multiple modalities the attention, which we call modular duplex attention (MODA), MODA first splits multimodal attention into the modality alignment part and the token focus correction part.
K K K T GT
(6)
Duplex Attention Alignment To reduce the modality inconsistency, natural idea is to align them. Inspired by the recent advance of visual- language embedding space map-
where T represents the value representation from other modalities m. The mapped tokens are further fused with the original ones to enhance the token similarity among

---

MODA; MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion
Table Ablation Study, We conduct experiments on four types multimodal tasks, including general QA (G), knowledge QA (K). OCR&Chart QA (O), and vision cenornc QA (V). The lines with blue shallow indicate the optimal setting for our method. If not otherwise specified, this setting used for all subsequent experiments.
(a) Module
(b) Attention Alignment
(c) Attention Fusion
(d) Attention Mask
MDM DAA
G
K
0
V
align
G
0
V
fusion
G
K
V
mask
G
V
63.6
44.0
60.8
38.0
MLP
69.5
47.5
66.8
46.0
Xp
69.2
45.4
60.9
42.6
Inf
67.8
47.6
63.
48.1
69.2
45,4
60.9
42.6
+2xMLP
66.5
48.6
67.9
49.1
Xa
67.8
47.6
63.3
48.1
Fix
70.1
49.0
67.0
52.3
67.8
47.6
63.3
48.1
+GeLU
69.5
49.1
64.0
54.2
Con
69.3
48.3
67.0
54,3
Attn.
69.3
48.3
67.0
54.3
69.3
48.3
67.0
54.3
+CoV
69.3
48.3
67.0
54.3
Add
62.2
47.6
67.2
52.2
[M]
69.5
47.5
66.8
46.0
all the modalities, Due to the substantial computational expense associated with training a complete MILM,, we utilize token merging and LoRA-based tuning to develop the fuser. Notably, the computation in the alignment stage keeps linear complexity the token number, since the matrix among tokens is only conducted in the first round.
matrix to serve as critical indicator, guiding the model in identifying which components should leverage modality location priors. This separation allows for more precise control over how tokens from the same modality interact with each other versus how they engage with tokens from other modalities. The self-modal attention, represented by O self, focuses on refining the relationships within the same modality, ensuring that relevant information is effectively propagated through the layers. Conversely, the cross-modal attention, denoted by O Ocross. facilitates the exchange of information between distinct modalities, enabling the model to leverage complementary features.
Modular Attention Mask Attention mask controls the flow of tokens across transformer layers and induces the positional bias for MILM (Wu al., 2024). To better fit the requirements of the multimodal token sequence, assign modulated attention mask for each modality, where the mask is split into M and MM responsible for self- and
cross-modality, respectively.
4. Experiment
nTh Km T + M")V""
O self = Softmax
4.1. Benchmark Datasets
QTK.m. Km
Perception: Following (Tong et al., 2024a), we conduct experiments on types of perception task (i.e., general, knowledge, ocr, and vision-centric) across 16 benchmarks:
O Ocross Softmax
+ +""
(8)
MME (Fu et al., 2023), MMBench (Liu et al., 2025),
To alleviate the collapsed attention matrix and prevent it from under-smoothed tokens. We first introduce a modular attention mask that stores unnecessary attention values as neeudo-attention scores (Yin et al., 2024). For each row, representing the attention scores for the i-th token, the se- quence length that the token can attend to is fixed at n. Consequently, each row contains scores, which are allocated to the excess values. The at- tention scores are formally represented using a masking strategy with decay rate as follows:
SEED (Li et al., 2024), GQA (Hudson & Manning, 2019).
ScienceQA (Lu et al., 2022), MMMU (Yue et al., 2024),
Math Vista (I.u et al., 2024), A12D (Kembhavi et al., 2016),
ChartQA (Masry et al., 2022), OCRBench (Liu et al., 2024),
TextVQA (Singh et al., 2019), DocVQA (Mathew et al., 2021), MMVP (Tong et al., 2024b), RealworldQA (xAI, 2024), and CV-Bench (Tong et al., 2024a). We adopt GPT4 score evaluate response.
pseudo-attention
Cognition:
Following (Dai al., 2025), we conduct exper- iments on MMRole to evaluate role-playing performance instruction adherence, fluency, coherency, image-text relevance, response accuracy, personality consis- tency, knowledge consistency, and tone consistency.
q _2kl
PIL
Pi(n-1) Pi(n-2)
from 8 aspects:
AMM-
(9)
k
k
Emotion: Following (Yang et al., 2023; Huang et al., 2024). conduct experiments 4 benchmark datasets. MVSA-S and MVSA-M (Niu et al., 2016) are datasets used for sen- timent polarity classification (positive or negative), while TumEmo (Yang et al., 2021) is a multimodal dataset de- signed for classifying six basic emotions. Additionally, HFM (Liu et al., 2022) a multimodal dataset focused on recognizing high-level implicit emotion of sarcasm.
Pbase
Pij
Pbase
(j 1)B
(10)
Except for the absolute location prior information, we fur- ther introduce the modality location to enforce the model to correct the token flow. We introduce the normed gram ma- trix as an indicator, to find out the part should be carried with modality location priors. We introduce the normed Gram
5

---

MODA; MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion
Model
General
Knowlodge
OCR & Chart
Vision-Centric
Mall Vista
OCRBench
RealworkOQ
CV BeneBd 30
CV VV-Beneh to
MME
MMB
SEED
GQA
Avg
SON
MMMU
A12D
AV8
ChanQA
TouVQA
DoeVQA
AVE
MMVP
Method
GPT-4V
63.0
1409.4
75.8
69.1
36.8
65.2
75.7
56.8
49.9
78.2
77.4
78.5
64.5
78.0
88.4
62.4
50.0
61.4
64,3
73.8
Gemini-1.0 Pro
1496.6
73.6
70.7
79.5
47.9
45.2
65.9
Gemini-1.5 1,5 Pro
58.5
52.1
80.3
81.3
73.5
86.5
67.5
G.ok-1.5
53.6
52.8
88.3
76.1
78.1
85.6
68.7
MM-1-8B
1529.3
72.3
69.9
72.6
37.0
35.9
MM-1-30B
1637.6
75.1
72.1
81.0
44.7
39.4
Base LLM: Llama 3 3-Jns-Bh 8B
Mini Gemini IID 8B
72.7
1606.0
72.7
73.2
64.5
55.7
75.1
37.3
37.0
73.5
62.9
59.1
47.7
70.2
74.6
51.5
18.7
62.1
62.2
63.0
LLaVA-NeXT-sl NeXT-8B 8B
72.5
1603.7
72.1
72.7
65.2
55.6
72.8
41.7
36.3
71.6
63.9
69.5
49.0
64.6
72.6
56.6
38.7
60.1
62.2
65.3
Camhrian- 18B
73.1
1547.1
75.9
74.7
64.6
61.3
80.4
42.7
49.0
73.0
71.3
73.3
62.4
71.7
77.8
65.0
51.3
64.2
72.3
72.0
MODA-8B 8B
72.1
1535.9
73.8
74.9
63.0
61.5
80.4
43.1
48.8
73.6
72.0
74.3
65.2
70.4
78.1
66.0
52.6
64.1
73.5
73.8
Base LLM: Hermes2 Yi :4B
Mini Gemini HID 34B
76.2
1689.0
80.6
75.3
65.8
62.4
77.7
48.0
43.4
80.5
68.1
67.6
51.8
74.1
78.9
63.8
37.3
67.2
71.5
79.2
LLAVA-NeXT-344
76.0
1633.2
79.3
75.9
67.1
62.5
81.8
46.7
46.5
74.9
67.7
68.7
54.5
69.5
78.1
64.0
47.3
61.0
73.0
74.8
Cambrian-13344
1-34-
76.8
1689.3
81.4
75.3
65.8
67.0
85.6
49.7
53.2
79.7
71.9
75.6
60.0
76.7
75.5
68.5
52.7
67.8
74.0
79.7
MODA-34B
76.7
1639.2
82.3
75.8
66.2
69.5
88.1
52.5
54.0
83.4
74.7
79.8
62.7
78.3
78.2
69.9
53.8
68.5
75.8
81.3
Table 2. Comparison of MODA with other leading MILM framework on twelve perception benchmarks. MODA outperforms other open-source models and achieves competitive performance on a number of benchmarks, compared to proprietary models such as GPT-4V, Gemini, and Grok-1.5. The reported numbers of leading MLLMs come from (Tong al., 2024a).
4.2. Settings
enhance multimodal attention?
We set the same experiment setting as (Tong et al., 2024a; Liu al., 2023). We adopt CLIP (ViT-L/14) (Radford et al., 2021) the visual encoder. For the foundational large language model, we choose models from different scales, i..., 8B: 3
Response to RQ1: Modality Axis Transfer we analyze the effectiveness of duplex attention alignment in facilitating cross-modal feature transfer by examining its ability to align modality-specife features along a shared latent axis. This motivated by the need to reduce modality gaps and ensure effective information exchange between modalities. We de- sign experiments to test different variants of duplex attention alignment, such as using covariance matrices, attention head configurations, and linear vs. non-linear transformations.
(Dubey et al., 2024) and 34B: Hermes2-Yi-34e (Young et al., 2024). MODA is trained for epoch with a batch size of 2048, using the AdamW (Loshchilov & Hutter, 2019) optimizer with cosine learning rate schedule. The learning rate is set to 2e-5 for LLM and 2 for visual encoder, respectively. The warmup rate is 0.03.
Response to RQ2: Modality Position Bias we investi- gate the role of the modular attention mask in addressing modality position bias and improving attention distribution. This analysis is crucial for understanding how the mask pre- vents attention collapse and ensures balanced contributions from all modalities. We experiment with different masking mechanisms, such as traditional infinity masking, fix-valued masking, and learnable masking. These variants are eval- uated on tasks involving long sequences and imbalanced modality contributions, such as vision-centric perception and knowledge understanding.
4.3. Ablation Study
To investigate the effectiveness of duplex attention align- ment and modular attention mask, we conduct a component- wise ublation study in Table 1. For ablation studies, we train the MLLMs at the scale of 8B, with the base LLM of Llama- I-s-BBB. For a fair comparison, all models are trained on 700K data samples for epoch. We further discuss each component by conducting in-depth analyses of their variants to answer the following research questions.
RQ1: How does the design of duplex attention align- ment impact cross-modal feature transfer?
Response RQ3: Multimodal Attention Matrix we analyze the interaction between duplex attention alignment and modular attention mask by studying their combined effect on the multimodal attention matrix. This is motivated by the hypothesis that the two components work synergis- tically to improve multimodal representation learning by enhancing both alignment and attention distribution. We de-
RQ2: How does the modular attention mask address modality position bias and improve attention?
RQ3: How do the proposed duplex attention alignment and modular attention mask respectively interact to
6

---

MODA; MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion
Model
Cognition
Text
Visual
Text
Visual
Instectioon Adllernnce
Brage Text relevanee
Peronnllty Comiseency
Knowkold Consistrncy
Responie Accuncy
Trae Ovmismnc
Per Layer Att. score
Chlkaenry
Flancyy
Method
Arg
La
GPT-4 Turbo
1,099
1.055
1,082
1,0844
1,097
1,022
1.368
1.103
1.16T
Gemini 1.0 Pro
1.007
1062
1.013
1.050
Chodee 3 Opus
1.157
1.127
1.070
1,149
1.167
1,146
1219
1.168
1,213
OWen VL
1.014
1,012
1,035
1,034
1,029
1,022
1,021
1.001
Base: Lomar 3-fos 8B
Mini Gemini HLD 9B
0,878
0.884
0.912
0.898
0.864
0.853
0.855
0.876
0.852
LLAVA-NeXT-58
0.988
0,980
01966
0.964
0,939
Aug.
50F Cross
$6.97%
62.44%
Aug. 50.1%%
Aug, 41ff:41.0154
Self Croos
Cambrian- 1-88
0.805
0.9001
0.987
0.934
01886
0.880
0.860
0.892
0.838
(a) Multimodal Attention
(b) MODA
MUDA-SB
0.972
0.976
0.992
0.985
0.970
0.970
0.969
0.945
MMRole-91
0.994
1.000
0.997
0,993
1.000
0.992
0.988
Figure 4. Analysis of multimodal attention and MODA. (a) Multi- modal attention: average difference between self and cross-mes modda attention is 56. 56.97% for text and 62.44% for visual. (b) MODA: average difference is 50.31% for text and 41.01% % for visual.
MODA-SI SD
0,995
1,000
1,001
0.999
0,993
0,989
1,001
0.991
0,988
MMRole-9B (In-Tex))
1.000
1.012
0.997
MODA: HB (In Test)
1,000
1,002
1.001
1.000
0,998
0.992
1,013
0.996
0.996
MMRals-9B (Ous: Test)
0,592
0,999
0.933
0.979
0.963
0.77
0.962
MODA-SB (Ous-Test)
(1.984
0.995
1,002
0.99
0.981
0
0.970
0.980
0.965
Table 3. Comparison of MODA with other leading MLLMs and cognition task-specialized methods on MMRole benchmark. The numbers of leading MLLMs come from (Dai al., 2025).
Perception Benchmark. To assess the effectiveness of our proposed model, we compare it against state-of-the-art Multimodal Large Language Models (MLLMs), including the Mini-Gemini-HH series, the LLa VA-NeXT series, and the Cambrian-1 1 series. We conduct comparison under two settings, where we tune these MLLMs from 8B and 34B scale large foundation models. Our proposed MODA out- performs other models of similar scale, including LLaVA- NeXT and Cambrian, achieving an average improvement of 1.0 for the base Lllama-3-Is-8B model and 0.9 for the base Hermes2-Yi-34B model. In vision-centric and OCR tasks, which require fine one-grained understanding, MODA consistently performs better, achieving metric of 66.0 for the vision-centric average and 74.7 for OCR & Chart tasks.
Model
Emotion
MVSW (ACE))
MVSW (A00)
MNSM (91
MVSA (FI)
HPM (ACC)
TumEmoo
TumFina
HPM (FI)
Avg
Method
GPT-4V
0.633
0.507
0.570
0.609
0.631
0.608
0.612
0.764
0.765
Gemini Pro
0.666
0.634
0.637
0.699
0.657
0.598
0,582
0.574
0.683
Claude 3 Opus
0.628
0.626
0,613
0.635
0.629
0,580
0.574
0,879
0.687
OWen VL Max
0.643
0,647
0.645
0.669
0.627
0.565
0,595
0.886
0.701
Bore: Ziame J Jus 80
Mini-GemiiniHHD..0
0.482
0.571
0.487
0.643
0.246
0.49%
0.593
NeXT
0,576
0.59]
0.543
0.617
0.607
0.547
0,533
0,572
0.551
Cambriann1-88
0.547
0.694
0.661
0.662
0.579
0.439
0.344
0,512
0.477
This highlights the model's 's ability in tasks demanding fine- grained perception, further reinforcing its superiority.
MODA-SR
0,588
0.702
01706
0.628
0.619
0,559
0.548
11585
0.563
Emotion Specinliee
MFCI
0.755
0.742
0.732
0.705
0.688
0.687
MULSER
0.757
0.755
0.739
0.738
0.775
0.775
CMGCN
0,733
0.720
0.697
0.683
0.575
0.441
-ognition-specific & Emotion-specific Benchmark. We evaluate a diverse set of MLLMs on both cognition- centric and emotion-centric benchmarks, designed to as- sess key dimensions of cognitive and emotional understand- ing across various aspects. On the cognition benchmark, MODA outperforms open-ended models of Cambrian-1 I (0.981) and I.LaVA- (0.979), achieving an average score of 0.996, and performs comparably to close-ended SOTA models like Claude 3 Opus (0.995). With cognition task-specific tuning, MODA achieves better performance, particularly excelling in fluency (0.999) and personality consistency (0.998). On emotion benchmarks, MODA out- performs open-ended models like Cambrian-1 (0.628) and LLaVA-NeXT (0.624), with an average score of 0.657, and achieves comparable performance to aask-specallized SOTA models like SPFVTE (0.738) and MULSER (0.739) No- tably, MODA excels in TumEmo (0.747) and HFM (0.753) benchmarks, demonstrating strong performance in emotion recognition tasks due to its ability to capture fine-grained emotional features and fine- grained details.
SPFYTE
0.806
0,801
0.799
0.788
0.883
0.79
MODA-8R
0.841
0
0.802
0.790
0.778
0.885
0.881
Table Comparison of MODA with other leading MLLMS as well as emotion task-specialized methods on four emotion benchmarks. The reported numbers of emotion specialliee meth- ods come from their official manuscripts. The missed average per- formance of emotion speciaaiise methods due to missed datasets.
sign experiments that compare the joint use of these compo- nents against their individual use, as well as against baseline models without either component. Tasks such as question answering and multimodal summarization are chosen to simultancously evaluate alignment and distribution.
4.4. Results
As shown in Table 2, Table 3, and Table 4, we demonstrate the main results on 21 popular benchmarks for multimodal perception, cognition, and emotion tasks, respectively.
7

---

MODA; MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion
ROSES ARE RED
Human: Why this image funny?
The humorr image comees from the unexpected twist c assic poem Roses are red, violets are blue! Instead of romantic sweet sentiment, the poer is humoroussy altered to a more mundare and slightly unsettling scenarto, which is bug in the bed. This justaposition of traditional romantic poom with common, albeir slightly unpleasant, hausehold occurrence
Cambrian-1:
comedic effect.
MODA: The image is funny because features cartoon character, specifically Homer Simpson from the popular television show "The Simasons," who is depicted humorous and unexpected situation. He is lving bed with a red rese in his mouth, which an unusual and comical scenaric for the character. The in age also includes a caption that reads "Roses are red, bug yo bed," adding to the humor of the scene. This combination of elements crnates lighthearted and amusing visual that likely elicit luggttr from viewers far liar with the character and the show.
BUG YO BED
(a) Fine-grained Perception
Phoebe Buffay: Look at this photol lin't it just a fairy-tale weddling, until, you know, it wasn't?
Ross Geller played by Cambrian-1: Phoebo, you're right, the wedding does lock fairy-tall tale soene, but it's not quite perfect see, bride, Fmilly mistakerly called Rachel by the groom, Ross, It's a liit of mileup, but it's all good. It's just one of those little hiccups that happens in real life, you know?
Ross Geller played MODA: Oh, Phoebe, you're right. It's like fairy-tale wedding, with besutiful bride and handsome groom, and the elegant setting. But, as we all know; fairy tales don't always have a happy ending. In this case, Ross made mistake that che changed the course of weddling, and it's a moment that will remembered everyone. It's reminder that even the most perfect of settings, life can throw curveballs that we have to navigate.
(b) Cognitive Conversation Analysis
Post: just that the weekends gone.
My photogenic little man
Post: New GIPHY!
Human: is is the sentiment of the above image tec par?
Human: What is the somtiment of above mage zai??
Human: What is the sent ment of the above image text
Cambrian-1: Sad V
Cambrian 1: Anticipation X
ARK
Cambrian 1: Happy X
MODA: Sad
MODA: Love
V
MODA: Angry V
(c) Emotion Understanding
Figure 5. Visualization results of state-of-the-art (SOTA) Multimodal Large Language Models (MLLMss) and our proposed MODA on
fine- fine-grained multimodal tasks, including perception, cognition, and emotion understanding,
Johneay: ugo bought the book, wouldntt even have act, just mystt On Gudfsther don't know what do. don't know do (oryin)
main charameris agg just
Analysis of Attention. We analyze the attention distribu- tion to evaluate the effectiveness of the proposed MODA in terms of attention distribution (Fig. 4). We conduct exper- iments on fine-grained understanding tasks, i.e., emotion recognition based on visual-textual affective cues. Base- line attention exhibits imbalanced scores, particularly in deeper layers, highlighting its tendency to over-focus on dominant modalities while neglecting others (i.e., visual af- fective cues). Our MODAachieves consistently high across all layers, preventing degradation and ensuring balanced contributions from all tokens and modalities. This improve- ment directly correlates with superior performance on multi- modal tasks, particularly those requiring fine-graines under- standing, such as TumEmo and OCRBench, demonstrating MODA's ability to avoid attention collapse and capture mul- timodal interactions effectively.
Human: few the Godbetee er lonks like?
Cambrian-- 1: The Godhtter, as depiiedd the image,
po portroud by an actor in classc black tueodoo which s traditional and leanic look the sharwher
The Sodfather or dresses in daskkc black tuoodoo which is enb emank of role a powerful and respectedd flgg in the flnm His pepressionn is one of concern, the littns to cntnny's delemns, Indicating his deep invobernent to Johnn''s life and the weight of resporuiiiiisss
MODA:
Human: What
Godfuthe'ss influence people
circle?
Cambrian-le The scace from The Grebbeee illustmtee the profound impart the chancctee fin Cortenne, has on those around ham it stouse how his influence ectends beyond his inmeedien ste famil affecting even those who are not di dineclly ne sted to him.
MODA; spene reveals Gostfiher's infuuenc over people his cincle as digiificrr wheliininng
force. Gostither's presence room, along with demeanor, further emphasiees control hight ihning
shuaton
dymamins play
(a) Understanding Godfather
Human: How night the Godtathe oct to proseed with the next conversation? we a fow possiblc options.
Cambrian-31 the portraped the man in the foreground might proceed with the next conversation by wther continuing to innertyyl to the
image, which suggett scene from Godlotherr draractrr
actter who
be sceaking by interfecting with question
MADA: The Eodfbrhee, upen wiiteeding inbrrey' didnwss right recpand with combinution of empathy and strattggcc thinking. Ite onuld offer a comforting word, such as. "I's a to tabe in, but wee I digue out trgethec" Abbemeihelyy he might staft the conversation to more practical matter, bire discundin the lngirtics of the book'ss i adaptation on or the potential benwlits of the project. Another posubiiity is that he might offer advica on
Visualization To highlight the advantages of Modular Duplex Attention (MODA), we design visualization exper- iment focusing on output answers, showcasing its capacity to generate fine-grained and accurate responses. The output answers (Figure 5) further demonstrate MODA's superior capacity for fine-grained understanding. By accurately cap- turing the humorous twist on the classic poem, MODA has demonstrated its unparalleled ability to reason about fine-grained multimodal details. These results underscore MODA's transformative potential in advancing the state-of- the-art in multimodal understanding, where it seamlessly integrates information across modalities to achieve excep- tional precision in complex cognitive and emotional tasks.
how to handle the situation, such as seeking legal counsel or connutting with other members of the familyy The Gotthheer' responne reupoooe
would Biely the meosured and thoughtful, refleting his rale as weter.
(b) Planning for Godfather
Figure 6. MODA-enabled apps in The Godjather. (a) With a deep and fine-grained understanding of conversation, MODA captures both the emotional and cognitive states of the character. (b) MODA further simulates the Godfather's 's strategic thinking, plan- ning the next steps while considering the character's 's traits.
8

---

MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion
5. Discussion
Impact Statement
MODA-based MLLM can boost various downstream direc- tions. Here, we envision two potential uses.
This paper introduces a novel multimodal attention mech- anism designed enhance Multimodal LLMs for fine- grained content understanding. However, as with most MLLMs, the quality of MODA's output is influenced by the fine-tuning data and the quality of the base models, which may result in the generation of low-quality or hallucinated content. Such outputs could potentially be harmful, and users are advised to interpret the results with caution, adher- ing to licensing restrictions, with commercial use explicitly prohibited. All the personal information is anonymized or obfuscated to ensure confidentiality.
Leveraging the fine-grained understanding of multimodal content, MODA demonstrates enhanced comprehension capabilities that facilitate human-centric interpretation. Integrated with the strong sensory grounding capacity, MODA can effec- tively process and interpret high-level human cognition and emotion. Notably, we present the first demonstration of a Multimodal Large Language Model (MLLM) that captures emotions expressed by individuals, going beyond simple classification into discrete categories. As illustrated in Fig.6 (a), MODA successfully captures the micro-exy exprrssspessssnss of the Godfather, reflecting the character's personality and the cultural context embedded within the scene.
Godfather-centric Understanding.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. In arXiv, 2023.
Conversation with Godfather. Further, the comprehen- sive understanding enabled by MODA enhances its capacity to plan conversations. We integrate MODA into a conversa- tion system between the human and the agent in three stages: description, analysis, and planning. As shown in Fig.6 (b), MODA takes the historical conversation context (e.g.. video keyframe, transcript, and audio) as input and generates the desired target (e.g., behavior, characteristics, and emotion). This allows the model to simulate a conversation flow, as in the Godfather scenario, where the agent responds with strategic plan based on the established emotional context.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision- model with versatile abilities. In arXiv, 2023.
Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer. In arXiv, 2020.
6. Conclusion
Binz, M. and Schulz, E. Using cognitive psychology to understand gpt-3. PNAS, 120(6):2218523120, 2023.
This work introduces the MOdular Duplex Attention (MODA) to tackle attention deficit disorder in multimodal large language models, characterized by inconsistent cross- modal attention and layer decay. MODA enhances multi- modal perception, cognition, and emotion understanding by modularly processing diverse data streams, outperform- ing existing MLLMs across 21 benchmark datasets. This advancement not only improves modality alignment but also supports deeper cognitive and emotional insights, with source code and demo available for further exploration.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In NeurIPS, 2020.
Chen, J.. Wang, X., Xu, R., Yuan, S., Zhang, Y., Shi, W., Xie, J., Li, S., Yang, R., Zhu, T., et al. From persona to personalization: A survey on role-playing language agents. TMLR, 2024.
Chiang, W.-L., Li, Z., Lin, 2., Sheng, Y., Wu, 2., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. https: /ttps://viiuut Imsys. org, 2023.
Acknowledgement
This work was supported by the National Natural Science Foundation of China (No. 623B2056), the Natural Science Foundation of Tianjin, China (N0.24I2Z),000400 the Fundamental Research Funds for the Central Universities, the Supercomputing Center of Nankai University (NKSC). We sincerely thank the reviewer team (cYUZ, XinM, Mf8z, and Ack2) for their invaluable feedback to improve our manuscript.
Child, R., Gray, S., Radford, A., and Sutskever, Gener- ating long sequences with sparse transformers. In arXiv, 2019.
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin,

---

MODA; MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion
A., Kaiser, L., et al. Rethinking attention with performers. In ICLR, 2021.
Li, Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
Dai, Y., Hu, H., Wang, L., Jin, S., Chen, X., and Lu, Z. Mmrole: A comprehensive framework for developing and evaluating multimodal role-playing agents. In ICLR, 2025.
Lian, Z., Liu, B., and Tao, J. Smin: Semi-supervised multi- modal interaction network for conversational emotion recognition. TAC, 2022.
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., and Narasimhan, Toxicity in chatgpt: Analyzing persona- cassona-ssigned language models. In EMNLP, 2023.
Liu, H., Wang, W., and Li, H. Towards multi-modal sar- casm detection via hierarchical congruity modeling with knowledge enhancement. In EMNLP. 2022.
Dong, Y., Cordonnier, J.-B., and Loukas, A. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In ICML, 2021.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tuning. NeurIPS, 2023.
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L.,, Jin, I.., and Bai, X. Ocrbench: the hidden mystery of ocr in large multimodal models. SCIS, 67(12):220102, 2024.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Liu, Y.. Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W. Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, 2025.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. In arXiv, 2024.
Loshehilov, and Hutter, Decoupled weight decay regu- larization, In ICLR, 2019.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al. Mme: A comprehen- sive evaluation benchmark for multimodal large language models. In arXiv, 2023.
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, 2022.
Han, D., Ye, T., Han, Y., Xia, Z., Pan, S., Wan, P., Song, S., and Huang, G. Agent attention: On the integration of softmax and linear attention. In ECCV, 2024.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Math- vista: Evaluating mathematical reasoning of foundation models visual contexts. In ICLR, 2024.
Huang, S., Xu, B., Li, C., Ye, J., and Lin, X. A senti- mental prompt framework with visual text encoder for multimodal sentiment analysis. In ICMR, 2024.
Mai, S., Zeng, Y., Zheng, S., and Hu, H. Hybrid contrastive learning of tri-modal representation for multimodal senti- ment analysis. TAC, 2022.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACI, 2022.
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram worth dozen images. In ECCV, 2016.
Mathew, M., Karatzas, D., and Jawahar, C. Doevqa: A dataset for vqa document images. In WACV, 2021.
Kitaev, N., Kaiser, t.., and Levskaya, A. Reformer: The efficient transformer. In ICLR, 2020.
Niu, T., Zhu, S., Pang, L., and El-Saddik, A. Sentiment analysis on multi-view social data. In MMM, 2016.
Kong, A., Zhao, S., Chen, H., Li, Q., Qin, Y., Sun, R., Zhou, X.. Wang, E., and Dong, X. Better zero-shot reasoning with role-play prompting. In NAACL, 2024.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. ICCV, 2023.
Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed-bench: Benchmarking multimodal large language models. In CVPR, 2024.
Pessoa, L. The entangled brain: How perception, cognition, and emotion are woven together. MIT Press, 2022.
10

---

MODA; MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Wu, X., Ajorlou, A., Wang, Y., Jegelka, S., and Jadbabaie,
A. On the role of attention masks and layernorm in transformers. In NeurIPS, 2024.
XAI. grok. http::://x.aa//blg/ggooo///
2024. Yang. J., She, D., Lai, Y.-K., Rosin, P. L.,, and Yang, M.-H. Weakly supervised coupled networks for visual sentiment analysis. In CVPR, 2018.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Ryu, J., Han, D., and Lim, J. Gramian attention heads are strong yet efficient vision learners. In ICCV, 2023.
Yang, Q. Ye, M., and Du, B. Emollm: Multimodal emo- tional understanding meets large language models. In arXiv, 2024.
Salemi, A., Mysore, S., Bendersky, M., and Zamani, H. LaMP: When large language models meet personalization. In ACL, 2024.
Yang, X., Feng. S., Wang. D., and Zhang, Y. Image- mul- timodal emotion classification via multi-view attentional network. TMM, 23:4014-4026, 2021.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In CVPR, 2019.
Yang, X., Feng, S., Wang, D., Hong, P., and Poria, S. Multi- ple contrastive learning for multimodal sentiment analy- sis. In ICASSP, 2023.
Teknium, R., Quesnelle, J., and Guang, C. Hermes 3 techni- cal report. In arXiv, 2024.
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., lyer, A., Pan, X., et al. Cambrian-1: A fully open, vision-centric exploration of multimodal 1lms. In NeurlPS, 2024a.
Yin, Q. He, X., Zhuang, X., Zhao, Y., Yao, J., Shen, X., and Zhang, Q. Stablemask: refining causal masking in decoder-only transformer. In ICML, 2024.
Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Wang, G., Li, H., Zhu, J., Chen, J., et al. Yi: Open foundation models by 01.ai. arXiv, 2024.
Tong, S., Liu, Z., Zhai, Y., Ma. Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, 2024b.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. Mmmu: A massive multi-disciplin multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. In arXiv, 2023.
Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Al- berti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q.. Yang, L.,, et al. Big bird: Transformers for longer se- quences. In NeurIPS, 2020.
Wang, C., Luo, W., Dong, S., Xuan, X., Li, Z., Ma, L., and Gao, S. Milm-tool: A multimodal large language model for tool agent learning. In WACV, 2025.
Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Lin- former: Self-attention with linear complexity. In arXiv, 2020.
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., and Weston, J. Personalizing dialogue agents: have a dog, do you have pets too? ACI, 2018
Wang, X., Xiao, Y., Huang, J.-t., Yuan, S., Xu, R., Guo, H., Tu, Q., Fei, Y., Leng, Z., Wang, W., Chen, J., Li, C., and
Zhang, Z. and Yang, J. Temporal sentiment localization: Listen and look in untrimmed videos. ACM MM, 2022.
Xiao, Y. InCharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In ACL, 2024a.
Zhang, Z., Wang, L., and Yang, J. Weakly supervised video emotion detection and prediction via cross-modal temporal erasing network. In CVPR, 2023.
Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., Zhang, F., Wang. Y., Li, Z., Yu, Q., al. Emu3: Next- token prediction is all you need. In arXiv, 2024b.
Zhang, Z., Zhao, P., Park, E., and Yang. J. Mart: Masked affective representation learning via masked temporal distribution distillation. In CVPR, 2024.
Wang, Z. M., Peng, Z., Que, H., Liu, J., Zhou, W., Wu, Y., Guo, H., Gan, R., Ni, Z., Yang, J., Zhang, M., Zhang, Z., Ouyang, W., Xu, K., Huang. S. W., Fu, J., and Peng. J. Rolellm: Benchmarking, eliciting, and enhancing role- playing abilities of large language models. In ACI. 2024c.
Zhao, S., Jia, G., Yang, J., Ding, G., and Keutzer, K. Emo- tion recognition from multiple modalities: Fundamentals and methodologies. SPM, 38(6):59-73, 2021.

---

MODA; MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024.
12


Reviews:
[{"review_id": "tlTWwn61fi", "reviewer": "Reviewer_Mf8z", "summary": "The paper identifies a critical limitation in MLLMs, where inconsistent attention across layers leads to errors in fine-grained emotion understanding (\"deficit disorder attention problem\"). To address this, the authors propose MOdular Duplex Attention (MODA), which separates attention into self-modal and cross-modal components, each governed by a dedicated modulated attention mask. Extensive evaluations on various benchmark datasets demonstrate MODA’s effectiveness.\n\n## update after rebuttal\n\nThank you to the authors for their responses. Most of my concerns have been addressed, and I am happy to raise my rate.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer Mf8z**\n\nThank you for your insightful comments and questions. For your reference, we summarized the main results and included the attached file in our response to Reviewer XinM.\n\n**A1. Claims on DDA**\n\n(1) Actually, DDA is supported by evidence from two aspects: rationale and observation.\n\n- **Rationale**: Based on the graph theory, the interaction of multimodal tokens are discorrupted from the input level, where most important visual tokens are throw away. Further, the layer-by-layer propagation introduces the coefficient of accumulated ignorance. The illustration is included in Fig.1 of the attached file.\n- **Observation**: First, attention scores exhibit a significant bias toward the textual modality, as shown in Fig.2a, where visual features are underrepresented. Second, Fig.2b&2c highlight a clear layer-wise attention decay, with attention inconsistencies becoming more pronounced in deeper layers. Finally, qualitative analyses in Fig.4&6 reveal that baseline models fail to capture subtle multimodal cues, resulting in incorrect or overly generic responses.\n\n(2) We thank the reviewer for the valuable comments and will **clarify this** in the revised manuscript accordingly.\n\n**A2. Discussion on DDA** \n\n(1)  **Similar visual tokens lead to high attention scores.** The authors would like to clarify that the statement of the reviewer 'adjacent tokens often exhibit higher similarity due to the spatial structure of images, naturally resulting in smaller self-attention scores.' is *wrong*. In fact, the similar visual tokens have a high attention score [A], according to the definition of attention $A = Softmax(QK/\\sqrt{d})\\propto QK$. $QK$ represents the similarity between tokens and is directly proportional to the attention score. Therefore, DDA suffers from imbalanced attention scores. The visual part is assigned a low attention score, leading to the neglect of important visual details.\n\n(2) **DDA brings negative impact in neglecting fine-grained details.**  We use a grounded MLLM (SA2VA) prompt by the ground truth answer to segment the key regions in visual content. Experimental results on 9 vision-centric perception, cognition, and emotion benchmarks revealed that the SOTA MLLM throws away 17% of the crucial visual tokens. For visualization results, please refer to Fig.2 of the attached file.\n\n(3) **DDA limits real-world applications.**  The imbalanced multimodal attention introduces bias in the flow of tokens, which can lead to inadequate token fusion and yield hallucination results [B]. As shown in Fig.1a &1b of our original manuscript, the model failed to recognize the waiter in the image background, posing critical challenges for practical scenarios, including OCR and conversational agent tasks.\n\n[A] Attention is all you need, NIPS, 2017.\n\n[B] Mitigating modality prior-induced hallucinations in multimodal large language models via deciphering attention causality, ICLR, 2025\n\n**A3. Comparison on fine-grained understanding:** We would like to clarify that our manuscript already *INCLUDES BOTH quantitative and qualitative direct comparisons with other leading MLLMs* in fine-grained content understanding, as presented in Tab.2, 3, 4, and Fig.4, 7.\n\n(1) **Tab.2**: We compare MODA with 6 close-sourced and 6 open-sourced MLLMs (GPT4V, Gemini 1.5 Pro, Grok-1.5, MM-1-30B, Mini-Gemini-HD, LLaVA-NeXT, Cambrian-1) on vision-centric and OCR tasks, which rely on fine-grained cue understanding.\n\n(2) **Tab.3&4**: We compare MODA with 4 close-sourced and 7 open-sourced MLLMs (GPT4o, Gemini 1.5 Pro, Claude 3 Opus, Owen-VL-Max, Mini-Gemini-HD, LLaVA-NeXT, Cambrian-1, MMRole)  on cognition and emotion tasks requiring contextual understanding.\n\n(3) **Fig.4&7**: we provide qualitative comparisons with SOTA Cambrian-1.\n\n(4) **Fig.3 of the attached file:** Additionally, we conduct further quantitative comparisons with Cambrian-1 on human-centric understanding and planning tasks.\n\n**A4. Framework and pipeline in Fig.3 and Sec. 3:** Thanks for the reminder and we include the modified pipeline and caption as Fig.4 of the attached file. Besides, we will add an overview section in Sec 3.3.\n\n**A5. Illustration of Fig.2:** The x-axis and y-axis represent the attention values and the corresponding probability. The x-axis ranges from 1e-4 to 1e-1 and is a logarithmic scale.\n\n**A6. Attention score:** Following [C], we compute token-level attention scores on all the testing set and perform average. The pseudocode is included as Alg.1 in the attached file.\n\n[C] Efficient streaming language models with attention sinks, NeurIPS, 2024\n\n**A7&A8. Writing:** (1) [M] replaces modular masked attention with learnable tokens to control attention distribution. (2) The term refers to the residual features before Gram matrix mapping. (3) For the token sequence $X^m \\in \\mathbb{R}^{N_m \\times D}$ of modality $m$, the mask controls its visibility with the entire multimodal sequence $X \\in \\mathbb{R}^{(N_m+N_\\bar{m}) \\times D}$.", "claims_and_evidence": "- The authors introduce the deficit disorder attention problem, arguing that it leads to neglect fine-grained details. However, this claim seems to be an overgeneralization. \n- Also, the visualizations of attention scores across different modalities (Figures 1(c), 2(c), and 5) may not provide convincing evidence of an inherent problem. Different modalities inherently contain varying levels of similarity between tokens. For instance, in visual inputs, adjacent tokens often exhibit higher similarity due to the spatial structure of images, naturally resulting in smaller self-attention scores. This phenomenon may not necessarily indicate a limitation but rather a characteristic of multimodal attention behavior.", "methods_and_evaluation_criteria": "The authors claim that their proposed approach enhances fine-grained content understanding and support this with qualitative results (e.g. Figure 6). However, they do not provide a direct comparison with other MLLMs.", "theoretical_claims": "The primary concern from this reviewer is in Section 3, particularly regarding the clarity and alignment of the model description.\n- The overall approach and flow of the proposed model architecture is unclear. While Figure 3 is intended to illustrate the framework, it does not clearly align with the explanation in Section 3, making it difficult to understand how the components interact.\n- In Figure 2(a), what do the x-axis and y-axis represent? How does the visualization support their claim about attention inconsistency across layers (Lines 133–137)?", "experimental_designs_or_analyses": "Some details are missing from the experimental results section. \n- How do the authors compute the attention scores? Are these derived from a single sample or aggregated from multiple samples? Additionally, which dataset(s) were used for this analysis?\n- What does [M] represent in Table 1? \n- In Line 184, the term \"the original ones\" is vague.\n- While the paper states that the mask is split into $\\mathbf{M}^m$ and $\\mathbf{M}^\\bar{m}$, it is unclear what the mask itself represents.", "relation_to_broader_scientific_literature": "It may be benefitial for multimodal perception, cognition, and emotion understanding.", "essential_references_not_discussed": "No major references appear to be missing.", "other_strengths_and_weaknesses": "**Weaknesses:**\n- The paper requires proofreading as there are several missing details (as noted in my comments above), and some sentences are incomplete or unclear.\n```(Line 204) To alleviate the collapsed attention matrix and prevent it from under-smoothed tokens. We propose a modular attention mask that chooses to store unnecessary attention values on these pseudo-attention scores.```", "other_comments_or_suggestions": "It might be useful to provide a more explanation of the Gram matrix, as MODA relies on it.", "questions_for_authors": "Please check my comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oqa6LDJ77c", "reviewer": "Reviewer_XinM", "summary": "This paper proposes a novel attention mechanism called Modular Duplex Attention (MODA) to tackle the attention inconsistency problem in Multimodal Large Language Models (MLLMs). MODA showcases outstanding performance in multimodal perception, cognition, and emotion understanding tasks. Specifically, the 34B version of MODA outperforms GPT-4V comprehensively.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **General Response**\n\nWe sincerely appreciate all the Reviewers and the Area Chair for their time and effort in reviewing our paper. Following the valuable suggestions and insights provided in the reviews, we summarize **the additional results and evidence** included in the rebuttal based on the reviewers' suggestions:\n\n- We provided the theoretical explanation of the deficit disorder attention problem and illustrated the diagram **[A1&A2 for Ack2 and Fig.1 of the attached file]**.\n- We conducted experiments on 9 benchmark datasets to discuss the negative impact of the critical issue of MLLM, i.e., the deficit disorder attention problem **[A1 for Mf8z and Fig.2 of the attached file]**.\n- We conducted more qualitative comparisons with SOTA MLLMs **[Fig.3 of the attached file]**.\n- We provided the pseudo-code for attention score computation and MODA for clarity **[A6 for Mf8z, A3-4 for Ack2,  and Alg. 1 & Alg. 2 of the attached file]**.\n- We provided the computational complexity analysis of the proposed module and conducted comparisons among existing MLLMs and alternative attention mechanisms **[A1 & A2 & A3 for cYUZ]**.\n- We provided the detailed experimental setup and illustrated more information, including training time and hardware **[A1 for XinM]**.\n- We corrected the mentioned typos and carefully checked our manuscript again. Moreover, we also modified the figures and added the discussions as suggested **[Fig.4 of the attached file]**.\n\nAdditionally, we follow the guideline of ICML 2025 and attach the figures as well as the pseudo-code in the [`anonymous link`](https://anonymous.4open.science/r/icml25_rebuttal-F06A/README.md): https://anonymous.4open.science/r/icml25_rebuttal-F06A/README.md\n\n---\n\n## **Response to Reviewer XinM**\n\nThank you for your insightful comments and questions.\n\n**A1. Training details, including backbone, data, hyperparameter, and hardware&time**\n\n(1) **Same setting as Cambrian-1:** To ensure a fair comparison, we adopt the same experiment setup, including the backbone and data, as Cambrian-1. Specifically, our training begins with a one-stage SFT following pretraining, which utilizes the pre-trained ViT and adaptor. During the training, we unfreeze the LLM backbone to fine-tune the attention part in the LLM and fully exploit the power of MODA.\n\n(2) **Details.** We attach a configuration table below, which outlines the backbone, data, hyperparameter, and details during the training process. We will include details about training in the revised manuscript.\n\n|               | Backbone        |                          | Data        | Param. |        |        | Details      |          |\n| ------------- | --------------- | ------------------------ | ----------- | ------ | ------ | ------ | ------------ | -------- |\n| **Model**     | **LLM**         | **Vision**               | -           | **lr** | **wd** | **bs** | **Hardware** | **Time** |\n| MODA-8B       | LLaMA3-Ins-8B   | OpenAI CLIP ViT-L/14@336 | Cambrian-7M | 2e-5   | 0      | 1024   | 2x 8 A800    | 6 days   |\n| Cambrian-1-8B | LLaMA3-Ins-8B   | 4 Vision Encoders*       | Cambrian-7M | 2e-5   | 0      | 512    | 128 TPUv4    | -        |\n| MODA-34B      | Hermes-2-Yi-34B | OpenAI CLIP ViT-L/14@336 | Cambrian-7M | 2e-5   | 0      | 2048   | 4x 8 A800    | 14 days  |\n| Cambrian-34B  | Hermes-2-Yi-34B | 4 Vision Encoders*       | Cambrian-7M | 2e-5   | 0      | 1024   | 512 TPUv4    | -        |\n\n- Backbone: Cambrian-1 uses 4 vision encoders* including OpenAI CLIP ViT-L/14@336, SigLIP ViT-SO400M/14@384, DINOv2 ViT-L/14@518, and Open-CLIP ConvNeXt-XXL@1024. In contrast, we only use OpenAI CLIP ViT-L/14@336 as other popular MLLMs.\n- Data: We follow the same setting as Cambrian and train our MODA by Cambrian-7M.\n- Hyperparameter: We follow the common setting and train the model by using different lr for LLM and vision encoder.\n- Hardware&time: For the 8B model, we use 2x A800 nodes to train for 6 days. For the 34B model, we use 4x A800 nodes to train for 14 days.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make eminent sense for the problem and application at hand.", "theoretical_claims": "Yes, I've checked the theoretical claims' proofs. For MODA, the duplex attention alignment's math based on gram matrix vectors is logical. The modular attention mask's equations and strategies are well - defined. The use of the normed gram matrix in both components is sound. Overall, the proofs are correct and support the MODA mechanism.", "experimental_designs_or_analyses": "Yes, I have checked the soundness and validity of the experimental designs and analyses in the paper. In the ablation study, different components of MODA, such as duplex attention alignment and modular attention mask, were systematically removed or modified. This design is valid as it helps to isolate the impact of each component on the overall performance, answering important research questions about their individual contributions.", "relation_to_broader_scientific_literature": "Prior studies have highlighted the importance of attention mechanisms in multimodal learning, but faced issues with inconsistent cross-modal attention. MODA builds on these by specifically addressing this problem.", "essential_references_not_discussed": "none", "other_strengths_and_weaknesses": "Weakness: \n1. As mentioned in **Supplementary Material**, I wanted to know if the model only trained on a SFT stage or followed a two stage training process like common models. Also, I'd like to know more details like learnable parameters during the training process.", "other_comments_or_suggestions": "none", "questions_for_authors": "I was wondering why MODA has such a significant improvement in emotion tasks. I'm willing to raise my score if deeper explanations and training details can be provided.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dXCaS9cS6E", "reviewer": "Reviewer_ACk2", "summary": "This paper proposes a MOdular Duplex Attention (MODA) for multimodal perception, cognition and emotion understanding. The proposed method is evaluated and the paper is well organized.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer Ack2**\n\nThank you for your insightful comments and questions. For your reference, we summarized the main results and included the attached file in our response to Reviewer XinM.\n\n**A1&A2. Discussion on DDA**\n\n(1) **Actually, DDA can be interpreted from the perspective of a multimodal token graph.** The core of DDA, the attention mechanism, builds the linkage across multimodal tokens by computing the similarity of each pair of tokens [A,B]. The attention output is yielded by a weighted sum among tokens. Therefore, the computation can be seen as a densely connected direct graph through layer-by-layer attention, and the DDA problem can be reformulated as a mislinked graph problem, where the multimodal token flows to the textual part and misses the visual token from the input layer. DDA, therefore, decreases the interaction between visual and textual modality, and throws away 17% of important visual cues and is lost in the last embedding layer of MLLM. For the detailed diagram of multimodal token flow, please refer to Fig.1 in the attached file.\n\n(2) **Formula definition of DDA.** Given the visual tokens $x_v^l$ and text tokens $x_t^l$ in the block $l$, the multimodal attention builds the link from two parts (*i.e.*, self-modal $x_t^l \\rightarrow x_t^{l+1}, x_v^l \\rightarrow x_v^{l+1}$ and cross-modal $x_t^l \\rightarrow x_v^{l+1}, x_v^l \\rightarrow x_t^{l+1}$ ),  where the links are commonly implemented by the pair-wise token similarity and weighted sum. However, the modality gap between tokens decrease the magnitude of links, as we observed, the link value of $x_v^l \\rightarrow x_v^{l+1}$ and $x_t^l \\rightarrow x_v^{l+1}$ decays exponentially with depth ($\\alpha_{v \\rightarrow v}^l \\propto \\gamma^l,\\gamma \\neq 1$). This misalignment propagates layer-wise, causing the cumulative error in cross-modal interaction to grow as $\\mathbb{E}_L = \\sum_l \\gamma^l \\epsilon_l$, where $\\epsilon_l$ denotes the layer-specific alignment error. This phenomenon aligns with theoretical insights in [B], where pure attention mechanisms suffer from **rank collapse** – a critical factor exacerbating the attention distribution.\n\n(3) **MODA vs existing methods:** To address this issue, we propose MODA, which introduces modality-aware gating and layer-wise error compensation to mitigate DDA. Unlike existing methods, MODA dynamically adjusts cross-modal interactions through modality-specific gates and compensates for alignment errors using adaptive propagation mechanisms. Further, MODA extracts the modality-specific feature and introduces the gram matrix as basis vectors for adaptive mapping. Thanks for the suggestions and we will re-organize the literature review in the introduction and related work.\n\n[A] On the role of attention masks and layernorm in Transformers, NeurIPS, 2024\n\n[B] Attention is not all you need: Pure attention loses rank doubly exponentially with depth, ICML, 2021\n\n**A3-1. Mapping function in V&T-aligner:** The mapping functions in V-Aligner and T-Aligner are designed to address modality alignment issues and exploit four alternative designs as shown in Tab.1c of our manuscript. We explore direct replacement ($X_a$ as the original feature, $X_p$ as the mapped feature), concatenation, and element-wise addition for token fusion. The use of basis vectors enables structured modality space representation, improving alignment.\n\n**A3-2. Formulas:** $||$ in Equ.5 represents the normalization operation.\n\n**A3-3. Self-attention&cross-attention:** We modularize the influence of self- and cross-attention separately from two aspects: pull the tokens by alignment and correct the focus by masking.\n\n**A3-4. MODA for broader MLLMs:** MODA can be integrated into the existing MLLMs by simply replacing each attention layer in the Transformer block. In our paper, we implement the most commonly used pre-norm type MLLM and verify its effectiveness on LLaMA-based MLLM. As a result, MODA can be integrated into the LLaVA series and Vicuna-based, Yi-based, Wizardlm2-based MLLMs because they share the same architecture as the implemented one of MODA. For clarity, we have prepared a torch-style pseudocode in Alg. 2 of the attached file.\n\n**A4-1. Detailed experimental setup:** As suggested, we provide all the experimental settings, including the model, data, hyperparameter, and details. Due to the character limits, please refer to the Q1 of reviewer XinM.\n\n**A4-2&A4-3&A4-4. Analysis and Discussion of MODA:** We thank the reviewer for the feedback. To address this, we added a detailed analysis of the advantages and disadvantages of each model and module across perception, cognition, and emotion tasks. MODA, as most MLLMs are, suffer from the risk of generating counterfactual responses, leading to false narratives or misrepresentations.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "The authors propose a novel modular and duplex attention mechanism which could be used in multimodal LLMs.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "1. Introduction\n (1) Although the authors propose the phenomenon of deficit disorder attention, the discussion of its underlying causes and theoretical foundations is weak. The article focuses more on the description of the phenomenon and lacks strict mathematical definitions and theoretical explanations, which makes the concept appear to be more of an analogy than a well-proven theoretical basis. \n(2) only mentions the inadequacy of MLLM for advanced tasks, but does not provide a detailed compendium and comparison of current research methods and advances to address this inadequacy.\n\n2. Related work\nWhen presenting related work, it is mostly descriptive, without clearly stating the direct correlation and difference between these studies and the methods proposed in this paper, and fails to clearly demonstrate the unique advantages of MODA in dealing with the problems of inconsistent attention and layer-by-layer attenuation.\n\n3. Methods\n(1) For the specific implementation of V-Aligner and T-Aligner, only the mapping based on the basis vectors of modal space is mentioned, but the specific form of the mapping function and why the basis vectors are used are not explained in detail. There is a lack of detailed description of the fusion method of the mapped tokens, such as whether weighted summing and splicing are used, as well as the parameter estimation and optimization strategy in the fusion process.\n(2) Insufficient explanation for some formulas. For example, “||” in Equation 5.\n(3) Although the attention is divided into two parts, self-modal and cross-modal, how to coordinate these two parts in the overall architecture and how to deal with their interactions and information fusion are not described clearly enough. There is a lack of detailed discussion on how the modules in the overall process work together.\n(4) There is a lack of specific guidance and examples on how to integrate MODA into existing multimodal large-scale language models, which cannot provide effective references for actual model development and application.\n\n4. Experimental Part\n(1) The description of the experimental setup is not exhaustive enough, and key settings such as data preprocessing, hyperparameter selection, and training details (e.g., hardware environment, running time) are not fully developed.\n(2) Despite the ablation experiments, there is a lack of detailed discussion on the individual contributions and interactions of the two modules (Duplex Attention Alignment and Modular Attention Mask) to the overall performance.\n(3) In the comparison with existing SOTA models, only a large amount of performance data is given, while the lack of analysis of the advantages and disadvantages of each model on different tasks makes the discussion of the reasons for the performance improvement insufficient.\n(4) The experimental results present more data, but fail to discuss in depth the performance differences between different tasks and metrics, potential bottlenecks, and directions for future improvement, limiting a comprehensive understanding of the advantages and limitations of the method.", "other_comments_or_suggestions": "no", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fGOntZ2KF3", "reviewer": "Reviewer_cYUZ", "summary": "The paper identifies the attention deficit disorder problem in SOTA MLLMs, characterized by inconsistent cross-modal attention and layer-by-layer decay of attention activation. Then, the authors introduce a linear-based attention mechanism that simultaneously conducts inner-modal refinement and inter-modal interaction. Experimental results show an improvement on fine-grianed detail understanding. The evaluation setting follows Cambrian-1, MMRole, and some emotion benchmarks.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer Ack2**\n\nThank you for your insightful comments and questions. For your reference, we summarized the main results and included the attached file in our response to Reviewer XinM.\n\n**A1. Analysis for linear complexity of duplex attention alignment** \n\n**(1) Complexity Analysis:** As suggested, we provide the complexity analysis of the proposed duplex attention alignment. Given a multimodal token sequence $X\\in\\mathbb{R}^{N\\times D}$, the computation cost comes from three operations, including Gram Matrix and alignment. In the Gram matrix, the computation cost is $O(D^2N)$. In the alignment, the computation cost is $O(DN)$, which in total is $O(D^2N+DN)=O(D^2N)$. Therefore, the proposed duplex attention alignment has a linear complexity in terms of token length, thus yielding a length extrapolation ability towards long multimodal context.\n\n**(2) Complexity comparison with other attention mechanism:** In contrast, the baseline attention owns a nature of $O(DN^2)$ due to the similarity computation between each pair of tokens. Therefore, they have a higher computation complexity of $O(DN^2)$, where in most cases $N>>D$, especially for multimodal tokens, the number can be up to 128000 (e.g., max model len of  Qwen2.5-VL).\n\n**A2. Ablation study on Attention mechanism**\n\n(1) As suggested, we further conduct a comparison between our MODA and other attention mechanisms, including baseline attention and SOTA attention mechanism. As shown below, MODA shows better performance among leading attention mechanisms due to its balance on multimodal tokens, while sparse and baseline attention may wrongly focus on the textual part.\n\n| Model                      |  G   |  K   |  O   |  V   |\n| -------------------------- | :--: | :--: | :--: | :--: |\n| MODA                       | 69.3 | 48.3 | 67.0 | 54.3 |\n| DeepSpeed Sparse Attention | 67.8 | 47.6 | 63.3 | 48.1 |\n| Multi-head Attention       | 63.6 | 44.0 | 60.8 | 38.0 |\n\n(2) Besides, as shown in Tab.1 of the manuscript, we provided an ablation study on alternative designs from three aspects: how to align attention tokens (Tab.1b), how to fuse attention tokens (Tab.1c), and how to mask attention  (Tab.1d). \n\n**A3. Computation cost:** As suggested, we compare the computation cost of MODA and other attention mechanisms in terms of FLOPs, memory, and latency as below. We can observe that MODA achieves better performance on fine-grained understanding tasks (e.g., four vision-centric benchmarks) with a few increased computation costs.\n\n| Model      | FLOPs  | MACs  | Latency | Vision-Centric |\n| ---------- | :----: | :---: | :-----: | :------------: |\n| MODA       | 134.9T | 76.7T |  2.04s  |      66.0      |\n| LLaVA-NeXT | 123.3T | 61.6T |  1.87s  |      56.6      |", "claims_and_evidence": "-The claimed linear complexity is not supported by any details. The paper does not provide detailed analysis of MODA's computational cost, especially compared to simpler baselines.\n\n-While the modular masked attention and duplex alignment are well-motivated, the paper does not explore alternative designs or compare MODA to other attention mechanisms (e.g., sparse attention or low-rank attention).", "methods_and_evaluation_criteria": "-Yes, the evaluabtion setting follows The evaluation setting follows Cambrian-1, MMRole, and some emotion benchmarks.", "theoretical_claims": "-The computation method of attention distribution is not mentioned with details", "experimental_designs_or_analyses": "-The paper does not provide detailed analysis of MODA's computational cost (e.g., FLOPs, memory usage, or inference time) compared to simpler baselines.", "relation_to_broader_scientific_literature": "-The paper demonstrates MODA's effectiveness across a wide range of tasks, including perception, cognition, and emotion understanding, setting new benchmarks in multimodal understanding.\n\n-The paper highlights the potential of MODA to advance multimodal understanding in real-world applications, such as human-computer interaction, robotics, and healthcare.\n\n-Identify the Attention Deficit Disorder Problem for multimodal large language model area.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "see above", "other_comments_or_suggestions": "N/A", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission introduces MODA, a novel modular duplex attention mechanism designed to address the deficit disorder attention problem in multimodal large language models (MLLMs). Reviewers unanimously recognize the paper’s significant contributions, highlighting its clear novelty, comprehensive theoretical grounding, and extensive experimental validation.

While initial concerns regarding clarity and computational cost were raised, these were adequately addressed in the authors' rebuttal, resulting in reviewers increasing their ratings. Given the paper’s strong innovation, technical depth, and potential impact on multimodal AI research, the consensus among reviewers supports a clear acceptance. The AC agrees with the reviewers to accept the submission.
Decision: Accept (spotlight poster)