The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to  Visual  Information Loss
Authors: Bozhou Li
Xinda Xue
Sihan Yang
Yang Shi
Xinlong Chen
Yushuo Guan
Yuanxing Zhang
Wentao Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: MultiModal Large Language Model;Pre-Normlization
Primary area: foundation or frontier models, including LLMs

Abstract: Multimodal Large Language Models (MLLMs), which couple pre-trained vision encoders and language models, have shown remarkable capabilities. However, their reliance on the ubiquitous Pre-Norm architecture introduces a subtle yet critical flaw: a severe norm disparity between the high-norm visual tokens and the low-norm text tokens. In this work, we present a formal theoretical analysis demonstrating that this imbalance is not a static issue. Instead, it induces an "asymmetric update dynamic", where high-norm visual tokens exhibit a ''representational inertia,'' causing them to transform semantically much slower than their textual counterparts. This fundamentally impairs effective cross-modal feature fusion. Our empirical validation across a range of mainstream MLLMs confirms that this theoretical dynamic---the persistence of norm disparity and the resulting asymmetric update rates---is a prevalent phenomenon. Based on this insight, we propose a remarkably simple yet effective solution: inserting a single, carefully initialized LayerNorm layer after the visual projector to enforce norm alignment. Experiments conducted on the LLaVA-1.5 architecture show that this intervention yields significant performance gains not only on a wide suite of multimodal benchmarks but also, notably, on text-only evaluations such as MMLU, suggesting that resolving the architectural imbalance leads to a more holistically capable model.

Published as a conference paper at ICLR 2026
THE UNSEEN BIAS: How NORM DISCREPANCY IN PRE-NORM MLLMS LEADS TO VISUAL INFORMA- TION LOss
Bozhou Li Xinda Xue', Sihan Yang Yang Shi', Xinlong Chen', Yushuo Guan Yuanxing Zhang Wentao Zhang Peking University
#Zhongguancun Academy
*Beijing Key Laboratory of Data Intelligence and Security (Peking University) School of Artificial Intelligence, University of Chinese Academy of Sciences 'Xi' an Jiaotong University
'llnng Team, Kuaishou Technology
ABSTRACT
Multimodal Large Language Models (MLLMs), which couple pre-trained vision encoders and language models, have shown remarkable capabilities. However, their reliance on the ubiquitous Pre-Norm architecture introduces a subtle yet criti- cal flaw: severe norm disparity between the high-norm visual tokens and the low- norm text tokens. In this work, we present formal theoretical analysis demon- strating that this imbalance is not static issue. Instead, induces an "asymmetric update dynamic," where high-norm visual tokens exhibit "representational iner- tia," causing them to transform semantically much slower than their textual coun- terparts. This fundamentally impairs effective cross- moos- feature fusion. Our empirical validation across a range of mainstream MLLMs confirms that this the- oretical dynamic- the persistence of norm disparity and the resulting asymmetric update rates-is a prevalent phenomenon. Based on this insight, we propose a remarkably simple yet effective solution: inserting a single, carefully initialized LayerNorm layer after the visual projector to enforce norm alignment. Experi- ments conducted on the VL. architecture show that this intervention yields sigmificant performance gains not only on a wide suite of multimodal benchmarks but also, notably, on text-only evaluations such as MMLUU suggesting that resolv- ing the architectural imbalance leads to a more holistically capable model.
INTRODUCTION
In recent years, Multimodal Large Language Models (MLLMs) have achieved significant progress, demonstrating robust performance across a wide range of cross-modal tasks (Comanici et al., 2025; Hurst et al., 2024; Wu et al., 2024; Bai al., 2025). A prevailing architectural paradigm involves augmenting a pre-trained Large Language Model (LLM) with visual capabilities by coupling it with pre-trained Vision Encoder (VE). The VE, typically Vision Transformer (ViT) (Dosovitskiy et al., 2020), first partitions an image into sequence of patches and encodes them into series of feature vectors, or "'isuull tokens." " To bridge the modality gap, lightweight adapter module is then introduced. This module's 's core function is to act as a translator, projecting these visual tokens into the LLM's word embedding space, thereby making visual information comprehensible to model originally designed for text (Zhang et al., 2024).
Despite their powerful general-purpost capabilities, emerging research has revealed inherent limi- tations in MLLMs. For instance, many models struggle with the perception of fine-grained visual details (Rabmanzadehgervi et al., 2024). Furthermore, within their self-attention mechanisms-the core component for weighing the importance of different inputs-vists tokens often receive less
"Email: libozhou @wt.oouuuuuu.u..cc. 1 "Correspnndigg aunhor:Wenaao Zhang

---

Published as a conference paper at ICLR 2026
focus than their textual counterparts (Chen al., 2024a). To address these challenges, identify more fundamental problem rooted in the now-ubiquitous Pre-Norm Xiong et al. (2020) architectural design. In this paradigm, normalization is applied before the main computational block (F), with the residual update defined as:
h h(+1)) = h F(Norm(h!)))
This architecture is widely adopted because it is easier to train. By leaving the residual path h()) unaltered, it creates an identity-like connection that ensures smooth gradient flow, preventing van- ishing gradients in deep networks. However, this design has critical side effect: since the output of the residual sum is never re-normalized, the variance- and consequently, the L2 norm of the hidden states tends to accumulate and grow with network depth (Kim et al., 2025). As is shown in Figure 1b, creates a particularly acute imbalance MLLMs where high-norm visual tokens and lower-norm text tokens processed together within a shared Pre-Norm LLM backbone- as the visual tokens themselves are generated by a deep, Pre-Norm VIT.
Our formal theoretical analysis reveals critical dynamic: a fundamental asymmetry in the evolu- tionary pace of visual and textual representations through the LLM's layers. We demonstrate that for high-norm visual tokens, the Pre-Norm update mechanism induces high "representational inertia", causing them to undergo a much slower semantic transformation. In contrast, lower-norm textual to- kens adapt their representations more readily, leading mismatched rate of convergence towards a unified multimodal space. Notably, this dynamic divergence arises not from an intrinsic property of visual versus textual information, but from an architectural artifact: the interplay between the Pre-Norm design and the prevailing MLLM paradigm.
Bridging theory and practice, we first confirmed that these norm disparities and asymmetric update rates are prevalent across mainstream open-source VL models. Based on this validation, we propose a targeted intervention: inserting normalization layer enforce strict norm alignment. However, practical implementation reveals a critical optimization bottleneck. Since text embeddings in mod- ern LLMs (e.g., Qwen2.5) exhibit extremely low magnitudes, aligning visual tokens to this target requires initializing the normalization gain to minute value. This naively triggers a vanishing gra- dient problem, detaching the vision encoder from supervision. To resolve this, we introduce a Global Weight Compensation (GWC) mechanism. This technique decouples the forward norm compression from the backward gradient magnitude, ensuring effective learning even under extreme alignment constraints.
In this work, our key contributions are threefold:
Theoretical Identification of Asymmetric Dynamics. We are the first to identify and theo- retically formalize the issue of cross-modal norm disparity in Pre-Norm MLLMs. Our analysis reveals an "asymmetric update dynamic" where high-norm visual tokens exhibit "representational inertia," leading slower semantic evolution compared to text tokens.
Extensive Empirical Validation. We provide extensive empirical validation across suite of mainstream open-source MILLMs, demonstrating that the predicted norm disparities and asym- metric update rates exist, confirming our theoretical model in practice.
Simple and Robust Solution. We propose Gradient-Awart Norm Alignment, incorporating novel WC mechanism. This approach resolves the optimization dilemma caused by extreme norm compression. Our experiments show that this method yields significant performance gains not only multimodal tasks but also, notably, on text-only benchmarks, indicating more holistic improvement to the model's capabilities.
2 PRELIMINARIES
Our analysis is grounded in the core components of modern Transformers: self-attention, normal- ization, and the choice of residual architecture.
Self-Attention and Normalization Layers. The self-attention mechanism projects input se- quences into queries (Q), keys (K), and values (V) compute unnormalized dot-product scores
2

---

Published as a conference paper at ICLR 2026
A
X
Pre Norm
Large
Language
Model
Norm
Eljtextily 0.86 56.88 Ellimagelyy
Embeddlngs
PFN
NORM MISMATH!
FFV
prG
Most
Embedding
Layer
ViT
Artention
Mork
Norm
Artention
Text Tokena
Iratoforrser
Morh
X
Image Patches
Norm
NNrm
(a) Pre-Norm vs Post-Norm
(b) The Norm Mismatch Problem Induced by MILM Architectures
Q Q To stabilize training and manage activation scales, normalization is applied. LayerNorm (Ba al., 2016) normalizes features per token, where g and [3 are learnable gain and bias parameters:
B[x
LayerNorm(z)
g
Var
RMSNorm (Zhang & Sennrich, 2019) computationally efficient variant that omits mean- centering and is widely adopted in modern LLMs:
R.M.Nor((z)
+ +
Pre-Norm Post-Norm Residuals. A Transformer block refines representations via an additive
update: ((+1) h + (!) The placement of normalization dictates the network's training
dynamics:
Normalizes after the residual sum, preserving strong representational fidelity but often impeding gradient flow deep networks:
Post-Norm:
(1+1) = Norm (h + Sublayer(h
Pre-Norm: Normalizes within the residual branch, leaving the skip connection unaltered:
Ah Sublayer(Norm(A" (1))
h (++ ( + Ah
This uninterrupted connection guarantees stable gradients. However, because the residual out- put (1+1) bypasses normalization, the variance- and consequently the Lg norm- nom-mmootonicclly accumulates with depth (Kim et al., 2025).
Building on these definitions, we formalize our central argument: the Pre-Norm design inherently creates a destructive dynamic imbalance MLLMs. By injecting features from a deep, pre-trained vision encoder (where norms have already accumulated) into the relatively low-norm embedding space of LLM, severe initial L2 norm disparity established exactly at the cross-modal inter- face.
3 THEORETICAL ANALYSIS OF FORM-INDUCED DECOUPLING EFFECT
In this section, we present a theoretical proof that this initial norm imbalance is not a static issue but rather the catalyst for accelerated geometric divergence between the two modalities, ultimately suppressing the cross-modal attention signal. The full mathematical derivation is provided in the Appendix.
3

---

Published as a conference paper at ICLR 2026
3.1 ANALYTICAL FRAMEWORK AND ASSUMPTIONS
Our proof is predicated on a set of simplifying assumptions that capture the core dynamics of the Pre-Norm architecture:
Modality Norm Imbalance: We analyze two cases: the imbalanced case (I: the ideal balanced case (k = 1).
| l2 A > 1) and
Uniform Update Magnitude: Due to the Pre-Norm design, the magnitude of the update vector, 1l2. is decoupled from the input norm ||2 We denote this uniform magnitude as C( for given layer.
||
Consistent Update Geometry: We assume the update vector Ah forms consistent expected angle, o, with the hidden state h for all tokens within a given layer.
Random Rotational Direction: We assume the direction of the rotational component of the up- date is drawn from a symmetric distribution over the relevant subspace.
3.2 ASYMMETRIC ANGULAR VELOCITY AND GEOMETRIC DIVERGENCE
To quantify the rate of directional change, we introduce the concept of effective angular velocity. The update vector Ah can be decomposed into component parallel to the hidden state h (which only scales its length) and a component orthogonal to it (which causes rotation). The effective angular velocity, measured by the angle of pure rotation fan. is driven solely by this orthogonal component. As derived in Appendix B, its tangent is given by:
( sin(Ø)
tan(fen) = ||1|| + CO cos(1))
(2)
A direct and critical consequence of our framework is that this angular velocity becomes asymmetric in the imbalanced case. Because uniform update magnitude C(0)) is applied to hidden states of different norms, the high-norm vision tokens exhibit lower effective angular velocity than the low-norm text tokens. Formally, for ||l > || 2|2, it follows that:
tan(fent. vis) < tan(Oen,
This disparity imparts higher "representational inertia" to visual tokens. In Appendix B, we rigor- ously prove that this asymmetry leads to an accelerated geometric divergence between the represen- tations of the two modalities, which in turn weakens the underlying similarity signal available to the attention mechanism.
3.3 SIGNAL DEGRADATION AND ATTENTION COLLAPSE
The geometric divergence derived above has a specific and destructive consequence for the attention mechanism: it prevents the semantic alignment of related tokens. In functioning multimodal model, text query q seeks a semantically relevant visual key kt. Specifically, this retrieval process involves evolving the hidden states into configuration where their projections are geometrically aligned in the shared metric space, allowing their dot product be maximized.
However, the "representational inertia" creates mechanical barrier to this alignment. While the low-norm text token rapidly updates its orientation query the image, the high-norm visual token updates too slowly to reciprocate. This results in persistent angular misalignment for semanti- cally related pairs. Mathematically, the dot product signal for a relevant pair, Sia x q kt, is fundamentally capped by this geometric lag.
In contrast, for irrelevant pairs (noise), the vectors remain approximately orthogonal in high- dimensional space regardless of the inertia. The critical failure mode is thus collapse in the Signal- to-Noise Ratio (SNR). Because the score of the relevant visual token Sjel is suppressed while the by the Softmax oper-
background noise levels remain static, the attention rechanism-govemeee its ability to sharply distinguish the target visual region. Formally, we prove in Ap- pendix B.4 that for the expected attention scores of semantically relevant pairs:
E (((((mh))
E E[stbal))
4

---

Published as a conference paper at ICLR 2026
This provides first-principles explanation for the diffuse and unfocused attention maps observed in baseline models (Appendix G). The norm imbalance denies the model the geometric agility required to establish strong correlations, leading to the effective loss of visual information.
4 EMPIRICAL VALIDATION: PROBING THE DYNAMICS OF NORM IMBALANCE
Our theoretical analysis provides a formal, first-principles explanation for how norm imbalance can impair multimodal fusion. However, this framework relies on a set of simplifying assumptions to ensure analytical tractability, while the dynamics of large-scale MI.LMs are considerably more com- plex. Therefore, to bridge the gap between our idealized model and real-world behavior, conduct a series of empirical investigations. These experiments are designed to probe whether the core con- sequences predicted by our nhmely.namely, the persistence of norm imbalance and the resulting asymmetric update dynamics manifest state-of-the-an Pre-Norm MLLMs. Our investigation is guided by the following research questions:
RQI: Existence of Initial Norm Disparity. Do visual and text tokens exhibit significant norm mismatch at the modality interface?
To answer this, we benchmarked the L2 norms from both sides of the modality interface. For the visual modality, coders-CLIP (Radford et al., 2021), SigLIP (Zhai et al., 2023), SigLIP-v2 (Tschannen et al., 2025), and Moon (Team et al., 2025). For the text modality, we established a baseline by com- puting the average L2 norm of the text embedding layers from prominent LLMs: Qwen2.5 (Bai al., 2025), Qwen3 (Yang al., 2025), and Llama3.2 (Grattafiori et al., 2024). The analysis of vision encoders was conducted on dataset of 1000 samples drawn from the MMBench, POPE, and MM-Star benchmarks, which serves as the foundation for all subsequent experiments this section. The combined results are presented in Table 1.
measured the output norms of four representative vision en-
Table 1:
Le norms and hidden dimensions at the modality interface: Vision Encoder outputs vs.
LLM text embeddings (mean + std).
Modality
Model
Dimension
Average L2 Norm
CL.I--ViT-large-pttch
1024
29.30 ::2 17.12
4-3884
1152
71.78 13.95
Visual
10L1I2--0--400-pplle
14-384
1152
59.37 + 106.08
MoonViT-SO-Aee ViT-SO-400M
1152
72.17 7.13
Swen2.5-7B-Innstren
3584
0.80
Text
@wen3-88--Insrrue
4096
1.38
1lama3.2-3BB --I-ttuutttt
3072
1.09
As shown in 1, vision encoder output norms are substantially larger than those of text embed- dings. This disparity persists because the encoders' contrastive pre-training-ever with final post-norm- not designed to align with the norm scale of an external LLM's embedding space.
RQ2: The Efficacy of the Adapter. Does the projection adapter harmonize the initial norm disparity before the tokens enter the LLM backbone?
Within MLLMs, the projector's role is to map visual tokens into the LLM's textual embedding space. A critical question is whether this process also serves to align their norms. To investigate this, analyzed a suite of prominent models: LLaVA-v1.5 (Li et al., 2024a), Qwen-2.5-VL (Bai et al., 2025), KimiVL (Team et al., 2025), and GLM-4. IV (Hong et al., 2025). For each model, we measured the La norm of visual tokens both before and after the projector and compared them to the text token norm. The results are summarized in Table 2.
The results in Table 2 reveal a clear spectrum of effectiveness across different projector designs. While sophisticated projectors like those in KimiVL and GLM-4.IV demonstrate a significant capability for norm compression, substantial disparity between visual and text token norms persists in all analyzed models. This varied effectiveness highlights a key finding: simply inserting normalization layer within the projector is not a guaranteed solution. With the exception of
5

---

Published as a conference paper at ICLR 2026
Table 2: L2 norms of visual tokens (before and after projector, mean + std) vs. text tokens.
Model
Visual (Before Proj.)
Visual (After Proj.)
Text (Embedding)
LLaVA-v1.5
28.71 +: 16.87
39.96
45.58
1.08
Qwen-2.5-VL.
3484.24 3882.22
56.88
25.73
0.86
KimiVL
137.93 1 34.43
4.78 1: 2.11
0.85
GLM-4.IV
47.44 4.58
4.58 + 2.03
0.80
LLaVA-v1.5, all other models incorporate internal norm layers, yet their final output norms differ by an order of magnitude.
This leads to broader discussion current design practices. We note that these architectural choices and their impact on cross-modal norm alignment are seldom, ever, addressed in the models' respective technical reports.
Notably, the systemic inflation of text embedding norms observed in Qwen-2.5-VV and Qwen- 2-VL (detailed in Appendix C) corroborates that norm discrepancy is fundamental bottleneck. This phenomenon suggests that the model is forced to inefficiently adjust its static embedding parameters passively compensate for the massive norm gap the modality interface.
RQ3: Asymmetry Update Dynamics. Do visual and textual hidden states exhibit different update rates, predicted by our theory of asymmetric angular velocity?
This question serves as the most direct empirical test of our theory's 's core mechanism. We use the cosine similarity between consecutive layers (1-1 and change, a metric conceptually linked to angular velocity. A higher similarity score implies a smaller angular change and thus slower update rate. We computed this metric for both modalities across all layers to determine if a systematic divergence in their update rates exists, as shown in Figure::
proxy for the rate of representational
The results in Figure confirm our theoretical predictions, revealing a consistent divergence in update rates between visual and text tokens across all analyzed models. Notably, the magnitude of this dynamic asymmetry appears to be directly correlated with the initial norm disparity identified in RQI and RQ2. Models with smaller initial norm gap, such as Kimi-VL. and GLM-4.5VV exhibit less pronounced difference in update rates. Conversely, models with more severe norm imbalance, like VA- and Qwen-2.5-V4 VL, demonstrate a significantly larger gap in their update dynamics, providing strong correlational evidence for our theory.
RQ4: Norm Discrepancy in Models with Visual Embedding Tables. Does the significant norm disparity persist in architectures utilizing probabilistic visual tokenization, such Ovis 2.5 (Lu et al., 2025y?
To verify whether the norm discrepancy and its dynamic consequences persist across different architectural paradigms, we conducted experiments on Ovis-2.5-9B, a model that employs prob- abilistic visual tokenization. Unlike LLaVA-style models that project continuous features, Ovis utilizes a discrete visual vocabulary derived from a SigLIP2 encoder.
10aa Bretarth
Analyii
Table 3: Lg norms in Ovis 2.5. Note that while the visual tokens strictly clustered (low std), their magnitude remains drastically higher than text tokens.
Modality Source
Avg L2 Norm
SigLIP2 Vocab (Frozen)
64.00
Visual
0.71
Text Vocab (Base LLM)
1.38
Figure 2: Layer-wise cosine similarity in Ovis 2.5. Despite the massive norm gap (64 vs 1.38), the update rates of visual and text to- kens remain remarkably synchronized.
Text
0.32

---

Published as a conference paper at ICLR 2026
As detailed in Table 3, substantial static norm discrepancy persists in Ovis 2.5: the visual vo- cabulary exhibits an average norm of 64.00 -0ll-lllllyyyyy stemming from its derivation via the SigLIP2 to the text norm of 1.38.
However, Figure indicates that the internal dynamics differ from those observed in continuous projection models, Despite the norm disparity, the update rates (proxied by cosine similarity) for visual and text tokens do not exhibit the marked asymmetry found in VLa architec- tures. The curves overlap significantly, suggesting that the discrete tokenization paradigm may inherently mitigate the representational inertia usually associated with high-norm inputs.
(Cues
Desaat Laper
Riscuk Libee
(a) LLVVA-155
(b) Qwen-2.5-VV.
Indentaper Cacoie Screlaity Anabyii
Intankayee Cocile Sindlaityy Analysic
(c) GLM4.1-V
(d) Kimi- VL
Figure Inter-layer cosine similarity of hidden states for visual vs. text tokens.
EXPERIMENTS
Our theoretical analysis in Section 3 posited mechanism whereby disparity leads update asymmetry and, consequently, suppressed visual attention. critical question remains, however: do these internal dynamics translate into a tangible degradation of the model's downstream capabilities? To investigate this link between internal mechanics and practical performance, we conducted a series of comparative experiments.
5.1 METHOD: GRADIENT-AWARD NORM ALIGNMENT
To enforce norm alignment between visual and text tokens, we introduce a straightforward interven- tion: inserting an additional LayerNorm layer immediately after the visual projector. Crucially, the leamable gain parameter g of this layer is initialized to explicitly match the average L2 norm of the text tokens within the LLM's embedding space.
Determining the Aligument Target. First, we compute the target L2 norm, T. by averaging the norms of all non-zero vectors from the language model's text embedding matrix, Wa:
T
Σ
||]]2,
where W = (w We ||w||2 > e).
(5)
W
WEW
Based this target, the gain parameter g is initialized to scalar Pinit T/ VD.

---

Published as a conference paper at ICLR 2026
The Optimization Dilemma. A critical challenge arises from the intrinsic properties of the LLM's embedding space. Specifically, pre-trained text embeddings exhibit an extremely small magnitude (e.g., ||wl2 R 1 even for D 4096). Consequently, aligning to this target requires initializing to minute value (e.g., finit Re 0.01). In standard backpropagation, the gradient flowing back to the vision encoder is scaled by this weight: VxL Vy g- Thus, minute initialization triggers vanishing gradient problem, detaching the vision encoder from supervision.
Resolution via Global Weight Compensation. To resolve this, we propose a Global Weight Compensation mechanism implemented via backward hook. Instead of redefining the gradient simply as identity, we actively rescale the gradients to counteract the dampening effect the small initialization. Formally, let E Di be the mean magnitude of the gain vector. We define the
backward pass dynamics as:
Backward ( V Ogga
(6)
Standard Gradient
Compensation Factor
By multiplying the gradient by the inverse of the global weight magnitude, effectively cancel out the scaling term (g g-1 Rs 1), restoring the gradient flow to unit scale while maintaining the precise architectural norm alignment in the forward pass.
5.2 EXPERIMENTAL SETUP
Our experiments are conducted within the LLaVA-1.5 architectural framework. Specifically, we employ 1itma--,--3B-Instteee (Grattafiori et al., 2024) and Qwen2.5-7B-Instru (Bai et al., 2025) as the base language model and SigLIP-S0400MM-atthh P4t as the vision encoder. Further details are provided in Appendix
A detailed list of the evaluation benchmarks is provided in the Appendix E; for all tasks, we em- ployed a greedy decoding strategy.
5.3 RESULTS AND ANALYSIS
5.3.1 MAIN PERFORMANCE GAINS
The results in Table 4 reveal backbone-dependent response. For Llama-3.2, simple norm alignment suffices to yield robust improvements. Conversely, Qwen2.5 exhibits the predicted vanishing gradi- enr pathology: the minute initialization stifles gradient flow the adapter, causing naive alignment to improve only text metrics while stagnating on multimodal tasks. Introducing our Global Weight Compensation (GWC) resolves this optimization bottleneck, unlocking gains across both domains. While this validates that norm alignment is fundamental for holistic model capability, the poten- tial for gradient oscillation with GWC suggests that exploring more stable compression strategies remains vital direction for future research.
We visualized the attention matrices in Appendix G. The analysis reveals that the baseline model, text-to-image attention is inappropriately and broadly concentrated the bottom regions of the im- age. This suggests a failure in semantic fusion, caused by the positional proximity bias introduced by RoPE's ' distance-decay property. In stark contrast, norm- memmaligged model's eoxt-to-image atten- tion correctly converges on the specific image regions that are semantically relevant to the text query. This visual evidence provides direct confirmation that method successfully restores meaningful cross-modal attention by correcting the underlying dynamic imbalance, thus enabling true feature fusion.
We also analyze the temporal evolution of layer-wise similarity and its convergence behavior during pre-training Appendix
5.3.2 ABLATION STUDY: THE CRITICAL ROLE OF INITIALIZATION
To isolate the effect of our proposed initialization strategy, conducted a crucial ablation study. We compared our method against a baseline where the added LayerNorm layer was initialized with default parameters (gain=1, bias=0). We analyzed the learned parameters immediately after the
8

---

Published as a conference paper at ICLR 2026
Table 4: Performance comparison on various benchmarks across different backbones. w/ Norm
(w/o GWC) denotes norm alignment with naive initialization (leading to gradient vanishing). w/
Norm (w/ GWC) denotes our proposed Global Weight Compensation mechanism.
Model
Method
MMBenchare
MM-Star
POPE
SEED-Bench-2
OCRBench
w/o Norm
71.39
37.72
88.14
42.86
40.70
Llama
w/ Norm (w/o GWC)
72.16
(1.49
88.88 (+0.74)
47.26
45.60 (+1.90)
w/ Norm (w/ GWC)
71.82
(1.54
(8.26
45.56 (+2.70)
44.10
w/o Norm
76.80
50.34
87.51
56.65
47.00
Qwen
w/ Norm (wo GWC)
75.60
48.08 (-2.26)
87.83 (40.32)
59.51 (+2.86)
47.60 (40.00)
w/ Norm (w/ GWC)
77.66 (10.66)
50.58 (40.24)
87.87 (40.36)
58.27 (+1.62)
49.40 (42.40)
Model
Method
ScienceQA
A12D
HellaSwag
MMLU
Avg
w/o Norm
78.99
60.17
65.96
45.19
59.01
Llama
w/ Norm (w/o GWC)
80.83 (+1.8)
(3.24
66.01 (+0.05)
53.21 (+8.02)
62.04(43.00
w/ Norm (w/ GWC)
81.00 (+2.01)
(1.65
65.99 (+0.0))
51.60 (+6.1)
61.27 (+2:26)
w/o Norm
82.81
73.74
70.56
71.02
68.49
Qwen
w/ Norm (wlo GWC)
82.20 (40.6))
(2.70
73.73
71.14 (+0.12)
68.71 (+0.22)
w/ Norm (w/ GWC)
82.93 (+0.12)
74.61
71.64
71.74 (+0.72)
69.41 (+0.90)
LLaVA Stage I pre-training phase. As shown in Table the parameters of the defaull-initializzed layer remained largely unchanged from their initial state, indicating that the optimization process failed to begin effectively without reasonable starting point. In contrast, our method shows mean- ingful parameter updates even after this initial stage. This demonstrates that simply adding a norm layer is insufficient; our targeted initialization essential place the parameters a gradient-rich region of the loss landscape, enabling effective learning.
Table 5: Learned parameters of the added LayerNorm layer after Stage I pre-training, comparing default initialization with our proposed strategy.
Parameter
Metric
Default Init
Our Init (After Stage 1)
(After Stage 1)
L2 Norm
53.2500
2.2812 0.0400 (1: 0.0001)
Gain (g)
Mean of Abs.
0.9609 (1: 0.0005)
Bias (3)
Mean of Abs.
0.0175 (1L 0.0002)
0.0152 (L 0.0001)
5.3.3 DIAGNOSTIC ANALYSIS: VERIFYING THE MECHANISM OF IMPROVEMENT
Finally, we performed a diagnostic analysis to investigate whether the performance gains correlate with the mitigation of the dynamic imbalance we identified. We analyzed the internal states of the fully trained model (after Stage 2) comparing the baseline against our norm-aligned method. Figure visualizes two key metrics:
Layer-wise L2 Norms (Fig. 4a): The left panel indicates that our method effectively har- monizes the visual token norms with the text token norms starting from the initial layers and maintains this alignment throughout the model's depth. In contrast, the baseline model exhibits a marked and persistent norm divergence.
Inter-layer Cosine Similarity (Fig. 4b): The right panel illustrates the corresponding shift in update dynamics. With our intervention, the update rates (proxied by cosine similarity) of visual and text tokens exhibit significantly improved synchronization. This observa- tion suggests that our method successfully mitigates the extreme asymmetry present in the baseline, where the persistently high similarity of visual tokens pointed to substantial "rep- resentational inertia."
Collectively, these findings corroborate theoretical framework: correcting the static norm dispar- ity appears to alleviate the asymmetric update rates, thereby facilitating the observed improvements in downstream performance.

---

Published as a conference paper at ICLR 2026
12 Norn meddenn States Leber Depth
(90be Staee laffy Degon
(a)
b
Figure 4: A comparison of token dynamics with and without our norm alignment method. (a) shows the layer-wise L2 norm evolution, while (b) shows the inter-layer cosine similarity, which acts as proxy for update rate.
CONCLUSION
Our analysis reveals critical, previously undiscovered dynamic within Pre-Norm MLLMs: an "asymmetric update." We have formalized this dynamic theoretically and validated it empirically, showing it to be direct consequence of the severe norm disparity between visual and text tokens. This analysis demonstrates that the dynamic manifests as "representational inertia" high-norm visual tokens, fundamentally impairing cross-modal fusion at an architectural level. It was this deep analysis of the mechanism that motivated our targeted solution of enforcing norm alignment via single LayerNorm with global Weight Compensation. The resulting significant performance gains on both multimodal and, critically, text-only tasks, serve as compelling validation for our core analysis, confirming that resolving this dynamic imbalance unlocks the model's full potential.
ACKNOTLEDGEMENT
This work is supported by National Natural Science Foundation of China (92470121, 62402016). National Key R&D Program of China 10040033,, Zhongguancun Academy (Grant No.s C20250204, C20250602), and High-performance Computing Platform of Peking University.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Yean-Baptist Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 23736, 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502. 13923, 2025.
Tianyi Bai, Hao Liang, Binwang Wan, Yanran Xu, Xi Li, Shiyu Li, Ling Yang, Bozhou Li, Yifan Wang, Bin Cui, et al. A survey of multimodal large language model from a data-centric perspec- tive. arXiv preprint arXiv:2405. 2024.
Junbum Cha, Wooyoung Kang. Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhancee projector for multimodal 1lm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13817-13827, 2024.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056-27087 2024b.
Xinlong Chen, Yue Ding, Weihong Lin, Jingyun Hua, Linli Yao, Yang Shi, Bozhou Li, Yuanx- ing Zhang, Qiang Liu, Pengfei Wan, et al. Avocado: An audiovisual video captioner driven by temporal orchestration. arXiv preprint arXi::2510.10395, 2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint ar5i 14683, 2025.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
Yingjie Fu, Bozhou Li, Linyi Li, Wentao Zhang, and Tao Xie. The first prompt counts the most! an evaluation of large language models iterative example-based code generation. Proceedings of the ACM on Software Engineering. 2883 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The laama 3 herd of models. arXiv preprint 24Xiv:2407.2117 1783, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.088 03300, 2020.
11

---

Published as a conference paper at ICLR 2026
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang. Jiale Cheng, Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, arXiv e-prints, pp. arXiv 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2. 1276, 2024.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456. pmlr, 2015.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In European conference on computer vision, pp. 235 251. Springer, 2016.
Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri-In: Revisiting normaliza- tion layer in the transformer architecture. arXiv preprint 2rXiv:2502.02732. 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, al. Llava-onevision: Easy visual task transfer. arXiv preprint arXhv:2408.03322 2024a.
Bohao Li, Yuying Ge, Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-22plus Benchmarking multimodal large language models with text-rich visual comprchension. arXiv preprint arXiv:2404.1677 2024b.
Bozhou Li and Wentao Zhang. Id-align: Rope-conscious position remapping for dynamic high- resolution adaptation in vision-language models. arXiv preprint arXiv:2505.21445, 2025.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024a.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 2212)220000, 2024b.
Zheng Liu, Hao Liang, Bozhou Li, Tianyi Bai, Wentao Xiong, Chong Chen, Conghui He, Wen- tao Zhang, and Bin Cui. Synthvlm: High-efficiency and high-quality synthetic data for vision language models. arXiv preprint axXiv:2407.20775, 2024c.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2522 2022.
Shiyin Lu. Yang Li, Qing Guo Chen, Zhao Xu. Weihua Luo, Kaifu Zhang, and Han Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797 2024.
12

---

Published as a conference paper at ICLR 2026
Shiyin Lu, Yang Li, Yu Xia, Yuwei Hu, Shanshan Zhao, Yanqing Ma, Zhichao Wei, Yinglun Li, Lunhao Duan, Jianshan Zhao, et al. Ovis2. 5 technical report. arXiv preprint arXiv:2508. 2025.
Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for Imms. Advances in Neural Information Processing Systems, 37:23464-225 23487, 2024.
Jianing Qi, Jiawei Liu, Hao Tang, and Zhigang Zhu. Beyond semanties: Rediscovering spatial awareness in vision-language models. arXiv preprint arXiv:2503.17344 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Pooyan Rahmanzadehgervi. Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision, pp. 18- 34, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127063 2024.
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint DrXiv:2504.07491, 2025.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint 2577 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, al. Videorope: What makes for good video rotary position embedding? arXiv preprint arXiv:2502.05173. 2025.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseck-v12: Mfxture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International conference on machine learning, pp. 10524-10533. PMLR, 2020.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint M.Xi::2505.09388, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint arXiv:/905.000 2019.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov. and Lucas Beyer. Sigmoid loss for language image pre-training. Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-1119999 1986, 2023.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural infor- mation processing systems, 32, 2019.
13

---

Published as a conference paper at ICLR 2026
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm- 1lms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13600 2024.
Zhijian Zhuo, Yutao Zeng, Ya Wang, Sijun Zhang, Jian Yang. Xiaoqing Li, Xun Zhou, and Jinwen Ma. Hybridnorm: Towards stable and efficient transformer training via hybrid normalization. arXiv preprint 25X::25033.04598 2025.
14

---

Published as a conference paper at ICLR 2026
A BACKGROUND & RELATED WORK
A.1 MULTIMODAL LARGE LANGUAGE MODELS
The remarkable success and emergent capabilities of Large Language Models (LLMs) in natural language processing have catalyzed efforts to generalize their powerful abilities other modalities (Achiam et al., 2023; Hurst et al., 2024; Comanici et al., 2025; Fu et al., 2025; Bai et al., 2025; Yang et al., 2025; Wu et al., 2024; Chen et al., 2025). In the multimodal domain, this trend has spurred the rapid development of Multimodal Large Language Models (MLLMs).
Early explorations in MLLMs, such as Flamingo (Alayrac et al., 2022) and BLIP-2 (Li et al., 2023a), primarily relied on the cross-attention mechanism for modality fusion. A subsequent evolu- tion witnessed paradigm shift toward a simpler and more efficient approach, an approach popular- ized by LLaVA (Liu et al., 2023) that has now become the undisputed mainstream. The LLaVa-style architecture eschews the complexity of cross-attention in favor of a more direct solution: it employs simple projection module, typically Multi-Layer Perceptron (MLP), to map visual token features directly into the LLM's word embedding space. Conceptually, this treats the image as a sequence of special "visual words" prepended to the text input, which are then processed uniformly by the LLM in an auto-regressive manner. The simplicity, scalability, and powerful performance of this paradigm fasemmm-aattuarlllyl when combined with visual instruction tuning havinghhve firmly established it as the foundational blueprint for the vast majority of today's 's advanced MLLMs
Despite the dominance of the LLaVA paradigm. the pursuit of optimal cross-modal fusion remains an active area of research. Investigators continue to experiment with more sophisticated projector de- signs (Team al., 2025; Hong et al., 2025; Cha al., 2024), alternative representation schemes like visual vocabularies (Lu et al., 2024), or deeper fusion strategies (Meng et al., 2024), novel methods for adapting the core architectures of large models for multimodal scenarios (Deng et al., 2025; Wei et al., 2025; Li & Zhang, 2025). Beyond these methods, other researchers have approached the challenge from data-centric perspective (Bai al., 2024; Liu et al., 2024c).
A.2 NORMALIZATION
Normalization layers are cornerstone of modern deep learning, designed to stabilize the training process and accelerate model convergence. By re-scaling the distribution of activations between layers, normalization effectively mitigates the internal covariate shift problem and ensures smooth gradient propagation in deep networks. While Batch Normalization (BN) (loffe & Szegedy, 2015) was seminal work in this area, its dependency on batch size makes it less suitable for natural language processing tasks with variable sequence lengths. Layer Normalization (LayerNorm) was therefore introduced, performing normalization along the feature dimension independently of the batch, and it quickly became the standard for Transformer architectures (Vaswani et al., 2017). This paradigm was further refined by RMSNorm Zhang & Sennrich (2019), which improves computa- tional efficiency by removing the mean re-centering step while maintaining performance, leading to its widespread adoption in many modern L.MSs such Llama.
A critical design axis in Transformer architectures is the placement of the normalization layer rela- tive to the residual connection, giving rise to the Pre-Norm and Post-Norm paradigms Xiong et al. (2020). The original Post-Norm design applies normalization after the residual addition, which can help preserve strong representational fidelity but is often prone to training instability in deep mod- els. In contrast, the Pre-Norm approach places normalization within the residual branch, greatly improving gradient flow and training stability by maintaining "clean" skip-connection path. This has made it the de facto standard for large-scale language models. However, the Pre-Norm archi- tecture has well-documented side effect: because the hidden states on the main path are never re-normalized, their L2 norm tends to accumulate and grow with network depth (Zhuo al., 2025).
Qi et al. (2025) have also observed the norm discrepancy between visual and textual tokens in MLLMs, they predominantly attribute the resulting performance degradation to the failure of Rotary Positional Embeddings (RoPE (Su et al., 2024) ) in handling high-magnitude features. However, argue that this attribution overlooks the fundamental mechanics of the Pre-Norm architecture. Specifically, in this paradigm, normalization is applied before the query and key projections; conse- quently, the vectors processed by RoPE are already normalized.
15

---

Published as a conference paper at ICLR 2026
Recently, the community has begun re-evaluate this classic dichotomy, spurring research into alternative placement strategies. Recent works, like Peri-Norm (Kim et al., 2025) and Hybrid- Norm (Zhuo et al., 2025), have begun to explore combining normalization different points of the residual connection to merge the benefits of both paradigms. These efforts, however, aim to find universally optimal static design for unimodal models. In contrast, work takes diagnostic perspective: rather than proposing a new general architecture, we are the first to deeply analyze and reveal how the de facto standard Pre-Norm design itself directly induces destructive dynamic imbalance within the multimodal context.
B APPENDIX: DETAILED DERIVATION AND PROOFS
This appendix provides the full mathematical derivation for the claims made in Section 3.3. We connect the architectural cause (norm imbalance) to the functional failure (SNR collapse) through continuous four-step theoretical framework.
B. STEP I: KINEMATICS NORM IMBALANCE INDUCES VELOCITY ASYMMETRY
We begin by establishing the kinematic relationship between token norm and update speed. Con- sider hidden state h updated by residual vector Ah. The update can be decomposed into a parallel component (magnitude scaling) and an orthogonal component (direction rotation). The fective angular velocity, len, is determined by the ratio of the orthogonal update to the current state magnitude.
Under the assumptions of Uniform Update Magnitude (|Ah|l ~ C) and Consistent Update Geometry, the tangent of the effective rotation angle derived as:
C (i)(
tan()
(7)
|b/2 + C Cos(())
This formula establishes an inverse relationship: higher norm implies lower angular velocity. Applying this to the visual (hvis) and text (haa) tokens, where ||bvis] > ||aaa arrive fundamental velocity asymmetry:
Ovis 1xx
Having established that the modalities rotate at significantly different speeds, we next model how this velocity mismatch affects the evolution of their alignment over time.
Transition:
B.2 STEP 2: DYNAMICS - RECURSIVE DECAY OF SEMANTIC SIMILARITY
We now derive the law governing the evolution of the cosine similarity between the two modalities across layers. Let ( be the angle between the visual and text hidden states at layer I.
Recursive Decay. Assuming the rotational updates are drawn from a symmetric distribution in the orthogonal subspace, the expected cosine similarity evolves according to:
Theorem 1:
((coo(((((t)))
O J (os(O()))
where the retention factor Tell is the product of the cosines of the individual angular velocities:
(C]
cos
@ vis
cos
(10)
of
This theorem tells us that alignment retention depends on the product cos(O cos(O.x). The critical question is: Does the extreme asymmetry derived in Step make this retention factor better or worse compared to a balanced scenario?
Transition:
B.3 STEP 3: OPTIMIZATION ASYMMETRY ACCELERATES GEOMETRIC DIVERGENCE
We compare the decay factor 7 in the Imbalanced case (B & Oxx) against the Balanced case
(0 Re Owa).
16

---

Published as a conference paper at ICLR 2026
Lemma 1 (Asymmetry Maximizes Decay Rate). For fixed total update potential (represented by the geometric mean of angular velocities). the similarity retention factor = cos(f1) cos(02) is maximized when velocities are symmetric (0, = 82). Conversely, asymmetry strictly decreases "V-
Proof. Maximizing y is equivalent to minimizing 1/7² = (1 + tan 01) + tan 02). By the AM-GM inequality, for fixed product of tangents, the sum tan 0, + tan 02 is minimized when tan 0 = tan Oz. Thus, the asymmetric condition ||h,s] || >> || forces the system into a subopti- mal regime where similarity decays strictly faster than the balanced case.
Definition of the Lag Angle Ag. Since the similarity decays faster in the imbalanced case, the final angle between the vectors will be strictly larger. We define this accumulated geometric deficit as the Lag Angle:
A = imb ral >
(11)
Transition: We have proven that visual tokens lag behind the optimal alignment direction by AΦ. Finally, we must determine if the attention mechanism can tolerate this lag or if it leads to functional failure.
B.4 STEP 4: FROM GEOMETRIC MISALIGNMENT TO SIGNAL-TO-NOIEE RATIO COLLAPSE
Finally, we connect the geometric lag Ae derived in Step 3 to the failure of the attention mecha- nism. We analyze the interaction between the learnable projection matrices W W Wx k and the degraded input statistics.
Spectral Degradation of the Semantic Signal. The expected attention score is given by the bilinear form E[S] = TC(WC. where C.u B[vu"] is the cross-covariance matrix of the inputs. In the Imbalanced case, the geometric lag AΦ causes the visual vector Vimb to deviate from the optimal alignment direction. By performing an orthogonal decomposition along the signal direction (aligned with the text vector u), we have:
Vimb (os) + sin(AØ)V
(12)
Since the orthogonal component V1 contains correlation with the query, the singular values (spectral energy) of the covariance matrix are dampened:
Cimb R cos( C
(13)
2. The Fundamental Spectral Upper Bound. We consider the maximization of the attention score under the constraint that the model parameters must remain finite. Let the magnitude of the projection matrix be bounded by some fixed radius R (i.e., || [|W]] S R). We invoke Von Neumann's Trace Inequality, which states that the trace of a matrix product is bounded by the dot product of their singular values: Tr(WC) < Σ a, (W)); (C). Applying this, the maximum achievable expected score is strictly limited by the spectrum of the input covariance:
max EE [[Sin
I R Σ oj (Cimb) ss cos(Ae) R Σ of (CCoal))
(14)
W PSR
Since Ao > O, we have cos(Ad) < 1. This proves a structural theorem: =*For any fixed capacity R of the projection layers, the norm-induced geometric lag strictly reduces the maximum extractable attention score. The model cannot recover the lost signal energy without unboundedly increasing its weights.
3. Signal-to-Noise Ratio (SNR) Collapse. The attention mechanism's 's efficacy relies on the gap between signal and noise:
Gapma = Sig [
(15)
Signal: As proven above, the maximum signal is capped by the factor cos( (A))
Noise: For irrelevant tokens, the vectors are initialized to be approximately orthogonal
(Oimel R 90%). The geometric divergence acts as an isotropic random walk, introducing no systematic bias. Thus, the noise floor remains constant: IR] ([5m) A O.
Consequently, the -o-nal-to-Noise Ratio collapses (Gapıb Gapha). This reduced gap forces the Softmax function to produce a higher-entropy. more uniform distribution, mathematically explaining the diffuse attention maps observed in experiments.
17

---

Published as a conference paper at ICLR 2026
C EMPIRICAL ANALYSIS OF EMBEDDING NORM SHIFTS
To investigate the impact of multimodal training on the embedding space, we analyzed the statistical properties of token embeddings across the Qwen2 and Qwen2.5 model families. We systematically compared the pre-trained pure language models (Base) against their visual-language counterparts (VL).
C.I GLOBAL STATISTICS
Table 6 presents the global statistics (Mean and Standard Deviation) of the embedding norms. We observe that while Qwen2-V VL maintains a mean norm similar its base model, Qwen2.5-VL ex- hibits a noticeable systemic inflation (Mean: 0.80 - 0.90), indicating a broader shift in the embed- ding distribution during multimodal alignment.
Table 6: Comparison of global embedding statistics (L2 Norm Mean and Standard Deviation) be- tween Base (Pure Text) and VL (Multimodal) models.
Qwen2 Family (7B)
Qwen2.5 Family (7B)
Metric
Base
VL
Base
VL
Mean Norm (w)
0.6908
0.6820
0.8031
0.8965
Std. Deviation (a)
0.1610
0.1588
0.1801
0.1821
C.2 DETAILED ANALYSIS OF TOP-NORM TOKENS
Tables 7 and detail the Top-10 10 tokens with the highest L2 norms. The comparison reveals fundamental structural change in the embedding space:
Base Models (Pure Text): In pure language models, the tokens with the highest norms typically rare subwords or specific syntactie markers (e.g., 'áves', "=>²), The maximum norm relatively contained ( 1.0 - 1.1).
2. VL Models (Multimodal): Upon multimodal training, the visual boundary tokens (< visi on_start >>, lvvit on_end >> emerge extreme outliers. In Qwen2-VI., they reach norms of 2.4, far exceeding the previous maximums. This explicitly confirms that the model allocates significantly larger magnitudes to visual anchors to accommodate the high-norm visual features.
Table 7: Qwen2 Family Comparison: Top- 10 tokens by Lg norm. Note how the visual tokens (in VL) far exceed the magnitude of the outliers in the Base model.
Qwen2-7B- (Base)
Oweu2-VL-7B-Instree (VL)
Rank
Token
Lg Norm
Token
Norm
1
'vvis''
1.0273
2.4590
2
m=>?
0.9741 <vision_end/>
2.3320
3
> () 7 Anlla
0,9258
áveis'
1.0049
4
/
0,9072
m=>
0.9458
5
'is'
0,9004
's'
0.9067
6
1 'an
0.8999
() Vinlia
0,9009
7
'1e'
0.8994
0.8999
8
'ed'
0.8979 'is'
0.8989
9
0.8965 'an'
0,8975
10
1 far'
0.8965 1ee
0.8965
D DETAILED IMPLEMENTATION OF NORM ALIGNMENT
Our norm alignment layer, denoted as Compenssttttm www.ssattddaaeeerrrr.... is designed enforce norm compression in the forward pass while preserving gradient magnitude in the back-
18

---

Published as a conference paper at ICLR 2026
Table 8: Qwen2.5 Family Comparison: Top-10 10 tokens by L2 norm. In the Base model, visual
tokens are initialized to zero or unused. In the VL model, they become the largest vectors.
Qwen2.5-7B-Inse (Base)
Qwen2.5-VV-77--nstrree (VL)
Rank
Token
L2 Norm
Token
L2 Norm
'ávei''
1.1201
< viisioneend//>
2.1523
2
1.1025
< vission.start:
1,4004
3
'is'
1,0322
1.2764
's"
1,0303
unb
1.2158
5
'an'
1,0293 áveis'
1.1748
"en'
1,0283
: Mnnlnn
1.1670
7
(Chinese Char)
1.0264
'en'
1.1670
out!
1.0264
's'
1.1650
'on'
1.0254
(Chinese Char)
1.1650
'Le'
1,0254
(Chinese Char)
1.1650
ward pass. Unlike standard LayerNorm, we introduce gradient compensation mechanism to handle the extremely small initialization of the gain parameter.
Forward Pass (Standard LayerNorm). Given the input vector E TR. (representing the pro- jected visual tokens), first compute the mean μ and variance o² across the feature dimension:
D Da 14)2.
Σ TG
o²
(16)
The input then normalized and transformed by the affine parameters, the learnable gain g and bias [
V y=xo g B.
(17)
Crucially, consistent with our method description Section 5.1, g initialized to a small sealar value (derived from the target text norm 7) to ensure immediate alignment, while &3 is initialized to zero.
Backward Gradient Compensation. Since is initialized to minute value, standard backprop- agation would attenuate the gradients flowing back to the input & (and consequently to the vision encoder) by factor proportional to ||g||- To prevent this, we register backward hook the normalized tensor & to dynamically rescale the gradients.
The compensation process proceeds as follows during training:
1. Global Gain Aggregation: We compute the mean absolute value of the gain parameter vector g to obtain global scaling scalar:
D Σ 19d.
Mg D
(18)
1=1
2. Safety Clamping: To ensure numerical stability and prevent division by zero (in the rare event of parameter collapse), we clamp the scalar with minimal threshold d = 10 10-3
Psate max [gg 8).
(19)
3. Compensation Factor Application: The gradient V&L flowing backwards through the normalization operation is scaled by the inverse of this scalar:
V Lscaled V&L X
(20)
Msaae
This mechanism effectively decouples the forward scale (controlled by g) from the backward gra- dient scale (restored to ~ 1.0). By preserving the gradient magnitude, ensure that the vision encoder receives effective supervision signals from the very first training iteration, despite the se- vere norm compression applied at the interface.
19

---

Published as a conference paper at ICLR 2026
E
TRAINING DETAILS
E EXPERIMENTAL. SETUP
Our experiments are conducted within the LLaVA-1.5 architectural framework. To ensure a com- prehensive evaluation, we employ two distinct base language models: Llama-3.2-388 2--ttunnttt and Qwen2.5-7B-Instructt Both models are coupled with L.L.I--S0400MPatcchhh
as the vision encoder. We follow a unified two-stage training protocol for both backbones: the first stage consists of one epoch of feature alignment pre-training on the LLaVA-558K dataset, using a learning rate of le-3, per-device batch size of 2, and gradient accumulation steps, resulting in global batch size of 256. This is followed by one epoch of full-model instruction tuning on the LLaVA-NeXT dataset, for which the learning rate is decreased to le-5 for the language model and 2c-6 for the vision en- coder, with a per-device batch size and 4 gradient accumulation steps, corresponding to a global batch size of 128. Across both stages, we utilize cosine learning rate scheduler with a warmup ratio of 0.03 and set weight decay to O. Notably, we do not employ dynamic high-resolution strategies; all images are uniformly resized to 384 384. To ensure reproducibility, we the random seed to 42 for all experiments.
To comprehensively evaluate the model's performance, we assessed its capabilities on both mul- timodal and text-only tasks. The model's multimodal abilities were benchmarked against a com- prehensive suite of benchmarks, including MMBench-EN (Liu et al., 2024a), MM-Star (Chen al., 2024b), OCRBench (Liu et al., 2024b),
(Li et al., 2024b), ScienceQA (Lu et al., 2022), A12D (Kembhavi al., 2016), and POPE (Li et al., 2023b). Furthermore, to gauge its core language understanding and commonsense reasoning skills, we evaluated its performance on the HellaSwag (Zellers et al., 2019) and MMLU (Hendrycks al., 2020) benchmarks.
F TRAINING DYNAMICS
To further probe the temporal evolution these dynamies, we tracked the layer-wise cosine simi- larity of image and text tokens across varying checkpoints during the multimodal pre-training phase (Figure 5a) and 5b). As shown in Figure 5a, without the additional normalization layer, visual to- kens exhibit persistently high inter-layer similarity. This pattern remains virtually static throughout the training process, confirming that the "representational inertia" induced by norm discrepancy is a persistent barrier, effectively locking visual features against semantic transformation from the very beginning. In contrast, our method (Figure 5b) initiates with a significantly lower visual similarity, indicating active feature updates. Interestingly, as training progresses, we observe gradual upward trend in similarity, which correlates with the optimization of the added LayerNorm' 's gain parameter (g). Crucially, as observed in the figure, this upward trend has converged. This phenomenon implies that perfectly consistent update rates between image and text tokens might not be optimal, sug- gesting that a certain degree of divergence between the two could be appropriate. However, this fundamentally distinct from the significant update inconsistency driven by the massive initial norm discrepancy observed in the baseline.
G APPENDIX: ATTENTION VISUALIZATION
In each pair heatmaps, the bottom image shows the model with norm applied, while the top image shows the baseline model. The caption for each pair corresponds to the text query used.
20

---

Published as a conference paper at ICLR 2026
Layer-wise Similanty Analysis Checkpoints (w/e Norm)
Similarity
17 Crttpion
(a) Evolution of layer-wise cosine similarity across different checkpoints during the multimodal pre-training phase (w/o Norm).
Layer-wise Similarity Analysis Across Chackpaints (w) Norm)
Image Token Similarity Evolution
Text Token Similarity Evolution
(b)
Figure 5: Evolution of layer-wise cosine similarity across different checkpoints during the multi- modal pre-training phase (w Norm).
21

---

Published as a conference paper at ICLR 2026
(a) Which mood does this image convey?
(b) What is the main subject of the flyer in the image?
(c) How many different pointedkinds are there?
(d) What type of family is shown in the image
22

---

Published as a conference paper at ICLR 2026
(a) What emotion portrayed in this image?
TUM
TOI
TOY
TOE
(b) How many people are performing on the stage?
(c) which image is more colorful?
(d) What is the main focus the image?
23


Reviews:
[{"review_id": "knUlBmqVXM", "reviewer": "Reviewer_uNHA", "summary": "This paper identifies a critical flaw in Pre-Norm MLLMs: a severe norm disparity between high-norm visual tokens and low-norm text tokens. The authors' theoretical analysis shows this imbalance creates an \"asymmetric update dynamic\". High-norm visual tokens exhibit \"representational inertia\", transforming semantically much slower than text tokens. This mismatch impairs cross-modal fusion and suppresses attention signals. The proposed solution is simple yet effective: inserting a single, carefully initialized LayerNorm after the visual projector to enforce norm alignment. This intervention yields significant performance gains on both multimodal benchmarks and, notably, on text-only evaluations like MMLU, suggesting a more holistically capable model.", "questions": "1. With sufficient training data, can the model learn to mitigate the \"norm discrepancy\" phenomenon on its own? \n2. For models like Ovis[1] that use a \"visual embedding table\", does a similar \"norm discrepancy\" phenomenon exist?\n3. In Table 2, the post-projector norms for KimiVL (4.78) and GLM-4.1V (4.58) are relatively close to the norm achieved by the author's method in Table 4 (2.2812). Does this imply these two models have already largely solved this problem with their more complex projectors?\n\n[1] Ovis: Structural Embedding Alignment for Multimodal Large Language Model", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents a novel perspective, using effective theoretical and empirical analysis (like inter-layer cosine similarity) to confirm the impact of \"norm discrepancy\" on the model's \"asymmetric update dynamics\".\n2. The solution (inserting a single LayerNorm) is simple, effective, and easy to apply, achieving significant gains on multimodal and even text-only benchmarks.", "weaknesses": "The solution lacks validation on more model combinations. The paper's experiments are primarily on the SigLIP + Llama-3.2-3B combination. To prove generalizability, the authors should test on more structures, such as the LLaVA-1.5 (CLIP + Vicuna) combination.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ArrkkOfNq", "reviewer": "Reviewer_Wqxf", "summary": "In this paper, the authors investigate a critical flaw in pre-normalized architectures within multimodal large language models (MLLMs): the significant disparity between high-norm visual tokens and low-norm textual tokens. This imbalance leads to a phenomenon referred to as “representational inertia,” where the semantics of high-norm visual tokens change at a considerably slower rate than those of text tokens, ultimately reducing the efficiency of cross-modal fusion between visual and textual features.", "questions": "While the proposed layer normalization solution effectively addresses norm differences between visual and textual tokens, how might this approach impact the model's ability to leverage subtle, nuanced variations within the visual or textual data that might be important for certain tasks, potentially leading to information loss or a homogenization of feature representations?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Identifies a Critical Issue: The paper highlights a significant and often overlooked issue regarding the norms of visual and textual tokens in multimodal large language models, shedding light on how this disparity can impair effective cross-modal integration.\n\nTheoretical Contributions: By introducing the concept of \"representational inertia,\" the authors provide a new theoretical framework for understanding the dynamics of feature representation in MLLMs. This contributes to the academic discourse on the challenges in multimodal learning.\n\nEmpirical Validation: The experiments conducted across multiple open-source multimodal models provide robust empirical support for the theoretical claims made, enhancing the credibility of the findings and their implications for model design.\n\nPractical Solutions: The proposed solution of adding a layer normalization layer to enforce norm alignment is practical and relatively easy to implement, making it accessible for researchers and practitioners aiming to improve their multimodal models.\n\nEnhanced Model Performance: The experimental results demonstrate significant improvements in model performance for both multimodal and text-only tasks, showcasing the effectiveness of the proposed intervention and its potential to advance the state of the art in multimodal understanding.", "weaknesses": "Lack of Broader Contextual Analysis: The paper primarily focuses on the norm differences without providing a comprehensive analysis of other factors that could affect multimodal performance, such as data quality or model architecture variations. This could limit a holistic understanding of the challenges in multimodal learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AxbId1SrpV", "reviewer": "Reviewer_XYFm", "summary": "This paper identifies and analyzes a subtle but consequential failure mode in Pre-Norm multimodal LLMs: a systematic L2-norm mismatch between high-norm visual tokens and low-norm text tokens induces an “asymmetric update dynamic,” in which visual tokens exhibit higher “representational inertia” and thus evolve semantically more slowly than text tokens. The authors formalize the effect under a geometric update model. They further argue that this suppresses the learnable attention signal in expectation and propose a minimal intervention: insert a single, carefully initialized LayerNorm after the visual projector to enforce norm alignment. On LLaVA-1.5 with a specific backbone and vision encoder, this yields consistent gains on multimodal benchmarks and text-only task MMLU, supporting the claim that resolving the architectural imbalance improves overall capability.", "questions": "1) Table 1 only reports the modality-interface L2 norms at initialization, but does not provide a post-training comparison. Was the norm disparity mitigated after training, especially in settings where the vision encoder is unfrozen?\n\n2) The caption of Table 1 states mean ± std, but the table reports only the mean values. The standard deviations are missing.\n\n3) Lines 307–309 assert that “multi-modal training increases the text embedding norm,” but Table 2 does not present experimental evidence to support this claim.\n\n4) Figure 2 visualizes the evolution rate of representations along depth. I suggest adding the similarity of one layer before vs. after parameter updates to show temporal representational change and test whether there is an asymmetry in the evolution speed of different modalities.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Mechanistic, interpretable theory-to-practice link.** The geometric update analysis connects norm magnitude to depth-wise angular dynamics and expected attention suppression, and the proposed diagnostic metrics and solution are well aligned with the theory.\n- **Minimal, low-risk intervention.** A single post-projector LayerNorm with principled initialization is simple to implement, does not alter the LLM backbone, and is easy to deploy or ablate in real systems.\n- **Clear diagnostic visualizations.** Attention maps and depth-wise metrics provide qualitative and quantitative evidence that norm alignment improves cross-modal focus and reduces modality asymmetry.", "weaknesses": "1. The theoretical model adopts strong simplifying assumptions—uniform update magnitude per layer, a layer-wise constant expected angle, symmetric distribution of orthogonal components—that are not empirically validated. Reporting empirical distributions of $C^{(l)}=\\|\\Delta h^{(l)}\\|_2$, $\\cos\\phi=\\langle \\widehat h,\\widehat{\\Delta h}\\rangle$, and the orthogonal/parallel energy split would substantiate the premises.\n\n2. The paper’s theoretical narrative emphasizes that an L2 norm mismatch leads to “asymmetric angular velocity.” If this is the causal core, then RMS-style normalization—or even a simple fixed scalar rescaling to bring visual-token norms to the same scale as text—should in principle deliver most of the gains. Conversely, if significant improvements arise only from the current solution, namely LayerNorm with a learnable gain and specific initialization, that would indicate mechanisms beyond mere norm magnitude are at play. The paper lacks corresponding experiments and discussion.\n\n3. Statistical robustness and scope are limited. Results appear to rely on a single seed, one backbone size, and one vision encoder, with no confidence intervals or significance tests. \n\n4. Some language overstates the effect. Phrasings like “eliminated” asymmetry are not fully supported by Fig. 3b, where a residual gap remains. The authors need to quantify the mean/max gap and the relative reduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dHcsuVjkQ4", "reviewer": "Reviewer_hK3Y", "summary": "The paper diagnoses a modality-scale mismatch in pre-norm MLLMs: vision tokens enter the LLM with much larger L2 norms than text tokens, inducing asymmetric update dynamics (“representational inertia” for vision) and consequently miscalibrated cross-modal attention. The fix is simple: insert a single LayerNorm after the visual projector, with its gain closed-form initialized to match the average text-embedding norm of the target LLM. The authors support the mechanism with theory and internal diagnostics (layer-wise norms and inter-layer cosine similarity), then show empirical gains in LLaVA-1.5 (multimodal benchmarks and even text-only MMLU). Post-intervention, the model displays aligned layer-wise norms and balanced update rates between modalities, consistent with the proposed explanation.", "questions": "- The paper can be stronger if address the empirical questions in the weakness section.\n- Any benchmark that the solution does not provide a performance gain?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- **Clarity and logically sound.** The paper is well written: key concepts (norm disparity, asymmetric update dynamics) are introduced with clear intuition, then formalized without losing readability. The work tightly connects diagnosis → theory → solution → gains → diagnostic analysis, making the mechanism logically sound.\n- **Simplicity of the fix.** The proposed solution—one LayerNorm after the visual projector with a closed-form gain init—is easy to implement, and compatible with existing MLLM stacks. The method adds negligible overhead, requires no data or architecture burden.\n- **Strong theoretical support.** The theoretical part (update geometry / angular velocity view) is well motivated and aligns with observed behaviors.", "weaknesses": "- **Limited model coverage.** Results are centered on the LLaVA-1.5 stack; the claim would be stronger with evaluations across more backbones/architectures (e.g., Qwen-VL, InternVL, Idefics2, GLM-4V) and different visual encoders (CLIP/SigLIP).\n- **Missing normalization ablations.** The paper focuses on LayerNorm with a specific init. Adding RMSNorm vs LayerNorm can be a valid ablation.\n- **Scale experiments.** The work can further show effectiveness if validate on larger data and larger LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EtS7jNH4NX", "reviewer": "Reviewer_K4C3", "summary": "**The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss** observes a critical problem in VLMs where visual tokens have much larger vector norms than text tokens, causing a large inertia (reducing angular velocity) for visual parameter updating. The authors use a simple but elegant method to rescale visual tokens, and found satisfactory improvements in experiments with LLaVA-1.5. Generally speaking, this paper makes valuable observations, and the claims and findings are well-supported by its experiments.", "questions": "Can this method also be applied to modern models like Qwen2.5-VL? In table 2, the same visual token magnitude problem is shown on Qwen2.5-VL, so the paper could be strengthened with an experiment on Qwen2.5-VL, to verify that this is a fundamental problem and a generalizable solution.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. This paper is novel in observation. The observation that visual features differ in magnitude with text features directly questions the widely accepted assumption that connecters align visual and text modalities. This is valuable itself. However, I am not an expert in VLM representations, and I am not sure whether this is the first paper proposing this idea. So I will look to other reviewers and may change this comment.\n2. The use of simple layer-norm to address this is clear and generalizable. \n3. Appendix D is informative. It shows that with paper's method, image features can align better with text features, which supports their claim.", "weaknesses": "1. The experiment is done on a rather old model. LLaVA-1.5 is not competitive in today's VLMs, so this might weaken the paper's claim.\n2. The paper is written in a hurry, and the subcaptions for Figure 3 are just placeholders. A typo ' in the title.\n3. The captions of the other figures can be more informative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)