Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding
Authors: name: Lin Long, name: Changdae Oh, name: Seongheon Park, name: Yixuan Li
Organization: University of Wisconsin-Madison

Large vision-language models (LVLMs) achieve strong performance on multimodal
tasks, yet they often default to their language prior (LP) -- memorized textual
patterns from pre-training while under-utilizing visual evidence. Prior
analyses of LP mostly rely on input-output probing, which fails to reveal the
internal mechanisms governing when and how vision influences model behavior. To
address this gap, we present the first systematic analysis of language prior
through the lens of chain-of-embedding, which examines the layer-wise
representation dynamics within LVLMs. Our analysis reveals a universal
phenomenon: each model exhibits a Visual Integration Point (VIP), a critical
layer at which visual information begins to meaningfully reshape hidden
representations and influence decoding. Building on this observation, we
introduce the Total Visual Integration (TVI) estimator, which aggregates
representation distance beyond the VIP to quantify how strongly visual query
influences response generation. Across 54 model-dataset combinations spanning 9
contemporary LVLMs and 6 benchmarks, we demonstrate that VIP consistently
emerges, and that TVI reliably predicts the strength of language prior. This
offers a principled toolkit for diagnosing and understanding language prior in
LVLMs.