Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval
Authors: Constantin Venhoff
Ashkan Khakzar
Sonia Joseph
Philip Torr
Neel Nanda
Venue: NeurIPS 2025 poster ()
Keywords: Mechanistic Interpretability
Vision Language Models
Multimodal Alignment
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: VLMs often perform worse at recalling facts than their LLM backbones because visual representations are formed too late in the forward pass to trigger the LLMs factual recall circuit.

Abstract: Training vision language models (VLMs) aims to align visual representations from a vision encoder with the textual representations of a pretrained large language model (LLM). However, many VLMs exhibit reduced factual recall performance compared to their LLM backbones, raising the question of how effective multimodal fine-tuning is at extending existing mechanisms within the LLM to visual inputs. We argue that factual recall based on visual inputs requires VLMs to solve a two-hop problem: (1) forming entity representations from visual inputs, and (2) recalling associated factual knowledge based on these entity representations. By benchmarking 14 VLMs with various architectures (LLaVA, Native, Cross-Attention), sizes (7B-124B parameters), and training setups on factual recall tasks against their original LLM backbone models, we find that 11 of 14 models exhibit factual recall degradation. We select three models exhibiting high- and two models with low performance degradation, and use attribution patching, activation patching, and probing to show that degraded VLMs struggle to use the existing factual recall circuit of their LLM backbone, because they resolve the first hop too late in the computation. In contrast, high-performing VLMs resolve entity representations early enough to reuse the existing factual recall mechanism. Finally, we demonstrate two methods to recover performance: patching entity representations from the LLM backbone into the VLM, and prompting with chain-of-thought reasoning. Our results highlight that the speed of early entity resolution critically determines how effective VLMs are in using preexisting LLM mechanisms. More broadly, our work illustrates how mechanistic analysis can explain and unveil systematic failures in multimodal alignment.

Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval
Constantin Venhoff University of Oxford
Ashkan Khakzar
Sonia Joseph
Philip Torr
Neel Nanda
McGiil University
Meta
*ATS
cvenhofflylm-twoohhp
Abstract
Training vision language models (VLMs) aims to align visual representations from vision encoder with the textual representations of pretrained large language model (LLM). However, many VLMs exhibit reduced factual recall performance compared to their LLM backbones, raising the question of how effective multi- modal fine-tuning is at extending existing mechanisms within the LLM to visual inputs. We argue that factual recall based visual inputs requires VLMs to solve a two-hop problem: (1) forming entity representations from visual inputs, and (2) recalling associated factual knowledge based these entity representations. By benchmarking 14 VLMs with various architectures (LLaVA, Native, Cross- Attention), sizes (7B-124B parameters), and training setups factual recall tasks against their original LLM backbone models, we find that 11 of 14 models exhibit factual recall degradation. We select three models exhibiting high- and two models with low performance degradation, and use attribution patching, activation patching, and probing to show that degraded VLMs struggle to use the existing factual recall circuit of their LLM backbone, because they resolve the first hop too late in the computation. In contrast, nigh-performing VLMs resolve entity representations early enough to reuse the existing factual recall mechanism. Finally, we demon- strate two methods to recover performance: patching entity representations from the LLM backbone into the VLM, and prompting with chain-of chain-of-thought reasoning. Our results highlight that the speed of early entity resolution critically determines how effective VLMs are in using preexisting LLM mechanisms. More broadly, our work illustrates how mechanistic analysis can explain and unveil systematic failures in multimodal alignment.
Introduction
Vision-Language Models (VLMs) achieve strong multimodal task performance by integrating vision transformers (ViTs) with large language models (LLMs) via adapter mechanisms Liu al. (2023]
Lin et al. [2024], Steiner et al. [2024), Liu et al. [2024a], Cocchi et al. [2025), Touvron et al. 2023 These adapters project visual representations from the ViT into the representation space of the backbone LLM, enabling text-based reasoning over visual inputs. However, how exactly preexisting
mechanisms in the backbone L.LM adapt to visual inputs, and the resulting implications or failure modes, remain severely understudied.
For example, previous studies have found that LLaVA-1.5-7 performs worse on factual recall tasks than its LLM backbone Cohen et al. [2024]. Explaining this requires understanding how visual information flows through VLMs and how factual recall mechanisms function in their LLM backbones. Prior work on factual recall mechanisms identified early-layer MLPs as crucial component that
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Vision Language Model
Language Model
Spain
Argentina
Factual Recall
Giadiator, 1786, Spain.
Middie Layer MLPs
Entity Recognition
Perito Moreno Glacler
Santa Cruz Province, Argentina, Andes.
Factual Recall
Early layer MLPs
Distributed Visual
No Subject
Information
Tokens
o000
O
VIT
which country entity located?
which country
the is
Perito Moreno Glacier located?
Figure I: Factual Recall in VLMS vs. LLMs Illustration of the Two-Hop Problem. This figure compares how Vision Language Model (VLM, left) and a text-only Language Model (LL.M, right) perform factual recall task. The VL. receives image of the Perito Moreno Glacier and question: "In which country the entity located?" The image is processed by a Vision Transformer (VIT), producing distributed visual embeddings that do not align with the LLM's pretrained token space. The entity ("Perito Moreno Glacier") is only recognized the middle layers, bypassing the early-layer MLPs responsible for factual recall and resulting in incorrect answer. In contrast, the LLM given the full question, with the subject tokens "Perito Moreno Glacier" available. This enables early-layer MLPs to access factual knowledge ("Argentina") and produce the correct answer. The comparison highlights the core issue: VLMs must first infer the subject before retrieving facts, but because recognition occurs late, it cannot engage early factual recall mechanisms. This "two-hop" problem leads to degraded factual accuracy in VLMs, even when visual recognition succeeds.
reads in subject token representations and then produces subject-related factual representations Meng al. [002), Chughtai al. 2024). Separate work on VLM information flow, however, shows that visual projections not aligned with the backbone LLM's token space and instead only gradually
align with textual representations deeper in the LLM Venhoff al. 20255) Wu et al. [2025] Masry
al. [2025].
Combining these observations, we hypothesize that VLMs fail to adapt to the preexisting factual recall circuit of their LLM backbone because visual representations in early layers are misaligned with the backbone LLM's s token space and therefore "skip over" early-layer MLPs. Factual recall in VLMs can therefore be viewed two-hop problem: the VLM must first construct robust entity representations from visual inputs before it can access the LLM's preexisting factual knowledge.
To investigate this, we benchmark 14 VLMSs across various architectures (LLaVA, Native, Cross- Attention), model sizes (7B-124B parameters), and training setups on factual recall tasks, comparing their performance to their original LLM backbone models. We find that 11 out of 14 models exhibit factual recall degradation. We then conduct comparative analysis of well- and coorly-performing VLMs using attribution patching, activation patching, and linear probing. We find that VLMs with degraded performance rely on different sublayers for factual recall than their LLM backbones, while high-perforning VLMs reuse the same components. In degraded VLMs, visual entity representations emerge too late in the backbone L.M''s forward pass, bypassing early-layer ML.Ps of the factual recall mechanism, whereas well-aligned models exhibit early resolution of such representations. By patching early-layer MI.P outputs from the backbone I.L.Ss into degraded VLMS, we can recover factual recall performance, providing strong causal evidence for our hypothesis. Finally, we explore

---

Table 1: Factual-recall accuracy of VLMs their text-only LLM backbones on 1000 questions.
VLM Model
LLM Model
Architecture
LLM (%)
VLM (%)
A (%)
LLaVA-MORE-8B
-lama-3.1-8B-11
Adapter
41
23
43,9
LLaVA-NEXT-SB
Llama-3-8B-it
Adapter
41
41.5
LLaVA-1.5-7B
Llama-2-7BB-hhe
Adapter
30
19
366.7
LLaVA-1.5-13B
Llama-2 Muma-2-13B-chanr
Adapter
44
36.44
Pixtral Pixtral-Large-
Mistral- 2arge-2
Adapter
68
56
17.6
Pixtral-12B
Mistral-NeMo
Adapter
41
36
12.2
Qwen2.5-VL-7B-it
Qwen2.5-7B
Adapter
30
28
6.7.
Qwen2.5-VL-72B-it
Qwen2.5-72B-it
Adapter
49
53
-
Jlama-4-Maveriee
Native
75
71
J.333
Gemini-2.0-Flaat
Native
66
63
44.5
Demma-3-27B-.
Native
46
46
0.0
Gemma-3-12B-it
Native
40
0.0
GPT-4o
Native
77
73
4.5.2
Llama-3.2-Vision--11 Llama-3.1-8Bii
Cross-attn
36
31
13.9
thain-of-though prompting as a mitigation strategy, finding varying but promising improvements, particularly for larger VLMs.
Our findings highlight that successful multimodal alignment requires more than representational compatibility: depends on integrating visual information into the functional circuits the LLM backbone.
2 Benchmarking Factual Recall in Vision Language Models
To systematically evaluate factual-recall degradation vasion-language models (VLMs), we intro- duce benchmark that directly compares each VLM with its original LLM backbone model. By feeding equivalent information to both systems, the benchmark isolates factual-reest ability from other confounding factors.
2.1 Benchmark Design
Our benchmark comprises 15000 multimodal factual-recall questions. We sample images from the Wikipedia based Image-Tex: (WIT) dataset (Srinivasan et al. 2021] and use GPT-4.1 to generate eptiiy-specific factual prompts (e.g. "Who invented the entity shown in the image?"). GPT-4.1 was chosen for its strong factual accuracy on the WildHallucinations benchmark (Zhao et al. 2024]. Ap- pendix B details the genareeeeettion pipeline, including entity verification and question construction.
During evaluation, a VLM first asked to identify the main entity each image. If the entity is misidentified, the sample discarded to avoid conflating recognition errors with factual retrieval. We conduct a brief case study into rejected examples in Appendix G to ensure there is no structural bias in the images discarded. For correctly identified entities, we compare the VLM's answer with that of its language-only backbone, ensuring a controlled, like-for-like comparison (for prompts see Appendix[F Fp.
We test broad range architectures, including Adapter-based VLMs, where a pretrained LLM is connected to vision transformer (ViT) through lightweight adapters, Native VLMMs, which are trained end-to-end with multimodal data from the beginning, and Cross-Attention VLMs where the L.LM backbone cross-attention blocks to attend to representations of the VIT. Each VLM/backbooee pair answers factual recall questions until 1000 valid samples are answered from the 15000-question pool.
2.2 Benchmark Results
Table [ summarizzes the results: out of VLMMs exhibit lower factual-recall accuracy than their language-only backbones. Adapter-based and Cross-Ateemtion-modde. suffer the steepest drop.

---

Attribution LLMs
1..-A-3.1188
LLaMA-2-13B
LLaMA-2-78
Qwen2.5-78
Gemma-3-1228
Attibation Scome
0.8
Attibution Scom
Attibation Score
OOESTIOO
0A
0.6
2.6
Mtrlationn
0.53
0.
0.2
Toom
8
15
16
24
15
24
15
24
15 24:22 49 Layor
Layer
Layer
TENTIT]]
0.4
Atribulie Score
0.5
Altrbuutt Score
Attention
0.6
244
QUESTION
Alsibution
Attibalion
0.4
22
0.1
Toom
8
15 24
0.0
16 24
0.0
3.5 Layer
24
5.0
1.5
24
16 24 40
Layer
Layer
Layer
Layer
Figure 2: Attribution scores of each MLP and Attention sublayer for the original LLM backbone models. Higher values indicate higher causal relevance for factual recall.
Attribution
LLaVA-1.5-13B
LLaVA-1.5-78
Qwen2.55VL-78
Gemma-3-128
[PAAGE Imran)
SCOre
Almbation Scon
Atriaatinn S.o0e
Aertbuston Sc00o
0.75
[QUESTION]
2.2
0.2
0.25
Lact Thden
16
32
36
24
16
Layer
Layerr
Layer
Layer
Layyer
[IMAGE] (max)
0.8
0.8
Attribttio 540re
Antibuttoo Sonte
Adributton Score
Attention
0.4
0.6
JOUESTION]
0.4
1.0
0.2
Last Tiken
Figure 3: Attribution scores each MLP and Attention sublayer for VLM models. Higher values indicate higher causal relevance for factual recall.
Degradation persists even for very large models such as Pixtral-Large- 124B. In general, Adapter-based models degrade more than Native VLMSs, with the notable exception of Qwen2.5-VI. We hypothesize that 2-sLL extensive multimodal fine-tuning on over 4 trillion tokens mitigates factual recall degradation, which may also explain why the Qwen2.5-VL-72B-it variant slightly outperforms its backbone.
In summary, almost all tested VLMs show pronounced factual-recall degradation, especially stapter-based VLMs, while Native models such as Llama-4-Maverick and GPT-4o degrade less. Qwen2.5-VL3 success suggests that massive multimodal fine-tuning can enable the VLM to utilize the factual knowledge of its backbone LLM.
2.3 Selecting Models for Comparative Analysis
In the following sections we describe several experiments to understand the drivers of factual recall degradation in VLMs. We select three Adapter-basee models with very high degradation, LLa VA- MORE-8B (Cocchi et al. 2025). LLaVA-1.3 1.5--B [Liu et al. 2023 and LLLVA-1.55-3BB [Liu et al. Additionally, we select two models with very low degradation, Gemma-3-12B-4 (Team et al.
2023
2025 to study Native models, and Qwwn225-VIL-7BBit (Bai et al. 2025 understand the impact of massive multimodal fine-tuning Adapter-basee based models.

---

3 Comparing Factual Recall Mechanisms
In this section aim to find the sublayers the original LLM backbone model uses to perform factual recall, and compare them against the sublayers that the respective VLM model uses.
3.1 Attribution Patching
To determine which MLP- and Attention-sublayerr in the LLM and VLM contribute most to factual recall, we employ an attribution patching methodology inspired by Nanda and Meng et al. First we sample 100 correctly answered examples from the benchmark dataset for each VLM and their original LLM backbone model. Then we use the following steps compute the attribution scores:
Targeted corruption: Let S be the token span necessary to identify the entity (LLM: entity text tokens; VLM: image tokens). Compute embed as the standard deviation of input embeddings over S across the examples of the attribution dataset. Perform a corrupted run, by injecting Gaussian noise in the embeddings across S:
H + E, 8 ~N, a (emeeee)
where a a noise multiplier. We ablate a for each model over the values {1, ......... 9, 10} and measure the corruption effect via the KL divergence between the clean predicted token distribution, and the corrupted predicted token distribution. More details on how we choose a can be found in Appendix D
2. Clean/corrupted passes and KI. objective: Run the model without the corruption (clean) and with the corruption (corrupted) to obtain next-token prediction logits (Zelean Rcorr) and define
L (=x(se)tmaxxxatee)) soottmax(2oor
3. Per-layer branch activations and gradients: For each layer é and branch E {mlp, attn). cache clean and corrupted outputs Helean Hy and backpropagate to obtain
0L
Ge, =
HH
4. GradxDelta attribution and aggregation: For token t and hidden dimension d. (t)
Σ de (Hr (t, d) - H (t, d))
Figure 2 shows the attribution scores for the LLMs and Figure 3 shows the attribution scores for the VLMs averaged over the 100 correct factual recall examples from the benchmark dataset, aggregated across different token positions (entity/image, question, and last token positions).
3.2 Attribution Patching Results
We find the same two sites across all LLMs tested: the early-layer MLP- and Attention Attention-sublayer across the entity tokens, and the middle-to-late layer MLP- and Attention-sublayers across the last token positions. This is consistent with findings from Meng et al. 2022]. For all LLaVA-style VLMs we find that there is only the mid-to-late layer site across the last token position active, indicating that the model is not able to use the same pathway as the LLM backbone. For Gemma-3- 12B and Qwen2.5-Ve 5-VL-7B however, whether the entity is given in token space, or as an image.
find the same two sites, showing that the model uses similar pathway,
Causal Analysis via Activation Patching
In Section 3
we found that for the three LLaVA models, the LLM backbone uses early-layer MLPs the entity token positions to perform factual recall, while the LLaVA models that are trained on top of these LLMs are not able to do that. We hypothesize therefore, that these early-layer MLPs are key breaking point in the factual recall mechanism the LLaVA models. To test this, we patch MLP outputs from the LLM backbones of the three LLaVA models into their forward pass to see whether this yields relevant performance recovery, which would support that the L.LVV VA models are not able to utilize the factual recall circuit of their LLM backbone properly.
5

---

Heuristic vs Random vs Back-Patch
0.5
Heuristic Patch
Random
0.44
BackkPPatch
0.4
0.30
0.30
Patched VLM
0.24
0.2
0.18
0.14
0.1
0.0
LLavA 1.5 138
LLaVA 1.5
LLaVA-MORE 8B
Figure 4: Factual recall performance recovery for LLaVA-1.5-7B. LLaVA-1.5-13B, and LLaVA- MORE models when MLP outputs across entity tokens from corresponding LLM backbones patched into early VLM layers. We test only on examples where the VLM was originally wrong, so the y-axis directly shows the recovered performance gap. We compare our heuristic patching approach against random baseline (randomly selecting patching positions), and back-patching
4.1 Heuristic Patching
We propose heuristic patching to determine which token position in the VLM we patch the backbone LLM's s MLP outputs (as the VLM doesn't 't have corresponding entity token positions). Heuristic patching uses steps:
1. Caching MIP outputs:
We define a of source layers L. For each layer e C, cache the LLM's MIP outputs over the entity tokens.
2. Define patching objective: We define an objective, that quantifies whether patching the cached LLM MLP outputs into given VLM token span, from given start position p. increases the likelihood of the VLM to change its generated answer to the LLM answer. To do this, we obtain clean next-token predictions from both models. If the next-tokens predictions of the LLM and VIM differ, we use a logit-difference objective that increases the L...-chosen token and decreases the VLMM-cosen token. If they match, objective over the LLM's ss top 10 tokens, renormalized. We treat both as a signed objective that higher better; we do not take absolute values.
KI.
3. Position-wise attribution: For every candidate token start position in the VLM, compute an attribution score by backpropagating the chosen objective the VLM MLP outputs and taking Grad A, where A is the activation difference between LLM MLP activations and the current MLP activations across the VLM slice with start positions p. Average the scores across the slice token positions, keep the sign, and sum the scores across all layers in L for each token start position p to obtain single scores sp)).
4. Selection and patching: Choose the top start position with the highest s(p). We patch all layers in L from the selected start token position during VLM generation and record whether the VLM now predicts the correct answer.
The results of heuristic patching are hard to interpret in isolation, as 1) LLaVA models are fine-tuned end-to-end, hence the LLM backbone in the LLaVA model is not the same the original LLM backbone model, and 2) there is no designated token position to patch into. Therefore it is unclear what accuracy expect consider good. Thus, compare against (i) Random patching, which uses the same patching rule as heuristic patching, but chooses the start token positions uniformly at random, and (ii) Back-patching. two-pass procedure both used in prior work to address two-hop problems LLMs [Biran et al. (2024) and VL.Ms [Nikankin et al.) (2025). that copies entire layer outputs across the image-tokens from set of source layers S a of (carlier) destination layers D. Concretely, we use:
Baselines.
-LRVAA-MORE (LLaMA-3.. 3.1 1-8B backbone): S=)6.. D=

---

ImageNet Probe Accuracies
1LaVA-155
1LaVA-1.5
HaVA-MORE
Qwen2,5NL.
Comma-3 (128)
1.0
1.0
1.:
1.00
0.
0.0
0.8
0.0
0.8
Pobe
Layer 07
16
24
8
12 Layee
0 6
12
Layez
Lagar
Layee
Figure 5: Accuracy of linear probes trained on residual-stream sep.ssaaatttttaaeppppppeeettnttt at each transformer layer of LLaVA-1.5-7B, LLaVA-115--311 LLaVA-MORE, Gemma-3-12B, and Qwen2.5-VL-77B measured on ImageNet-100 entity prediction. The three LtaVA-style models exhibit consistent pattern: probe accuracy remains poor in early layers and rises sharply between middle-to-la layers. Gemma-3-12B and Owen2.5-VL-7B on the other hand show consistently high probe accuracies.
-LLVV--.5--B (LLaMA-2-7B backbone): S-[6..16], D=[L..11].
LLLVAA-L.--3BB (LLaMA-2-13B backbone): S=[277.37], D=113..233
We select for each VLM/LLM pair set of 100 factual recall questions from our dataset, which were incorrectly answered by the VLM and correctly answered by the corresponding LLM. Then we run heuristic patching with different layer choices (see Appendix for details), and compare against the two baseline methods.
4.2 Patching Results
Results shown in Figure 4 recall accuracy difference between the VLM and its language-only counterpart, while back- patching only recovers 13%, and random patching recovers 16%. We therefore show that the early-layer MLP outputs used in the LLM's factual recall circuit are indeed a core missing piece in the degraded factual recall cireuit used by the VLaVttst VLMs.
We find that average heuristic patching recovers 35% of the factual
Probing for the Emergence of Visual Entity Representations
Our previous experiments demonstrate that staVe VLMs show degraded factual recall perfor- mance, because they are not able utilize the factual recall circuit of their LLM backbone correctly. The Native model Gemma-3-12B and the Adapter-based model Qwen2.5-VL.-3 VLL trained billions of image-text pair tokens, on the other hand, are seemingly able to use the same circuit, whether they prompted with image of entity the entity in text tokens.
know already from prior work that the visual representations passed to the LLM backbone in LLaVA-style models are not aligned with the token space (reproduced in Appendix C There- fore, in order to use the same circuit as the LLM backbone, VLMs need to produce robust entity representations in early layers of the LLM backbone. We hypothesize that the reason for the de- graded performance of the VLaVy--tyle models is that entity representations emerge too late in the forward pass, compared to Gemma-3- 3-10--3------ 1-B and Qwen2.5-VL-77 VL-7B which might be able produce such representations much earlier due to native pretraining and massive multimodal fine-tuning respectively.
5.1 Linear Probing for Visual Entity Representations
To systematically assess where visual entity representations emerge within the L.M backbone of VLMs, we employ probing methodology. Specifically, we train layer-wise linear classifiers (probes) on the residual stream outputs of each transformer layer to predict visual entities depicted in input images.
Experimental Setup:
We use subset of ImageNet- 100 [Deng al. 2009 restricted to 50 of the 100 classes, as our evaluation dataset, due to its controlled set of classes and multiple images per entity, ensuring reliable estimation of representational capacity. We randomly select 2500 images and

---

Table 2: dictual-recall accuracy using Chain-of-Thought (CoT) prompting. Green arrows denote
improvement compared to the base (non-CoT) runs.
VLM Model
LLM Model
Architecture
LLM (CoT,, %)
VLM (CoT, %)
LLaVA-MORE-SB
Llama-3.1
Adapter
461
+8.
LLaVA-NEXT-8B
Llama-3-8B-it
Adapter
451
301+6.0
LLaVA-1.5-7B
Llama-2-7 2-h-chhtt
Adapter
351
18 18
LLaVA-1.5-13B
Llama-2-133 2--3--1h1BBa
Adapter
451
+7.
Pixtral-Large 124B
Margee
Adapter
721+4.0
711+15.0
Pixtral- 12B
Mistral-NeMa
Adapter
45 +4.0
48 +12.0
Qwen2.5-VL-77Bit
Adapter
+5.0
431+15.0
Qwen2.5-VL-72B-it
Qwen2.5-72B-it
Adapter
52 $21+3.. +3.0
531+
Llama Alama-4-Maveen 4-Maverick
Native
791+4.0
771
Demini--2.0-Flan
Native
70 +4.0
70
Demma-3-27BB
Native
13.+7.0
571+11.0
Gemma-3-12B-it
Native
+1.+
46.0
GPT-4o
Native
791
791+6.0
Ilama-3.22 --iionn--
Llama-3.1-8B-i
Cross-attn
421 +6.0
40
generate for each image a factual recall question, using a similar methodology as in Section We provide GPT-4.1 with the entity class, and prompt for concise factual recall question. We keep track of the last few generated questions and prompts to produce a different type of question, to ensure sufficient question diversity. At each transformer layer, train independent linear probe the extracted and averaged residual-stream representations over the question token positions, to predict the correct entity label. We use 20%/80% train-test split to evaluate the probes.
Results and Analysis: Figure depicts the accuracy the linear probes across layers for LLaVA- 1.5-7B, LLaVA-1 -.VA-1.5. 1.5-13B, LLaVA-MORE, Gemma-3-12B, and Qwen2.5-VL-7B The results reveal clear trend for the LLaVA-style models: linear representations capable of reliably encoding visual entity information do not emerge until the middle-to-late layers. Prior to these middle-to-late layers, probe accuracy remains poor, indicating that early layers do not encode robust entity representations. Gemma-3-122 1-BB and Qwen2.5-VL.
on the other hand show consistently high probe accuracies across all layers (with small variance), indicating that these models are capable of producing robust entity representations from visual inputs in the LLM backbone immediately.
These findings reveal the core challenge the two-hop problem multimodal knowledge retrieval: the model must first form robust entity representation (first hop) before being able to access the latent knowledge of the LLM backbone encoded in early-layer MLPs (second hop). However, since entity representations only emerge in deeper layers in VA-Sttle models, they bypass the early-layer MLPs, causing factual recall degradation.
Reasoning as Potential Mitigation
Previous experiments have shown that the LLaVA-style VLMs are fundamentally constrained in accessing the factual knowledge their LLM backbone. The only mitigation seems either native pretraining, or massive multimodal fine-tuning, which both require orders of magnitude more compute and data than VLaVA-style models. Therefore, we experimented with another, simpler, mitigation strategy. The VLMs could use inference- time compute, to describe the visual entities relevant for factual recall in text space, and then answer the question. In this setup, the VLM might be able to access the factual recall knowledge of its LLM backbone again, as it produces token representations related visual entities and therefore doesn' 't rely on pure visual representations anymore.

---

6.1 VLM Chain of Thought Prompting
Experimental Setup: To test this, we alter the prompting template in the benchmark and include chain of thought prompt (see Appendix for prompt). Then we re-run the benchmark on the same data for each model and record the increase factual recall accuracy.
Results and Analysis: Table 2 shows the chain of thought benchmark accuracies for each model, alongside the increase aceuracy compared the original accuracies without chain of thought. We find that for most models the chain of thought leads to larger increase for the VLM compared to the LLM backbones, supporting our hypotheses. For Pixtral- 12B and Larial--ar---124 we see that performance gap is fully closed, while for the LLaVA-style models we see an increasing gain with the model size, with performance gap to begin with, closing over half its performance gap. Albeit having a smaller that (emini-2.00-Flase GPT-4o, and LLama-4-Maverick close their respective performance gaps when using chain of thought prompting.
In summary, the effect of chain of thought prompting seems to be very model dependent (e.g. LLaVA-1.5-7B even loses percentage of accuracy, as chain of thought prompting often leads to less structured responses), however models with substantial reasoning capabilities are able to close significant fraction or even the full performance gap. We therefore think that reasoning a promising approach for future research to improve factual recall performance of VLMs for which only limited data and compute resources are available.
Related Work
We review prior research on Vision Language Models (VLMs) and their handling of visual and textual representations. We first cover methods for aligning visual features with language embeddings and their effect multimodal reasoning. We then summarize findings factual recall in unimodal and multimodal models, highlighting evidence of recall degradation in VLMs.
7.1 Vision Language Models
VLMs integrate visual and textual information by mapping both modalities into shared embedding space processed by a pretrained LLM Liu et af. [2023]. A key challenge is ensuring that visual features retain critical information (e-g., entities) when projected into the token space Masry et al.)
[2025]. The standard approach uses an image encoder and a projector (typically an MLP) to map visual data into the LLM's embedding space Liu al. [2023] 2024b), Chen et al. [2024). However, this mapping often fails to achieve meaningful alignment, limiting VLM performance on multimodal tasks Masry al. [025]] Lin al. 2024]. To improve this, several works propose alternatives that refine the mapping process or introduce pross-modal layers deeper in the network Alaayra et al.
[2022), Yan et al. [2024, Li et al. [2024]. Upadhyay et al. [2023). While these methods enhance interaction between modalities, they add significant computational cost.
7.2 Multimodal Mechanistic Interpretability
Recent interpretability research has explored vision language models through various approaches. Chen et al. [003) and Gande sman et al. [023) analyzed the vision encoder CLIP, identifying visual features Schwettmann al. [2023) examined multimodal neurons in transformer models that were solely trained on text data. Diang et al. 20244] investigated VLM responses to hallucinated versus real objects, and showed how to edit their internal representations to mitigate hallucinations. Work by Neo et al. (2244] investigated information flow within VLMs by projecting visual representations from intermediate layers onto language vocabulary, identifying gradual alignment with text vocabulary. Naghashyar al. (2025) use sparse autoencoders show how visual features emerge during multimodal fine-tuning in LLaVA.
7.3 Factual Recall LLMSs and VLMs
LLMSs implement important mechanisms of factual recall in early-layer ML.Ps Chughtai et al. 2024
Meng et al. [2022). Geva al. [2023. This layer localization has been linked to their struggles les with multi-hop reasoning, which requires resolving intermediate entities before answering the full query

---

Biran et al. [2024), Sakarvadia et al. [2023], Yang et al. [2024]. For example, answering "Who is the mother the inventor of transistors?" requires first identifying the inventor, [Biran et al. [2024] suggests early layers resolve these "bridge entities." leaving later layers with insufficiet capacity for the follow-up reasoning Chughtai et al. [2024]. [Cohen et al. 2024] reports that 1A-1VA---.-77 struggles with factual recall when processing images. Their patching experiments reveal that visual tokens stop influencing model behavior beyond middle layers, implying that visual processing finishes before later layers handle factual reasoning. They hypothesize that factual recall failures occur due to insufficient remaining layers after visual processing.
8
Discussion and Conclusion
Despite the growing capabilities of Vision Language Models (VLMs), their ability to reliably ground outputs in factual knowledge remains brittle. This is especially concerning in high-stakes applications, where factual hallucination carries real-world risks. In this work, we identify structural failure mode in VLaVt-style VLMs: systematic degradation of factual recall caused by insufficient early layer alignment of textual- and visual representations in the LLM backbone of the VLM.
Our findings show that such misalignment causes VLMs to use different the original LLM backbone model to perform factual recall. This restricts them from accessing the full factual knowledge of their backbone LLM. We confirm this hypothesis using attribution patching, activation patching and probing experiments. We show that Native VIMMs and adapter- based VLMs with massive multimodal fine-tuning do not exhibit the same degradation, as they are able to form consistent entity representations in early layers of the LLM backbone, regardless of the input modality.
than
These findings expose fundamental limitations in current LLaVA-style VLMs that are trained with limited data and compute resources. We conduct promising early experiments with chain thought prompting to utilize inference-time compute to mitigate the problem, suggesting potential avenue for future research. Additionally, future work should look into better, more data efficient, adapter mechanisms and multimodal alignment techniques to integrate visual representation earlier into the LLM's embedding space. While our study focuses on factual recall, future work should examine other tasks, comparing VLMS against their LLM backbone.
Overall, this study provides strong empirical evidence that representation misalignment limits VLMs to re-use the existing task circuits in their LLM backbone models. Further investigating and mitigating this issue is crucial for improving multimodal reasoning.
Contributions
Constantin Venhoff conceived and led the project, designed the methodology, and conducted all ex- periments. Ashkan Khakzar and Sonia Joseph provided in-depth feedback throughout and contributed the manuscript. Philip Torr and Neel Nanda advised on the research and provided high-level feedback.
Acknowledgements
This work was carried out as part of the MI. Alignment & Theory Scholars (MATS) program. We thank Josh Engels, Julian Minder, Clément Dumas, Iván Areuschin, Nick Jiang and Sharan Maiya for helpful comments, discussions and feedback.
References
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre- me-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (1689922699,,9811 2024.
10

---

Andreas Steiner, André Susano Pinto. Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer. Anthony Sherbondy. Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdulmohsin, Lucas Beyer, and Xiaohua Zhai. Paligemma 2: A family of versatile vlms for transfer, 2024, URL/https: sttttps://arxivvi oxg/ org/ass//////eeaeooo abs/2412. 03555
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lec. Improved baselines with visual instruction tuning. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 26296-26306,
2024a.
Federico Cocchi, Nicholas Moratelli, Davide Caffagni, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. LLaVA-MORE, A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ido Cohen, Daniela Gottesman, Mor Geva, and Raja Giryes. Performance gap in entity knowledge extraction across modalities in vision language models. arXiv preprint arXiv:24/2.44133 2024.
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359-1177 17372, 2022.
Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual recall in llms, arXiv preprint arXiv:2402.07321, 2024.
Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip Torr, and Neel Nanda. How visual representations map language feature space multimodal llms. 2025. URL htttps://ar/rtttt hwtp://////b//aeeeee 11976
Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lui, and Yoon Kim. The semantic hub hypothesis: Language models share semantic representations across languages and modalities. In The Thirteenth Inter- national Conference on Learning Representations, 2025. URL. httptttt///ppeeeeeeeseee
wwt./ee//ffooee foo/ffrum?id
FrFUpAgnGE
Ahmed Masry, Juan A Rodriguez, Tianyu Zhang, Suyuchen Wang, Chao Wang. Aarash Feizi, Akshay Kalkunte Suresh, Abhay Puri, Xiangru Jian, Pierre-Ande Nočl, al. Alignvim: Bridging vision and language latent spaces for multimodal understanding. arXiv preprint arX1v:2502.01331, 2025.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 2443-2449, New York, NY, USA, 2021. Association for Computing Machinery, ISBN 978 9781450380379, doi: 10.1 1455/ 3404835.346322 URL https://dol.. org/10 10. 0145/3404835 3442433336666365577 3463257
Wenting Zhao, Tanya Goyal, Yu Ying Chiu, Liwei Jiang, Benjamin Newman, Abhilasha Ravichander, Khyathi Chandu, Ronan Le Bras, Claire Cardic, Yuntian Deng, al. Wildhalluccnationns: Evaluating long-form factuality Ilms with real-world entity queries. arXiv preprint arXiv:2407. 174.66 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Cashon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa- Fekete, Alex Feng, Noveen Sachdeva, Renjamin Coleman, Yi Gao, Basil Mustafa, Tain Barr, Emilio Parisotto, David Tian, Matan Eyal. Colin Cherry. Jan Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju. Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma. Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer. Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Micch, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlic Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas. Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan. Glenn Cameron, Hadi Hashemi, Hanna Klimezak- Pluucinska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abaade
Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai. Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji. Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter.
11

---

Matt Hoffmann Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma. Nabila Babar. Natasha Noy. Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton. Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Pöder, Sijal Bhatnagar. Sindhu Raghuram Panyam. Sivan Higer, Susan Zhang. Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kaira, Utku Evci, Vedant Misra, Vincent Roseherry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoche Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lueas Gonzalez. Zach Gleicher. Tris Warkentin, Vahab Mirrokni. Evan Senter, Eli Collins. Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer. Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu. Clement Farabet, Elena Buchatskaya, Jean- San-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Schastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. https://:///axx/ org/ 2553/ 2503. 19786
Shuai Bai, Kegin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song. Kai Dang, Peng Wang. Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Ii, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. 2025. URL https: //arxiv org/ /2022. 13923
Neel Nanda. Attribution patching: Activation patching at industrial scale. URL: https://www. neelnanda. 2023.
Eden Biran, Daniela Gottesman, Sohee Yang. Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models on multi-hop queries, arXiv preprint arXiv:2406. 12775, 2024.
Yaniv Nikankin, Dana Arad, Yossi Gandelsman, and Yonatan Belinkov. Same task. different circuits: Disentan- gling modality-specific mechanisms in vlms. 2025. htttp:////axxttt wwtp:og///aaaaaaaaoeo 09047
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEFE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10. wttp:/CCP....00.500000
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lee. Llavanext: Improved reasoning, ocr, and world knowledge, 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generie visual- lingul-in tasks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185-24198, 2024.
Jcan- Caan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for fow-shot learning. Advances in neural information processing systems, 35:23716-237366 2022.
Dawei Yan, Pengeheng Li, Yang Li, Hao Chen, Qingguo Chen, Weihua Luo, Wei Dong, Qingsen Yan, Haokui Zhang, and Chunhua Shen. Tg-llava: Text guided llava via learnable latent embeddings. arXiv preprint arXiv:2409. 4.44544409 2024.
Jinhao Li, Haopeng Li, Sarah Erfani, Lei Feng, James Bailey, and Feng Liu. Visual-text cross alignment: Refining the similarity score in languaee models. arXiv preprint arXiv:2406.00 0210000029....2..0 2024.
Uddeshya Upadhyay, Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. Probvlm: Probabilistic adapier for frozen language models. In Proceedings of the IEEÉ/CVF International Conference on Computer Vision, pages 1899-19111 1910, 2023.
Haozhe Chen, Junfeng Yang, Carl Vondrick, and Chengzhi Mao. Interpreting and controlling vision foundation models via text explanations. arXiv preprint 10 10591, 2023.
Yossi Gandelsman, Alexci A Efros, and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition. arXiv preprini arXiv:2310.05916 2023.
Sarah Schwettmann Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2862-2867 2023.
Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-languee representations to mitigate hallucinations. arXiv preprint 4rXiv.2410.02762. 2024.
12

---

Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing vision-language models. arXiv preprint arXiv:2410.07149, 2024.
Lachin Naghashyar, Hunar Batra, Ashkan Khakzar, Philip Torr Ronald Clark, Christian Schroeder de Witt, and Constantin Venhoff. Towards understanding multimodal fine-tuning: case study into spatial features. In Mechanistic Interpretability Workshop NeurIPS 2025, 2025. URL httpsss//pppeeereii..oooo//////e
[idJ1] j6hmk2]
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations auto-regressive language models. arXiv preprint arXiv:2304.14700 2023.
Mansi Sakarvadia, Aswathy Ajith, Arham Khan. Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Memory injections: Correcting multi-hop reasoning failures during inference in transformer-basee language models. arXiv preprint arXiv:2309 05605, 2023.
Sohee Yang. Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprini arXiv:2402.16837 2024.
13

---

A Compute Resources
We used an NVIDIA A100 GPU with 80GB memory for all local experiments. Factual question generation for the benchmark was performed via API calls to GPT-4.1.
B Dataset Generation Pipeline
"Name entity in image"
"Half Dome
correct
Half Dome
Q: Where is the entity in the image located?
"Create factual recall Q/A"
A:Yosemite Valley
Hall Dome monzonie botholith the
Figure Overview of the dataset construction pipeline for the multimodal factual recall benchmark.
We illustrate the pipeline to generate factual recall questions in Figure 6 The procedure has five stages:
Images are randomly selected from WIT and paired with their ground- truth entity names from the page title:
Image Sampling:
Entity Detection: GPT-4. is prompted to list the central identifiable entity in the image;
Entity Verification: GPT-4.1 checks whether any detected entity matches (or paraphrases)
the ground-truth entity name;
Question Generation: Given the entity name and context from the Wikipedia page, factual recall question about the entity is generated, without naming the entity explicitly;
Paraphrase Generation: minimize API calls during the benchmark run, we sample list of common paraphrases for the entity name, enabling us use simple string matching for entity guesses.
Save to Dataset: The image, verified entity name (and paraphrases), and generated question are stored as finalized benchmark sample.
C Token Space Misalignment
which shows that projector outputs of LLaVA-style VLMs are aligned with the token space of their backbone LLMs, using LLaVA-1.5.
We reproduce the results of prior work Masry et al.) 2025
7B, LLaVA-1.55-33B and LLaVA- VA-MOE.. We extract visual projector outputs from 1,000 randomly selected ImageNet images. Each image produces 575 visual token activations, yielding a total of 575,000 activations model. To assess alignment, we randomly sample 10,000 visual token embeddings per model and compute their cosine similarity with textual token embeddings from the LLM backbone.

---

Cosine Sim of Projector Output to LLM Token Embeddings
100
LLaVA-NeXT
LLaVA-1.5 (13B)
80
LLaVA-1.5 (7B)
Density
60
40
20
0.04
0.06
0.08
0.10
0.12
0.14
Cosine Similarity
Figure 7: Distribution of absolute cosine similarity between visual projector outputs and text embed- dings for LLaVA-1.5-7B. LLaVA-1.5-13BB and LLaVA-MORE All models exhibit similar distribu- tions, with means ranging from approximately 0.06 0.08. The distributions are characterized by their sharp. concentrated profiles with high density around their respective means. This concentrated distribution pattern strongly indicates that visual token embeddings consistently oceupy subspaces nearly orthogonal to the pretrained text token embedding space across all evaluated models.
Results and Analysis: Figure presents the distribution of cosine similarity scores between visual projector outputs and textual embeddings. The results indicate pronounced misalignment across all evaluated models, with cosine similarity scores tightly concentrated near zero. This suggests that visual tokens predominantly occupy an embedding subspace orthogonal to pretrained textual representations.
These findings provide direct empirical support for our hypothesis that visual embeddings fail to integrate naturally into the LLM backbone's structured token space. As result, early-layer factual recall mechanisms remain disengaged when processing visual inputs, reinforcing the idea that factual recall degradation in VLMs stems from fundamental adapter misalignment rather than insufficient computational depth alone.
D Attribution Patching Noise Multiplier
Models
Models
Divergen gencc
KLm2.0 (larget)
KLm2.0 (larget)
LLaMA-3.1 3.188
LLaVA MORE
LLaMA-2-78
L1aVA-1.5-78
Dwe-225-70-17
0wen2.5-VL77
Gemmaa-3 (LUM)
Germma-3 (VLM)
5.0
6.0
7.0
9.0
10.0
6.0
7.0
8.0
9.0
10.0
Naise Multiplier
Noise Multiplier
Figure 8: The x-axis shows different choices for the noise multiplier a. The y-axis shows the average KL Divergence of the clean predicted token distribution and the corrupted predicted token distribution, using the indicated for the corruption. (For LA-1.55-577 we only show values for a E {1, as other values yielded NaN for the KL Divergence.)

---

The noise multiplier determines how strongly we corrupt the entity input embeddings (text tokens or image tokens). We aim for choice that leads the model to predict false answer, without completely comrppinng its generated outpur. By experimenting with different values we find that KI. Divergence around 2 usually works well for the tested models. Therefore, we ablate a for each model and choose the value that causes KL Divergence closest to 2. Results are shown in [
For LLaVA-1.5-7B we choose 1, for for -nstttrrutt we choose 2, for Gemma-3-12B (LLM) we choose 1, for Qwen2.5-VL-7B- Instruct we choose and for Gemma-3-122 (VLM) we choose
we choose for LLaVA-More we choose
E Layer Selection for Heuristic Patching
Layer Ablation LLaVA-MORE
Potch from LLM
0.28
0.26
0.22
0.18
93
6
03
0.9
0:10
0:11
Figure The x-axis shows the layer range used cache and patch MLP activations from the original LLM backbone model into the according LtyVe-style VLM. The y-axis shows the recovered performance.
Layer Ablation LLavA 1.5 7B
0.44
Patch from LLM
Accuracy Imprevement
0.43
0.42
0.41
0.3
0.5
6-1
6.8
6.9
0-10
0-11
Figure 10: The x-axis shows the layer range used to cache and patch MLP activations from the original LLM backbone model into the according LLaVA-style VLM. The y-axis shows the recovered performance.
We run heuristic patching across several layer ranges, to capture the impact of the layer choice. For the final plot we use the best performing layer range. We across all models relatively consistent maximum, with a decrease in accuracy towards larger layer ranges, indicating that the early-layer MLPs are indeed causally most relevant for factual recall. Results for LLAVA-MORE shown in for LLaVA-1.5-7B in 10 and for LLaVA-1.5-13B in
Benchmark Prompt Templates
The placcholders cantity_namee and {question} are programmatically substituted runtime.

---

Required JSON fields: Layer Ablation LLavA 13B
0.31
0.20
0.27
0.26
0.1
u-10
0-11
Figure 11: The x-axis shows the layer range used to cache and patch MLP activations from the original LLM backbone model into the according LLaVA-style VLM. The y-axis shows the recovered performance.
F.1 Image entity recognition (VLM)
I provide you an image from wikipedia. Can you tell me what the specific name of the entity shown in the image is? The specific name is the name the entity is commonly known by (name of person e.g. Julias Ceasar, name of location e.g. Eiffel Tower, name of building e.g. White House, etc.).
answer: A string containing the specific entity name
Example 1:
Input: Image of the Eiffel Tower in Paris
Output: {{
"answer"; "Eiffel Tower"
}}
Example
Input: Image of famous scientist
Output: {{
"answer"; "Albert Einstein" }}
Answer in the following json format:
{{
"answer"; "Kspecific_entityy.u
}}
F.2 LLM factual recall (base)
Consider the following entity: pattiyynnmmel" Answer the following question: {question)
Required JSON fields:
answer: string containing the final answer in 1-5 words
Example
Input: Entity: Eiffel Tower, Question: When was it completed?
Output: {{
"ansser": "1889"
}}
Example 2:

---

Input: Entity: Albert Einstein, Question: What theory is he most famous for?
Output: {
"answer": "Theory of Relativity" }}
PROVIDE YOUR FINAL ANSWER IN 1-5 WORDS IN THE FOLLOWING FORMAT AND PROVIDE NO OTHER TEXT:
{{
"answer"; "<your answer>"
}}
F.3 VLM factual recall (base)
provide you with image of entity. Answer the following question: {question]
Required JSON fields:
answer: A string containing the final answer in 1-5 words
Example 1:
Input: Image of the Eiffel Tower, Question: When it completed?
Output: {{
"aanserr; "1889"
}}
Example 2:
Input: Image of Albert Einstein, Question: What theory is he most famous for?
Output: {{
"answer": "Theory of Relativity"
}}
PROVIDE YOUR FINAL ANSWER IN 1-5 WORDS IN THE FOLLOWING FORMAT PROVIDE NO OTHER TEXT:
{(
"answer": "cyour answer>"
}}
F.4 LLM factual recall (CoT)
Consider the following entity: pontity_name) Answer the following question: {question} Reason through this question by:
1. Summarizing relevant background knowledge about the entity
2. Deriving the answer to the question
Required JSON fields:
reasoning: A string containing 2-3 sentences explaining your thought
process
answer: A string containing the final answer in 1-5 words
Example 1:
Input: Entity: Eiffel Tower, Question: When it completed?
Output: {{
"reasoning": "The Eiffel Tower was built for the 1889 World's Fair in Paris. Construction began in 1887 and took about years to complete.". ",
"answer": "1889"
}
5

---

Example
Input: Entity: Albert Einstein, Question: What theory is he most famous for?
Output: {{
"reasoning": "Einstein revolutionized physics with his theory of + relativity. His most groundbreaking work was the theory of general relativity, which he published in 1915.", "answer": "Theory of Relativity"
}}
PROVIDE YOUR REASONING OF 2-3 SENTENCES AND FINAL ANSWER IN of 1-5 WORDS IN THE FOLLOWING FORMAT AND PROVIDE OTHER TEXT:
{{
"reasoning"; "<your reasoning>", "answer"; "cyour answer>"
}}
F,5 VLM factual recall (CoT)
I provide you with an image of entity. Answer the following question: {question) Reason through this question by:
1. providing a brief description of the image
2. Identify the entity the question refers to and summarize relevant background knowledge about the entity
3. Derive the ansver to the question
Required JSON fields:
reasoning: A string containing 3-4 sentences explaining your thought
process
answer: A string containing the final answer in 1-5 words
Example 1:
Input: Entity: Eiffel Tower, Question: When was it completed?
Output: {{
"reasoning"; "This is image of the Eiffel Tower in Paris, + wrought-iron lattice tower. It was built for the 1889 World's Fair and took about 2 years construct, with completion in 1889.", "answer": "1889"
}}
Example 2:
Input: Image of Albert Einstein, Question: What theory is he most famous for?
Output: {{
"reasoning": "This is an image of Albert Einstein, renowned physicist + known for his revolutionary contributions to physics, His most groundbreaking work was the theory of general relativity, which he published 1915 and fundamentally changed our understanding space and time.", "answer": "Theory of Relativity"
-
}}
PROVIDE YOUR REASONING OF 3-4 SENTENCES AND FINAL ANSWER IN of 1-5 WORDS IN THE FOLLOWING FORMAT AND PROVIDE NO OTHER TEXT:
{{
"reasoning": "cyour reasoning?", "answer": "<your answer>"
}

---

G Case Study into Excluded Images
We conduct brief case study into the images for which entities are correctly identified by VLMs. We check the entity recognition results of Pixtral-12B and find that about 24% of examples are excluded due Pixtral- 12B identifying the correct entity. A manual audit of 100 rejected cases reveals four overall failure modes:
1. Paraphrase mismatch e.g.. "FotoArtFestival" -
"6 Foto Art Festival"
2. Near-entity confusion e.g., "SS Monterey" - "RMS Mauretania"
3. Imprecision - e.g., "City Point, Wisconsin" "Jackson County"
4. Completely off-target rare; e.g., "Grand Palace Hotel" + "European Union"
We find that paraphrase mismatches and imprecision occur most often, likely because the VLM genuinely does not know the correct entity. Paraphrase mismatches in general are infrequent, which supports our approach of pre-generating list of paraphrases avoid an API call for every entity guess. Lastly, we find very few "completely off-target" cases. These seem to arise when the VLM misidentifies the central entity, e.g., the image above contains European Union flag on the hotel, and the VLM assumes the target entity is the European Union rather than the hotel. No systematic bias toward any particular domain (e.g.. geography, people, or protected classes) was observed. Images span landscapes, historical objects, banknotes, architecture, etc.; the error distribution looks random rather than skewed. Consequently, we believe these exclusions are unproblematic.

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's contributions: (1) identifying factual recall degradation in VLMs using a new benchmark, (2) providing evidence of visual-text token misalignment, and (3) showing via probing and patching that visual entity representations emerge later than the early-layer factual recall mechanisms. These findings, presented in Sections 3 3-6, support the two-hop hypothesis through empirical evidence.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper discusses several limitations in the Discussion and Conclusion section. It acknowledges that the study focuses primarily on LLaVA-style VLMs and suggests that future work should examine whether similar issues exist other architectures. It also notes the limitation in scope regarding model scale and hints at the possibility that larger models may behave differently. Furthermore, recognizes that alternative probing and intervention techniques may yield additional insights. While a separate "I "Limitations" section is not explicitly present, these points are conveyed within the discussion.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ter system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
The paper does not present any formal theoretical results, theorems, or proofs. Instead, it an empirical and experimental study focused on benchmarking, latent analysis,
Justification:
attribution, probing, and patching experiments to support its claims. Therefore, the question is not applicable.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides details to reproduce the main experimental results, in- cluding benchmark construction (2) with additional implementation details in Appendix
Justification:
B), patching methodology (4), attribution scoring (4), and probing setup (5). All evalu- ated datasets (ImageNet. CIFAR-I00 (00) publicly available, and the question-generation procedure using GPT-4o is clearly described. The models used (LLaVA variants, GPT-4o, Claude Opus) are also publicly documented. Additionally, the code for all experiments and benchmark generation will be made publicly available to facilitate full reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper provides open access to both the code and the benchmark dataset. These resources are included in the supplementary materials for reviewers and will also be made publicly available upon publication. The release includes scripts for data generation, experiment execution (c.g., probing and patching), and evaluation to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/ public/guiden https///ooeeeubaaasiooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https httpp//i//giiieeeeeeeeiiiiiissss Cwt///oooeesssssseeooeee
wwi.bbiissinooiiiici.cooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper provides all necessary details understand and reproduce the experimental setup. including dataset sources, number of examples, model variants, and
10

---

evaluation methodologies, The probing setup is described in Section patching method- ology Section 4 and benchmark construction in Section with implementation details in Appendix B As this is an evaluation and analysis paper- not focused on training new hyperparameters (e-g.. optimizer, learning rate) are not appli- cable. All code, data generation scripts, and the benchmark dataset are included in the supplementary materials and will be made publicly available upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper uses large, randomly sampled evaluation sets, and the observed differences- succh as the VLM-LLM factual recall gap and patching improvements are substantial and consistent across models (Sections [2). Due to the clear empirical trends and large sample sizes, variance from different samplings was not expected to meaningfully affect conclusions. That said, we acknowledge that including statistical significance measures would strengthen the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma eтror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eтrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies that all experiments were conducted on an NVIDIA A100 GPU with 80GB memory. Since the work involves only inference (not training) relatively small datasets (5K examples), the experiments are lightweight and not resource-intensive. Additionally, part of the benchmark generation involved API calls to GPT--4.1. Given the low computational demands, detailed runtime metries were not included but the hardware setup is clearly documented for reproducibility.

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httph:
Answer: [Yes]
The research conforms to the NeurIPS Code of Ethics. It involves standard datasets (ImageNet, CIFAR- 100) and public vision-language models (c.g., LLaVA, GPT- 40) and does not involve human subjects, personally identifiable information, or ethically sensitive data.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The paper discusses broader societal implications in the Discussion and Con- clusion section (see Section 8). It highlights the potential risks of factual recall failures in high-stakes settings, where misleading outputs from VL.Ms could have harmful conse- quences. At the same time, the work contributes positively by identifying architectural limitations that, addressed, can improve the safety and reliability of future multimodal systems.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
12

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper does not release new models or datasets that pose high risks for misuse. It evaluates existing publicly available proprietary models (e.g.. LLaVA variants, GPT-4o, Claude Opus) and does not introduce new generative systems or web-scraped data. The released dataset is benchmark constructed from subset of ImageNet, which is well-established and widely used dataset. As such, no special safeguards are necessary.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets used in the paper-including models (e.g., LLaVA variants, GPT-4o, Claude Opus) and datasets (ImageNet, CIFAR-100)-- are properly cited, and their licenses and terms of use are respected. ImageNet and CIFAR-I00 are widely used public datasets with established licenses, and all referenced models are either open-source or accessed via official APIs under their respective terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
13

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The paper introduces a new benchmark for evaluating factual recall VLMs, constructed from subset of ImageNet with GPT-4.1-generee questions. The benchmark is well documented, with details provided in Section and Appendix [ The dataset and
Justification:
associated seripts are included in the supplementary materials and will also be publicly released to ensure transparency and reproducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any crowdsoureing or research with human subjects. All data used (ImageNet, (IFAR-100) is publicly available, and no new human annotations or interactions were collected for this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve research with human subjects and therefore does not require IRB approval. All experiments were conducted using publicly available datasets (ImageNet, CIFAR-100) and pre-existing models, with no human participation or data collection involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
14

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
The paper uses GPT-4.1 generate entity-specific factual questions as part
of the benchmark construction process (see Section 2 This usage is methodological component and clearly described the paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy for what should or should be described.
15


Reviews:
[{"review_id": "HyVMD31zy5", "reviewer": "Reviewer_wjJF", "summary": "This paper investigates the poor performance of multi-modal large models in factual recall tasks. First, the authors propose that the distributed representations of visual information across visual tokens in early layers bypass the factual recall mechanism residing in the early-layer MLPs of the LM backbone, which is the root cause of this phenomenon. Based on this analysis, they then patch early-layer MLP outputs from the LM backbone into the corresponding VLM layers, significantly improving factual recall performance. Finally, the authors introduce a benchmark to systematically evaluate factual recall accuracy and knowledge hallucination in multi-modal settings.", "questions": "1. I'm intrigued by the inherent contradiction in VLMs: During pretraining, the primary objective is vision-language alignment, yet empirical evidence shows persistent token space misalignment. Would scaling up pretraining (e.g., with more data/compute) effectively mitigate this issue?\n2. Does the proposed activation patching technique impact inference speed? Quantitative metrics are required to evaluate its computational overhead.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The authors hypothesize that adapter misalignment is the root cause of factual recall degradation in VLMs, and gives the illustration of the Two-Hop problem.\n2. The manuscript is explicit and well-organized, and provides visualizations.\n3. Experimental validation is sufficient and solid. The proposed method achieves remarkably high accuracy, demonstrating impressive performance compared to baselines, as shown in Figure 5.\n\n**Weaknesses:**\n\n1. While the analysis in Chapter 2 provides illustrative insights, its exclusive reliance on LLaVA-series models raises concerns about generalizability. The observed phenomena may not necessarily hold for other architectural paradigms, requiring validation across more diverse model families.\n2. The authors conducted experiments only on the LLaVA-series architectures. Including evaluations on Qwen-VL and InternVL would further strengthen the confidence of the findings.\n3. The citation format in the paper has some minor issues. please check it.", "was_revised": true, "final_justification": "Thank you for answering my question; I decided to keep my original score.", "rebuttal": "We sincerely appreciate wjJF’s thoughtful comments and suggestions.\n\nWe address all questions and concerns raised, specifically:  \n1. Generalization of our results  \n2. Influence of scaled-up pre-training and alternative architectures on representation and token space  \n3. Inference speed of the patching experiment  \n\nWe also thank wjJF for careful proofreading and for pointing out minor citation-format issues; we have double-checked all citations and fixed the problems.\n\n## 1. Generalization of our results  \nWe appreciate the suggestion to investigate the generalization of our findings by including more diverse models.\n\nTo mitigate concerns about the generality of our results, we have:  \n1. Updated the benchmark dataset to ensure a more diverse, realistic, and factually grounded distribution.  \n2. Run the benchmark across a wide range of models, including adapter-based models (LLaVA, Pixtral, Qwen-VL), native models (Llama-4, Gemma 3, Gemini 2.0), and a cross-attention model (Llama 3.2-Vision) covering model sizes from 7B to 124B.\n\nGiven inconsistencies in model classifications in the literature, we adopt the following definitions:  \n- **Adapter-based**: models that train on top of a pretrained LLM and a pretrained ViT.  \n- **Native**: models that train the autoregressive decoder on textual and visual data from the beginning (no separate pretrained LLM).  \n- **Cross-attention**: models that use cross-attention layers to incorporate visual information (“Flamingo style”).  \n\n### Results  \n\n| **VLM Model**            | **LLM Model**      | **Architecture** | **LLM (%)** | **VLM (%)** | **Δ VLM → LLM (%)** |\n|--------------------------|--------------------|------------------|-----------|-----------|---------|\n| LLaVA-MORE-8B            | Llama-3.1-8B-it    | Adapter          | 41 | 23 | **–43.9** |\n| LLaVA-NEXT-8B            | Llama-3-8B-it      | Adapter          | 41 | 24 | **–41.5** |\n| LLaVA-1.5-7B             | Llama-2-7B-chat    | Adapter          | 30 | 19 | **–36.7** |\n| LLaVA-1.5-13B            | Llama-2-13B-chat   | Adapter          | 44 | 28 | **–36.4** |\n| Pixtral-Large-124B       | Mistral-Large-2    | Adapter          | 68 | 56 | **–17.6** |\n| Pixtral-12B              | Mistral-NeMo       | Adapter          | 41 | 36 | **–12.2** |\n| Qwen-2.5-VL-7B-it        | Qwen-2.5-7B-it     | Adapter          | 30 | 28 | **–6.7** |\n| Qwen-2.5-VL-72B-it       | Qwen-2.5-72B-it    | Adapter          | 49 | 53 | **+8.2** |\n| Llama-4-Maverick         | —                  | Native           | 75 | 71 | **–5.3** |\n| GPT-4o                   | —                  | Native           | 77 | 73 | **–5.2** |\n| Gemini-2.0-Flash         | —                  | Native           | 66 | 63 | **–4.5** |\n| Gemma-3-27B-it           | —                  | Native           | 46 | 46 | **0.0** |\n| Llama-3.2-Vision-11B     | Llama-3.1-8B-it    | Cross-attention  | 36 | 31 | **–13.9** |\n\n### Findings  \n\n- Factual-recall degradation appears in **11 of 13 models tested**.  \n- The problem **extends to very large adapter-based models** (Pixtral-Large-124B, –17.6 %) and to the cross-attention model (–13.9 %).  \n- Native models exhibit smaller drops (≈ 4–5 %), and Qwen-2.5-VL actually outperforms its backbone (+8.2 %), likely due to its >4T-token multimodal alignment training corpus.  \n\n### Summary  \n\n1. Factual-recall degradation is prevalent (11 / 13 models).  \n2. It affects large adapter-based and cross-attention models.  \n3. Native models, or adapter-based models trained on massive multimodal corpora, show less degradation.  \n\nThus, unless one can afford native end-to-end training or huge multimodal datasets, understanding and mitigating factual-recall degradation is a critical challenge.\n\n## 2. Pre-training, architecture, and visual representations  \n\nWe selected Qwen-2.5-VL-7B (adapter-based, >4T alignment training tokens) and Gemma-3-11B (native) to compare with LLaVA-1.5-7B.\n\nWe performed:  \n1. Layer-wise linear probing to detect when entity information becomes linearly represented.  \n2. Token-space-alignment experiments measuring cosine similarity between projector outputs and LLM token embeddings.\n\n### Probing results  \n\n| Layer # (32 total) | 0 | 5 | 10 | 15 | 20 | 25 | 30 |\n|--------------------|----|----|----|----|----|----|----|\n| LLaVA-1.5-7B       | **0.18** | 0.40 | 0.81 | 0.96 | 0.98 | 1.00 | 1.00 |\n\n| Layer # (28 total) | 0 | 4 | 8 | 12 | 16 | 20 | 24 |\n|--------------------|----|----|----|----|----|----|----|\n| Qwen-2.5-VL-7B     | **0.78** | 0.91 | 0.96 | 0.99 | 1.00 | 1.00 | 1.00 |\n\n| Layer # (48 total) | 0 | 7 | 14 | 21 | 28 | 35 | 42 |\n|--------------------|----|----|----|----|----|----|----|\n| Gemma-3-11B        | **0.63** | 0.89 | 0.97 | 1.00 | 1.00 | 1.00 | 1.00 |\n\nBoth Qwen-2.5-VL-7B and Gemma-3-11B form linear entity representations far earlier than LLaVA-1.5-7B, aligning with their stronger benchmark performance.\n\nTwo mechanisms could account for this:  \n1. Better projection alignment with the LLM token space.  \n2. Adaptation of LLM factual-recall circuits to non-textual representations.\n\n### Token-space alignment  \n\n- **Qwen-2.5-VL-7B**: avg. cosine = 0.0778 ± 0.0023  \n- **Gemma-3-11B**: avg. cosine = 0.1283 ± 0.0043  \n\nGemma-3’s higher cosine stems largely from tokens aligning with the beginning-of-sentence embedding (≈ 0.33) and therefore carrying no visual content. Thus, early representational availability, not projection/token-space overlap, likely yields the improvement, supporting hypothesis 2. We think carrying out a detailed analysis of how visual representations integrate with LLM mechanisms like factual recall is an exciting avenue for future research.\n\n## 3. Computational overhead of patching  \n\nOur patching round requires:  \n- One forward pass through the LLM (text input) to cache MLP activations.  \n- One forward pass through the VLM (image + text) to cache MLP activations.  \n- One backward pass through the VLM to obtain KL-divergence gradients.  \n- A final matrix multiplication of gradient × activation differences.\n\nTotal overhead: two forward passes, one backward pass, plus a single matrix multiply. On a single A100, running 100 examples requires ≈ 40 minutes. We emphasize that the method is intended as an experimental setup to support our hypothesis about the role of early-layer MLPs, and not as a practical approach to reduce factual-recall degradation.\n\n## Additional: reasoning / chain-of-thought (CoT)  \nWhile our patching experiment shows that factual-recall degradation can be reduced by repairing the early-layer MLP outputs, it is not a practical approach. Similarly, it is often not feasible to train a model natively or run large-scale alignment training like Qwen-2.5-VL. Therefore, we ran an experiment where we include a chain-of-thought prompt, allowing the model to reason about the image content and the question for a couple of sentences.\n\n| **VLM Model**           | **LLM (Base → CoT)** | **VLM (Base → CoT)** | **Δ Gap (%)** |\n|-------------------------|---------------------|---------------------|--------------|\n| LLaVA-MORE-8B           | 41 → +5             | 23 → +5             | –43.9 → –39.1 |\n| LLaVA-NEXT-8B           | 41 → +4             | 24 → +6             | –41.5 → –33.3 |\n| LLaVA-1.5-7B            | 30 → +5             | 19 → +0             | –36.7 → –45.7 |\n| LLaVA-1.5-13B           | 44 → +0             | 28 → +9             | –36.4 → –15.9 |\n| Pixtral-Large-124B      | 68 → +4             | 56 → +15            | –17.6 →  –1.3 |\n| Pixtral-12B             | 41 → +4             | 36 → +12            | –12.2 → +6.6 |\n| Llama-3.2-Vision-11B    | 36 → +6             | 31 → +9             | –13.9 →  –4.7 |\n\nWe find significant recovery of the performance gap for Pixtral-12B, Pixtral-Large-124B, Llama-3.2-Vision-11B, and LLaVA-1.5-13B. We therefore suggest reasoning as a potential avenue to mitigate factual-recall degradation in VLMs, without needing excessive pre-training datasets or the resources for native training.\n\n---\n\nWe hope these clarifications and additional results resolve all questions and outstanding concerns and demonstrate the robustness of our findings. We are happy to answer any remaining questions. Thank you for your thoughtful review."}, {"review_id": "LC8R4i2RMa", "reviewer": "Reviewer_nkVQ", "summary": "The paper investigates the factual recall degradation in LLaVA-style Vision-Language Models (VLMs) compared to their language backbone. It attributes this degradation to the architectural design of VLMs, where visual information is processed too late to effectively engage the early-layer factual recall mechanisms of the language model. The paper introduces the concept of a \"two-hop\" problem, provides empirical evidence through linear probing and patching experiments, and proposes a benchmark to evaluate factual recall accuracy and knowledge hallucination in VLMs. It also demonstrates that patching early-layer MLP outputs from the language model into the VLM can significantly improve factual recall performance.", "questions": "1. The authors should provide more details about the experiment setup.\n2. The code and datasets are not included in the supplementary materials or publicly released.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## **Summary Of Strengths:**\n\n1. The paper presents a clear and novel explanation for the factual recall degradation in VLMs by highlighting the architectural limitations and the \"two-hop\" problem. This provides a fresh perspective on why VLMs underperform compared to their language backbones in factual recall tasks.\n2. The paper support the hypothesis with a variety of empirical experiments, including linear probing, activation patching, and a newly introduced benchmark. These experiments help to validate the two-hop hypothesis and demonstrate the impact of visual-text misalignment in VLMs.\n3. The introduction of fact recall benchmarks can better evaluate the accuracy of fact recall and knowledge illusion, which is of great significance for improving the reliability of VLMs.\n\n## **Summary Of Weaknesses:**\n\n1. While the paper provides empirical evidence for the two-hop problem, it lacks a more thorough comparison with other possible explanations for factual recall degradation in VLMs. This makes it difficult to determine if the two-hop problem is the sole or primary cause.\n2. The models evaluated were small in size, and the authors should examine whether the behavior of larger models alleviates or exacerbates these problems.\n3. The paper's focus is primarily on LLaVA-style VLMs, which limits the generalizability of the findings. It would be beneficial to explore whether similar issues exist in other VLM architectures.", "was_revised": true, "final_justification": "I have read the author's rebuttal carefully, along with the other reviews, and most of my concerns have been resolved. The experiment on generalization is a good additional experiment. Given the paper's scope and comprehensive experiments, I remain mostly positive and will keep my scores.", "rebuttal": "We truly thank **nkVQ** for the nice comments and helpful suggestions.\n\nWe address all questions and concerns raised, specifically:  \n1. Generalization of findings  \n2. Considering alternative hypotheses  \n\n## 1. Generalization of findings  \nWe appreciate the suggestion to investigate the generalization of our results by including more diverse—and larger—models.\n\nTo mitigate concerns about generality, we have:  \n1. Updated the benchmark dataset to ensure a more diverse, realistic, and factually grounded distribution.  \n2. Run the benchmark across a wide range of models, including adapter-based models (LLaVA, Pixtral, Qwen-VL), native models (Llama-4, Gemma-3, Gemini-2.0), and a cross-attention model (Llama-3.2-Vision) covering model sizes from 7B to 124B.\n\nGiven inconsistencies in model classifications, we use:  \n- **Adapter-based**: pretrained LLM + pretrained ViT, then finetuned together.  \n- **Native**: the decoder is trained on text + vision from scratch (no standalone LLM).  \n- **Cross-attention**: Flamingo-style cross-attention layers inject vision.\n\n### Results  \n\n| **VLM Model**            | **LLM Model**      | **Architecture** | **LLM (%)** | **VLM (%)** | **Δ VLM → LLM (%)** |\n|--------------------------|--------------------|------------------|-----------|-----------|---------|\n| LLaVA-MORE-8B            | Llama-3.1-8B-it    | Adapter          | 41 | 23 | **–43.9** |\n| LLaVA-NEXT-8B            | Llama-3-8B-it      | Adapter          | 41 | 24 | **–41.5** |\n| LLaVA-1.5-7B             | Llama-2-7B-chat    | Adapter          | 30 | 19 | **–36.7** |\n| LLaVA-1.5-13B            | Llama-2-13B-chat   | Adapter          | 44 | 28 | **–36.4** |\n| Pixtral-Large-124B       | Mistral-Large-2    | Adapter          | 68 | 56 | **–17.6** |\n| Pixtral-12B              | Mistral-NeMo       | Adapter          | 41 | 36 | **–12.2** |\n| Qwen-2.5-VL-7B-it        | Qwen-2.5-7B-it     | Adapter          | 30 | 28 | **–6.7** |\n| Qwen-2.5-VL-72B-it       | Qwen-2.5-72B-it    | Adapter          | 49 | 53 | **+8.2** |\n| Llama-4-Maverick         | —                  | Native           | 75 | 71 | **–5.3** |\n| GPT-4o                   | —                  | Native           | 77 | 73 | **–5.2** |\n| Gemini-2.0-Flash         | —                  | Native           | 66 | 63 | **–4.5** |\n| Gemma-3-27B-it           | —                  | Native           | 46 | 46 | **0.0** |\n| Llama-3.2-Vision-11B     | Llama-3.1-8B-it    | Cross-attention  | 36 | 31 | **–13.9** |\n\n### Findings  \n\n- Factual-recall degradation appears in **11 / 13 models tested**.  \n- The issue **extends to very large adapter-based models** (Pixtral-Large-124B, –17.6 %) and to the cross-attention model (–13.9 %).  \n- Native models drop only ≈ 4–5 %, and Qwen-2.5-VL actually beats its backbone (+8.2 %), likely due to its > 4 T-token multimodal alignment corpus.  \n\n### Summary  \n\n1. Factual-recall degradation is prevalent (11 / 13).  \n2. It affects large adapter-based and cross-attention models.  \n3. Native models—or adapter-based ones trained on massive multimodal corpora—show less degradation.  \n\nThus, unless one can afford native end-to-end training or huge multimodal datasets, understanding and mitigating factual-recall degradation remains critical.\n\n## 2. Considering alternative hypotheses  \n\nWe thank nkVQ for suggesting adding experiments that challenge our main hypotheses and provide further context.\n\n### 2.1  Late-layer representations instead of early-layer MLPs  \nOur attribution experiment focuses solely on early-layer MLPs. However, prior work on two-hop queries in LLMs (Biran et al. 2024, https://arxiv.org/abs/2406.12775) and modality-specific mechanisms in VLMs (Nikankin et al. 2025, https://arxiv.org/html/2506.09047v1) shows that “back-patching” later-layer representations into earlier layers can also help recover performance.  \nWe implemented the exact back-patching setup of Nikankin et al. as a baseline and find that it closes only **6–8 %** of the performance gap, compared with **35–40 %** from our early-MLP patching.  \nThis further supports our hypothesis that early-layer MLPs are the sub-module that breaks in multimodal factual recall due to token-space misalignment, in contrast to the alternative explanation that VLMs could perform correct factual recall if they had only access to their later-layer representations.\n\n### 2.2  Additional probing experiments  \nNative VLMs and adapter-based VLMs with large-scale alignment show far less degradation, so we can use them as counterfactuals in the probing experiment. If our early-layer claim is right, probing should reveal that linear entity representations occur earlier in those models compared with the LLaVA-style VLMs.\n\n| Layer # (32 total) | 0 | 5 | 10 | 15 | 20 | 25 | 30 |\n|--------------------|----|----|----|----|----|----|----|\n| LLaVA-1.5-7B       | **0.18** | 0.40 | 0.81 | 0.96 | 0.98 | 1.00 | 1.00 |\n\n| Layer # (28 total) | 0 | 4 | 8 | 12 | 16 | 20 | 24 |\n|--------------------|----|----|----|----|----|----|----|\n| Qwen-2.5-VL-7B     | **0.78** | 0.91 | 0.96 | 0.99 | 1.00 | 1.00 | 1.00 |\n\n| Layer # (48 total) | 0 | 7 | 14 | 21 | 28 | 35 | 42 |\n|--------------------|----|----|----|----|----|----|----|\n| Gemma-3-11B        | **0.63** | 0.89 | 0.97 | 1.00 | 1.00 | 1.00 | 1.00 |\n\nIndeed, Qwen-2.5-VL-7B and Gemma-3-11B form linear entity representations far earlier than LLaVA-1.5-7B, matching their stronger benchmark scores and supporting our hypothesis.\n\n### Summary  \nWe added the back-patching baseline from related work and a counterfactual probing study. These additions contextualize and support our claims and show that alternative hypotheses are not supported by the empirical results.\n\n## Additional: reasoning / chain-of-thought (CoT)  \nIt is often not feasible to train a model natively or run large-scale alignment training like Qwen-2.5-VL. Therefore, we tested an alternative: reasoning. We reran the benchmark for models most affected by factual-recall degradation, adding a chain-of-thought prompt that lets the model reason about the image and question for a couple of sentences.\n\n| **VLM Model**           | **LLM (Base → CoT)** | **VLM (Base → CoT)** | **Δ Gap (%)** |\n|-------------------------|----------------------|----------------------|--------------|\n| LLaVA-MORE-8B           | 41 → +5              | 23 → +5              | –43.9 → –39.1 |\n| LLaVA-NEXT-8B           | 41 → +4              | 24 → +6              | –41.5 → –33.3 |\n| LLaVA-1.5-7B            | 30 → +5              | 19 →  0              | –36.7 → –45.7 |\n| LLaVA-1.5-13B           | 44 →  0              | 28 → +9              | –36.4 → –15.9 |\n| Pixtral-Large-124B      | 68 → +4              | 56 → +15             | –17.6 →  –1.3 |\n| Pixtral-12B             | 41 → +4              | 36 → +12             | –12.2 → +6.6 |\n| Llama-3.2-Vision-11B    | 36 → +6              | 31 → +9              | –13.9 →  –4.7 |\n\nWe find significant recovery of the performance gap for Pixtral-12B, Pixtral-Large-124B, Llama-3.2-Vision-11B, and LLaVA-1.5-13B. We therefore suggest reasoning as a potential avenue to mitigate factual-recall degradation in VLMs without needing excessive pre-training datasets or the resources for native training.\n\n---\n\nWe hope these clarifications and additional results resolve all outstanding concerns and demonstrate the robustness of our findings. We are happy to answer any remaining questions. Thank you for your thoughtful review."}, {"review_id": "QnMlecZLqW", "reviewer": "Reviewer_ppcn", "summary": "The paper investigates why LLaVA-style Vision-Language Models (VLMs) underperform at factual recall compared to their underlying LLM backbones. Rather than blaming insufficient compute after the visual encoder, the authors argue that misaligned visual token embeddings bypass the early-layer MLPs responsible for storing world knowledge. Through a series of linear‐probe analyses and “attribution patching” experiments, they show that (a) visual information only takes on a clean, linearly recoverable form in mid-to-late layers, and (b) if you inject the early MLP activations from the unimodal LLM into the VLM, you recover almost 40% of the factual recall gap. Finally, they release a benchmark that systematically compares VLM vs. LLM recall on 1,000 ImageNet-derived questions. Their main takeaway is that current adapter architectures delay entity formation until it’s too late to hook into the LLM’s fast recall pathways, creating a “two-hop” bottleneck.", "questions": "1. How uniform is the difficulty across GPT-4.1-generated questions? Could some entities consistently yield easier or harder questions, skewing VLM vs. LLM gaps?\n\n2. Do findings hold for other adapter-based VLMs (e.g., Flamingo, InternVLM) or fusion architectures that interleave vision and language layers?\n\n3. The “greedy matching” to align entity positions in the VLM seems critical. Can you quantify how often it picks the correct slot vs. noise spots, and does that affect recovery magnitude?\n\n4. For the cases you exclude because the VLM misidentifies the entity, how often does that happen and does it correlate with certain image types?\n\n5. Have you tried patching at a sliding window of layers (e.g., layers 1–3, 3–5, etc.) to pinpoint the minimal “hop” window needed for a meaningful recovery?", "limitations": "The authors acknowledge that their study is limited to LLaVA-style models and suggest future work should explore other architectures and scales. However, they do not address potential negative societal impacts of deploying misaligned VLMs (e.g., hallucinating harmful misinformation, misidentifying medical images).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Well-structured narrative: hypothesis → misalignment proof → patching → probing → success/failure comparison.\n\nFocused solely on LLaVA variants; it’s unclear if patching tricks or adapter redesigns generalize to fusion-style VLMs.\n\nBuilds heavily on prior work without clearly delineating incremental vs. groundbreaking contributions", "was_revised": true, "final_justification": "All my concerns have been resolved, but in the paper, the author said the code and benchmark data are included in the supplementary material, but I can't see them.", "rebuttal": "We thank **ppcn** for the helpful suggestions and precise questions.\n\n---\n> **How uniform is the difficulty across GPT-4.1-generated questions? Could some entities consistently yield easier or harder questions, skewing VLM vs. LLM gaps?**\n\nWe replace ImageNet with **WIT: Wikipedia-based Image–Text Dataset**, which comprises images of entities and their corresponding Wikipedia paragraphs.  Instead of relying fully on GPT-4.1 to create a diverse and factually correct distribution of question–answer pairs, we now prompt it with both the image of an entity and its Wikipedia content. This ensures a diverse, realistic distribution of entities and factual grounding of questions in Wikipedia.\n\nWe manually inspected hundreds of questions and found that GPT-4.1 adheres well to the given context; the questions consistently ask for factual knowledge mentioned in the Wikipedia paragraph.\n\nBecause we test VLMs and LLMs on the exact same questions, providing the entity either visually or textually, the performance gap stems solely from the modality-specific performance difference.\n\n---\n> **Do findings hold for other adapter-based VLMs (e.g., Flamingo, InternVLM) or fusion architectures that interleave vision and language layers?**\n\nWe ran the benchmark across a wide range of models: adapter-based (LLaVA, Pixtral, Qwen-VL), native (Llama-4, Gemma-3, Gemini-2.0) and a cross-attention model (Llama-3.2-Vision) covering sizes from 7 B to 124 B. Flamingo proved hard to elicit structured output from, so we used Llama-3.2-Vision as a cross-attention proxy.\n\n| **VLM Model**            | **LLM Model**      | **Arch.** | **LLM (%)** | **VLM (%)** | **Δ VLM→LLM** |\n|--------------------------|--------------------|-----------|-----------|-----------|--------------|\n| LLaVA-MORE-8B            | Llama-3.1-8B-it    | Adapter   | 41 | 23 | **–43.9** |\n| LLaVA-NEXT-8B            | Llama-3-8B-it      | Adapter   | 41 | 24 | **–41.5** |\n| LLaVA-1.5-7B             | Llama-2-7B-chat    | Adapter   | 30 | 19 | **–36.7** |\n| LLaVA-1.5-13B            | Llama-2-13B-chat   | Adapter   | 44 | 28 | **–36.4** |\n| Pixtral-Large-124B       | Mistral-Large-2    | Adapter   | 68 | 56 | **–17.6** |\n| Pixtral-12B              | Mistral-NeMo       | Adapter   | 41 | 36 | **–12.2** |\n| Qwen-2.5-VL-7B-it        | Qwen-2.5-7B-it     | Adapter   | 30 | 28 | **–6.7** |\n| Qwen-2.5-VL-72B-it       | Qwen-2.5-72B-it    | Adapter   | 49 | 53 | **+8.2** |\n| Llama-4-Maverick         | —                  | Native    | 75 | 71 | **–5.3** |\n| GPT-4o                   | —                  | Native    | 77 | 73 | **–5.2** |\n| Gemini-2.0-Flash         | —                  | Native    | 66 | 63 | **–4.5** |\n| Gemma-3-27B-it           | —                  | Native    | 46 | 46 | **0.0** |\n| Llama-3.2-Vision-11B     | Llama-3.1-8B-it    | X-attn    | 36 | 31 | **–13.9** |\n\n### Findings  \n- Factual-recall degradation appears in **11 of 13 models tested**.  \n- The problem **extends to very large adapter-based models** (Pixtral-Large-124B, –17.6 %) and to the cross-attention model (–13.9 %).  \n- Native models exhibit smaller drops (≈ 4–5 %), and Qwen-2.5-VL actually outperforms its backbone (+8.2 %), likely due to its >4T-token multimodal alignment training corpus.  \n\n### Summary  \n\n1. Factual-recall degradation is prevalent (11 / 13 models).  \n2. It affects large adapter-based and cross-attention models.  \n3. Native models, or adapter-based models trained on massive multimodal corpora, show less degradation.  \n\n---\n> **The “greedy matching” to align entity positions in the VLM seems critical. Can you quantify how often it picks the correct slot vs. noise, and does that affect recovery magnitude?**\n\nThere is no direct ground-truth slot, because we must patch LLM entity-token positions into VLM token positions. Since there is no entity token in the VLM template, we use a heuristic based on attribution to choose a slot.\n\nTo quantify the heuristic as requested, we added two baselines:\n\n* **Random token patching** → **4%** average recovery  \n* **Back-patching** (Nikankin et al. 2025) → **6%** \n\nThe large gap between those results and our **36%** recovery rate, shows that our patching approach reliably selects a good slot.\n\n---\n> **For cases excluded because the VLM misidentifies the entity, how often does that happen, and does it correlate with certain image types?**\n\nUsing Pixtral-12B for entity recognition, about **24 %** of examples are excluded. A manual audit of 100 rejected cases reveals four failure modes:\n\n1. **Paraphrase mismatch** – e.g., “FotoArtFestival” → “6 Foto Art Festival”  \n2. **Near-entity confusion** – e.g., “SS Monterey” → “RMS Mauretania”  \n3. **Imprecision** – e.g., “City Point, Wisconsin” → “Jackson County”  \n4. **Completely off-target** – rare; e.g., “Grand Palace Hotel” → “European Union”\n\nWe find that paraphrase mismatches and imprecision occur most often, likely because the VLM genuinely does not know the correct entity. Paraphrase mismatches in general are infrequent, which supports our approach of pre-generating a list of paraphrases to avoid an API call for every entity guess. Lastly, we find very few “completely off-target” cases. These seem to arise when the VLM misidentifies the central entity, e.g. the image above contains a European Union flag on the hotel, and the VLM assumes the target entity is the European Union rather than the hotel.\n\nNo systematic bias toward any particular domain (e.g., geography, people, or protected classes) was observed. Images span landscapes, historical objects, banknotes, architecture, etc.; the error distribution looks random rather than skewed. Consequently, we believe these exclusions are unproblematic for our gap analysis.\n\n---\n> **Have you tried patching with a sliding window of layers (e.g., 1–3, 3–5) to pinpoint the minimal “hop” window needed?**\n\nWe conducted an ablation study over the first ten layers (25 examples per window) on the three LLaVA models. Because the earliest layers are necessary and causally very relevant (attribution experiment), we ablated in an increasing window:  \n(0, 1), (0, 1, 2), …, (0–9).\n\n| Layers window | 0-2 | 0-3 | 0-4 | 0-5 | 0-6 | 0-7 | 0-8 | 0-9 |\n|---------------|----|----|----|----|----|----|----|----|\n| **LLaVA-1.5-7B**   | 0.44 | 0.44 | 0.40 | **0.46** | 0.44 | 0.44 | 0.40 | 0.44 |\n\n| Layers window | 0-2 | 0-3 | 0-4 | 0-5 | 0-6 | 0-7 | 0-8 | 0-9 |\n|---------------|----|----|----|----|----|----|----|----|\n| **LLaVA-1.5-13B**  | 0.32 | 0.40 | 0.28 | 0.40 | 0.32 | 0.40 | **0.44** | 0.40 |\n\n| Layers window | 0-2 | 0-3 | 0-4 | 0-5 | 0-6 | 0-7 | 0-8 | 0-9 |\n|---------------|----|----|----|----|----|----|----|----|\n| **LLaVA-MORE-8B** | 0.40 | 0.28 | 0.40 | 0.40 | **0.44** | 0.40 | 0.32 | 0.28 |\n\nOn 100 examples with the best window we obtain **0.37, 0.38, and 0.36** recovery for LLaVA-1.5-7B, LLaVA-1.5-13B, and LLaVA-MORE-8B, respectively.\n\n\n---\n## Some additional experiments\n\n### Probing other architectures  \nTo improve generalization of our findings, we ran probing on Qwen-2.5-VL-7B (adapter-based, >4T alignment tokens) and Gemma-3-11B (native).\n\n| Layer # (32 total) | 0 | 5 | 10 | 15 | 20 | 25 | 30 |\n|--------------------|----|----|----|----|----|----|----|\n| LLaVA-1.5-7B       | **0.18** | 0.40 | 0.81 | 0.96 | 0.98 | 1.00 | 1.00 |\n\n| Layer # (28 total) | 0 | 4 | 8 | 12 | 16 | 20 | 24 |\n|--------------------|----|----|----|----|----|----|----|\n| Qwen-2.5-VL-7B     | **0.78** | 0.91 | 0.96 | 0.99 | 1.00 | 1.00 | 1.00 |\n\n| Layer # (48 total) | 0 | 7 | 14 | 21 | 28 | 35 | 42 |\n|--------------------|----|----|----|----|----|----|----|\n| Gemma-3-11B        | **0.63** | 0.89 | 0.97 | 1.00 | 1.00 | 1.00 | 1.00 |\n\nBoth Qwen-2.5-VL-7B and Gemma-3-11B form linear entity representations much earlier than LLaVA-1.5-7B, matching their stronger benchmark results.\n\n### Reasoning / chain-of-thought (CoT)  \nTraining a model natively or at Qwen scale is often infeasible, so we tested reasoning as a lightweight fix:\n\n| **VLM Model**        | **LLM (Base→CoT)** | **VLM (Base→CoT)** | **Δ Gap** |\n|----------------------|--------------------|--------------------|-----------|\n| LLaVA-MORE-8B        | 41 → +5 | 23 → +5 | –43.9 → –39.1 |\n| LLaVA-NEXT-8B        | 41 → +4 | 24 → +6 | –41.5 → –33.3 |\n| LLaVA-1.5-7B         | 30 → +5 | 19 → 0  | –36.7 → –45.7 |\n| LLaVA-1.5-13B        | 44 → 0  | 28 → +9 | –36.4 → –15.9 |\n| Pixtral-Large-124B   | 68 → +4 | 56 → +15| –17.6 → –1.3 |\n| Pixtral-12B          | 41 → +4 | 36 → +12| –12.2 → +6.6 |\n| Llama-3.2-Vision-11B | 36 → +6 | 31 → +9 | –13.9 → –4.7 |\n\nReasoning recovers much of the gap for Pixtral-12B/124B, Llama-3.2-Vision-11B, and LLaVA-1.5-13B, suggesting an inexpensive mitigation.\n\n---\n\nWe hope these clarifications and additional results address all outstanding concerns and demonstrate the robustness of our findings. We are happy to answer any remaining questions. Thank you for your thoughtful review."}, {"review_id": "ldpFWYtrVK", "reviewer": "Reviewer_zisd", "summary": "This paper investigates factual recall degradation in LLaVA-style Vision-Language Models (VLMs) compared to their underlying language models. The authors propose that this degradation stems from misalignment between visual projector outputs and pretrained textual embeddings, rather than insufficient computational depth as previously suggested. Through a systematic analysis involving cosine similarity measurements, a new benchmark, linear probing, and activation patching experiments, they demonstrate that visual entity representations emerge too late in the model's layers to engage early-layer factual recall mechanisms. The paper introduces the \"two-hop problem\" hypothesis: VLMs must first resolve visual inputs into entity representations before retrieving factual knowledge, but this resolution occurs after the critical early-layer recall mechanisms. Their patching experiments show ~40% recovery of the performance gap, providing causal evidence for their hypothesis.", "questions": "Have you tested or do you have hypotheses about whether this two-hop problem exists in non-LLaVA architectures like Flamingo that use different visual-linguistic integration mechanisms?\n\nWhat factors contribute to the 60% of the performance gap that remains after patching? Have you explored combining your approach with other interventions?\n\nIn the ~35% of cases where VLMs successfully recall facts, what distinguishes these examples? Do certain types of entities or facts show better alignment timing?\n\nBased on your findings, what specific architectural modifications would you recommend to address the timing misalignment? Would earlier cross-modal fusion help?\n\nHow does the two-hop problem manifest in larger VLMs? Does increased model capacity provide any natural mitigation?", "limitations": "The authors adequately acknowledge that their study focuses on LLaVA-style models and suggest future work should examine other architectures. However, they could better discuss:\n\n* The potential bias introduced by using GPT-4.1 for question generation\n* The effectiveness and computational costs of their probing and patching approaches at scale\n* Potential negative impacts if their insights are used to develop VLMs that hallucinate more convincingly", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper provides a compelling alternative explanation for VLM factual recall degradation, shifting focus from computational depth limitations to timing misalignment between visual processing and factual recall mechanisms.\n\nThe work combines multiple complementary approaches - cosine similarity analysis, benchmarking, probing, and causal intervention through patching - providing converging evidence for their hypothesis.\n\nThe factual recall benchmark with 1,000 examples provides a valuable resource for future research, with careful design to isolate factual recall from visual recognition errors.\n\nThe cross-model patching experiments provide compelling causal evidence, showing ~40% performance recovery when early-layer representations are restored.\n\nThe paper is well-written with effective visualizations, particularly Figure 1 which clearly illustrates the two-hop problem concept.\n\nWeaknesses:\n\nThe paper only evaluates LLaVA-style architectures. It's unclear whether these findings generalize to other VLM architectures (e.g., cross-attn Flamingo, end2end chameleon, or more recent models).\n\nThe benchmark uses only ImageNet images and GPT-4.1 generated questions, which may not capture the full diversity of real-world factual queries about images. Also, how about object detection tasks?", "was_revised": true, "final_justification": "The authors addressed my concerns. I would like to keep this score.", "rebuttal": "We sincerely thank **zisd** for the constructive suggestions.\n\n---\n> **The benchmark uses only ImageNet images and GPT-4.1-generated questions, which may not capture the full diversity of real-world factual queries about images. Also, how about object-detection tasks?**\n\nWe replace ImageNet with **WIT: Wikipedia-based Image–Text Dataset**, which comprises images of entities and their corresponding Wikipedia paragraphs. Instead of relying fully on GPT-4.1 to create a diverse and factually correct distribution of question–answer pairs, we now prompt it with both the image of an entity and its Wikipedia content. This ensures a diverse, realistic distribution of entities and factual grounding of questions in Wikipedia.\n\nWe manually inspected hundreds of questions and found that GPT-4.1 adheres well to the given context; the questions consistently ask for factual knowledge mentioned in the Wikipedia paragraph.\n\nWe agree that object detection could be another interesting instance of a two-hop problem. However, as most of **our** analysis is very specific to the factual-recall mechanism in the LLM backbone, adding more domains of two-hop questions is out of scope for this work—though an interesting avenue for future research.\n\n---\n> **Have you tested or do you have hypotheses about whether this two-hop problem exists in non-LLaVA architectures like Flamingo that use different visual-linguistic integration mechanisms?**\n\nWe ran the benchmark across a wide range of models: adapter-based (LLaVA, Pixtral, Qwen-VL), native (Llama-4, Gemma-3, Gemini-2.0) and a cross-attention model (Llama-3.2-Vision) covering sizes from 7 B to 124 B. Flamingo proved hard to elicit structured output from, so we used Llama-3.2-Vision as a cross-attention proxy.\n\n| **VLM Model**            | **LLM Model**      | **Arch.** | **LLM (%)** | **VLM (%)** | **Δ VLM→LLM** |\n|--------------------------|--------------------|-----------|-----------|-----------|--------------|\n| LLaVA-MORE-8B            | Llama-3.1-8B-it    | Adapter   | 41 | 23 | **–43.9** |\n| LLaVA-NEXT-8B            | Llama-3-8B-it      | Adapter   | 41 | 24 | **–41.5** |\n| LLaVA-1.5-7B             | Llama-2-7B-chat    | Adapter   | 30 | 19 | **–36.7** |\n| LLaVA-1.5-13B            | Llama-2-13B-chat   | Adapter   | 44 | 28 | **–36.4** |\n| Pixtral-Large-124B       | Mistral-Large-2    | Adapter   | 68 | 56 | **–17.6** |\n| Pixtral-12B              | Mistral-NeMo       | Adapter   | 41 | 36 | **–12.2** |\n| Qwen-2.5-VL-7B-it        | Qwen-2.5-7B-it     | Adapter   | 30 | 28 | **–6.7** |\n| Qwen-2.5-VL-72B-it       | Qwen-2.5-72B-it    | Adapter   | 49 | 53 | **+8.2** |\n| Llama-4-Maverick         | —                  | Native    | 75 | 71 | **–5.3** |\n| GPT-4o                   | —                  | Native    | 77 | 73 | **–5.2** |\n| Gemini-2.0-Flash         | —                  | Native    | 66 | 63 | **–4.5** |\n| Gemma-3-27B-it           | —                  | Native    | 46 | 46 | **0.0** |\n| Llama-3.2-Vision-11B     | Llama-3.1-8B-it    | X-attn    | 36 | 31 | **–13.9** |\n\n### Findings  \n- Factual-recall degradation appears in **11 of 13 models tested**.  \n- The problem **extends to very large adapter-based models** (Pixtral-Large-124B, –17.6 %) and to the cross-attention model (–13.9 %).  \n- Native models exhibit smaller drops (≈ 4–5 %), and Qwen-2.5-VL actually outperforms its backbone (+8.2 %), likely due to its > 4 T-token multimodal-alignment corpus.  \n\n### Summary  \n\n1. Factual-recall degradation is prevalent (11 / 13 models).  \n2. It affects large adapter-based and cross-attention models.  \n3. Native models, or adapter-based models trained on massive multimodal corpora, show less degradation.  \n\n---\n> **How does the two-hop problem manifest in larger VLMs? Does increased model capacity provide any natural mitigation?**\n\nAs the previous section shows, very large models like Pixtral-Large-124B still exhibit significant factual-recall degradation, while smaller models like Qwen-VL-2.5-7B (with extensive alignment data) show less degradation.  \n\nTherefore, we hypothesize that data mixture and the timing of modality fusion are crucial, even though the larger Qwen-2.5-VL-72B gains performance—implying that size is still beneficial given sufficient data.\n\nWe also ran our probing experiment on Qwen-2.5-VL-7B and Gemma-3-11B to investigate why these models perform better than LLaVA.\n\n| Layer # (32 total) | 0 | 5 | 10 | 15 | 20 | 25 | 30 |\n|--------------------|----|----|----|----|----|----|----|\n| LLaVA-1.5-7B       | **0.18** | 0.40 | 0.81 | 0.96 | 0.98 | 1.00 | 1.00 |\n\n| Layer # (28 total) | 0 | 4 | 8 | 12 | 16 | 20 | 24 |\n|--------------------|----|----|----|----|----|----|----|\n| Qwen-2.5-VL-7B     | **0.78** | 0.91 | 0.96 | 0.99 | 1.00 | 1.00 | 1.00 |\n\n| Layer # (48 total) | 0 | 7 | 14 | 21 | 28 | 35 | 42 |\n|--------------------|----|----|----|----|----|----|----|\n| Gemma-3-11B        | **0.63** | 0.89 | 0.97 | 1.00 | 1.00 | 1.00 | 1.00 |\n\nBoth Qwen-2.5-VL-7B and Gemma-3-11B form linear entity representations much earlier than LLaVA-1.5-7B, matching their stronger benchmark results. This supports that model size is **not** as important as data budget and/or timing of multimodal training, both of which help represent visual entities earlier and enable correct factual recall.\n\n---\n> **Based on your findings, what specific architectural modifications would you recommend to address the timing misalignment? Would earlier cross-modal fusion help?**\n\nWhile our findings show that native training or large-scale multimodal training can help, this is often not feasible for smaller labs or open-source projects.\n\nAnother idea is to utilize reasoning. If the VLM can articulate important entities robustly in LLM token space through reasoning before attempting to recall factual knowledge, it could fix the token-space misalignment. We tested reasoning as a lightweight fix across several models:\n\n| **VLM Model**        | **LLM (Base→CoT)** | **VLM (Base→CoT)** | **Δ Gap** |\n|----------------------|--------------------|--------------------|-----------|\n| LLaVA-MORE-8B        | 41 → +5 | 23 → +5 | –43.9 → –39.1 |\n| LLaVA-NEXT-8B        | 41 → +4 | 24 → +6 | –41.5 → –33.3 |\n| LLaVA-1.5-7B         | 30 → +5 | 19 → 0  | –36.7 → –45.7 |\n| LLaVA-1.5-13B        | 44 → 0  | 28 → +9 | –36.4 → –15.9 |\n| Pixtral-Large-124B   | 68 → +4 | 56 → +15| –17.6 → –1.3 |\n| Pixtral-12B          | 41 → +4 | 36 → +12| –12.2 → +6.6 |\n| Llama-3.2-Vision-11B | 36 → +6 | 31 → +9 | –13.9 → –4.7 |\n\nReasoning recovers much of the gap for Pixtral-12B/124B, Llama-3.2-Vision-11B, and LLaVA-1.5-13B, suggesting an inexpensive mitigation. However, we find that LLaVA-1.5-7B reasons poorly: it often over-describes details and never names the entity. We believe that reasoning—perhaps reinforced with RL—is a promising future direction.\n\n---\n> **What factors contribute to the 60 % of the performance gap that remains after patching? Have you explored combining your approach with other interventions?**\n\nWe tried patching middle-layer attention heads at the final token positions. Although this recovers a few additional examples, it lags behind the MLP approach, and combining both does not improve the score.\n\nOur main hypothesis is that end-to-end visual instruction tuning slightly alters some attention heads and MLPs that were part of the factual-recall circuit, so full recovery is impossible without re-training.\n\n---\n> **In the ~35 % of cases where VLMs successfully recall facts, what distinguishes these examples? Do certain types of entities or facts show better alignment timing?**\n\nIn our last patching experiment we found that, for examples answered correctly by the VLM, linear probes can already recover the entity from early-layer visual representations.  \n\nThis is further supported by the models (Gemma-3, Qwen-2.5-VL) that show less degradation and also allow probing for linear entity representations in earlier layers—supporting this as the main difference between examples where factual recall works and where it fails. Which entities precisely enjoy superior early-layer representations likely depends on the data used for visual instruction tuning and multimodal pre-training.\n\n---\n\nWe hope these clarifications and additional results resolve the reviewer’s concerns and strengthen the paper. Thank you again for the thoughtful feedback."}]

Author rebuttal:

Meta-review: This paper deals with multomodal knowledge retrieval and investigates poor performance of LLaVa-style VLMs in factual recall tasks.  The paper attributes this to the VLM architecture in whicb visual information is processed too late.  To counter this the paper treats this as a "two-hop" problem and shows that visual information takes a clean linear form in mid-to-late layers and injecting MLP activations from unimodal model into the VLM can help.

The paper received four positive reviews, albeit all borderline.  Authors are encouraged to integrate feedback from the reviews and rebuttal/discussion phase into the subsequent version of the paper.  The code and benchmark should be open-sourced as it seems it's not explicitly stated.
Decision: Accept (poster)