Title: Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval
Venue: NeurIPS 2025
Accepted: yes
Keywords: Mechanistic Interpretability

Vision Language Models

Multimodal Alignment
Abstract: Training vision language models (VLMs) aims to align visual representations from a vision encoder with the textual representations of a pretrained large language model (LLM). However, many VLMs exhibit reduced factual recall performance compared to their LLM backbones, raising the question of how effective multimodal fine-tuning is at extending existing mechanisms within the LLM to visual inputs. We argue that factual recall based on visual inputs requires VLMs to solve a two-hop problem: (1) forming entity representations from visual inputs, and (2) recalling associated factual knowledge based on these entity representations. By benchmarking 14 VLMs with various architectures (LLaVA, Native, Cross-Attention), sizes (7B-124B parameters), and training setups on factual recall tasks against their original LLM backbone models, we find that 11 of 14 models exhibit factual recall degradation. We select three models exhibiting high- and two models with low performance degradation, and use attribution patching, activation patching, and probing to show that degraded VLMs struggle to use the existing factual recall circuit of their LLM backbone, because they resolve the first hop too late in the computation. In contrast, high-performing VLMs resolve entity representations early enough to reuse the existing factual recall mechanism. Finally, we demonstrate two methods to recover performance: patching entity representations from the LLM backbone into the VLM, and prompting with chain-of-thought reasoning. Our results highlight that the speed of early entity resolution critically determines how effective VLMs are in using preexisting LLM mechanisms. More broadly, our work illustrates how mechanistic analysis can explain and unveil systematic failures in multimodal alignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates factual recall degradation in LLaVA-style Vision-Language Models (VLMs) compared to their underlying language models. The authors propose that this degradation stems from misalignment between visual projector outputs and pretrained textual embeddings, rather than insufficient computational depth as previously suggested. Through a systematic analysis involving cosine similarity measurements, a new benchmark, linear probing, and activation patching experiments, they demonstrate that visual entity representations emerge too late in the model's layers to engage early-layer factual recall mechanisms. The paper introduces the \"two-hop problem\" hypothesis: VLMs must first resolve visual inputs into entity representations before retrieving factual knowledge, but this resolution occurs after the critical early-layer recall mechanisms. Their patching experiments show ~40% recovery of the performance gap, providing causal evidence for their hypothesis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper provides a compelling alternative explanation for VLM factual recall degradation, shifting focus from computational depth limitations to timing misalignment between visual processing and factual recall mechanisms.\n\nThe work combines multiple complementary approaches - cosine similarity analysis, benchmarking, probing, and causal intervention through patching - providing converging evidence for their hypothesis.\n\nThe factual recall benchmark with 1,000 examples provides a valuable resource for future research, with careful design to isolate factual recall from visual recognition errors.\n\nThe cross-model patching experiments provide compelling causal evidence, showing ~40% performance recovery when early-layer representations are restored.\n\nThe paper is well-written with effective visualizations, particularly Figure 1 which clearly illustrates the two-hop problem concept.\n\nWeaknesses:\n\nThe paper only evaluates LLaVA-style architectures. It's unclear whether these findings generalize to other VLM architectures (e.g., cross-attn Flamingo, end2end chameleon, or more recent models).\n\nThe benchmark uses only ImageNet images and GPT-4.1 generated questions, which may not capture the full diversity of real-world factual queries about images. Also, how about object detection tasks?\n\nQUESTIONS:\nHave you tested or do you have hypotheses about whether this two-hop problem exists in non-LLaVA architectures like Flamingo that use different visual-linguistic integration mechanisms?\n\nWhat factors contribute to the 60% of the performance gap that remains after patching? Have you explored combining your approach with other interventions?\n\nIn the ~35% of cases where VLMs successfully recall facts, what distinguishes these examples? Do certain types of entities or facts show better alignment timing?\n\nBased on your findings, what specific architectural modifications would you recommend to address the timing misalignment? Would earlier cross-modal fusion help?\n\nHow does the two-hop problem manifest in larger VLMs? Does increased model capacity provide any natural mitigation?\n\nLIMITATIONS:\nThe authors adequately acknowledge that their study focuses on LLaVA-style models and suggest future work should examine other architectures. However, they could better discuss:\n\n* The potential bias introduced by using GPT-4.1 for question generation\n* The effectiveness and computational costs of their probing and patching approaches at scale\n* Potential negative impacts if their insights are used to develop VLMs that hallucinate more convincingly", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates why LLaVA-style Vision-Language Models (VLMs) underperform at factual recall compared to their underlying LLM backbones. Rather than blaming insufficient compute after the visual encoder, the authors argue that misaligned visual token embeddings bypass the early-layer MLPs responsible for storing world knowledge. Through a series of linear‐probe analyses and “attribution patching” experiments, they show that (a) visual information only takes on a clean, linearly recoverable form in mid-to-late layers, and (b) if you inject the early MLP activations from the unimodal LLM into the VLM, you recover almost 40% of the factual recall gap. Finally, they release a benchmark that systematically compares VLM vs. LLM recall on 1,000 ImageNet-derived questions. Their main takeaway is that current adapter architectures delay entity formation until it’s too late to hook into the LLM’s fast recall pathways, creating a “two-hop” bottleneck.\n\nSTRENGTHS AND WEAKNESSES:\nWell-structured narrative: hypothesis → misalignment proof → patching → probing → success/failure comparison.\n\nFocused solely on LLaVA variants; it’s unclear if patching tricks or adapter redesigns generalize to fusion-style VLMs.\n\nBuilds heavily on prior work without clearly delineating incremental vs. groundbreaking contributions\n\nQUESTIONS:\n1. How uniform is the difficulty across GPT-4.1-generated questions? Could some entities consistently yield easier or harder questions, skewing VLM vs. LLM gaps?\n\n2. Do findings hold for other adapter-based VLMs (e.g., Flamingo, InternVLM) or fusion architectures that interleave vision and language layers?\n\n3. The “greedy matching” to align entity positions in the VLM seems critical. Can you quantify how often it picks the correct slot vs. noise spots, and does that affect recovery magnitude?\n\n4. For the cases you exclude because the VLM misidentifies the entity, how often does that happen and does it correlate with certain image types?\n\n5. Have you tried patching at a sliding window of layers (e.g., layers 1–3, 3–5, etc.) to pinpoint the minimal “hop” window needed for a meaningful recovery?\n\nLIMITATIONS:\nThe authors acknowledge that their study is limited to LLaVA-style models and suggest future work should explore other architectures and scales. However, they do not address potential negative societal impacts of deploying misaligned VLMs (e.g., hallucinating harmful misinformation, misidentifying medical images).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the factual recall degradation in LLaVA-style Vision-Language Models (VLMs) compared to their language backbone. It attributes this degradation to the architectural design of VLMs, where visual information is processed too late to effectively engage the early-layer factual recall mechanisms of the language model. The paper introduces the concept of a \"two-hop\" problem, provides empirical evidence through linear probing and patching experiments, and proposes a benchmark to evaluate factual recall accuracy and knowledge hallucination in VLMs. It also demonstrates that patching early-layer MLP outputs from the language model into the VLM can significantly improve factual recall performance.\n\nSTRENGTHS AND WEAKNESSES:\n## **Summary Of Strengths:**\n\n1. The paper presents a clear and novel explanation for the factual recall degradation in VLMs by highlighting the architectural limitations and the \"two-hop\" problem. This provides a fresh perspective on why VLMs underperform compared to their language backbones in factual recall tasks.\n2. The paper support the hypothesis with a variety of empirical experiments, including linear probing, activation patching, and a newly introduced benchmark. These experiments help to validate the two-hop hypothesis and demonstrate the impact of visual-text misalignment in VLMs.\n3. The introduction of fact recall benchmarks can better evaluate the accuracy of fact recall and knowledge illusion, which is of great significance for improving the reliability of VLMs.\n\n## **Summary Of Weaknesses:**\n\n1. While the paper provides empirical evidence for the two-hop problem, it lacks a more thorough comparison with other possible explanations for factual recall degradation in VLMs. This makes it difficult to determine if the two-hop problem is the sole or primary cause.\n2. The models evaluated were small in size, and the authors should examine whether the behavior of larger models alleviates or exacerbates these problems.\n3. The paper's focus is primarily on LLaVA-style VLMs, which limits the generalizability of the findings. It would be beneficial to explore whether similar issues exist in other VLM architectures.\n\nQUESTIONS:\n1. The authors should provide more details about the experiment setup.\n2. The code and datasets are not included in the supplementary materials or publicly released.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the poor performance of multi-modal large models in factual recall tasks. First, the authors propose that the distributed representations of visual information across visual tokens in early layers bypass the factual recall mechanism residing in the early-layer MLPs of the LM backbone, which is the root cause of this phenomenon. Based on this analysis, they then patch early-layer MLP outputs from the LM backbone into the corresponding VLM layers, significantly improving factual recall performance. Finally, the authors introduce a benchmark to systematically evaluate factual recall accuracy and knowledge hallucination in multi-modal settings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The authors hypothesize that adapter misalignment is the root cause of factual recall degradation in VLMs, and gives the illustration of the Two-Hop problem.\n2. The manuscript is explicit and well-organized, and provides visualizations.\n3. Experimental validation is sufficient and solid. The proposed method achieves remarkably high accuracy, demonstrating impressive performance compared to baselines, as shown in Figure 5.\n\n**Weaknesses:**\n\n1. While the analysis in Chapter 2 provides illustrative insights, its exclusive reliance on LLaVA-series models raises concerns about generalizability. The observed phenomena may not necessarily hold for other architectural paradigms, requiring validation across more diverse model families.\n2. The authors conducted experiments only on the LLaVA-series architectures. Including evaluations on Qwen-VL and InternVL would further strengthen the confidence of the findings.\n3. The citation format in the paper has some minor issues. please check it.\n\nQUESTIONS:\n1. I'm intrigued by the inherent contradiction in VLMs: During pretraining, the primary objective is vision-language alignment, yet empirical evidence shows persistent token space misalignment. Would scaling up pretraining (e.g., with more data/compute) effectively mitigate this issue?\n2. Does the proposed activation patching technique impact inference speed? Quantitative metrics are required to evaluate its computational overhead.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}