Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs
Authors: Yaniv Nikankin
Dana Arad
Yossi Gandelsman
Yonatan Belinkov
Venue: NeurIPS 2025 poster ()
Keywords: mechanistic interpretability
vision language models
VLMs
Primary area: deep_learning

Abstract: Vision-Language models (VLMs) show impressive abilities to answer questions on visual inputs (e.g., counting objects in an image), yet demonstrate higher accuracies when performing an analogous task on text (e.g., counting words in a text).
We investigate this accuracy gap by identifying and comparing the circuits---the task-specific computational sub-graphs---in different modalities.
We show that while circuits are largely disjoint between modalities, they implement relatively similar functionalities: the differences lie primarily in processing modality-specific data positions (an image or a text sequence).
Zooming in on the image data representations, we observe they become aligned with the higher-performing analogous textual representations only towards later layers, too late in processing to effectively influence subsequent positions. 
To overcome this, we patch the representations of visual data tokens from later layers back into earlier layers. 
In experiments with multiple tasks and models, this simple intervention closes a third of the performance gap between the modalities, on average.
Our analysis sheds light on the multi-modal performance gap in VLMs and suggests a training-free approach for reducing it.

Same Task, Different Circuits: Disentangling Modality-Specift Mechanisms in VLMs
Yaniv Nikankin Dana Arad
Yossi Gandelsman"
Yonatan Belinkoy!
Technion - Israel Institute of Technology
UC Berkeley
Abstract
Vision-Language models (VLMs) show impressive abilities to answer questions on visual inputs (e.g., counting objects in an image), yet demonstrate higher accura- cies when performing an analogous task on text (e.g., counting words in a text). We investigate this accuracy gap by identifying and comparing the circuits- the task-specific computational sub-graphs-in different modalities. We show that while circuits are largely disjoint between modalities, they implement relatively similar functionalities: the differences primarily processing modality-specine data positions (an image sentations, we observe they become aligned with the higher- segher-performin analogous textual representations only towards later layers, too late in processing to effectively influence subsequent positions. To overcome this, we patch the representations visual data tokens from later layers back into earlier layers. In experiments with multiple tasks and models, this simple intervention closes a third of the perfor- mance gap between the modalities, on average. Our analysis sheds light the multi-modal performance gap in VLMs and suggests a training-free approach for reducing it.?
text sequence). Zooming in the image data repre-
(A) Different model components are used for vision and language
(B) Query and generation components are monality-agnostic
(c) Back-patching visual data representations improves performance
many "bonana" sequence/fiaage?
flower bonana banona
Data
Taker
Query
How
Gon
Query
mere
simal
choir flower
banana"
Generation
Data components are modality specific
many
Data
sing?
<mg
singe
Generation
Query
many
3X
Query
How
N/A
How
Figure 1: Overview of our analysis. (a) We find circuits for analogous vision and language tasks and show they are structurally disjoint-differeen different model components are responsible for each modality. (b) Swapping sub-circuits across modalities (shown for the language circuit, but applies similarly to vision) reveals that query and generation components preserve performance when swapped between modalities, while swapping data components degrades performance. (c) To address the performance gap, we apply mack-patching: re-injecting visual data activations from later layers into earlier ones. representations from deeper layers available during visual prompt
This makes processing, enhancing performance on visual tasks.
"Correspondence to: maniv.@@cs.technion
"Code and data available https://githuu http:///httt...cmmee/emaaaaaaaaa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Recent years have seen major progress in building vision-and-languaae models (VLMs) that can handle both image and text inputs (Li et al., 2025). However, significant performance gap persists between textual tasks and their visual counterparts (Fu et al., 2024; Wang et al., 2024a: Cohen et al., 2024). For example, models are considerably less accurate at counting objects in an image than at counting words in list (Figure la, Appendix C), or at identifying winner in a board game when it is supplied as an image rather than as a textual description (Fu al., 2024). Curiously, this performance gap persists even in VLMs with decoder that is jointly trained from scratch on both modalities (e.g. Team et al. (2024)). To address the performance gap between analogous image and text tasks in VLMs, one should first understand its source.
In this paper, we aim explain the performance gap in VLMs between analogous visual and textual tasks from structural and functional perspectives. First, we create a dataset of five diverse tasks, each comprising pairs of analogous textual and visual prompt variants (Figure 2). Each prompt consists of data, query, and generation positions, that contain the subject of the prompt (an image or short text), the task description, and the last token position, respectively. For example, the data for an object counting task can be a text listing objects ("car flower banana...") or an image with these objects, and the query for this task will ask "How many bananas are there?". We use this dataset extract and examine the circuits- the computational sub-graphs tee-grpps-rresposibbeee responsible for performing each of the variants. We measure the degree of structural overlap between these codallity-speciffe circuits, and find that vision and language tasks are performed by relatively disjoint VLM components (Figure la), with an average of only 18% components shared between modalities.
To analyze the functional differences of intermediate sub-circuits, we swap them between the two modalities, and measure how interchangeable they are for task performance (Figure lb). This reveals two main findings: first, despite being structurally disjoint, the sub-circuit that processes the query tokens, as well as the sub-circuit that generates the answer, are functionally equivalent between modallties--eadeee
to similar performance when swapped. Second, there is major functional difference in the processing of data tokens between modalities, resulting in much lower performance after swapping the relevant sub-circuit. Hence, we conclude that the difference in data processing is the leading cause behind the accuracy gap.
We utilize these findings to bridge the performance gap, without any additional training, by automat- ically intervening in the model's computation test time. Specifically, we patch the visual token embeddings from later layers of the moeel-where these embeddings are more aligned to their textual counterparts- to earlier layers (Figure 1c). This results in visual processing that is more aligned with analogous textual prompts, leading to higher accuracy on visual prompts. Our results suggest that understanding the differences in the mechanisms between modalities attainable, and provides a path to improving visual processing in VLMs.
2 Preliminaries
In this section we present our main unit of -model circuits. We discuss how circuits for specific tasks are discovered and evaluated. These discovery and evaluation techniques are used in Section 4 to identify the common and distinct sub-circuits for analogous tasks in different modalities.
2.1 Model Circuits
A circuit is minimal subset of intercomnected model components that perform the computations required for a specific task or prompt (Elhage et al., 2021). In our analysis, we consider circuit component to be either an entire attention head or an individual multilayer perceptron (MLP) neuron, at a specific output position. We follow the existing definition of MLP neurons (Geva et al., 2021; Nikankin et al., 2024), where the th neuron in an MLP block is the combination of the thth row vector of the first MI.P layer and the th column vector of the second MI. layer. A full definition for ML. neurons and circuits is given in Appendix A.

---

2.2 Circuit Discovery
To identify circuits for both textual and visual task variants, we employ causal analysis techniques that score the importance of each component's activation (Vig et al., 2020). For each textual or visual task, we divide its prompts into two subsets: one for discovery and another for evaluation (See Appendix B.3). From the discovery subset, for each prompt p with answer r, we select random counterfactual prompt p' that yields a different answer r'. Ideally, we would find the importance of a component for prompt p by intervening on its activation du.p with the counterfactual activation au,p', and calculate the difference logits of r and r' post-intervention (Wang et al., 2022), marked (D(r, r'I do(au.,p du.p')). However, this process becomes unfeasible due to the large amount of model components. Instead, we approximate this effect by applying attribution patching (Nanda, 2022) with integrated gradients (AP-IG; Sundararajan et al., 2017; Hanna et al., 2024):
R OLD (r, do (e
= = (e-
((u.p.P) = (avr'' au.p)
(I)
ddu.
where e and e' are the input embeddings for the prompts and p', respectively, and k = 5 is the number of integration steps, following Hanna al. (2024). This value first-order Taylor-series approximation to the logit difference, and can be calculated once, for all model components in parallel. The value is averaged across prompts and calculated per circuit component. higher value indicates greater component importance for the analyzed task prompts. Given a score for each component, we construct a eircuit e by selecting a specific percentage of the highest-scoring components.
2.3 Circuit Evaluation
We evaluate circuit с by measuring its faithfulness (Wang et al., 2022) the proportion of the full model's task performance explained by the circuit. For each prompt with answer r in the evaluation subset, we randomly choose counterfactual prompt p' with answer r' and compute the activations of the model for p'. We then pass p through the model while ablating all non-circuit components using the pre-computed counterfactual activations, as per standard procedure (Mueller et al., 2025). The intervention's 's effect is quantified by the logit difference between correct and counterfactual answers. denoted (r r'I do(Vu & c: Gu.p = (x,z)) )). This effect is normalized calculate the faithfulness of circuit с on task T (Mueller et al., 2025):
1 T
LD(r, r' d'o au.p au, )) (D,(),r')
F(c, 1)
LD(r, LDM((r,")
where LD(r, r") is the logit difference without ablations and LDv r') is the logit difference when all components in model M are ablated. This normalization typically constrains faithfulness to [0.0, 1.0] range.
3 Experimental Settings
To investigate the accuracy gap between textual and visual prompts in VLMs, we construct a dataset of five question-answer answering tasks, seen in Figure 2. Each task consists of query paired with data presented in one of two analogous formats: as either an image or a text. This analogous design ensures direct comparability for every image-based prompt, we create analogous text- based prompt that describes the same information. Our dataset contains the following tasks:
Object counting:
The model is given a textual sequence of objects or an image containing these objects. The task is to count how many objects of specific type appear in the data.
Two-operand Arithmetic: The model is given two-operand arithmetic calculation (e.g., "56 + 14"), either as text or as backkrbbccground image with the calculation written on it. The task is to complete the prompt with the correct answer.
Spatial Ordering: The model is presented with scene with colored objects, either as an image from the CLEVR dataset (Johnson et al., 2017) or as a text describing the scene. The task to identify the color of an object based on its position in the scene.
3

---

Object Counting
Arithmetics
Spatial Ordering (CLEVR) In with four objects a object, cyan object,
Factual Recall
Sentiment Analysis
Consider Eiffel Tower.
A child taughs joyfully, running through a
Textual Data
book bird bag book book book.
14 98
Visual Data
4 X 9
How mony "book"
What the result of the calculation?
What is the color of the first object from the right?
Which country is this iandmark locoted in?
Is this scene happy, sod, neutral?
Query
there?
Figure 2: Analogous Vision-Language Tasks. We create dataset of five question-answering tasks, each with a textual and visual variants. A task prompt is made up of query (bottom row) asked either about image (middle row) for the visual variant analogous text (top row) for the textual variant.
Factual Recall:
The model is presented with entity- either person landmark- by name for the textual variant, or by image for the visual variant. The task is to complete a specific property of the entity. For example, when presented with famous landmark, the task is to recognize which country is it located in.
Sentiment Analysis:
The model is presented with scene, either as image or as the caption of an image. The task is to identify whether the sentiment in the scene is happy, sad, or neutral.
These were selected to span diverse spectrum of tasks, and to incorporate both existing and generated data (i.e., factual recall uses real entity images while sentiment analysis and counting rely on generated images).
To perform circuit discovery these tasks, we enforce constraints the prompts. First, only use prompts where the answer a single word. Second, for consistency within a task, we align all prompts to positional template, such that each position in each prompt always contains the same type of token in it. (For instance, in factual recall, the first four tokens contain the entity name.) Prompts that do not align well the template are filtered out. These constraints are standard in circuit discovery (Vig et al.. 2020; Mueller et al., 2025). Prompt examples per task and further details on prompt generation are provided in Appendix B.2.
We analyze three transformer-based VLMs: Qwen2-7B- Owen2-7B-VLLInnstrus (Wang et al., 2024b), Pixtral-12B (Agrawal al., 2024), and Gemma-3-1 3-1--it (Team et al., 2024). All these models use an "adapter" architecture (Liu et al., 2023), where the VLM comprises of a pre-trained image encoder that converts sequence of visual token embeddings, each embedding to the language embedding space, and a language decoder. The decoder concatenates the visual embeddings with the of the textual prompt embeddings, and processes them to generate an answer. The models differ in their training schemes (c.g., Gemma-3's s decoder is pre-trained from scratch on visual prompts, whereas Qwen2-7B-VL's decoder initialized from the weights of text-only LLM), training data, and additional techniques deployed to process visual information. This variety makes our findings robust a wide array of settings.
input image patches
optional adapter layer that projects
To ensure we identify meaningful circuits within these models, we follow standard procedures (Mueller et al., 2025) and verify high VLM performance (substantially above-chance accuracy) each task. We report accuracies in Appendix C and focus on the more common case where models achieve higher accuracy textual variants compared visual variants.
4 Cross-modality Circuit Analysis
To investigate the performance gap, we aim to pinpoint the differences between the circuits used for analogous visual and textual tasks. Namely, we analyze both the structural and functional intersection between these cireuits. We start by identifying the circuits responsible for each combination of model, task, and modality.

---

Textual Task Patching Effects
Visual Task Patching Effects
20
20
20
15
10
Data (Text)
Query
Gen.
Data (Image)
Query
Gen.
Position
Position
Figure 3: Patching effects for Qwen2-7B-VL. for the textual (left) and visual (right) counting
task. We sum the patching effect (described in Section 2.2) across all model components for specific position and layer. This reveals different patterns of component importance by position, motivating the separation of each cireuit three sub-circuits data, query and generation.
First, we use the methods described in Section 2 to find the patching effect of each model component, estimating their importance for the task. For instance, Figure shows importance scores for counting task in Qwen2-VI, summed per position and per layer (other models and tasks are shown in Appendix D). Next, we construct a circuit from the set of top-p" percent of components and measure its faithfulness, repeating this process for different circuit sizes. Following earlier work (Nikankin et al.. 2024; Ameisen et al., 2025), a circuit is considered sufficient for a task if the minimal circuit that achieves faithfulness of over 80%. As Figure shows, the circuits discover obtain high faithfulness results for relatively small circuit sizes, indicating that we find the most important components for the analyzed tasks, in both textual and visual modalities." This allows to further focus on these circuits to analyze the structural (Section 4.1) and functional (Section 4.2) intersection between the modalities.
Owen2-VL-7B
Pixtral-12B
Gemma-3-122
thrcshold
Faithfulness
0.6
D.4
0.25
1%
10%
100%
1%
10%
100%
1%
10%
100%
Circuit size (% Components)
Circuit size (% Components)
Circuit size (% Components)
Textual Task
Counting
Arithmetic
ordering
Factual
Sentiment analysis
Figure 4: Circuit faithfulness across models and tasks. We measure the faithfulness each eircuit size, for each model, task and modality. The circuits we further analyze the minimal circuits that achieve faithfulness of over 80%.
4.1 Vision and Language Circuits are Implemented by Different Components
Given the discovered sufficient circuits for each textual and visual task variants, start by analyzing their structural intersection: how much component overlap is there between the two circuits?
Similarly to Kaduri et al. (2024), and motivated by Figure 3, we separate our prompts into three parts: data positions, query positions, and the answer generation position the last token of the prompt. The positions are consistent within each modality, as detailed in Section 3, but can differ between
0.01 *On average, the textual task circuits we find include 7% the components while the visual task circuits include 15% of the components. The larger amount of components necessary for visual tasks is expected since images have tokens, requiring larger number of position- components them all.
3p E (0.001,
1issis 100}

---

Owen2-VL-7B
Pixtral-128
Gemma-3-128
Normalized toU
0.8
0.6
0.4
Counting
Arthmetic
Color Ordering
Factual
Recall
Sentiment
Analysnn
Courung
Anthmetic
Color
Ordering
Facital
Recall
Sentinens
Analyats
Caunting
Arthmeti
Color
Ordering
Factual
Recall
Sentiment
Analyon
Data positions components
Query positions components
Generation position components
Figure 5: Normalized IoU scores. We measure the IoU between the component set the textual
and visual circuits for each model and task and normalize it using random baseline. We find that across models and tasks, the intersection of components in data token positions (blue) is close to zero and the intersection of components in query token positions (orange) is very low. The intersection of components in the last token position (green) varies between tasks.
modalities. Since circuit components position-specific, have to map the positions between the two modalities in order to evaluate their overlap. Query positions and generation positions are easily mapped, since they consist of the same textual sequence for both textual and visual inputs (e.g., "How many 'banana' are in the image/sequence" ?). A special case of alignment in the data positions, where single text position cannot be mapped to single image patch. Thus, do not consider data positions when calculating intersections for the data processing components (i.e., we only compare the layer and attention head index MLP neuron index). This is formally defined in Appendix A.3.
Consequently, we split each circuit to three sub-circuits, split by positions: data (cP), query (cQ) and generation (c sub-circuits, which each include the circuit components in the corresponding positions, such that с = c U U UCC, A formal definition is given in Appendix A.2.
We use intersection over union (IoU) to quantify intersection between textual and visual circuits. Since the positional mapping between modalities not one-to-one (due visual prompts containing special tokens), we calculate IoU bidirectionally to ensure accurate comparison. First, we map the textual circuit's positions to the visual domain and measure its IoU with the visual circuit. Then, we perform the reverse operation, mapping the visual circuit's positions to the textual domain and measuring loU with the textual circuit. These two measurements are averaged to produce our intersection metric:
1 C1. C n cvl Ucyl IoU (c,,cv) = C U (
cr. ncv Jcz Uc
+
where Cz, Cv are the textual and visual circuits, respectively. CL-+V and CV- the textual and visual circuits with their positions mapped to the other modality, respectively (formally defined in Appendix A.3). Due to the significant numerical imbalance between MLP neurons and attention heads, we calculate the intersection in eq. (3) separately for MLP neurons and attention heads, and average the results.
To account for potential bias from varying circuit sizes (larger circuits may yield higher loU scores), establish baseline using random circuits. For each model and task, create two circuits caL. CRV from a set of random model components, each containing the same amount of components as the textual and visual circuits, respectively. We normalize the calculated IoU of the textual and visual circuits using the random baseline and perfect loU score of 1.0:
NIoU (cL, IoU (cz, cv) IoU (CLV,CRV)
1.0 IoU (CLVCCRV
The normalized IoU results (Figure 5) reveal varying degrees of intersection across sub-cireuits. In data positions, the modality-specific sub-circuits exhibit virtually no intersection across tasks and models-an expected finding given their specialized function in processing distinct data token distributions. In query positions, there is low overlap between modalities (12% on average), despite operating on identical input query tokens. The generation position sub-cireuits show moderately higher intersection rates (38% average), reflecting these components' shared role in promoting

---

Owen2-VL-7B
Pixtral-128
Gemma-3-122
Interchang Carccaaggggg. Faithfuliness
0.8
Nermalized
Counting
Anthmettic
Color
Ordering
Factual
Recill
Sentiment
Anatyols
Counting
Artthmetic
Color
Ordering
Factual
Recall
Sentiment
Aralyals
Counting
Arttimetic
Color
Ordern
Factual
Recall
Sentiment
Anabysia
positions components
Query positions companents
Generation posibin companents
Figure Sub-cireuit interchange faithfulness. We measure the faithfulness of the textual and visual circuits while interchanging one of the sub-circuits with its parallel from the second modality. Across tasks, replacing the query (orange) or generation (green) sub-circuits between modalities has little effect on faithfulness, indicating shared functionality across modalities. In contrast, interchanging the data sub-circuit (blue) between modalities has a large negative effect on faithfulness.
consistent answers across modalities. Overall, these results indicate that while some components important in both visual and textual tasks, the different modalities utilize relatively disjoint circuits.
4.2 Query Processing and Answer Generation are Functionally Equivalent; Data Processing is Modality-Specifie
Despite being largely structurally disjoint, two distinct component sets may still implement similar functionality (Hanna et al., 2025). To measure the shared functionality of the textual and visual circuits, we build upon the partitioning of each circuit to three sub-cireuits (Section 4.1). We investigate whether these sub-cireuits are functionally interchangeable between the textual and visual cireuits- for instance, we replace el the sub-cireuit in the query positions of the language circuit, with of the sub-cireuit the query positions of the visual circuit, while maintaining high circuit faithfulness? Such interchangeability would indicate shared functionality, Formally, we interchange the query sub-cireuit between modalities and measure the average faithfulness (as in eq. (2)):
F(cL U UC V-L UCE T2) F(cD U U Lo U UCG TV).
(5)
We apply this process to data and generation sub-circuits as well. When interchanging data sub- cireuits, we ignore positional information (as in Section 4.1) and treat components active across all data positions. As in eq. (4), we normalize these scores using a random baseline as lower bound and the original faithfulness as upper bound. The random baseline is measured by interchanging sub-circuit with random components, and the upper bound is the
sub-circuit with an identically-sizee original faithfulness of the entire circuit (cy or cv).
Our results (Figure 6) indicate that in the query and generation positions, textual and visual sub-circuits interchangeable. In query positions, the visual sub-circuit can be replaced with position-aligned textual sub-circuit (and vice versa), while maintaining high faithfulness to the entire model. This high interchange extends to the generation sub-circuits, showing a functional equivalence. Given the low structural overlap (12% and 38% on average in the query and generation positions, respectively, shown in Section 4.1), this implies that different components at these positions implement similar functionality, indicating a form of redundancy. A different pattern emerges when interchanging the sub-circuits at the data positions: across all tasks, the faithfulness drops match the random baseline, indicating that the data sub-cireuits c and cß implement completely different functionality in textual and visual prompts. This indicates that the functional difference lies mainly the components responsible for processing the data tokens, prior to being processed by the rest of the model.
5 Improving Visual Tasks Performance
Since textual and visual circuits functionally differ mainly how they process analogous data tokens, examine the representations at these positions to address the accuracy gap.
7

---

Cosine Similarity
Qwen2-VL-78
Pixtral-128
Gemma-3-128
0.6
0.4
0.2
10
20
0
10
20
30
10
20 Layer
30
40
Layer
Layer
Counting
Arithmetic
Color ordering
Factual recall
Sentiment analysis
Figure 7: Similarity of visual token activations to analogous textual token activations. We compute cosine similarity between visual data activations and their analogous textual data activations. The maximum similarity for each visual token is selected and averaged across visual tokens. This similarity score increases deeper in the model, with varying rates across different models.
Recent work (Wu et al., 2024) has shown that as visual tokens move through multi-modal model, they gradually align with the highest-matching textual token (e.g. visual tokens of a cat gradually align with the embedding of the token "cat"). We build upon this and measure the alignment of whole image with its entire analogous textual prompt, across layers. We compute model activations for each analogous prompt pair. For each visual data token activation, find the highest cosine similarity among all textual data token activations the analogous prompt at the same layer. We average these to produce cross modoltto alignment score per layer.
We find that alignment between images and analogous texts gradually increases across layers (Figure 7). However, this increase is slow, with higher similarities only occurring later in the model, after information from data tokens moves to later positions (this information flow was shown in Kaduri et al. (2024), and is seen in Figure 3). We hypothesize that if these text-aligned representations were available carlier in the model's processing, it could improve performance on visual tasks.
To utilize the more textually-aligned representations of visual tokens, we employ (Biran et al., 2024; Lepori al., 2024): the process of patching the late representations back to earlier layers (Figure lc). For each visual prompt, we first perform a forward pass through the model and record activations a' for window of consecutive layers centered at lares specifically lave +i} where E {0, 2, determines the window size (1, 3, or 5 layers). We then perform another forward pass, but this time replace the activations at corresponding window centered at last (where last < (sre) with our previously recorded activations from around (arc- This replacement is performed parallel across all visual token positions. systematically explore this back-patching by varying the source layers arre {1 + 1,..., T. 1}
I E flure
model with layers, we
and destination layers last 10.1..... 1 L }}. while maintaining the same window size for both source and destination, and choose the best performing (ares last (reported in Appendix E.2).
To show statistical significance, the lower bound of the back-patched accuracy is higher than the baseline accuracy. We report the back- back-patched accuracies, the improvement compared to the baseline model accuracy, and standard deviations across resamples in Table
bootstrap re-sampling with 1000 iterations and verify that
Table 1: Back-patching visual token embeddings increases accuracy across tasks. We measure the improvement on visual tasks while back-patching visual token embeddings from later layers back earlier layers. We report the back-patched accuracies and standard deviations, as well the change from baseline model performance. Green marks statistically significant improvement. Back-patching shows increase from baseline accuracies (reported in Appendix C) across most settings.
Task
Qwen2-7B-VL
Pixtral-12B
Gemma-3-12B
Counting
75.0% (4.44%4444.999 1.9%)
(+2.2%
1.1%)
75.4% (+2.0% + 2.1%)
Arithmetic
76.9% (49.4% +1..4%+..339 1.3%)
29.4% (46.7% + 1.6%)
94.2% (+6.7% + 0.7%)
Color Ordering
74.8% (+0.6% + 0.6%)
84.7%
(44.1% 1.4%)
47.5% (+1.5% 0.9%)
Factual Recall
69.7% (+1.6% + 2.2%)
49.1% (44.1% 2.3%)
68.6% (+2.1% + 1.8%)
Sentiment Analysis
94.8% (+2.2% + 1.4%)
87.4% (+17.7% 2.2%)
98.0% (+5.4% + 0.9%)
8

---

Our results show that back-pan patchhng leads to an average absolute improvement of 4.6% on visual prompts, closing approximately 32% of the performance gap between visual and textual accuracy. Additional back-patching experiments are detailed in Appendix 1. and Appendix E.4.
To confirm that these improvements stem from leveraging textually-alinned representations rather than mere additional computation, we conduct a control experiment and apply similar back-patchin textual prompt variants. This control yields smaller accuracy gains in roughly 79% of cases (further shown in Appendix E.3), supporting our hypothesis that improved visual performance is more due to the early transition of visual tokens to eetuuall--aligned representations.
6 Related Work
Interpretability of VLMs. Interpretability research has examined VLMs' internal mechanisms, investigating how they integrate visual and textual inputs to generate textual responses, starting from early visual question answering models (Agrawal et al., 2016) and requence-to-sequenee (Chefer et al., 2021; Cao et al., 2020). Within this field, many studies have analyzed model weights and components and their effect on model outputs, both in discriminative VLMs (Gandelsman et al., 2025, 2024; Dravid et al., 2023; Goh et al., 2021) and generative VLMs (Kaduri et al., 2024; Zhang et al., 2024b; Jiang et al., 2024b; Neo et al., 2024; Luo et al., 2024; Yu & Ananiadou, 2024). Some studies have used causal analysis (Pearl, 2001), the study of causal mechanisms in computational graphs, in VLMS to identify key components in varied tasks (Li et al., 2022; Basu et al., 2024; Golovanevsky et al., 2024). Building on these, we identify model cireuits for a range of analogous tasks, and measure their structural and functional overlap. In LLMs, recent work shows high circuit overlap for similar tasks (Merullo et al., 2024; Zhang et al., 2024a; Mondorf al.. 2024; Hanna et al., 2025). We demonstrate this phenomenon occurs in VLMs at a different granularity by investigating position- based sub-circuits, and propose simple inference-time method to improve visual accuracy.
architectures
Multi-modal representations. Previous work has shown that representations from models trained on different modalities, like vision and language, can be aligned using transformations to build multi-modal models (Merullo et al., 2022; Koh et al., 2023). The VI.Ms we investigate implement similar approaches. However, since models perform differently across modalities (Fu et al., 2024; Cohen et al., 2024; Wang et al., 2024a; Ventura et al., 2024), it raises the question: how do these modalities align, and where do they differ? Recent studies have explored multi-modal alignment in different ways: showing a gap between textual and visual representations (Liang et al., 2022; Jiang al., 2024a), investigating how in-context tasks represented (Luo et al., 2024) and how semantically similar tokens are aligned (Wu et al., 2024). While these findings help us understand modality alignment better, they do not explain why accuracy varies across modalities. Building upon previous research that investigated a representational gap in contrastive VI.Ms (I (Liang et al., 2022) and its correlation with accuracy (Schrodi et al., 2024), propose a way to modify under-performing visual inputs so they can better match textual representations and improve model accuracy.
7 Conclusions
Our research explores why VLMs perform differently when processing the same tasks on textual or visual data. We create a dataset of five analogous tasks in both modalities to investigate this accuracy gap. We discover that even models trained simultaneously on both modalities develop separate circuits for each. Despite being separate, the circuits are functionally equivalent for the most part-when processing query tokens and generating the answer-but functionally differ when processing data tokens. Using these findings, we utilize a simple test-time method, back potchpathhnng to increase VLM accuracy visual prompts and reduce the accuracy gap between modalities.
Our results demonstrate that visual tokens benefit from textual alignment through further model processing. This points to the benefit of flexible processing for different tokens in VLMs. Relatedly, Geiping et al. (2025) presented a new paradigm of test-time compute scaling, where models use a varying amount of layers per prompt. Future work can extend this and explore the benefit of varied computational resources per token, particularly for visual inputs which may require more processing than text.

---

8 Limitations
We focused on question-answeriing tasks involving single images, where the image appears prior to the query. Further work could explore multi-image scenarios, different prompt order, or more complex and visually-inclined reasoning tasks, where the model is expected to perform better on visual representations (e.g., navigating through a visual environment). We limited our scope to simpler tasks with single-token completions due to two key constraints. First, current open models struggle with more complex visual reasoning. making mechanism analysis less meaningful in these cases, and second, current circuit analysis methods can only be applied to single-token completions.
Our experiments show that patching visual token representations from later to early layers closes 32% of the performance gap between analogous visual and textual prompts. Yet. the causes for the remaining gap remain unclear, suggesting potential for further visual processing improvements.
Finally, we focus our analysis adapter-based VLMs, this is the current state-of-the-ar approach for aligning vision and language. Our findings may not immediately apply to models utilizing different fusing approaches, which we leave for future work.
Acknowledgments
We thank Tal Haklay for her feedback on this project, as well as Simon Schrodi, Jiahai Feng. and Tomer Ashuach for their helpful comments on drafts of this paper. We thank the anonymous reviewers of this paper for providing useful feedback. This research was supported by an Azrieli Foundation Early Career Faculty Fellowship and Open Philanthropy. DA supported by the Ariane de Rothschild Women Doctoral Program. YG is supported by the Google Fellowship. This research was funded by the European Union (ERC, Control-LM, 101165,. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
References
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 1955-1960, Austin, Texas, November 2016. Association for Computational wwtee//org//////000
Linguistics. doi: 10.18653/v1/D16-12200 URL
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint a7X00:2410.07073, 2024.
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL https: //transformer: wwrccittt...... wttiitts..uu///00 pub/2025/ attribution atrr.bbuiiaa-gggesss/netthhtt html.
Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, and Daniela Massiceti. Understanding information storage and transfer in multi-modal large language models. arXiv preprint
2024.
Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models on multi-hop queries. arXiv preprint arXiv:2406. 12775, 2024.
Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained ridion-and-langguag models. In Computer -cion-ECCVV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pp- 565-580. Springer, 2020.
10

---

Hila Chefer, Shir Gur, and Lior Wolf. Generie attention modell explainability for interpreting bi-ml and encoder-decoder transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 397-406, October 2021.
Ido Cohen, Daniela Gottesman, Mor Geva, and Raja Giryes. Performance gap in entity knowledge extraction across modalities vision language models. arXiv preprint arXiv:2412. 14133, 2024.
Amil Dravid, Yossi Gandelsman, Alexei Efros, and Assaf Shocher. Rosetta neurons: Mining the common units in model zoo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1934-1943, 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/transffommr-iiccct htte:auisspppp222222//ammooommm html..
Deqing Fu, Ruohao Guo, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama. Robin Jia, and Willie Neiswanger. Isobench: Benchmarking multimodal foundation models on isomorphic representations. arXiv preprint arXiv:2404.0/266, 2024.
Yossi Gandelsman, Alexci A Efros,, and Jacob Steinhardt. Interpreting CLIP's image representation via text-based decomposition. In The Twelfth International Conference on Learning Representations, 2024. URL htRphtttss///peeeeee
www. orrm? www.rrunnnnnnnanaaaeeea
Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting the second-order effects of neurons in CLJP. In The Thirleenth International Conference on Learning Representations, 2025.
URL
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprini arXiv:2502.05/71, 2025.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021.
Gabriel Goh, Nick Cammarata +, Chelsea Voss Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi:
http::/ddittillpuub/Z2 http://22//mmmiioallllllllao
Michal Golovanevsky, William Rudman, Vedant Palit, Ritambhara Singh, and Carsten Eickhoff. What do vlms notice? a mechanistic interpretability pipeline for gaussian-noise-five text-image corruption and evaluation. arXiv preprint arXiv:2406. Ttt.:2406..632000 16320, 2024.
Yash Goyal, Tejas Khot, Douglas Summers- Stanmer--Sty,.... Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In First Conference on Language Modeling. 2024.
Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Are formal and functional linguistic mechanisms dissociated? arXiv preprint arXiv:2503. 13302, 2025.
Evan Hernande7, Arnb Sen Sharma, Tal Haklay, Kevin Meng. Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. arXiv preprint 230.022308.09/22 2023.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card, arXiv preprint arXiv:2410.21276, 2024.
11

---

Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang. Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 77036-27046, 2024a.
Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language representations to mitigate hallucinations. CoRR, 2024b.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2901-2910, 2017.
Omri Kaduri, Shai Bagon, and Tali Dekel. What's in the image? deep-dive into the vision of vision language models. arXiv preprint arXiv:2411. 2024.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. International Conference on Machine Learning, pp. 17283- 17300. PMLR, 2023.
Black Forest Labs. Flux. http:://ggtthhhc.o////bbeeeeeeee
2024.
Michael A. Lepori, Michael Mozer, and Asma Ghandeharioun. Racing thoughts: Explaining large language model contextualization errors. ArXiv, abs/2410.02102, 2024. URL https:
//api. www.stiimmooolteeeee.. org/ Trrg/CorpusID... :
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learming, pp. 12888-12900 PMLR, 2022.
Zongxia Li, Xiyang Wu, Hongyang Du, Fuxiao Liu, Huy Nghiem, and Guangyao Shi. A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges, 2025. URL attpp://ont //axivv../iiv////a org/ abs//b/1..22011 02189.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 00211762-176633 2022.
Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay attention to MLPs. Advances in neural information processing systems, 34:9204-9215, 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3499 2023.
Grace Luo, Trevor Darrell, and Amir Bar. Task vectors are cross-modal. arXiv preprint -xXiv:4110.22300. 2024.
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint arXiv:2209. 250.:2299./162, 2022.
Jack Merullo, Carsten Fickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models. The Twelfth International Conference on Learning Representations, 2024.
Philipp Mondorf, Sondre Wold, and Barbara Plank. Circuit compositions: Exploring modular structures in transformer-based language models. arXiv preprint arXiv:2410.01434 2024.
Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Pootto-Kaufman Tal Haklay, Michael Hanna, et Mib: A mechanistic interpretability benchmark. arXiv preprint arXiv:2504.1315// 2025.
Neel Nanda. Attribution patching: Activation patching industrial scale, 2022. URL https: // www.:.neelaanda... io/ htt////channiiiicccm preecaasttc--eepppeeaal//ttttttr patchhi
Neel Nanda and Joseph Bloom. Transformerl.ens. http://///gggaooooo///////ttataa TransformerLens 2022.
12

---

Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing vision-language models. arXiv preprint arXiv:2410.07149, 2024.
Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without algorithms: Language models solve math with bag of heuristics. arXiv preprint arXtv:2410.21272. 2024.
Judea Pearl. Direct and indirect effects. Direct and Indirect Effects, 2001. URL httpttt/ftt..... ucla. www/pub/ aeaa_ssr//Rae pdf.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Simon Schrodi, David Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap. object bias, and information imbalance in contrastive vision- lasiun-language models. arXiv preprint 1.Xi::240400798. 2024.
Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey A Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, et al. Paligemma 2: family of versatile vlms for transfer. CoRR, 2024.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pp. 3319-3328, PMLR, 2017.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint 20X0::2408.00/ 18, 2024.
Mor Ventura, Michael Toker, Nitay Calderon, Zorik Gekhman, Yonatan Bitton, and Roi Reichart. Nl-eye: Abductive nli for images, 2024.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388- 12401, 2020.
Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is picture worth thousand words? delving into spatial reasoning for vision language models. In A. Globerson, L. Mackey, D. Belgrave, Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Syxtems, volume 37, pp. 75392-75421. Curran Asso- ciates, Inc., 2024a. URL https: ftte//Scccee13344000022226666660 oorp:a/eeeeeeeooooooeeereeeeeeee pdf.
neurips cc /aperr.p..... paaaer_fill wwwes/ p221///
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: A circuit for indirect object identification in GPT-2 small. In The Fleventh International Conference Learning Representations, 2022.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12/9/, 2024b.
Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, and Yoon Kim. The semantic hub hypothesis: Language models share semantic representations across languages and modalities. arXiv preprint arXiv:2411.044 044986, 2024.
xAI. Grok- 1.5 vision preview., 2024. URL
Zeping Yu and Sophia Ananiadou. Understanding multimodal 1lms: the mechanistic interpretability of 1lava in visual question answering. arXiv preprint arXiv:2411.10950 2024.
13

---

Ruochen Zhang, Qinan Yu, Matianyu Zang, Carsten Eickhoff, and Ellie Pavlick. The same but different: Structural similarities and differences in multilingual language modeling. In The Thirteenth International Conference Learning Representations, 2024a.
Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova, Cross-modal information flow in multimodal large language models. arXiv preprint arXiv:2411.18620, 2024b.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: abstract lines 1-2 are an introduction, lines 3-9 are reflected in section abstract lines 10-16 are reflected in section 5. The intro is extended version of these two main line groups
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
The last section includes limitations paragraph, mainly dealing with our experimental settings, which are the simple case and can be extended to more complex cases.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The full experimental settings are described in the appendix, and the code is added as supplementury.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We make the code available and will publish the datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ weaiiccgguie///eeee httee//iie///Sssiiiieeeeeeeeeeee Satps//ooSSuubbimia wwt.sSuwwwmssssooooooooooooooooe
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // //nips.cc//// ppwplcc/guidessseepppppppppopaee http///gguuieee/eeeuuuimiiiiee (CodeSubmissiooooooll.loooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We write all experimental details across the paper, and specifically in the Appendix that deals with Experimental Details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: All of experiments are performed across all available data in single exccution. Thus, we have no standard deviation results to report.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This is described in the Experimental details appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://mmtt /tte::////eurpeee cc cctppclccc///c/eccccccccccccllct Bttpai///ehcccuieelliiiaee
Answer: [Yes]
Justification: The paper does not deal with any research that might deviate from the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We do not publish any new models / technologies that can be used in malicious ways.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We not release any models, and believe there no risk publishing our dataset, which can be used mainly for specific types of analysis, and contain only data of types that already exist in public datasets. We manually verify all generated images to avoid publishing any unsafe images.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In our dataset, we partially rely on the existing CLEVR dataset, which is properly cited. In our "Factual Recall" task, we rely on publicly available images, that are collected from set of images with free usage licences only.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The assets introduced in the paper are the analysis dataset and code. The dataset is detailed in the appendix. Both dataset and code will be published.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not deal in research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not deal in research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
We describe the usage of LLMs for the curation of some annotations in our dataset in the Experimental details appendix. The LLM outputs were manually verified by the paper's s authors to avoid any labeling errors.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp://eeeuu..eeeeeee for what should or should not be described.
21

---

A Formal circuit definitions
A.1 Definition of MLP Neurons
Our definition of MLP neurons as circuit components follows existing literature (Nikankin et al., 2024), and is provided for completeness. the VLMs we analyze, the MLP layer of a transformer block is implemented by a Gated MLP (Liu et al., 2021). This MLP at layer can be described by the following equations:
h hypsst = o(hnn W o (h'n, )
(6)
h = h post W out
(7)
where h hbuu Rd hppatt E Rdmip are the MLP input, MLP output and post-non-lineartyy representations of the MLP block, respectively. W Wis W out Rdmipxd w gate RR xd are weight matrices, and a is non-linearity function, is Hadamard product, and biases are omittee.
In this formulation, we define the activation of the Th MLP neuron in layer as the n scalar value in the intermediate representation h This scalar is multiplied with the th row of W out- to produce the neuron's contribution to the output of the entire MLP block.
A.2 Definition of circuits
A circuit с is formally defined as set of components:
c = ((1pp.) IeN.PEN,iEN)
where each component is represented by a 3-tuple (4, i) denoting the layer, position, and index respectively. Each component represents either attention head (where i denotes the attention head index within layer 7) or an MLP neuron (where denotes the neuron index within layer I, as defined in Appendix A.
When decomposing a circuit into its data (cP), query (c?), and generation (cS) sub-circuits, the circuit components are partitioned according their position indices, such that c U U Formally, this decomposition is defined as:
c² ((l.p.i) PD)
Q = =clpp Pq]
c = ((1,p,))
where Pp. Pq. and Po are disjoint sets of position indices corresponding to the data, query, and generation regions respectively. our case, the generation region includes only the last input position.
A.3 Definition of circuit component mapping
A positional mapping is function M N P(N) that maps each position in textual task prompt to one or more positions in a visual task prompt containing equivalent information, and vice versa to map visual positions textual positions (marked M-1). We manually define the positional mapping for each task and model combination. For an individual circuit component (1, i), the mapping transforms it to set of components in the target modality:
M ((1, p. (( I M(p))
For example, in the object counting task with Qwen2-7B-V LL, the mapping function M maps textual position 23 to visual position 96, where both contain the first token of the query ("How"). The complete positional mappings for all tasks and models are provided in the code accompanying the paper.
To compute our similarity metrics (Equation (4), Equation (5)) over query and generation sub-circuits, map sub-cireuit from one modality to another (e.g. e.vv) by applying the mapping function to
22

---

each component individually. Formally, for query (c generation sub-cireuit (c²)) we obtain the mapped sub-circuit:
CL-V :ov= = U M((I,,,))
(1p..))c
The treatment data sub-circuits differs because a positional correspondence does not always exist between modalities in the data positions. For instance, in the sentiment analysis task, general words the textual sequence may not map to specific visual tokens, but rather convey an overall sentiment atmosphere that is distributed across the entire visual input. Therefore, when computing similarity metrics for data sub-circuits c., we discard positional information and represent components as 2-tuples (1, i).
B
Experimental and Technical Details
B.1 Prompt examples
Table 2 presents prompt examples for each task in both input modalities.
Table 2: Textual and Visual Prompt Examples per task
Task Name
Textual Prompt Example
Visual Prompt Example
Object Counting
"Sequence: book tree book cup cup ball tree. How many "tree" are in the sequence? Answer in a single number"
"How many "tree" are the image? Answer in single number."
Arithmetic
"Question: 10$48. What the result of the given arithmetic calculation? Answer in single number"
10.40- "What is the result of the given arithmetic calculation? An- swer in single number."
Spatial Or- dering
"In a scene with four objects ar- ranged horizontally, there is green object, a yellow object, a yellow ob- ject and cyan object. What is the color the third object from the left? Answer in single word."
"What is the color of the third object from the left? Answer in a single word."
Factual Recall
"Consider Dennis Rodman. What sport does this athlete play? Answer in a single word."
"What sport does this athlete play? Answer in single word."
Sentiment Analysis
"A child runs through a field of daisies, laughter echoing in the sun- shine, feeling the ...". Is this scene happy, sad, or neutral? Answer a single word."
"Is this scene happy, sad, or neutral? Answer in a single word." "
The amount of prompts per task is shown in Table 3. The amount of prompts for the same task and modality varies between models due to tokenization considerations (e.g. all object colors in the spatial ordering task must be tokenized to single token for positional alignment purposes- promptss that contain multi-token object colors are filtered out).
The full list of prompts will be published.
23

---

Table Prompt amounts for each model, task and modality.
Owen2-7B-VI
Pixtral-12B
Gemma-3-122
Task
I.
V
I.
V
I.
V
Counting
1524
383
1524
334
1524
383
Arithmetic
1000
1000
1000
1000
1000
1000
Spatial Ordering
1865
1925
472
497
1865
1925
Factual Recall
416
1265
454
1265
453
1265
Sentiment Analysis
232
245
237
245
222
245
B.2 Tasks Generation Details
In this section we describe in detail the prompt generation process for each task and modality in our dataset.
Object Counting: We use a base list of 30 possible object types (e.g. "banana", "fork", "book"). Each textual prompt contains seven randomly sampled objects drawn from up to four different object types-a configuration chosen balance task difficulty while maintaining high VL.M accuracy. For the visual prompts, use SD3-XL (Podell al., 2023) generate images containing the same amount objects described the text. Due to limitations in image generation models' ability to produce exact object counts, we manually verify each image, and filter out images that don't 't match the required object counts.
Arithmetic: For the textual variant, each prompt consists of two two-digit operands, ensuring positional alignment between different prompts (c.g., the second position always includes the singles digit of the first operand). This consistency is important because all analyzed models tokenize numbers into separate digits. For the visual variant, we create white- bhite-background images (75x338 resolution) displaying the identical arithmetic calculations in large black font centered in the image, maintaining simple presentation that focuses entirely on the calculation itself.
Spatial Ordering: For the visual variant, the images taken from the CLEVR dataset (Johnson et al., 2017). Each image depicts scene with four colored objects. We select this specific object count it provides optimal difficulty tradeoff-scemte with three objects prove easy for models, while scenes with five objects result in significantly lower model accuracy. The textual variant is generated out of these scenes. Each textual prompt depicts CLEVR colored objects, mentioning only their colors without shapes, to maintain positional alignment (as different shapes (e.g. "sphere") get tokenized into varying numbers of tokens).
with four
Sentiment Analysis: We use ChatGPT (Hurst et al., 2024) to generate 120 scene descriptions for each santiment-happy, sad, neutral, and manually drop scene descriptions that don't convey the target sentiment. To maintain positional alignment in the textual variant, we truncate each description exactly 20 tokens and pad the description with " Scene descriptions shorter than 20 tokens are filtered out. We use a scene token length of 20 since most scenes are longer than it, and we found it convey the sentiment of the scene well enough all cases. For the corresponding visual prompts, we use Flux -Schnell (Labs, 2024) to generate images based on these scene descriptions, conducting additional manual filtering to exclude any images that don't adequately match the described scene or fail to convey the intended sentiment.
Factual Recall:
We use three prompt templates from Hernandez et al. (2023):
1. "Consider [X]. What sport does this athlete play? Answer in a single word."
2. "Consider [X]. Which country is this landmark in? Answer in a single word."
3. "Consider [X]. What instrument does this person play? Answer in single word."
For the textual variant, we select entities (to replace the [X] in the prompt, e.g. "Michael Jordan")) that, when combined with the prefix "Consider [X]", result in exactly 5 tokens to ensure positional alignment. We use this number as it results in the highest number of possible entities in the given

---

dataset. For the visual variant, we collect entity images with public licenses from Wikimedia Commons. We rank 10 potential images per entity using CLIP (Radford et al., 2021) scores and select the best match for each entity. We resize and center-crop each image (to 256x256 resolution). We manually review the images remove any that contain "shortcut" hints (e.g., any image of basketball player holding basketball), to ensure models rely on factual recall rather than visual cues. In both variants, the counterfactual prompts for each template are sampled only from within the same template.
B.3 Circuit Discovery and Evaluation
For our circuit discovery experiments (detailed in Section 2 and Section 4), we allocate each modality- specific task dataset with 75/25 split: 75% of the prompts for discovery and 25% of the prompts for faithfulness evaluation. We divide prompts between the discovery and evaluation subsets such that all possible answers are distributed in the same ratio between the subsets.
When answers span multiple tokens (e.g. in the arithmetic task, where answers can be comprised of multiple digits), we measure the patching effect (Equation (1)) solely on the first token. We verify that the answers for each prompt and counterfactual prompt (r and r') differ the analyzed first token.
Our circuit discovery experiments use fork of the TransformerLeens library (Nanda & Bloom, 2022), in which we implement patching code for VLMs. This fork is available in our code release.
B.4 Compute Resources
Our experiments were conducted on an NVIDIA L40 node equipped with 8 GPUs, each containing 48GB of memory. Peak memory consumption occurred during circuit discovery operations on the Gemma-3-128 1-2ma---12BInsttuuee model, that required the parallel use of GPUs. The complete experimental suite, including studies not featured in the final paper, consumed roughly 200-300 GPU hours.
C Task Accuracies
In Table 4 we report the accuracies of models on each of the tasks. We observe that across most tasks, VLMs achieve higher accuracy the textual variant than the visual variant, motivating our analysis into this performance gap. This is compatible with earlier results on closed-source VLMs (Fu et al., 2024).
Table 4: Model Accuracies for textual and visual task variants.
Qwen2-7B-Vi L
Pixtral-12B L
Gemma-3-12 3-12B
Task
V
L
V
Counting 79.3%
70.8%
49.3%
66. %
89.3%
73.4%
Arithmetic
99.2%
67.5%
25.7%
22.7%
99.0%
87.5%
Color Ordering
86.8%
74.2%
77.4%
80.6%
76.1%
46.0%
Factual Recall
73.5%
68.1%
83.4%
45.0%
80.7%
66.5%
Sentiment Analysis
97.4%
92.6%
98.3%
69.7%
98.7%
92.6%
D Additional Circuit Discovery Results
In Table we report the circuit size, in percentage of the model's component count. As described in Section 4, the circuits chosen to be the minimum-sized circuit with faithfulness of over 80%. The faithfulness achieved by each circuit on each task and modality is reported in Table 6.
Furthermore, in Figure 10, Figure 1 11, Figure 12, Figure 13, Figure 14 we present the patching importance scores for each model and task, summed per position and layer. These figures are complementary Figure 3.
25

---

Table 5: Circuit sizes, in percentage of components out of the model's whole component set.
Qwen2-7B-VI.
Pixtral-12B
Gemma-3-12H
Task
I.
V
I.
V
I.
V
Counting
5%
8%
7%
20%
5%
20%
Arithmetic
1%
3%
6%
7%
2%
5%
Color Ordering
2%
8%
5%
20%
2%
7%
Factual Recall
2%
5%
20%
20%
Sentiment Analysis
2%
10%
20%
20%
10%
20%
Table 6: Circuit faithfulness scores for textual and visual task variants.
Qwen2-7B-VL
Pixtral-12B
Gemma-3-
Task
V
L
L
V
Counting 82.9%
82.0%
81.4%
82.7%
88.0%
89.7%
Arithmetic
86.0%
81.9%
82.6%
80.3%
80.9%
83.9%
Color Ordering
83.6%
81.2%
82.4%
84.1%
82.8%
81.3%
Factual Recall
82.6%
82.7%
80.7%
94.6%
82.3%
92.8%
Sentiment Analysis
88.2%
80.4%
92.4%
80.7%
85.3%
82.1%
E Back-Patching Ablations and Extended Results
E.1 Iterative Back-patching
Extending the back-patching experiments in Section we explore additional processing of visual data tokens yields further improvements. To do that, we perform back-patching iteratively (back- patching more than once). After identifying the optimal layers lere: last and layer window size for each model and task, perform back-patching iteratively. Namely, after each patching of the window centered at latts we continue the forward pass to calculate the window centered at lare- and re-patch the window centered at last. This process is repeated up 10 times. As shown in Figure 8, performance degrades after the first back-patching iteration across all but one task and model pairs, with each subsequent iteration further reducing model accuracy. We attribute this decline to representations becoming increasingly out-of disttiibbittno with each iteration, making them progressively less compatible with the model's learned parameters.
E.2
Best Layers For Back-Patching
For each model and task, Table shows the values of lress last and the layer window size that lead to the highest back-natching results, reported in Table 1.
These optimal layers match our hypothesis across models: replacing visual data token representations at model layers when they are less coxtually-alignee with more textually-alignedl representations,
Qwen2-VL-7B
Pixtral-12B
Gemma-3-122
Accuracy
Back-patching iteration
Back-patching iteration
Back-patching iteration
Counting Arithmetic
Color ordering
Factual recall
Sentiment analysis
Figure 8: Iterative back-patching results. Applying back bacchpatching several times in the highest- performing settings leads to a decrease in accuracy after the first back-patching application.
26

---

Table 7: Best back-patching settings. Each cell presents the lare, lss and layer window size (in parenthesis) that leads to the highest back patching accuracy.
Task
Qwen2-7B-VL
Pixtral-12B
Gemma-3-128
Counting
20 15 (3)
28 23
(5)
36 + +5 (3)
Arithmetic
1177 (3)
+
10
(5)
32 -5 (1)
Color Ordering
18 + 15 (1)
32 + 22
(5)
34 34-20
24 (1)
Factual Recall
11 + 6
(5)
32 + 18
(5)
34 +
20
(5)
Sentiment Analysis
15 + (5)
37 + 19
(5)
23
+
(5)
yields the highest accuracy improvements. This pattern persists even Gemma-3- 12B, where visual data tokens demonstrate high aligmment with textual analogs throughout most of the model (as shown Figure 7): tasks showing the greatest aceuracy gains from back-patching (counting, arithmetic, and sentiment analysis, detailed in Table 1) benefit from patching in the earliest layers where the alignment of tokens with textual analogs relatively low.
E.3 Back-Patching Control Experiments
This section presents our back-patching control experiment results across models and tasks. The accuracy increases from back pack-pat visual data tokens (reported in Table 1) can either stem from earlier eextually-alignee representations or from additional compute. To isolate the effect of adding computational layers, we conduct the same back- back-patching analysis on analogous textual prompts. In this control experiment, the alignment with textual tokens is inherent, so any accuracy increase task and layer window
must stem from added compute. Across each back patckkaathing setting-mode size-we check for how many values of lre and last is the accuracy increase caused by back-pate hack-patching larger than the control accuracy increase. Our findings (Figure 9) show that in most settings, visual back-patching yields greater accuracy improvements than the control for majority of layer values. notable exception occurs in factual recall tasks for Qwen2-VL. VL-7B and Gemma-3-12B, where back-patching shows minimal accuracy gains and is outperformed by the control in over half the cases. These results support our hypothesis that improved visual performance stems primarily from the early transition of visual tokens into textually-alignae representations, rather than from increased computational depth.
E.4 Back-Patching General VQA
To demonstrate that back-patching generalizes beyond our constructed dataset in broader visual question-answerte we evaluate its impact on model accuracy using the VQAv2 (Goyal al., 2017) and RealWorldQA (xAI, 2024) datasets, two common VQA benchmarks. We measure the accuracy of each VLM on each dataset without back-patching as a baseline, and measure it again when applying back-patchin (as described in Section 5). We present the results in Table 8. For VQAv2, the accuracy of each analyzed VLM on 3000 randomly sampled visual prompts (ensuring no image repetition). For RealWorldQA, limit the analysis to this amount of prompts due to computational requirements of exploring different back-pate beck-patching settings. For both datasets, also evaluate the mean and standard deviation of the accuracy by bootstrapping for 1000 iterations with a full sample size. In both datasets, following our experimental setup for other tasks (Appendix B.3, we sample the model for single forward pass, forcing an immediate answer. While this leads to generally lower results on the benchmarks, this setup necessary due the need search for the berf-performing back-patching settings, and ensures consistency across our experimental framework.
measure
measure the accuracy on the entire dataset of 765 prompts. We
The results confirm the average improvements observed in Section 5. Since the visual prompts in VQAv2 and WorldQA do not have textual analogs, we do not perform the control experiment in this setting.
E.5 Back-patching across model scales
To evaluate and how back-patching visual data representations affects accuracy across model scales, we perform back patchint on scales of the PaliGemma2 VLM suite (Steiner et al., 2024): 3B, 10B
27

---

Table 8: Back-ngg results on general VQA benchmarks. patching leads to significant increase in accuracy in general VQA benchmarks as well.
Dataset
Setting
Qwen2-7B-VL
Pixtral-12B
Gemma3-12B
Baseline
65.0%
62.2%
65.7%
VQAv2
Back-patching
68.2% + 0.9%
66.2% + 1.1%
70.3% + 1.3%
Baseline
58.8%
58.3%
59.1%
RealWorld
Back-patching
61.6% = 1.0%
60.1% + 1.2%
60.0% + 0.8%
and 28B parameters. We evaluate the models on our dataset five tasks, and measure the mean accuracy and it's standard deviation by bootstrapping for 1000 iterations with full sample size. The baseline accuracies are presented Table 9. The back-patched accuracies are reported in Table 10 and show that across most model scale and task pairs, back-patchie increases model accuracy, by aligning visual token representations with their analogous textual representations earlier the model.
Table 9; Baseline PaliGemma2 accuracies.
Model Size
Counting
Arithmetic
Color Ordering
Factual Recall
Sentiment Analysis
3B
64.0%
0.7%
15.4%
32.7%
48.7%
10B
66.0%
8.7%
14.0%
78.7%
56.7%
22.0%
6.7%
12.0%
30.7%
10.7%
Table 10: Back-patching accuracy increase across model scales. We evaluate how back-patching affects the accuracy of the PaliGemma2 VLM suite three model sizes.
Model Size
Counting
Arithmetic
Color Ordering
Factual Recall
Sentiment Analysis
3B
+1.3% + 2.9%
+31.6% + 3.8%
+8.1% = 3.4%
+6,5% + 3.9%
+7.6% + 3.9%
10B
+3.4% + 3.1%
+88.1% + 4.1%
+11.9% + 3.4%
+17.5% + 4.0%
+16.1% + 3.6%
28B
+4.2% + 3.6%
+21.1% + 3.8%
+33.5% 2.9%
+5.3% + 4.0%
+15.3% 3.6%
A note-worthy finding in the results is that the accuracy increase caused by back-patching is the largest in the 10B model size. We hypothesize this caused by the overall lower performance the 3B and the 28B model size. According to Steiner et al. (2024), the lower performance at the 28B size likely happens because this model scale is trained from scratch, as opposed to the smaller scales that result of model distillation.
28

---

Qwen2-VL-7B
Pixtral-12B
Gemma-3-12B
100 B.P. www.Adaantgg.... g...Advantg.....................
pack-patching percenlage
beats
50
Advantaa..
Counting
Arithmetic
Color ordering
Factual recall
Sentiment analysis
Layer window size:
Figure 9: Back- patching control results. We measure the percentage of lares last values for which the control accuracy increase is lower than the back patching accuracy increase. most model, task and layer window size combinations, back-patching shows higher accuracy increase (above the 50% line) compared to the control results.
29

---

Textual Task Patching Effects
Visual Task Patching Effects
25
20
Layer
10
Data
Query
Data (Image)
Query
(Text)
Position
Position
(a) Patching effects for Qwen-7B-VL for the textual (leff) and visual (right) counting task.
Textual Task Patching Effects
Visual Patching Effects
35
30
0
20
Layer
20
15
5
Data
Query
Data (Image)
Query
(Text)
Position
Position
(b) Patching effects for Pixtral-12B for the textual (left) and visual (right) counting task.
Textual Task Patching Effects
Visual Task Patching Effects
45
45
40
40
35
35
30
Layer
5
Data (Text)
Query
Data (Image)
Query
Position
Position
(c) Patching effects Gemma-3-128 the textual (left) and visual (right) counting task.
Figure 10: Patching effects for the counting task. The vertical lines split between data, query and generation positions.
30

---

Textual Task Patching Effects
Visual Task Patching Effects
25
25
Layer
10
Data
Query
Data (Image)
Query
(Text)
Position
Position
(a) Patching effects for Qwen-7B-VL for the textual (left) and visual (right) arithmetic task.
Task Patching Effects
Visual Patching Effects
35
30
Layer
20
5
Data
Query
Data (Image)
Query
(Text)
Position
Position
(b) Patching effects Pixtral-12B for the textual (left) and visual (right) arithmetic task.
Textual Task Patching Effects
Visual Task Patching Effects
45
45
40
40
35
35
30
25
20
15
Data
Query
Data (Image)
Query
(Text)
Position
Position
(c) Patching effects Gemma-3-12R the textual (left) and visual (right) arithmetic task.
Figure 11: Patching effects for the arithmetic task. The vertical lines split between data, query and generation positions.
31

---

Textual Task Patching Effects
Visual Task Patching Effects
25
20
Layer
Layer
10
Data
Query
Data (Image)
Query
(Text)
Position
Position
(a) Patching effects for Qwen-7B-VI. for the textual (left) and visual (right) spatial ordering task.
Textual Task Patching Effects
Visual Patching Effects
35
30
30
Layer
20
15
5
Data
Query
Data (Image)
Query
(Text)
Position
Position
(b) Patching effects for Pixtral-12B for the textual (left) and visual (right) spatial ordering task.
Textual Task Patching Effects
Visual Task Patching Effects
45
45
40
40
35
35
30
Layer
5
Data
Query
Data (Image)
Query
(Text)
Position
Position
(c) Patching effects for Gemma-3-128 for textual (left) visual (right) spatial ordering task.
Figure 12: Patching effects for the spatial ordering task. The vertical lines split between data, query and generation positions.
32

---

Textual Task Patching Effects
Visual Task Patching Effects
25
20
Layer
10
Data
Query
Data (Image)
Query
(Text)
Position
Position
(a) Patching effects for Qwen-7B-VI. for the textual (left) and visual (right) factual recall task.
Textual Task Patching Effects
Visual Patching Effects
35
30
30
Layer
20
20
15
5
Data
Query
Data (Image)
Query
(Text)
Position
Position
(b) Patching effeets for Pixtral-12B for the textual (left) and visual (right) factual recall task.
Textual Task Patching Effects
Visual Task Patching Effects
45
45
40
40
35
35
30
30
25
20
10
5
Data (Text)
Query
Data (Image)
Query
Position
Position
(c) Patching effects for Gemma-3-12B for textual (Ieft) and visual (right) factual recall task.
Figure 13: Patching effects for the factual recall task. The vertical lines split between data, query and generation positions.
33

---

Textual Task Patching Effects
Visual Task Patching Effects
25
20
Layer
Layer
10
Data
Query
Data (Image)
Query
(Text)
Position
Position
(a) Patching effects for Qwen-7B-VL. for the textual (left) and visual (right) sentiment analysis task.
Textual Task Patching Effects
Visual Patching Effects
35
30
30
Layer
20
20
15
5
Data
Query
Data (Image)
Query
(Text)
Position
Position
(b) Patching effects for Pixtral-12B for the textual (left) and visual (right) sentiment analysis task.
Textual Task Patching Effects
Visual Task Patching Effects
45
45
40
40
35
35
30
25
20
10
5
Data
Query
Data (Image)
Query
(Text)
Position
Position
(c) Patching effects Gemma-3-128 textual (left) and visual (right) sentiment analysis task.
Figure 14:
Patching effects for the sentiment analysis task. The vertical lines split between data, query and generation positions.
34


Reviews:
[{"review_id": "cbn5Z7KZxa", "reviewer": "Reviewer_NpXT", "summary": "This paper investigates the performance gap between vision and language tasks in VLMs by examining the underlying circuits for analogous problems (e.g., counting objects in an image vs. words in a text). The authors use causal analysis to identify task-specific circuits, revealing that vision and language modalities rely on structurally disjoint components. Despite this structural separation, they find that the sub-circuits responsible for processing the query and generating the answer are functionally interchangeable. The critical difference lies in the initial data integration, which is modality-specific. The authors hypothesize that the performance gap arises because visual representations align with their textual counterparts too late in the model's processing depth. As a remedy, they leverage back-patching -- a training-free intervention that injects these late-stage, text-aligned visual representations into earlier layers, to meaningfully reduce the performance gap.", "questions": "These are points I was curious about that could be interesting to discuss in the paper:\n\n* The authors may want to check out recent work by Campbell (2024) which looks at VLM failures on similar tasks from the perspective of the \"binding problem\" in cognitive science, particularly regarding parallel vs. serial processing. In many ways, this paper's findings could be interpreted as showing that VLMs lack the deep, sequential processing pathways for vision that are naturally afforded by language. I am speculating here, but I would be interested to hear how the authors' work ties into that literature.\n*  It would be very interesting to see the results of an experiment where the order of the query and the data in the prompt is swapped. This would likely change how information is integrated throughout the network and could impact the degree to which circuits are interchangeable and back-patching is effective.\n* Figure 7 shows a different pattern of activation similarity for Gemma compared to the other models. Its visual-textual alignment is high from very early layers. Could this be due to Gemma being trained from scratch on multimodal data? This seems to contradict the general hypothesis that the model first aligns image data into a textual representation. Despite this, back-patching is still effective for Gemma. A discussion of this would be a welcome addition to the paper.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\nIn general, I think this paper is very strong. The research question is well-motivated, the methodology for identifying and comparing circuits is compelling, and the proposed back-patching intervention is both simple and effective. The work builds upon a rich and relevant body of knowledge and doesn't try to reinvent the wheel. My review should be caveated with the fact that I am not an expert in mechanistic interpretability, so I have assumed that the application of methods like AP-IG and the faithfulness metric are consistent with standard conventions in the field.\n\n**Weaknesses**\n\nI would strongly shy away from framing the analysis around Marr's three levels. While I am a fan of Marr's framework, its introduction here (Line 140) seems to open a Pandora's box without adding significant value to the paper's core results. The paper doesn't elaborate on what Marr's levels are or how the structural and functional roles of circuits map onto that hierarchy. I assume the authors intend to map their analysis onto the implementational and algorithmic levels, respectively, but I don't believe it's that clean of a mapping. In fact, one could argue the entire analysis resides at the implementational level (i.e., identifying that different hardware implements the same algorithm/function). Removing the reference would improve the paper's focus and sidestep potential debates that detract from the empirical results.\n\nThe writing is not the most polished and seems rushed, which is a shame given the nice results. For instance:\n\n* On line 72, the notation $LD(r,r')$ is used before the term logit distance is explicitly stated. Introducing the term first would improve readability.\n* On line 83, the authors begin referring to circuits as $c$ without explicitly defining the notation (e.g., \"a circuit $c$ is a set of components...\"). This should be formalized.\n* There is a quotation mark bug on line 163.\n* The bar order in Figure 6 doesn't seem to match the legend, which can be confusing for the reader. While a quick check suggests they might be in the correct order, ensuring this is maximally intuitive is important. Furthermore, Figure 3, which illustrates the motivation for the sub-circuit analysis, is introduced very early and might be better placed closer to its detailed discussion in the text to improve the paper's flow.", "was_revised": true, "final_justification": "I thank the authors for addressing my concerns. I enjoyed reading the paper, and believe that the changes the authors plan to make will make the paper stronger. For these reasons, I will maintain my positive rating.", "rebuttal": "Thank you for your thoughtful and constructive feedback!\n\nWe're glad to hear that you found our paper to be very strong, with the motivation, methodology and impact of our work coming through clearly.\n \nTo address your comments and questions:\n\n**1. \"I would strongly shy away from framing the analysis around Marr's three levels\"**\n\nIn mentioning Marr’s levels of analysis, our intent was to loosely connect structural and functional circuit analysis with the implementational and algorithmic levels, respectively (as you correctly assumed).\nHowever, we acknowledge that this mapping is ambiguous and not essential for understanding our findings. We will remove this reference entirely in the revision to maintain focus.\n\n**2. “The authors may want to check out recent work by Campbell (2024)”**\n\nWe appreciate the reference to Campbell et al. 2024 [1], which we agree seems very relevant to our findings. Our work highlights that visual data representations align with their textual counterparts too late, which can be seen as a form of representational interference. Back-patching effectively injects aligned representations back into early layers of the model, functionally reducing this interference, similar in spirit to introducing more serial processing.\n \n**3. \"It would be very interesting to see the results of an experiment where the order of the query and the data in the prompt is swapped.\"**\n\nWe agree that reversing the order of the query and data might affect how information is integrated and may impact interchangeability and back-patching performance. However, some current VLMs are typically trained with the query following the image-text input, so reversing the order constitutes a distributional shift. While Qwen and Gemma 3 models do allow this reversal with some accuracy, we chose to focus on the more prevalent setting to match that of Kaduri et al., 2024 [2].  We will add this point to our limitations section.\n \n**4. \"different pattern of activation similarity for Gemma…Could this be due to Gemma being trained from scratch on multimodal data?\"**\n\nThis is a great question, which we discuss in Appendix E.2. While Figure 7 shows that Gemma’s activation similarities increase in earlier layers than other models, the best back-patching layers (presented in Appendix E.2) match this pattern, showing that in most tasks, back-patching achieves the best performance when patching embeddings from higher-similarity layers back into lower-similarity layers (where in Gemma3, these layers indeed come earlier in the model, perhaps due to it being trained from scratch on multimodal data, as you suggest). We will add an additional reference to this discussion appendix around lines 233-234 to make it more clear.\n \n**5.** We appreciate the detailed suggestions for improving the clarity of the paper, and we will address these issues in the revision of the paper. Specifically - we will improve the definition of logit difference in line 72, fix quotation mark bugs, and make the order of bars in figure 6 more intuitive by showing a minimal bar when the faithfulness is near-zero so the order will be clear.\n \n\nWe hope our answers are satisfactory. Please let us know if you have any other questions or concerns.\n\nReferences:\n\n[1] D. Campbell, Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem, 2024\n\n[2] O. Kaduri et al., What's in the Image? A Deep-Dive into the Vision of Vision Language Models, 2024"}, {"review_id": "OPo0xgGIPb", "reviewer": "Reviewer_KJ8j", "summary": "The authors investigate why vision–language models (VLMs) are consistently less accurate on visual than on textual versions of the same task. Based on prior works on circuit analysis (minimal subset of connected model components required for a specific task), the authors extract the circuits across three VLMs. The analysis shows that vision and language circuits are largely structurally disjoint—yet the sub‑circuits handling the query and answer generation are functionally interchangeable; the accuracy gap stems mainly from modality‑specific data‑processing components. In addition, visual token representations align with their textual analogues in late layers. Based on these observations, the authors propose an inference-time method that \"back-patches\" these late-layer visual activations into earlier layers, and demonstrate its effectiveness experimentally.", "questions": "**Rationale for circuit‑level analysis**\n  - The study follows the circuit‑extraction procedure of Mueller et al. (2025), isolating a minimal, interconnected sub‑graph that is causally sufficient for each task. Could you clarify why this circuit‑based lens is preferable to more common attribution techniques—e.g., gradient‑ or perturbation‑based importance scores—when diagnosing modality gaps? What limitations of those alternatives (faithfulness, resolution, inter‑head interactions, etc.) motivated your choice? Or do you view circuit extraction as just one of several equally valid approaches?\n\n**Evaluation beyond toy settings**\n\n - Counting, spatial ordering and factual recall are established tasks, but the visual sentiment analysis and two‑operand arithmetic are not realistic. Overly templated captions rarely occur in practice. Likewise, restricting prompts to single‑word answers and short captions departs from real VQA workloads.\n\n - Can authors evaluate the effectiveness of back-patching on a couple of common VQA benchmarks that have paired realistic image-texts (e.g., object counting can also be done based on images with dense captions such as [1])? \n\n[1] Urbanek et al., A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions, CVPR 2024", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength\n- The empirical findings of the paper are interesting and show new insights.\n- The ablation studies and analysis part are carefully designed.\n\nWeakness\n- I am concerned about the validity of the findings beyond simple toy datasets. All experiments are run on highly simplified, synthetic datasets. Tasks such as two‑operand arithmetic or sentiment judgments over short, templated captions have little real‑world relevance, and the captions themselves are far simpler than those found in practical VQA settings. As a result, it is unclear whether the conclusions—and the proposed late‑layer “back‑patching” fix—would hold on more complex images, free‑form captions, or open‑ended answers.\n\n- Prompting setup is also restricted. Circuit discovery is performed only on prompts whose ground‑truth answer is a single word, further distancing the evaluation from realistic multimodal QA scenarios.\n\n- The writing of the paper needs to be improved for better clarity and accessibility. In particular, the Preliminaries Section could be made significantly more accessible. For example, \n\n  - Formula (1) appears without motivation (e.g., why k=5?) or derivation, and obscures rather than illuminates the Hanna et al. (2024) method.\n\n  - Key terms such as “textual task,” “visual task,” and how prompts are split into “discovery” and “evaluation” subsets are introduced without definition or concrete examples. It would be better to expand this paragraph. \n\nI would suggest authors to polish these sections, add illustrative examples, and remove unnecessary formalism.", "was_revised": true, "final_justification": "After discussing with the authors and reviewing other reviewers' opinions, I believe the writing of the original paper needs significant improvement for clarity. Multiple sections could be made more accessible. I would be disappointed if the manuscript were to remain largely unchanged. However, it would be great if the authors make the modifications they promised in the rebuttal.\n\nMy major concern is that the applicability of the method is very restricted to cases where the answer is a single token, and prompts need to be arranged in a very specific way for analysis (L117–123). All experiments in the paper are conducted on relatively simple, toy-like visual datasets (e.g., simple object counting, sentiment analysis). This also explains why the baseline performance on more **realistic tasks**, such as RealWorldQA and VQAv2, is much lower than publicly reported results. \n\nFor this reason, I take the claimed effectiveness of the back-patching method with a grain of salt. In their rebuttal, the authors claim this limitation is a fundamental issue in circuit analysis and not something they aim to solve in this work. Nevertheless, more experiments on real-world datasets are necessary to properly justify the empirical contribution of this paper.\n\nThat said, I agree with other reviewers that the analysis itself is interesting. I hope this justifies my rating: \"3: Borderline reject – Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good analysis.\"", "rebuttal": "Thank you for the feedback! \n\nWe appreciate you finding our findings interesting and insightful, as well as mentioning the thoroughness of our ablations.\nWe address your comments and questions one by one:\n \n**1.** Regarding your comments and questions on evaluation beyond toy settings:\n\n**\"Can authors evaluate the effectiveness of back-patching on a couple of common VQA benchmarks that have paired realistic image-texts (e.g., object counting can also be done based on images with dense captions such as [1])?\"** \n\nThank you for this suggestion. We have performed an evaluation of back-patching on general vision-question answering using the VQAv2 dataset [9] and the more up-to-date RealWorldQA [10] dataset, two common visual-question answering benchmarks. In the following table, we report the results, showing that *back-patching textually-aligned visual embeddings is effective even in general VQA prompts*:\n\n| | Qwen2-7B-VL | Pixtral-12B | Gemma3-12B |\n|---|---|---|---|\n| VQAv2 Clean | 65.0% | 62.2% | 65.7% |\n| VQAv2 Back-patched | 68.2% ± 0.9% | 66.2% ± 1.1% | 70.3% ± 1.3% |\n| RealWorldQA Clean | 58.8% | 58.3% | 59.1% |\n| RealWorldQA Back-patched | 61.6% ± 1.0% | 60.1 ± 1.2% | 60.0% ± 0.8% |\n\nThe RealWorldQA results are calculated on the entire dataset. The VQAv2 results are calculated as the average of 3000 randomly sampled visual prompts (without image repetition). We limit the analysis to 3000 prompts due to computational requirements of exploring different back-patching settings. Bootstrapping was performed (similarly to the settings in the response to reviewer 1QgH) for 1000 iterations with full sample size to validate statistical significance.\n\nWe will add these results to the appendix and update the code in the revision.\n\n**\"Restricting prompts to single‑word answers and short captions departs from real VQA workloads\"**,  \n**\"Circuit discovery is performed only on prompts whose ground‑truth answer is a single word\"**\n \nWhile this limits analysis approach, this stems from an inherent limitation of circuit discovery methods (as we mention in lines 121-122). Such methods (as in [1,2,3,4,5], and more) are all limited to identifying model components crucial to the completion of a single token. Extending these methods to analyze the completion of a complex, multi-token completion isn't trivial and *is an active research question in mechanistic interpretability*. Additionally, our circuit intersection analysis relies on the positional alignment of query tokens, making it less feasible to be applied on prompts with different-length questions.\n\nWhile this limits the capabilities of *all* circuit discovery research, we follow standard procedures ([3,6,7,8] and more) to isolate the behavior we are localizing, such that it could be expressed in a single token containing the answer, and will allow us to identify the key components for each task, which we further analyze.\n\nWe will add a discussion on this point to a revised limitations section.\n \n**2. Rationale for circuit‑level analysis: \"Could you clarify why this circuit‑based lens is preferable to more common attribution techniques—e.g., gradient‑ or perturbation‑based importance scores—when diagnosing modality gaps?\"**\n\nThis is an interesting question. Could you provide specific references or methods? As “gradient- or perturbation-based importance scores” could refer to different approaches.\n\nIf you mean input attribution techniques (identifying influential input features), these are valid and provide a different perspective for investigating modality performance gaps. We chose circuit extraction because it reveals the computational pathways that process different modalities, allowing us to characterize the role of different model components and how they interact. \nThis structural lens enabled our key analyses: examining circuit intersections (Section 4.1) and testing component interchangeability between modalities (Section 4.2) to identify similar or different functionalities. \nBy isolating causally sufficient subgraphs, we can understand how different modalities utilize shared versus distinct computational mechanisms.\n\nIf you mean component attribution techniques (scoring individual model components), we should clarify that our approach actually incorporates these methods. We use attribution patching with integrated gradients [1,4], which is activation-based and gradient-based, to score component importance and build sufficient circuits from high-importance components.\n\nWe view our circuit-based analysis as one rigorous lens among several valid methodological approaches. If you have any further questions in this manner, we would love to elaborate.\n\n \n**3.** Writing comments:\n\n**\"Formula (1) appears without motivation (e.g., why k=5?) or derivation, and obscures rather than illuminates the Hanna et al. (2024) method.\"**\n\nThank you for pointing this out. We wish to clarify: Formula (1), showing the importance metric we measure per model component, was taken almost as-is from Hanna et al. [4], and is a standard circuit discovery metric. The number of integrated gradient steps (k) is chosen to be 5 in the original work [4] to be a good tradeoff between accuracy and runtime.\nPer your comment, we will add a relevant section in the appendix to expand on this method instead of merely citing it.\n \n**\"Key terms such as “textual task,\"visual task,” and how prompts are split into “discovery” and “evaluation” subsets are introduced without definition or concrete examples. It would be better to expand this paragraph.\"**\n\nRegarding textual vs visual tasks - we present a quick example of textual and visual tasks in the abstract (e.g., an analog pair of a visual and textual tasks is \"counting a specific object in an image\" versus \"counting a specific word in a word sequence\"), but we realize this should be re-iterated. We will add such an example to the intro and connect to it in the caption of Figure 2.\n\nRegarding discovery vs evaluation subsets - we explain the division to discovery and evaluation datasets in Appendix B.3. We will add a reference to it to Section 2.2., where these split is first presented.\n \nPer your request, we will polish the preliminaries section to improve its presentation and remove unnecessary formalism.\n \n.\n\nWe hope our updated results, answers and fixes are satisfactory. Please let us know if you have any other questions or concerns.\n \n References:\n\n[1] N Nanda, \"Attribution Patching: Activation Patching At Industrial Scale\", 2022.\n\n[2] F Zhang, et al., \"Towards Best Practices of Activation Patching in Language Models: Metrics and Methods\", 2023\n\n[3] K Wang, et al., \"Interpretability in the wild: a circuit for indirect object identification in gpt-2 small\", 2022\n\n[4] M Hanna, et al., \"Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms\", 2024\n\n[5] E Ameisen, et al., \"Circuit Tracing: Revealing Computational Graphs in Language Models\", 2025\n\n[6] M Hanna, et al., \"How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model\", 2023.\n\n[7] N. Prakash et al., \"Language models use lookbacks to track beliefs\", 2025.\n\n[8] A. Stolfo et al., \"A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis\", 2023.\n\n[9] Y. Goyal et al., \"Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering\", 2017.\n\n[10] X.AI. Grok-1.5 vision preview. https://x.ai/blog/grok-1.5v, 2024a. 9, 15\t\n\n[11] A. Mueller et al., \"MIB: A Mechanistic Interpretability Benchmark\", 2025"}, {"review_id": "ctaTt6qp6u", "reviewer": "Reviewer_1QgH", "summary": "This paper investigates the accuracy gap for VLMs between the image-to-text inputs on tasks like counting objects/words or arithmetics by means of circuits, which are task-specific computational sub-graphs. They show that these circuits are disjoint between modalities primarily for data positions– especially so for Qwen2 & Pixtral – by means of the “Faithfulness” metric that uses counterfactual prompts and that in later layers the alignment between representations between text and visual tasks increases. Based on this finding, they propose Back-patching of visual tokens into earlier layers of the VLM, which can be applied post-hoc and show that this simple step improves performance on Counting, Arithmetic, Color Ordering, Factual Recall and Sentiment Analysis tasks for Qwen2, Pixtral, and Gemma2.", "questions": "For the most crucial points I would like to see addressed in this paper before giving a more beneficial rating, I refer to Weaknesses. \n\n- How does the back-patching of visual tokens perform for VLMs with very strong multimodal capabilities like Pali-Gemma [1,2]? \n- Would the authors expect the back-patching to lead to a higher focus of VLMs on image modality than on text modality as reported in, e.g., [3,4,5]? \n    - An experiment showing this explicitly would greatly increase the value of the paper, especially if the compute tradeoffs are not beneficial for the back-patching.\n\n**Citations**\n- [1] PaliGemma: A versatile 3B VLM for transfer\nsrc: https://arxiv.org/pdf/2407.07726 \n- [2] PaliGemma 2: A Family of Versatile VLMs for Transfer\nsrc: https://arxiv.org/pdf/2412.03555\n- [3] DO VISION & LANGUAGE DECODERS USE IMAGES AND TEXT EQUALLY?\nHOW SELF-CONSISTENT ARE THEIR EXPLANATIONS?\nsrc: https://arxiv.org/pdf/2404.18624\n- [4] Visual cognition in multimodal large language models\nsrc: https://www.nature.com/articles/s42256-024-00963-y \n- [5] WHY CONTEXT MATTERS IN VQA & REASONING: SEMANTIC INTERVENTIONS FOR VLM INPUT MODALITIES\nsrc: https://arxiv.org/pdf/2410.01690", "limitations": "With the exception of the Limitations I stated in the Weaknesses section, the limitations are discussed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "## Strengths\nThe idea to improve the performance of VLMs by means of patching vision tokens into earlier layers of the VLM is, to my knowledge, novel and represents an elegant solution to improve the performance of VLMs for vision tasks by allowing the adapter to make better use of the powerful LLM it connects to.\n- They also show that this does improve performance, even for models like Gemma-3, which showcases a much stronger cosine similarity as the multimodal aspect is much more enforced for these models than for the other 2 VLMs (Qwen2 and Pixtral).\n\nThe paper is clearly written, well-formatted, and presents its idea well, as indicated by:\n- Overview Figures 1 & 2 give immediate intuition about the main concept of the paper.\n- All figures have clear, descriptive descriptions with highlighted takeaways.\n- Use of italics to highlight key messages\n\nRich in information:\nThe appendix in Supplementary Material contains a lot of relevant information and is well organized.\n- Code in Supplementary Material is present and exhaustive (note: please make the README a little bit more descriptive)\n\n\n## Weaknesses\n**Evaluation does not incorporate scaling of models and compute cost** Evaluation is performed on 3 adapter models with one fixed size. One of their main findings is that image tokens from the adapter are assimilated by the LLM too late, so the natural question arises of whether this issue still persists for larger VLMs. It is crucial for potential practitioners to understand where back-patching is useful. Further, their patching approach increases the computational cost of inference, so this needs to be weighed against just using a larger model.\n\nIf it turns out that larger models are more efficient by having a better performance vs. compute-tradeoff than the proposed method, this significantly changes the contribution from a solution to a very interesting finding that requires more research before application.\n\nTo address this concern: Perform an evaluation where the same model family with different sizes is compared against back-patching, comparing the performance improvements and evaluating the inference cost.\n\n**No Statistical Significance**  The authors correctly state in the checklist that neither error bars nor statistical significance are used.\n However, their justification would be valid for almost any machine learning paper and, therefore, more of an excuse: “All of our experiments are performed across all available data in a single execution. Thus, we have no standard deviation results to report.”\n\nEspecially since it is possible to obtain error bars by means of bootstrapping in case one does not wish to rerun the experiments and/or better use multiple seeds during evaluation for the generation over the entire dataset. \n\nTo address this concern: Show that the performance improvements using back-patching are statistically significant (e.g. by one of the previously mentioned methods).\n\n**Nitpicks:**\n- Please add a reference to the absolute Task Accuracies in Appendix C to the table description in Table 1. \n- There appears to be something off with the format of the PDF (e.g., citations not clickable, text not highlightable); please fix this for the final version of the paper.", "was_revised": true, "final_justification": "The rebuttal convincingly addresses my main concerns. The authors demonstrate via bootstrapping that the improvements from back-patching are statistically significant in nearly all cases, strengthening the robustness of their claims. They also extend evaluation to the PaliGemma2 family across multiple scales, showing consistent gains and providing a plausible explanation for the unexpected 28B results. Finally, they clearly position back-patching as a proof-of-concept guided by mechanistic insights, rather than a ready-for-application method. With these clarifications, I believe the paper makes a meaningful and well-supported contribution, and I raise my score.", "rebuttal": "Thank you for your valuable comments and feedback!\n\nWe appreciate you recognizing the novelty, elegance and generality of our interpretability-guided improvement to VLM performance, and for finding our paper clear and well-presented. We address your comments one by one:\n\n**1. \"No Statistical Significance results…To address this concern: Show that the performance improvements using back-patching are statistically significant\"**\n\nThank you for bringing this up. Per your suggestion, we show that back-patching leads to statistically significant performance improvement by running a bootstrapping experiment (1000 iterations, full sample size). The results, across all but one combination of model+task, show high significance confidence. The full bootstrapping results are in the following table (showing the clean accuracies, taken from Appendix C, and the bootstrapped mean and std of the post back-patching accuracy):\n\n| Model | Counting | Arithmetic | Spatial Ordering | Factual Recall | Sentiment Analysis |\n|---|---|---|---|---|---|\n| Qwen Clean | 70.9% | 67.6% | 74.2% | 68.1% | 92.6% |\n| Qwen Back-patched | 75.0% ± 1.9% | 76.9% ± 1.3% | 74.8% ± 0.6% | 69.7% ± 2.2% | 94.8% ± 1.4% |\n| Pixtral Clean | 66.1% | 22.7% | 80.6% | 45.0% | 69.7% |\n| Pixtral Back-patched | 68.3% ± 2.2% | 29.4% ± 1.6% | 84.7% ± 1.4% | 49.1% ± 2.3% | 87.4% ± 2.2% |\n| Gemma3 Clean | 73.4%  | 87.5%| 46.0% | 66.5% | 92.6% |\n| Gemma3 Back-patched | 75.4% ± 2.1% | 94.2% ± 0.7% | 47.5% ± 0.9% | 68.6% ± 1.8% | 98.0% ± 0.9% |\n\nTo measure significance, we use a high confidence threshold of 0.99, and validate that the lower bound of the mean (which is the mean across 1000 iterations minus the error margin) is higher than the clean baseline accuracy. This shows all performance improvements are significant except for the Qwen+factural recall case, where back-patching leads to only a minor improvement.\nWe will add these results to Table 1 (back-patching results), describe the bootstrapping procedure in the revision's appendix, and add the relevant code.\n \n**2. \"Evaluation does not incorporate scaling of models…To address this concern: Perform an evaluation where the same model family with different sizes is compared against back-patching, comparing the performance improvements\"**\n \nTo incorporate evaluation across scales, as well as answer your later question on the back-patching effect on PaliGemma, we perform the back-patching experiment on PaliGemma2 models, at 3 model sizes - 3B, 10B and 28B.\nThe results, presented in the following table, show that across model sizes, back-patching is a valid fix to the \"lack of visual alignment\" problem, showing relative performance improvements.\n\n| Model | Counting | Arithmetic | Spatial Ordering | Factual Recall | Sentiment Analysis |\n|---|---|---|---|---|---|\n| PaliGemma2-3B-224 Clean | 64.0% | 0.7% | 15.4% | 32.7% | 48.7% |\n| PaliGemma2-3B-224 Back-patched | 65.3% ± 2.9% |  32.3% ± 3.8% | 23.5% ± 3.4% | 39.2% ± 3.9% | 56.3% ± 3.9% |\n| PaliGemma2-10B-224 Clean | 66.0% | 8.7% | 14.0% | 38.7% | 56.7% |\n| PaliGemma2-10B-224 Back-patched | 69.4% ± 3.1% | 36.8% ± 4.1% |  25.9% ± 3.4% | 56.2% ± 4.0% | 72.8% ± 3.6% |\n| PaliGemma2-28B-224 Clean | 22.0% | 6.7% | 12.0% | 30.7% | 10.7% |\n| PaliGemma2-28B-224 Back-patched | 26.2% ± 3.6% | 30.8% ± 3.8% | 15.5% ± 2.9% | 36.0% ± 4.0% | 26.0% ± 3.6% |\n\nAcross most tasks and scales, back-patching improves performance by a significant margin, across scales. The bootstrapping settings are identical to the ones in the previous table.\n\nWhile back-patching isn't a perfect solution across settings, *we don't view it as such*. To quote the reviewer, we view it as \"a very interesting finding that requires more research before application\" -- it's a PoC of a potential way to bridge a limitation of VLMs, based on the mechanistic insights which are the main focus of our paper.\n\nNevertheless, we will add this evaluation of back-patching across scales and its implications to the revised paper.\n\n**3. “their patching approach increases the computational cost of inference, so this needs to be weighed against just using a larger model.”**\n\nWe treat back-patching as a PoC based on mechanistic insights (which are the main focus of our paper) rather than a ready solution. With that said, the additional experiments on the PaliGemma family show that this method can be applied on models of different scales to achieve performance improvements. Thus, we argue that such interpretability-based insights can provide improvements orthogonally to using larger models, which can be limited by availability.\n\n\n**4. \"Would the authors expect the back-patching to lead to a higher focus of VLMs on image modality than on text modality as reported in, e.g., [3,4,5]?\"**\n\nThis is an interesting question. We speculate back-patching doesn't affect the amount of attention given to image positions, but mainly allows the propagation of more \"concise\" and textually-aligned information from image tokens from early layers. However, further analysis into the effects and limitations of back-patching on visual token processing is an interesting direction. As we view the main focus of our work in the mechanistic analysis, and view back-patching as a finding rather than a perfect solution, we leave these investigations to future work.\n \n**5.** We will fix the two additional comments (reference to Appendix C from Table 1, PDF formatting issues) in the revision of the paper.\n\nTo summarize, we wish to re-iterate that we don’t view back-patching as a limitation-free solution to the performance gap, but rather a fix showing that interpretability-guided insights can help bridge modality performance gaps. \n \n\nWe hope our updated results and answers are satisfactory. Please let us know if you have any other questions or concerns."}, {"review_id": "lmLsCE5G2G", "reviewer": "Reviewer_KWdk", "summary": "The paper introduces an improved method of identifying modality-specific processing circuits (attention heads and MLP output units that strongly influence the outputs) in VLMs. The paper treats the computations in VLMs as three stages: data, query, and generation. They found that while the circuits for data processing vary across modalities, the circuits for query and generation are largely the same for a given task. (Here, the circuits are a kind of averaged effect across the whole task dataset.)\nThe paper proposed to patch the high-level tokens back to lower layers when the VLM processes visual inputs. They showed that this method closes 32% of the gap between analogous visual and textual tasks.", "questions": "1. the authors might consider better explaining the relation between the observations and the proposed back-patching method. The authors might do the back-patching to both visual and textual inputs and contrast the improvement in both cases.\n2. the authors might consider improving the clarity and the conciseness of the presentation.\n3. I wonder if any of the non-adaptor architectures still have this property.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. the perspective and analysis are novel, especially when applied to VLMs.\n2. the circuit identification and comparison methods have some extent of novelty beyond the previous works. The components in the proposed methods have been carefully considered and controlled.\n3. the swapping experiment is strong evidence for the main claim. This evidence confirms the validity of the proposed normalised IoU methodology and the results.\n4. the proposed back-patching method is simple and effective.\n\n\nWeaknesses:\n1. I don't really immediately see why the proposed back-patching method is the logical solution to the circuit difference problem. For me, it is not surprising to see that the data circuits are different (but I am surprised that the query and generation circuits are not changing according to the input modalities), and therefore I don't understand the necessity to forcefully align the visual embeddings in the earlier layers to be more text-orientated. I wonder if the same technique can be used for textual inputs and we can still see performance improvement.\n2. the presentation is good but not ideal. For me, there are some key steps with limited explanation. E.g., around line 170 the authors mentioned that the special visual tokens lead to the non-trivial positional mapping, but from my understanding, in the previous paragraph the authors already mentioned that they will not consider the data positions. Another example is that it seems the \"patching effects\" in figure 3 are never clearly defined (I have to assume that it refers to the \"attribution patching\" in section 2.2). A final example is that in line 229 the authors referred to another work without clearly explaining \"unembedding matrix\".", "was_revised": true, "final_justification": "The paper offers a novel analysis that uncovers the drastic differences in the “circuits” of a VLM across different input modalities. Leveraging this, the authors propose a simple yet effective strategy—back-patching the high-level vision token into the input stream—that efficiently boosts performance and provides valuable insights for the VLM community.\n\nMy previous major concern was that the proposed strategy seems to be less relevant to the analysis. But with Appendix E.3 and the author's explanation, I am more confident in their relation.\n\nMy other concern was the clarity of some parts of the paper, which the authors have clarified and promised to improve.\n\nThe concerns of 1QgH are also addressed, which show the robustness of the observed performance improvement.\n\nI personally think the concerns of KJ8j are less relevant in that the \"real-world\" datasets will have complex information that diminishes the power of controlled experiments, which is the core of this paper. Nevertheless, the performance improvement is also reassuring, though quite small.", "rebuttal": "Thank you for your detailed comments and feedback!\n\nWe are glad that the novelty of our analysis and methodology came through clearly, and we appreciate the recognition of the strengths in our circuit comparison approach, the normalized IoU analysis, and the effectiveness of back-patching. Below, we address your concerns point by point.\n\n**1. \"I wonder if the same technique (back-patching) can be used for textual inputs and we can still see performance improvement\"**\n\nThis is a great suggestion, which we indeed executed and reported in Appendix E.3 (referencing it in line 255). Since textual tokens are already “aligned with themselves”, gains here would reflect added compute. We find that visual back-patching yields larger improvements than textual back-patching in most settings, supporting our hypothesis that gains stem from improved alignment rather than additional depth alone. We will highlight this more clearly.\n \n**2. \"I don't really immediately see why the proposed back-patching method is the logical solution to the circuit difference problem\"**\n\nWe appreciate your question, and want to clarify – the difference between circuits employed in the model to solve visual / textual tasks isn't necessarily a problem – it’s just a finding. This, in addition with our additional findings:\n\n(1) the difference lies mainly in the data positions; and\n\n(2) visual embeddings only converge towards the higher-performing textually-aligned embedding in later layers;\n\nleads us to utilize the aligned embeddings from later layers earlier in the model, to allow for higher performance with a simple intervention.\nPer your comment, we will better clarify this motivation in the introduction section.\n \n**3. \"I wonder if any of the non-adaptor architectures still have this property.\"**\n\nMost open-weight SOTA VLMs use adapter-based architectures, including those we study. Whether our findings extend to non-adapter architectures is an important direction for future work, and we’ll highlight this in the limitations section.\n \n**4.** Regarding comments on presentation:\n\n* **\"patching effects are not defined, I have to assume that it refers to the \"attribution patching\" in section 2.2\"**: This is correct! We will point to section 2.2 in this figure to make this definition more clear.\n \n* **\"in line 229 the authors referred to another work without clearly explaining \"unembedding matrix\"**: The unembedding matrix is another name for the language modeling head (or vocabulary projection), the last linear layer in the model that projects the embedding to the vocabulary space. Thank you for pointing this out - we will rephrase this line to avoid confusion.\n \n* **\"the authors mentioned that the special visual tokens lead to the non-trivial positional mapping, but from my understanding, in the previous paragraph the authors already mentioned that they will not consider the data positions\"**:  This non-trivial mapping happens in few cases where a single token is mapped to multiple tokens. One such example is a visual prompt that has a \"<vision_end>\" token and a \"\\n\" following token and the analog textual prompt containing a single '\"\"\"\\n' token which we map to both visual tokens. In this case, we consider these tokens to be query tokens (as they don't contain any visual information), and thus the mapping between query tokens isn't always one-to-one. We will make this part more clear in the revision.\n \nWe hope our answers are satisfactory. Please let us know if you have any other questions or concerns."}]

Author rebuttal:

Meta-review: The submission focuses on investigating the performance gap between vision and language tasks in vision-language models, which is achieved by examining the underlying circuits for analogous problems in the corresponding data modality. The authors observed that while the sub-circuits are functionally interchangeable, the initial data integration is modality-specific. The authors then proposed "back-patching", which is a training-free intervention approach to inject text-aligned visual representations into earlier layers, which indeed reduce the modality performance gap.

The submission received three accept and one borderline reject ratings after rebuttal. Reviewers generally appreciate the method and analysis from the submission as well-motivated, novel, and insightful. Reviewer NpXT provided valuable feedback on the presentation of the submission, and found all of their concerns resolved after the rebuttal. Reviewer KWdk had questions about the connection between the analysis and the proposed remedy, which was also mostly addressed during rebuttal. There were also concerns on the empirical evaluations: Reviewer 1QgH requested model scaling experiments and statistical significance analysis, which were provided by the authors. 1QgH found them to be convincing, but requested clarification in the final version that the proposed back-patching method is a proof-of-concept. Finally, reviewer KJ8j had concerns that the baseline performance on real-world tasks are much lower than publicly reported results, likely due to the implementation / prompting design choices made to facilitate the analysis.

Overall, the submission makes valuable contributions to the multimodal learning and mechinterp communities, and the AC believes it is ready to be shared with the NeurIPS audience, assuming the promised clarifications will be made in the camera ready version. The AC also shares concerns (KJ8j, 1QgH) on the experimental setup and the generalizability of the empirical observations, but does understand novel methods and insights with a proof-of-concept implementation would still be impactful for the relevant research communities.
Decision: Accept (poster)