Visual symbolic mechanisms: Emergent symbol processing in Vision Language Models
Authors: Rim Assouel
Declan Iain Campbell
Yoshua Bengio
Taylor Whittington Webb
Venue: ICLR 2026 Oral (2026)
Keywords: visual object binding
vision-langue model
symbolic reasoning
interpretability
Primary area: interpretability and explainable AI
TL;DR: We describe a set of symbolic-like mechanisms that VLMs use to bind to visual entities in context

Abstract: To accurately process a visual scene, observers must bind features together to represent individual objects. This capacity is necessary, for instance, to distinguish an image containing a red square and a blue circle from an image containing a blue square and a red circle. Recent work has found that language models solve this ‘binding problem’ via a set of symbol-like, content-independent indices, but it is unclear whether similar mechanisms are employed by Vision Language Models (VLM). This question is especially relevant, given the persistent failures of VLMs on tasks that require binding. Here, we identify a previously unknown set of emergent symbolic mechanisms that support binding specifically in VLMs, via a content-independent, spatial indexing scheme. Moreover, we find that binding errors, when they occur, can be traced directly to failures in these mechanisms. Taken together, these results shed light on the mechanisms that support symbol-like processing in VLMs, and suggest possible avenues for reducing the number of binding failures exhibited by these models.

Published as a conference paper at ICLR 2026
VISUAL SYMBOLIC MECHANISMS: EMERGENT SYMBOL PROCESSING IN VISION LANGUAGE MODELS
Rim Assouel!
Declan Campbell"
Yoshua Bengio'.
Taylor Webb!
Equal contribution.
1 Mila Québec AI Institute, Université de Montréal
2 Princeton Neuroscience Institute
3 CIFAR AI Chair
assouelr la. quebec, i d
la quebec, taylor w www.wwwwwil.cc..ommmmmmommoooo com
ABSTRACT
To accurately process visual scene, observers must bind features together rep- resent individual objects. This capacity is necessary, for instance, to distinguish image containing red square and blue circle from an image containing blue square and a red circle. Recent work has found that language models solve this 'binding problem' via set of symbol-like, content-independder indices, but it is unclear whether similar mechanisms are employed by Vision Language Mod- els (VLMs). This question is especially relevant, given the persistent failures of VLMs on tasks that require binding. Here, we identify previously unknown set of emergent symbolic mechanisms that support binding specifically in VLMs, via a content-indee innteentiiddeeedentt spatial indexing scheme. Moreover, we find that binding errors, when they occur, can be traced directly to failures in these mechanisms. Taken together, these results shed light on the mechanisms that support symbol- like processing in VLMs, and suggest possible avenues for reducing the number of binding failures exhibited by these models.
INTRODUCTION
The visual world is composed of many commonly recurring elements - shapes, colors, textures, etc. and visual inputs can be efficiently represented by making use of this compositional structure, for instance, by representing 'red square' as a combination of features for 'red' and 'square'. There growing evidence that neural networks, including language models and vision language models (VLMs), learn such compositional representations and use them represent novel combinations of familiar features (Lepori et al.) 2022)] Lewis et al.) 2022; Campbell et al.] 2024 Assouel et al.) 2025). This compositional coding approach, however, introduces fundamental representational challenge, sometimes referred to as the "binding problem' (Treisman & Gelade 1980) Girfff et al., 2020 Frankland et 2021). Given a set of compositional features, such as features representing basic colors and shapes, the binding problem refers to the question of how these features are bound together to represent the entities and relational configurations in specific context. For instance, in order to represent an image that contains red square and a blue circle, and to distinguish it from other potential combinations of the same features (e.g., an image containing blue square and red circle), it is necessary to form in-context associations between the features that correspond to the same entities (i.e., to bind 'red' to 'square', and 'blue" to 'eircle').
Recent work has begun to uncover the representations and mechanisms that support this capacity in language models. Of particular relevance the discovery of emergent binding IDs, vectors rep- resenting content-indeependent indices akin to symbolic variables that language models use to track the assignment of entities and attributes in-context (Feng & Steinhardt, 2023 Feng et al. 2024). Binding IDs additively incorporated into the token embeddings corresponding to their arguments, and can be intervened upon to produce systematic binding errors. Other recent work has identified set of emergent symbolic mechanisms that language models use to perform abstract

---

Published as a conference paper at ICLR 2026
reasoning (Yang al. 2025 These mechanisms convert input tokens to abstract, symbol-like vari- ables, which can then be processed independently of the tokens to which they're bound, before eventually being converted back to the corresponding tokens to perform inference. These and other recent findings (Griffiths et al. 2025) suggest that language models rely on an emergent form of symbol processing to represent and reason about entities and relations in-context, but it is open question whether VLMs employ similar mechanisms to bind visual entities in-context.
It is particularly notable that many of the most puzzling shortcomings of VLMs are directly related to the binding problem. VLMs perform very poorly on many tasks that are easy for humans, such as counting or visual search (Rahmanzadehgervi et al. ] and many of these tasks involve the need to accurately bind features in- in-context. Indeed, careful behavioral evaluations have found that VLMs display specific psychophysical signatures similar to those observed in human vision for conditions that interfere with binding (Camphell et al. 2024). These findings underscore the importance of understanding the mechanisms that support binding in VLMs, and especially how the failure of these mechanisms might explain the binding failures that limit the performance of these models.
In this work, we present evidence for a set of emergent symbolic mechanisms that support binding in VLMs. Similar to the content-independent binding IDs that have been identified in text-only language models, we find that VLMs use visual space as a content- content-independent scaffold to bind features and parse multi-object scenes. We therefore refer to these indices position IDs. We also identify a set of visual symbolic mechanisms that VLMS use to manipulate these indices: 1) ID retrieval heads retrieve the position ID associated with an object described in the prompt, based on the features of that object; 2) ID selection heads compute the ID of a target object; and 3) feature retrieval heads use this ID retrieve the features associated with that object. We present convergent evidence for these mechanisms across a set of representational, causal mediation, and intervention analyses. Finally, we analyze the role that these mechanisms play in the persistent binding failures exhibited by VLMs. We find that binding errors can be directly tied to failures of the identified mechanisms. Taken together, these results begin to uncover the mechanisms that support symbol-like binding and inference in VLMs, and illustrate how the breakdown of these mechanisms contributes to their persistent binding failures, suggesting avenues for further improvement of these models. Our specific contributions are as follows:
We identify and characterize the role of sets of attention heads involved in visual object binding (Section 3).
We define the sets using causal mediation analyses (Section 3.4).
We validate the role of these attention heads and the use of position IDs across a diverse range of VLMS (7 models) through representational analyses(Section 3.3) and intervention experiments (Section 4.2).
We show the generality and reuse of position IDs across several tasks and in photorealistic images (Sections 4.1 4.3 and (4.4).
We link binding failures VLMs to interference during the ID retrieval process, suggesting new avenues for improving visual grounding in VLMs (Section
2 RELATED WORK
A number of studies have investigated the emergent mechanisms that support symbol-like processing in language models and other neural networks. This work has identified a number of surprisingly interpretable and structured mechanisms, including induction heads (Olsson et al. 2022), function vectors (Todd et al. 2023), binding IDs (Feng & Steinhardj 2023 Feng et al.] 2024), and emergent symbolic mechanisms (Yang et al. 2025). Related work has found convergent evidence for such emergent mechanisms in transtormer language models and vision transformers that are trained in controlled settings (Lepori et al., 2024 Tang et al.) (025). Although a number of recent studies have begun to apply mechanistic interpretability techniques to understand VLMS (Neo et al. 2024] Golovanevsky et al. 2024 Kaduri et al.) 2024 Basu et al., 2024), it has not yet been established whether VLMs possess emergent mechanisms for symbol processing similar to those that have been identified in text-only language models, as our results suggest.
Beyond the difference in modalities (images vs. text), there are also several novel contributions of our work that go beyond the previously identified emergent symbolic mechanisms in text-only language models (Yang et al. 2025). First, although the identified mechanisms in both cases in- volve emergent form of symbol processing, the specific function that these mechanisms perform
2

---

Published as a conference paper at ICLR 2026
is different (parsing of multi-object scenes vs. induction of abstract relational patterns). This is not merely a translation of the same circuit from the textual domain into the visual domain, but entails a different circuit altogether. Second, unlike previous results in text-only models, the emergent visual symbolic representations are distributed across multiple tokens (each object spans multiple patches), demonstrating that emergent symbol processing extends more naturalistic and high-dimensional domains. Third, the identified visual symbolic mechanisms are directly related to a significant lim- itation faced by current VLMs (the binding problem), suggesting that these mechanisms may be of practical relevance for improving these models.
Several studies have probed the capacity of VLMs to process multi-objeed scenes, revealing number of failure modes for tasks such as counting (Rahmanzadehgervi et al. [0224] Rane et al. [2224] Zhang
Wang
2024), visual search (Campbell et al.) 2024), and visual analogy (Mitchell et al. 2023
Yiu et al. 2024: Fu et al., 2025). Notably, all of these failure modes appear to be directly related
to difficulty with binding object features and relations (Yuksekgonul et al.) [023] Campbell et al.)
2024 Assouel al. 2025). Our results reveal some of the key mechanisms that are involved in
these binding failures, suggesting potential avenues for further improvement of VLM architectures
and training.
Finally, our findings are closely related to work that has identified the neural and psychological correlates of visual binding. In cognitive science, visual indexing theory (Pylyshyn 200T) has pos- tulated the existence of priteettinddeeddeen, visual indices that are used for binding object features. In neuroscience, there is broad distinction between brain regions involved in representing con- crete features (c.g., shapes and colors), and brain regions involved in representing space (Goodale) & Milner 1992). The spatial representations in this latter of brain regions also appear to be more broadly involved in abstract, symbol-like processing (Whittington et al.) 2020; O O'Reilly et al.) 2022 Webb et al. 2024). The emergent mechanisms that we have identified VLMs, in which sual space serves as a content- content-indepemdeent scaffold for binding object features, thus have interesting parallels to findings from cognitive science and neuroscience.
SYMBOLIC MECHANISMS FOR VISUAL BINDING IN VLMS
3.1 SCENE DESCRIPTION TASK
To identify the mechanisms underlying visual binding in VLMs, we used a scene description task that tests the model's ability to bind features to objects in multi-object visual scenes. In that task, the model receives an image containing multiple objects (e.g.. colored shapes) alongside a prompt describing some, but not all, objects present. The model must identify and describe the missing object. For instance, given an image with a red square, blue circle, and green triangle, plus the prompt "This image contains red square, blue circle and ", the model must respond "green triangle."
This task requires the model to: (1) parse the visual scene, (2) match objects described in the prompt to visual representations, (3) determine the missing object, and (4) retrieve its features. Crucially, success in this task demands accurate spatial binding avoid erroneously combining features across objects.
3.2 POSITION ID ARCHITECTURE
We identified a three-stage architecture that VI.MS employ to solve the binding problem (Figure lap. These stages implement content-indepente indexing scheme using position IDs -spatial indices serving as symbolic variables for tracking visual objects.
Stage 1: Position ID Retrieval In this stage the model establishes correspondences between semantic content in the prompt and spatial locations in the image. Given an object description (e.g., "red square"), these heads retrieve the position ID for that object from the corresponding image tokens. The output of these heads is an abstract spatial index not the object's features, but a pointer to its location.
3

---

Published as a conference paper at ICLR 2026
Tedinaad Aboiio
purple criangle
Tutur
Position (Prompt Tokens)
Taken)
Cornetation 1
Festure (Last Token)
Heacs
D
0.0
15
20
orange trisngle
Layer
(a) Position ID architecture
(c) Representational similarity analysis
Layer
Layer
Position
Positton
Target Features
Features
Grean Trangls
Onde
Square
Purge
Moad bodra
Principal component analysis
(d) Causal mediation analysis
Figure 1: Overview of position ID architecture and supporting evidence. We identify three process- ing stages: ID retrieval heads that retrieve the position ID of objects described in the prompt, ID selection heads that select the position ID) for target object, and feature retrieval heads that use this position ID to retrieve the features of the target object. These stages are revealed both by (b,c) representational and (d) causal mediation analyses (see text for explanation).
Stage 2:
Position ID Selection In this stage. the model selects the position ID for the target object, i.e., the object that will be described next, based on the position IDs that have already been retrieved in the preceding prompt.
Stage 3: Feature Retrieval In this stage, the model uses the position ID from stage 2 as an index retrieve semantic features of the target object from the image tokens corresponding that object.
In the following sections, we first present representational analyses (Section 3.3 that confirm the progression of these three stages across layers and sequence positions, and then describe the causal mediation procedure (Section 3.4h used to identify the specific attention heads that implement these stages. We show results for Qwen2-Vi VI. in the main text, but results for additional models are shown
the Appendix sections A.5. and A.5.2
3.3 REPRESENTATIONAL ANALYSES
We carried out representational analyses using approaches: principal component analysis (PCA), and 2) representational similarity analysis (RSA). The PCA results for Qwen2-VL are shown Figure [b. These results provide an intuitive visualization of stages and 3. The results show the hidden state activations for the last token in the sequence, at which the model must generate description of the target object. These activations are projected on to the top 2 principal components (PCs), and colored according to either the spatial position of the object (top row), or the features (color and shape) of the object (bottom row). The results show that spatial position is clearly separa- ble in the embeddings for layer 19 (left column), whereas object features are completely overlapping. This corresponds to stage 2, during which the model computes the position ID for the target object. By contrast, in layer 27 (right column), spatial position is overlapping, while object features become separable. This corresponds to stage 3, during which the model retrieves the features of the target object based on its ID.

---

Published as a conference paper at ICLR 2026
To more comprehensively characterize this process, also performed RSA Kriegeskorte et al. (2008). In RSA, two embedding spaces can be compared by first computing the pattern of pairwise similarities within each space, and then computing the correlation between these similarities. Using this approach, we compared the hidden state embeddings in the model to two hypothesized embed- ding spaces, one that coded only object position, and one that coded only object features. The results (Figure Icb further confirmed the proposed three-stage progression (Figure Iaj. At the prompt to- kens describing an object, the position of that object was most strongly represented in layers 14-17, as predicted by the ID retrieval stage. At the final token, the position of the target object was most strongly represented in layers 18-21, as predicted by the ID selection stage, and the features of the target object were most strongly represented in layers 23-26, as predicted by the feature retrieval stage. We also found similar RSA results for the other models studied (Appendix A.5.T
3.4 IDENTIFYING ATTENTION HEADS VIA CAUSAL MEDIATION ANALYSIS
In the previous section, we presented representational evidence consistent with the proposed three- stage architecture. To identify the specific mechanisms that implement these stages, we next per- formed causal mediation analysis (CMA) (Pearl [022] Wang et al.] [022] Meng et al.) 2022 Yang eet al. 2025). This approach allows us to estimate the causal effect of an embedding at a particular layer, position,
attention head. In our case, given our hypothesized architecture, we focus on analyzing the causal effect of the output of attention heads in the prompt tokens. Our goal in this analysis was to identify the attention heads that are causally involved in the stages that we identi- fied: 1) retrieving the ID of the object mentioned in the prompt, 2) computing the ID of the missing target object, and 3) using this ID to retrieve the features of the target object. We refer to the heads that perform these computations as ID retrieval heads, ID selection heads, and feature retrieval heads respectively.
In order to isolate these sets of heads, we designed three conditions. Each CMA condition is defined by clean context e1, with correct answer d1, and a modified context cg. Activations are patched from 02 to c₁ yielding the patched context ct. with expected answer at. Given model M, we measure the causal mediation score as defined by (Wang et al. 2022):
s = (M(G)a]] - M(C))aa)) - (M(())
[a1)
(1)
where [c] corresponds to the logits of token a at the output of the model evaluated on input Intuitively, this score measures the extent to which patching activations from 02 to o has the expected effects on the model's outputs (i.e.. makes the model more likely to respond with of than a1). We performed causal mediation on a simpler version of the scene description task involving only two objects. All CMA scores are averaged across 50 samples for each condition. Below we define the three different conditions corresponding to the three hypothesized stages. We include a schematic illustration of the CMA patching intervention targeting all sets of head in Appendix A.33
ID Retrieval Heads To identify heads responsible for retrieving the position ID of objects men- tioned in the prompt, we designed condition where the clean context ci contains image with two objects and prompt mentioning one of them, with the clean answer 01 being the color of the unmentioned object. The modified context C2 uses the same prompt but with the object positions swapped in the image. We predicted that this would result in the position IDs assigned to these objects also being swapped. We performed causal mediation by patching attention head outputs at the prompt tokens for the object described in the prompt. If an attention head is causally involved in retrieving the position ID for the described object, then patching from the modified context (where object positions are swapped) should cause the model to retrieve the wrong position ID (namely, the position ID for the target object, rather than the object described in the prompt). This in turn should cause the ID selection heads to erroneously select the position ID for the object already described in the prompt (rather than the target object), which should ultimately cause the feature retrieval heads to retrieve the features for the object already described in the prompt (see Figure 10) for a schematic depiction). The expected answer aj in this condition is therefore that the model should repeat the features for the object already described in the prompt.
ID Selection Heads To identify heads that compute the position ID of the target object, we used the same clean and modified contexts as in the ID retrieval condition. However, we performed
5

---

Published as a conference paper at ICLR 2026
causal mediation by patching attention head outputs at the last token position, where the model must generate a description of the target object. If an attention head is causally involved in computing the target object's position ID, then patching from the modified context (where object positions swapped) should cause these heads to select the wrong target ID (namely, the ID) for the object already mentioned in the prompt). This in turn should cause the feature retrieval heads retrieve the features associated with the object already described in the prompt (see Figure [ and therefore the expected answer aj in this condition is that the model should repeat the features for the object already described in the prompt.
Feature Retrieval Heads To identify heads that retrieve object features, we used clean and mod- ified contexts that differed only in the features of the target object. The features and position of the object mentioned in the prompt, as well as the position of the target object, were the same for both contexts. We patched attention head outputs the last token position. If an attention head is causally involved in retrieving the features of the target object, then patching from the modified context (in which the target object has different features) should cause the model to retrieve the wrong feature information (see Figure 12), and the expected answer of is therefore the semantic features of the target object the modified context rather than the original target object.
CMA Results We performed CMA for these conditions on all models (Qwen2.5-VL- 3B, Qwen2.5-VL--7B, Qwen2.5-VL-32B, Qwen2-VL. Llaval.5-7B. Llaval.5-13B, and Llava- On) and report the results in the Appendix (see Figures 20 25 We include the CMA results for Dw in Figure [d] where the CMA scores are color coded by condition (i.e., head type). These results confirmed the predicted three-stage progression, with ID retrieval heads (in blue) primarily localized to layers 12-16, ID selection heads (in red) primarily localized to layers 18- 19, and feature retrieval heads (in green) primarily localized layers 20-27. These layers also corresponded closely to the layers identified for these stages in the representational analyses.
4 RESULTS
Having identified the mechanisms that support binding in VLMs, next performed series of analyses to characterize these mechanisms in more detail. These analyses provide a richer picture of the coding scheme that they employ, demonstrate their generalizability across tasks and image types, and implicate them in the binding failures displayed by VLMs.
4.1 ARE POSITION IDs RELATIVE OR ABSOLUTE?
In this section we focus on determining whether position IDs employ a relative or absolute spatial coding scheme. We designed an experiment using a 3x3 object grid. We created four separate 2x2 configurations, each positioned within different quadrants of the larger (3x3) grid (Figure 2 Crucially, across all four arrangements, one object was always placed at the same absolute gnd location (the center position, circled in Figure 2. We prompted the model to perform the scene description task from the previous analyses, in which single object is missing from the prompt. We analyzed the representations at the last token, where the model must predict the missing object.
We then performed RSA using similarity matrices that reflected either relative (within each inner 2x2 grid) or absolute position (within the larger 3x3 grid). We found that models from the Qwen family had a clear preference for a relative coding scheme (Figure shows results for Qwen2-VL). The effects were less pronounced for models from the LLava family (see Appendix I results from all models). This may be a result of differences in position embeddings for the Llava models, or may be due Llava's 's significantly smaller training set. Finally, further confirmed these results with a set of interventions that targeted relative vs. absolute positions (see Appendix 1.2 Figures 13 and 30
4.2 POSITION IDS GENERALIZE TO VISUALLY COMPLEX SETTINGS
We next investigated whether the symbolic mechanisms employed in synthetic visual tasks also used when processing more complex, naturalistic images. To do so. we generated a dataset us- ing the Photorealistic Unreal Graphics (PUG (Bordes et al. 2023)) environment, which allows for

---

Published as a conference paper at ICLR 2026
Representational Similarity Across
Texture
Corroation (Pearson ()
Absolule Paskior
Rebtive Pustion
Layer
Figure 2: Testing for relative vs. absolute spatial coding. Left: Example of each grid condition with centrally located object circled. Right: RSA results showing second-order correlation of last token embeddings (Qwen2-VL) VL) with relative position, absolute position, and object features.
the generation images that capture important properties real-world images, including three- dimensional structure, occlusion, and lighting/shadows. Our dataset was comprised of images that each contained two distinctly colored 3D animals in varying realistic backgrounds (see Appendix A for details). We randomly jittered the positions of the animals, but ensured that each image has clear leftmost and rightmost animal (to ensure that we have ground-truth labels for the position IDs that the models will assign).
Fredforwnn
red dolphin
Hatneval Heads
ID
HD. D0 IDD1T Intervention
Accuracy
ID
0.4
Sclection Heads
0.2
Bettieval
Selection
Posture
Uava-1.5 Uava-1.5-78 1.5-78
Ulav- Llava-1.5-128 1.5-188
Qwen-2 2.544 38
Owen 2 2.5-V-337 328
Qwer-24V
Ownn-2 2.85-
Liea One.ision---nn
Hetrieval Heada
There is red dolphin and
(b) Intervention accuracy across all models and sets of heads.
(a) Intervention targeting ID selection hcads.
Figure 3: Photorealistic intervention results targeting the three position ID processing stages as identified by CMA.
Our position ID hypothesis predicts that by intervening on the position IDs, we can steer the model to predict the color of either of the animals based solely on position ID (left vs. right) assigned to them. Figure 3a shows illustration of this intervention for the outputs the selection heads. The basic logic of the intervention is to first estimate the embeddings for position IDs 0 and (by averaging over several instances of these IDs), and then to "edit" the IDs computed by the model (i.e., subtract the ID computed by the model and add the other ID).
We performed this intervention targeted to each of the 3 identified sets heads, For the ID retrieval and selection heads, we performed the intervention feature retrieval heads, we performed the intervention on the queries, since our hypothesis that these heads use position IDs queries retrieve features stored in the image tokens.
the output of the attention heads. For the

---

Published as a conference paper at ICLR 2026
The intervention can be formally defined as follows:
(x())) = (x) + a* (d: do)
diet,o = Enxx [xEx((ox(r]]
where on (3) is the output of attention head h given input æt, de is the estimate of the target binding ID (e.g., ID 0 FigureB3a), d, is the estimate of the original binding ID (e.g.. ID in Figure Baj. and is a coefficient that controls the magnitude of the intervention. The intervention was applied to the top K heads as defined by the CMA conditions for cach head type. We optimized the intervention by sweeping over range of values for and K (see Appendix A.5.5 for results across all values), and show the results for the best performing parameters in Figure 3bj The results indicated that this intervention was highly effective (> 79% efficacy) for all 3 head types and all models, demonstrating that position IDs also play a central role in processing more complex images. Furthermore, we present intervention results in section B of the Appendix showing that position IDs also play a role in processing of real-world images (using the COCO dataset).
4.3 POSITION IDS ARE LOCALIZED WITHIN VISUAL OBJECT PATCHES
We then sought to determine whether position IDs are localized in the residual stream of visual patches spanning the objects in the image. This analysis employed simple color retrieval task, in which the model is prompted to identify the color of a specified object. The following prompt template was used: "In this image what is the color of the (SHAPE). Answer with the correct color only." Our objective was to steer the models to retrieve the color of an arbitrarily specified object by intervening on the keys of the visual object patches. Let Kî R.Noxd be keys vectors spanning the No visual patches of object o. We perform an additive intervention on the K7 in order to swap the ID of object on with the ID of object 0r at layer / such that:
K = KTn + aad - d()
K = Kt + a(dh - dh)
with d, = Eogo,KK? and O being the set of object patches with ID i. Importantly, we performed the intervention before the RoPE embedding module, and therefore any ef- fects of this intervention on position IDs is independent of the RoPE position embeddings. For this experiment, taining the top-20 highest scoring feature retrieval heads (according to the CMA scores).
fixed a to 2, and we applied the intervention to the layers con-
The results are shown in Figure 4 This intervention highly effective for all models, confirming that position IDs are stored locally in the keys of the image patches spanning each object.
Intervention Accuraty
8.6-
0.4
0.2
4.4 POSITION IDS ARE REUSED ACROSS TASKS
6.0
Owin 54V
Green22 Sin
Ownn-2. 5.
30
Qwnn-aat
Liav--1 1.5 78
Liava 5-130
We next sought to determine whether position IDs are also used in more complex tasks. To do so, we focused on a visual reasoning task involving spatial relations. The task involved questions with the following template "In this image, what is the color of the object that is directly (RELATION) of [REF). Answer with the relevant color only." where RELATION was one of {above, below, left, right} and REF was one of the reference objects in the image. Importantly, we first estimated position IDs based on the scene deseription task, and then applied these IDs in an intervention in the visual reasoning task, thus assessing the extent to which position IDs are reused across tasks. More precisely, given the output og (x) of head h with input context x, we performed an additive intervention such that
Model
Figure 4: Color Retrieval Intervention Accuracy for all models.
on (I:) = 0% (x) + * dhese
where d'ase ErEx, (a))), with X₃ being the set of scene description input context in which the target object is assigned to ID i. The intervention was applied to the top 100 ID selection heads (based on the CMA scores), and a was set by 2.
8

---

Published as a conference paper at ICLR 2026
Uave-1.5-138
Uava-115578
100
100
Accuracy (%)
80
Accuracy (%6
80
Intervention
60-
60
After
Before
beloo aboovs
left
nightt
aboyte
below
left
right
Owen-2-VL
Owen-2.5-VL-BB
Owen-2. Swen-2.5-ML-78
Accuracy (%)
80
Accuracy
60
60
40-
40
40-
20
20
20
above
below
leff
right
above
below
left
right
above
below
leff
right
Figure 5: Intervention performance on spatial reasoning task.
Figure 5 shows the effects of this intervention for 5 out of models (Qwen2.5-32B and Llava- Vnsion were already at ceiling-level performance this task, making an intervention uninfor- mative). The results show that transferring position IDs from the scene description task where models generally performed well significantly improved performance spatial reasoning task. These results highlight not only that position IDs are employed in more com- plex tasks, such spatial reasoning, but that the same position IDs are used across tasks, indicating a general-purpose symbolic indexing scheme.
the more challenging
4.5 BINDING ERROR ANALYSIS
Prompt Tokens Position RSA
Last Token Position RSA
0.8
0.8
Low Entropy
High Entropy
0.6
RSA Correlation
RSA Correlation
0.6
(a) High entropy
0.2
Layer
Layer
(b) Low entropy
(c)
(d)
Figure 6: Binding analysis showing effect of feature entropy position ID mechanisms.
We next investigated the role that position IDs and visual symbolic mechanisms play in binding errors. Specifically, we analyzed how position IDs influence the model's ability to bind features to objects or locations, using scene description task with varying feature entropy factor known to impact binding errors (Greff al.) 20200] Campbell et al. 2024). Details of the generation process are provided in Appendix A.5.4 and examples of low (resp. high ) entropy stimuli are shown in Figure 6b (resp. 6a). Our results (Figure 6) show that entropy level strongly affects position representations at two critical stages: ID retrieval Figure (c) and ID selection (Figure (6d). We observe that lower entropy leads to less accurate ID retrieval and more ambiguous ID selection, reflected in performance differences between low and high entropy settings (Figure 77. We also report overall RSA (averaged across layers) for prompt tokens and last tokens Figure Layer- wise RSA details for all models included in Appendix A.5.4 We also show the causal relation of the binding ID mechanisms failure and the binding emrors in this task in Appendix D

---

Published as a conference paper at ICLR 2026
Model Porformance (3x3 Grid)
Prompt Tokens- - Position RSA (3x3)
Last Token Position RSA (3x3)
High Entropy
Enrrpy
RSA Coreaatin in
RSA Corretation
Accuracy
0.6
9.3
0.4
0.8
Opeen
2.5
LLab% 1.5
Gaen 23
25
Oven
Qwen Owan
25
329
Figure 7: Link between binding errors and failure position ID mechanisms.
5 CONCLUSION AND FUTURE WORK
In this work, we have identified and characterized set of emergent symbolic mechanisms that VLMs use to perform visual binding (Greff et al. 2020; Lewis et al. 2022 Campbell et al., 2024] Assouel et al. 2025) in multi-object scenes. Through combination of representational analyses, causal mediation, and interventions across seven different VLM models, we uncovered a stage architecture that employs position IDs as content-indepenae tures. These mechanisms are similar to mechanisms that support symbol-like processes in language models, most notably including binding IDs (Feng & Steinhardi, (023) and emergent symbolic mechanisms Yang et al. 2025). but our results extend these findings to visual processing.
spatial indices for binding object fea-
Our key findings demonstrate that VLMs implement binding through: (1) ID retrieval heads that establish correspondences between semantic content in prompts and spatial locations in images, (2) ID selection heads that compute the position ID of target objects, and (3) feature refrieval heads that use these IDs to retrieve object features. Importantly, we show that this architecture is remark- ably consistent across multiple model families and scales, suggesting it represents fundamental solution to visual binding in current VI.Ms. Our analyses reveal several critical properties of these mechanisms: position IDs employ relative rather than absolute spatial coding; they generalize be- yond synthetic stimuli to complex, photorealistic images with naturalistic backgrounds; they are localized within the image patches corresponding to each object; and they are reusable across dif- ferent tasks, indicating general-purpon symbolic indexing system that supports diverse forms of visual reasoning.
Crucially, demonstrate that the persistent binding failures exhibited by VLMs can be directly traced to failures in these symbolic mechanisms. We find that position IDs are less accurately rep- resented in conditions that typically lead to binding errors, such images where multiple objects share features. These findings have important implications for the development of more capable VLMs. Our results suggest that binding performance may be improved either via architectural in- novations that better support spatial indexing (such as object-centric architectures (Locatello et al.) 2020 training strategies that explicitly strengthen these symbolic mechanisms, such spatial pointing tasks (Deitke et al. 2024).
is worth considering the extent to which the identified mechanisms are truly emergent, particularly given the potential relationship between a model's 's innate position embeddings and the structure of the position IDs. is important to emphasize that these mechanisms involve several operations that are not entailed by position embeddings alone, and which are not built into the architecture, including the clustering position embeddings for patches belonging to the same object, and the specific function of the three identified attention heads (mediated by the content of their queries, keys, and values). However, is an open question whether the emergence of these mechanisms may be driven by architectural inductive biases, such as the use of distinct query/key and value embeddings (enabling a form of indirection, or the use of 'pointers') ) or distributional aspects of the training data. We leave the investigation of these questions to future work.
10

---

Published as a conference paper at ICLR 2026
6 REPRODUCIBILITY STATEMENT
We included all the generation details of our synthetic datasets Appendix A A detailed expla- nation of the RSA is provided in A.2 and the CMA score in Section A.552 of the main text. We additionally referenced all the analyses for all the models we study in Appendices A.5.5 A.5.2 and A.2 Datasets, analysis and intervention code will be released.
ACKNOWLEDDGMENTS
RA would like to thank Sjoerd Van Steenkiste for all the brainstorming sessions during the course of the project. DC is supported by the NSF GRFP Fellowship and the Natural and Artificial Minds Graduate Fellowship. TW supported by the Courtois Chair in Fundamental Research V (Neuro- science).
REFERENCES
Rim Assouel, Pietro Astolfi, Florian Bordes, Michal Drozdzal, and Adriana Romero-Soriano. Object-centric binding contrastive language-image pretraining, 2025. URL https://
arxiv. Trxiv.org/abe.
Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, and Daniela Mas- siceti. Understanding information storage and transfer in multi-modal large language models. arXiv preprint arXiv:2406.042: 2024.
Florian Bordes, Shashank Shekhar, Mark Ibrahim, Diane Bouchacourt, Pascal Vincent, and Ari S. Morcos. Pug: Photorealistic and semantically controllable synthetic data for representation learn- ing, 2023. htttttt:////xiii//// org/ abw/abas//22/ 03977
Declan Campbell, Sunayana Rane, Tyler Giallanza, Camillo Nicolò De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven Frankland, Tom Griffiths, Jonathan D Cohen, et al. Understanding the limits of vision language models through the lens of the binding problem. Advances in Neural 2024.
Information Processing Systems, 37:1
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salchi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409. 17146, 2024.
Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? arXiv preprint arXiv:2310.111 2023.
Jiahai Feng. Stuart Russell, and Jacob Steinhardt. Monitoring latent world states in language models with propositional probes. arXiv preprint arXiv:2406. 19501, 2024.
Steven M Frankland, Taylor Webb, Richard L Lewis, and Jonathan D Cohen. No coincidence, george: Capacity-limits as the curse of compositionality. PsyArXiv preprint, 2021. htthttps:
cjubb_v
Shuhao Fu, Andrew Jun Lee, Anna Wang, Ida Momennejad, Trevor Bihl, Hongjing Lu, and Tay- lor W Webb. Evaluating compositional scene understanding in multimodal generative models. arXiv preprint arXiv:2503.23125. 2025.
Michal Golovanevsky, William Rudman, Vedant Palit, Ritambhara Singh, and Carsten Eickhoff. What do vlms notice? mechanistic interpretability pipeline for mqussian-noise-fiie text-image corruption and evaluation. arXiv preprint arXiv:2406. 2oXiv:2406.16320, 2024.
Melvyn A Goodale and A David Milner. Separate visual pathways for perception and action. Trends in neurosciences, 15(1):20-25 1992.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208. 2020.

---

Published as a conference paper at ICLR 2026
Thomas L Griffiths, Brenden M Lake, R Thomas McCoy, Ellie Pavlick, and Taylor W Webb. Whither symbols in the era of advanced neural networks? arXiv preprint arXiv:2508.05776, 2025.
Omri Kaduri, Shai Bagon, and Tali Dekel. What's in the image? deep-dive into the vision vision language models. arXiv preprini 1. 174911, 2024.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249, 2008.
Michael Lepori, Thomas Serre, and Ellie Pavlick. Break it down: Evidence for structural compo- sitionality in neural networks. Advances in Neural Information Processing Systems, 36:42623- 42660, 2023.
Michael Lepori, Alexa Tartaglini, Wai Keen Vong, Thomas Serre, Brenden M Lake, and Ellic Pavlick. Beyond the doors of perception: Vision transformers represent relations between ob- jects. Advances in Neural Information Processing Systems,
2024.
Martha Lewis, Nihal V Nayak, Peilin Yu, Qinan Yu, Jack Merullo, Stephen H Bach, and Ellie Pavlick. Does clip bind concepts? probing compositionality in large image models. arXiv preprint arXiv:2212. 10537, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in neural information processing systems, 33:1 13:11125511538, 2020.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 15:9773--77372, 2022.
Melanie Mitchell, Alessandro B Palmarini, and Arseny Moskvichev. Comparing humans, gpt-4, and gpt-4v on abstraction and reasoning tasks. arXiv preprint #rXiv:331..0924/ 2023.
Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpret- ing visual information processing in vision-language models. arXiv preprint arXiv:2410.07149, 2024.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895 2022.
Randall O'eell,,, Charan Ranganath, and Jacob L Russin. The structure of systematicity in the brain. Current directions in psychological science, 31(2):124-130, 2022.
Judea Pearl. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pp. 373-392, 2022.
Zenon W Pylyshyn. Visual indexes, preconceptual objects, and situated vision. Cognition, 80(1-2): 127-158, 2001.
Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision, pp. 18- 34, 2024.
Sunayana Rane, Alexander Ku, Jason Baldridge, Ian Tenney, Tom Griffiths, and Been Kim. Can generative multimodal models count to ten? In Proceedings of the Annual Meeting of the Cogni- tive Science Society, volume 46, 2024.
Cheng Tang, Brenden Lake, and Mehrdad Jazayeri. An explainable transformer circuit for compo- sitional generalization. arXiv preprint arXiv:2502.15800 1580/, 2025.
12

---

Published as a conference paper at ICLR 2026
Eric Todd, Millicent L Li, Arna Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXi::2310.15213, 2023.
Anne M Treisman and Garry Gelade. A feature-integration theory of attention. Cognitive psychol- ogy, 12(1):97-136, 1980.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00591, 2022.
Taylor W Webb, Steven M Frankland, Awni Altabaa, Simon Segert, Kamesh Krishnamurthy, Declan Camphell, Jacob Russin, Tyler Giallanza, Randall O'Reilly, John Lafferty, et al. The relational bottleneck as an inductive bias for efficient abstraction. Trends Cognitive Sciences, 2024.
James CR Whittington, Timothy Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy EJ Behrens. The tolman--ichenbann machine: unifying space and relational mem- ory through generalization in the hippocampal formation. Cell, 183(5):1249-1263 2020.
Yukang Yang, Declan Campbell, Kaixuan Huang, Mengdi Wang, Jonathan Cohen, and Taylor Webb. Emergent symbolic mechanisms support abstract reasoning in large language models. arXiv preprint arXiv:2502.203:: 2025.
Eunice Yiu, Maan Qraitem, Anisa Noor Majhi, Charlie Wong, Yutong Bai, Shiry Ginosar, Alison Gopnik, and Kate Saenko. Kiva: Kid-inspired visual analogies for testing large multimodal mod- els. arXiv preprint MxXv:2447..77733 2024.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words and what to do about it? 2023, URL
fittss:/aarxe www.orgg//ssooo abs aorpooo/oooi..o///aaaooos 2210 01936
Chenhui Zhang and Sherrie Wang. Good at captioning bad at counting: Benchmarking gpt-4v on earth observation data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7839 7849, 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DATASET GENERATION DETAILS
A.1.1 SYNTHETIC SCENE DESCRIPTION TASK
This section describes the synthetic image datasets generated for representational similarity analysis (RSA), patching experiments, and principal component analysis (PCA). All images used colored shapes that each occupied a 56x56 pixel region, comprising a 2x2 grid of patches (cach patch had a size of 28x28 pixels). RSA and intervention experiments were conducted using datasets generated by arranging objects within a 2x2 or 3x3 grid, while PCA analyses used 3x2 grid configuration.
Dataset generation protocol: For each dataset, there were unique objects defined by unique color-shape conjunctions, where N was equal to the number of grid positions. Each image contained one target object at fixed position, with the remaining N-1 objects randomly permuted across remaining positions. We generate K trials for each combination of object identity and grid position, yielding K x N total trials per dataset.
PCA dataset The 3x2 grid configuration (392x932 pixels) used for the PCA results shown in Figure [bb used 6 specific color-shape conjunctions: red circle, green triangle, blue square, purple star, yellow heart, and orange cross. Objects were placed in the leftmost and rightmost columns of 3x3 grid layout with the center column empty. 200 trials/combination were generated, resulting in 7,200 total trials.
A.1.2 DATASET FOR TESTING ABSOLUTE VS. RELATIVE SPATIAL CODING
Figure 8: Stimulus condition for characterizing relative vs absolute position IDs.
To determine whether position IDs use relative absolute position coding, designed an exper- iment using a 3x3 object grid. We created four different 2x2 sub-arrangements of objects, each positioned within different quadrants of the larger grid (Figure(8). Crucially, across all four arrange- ments, one object was always placed at the same absolute position (the center position).
A.1.3 SCENE DESCRIPTION TASK WITH NATURALISTIC IMAGES (PUG)
To validate the use of position IDs in more naturalistic settings, we performed additional experi- ments using the PUG (Photorealistic Unreal Graphics) environment, which allows for the genera- tion of images that capture important properties of real-world images, including dine--iimesiooal structure, occlusion, and lighting/shadows. We generated dataset of images that each contained two distinctly colored 3D animals with realistic backgrounds. We randomly jitterrd the positions of the animals, but ensured that each image has clear leftmost and rightmost animal (to ensure
14

---

Published as a conference paper at ICLR 2026
that we have ground-truth labels for the position IDs that the models will assign). An example is shown in Figure The dataset is comprised of 200 images with different animals (camel, dolphin, elephant) and different colors (green, red, white) in 3 different realistic environments (beach, salt desert, museum).
A.1.4 DATASET FOR INVESTIGATING BINDING ERRORS
2x2
3x3
High Entropy
Low Entropy
Figure 9: Image configurations for investigating binding errors.
To investigate the role of position IDss in binding errors, we created datasets varying in grid size and entropy. Low entropy datasets used all possible conjunctions from limited color and shape sets, while high entropy datasets used the most distinct conjunctions from larger sets. The 2x2 grid (280x280 pixels; 4 objects) used 2 colors x 2 shapes for low entropy and the 4 most distinct conjunctions from colors 4 shapes for high entropy (both involving 100 trialsccombination; 1600 total). The 3x3 grid (392x392 pixels; 9 objects) used 9 conjunctions from 3 colors x 3 shapes for low entropy and 9 objects with unique color and shape for high entropy (both involving 50 trialscoombination; 4,050 per condition).
15

---

Published as a conference paper at ICLR 2026
A.2 REPRESENTATIONAL SIMILARITY ANALYSIS PROTOCOL
We used representational similarity analysis (RSA) to assess the alignment of the model's internal representations with either the spatial position the semantic features (color and shape) of objects. RSA quantifies representational alignment by comparing pairwise representational similarity matri- ces (RSMs) between model embeddings and hypothesized representational structures, as quantified by correlation metrics.
A.2.1 TOKEN POSITIONS USED FOR RSA
We performed RSA on embeddings from two different token position sources. The first was the last token in the sequence, positioned immediately before the description of the target object. This token represents critical decision point where the model must integrate all relevant information to generate the appropriate object description. Analyzing this single token position yields RSMs of shape [1. T. TJ, where T represents the number of trials. The second source was the comma tokens that punctuate each object description in the prompt. Specifically, we analyze the comma tokens that follow each of the N-1 object descriptions preceding the target. These tokens serve as natural boundaries between object descriptions and we found that they encode information about the preceding object in the description. This multi-token approach produces RSMs of shape [N-1, T, TJ. For both token position sources, we extract activations from two locations within the model: the residual stream and the attention block outputs.
A.2.2 MODEL. RSM CONSTRUCTION
We constructed model RSMs by computing pairwise cosine similarities between activations across samples. All our RSAs were performed on residula stream. We systematically extract activations at the specified token positions, yielding single representation per layer. Given set of activations Act(i, t) for token and trial f, we computed pairwise similarities across all trial pairs. Each entry in the resulting RSM reflects the similarity of representations across two trials.
A.2.3 TARGET RSM CONSTRUCTION
To evaluate what type of information these representations encode, we constructed two types of target RSMs. The position-based RSM captures spatial relationships using ground-truth (æ; y) co- ordinates coord(i, () of object i in trial I, with normalized Euclidean distances:
Dicoord 41), co,rd (2))
RSMpos(i, 4,4
max D
The feature-based RSM represents semantic similarity through visual attributes. We constructed separate matrices for color and shape attributes, then combined them:
RSMcolor(i, f1, t2] = (coloori,, t1) = color(i, t2)) RSMshape 41,42] (2] H (shape(i, (1) shape (2))
(3)
RSMfeat(i, f2, 2 (RSMcolor(i, t,, RSM [1,, t2])
(5)
A.2.4 QUANTIFYING ALIGNMENT
We quantified the alignment between model representations and object features by computing Pear- son correlations between model RSMs and each target RSM. These correlations produce sealar similarity scores that vary depending on the source of the activations - for residual stream analysis, we obtained one score per layer, while for attention block analysis, we obtained one score per head per layer. By comparing these alignment patterns across two token sources, and between the residual stream and attention mechanisms, we can trace how different types of object information encoded and transformed throughout the model's processing hierarchy.
16

---

Published as a conference paper at ICLR 2026
A.3 ILLUSTRATION OF CMA CONDITIONS
For the CMA analysis all of the prompts follow the simple Scene description template In this image there is [COLOR] [SHAPE] and and the model is tasked to complete the sen- tence with the missing colored shape.
Here present schematic illustrations the different CMA conditions used identify the differ- ent sets of attention heads described in Section 3.4 namely the ID retrieval heads (Figure I0), ID selection heads (Figure IT, and the feature retrieval heads (Figure 12
Attertion
purple triangle
orange triangle
Patthing
Foature Retrieval Heads
Feature Retrivval Heads
0
1
ID
T
Selection Heads
Setection Heads
10 Ratieevll Heads
Ratrioval Heads
Thare an ozange triangle
orange triangle
Figure 10: Causal mediation procedure used to identify ID retrieval heads.
Feecfonward
Astordon
Patching
purple triangle
orange triangle
ID
ID 1D
1
ID=0
1D=1
Selection
Selection Heads
ID
Retrioval
Heads
Thare is an orange triangle and
There is orange
triangle and
Figure 11: Causal mediation procedure used to identify ID selection heads.
17

---

Published as a conference paper at ICLR 2026
Feadforeard
pink triangle
Patshing
pink triangle
puzple triangie
Foelura Retrioval Heads
Feature Ratrivval Heads
ID
Selection Heads
Selection
ID=11
D=1
Heads
ID
Heads
is orange triangle
orange triangle
Figure 12: Causal mediation procedure used to identify feature retrieval heads.
A.4 ILLUSTRATION OF ABSOLUTE VS. RELATIVE POSITION INTERVENTION
Figure illustrates the intervention performed to test whether position IDs employ absolute or relative spatial coding scheme. The figure depicts a pair of images involving two different 2x2 grid configurations. For the first image, prompt presented including out of the objects present in the image (specific prompt: In this image there is [COLORO] ] [SHAPEO], [COLOR2] [SHAPE2] and a), and the output of the ID selec- tion heads was extracted at the final token position (where the model generated descrip- tion of the target object). For the second image, prompt was presented including 2 out of the objects (specific prompt: Tn this image there is [COLORO] ] [SHAPR0], [COLOR1] [SHAPR1], a). One of the missing objects appeared at the same absolute position as the target object in the first image (the center of the image), and the other missing object appeared at the same relative position as the target object from the first image (the lower right quadrant of the 2x2 configuration). The ID for the target object in the first image was patched into the output of the ID selection heads for the target object in the second image, and we assessed whether the object generated by the model matched the prediction of the absolute vs. relative position hypotheses. The results are shown in Figure 30
[COLOR1] [SHAPE1],
Feedforward
Attention
Position IDs
Position IDD
Patching
purple heart?
orange square?
Same elstive
abeolute posiion
blue triangle
Feature Retrieval Heads
Retrieval Heads
Patched ID
There is pink circle, orange square, purple heart, and
ID Heads
ID Selection Heads
There is pink circle,
blue triangle, and
Figure 13: Illustration of the intervention testing for relative vs. absolute spatial coding.
18

---

Published as a conference paper at ICLR 2026
A.5
ADDITIONAL RESULTS
A.5.1
REPRESENTATIONAL ANALYSES
Here we present RSA results across a range of different open-source VLMs (Qwen2-vl, LLaVa 1.5, LLaVa- VaeVVV-vviiion---- 7b, and Qwen 2.5) and model scales (Qwen 2.5 3b, 7b, 32b). We find convergent evidence for the same two stage processing (position followed by object features) across all models and model scales.
RSA Similarity by Layer
1.0
Target
Target Features
0.8
0.4
12
1.6
32
Layer Number
Figure 14: Representational analyses for Qwen2.55VLL3b.
RSA Similarity by Layer
Target Position
Target Features
0.8
Convatton))
0.4
0.0
Figure 15: Representational analyses for Qwen2.5-VL-22
Similarity Layer
Target Position
Features
0.8
Cametel
0.6
0.2
0.0
24
28
36
66
Layer Number
Figure 16: Representational analyses for Qwen2.5-VL-32b
19

---

Published as a conference paper at ICLR 2026
RSA Similarity by Layer
Target Position
Target Features
0.8
0.6
0.2
0.0
Number
Figure 17: Representational analyses for Llaval.5-1
RSA Similarity Layer
Target Position
Target Features
0.8
0.6
0.2
0.0
12
16
20
Lager
Figure 18: Representational analyses for LlavaOnevision-7b.
RSA Similarity by Layer
1.0
Target Position
Target Festures
0.8
0.4
0.2
12
16
20
24
Layer Number
Figure 19: Representational analyses for Idefics2-8b.
20

---

Published as a conference paper at ICLR 2026
A.5.2 CAUSAL MEDIATION ANALYSES
(a) ID retrieval heads
(b) ID selection heads
(c) Feature retrieval heads
Figure 20: CMA results for Qwen-2.5-VL-33
CHA Scor
CM
Hobd Indes
Head Indee
(a) ID retrieval heads.
(b) IDD selection heads.
(c) Feature retrieval heads.
Figure 21: CMA results for Qwen-2.5-VL-78
(a) ID retrieval heads.
(b) ID selection heads.
(c) Feature retrieval heads.
Figure 22: CMA results for Qwen-2.5-VL-326
21

---

Published as a conference paper at ICLR 2026
(a) ID retrieval heads.
(b) IID selection heads.
(c) Feature retrieval heads.
Figure 23: CMA results for Llava- 1.5-7b.5.bb
(a) ID retrieval heads.
(b) ID selection heads.
(c) Feature retricval heads.
Figure 24: CMA results for Llava- 1.5-13b.
(a) ID retrieval heads.
(b) ID selection heads.
(c) Feature retrieval heads.
Figure 25: CMA results for Llava-One Vision.
22

---

Published as a conference paper at ICLR 2026
A.5.3 ADDITIONAL RESULTS FOR RELATIVE VS. ABSOLUTE SPATIAL CODING
PCA Analyses We performed PCA analyses 26 at two levels of granularity. First, analyzed all
objects across conditions, plotting them according to either their relative grid position (Figure 26a eg. top left), their absolute grid position (Figure Figure2663 Second, we focused specifically the central object (circled in dotted lines in Figure 8), coloring these representations by either relative grid condition (Figure[66d or semantic identity (Figure 2 If VLMs use absolute position encoding. the central object's position ID should cluster together gardless of grid arrangement, since occupies the same absolute location across conditions. VLMs use relative position encoding, the central object's position ID should vary systematically with the surrounding spatial context. We vizualize the PCA results for Qwen2-VL. Our PCA re- sults showed that representations of the central object clustered primarily by grid arrangement for the Qwen2-V VL [6d rather than remaining invariant across conditions. This provides evidence that some VLMs like Qwen2-VI. VL employ relative position encoding scheme, where spatial indices computed based on local spatial relationships between objects rather than absolute grid coordinates.
their semantic features (Figure 26c
Layer
Layer
14
Layer 15
19
Layer 24
25
(a) PCA results coded by target object relative position for all objects.
Layer
Layer
Layer
Layer
Layer
Layer 5
Layer
Layer
Layer
Layer
Layer 10
Layer 11
Layer 12
Layer 13
Layer 14
Layer 25
Layer
Layer 17
Layer 18
Layer 19
Layer 20
Layer 21
Layer 22
Layer 23
Layer 24
Layer 25
Layer 26
Layer 27
(b) PCA results colored by target object absolute position all objects.
Layer
Layer
Layer
Layer
Layer
Layer
Layer 6
Layer?
Layer
Layer
Layer 10
Layer Layer 12
Layer
Layer 14
Layer
Layer
Layer
Layer 18
Layer 19
Layer 20
Layer
Layer
Layer
Layer 24
Leyer 25
Layer
Layer
(c) PCA results colored by target object feature central object.
Laver
Layer
Layer
Layer
Layer
Layer
Layar
Layer
Layer
Layer
Layer
Layer
Layer
Layer17
Layer
19
Layer
Layer
Layer
Layor
Layor
Layer
Layer
Layer
(d) PCA results colored target object relative position for central object.
Layer
Layer
Layer
Layer
Layer
Layer
14
Layer
(e) PCA results colored by target object features for central object.
PCA results for all objects. (c-g) PCA results for central object only. Hidden state embeddings at last token position projected onto the top 2 principal components, coded by relative position, absolute position, or object features.
Figure 26: Principal Component Analyses. Stimulus condition shown in Figure g
23

---

Published as a conference paper at ICLR 2026
RSA Analyses. We performed RSA the last token in scene description task. The task involved completing a caption that described all but one of the objects present in the image. Our results (Figures 27 and 28) confirmed that the Qwen family of models employ relative position coding scheme, while the difference was less pronounced for the Llava family, Interestingly, those results also reflected in the effectiveness of the interventions reported in Figure 30 We also tested model from the Idefics family, finding similar results as those seen for the Qwen models (Figure 29
Repretentation Similarity
Representational Similarity
1.0 Representational Similarity
War CA HANOEE
(a) Qwen2.5-VL-3b
(b) Qwen2.5-VL-7b
Qwen2.5-VLL332
Figure 27: RSA results comparing relative vs. absolute spatial coding schemes for Qwen2.5 models.
Represental lonal Similarity Across Layare
Representatioal Similarity Acroes Layers
Regresentationa Similarit Acroes Layers
Paaron Canmeaae
har
Paaron Cente
(a) QWen2-VL
(b) Llaval.5-7b
(c) Llaval.5-13b
Figure 28: RSA results comparing relative vs. absolute spatial coding schemes for Qwen2 and L models.
Representational Similarity Across Layers
1.0
Texture
0.8
Position
Pearson Correlation
0.6
0.4
0.0
10
15
20
25
30
Layer
(a) Idetics2-8b
Figure 29: RSA results comparing relative vs. absolute spatial coding schemes for Idefics2-8b.
24

---

Published as a conference paper at ICLR 2026
Intervention Setup. Figure 30 the results of the intervention testing for relative absolute
spatial coding, as described in Section A.4
Position Patching Performance
1.0
Absolute
Relative
0.8
Proportion Correct
0.6
0.4
0.2
0.0
Qwen 2.5
Qwen 2.5
Qwen 2.5
Qwen 2
LLaVA-1.5
LLavA-1.5
3B
7B
32B
7B
7B
13B
Figure 30: Intervention results testing for relative vs. absolute spatial coding schemes.
A.5.4
BINDING ERRORS
Here report RSA results in the low high entropy conditions, illustrating the relationship between position IDs and binding errors (similar to Figure for all models.
Prompt Tokens Position RSA
Last Taken Position RSA
Prompt Tokens Position RSA
Last Token Position RSA
Jow trtrees
Low Entrary Entrepy
0006 Corvetllion V7
righ
Cur Corrsaationn Irt
E.4
B00 conretaiion
Correttton
Ra
0.
0.
0.0
6.0
Layor
Layor
Layor
Layer
Figure 31: RSA for Qwen2- VL.
Prompt Tokana Position RSA
Last Takan Position R5A
Prompt Tokens Position R5A
Last Token Position Lon Entrayy
Curdistion (n
Hight Entospy
0104 Corretetoo
BAC Correfation
0.4
30
Lay
Layor
Layor
Layer
Figure 32: RSA for Qwen2.5-VL-3b.
25

---

Published as a conference paper at ICLR 2026
Prompt
Prompt
6.5
0025
Commtatioo or
ASA Commiation 6/1
6.4
High troney
DOSA Cormeatiin
0.20
92
0.00
60
Figure 33: RSA for Owcn2.5-VL-7b.
Prompt Tokens Pocition RSA
Last Token Position RSA
Prompt Tokens Position RSA
Last Token Position RSA
K.5
1004 wwteaattioono
RSA Cornoadoonn
044
02
122
0.3
0.3
0.0
60
42
60
20
63
20
40
Lay
Layer
Layer
Figure 34: RSA for Qwen2.5-VL-322
Low bringr
High Enklayy
B00 Chorelation 07
ISA Corriattonn o1
Box Conretation
0.3
0.20
0.15
6,12
02
8.05
0.00
5.00
Loy
Layer
Lay
Layer
Figure 35: RSA for Llaval.5-7b.
Prompt Tokens Position RSA
Last Token Position RSA
Prompt Tokens Position RSA
Last Token Position RSA
0.4
25A coereeitenn
044
0.
0.
ISA Conssdden
0.3
03
0.0
Figure 36: RSA for 1.33b1.5-13b.
Tokana Poeition
Last Takan Position R5A
Prompt Tokens Position R5A
Last Token Position
Lon Enirgyy
06
Hight Enlospy
0104 Correlttion
BAC Carrelatter
Comedtttoon
RSA Correaation
R4
122
22
3.0
6.0
12y/
Layor
Layor
Layer
Figure 37: RSA for LlavaOne-7b.
26

---

Published as a conference paper at ICLR 2026
A.5.5 OPTIMIZING HYPERPARAMETERS FOR POSITION ID INTERVENTION
In this section, we report the hyperparameter sweep results for the position ID intervention using the PUG environment. We report intervention performance as a function of the number of top-K heads that are intervened on and the magnitude of the intervention coefficient a, as described in Section 4.2
Retrieval Heads
Selection Heads
Feature Retrieval Heads
Tp Kwz
Ep Kw3
top---5
K-10
0.06
Top
lop-K-30
Top-kst0
Rpp K-50
1p-K--0
Top K-100
Magnitude Intervention
Magnitude Intervention
Magnitude Intervention
Figure 38: Qwen-2.5-VL.
2.5- VL5-VL-3B
Retrieval
Heads
Retrieval
3.0
Tpp
1ap-K-10
Top Ki-12
0.6
Top K-15
Tpp--220
Top K-30
lop-kmt
Tp-Kisto
10 Magnitude of Intervention
10
10a
10a
10'
10
Magnitude of Intervention
Magnitude of Intervention
Figure 39: Qwen-2.5-V1L77B
Id Retrieval H Heads
Selection Heads
Feature Retrieval Heads
lpp--2
Rp K-5
Tpp--110
Top K-12
kp-K-15
Tpp---20
Top
1pp---100
10a Magnitude Intervention
10
10a
10
10' Magnitude of Intervention
10'
10
Magnitude of Intervention
Figure 40:
27

---

Published as a conference paper at ICLR 2026
Retrieval
Selection
hp-Koz
K-5
Tip
0.6
Top
Top--20
Top K-30
0.4
Top K-40
kp-K-50
022
Rop--110n
Top K-200
10
10
10a
10a
10a
10*
Magnitude Intervention
Magnitude of Intervention
Magnitude of Intervention
Figure 41: Qwen-2-VL
Retrioval Heads
Selection Hleads
Retrieval
kpp-22
Tp-Ka5
Top
Tp-K-100
Top K-40
kppX-50
Tp-K-1100
Top K-200
0.0
Magnitude Intervention
Magnitude of Intervention
Magnitude of Intervention
Figure 42: Llava-1.5-7B
Retrieval Heads
Selection Heaads
Retrieval
bp-K-2
K-5
lopK-10
Top
lop-K=20
Tpp-iu30
Top K-50
lop--1100
Kp200
Magnitude Intervention
Magnitude of Intervention
Magnitude of Intervention
Figure 43: Llava-1.5-13B
Retrieval Heads
Solection Heads
Feature Retrieval
bp K-2
bp-K-5
0.8
0.6
Top
Top--330
Kud0
Top K-50
Top--1100
Top K-200
Magnitude Intervention
Magnitude Intervention
Magnitude Intervention
Figure 44: Llava-One Vision
28

---

Published as a conference paper at ICLR 2026
B GENERALIZATION OF POSITION IDS TO REAL-WORLD SETTINGS
Obtain ID Order (Source)
ID Estimation (Source)
STOP
car 2. tree....
STOP
=D
VLM
Selection Heade
There is 1.
There is 1.
Obtain ID Order (Target)
Intervention (Target)
cat
Feature Retrieval Heads
cat 2. mag.
ID 0
ID Selection Heads
VLM
There is
There is 1. 2.
Figure 45: Illustration of the intervention performed with the COCO dataset.
In this section, we verify that the use of position IDs generalizes to real-world images. Importantly, in contrast to the experiments conducted in the main sections, for real-world spatial arrangements we do not know the assignment IDs to objects in the image. We hypothesized that the assignment order would be reflected by the order in which objects were enumerated in open-ended scene description task. Accordingly, we obtained an ordering for the objects in an image based on the order in which the model described these objects, and then used this ordering perform intervention (Figure (45)
first created separate source and target sets using images from the COCO dataset (Lin et al., 2014). The source set was used to estimate position IDs that were then used intervene scene description for the target set. For each image in the source set, we first ob- tained the model's assigned order by presenting the prompt 'In this image there is 1. a'. We then parsed the response and extracted the first two objects (O,, O₂) listed by the model (fil- tering out cases in which the model described the same object twice). Then, using the same image and prompt, we extracted the output of the ID selection heads from the final position (at which the model generated On). We performed this procedure for all images in the source set and averaged the resulting embeddings, yielding IDo,, an estimate the position ID for object Og-
Specifically,
We then used this estimated position ID embedding to intervene on scene description with images from the target set. For each image in the target set, we again obtained the model's 's assigned order for the first two objects (Oo, O1). We then presented the model with the same image and the prompt 'In this image there is 1. a Ou 2.
(where O₂ was filled in based on the model's assigned order), and replaced the output of the ID selection heads at the last token position with the estimated position ID embedding ~ IDo.. Our prediction was that this intervention should cause the model repeat Og-
29

---

Published as a conference paper at ICLR 2026
For this experiment, used the COCO 2017 validation with 3 random splits for the source and target sets. We performed the intervention on the top 50, 100, and 200 ID selection heads (as identified by the CMA scores in A.5.2). We report the results for models from the Llaval.5 and Qwen2.5 families in Figure 46
0.8
Mean Accuracy
0.6
Intervention
top 50
top
top 200
LLava-1.5-133 b--13
0.0
Qwen-2.5-7b
Qwen-2.5-3b
LLava-1.5-7b
Figure 46: COCO intervention results. Intervention performed on ID selection heads.
C INVOLVEMENT OF VISUAL SYMBOLIC MECHANISMS IN COUNTING
Max
Bottom
Counting Accuracy
200
300
530
Top Heads
Figure 47: Ablation of top- vs. bottom-K K heads (as identified by CMA) on counting task. Note that the model contains a total of 2, 560 attention heads.
To test whether the visual symbolic mechanisms identified in the main text are involved in more complex tasks that require binding, we performed an experiment with counting task. We hypothe- sized that these mechanisms are involved in enumerating the objects present in an image, and that the ablation of these mechanisms would interfere with this enumeration process, and therefore interfere with the final count provided by the model. To test this, sorted the attention heads in the model according to their scores (as given by the maximum CMA score for each head from the identified stages) and ablated (by setting the output 0) either the top-K bottom-K heads according to
30

---

Published as a conference paper at ICLR 2026
those scores. We tested the impact of this ablation in counting task. The task involved images with between and unique objects with the following characteristics:
shapes sampled from: triangle, square, circle, cross
colors sampled from: red, blue, green, yellow
positions sampled randomly from a 3x3 grid
We used the following prompt for this experiment:
Counting Task Prompt
You are given image containing multiple colored objects. Your task is carefully observe the image and idanti fy all the que colored objects present.
Knumerate all the unique colored objects you find in the image, provi di numbered 1ist for clarity.
After listing the objects, provide the total count of these unique colored objects.
Format the total count by writing 'Answer:' followed by the number. It is crucial to adhere to this format: Annser: TOTAL_COUNT'..
shows the ablation results for the best performing model on this task, Qwen2.5-32b. Ablation of the top-K heads had a much stronger impact on task performance than than the bottom- K heads, with performance falling to 0% after the ablation of the highest-scoring 250 heads (out of 2,560 heads total), whereas the model still displayed performance close to 70% even after ablating the lowest-scoring 500 heads. These results indicate that the identified visual symbolic mechanisms play an important role in more complex tasks that require binding, such as counting.
Figure 47
D BINDING ERROR INTERVENTIONS
Model
High Entropy
Low Entropy
Improvement
No Intervention
With Intervention
LLaVA 1.5 7B
70.30%
27.50%
32.10%
4.60%
LLaVA VA 1.5 13B
75.00%
41.20%
51.60%
10.40%
Owen 2.5-VL. 3B
91.20% 89.90%
69.00% 51.20%
80.10% 62.30%
11.10%
Qwen 2.5-VI. 7B
11.10%
Table 1: Performance (accuracy) on high vs. low entropy scene description. Intervention performed on ID selection heads low entropy condition.
To causally the relationship between binding errors and failures in the position ID mechanism, we conducted an activation patching intervention targeting the ID selection heads identified in Sec- tion 3.4 We leveraged the observation that models generate robust position ID representations in high-entropy settings (distinct feature conjunctions) but suffer from representational collapse in low- entropy settings (shared features). We first computed the mean activations of the top-k ID selection heads during successful trials on the high-entropy dataset to extract robust prototype of the spatial symbol for each grid position. We then intervened during the processing of low-entropy trials by replacing the activations of the ID selection heads with those obtained from the high-entropy trials.
The results of this intervention are presented in Table I Injecting position IDs from high-entropy trials significantly improved performance in low-entropy trials across all tested architectures. For instance, Qwen 2.5-VL 3B improved by 11.1% %, while LLaVA 1.5 13B improved by 10.4%. Notably, the effectiveness of the intervention appears contingent the quality of the source representations; LLaVA 1.5 7B showed the smallest improvement (+4.6%), consistent with its lower baseline perfor- mance in the high-entropy setting (70.3%). This suggests that because the source position IDs were
31

---

Published as a conference paper at ICLR 2026
less precise, the resulting patch was less effective in improving performance in the low-entropy condition. Overall, these findings demonstrate that patching more precise position IDs from the high-entropy condition can attenuate binding emrors during the low-entropy condition, providing causal evidence that position ID failures are responsible for binding errors.
32


Reviews:
[{"review_id": "X1Z42cZL2d", "reviewer": "Reviewer_7kgv", "summary": "In this paper the authors shed light on how VLMs work, specifically how they solve the binding problem (associating features together to represent distinct objects). The authors identify the “position IDs” mechanism composed of 3 steps:  first retrieval of the position ID of the image objects corresponding to the input prompt. Then, selecting the position ID for the target object, and finally retrieving the semantic features of the target object. The authors provide evidence for this scheme across 7 models using synthetic and photorealistic datasets.", "questions": "How does the size of the model affect the results ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors identified an interesting internal working mechanism of VLMs, explaining also why VLMs sometimes fail in spatial reasoning. I think understanding how VLMs work is a high-impact problem given recent benchmarks showing that VLMs fail in spatial reasoning tasks.\n- I think the authors provide enough evidence to support the “position IDs” mechanism. They identify what layers correlate with each step and perform causal mediation analysis to identify the specific attention heads that are causally involved in the 3 steps. \n- The authors show that binding errors are correlated with failures in the position ID mechanism, potentially informing future improvements to VLMs architectures.", "weaknesses": "- It is not clear what the novelty is in the paper in terms of methodology and techniques compared to the paper of Yang et. al (2025) that identifies similar mechanisms yet for LLMs. I think the authors should discuss it in the paper.\n- The correlation between VLMs failures and position ID mechanism failures is just correlation, not causation. It is not clear if mechanism failures really cause binding errors.\n- I think the writing can be improved, e.g. provide the specific prompt that is used in each case, specifically in section 3.4 and in 4.1.\n- Minor: there are 2 “Figure ??” in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cdWsgYo4p9", "reviewer": "Reviewer_LPtx", "summary": "The paper studies how vision-language models address the binding problem, which involves linking visual features, such as color and shape, to specific objects. The paper proposes that these models use spatial \"position IDs\" as symbolic indices to perform this binding. Through representational similarity analysis, causal mediation, and targeted interventions on various VLM families, the authors identify three groups of attention heads responsible for retrieving, selecting, and using these position IDs to recover object features. The authors argue that this mechanism explains how VLMs achieve visual binding and why binding errors occur.", "questions": "Could the authors clarify what is genuinely new about these mechanisms in the visual domain beyond applying known LLM findings to spatial inputs?\n\nTesting whether the proposed mechanisms persist under more realistic conditions would be essential to support their claim of generality.\n\nCould the observed differences between models simply reflect their positional encoding priors (for example, relative RoPE embeddings in Qwen versus absolute sinusoidal embeddings in LLaVA) rather than evidence for an emergent symbolic mechanism?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses an important question about how VLMs perform visual binding and compositional reasoning.\n\nIt evaluates a diverse range of model architectures and scales (Qwen2.5-VL, LLaVA-1.5, and LLaVA-OneVision), differing in backbones, design choices, and training data, which strengthens the generalizability of the findings.\n\nIt employs multiple complementary methods, representational analysis (RSA), causal mediation, and targeted interventions, to support its hypotheses from several analytical perspectives.", "weaknesses": "The main claim that VLMs develop symbolic binding mechanisms similar to those in LLMs seems incremental since comparable mechanisms are already well-established in language models. The results show that related processes emerge in multimodal settings, which seems expected given the shared Transformer backbone. Could the authors clarify what is genuinely new about these mechanisms in the visual domain beyond applying known LLM findings to spatial inputs?\n\nThe paper aims to demonstrate general mechanisms of visual binding, yet the datasets are highly controlled and simplified. Even the “photorealistic” PUG scenes contain only two clearly separated, uniformly sized objects, avoiding occlusion or clutter. Testing whether the proposed mechanisms persist under more realistic conditions would be essential to support their claim of generality.\n\nThe explanation of the relative versus absolute index analysis could be made clearer, as the underlying idea is difficult to follow in its current form.  In addition, could the observed differences between models simply reflect their positional encoding priors (for example, relative RoPE embeddings in Qwen versus absolute sinusoidal embeddings in LLaVA) rather than evidence for an emergent symbolic mechanism? Clarifying this distinction would strengthen the interpretation of the results.\n\nMinor comments:\nLine 306: Typo in “wihtin”\nLine 347: Missing space.\nLine 427: The reference to the Appendix does not lead to the generation details as stated. The appendix structure should be cleaned up.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Txv0iyfZfT", "reviewer": "Reviewer_16UN", "summary": "This paper investigates whether vision-language models employ symbolic mechanisms similar to those found in language models for solving the binding problem: the challenge of correctly associating features to form coherent object-level representations. Using a scene description task where models must describe remaining objects in multi-object images, the authors employ a thorough suite of methodologies in order to uncover the underlying mechanisms across a range of VLM architectures: PCA, representational similarity analysis (RSA), causal mediation analysis (CMA), and targeted interventions. The core finding is a three-stage binding process: 1) Position ID heads compute spatial indices for objects mentioned in the prompt; 2) ID selection heads select the position ID for the target object; and 3) Feature retrieval heads use the selected position ID to access target object features. The authors then perform a series of analyses to characterize the binding mechanisms in greater detail. They find that models tend to use relative vs. absolute position IDS; that the binding mechanisms generalize to more realistic images; that position IDs are localized to image patches containing the relevant object; and that the same position IDs are used in more complex relational reasoning tasks. They finally perform an error analysis, demonstrating that binding errors correlate with failures in position ID processing. \n\n\nThe authors then demonstrate that binding errors correlate with failures in position ID processing, particularly in low-entropy conditions where objects share features.", "questions": "1. Do the authors have hypotheses or preliminary observations about when / how these position ID mechanisms emerge during training or what factors might influence their emergence (architecture, training data, etc.)?\n2. Given the discovery of these three distinct stages (retrieval, selection, feature retrieval), could architectures be explicitly designed to separate or reinforce them, similar to slot-attention or object-centric models?\n3. Is there evidence that the position IDs interact systematically with the text tokens (e.g., positional embeddings in the language stream), or are they confined to the vision encoder?\n4. In the scene description task, are object attributes (e.g. shape, color) repeated in the scene? (e.g. two or more green objects, two or more squares). If a scene of three objects contains three distinct colors and shapes, then it seems that models could solve the task by just picking whatever attributes are left out of the prompt rather than binding.\n5. The term \"symbol\" is strong and implies total content-independence. To what extent do the authors believe these mechanisms are truly symbolic, e.g. in the way cognitive scientists mean?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Importance of the research question**. The binding problem represents a fundamental challenge in AI systems, and understanding how VLMs solve it has significant implications for improving multi-object reasoning capabilities. \n2. **Novel mechanistic insight into VLMs**. The paper makes a strong conceptual and empirical contribution by identifying position IDs (content-independent, spatially grounded indices that play a symbolic role in feature binding). This is a novel mechanistic finding in multimodal interpretability.\n3. **Methodological rigor**. The paper employs a comprehensive multi-method approach. The progression from RSA to CMA to direct interventions provides converging evidence for the proposed mechanisms.\n4. **Cross-model validation & generalization**. The authors test their findings across multiple VLM architectures (Qwen-VL, LLaVa variants). The appendices contain detailed methodological descriptions, extensive additional analyses, and validation experiments.", "weaknesses": "1. **Limited exploration of real-world consequences.** While the authors link the discovered symbolic binding mechanisms to binding failures, they do not connect the mechanisms to the downstream behavioral consequences of binding failures gestured towards in the introduction (e.g. counting, visual search). Bridging this gap would strengthen the work.\n2. **No investigation into learning origins.** The paper identifies what the mechanisms are but not how they arise. Do they emerge naturally from pretraining data distributions, or are they artifacts of architectural bias? \n3. **Unclear how to improve binding**. Related to Weakness 2, the authors do not attempt to modify / strengthen the binding mechanisms. This would be practically useful and would also enable direct causal claims about the relationship between these mechanisms and downstream behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nUD66fWmJu", "reviewer": "Reviewer_LWd9", "summary": "This paper investigates emergent symbolic mechanisms for binding features and objects in vision language models. It presents evidence for a binding process where the model first computes a spatial index for the target object and then uses that index to retrieve its features. Overall, this paper looks like a good paper with solid methodology and evidential conclusions to me. The proposed binding mechanism is justified and intuitive, and more importantly generalisable across different models, which makes the paper relevant and useful to community.", "questions": "The paper says improving binding performance may require either architectural innovations or training strategies. Is there a good benchmark/criterion (i.e. the one used in this paper) to show progress? How generalisable is the conclusion to more complex open-domain?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The analysis methods used including PCA and RSA to localise when position and features dominate, CMA and intervention to find contributing heads are soild methods.\n3. Given that the experiments throughout 7 VLMs show a consistent pattern in the feature binding mechanism, the findings and conclusions are generalisable.", "weaknesses": "1. Although 7 VLMs are tested, they are from 2 model families (Qwen and Llava). The mechanism still generalises with the current setting but can be strengthened when tested on more model families. \n2. The experiment is conducted under controlled synthetic settings. While this is required for a grounded analysis, explanaing how the mechanism can be exploit/identified in open-domain can be useful.\n3. Some more explanation under Fig1 caption can help me to understand the paper better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)