Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers?
Authors: Yihao Li
Saeed Salehi
Lyle Ungar
Konrad Kording
Venue: NeurIPS 2025 spotlight ()
Keywords: Object binding
Vision Transformers (ViTs)
Neural coding
Cognitive science
Interpretability
Primary area: neuroscience_and_cognitive_science
TL;DR: Vision Transformers (ViTs) perform object binding.

Abstract: Object binding, the brain’s ability to bind the many features that collectively represent an object into a coherent whole, is central to human cognition. It groups low-level perceptual features into high‑level object representations, stores those objects efficiently and compositionally in memory, and supports human reasoning about individual object instances. While prior work often imposes object-centric attention (e.g., Slot Attention) explicitly to probe these benefits, it remains unclear whether this ability naturally emerges in pre-trained Vision Transformers (ViTs). Intuitively, they could: recognizing which patches belong to the same object should be useful for downstream prediction and thus guide attention. Motivated by the quadratic nature of self-attention, we hypothesize that ViTs represent whether two patches belong to the same object, a property we term *IsSameObject*. We decode *IsSameObject* from patch embeddings across ViT layers using a quadratic similarity probe, which reaches over 90\% accuracy. Crucially, this object-binding capability emerges reliably in DINO, CLIP, and ImageNet-supervised ViTs, but is markedly weaker in MAE, suggesting that binding is not a trivial architectural artifact, but an ability acquired through specific pretraining objectives. We further discover that *IsSameObject* is encoded in a low-dimensional subspace on top of object features, and that this signal actively guides attention. Ablating *IsSameObject* from model activations degrades downstream performance and works against the learning objective, implying that emergent object binding naturally serves the pretraining objective. Our findings challenge the view that ViTs lack object binding and highlight how symbolic knowledge of “which parts belong together” emerges naturally in a connectionist system.

Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers?
Yihao Li'
Saeed Salehi² Lyle Ungar' Konrad Kording'
University of Pennsylvania liyihao@seas upenn. edu, ai.neuro io@gmail. ungarocis. upenn. edu, koerdingggmiil.co com
Machhne Learning Group, Technical University of Berlin
Abstract
Object binding, the brain's 's ability to bind the many features that collectively represent an object into coherent whole, is central to human cogmition. groups low-level perceptual features into high-level object representations, stores those objects efficiently and compositionally in memory, and supports human reasoning about individual object instances. While prior work often imposes object-centric attention (e.g., Slot Attention) explicitly to probe these benefits, it remains unclear whether this ability naturally emerges in pre-trained Vision Transformers (ViTs), Intuitively, they could: recognizing which patches belong to the same object should be useful for downstream prediction and thus guide attention. Motivated by the quadratic nature of self-attention, we hypothesize that ViTs represent whether two patches belong to the same object, property term IsSameObject. We decode Objeect from patch embeddings across ViT layers using a quadratic similarity probe, which reaches over 90% accuracy. Crucially, this object-binding capability emerges reliably in DINO, CLIP, and ImageNet--uuprrisee weaker in MAE, suggesting that binding is not trivial architectural artifact, but an ability acquired through specific pretraining objectives. We further discover that IsSameObject encoded in low-dimensional subspace on top of object features, and that this signal actively guides attention. Ablating IsSameObject from model activations degrades downstream performance and works against the learning objective, implying that emergent object binding naturally serves the pretraining objective. Our findings challenge the view that ViTs lack object binding and highlight how symbolic knowledge of "which parts belong together" emerges naturally connectionist system.
ViTs, but is markedly
1 Introduction
Humans naturally parse scenes into coherent objects [ (e.g., grouping features such as rounded shape, smooth surface, and muted color into the mug) and further ground their identities in context (e.g., recognizing my coffee mug on the desk rather than just a mug). This is assumed to be made possible by what cognitive scientists call object binding [2. the brain's ability to group an object's low-level features (color, shape, motion, etc.) into unified representation. This in turn enables objects be stored efficiently and compositionally in memory and used as high-level symbols for reasoning. The binding problem is genuine computational challenge, as evidenced by humans' limited competence conjunction-searge tasks (3] and clinical dissociations such as Balint's 's syndrome, where feature perception remains intact but binding breaks down [4] If AI systems could replicate the human ability for object binding, that may help them ground symbols for
Code available at: wttttpr//shhhll..o//yyyylo/ooppp binding
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

perception and exploiting compositionality [3]. The key question is: do current Al systems solve the binding problem?
Instance Segmentation
dog
Gee2
Q: Where's is the dog with light brown ears?
A:
Visual Reasoning
Q: How many silverwwhite cats are there in the image?
Probe
A: Two.
SSammOOject 0.09
Probe
Differntt Coject
IsSameObject 0.75
(a)
(b)
Figure Assessing object binding in ViTs with IsSameOhject. (a) We use probe to decode IsSameObject, with scores near I for same-object pairs and near 0 for different-object pairs. (b) Downstream tasks that benefit from strong object binding include instance segmentation and visual reasoning (e.g., locating and counting objects with specific features), where patches triggered by certain features are bound to the rest their object to allow extraction of the entire object.
Object binding has received little attention in mainstream AI research. Cognition-inspired models [6] 77) build in human-like object-based attention. By contrast, mainstream vision models are assumed to implicitly learn handle multiple objects from training data, yet empirical studies show they often "attend" only to the most salient regions and overlook the rest [8] While ViT attention scores can capture global image structure and salient regions (often corresponding to target objects) (9), empirical evidence shows that self-attention tends to group patches by low-level feature similarity rather than reliably producing object-level binding [1O]. Object-centric methods like Slot Attention [II] fix this by allocating small set of learnable slots that compete for token features, enforcing binding by design. However, whether Al vision models, especially leading ViTs, can achieve robust object binding without explicit mechanisms remains open question.
Cognitive scientists have questioned whether ViTs can bind objects at all: arguing that they lack mechanisms for dynamically and flexibly grouping features [5; they lack recurrence necessary for iterative refinement of object representations 12 7 and purely connectionist models, they appear incapable of true symbolic processing binding from emerging through learning. object (JSSameObjec)), this signal can guide attention and improve prediction [ [3]. Human-labeled data also reflects object-level structure, so ViTs can acquire binding by imitation. This suggests that ViTs may learn to bind objects directly from large-scale training data, without requiring explicit architectural inductive biases.
[8]. However, these architectural limitations do not preclude model encodes whether two patches belong to the same
Here, we ask whether object binding naturally emerges in large, pretrained Vision Transformers, which is question that matters for both cognitive science and AI. We propose IsSameObject (whether two patches belong to the same object) and show that it is reliably decodable (with 90.20% accuracy) using a quadratic similarity probe starting from mid-layers of the transformer layers. This effect is robust across DINO, CLIP and ImageNet-supervised ViTs, but largely absent in MAE, suggesting that binding is an acquired ability rather than a trivial architectural artifact. Across the VIT's layer hierarchy, it progressively encodes IsSameObject in low-dimensional projection-space on top of the features of the object, and it guides self-attention. Ablating aSameObject from model activations hurts downstream performance and works against the pretraining objective.

---

Our main contributions are as follows: (i) We demonstrate that object binding naturally emerges in large. pretrained Vision Transformers, challenging the cognitive-scient binding isn't possible given their architecture. (ii) We show that ViT' encode a low-dimensions signature of IsSemeOhject (whether two patches belong to the same object) on top of their feature representations. (iii) We suggest that Garning-objective-e binding, pointing future work toward implicitly learned object- representations.
assumption that such
inductive biases can enable object
2 Related Work
Object Binding in Cognitive Science and Neuroscience. The object binding problem asks how the brain integrates features that are processed across many distinct cortical areas into coherent object representations (1). The concept of bindingrrests on three key hypotheses: First, visual processing is widely understood to be hierarchical, parallel, and distributed across the cortex [0) Second, perceive the world primarily in terms of objects, rather than collection of scattered features [12 M. This abstraction is fundamental to both perception and interaction with the world, allowing us to recognize, reason about, and manipulate
environment effectively [1 23) Third, feature binding requires a mechanism that correctly assigns features, represented in spatially distinct cortical areas, to their corresponding object [15] 24 2 This third hypothesis is where the core of the binding problem lies, and it has been a longstanding point of debate among neuroscientists and cognitive scientists (25
Despite their substantial difference, vision transformers (ViTs) share several key computational parallels with the mammalian visual system: they both rely parallel, distributed and hierarchical processes. More importantly. ViTs do have two of the three architectural and computational elements hypothesized to enable binding in the brain. The explicit position embeddings in VITs resemble
spatial tagging and the spatiotopic organization observed in the ventral stream [28] 26]; and the
self-attention mechanism is akin to dynamic tuning and attentional modulation, which are thought
to be primary mechanisms for object binding [26 29 30
(although attention is believed to be of
recurrent nature in the brain [1 32) These parallels position ViTs potential computational
models for exploring object binding in both artificial and biological systems.
Object-Centric Learning. Motivated by how humans naturally reason about individual objects, Object-Centric Learning (OCL (TT) aims to represent a scene as a composition of disentangled object representations. While segmentation only partitions an image into object masks, OCL goes further by encoding each object into its own representation 333 Unsupervised approaches such as MONet (34,,
IODINE [33), and especially Slot Attention [1T] encode scenes into small, permutation-invarian set "slos" that are iteratively refined, producing robust object representations on both synthetic [1 35
and real-world data (36] 377 and enabling compositional generation and manipulation (38 How- ever, since Slot Attention is added as an external module rather than integrated into the transformer architecture, introduces additional challenges for scaling and training 4T.. Other explicit object-
centric approaches include Tensor Product Representations 442 and Capsule Networks
4
Instead of object-centric approaches that explicitly enforce object-level attention, we propose an alternative view that ViT's may already encode implicit object-level structure. Prior work has assumed this and attempted to group patches into objects directly from activations or attention maps ViTs, using methods like clustering (4 or GraphCut (5). [46) conduct a behavioral experiment where participants judge whether two dots belong to the same object at varying distances, and show that patch-level feature similarity in self-supervised ViTs supports object-based grouping. Building on this line of work, we show that ViT patch embeddings intrinsically encode whether any two patches belong to the same object, and analyze how this information is structured through probing.
Binding in Transformers. Binding has received growing recognition in transformer-basee machine learning research and binding failures are seen as examples of performance breakdowns in modern applications [77-50]. Diffusion models rely on binding attributes entities, and failures cause attribute leakage (e.g.. both a dog and a cat end up wearing sunglasses and a sun-hat) [48] 47]. Vision- language models face similar binding challenges, struggling with differentiating multiple objects with feature conjunctions [50]. Despite these binding failures, transformers still demonstrate some binding
"The term binding was introduced to neuroscience by Christoph von der Malsburg in 1981, inspired by the notion of variable binding in computer science [15].
3

---

capability, yet the underlying mechanism is not well understood. Feng and Steinhardt [[]]. Dai et al. [52] study binding in language models, showing that attributes (e.g., 'lives Shanghai") are linked to their subjects (e.g., "Alice") via low-dimensional binding-IID code that is added to the activation and can be edited to swap or redirect relations. Binding mechanisms in vision transformers remain unexplored, and our study aims to fill this gap.
3 Assessing Object Binding in ViT's through IsSame
(a)
(b)
(c)
Figure IsSameObject predictions distinguish objects in highly complex scenarios [53] Quadratic probe results for DINOv2-Large at layer 18 show that overlapping deer can be dis- tinguished and that disconnected regions of the same deer are correctly retrieved in (c).
3.1 Probing IsSame representations
Vision Transformers (ViTs) tokenize images by dividing them into a grid of fixed-size patches [54]. Because the token is the minimal representational unit, any grouping of features into objects must arise through relations between tokens, not within them. The only mechanism ViTs have for such cross-token interaction is scaled dot-product attention, where attention scores can viewed as dynamic edge weights in graph that route information between tokens (5). Therefore, if ViTs perform any form object binding, we expect to observe a pairwise token-level representation that indicates whether two patches belong to the same object, which we term IsSameObject.
Since object binding is the ability to group an object's features together, decoding IsSameObject reliably from VIT patch embeddings would provide direct evidence of object binding (and its representation) in the model. We adopt probing, which takes measurements of ViT activations with lightweight classifiers (55, to determine whether /sSameObject is encoded or unrecoverable by simple operations.
Formally, we define the IsSameObject predicate on a pair of token embeddings (z) (9) at layer by
IsSameObject (z. (9) xx))) = oa x xx))).
Rdd x Rd [0, 1].
where ó scores the probability that tokens and j belong to the same object.
Here, ask whether models reliably encode IsJeme and, if so, what mechanisms they to do so. We consider the following hypotheses about how IsSameObjecr may encoded in the model's activations:
It may be linear (recoverable by weighted sum of features) or fundamentally quadratic (recoverable only through pairwise feature interactions).
It is a pairwise relationship versus pointwise mapping (i.e. the model first maps each patch to a discrete object identity or class, then compares).
The model tells objects apart using only broad class labels or object identities-i.e i..., may rely class-level recognition ("dog vs. chair") instead of explicitly binding pixels to objects, as class labels already encode coarse notion object identity.
The signal is stored in few specialized dimensions versus distributed across many dimensions. In the former case, binding information would be isolated to small subset of

---

channels, while in the latter it would be encoded diffusely (e.g., as rotated combinations of features) such that no single dimension carries the signal on own.
To test these hypotheses, decode IsSameObject using several probe architectures, each parame- terized by a learnable matrix and scalar bias All probes are constructed be symmetric in their inputs, reflecting the constraint IsSameObject(z; y) IsSameObject(y, z). Throughout, ar-) denotes the sigmoid function.
1. Linear probe.
IsSameOOjechh (x. o(Wx: Wy b)
W WERI beR.
2. Diagonal quadratic probe (specialized dimensions).
IsSameOOjectaang
W
where W E Rddd is constrained to be diagonal, the probe uses only d parameters, each corre- sponding to a single feature dimension.
3. Quadratic probe (distributed).
IsSameOOjeecguaaa v) a(x W W2 W2y+b).
where W1, W/2 Rexd with h < d. To enforce symmetry, we set W2 = SW1, where S is diagonal matrix with entries in {+1}, yielding low-rank quadratic form with ( parameters. Object-class object-identity probes (pointwise). We first map each embedding to a probability distribution:
p softmax(Worr + b), q softmax (Wyy + 6),
where W. is trained using multiclass cross-entropy object-class labels (and similarly WN for object-identity labels). The pointwise IsSaneObject score then defined as the inner product the two distributions:
N Σ p(c) ((c).
JsSamm@Ohjeccaass///eeemmii (I,
p_q
3.2
IsSameObject is best decodable in quadratic form
IsSameObjech Class. Acc.
0.90
Linear
Diagonal Quadratic
Quadratic
0.85
Object Class (Pointwise)
Object Identity (Pointwise)
0.80
Baseline
0.75
7 8
10 11
12
13
14
15
16
17
18
19
20
21
22
23
Layer Index
Figure 3: Quadratic probes excel at decoding the binding signal. Layer-wise accuracy of the IsSameObject probe on DINOv2-Large. The quadratic probe consistently outperforms all other probes from middle layers onward. Results for additional models are shown in Appendix A.3
extract DINOv2-Large 13 activations each layer and train the probes on the ADE20K dataset [56] using cross-entropy loss for all pairwise probes to classify same-object vs. different-object patch pairs (see Figure for /sSameObject visualizations). Figure 3 shows probe accuracy across layers. To test hypotheses about how IsSaneObject represented, compare:

---

Linear < quadratic probes: (Diagonal) quadratic probes significantly outperform lin- ear ones, suggesting that Zsjant is quadratic representation, consistent with the quadratic form used by the self-attention mechanism.
Quadratic (pairwise) object-identity probes (pointwise): Mapping each patch discrete object identity and then comparing them pointwise underperforms direct pairwise comparison embeddings, as the pointwise approach discards information by collapsing continuous representations into discrete classes.
Quadratic object class probes: The model encodes only shared object class but also finer-grained identity cues (e.g., distinguishing two identical cars of the same make and model).
Full > diagonal quadratic probes: ThelsSameObjer information more distributed across dimensions rather than restricted to specific channels.
3.3 Object binding emerges broadly across self-supervised ViTs
We extend our analysis beyond DINOv2 to a broader set pretrained Vision Transformers, including CLIP. MAE, and fully supervised ViTs. To enable direct comparison, we standardize input patch coverage by resizing all inputs so that each model processes the same spatial patch divisions as the DINOv2 family. Under this setup, every probe starts from the same trivial baseline of 72.6% accuracy, which corresponds to always predicting "different", reflecting the class imbalance that most patch pairs do belong to the same object the dataset.
Table reports Isject decoding accuracy across models. DINO models show the strongest binding signal, with large and giant variants exceeding +16 percentage points over baseline. ImageNet- supervised ViT and CLIP also exhibit clear object-binding ability, though to a lesser degree. In contrast, MAE yields poor object-binding performance, suggesting that binding is acquired ability under specific pretraining objectives rather than being a universal property of all vision models.
Table 1: Binding is consistently represented in DINOv2, CLIP and supervised ViT, but less in MAE. Probe accuracy on IsSameObject across pretrained ViTs. is reported percentage points (pp), and the peak layer index is normalized to [0, 1] within each model.
Model
Highest Accuracy (%)
A over Baseline (pp)
Peak Layer (0-1)
DINOv2-Sman
86.7
+14.1
1.00
DINOv2-Baa Base
87.5
+14.9
0.82
DINOv2-Large
90.2
+17.6
0.78
DINOv2-Giant
88.8
+16.2
0.77
Supervised (ViT-L)
84.2
+11.6
0.39
CLIP (ViT-L.)
82.9
+10.3
0.65
MAE (ViT-L)
76.3
+3.7
0.13
Our findings thus produce much wider coverage of ViTs and provide understanding of potential reasons why binding emerges:
DINO. The contrastive teacher-student loss enforces consistency across augmented views containing the same objects. This objective encourages the model to learn object-level features that persist under augmented views 9
Supervised ImageNet training. Although ImageNet labels correspond to the dominant object in each image [57], class-level supervision still provides useful signals for object identity, consistent with the strong performance of object-class probes.
CLIP. By aligning images with text captions, CLIP effectively assigns each object symbolic label (e.g., "the red car"), which can act like pointer that pulls together all patches of that object. This supervision likely encourages patches from the same object to cluster in feature space.

---

IsSameObject (h,, (h,
o(b
h1
ifi hz
big = SbiR. S = E [-1,1]
if2
b
Dek
W probelty!
It.
bik
Probe @G,,
bai
Probe
(b
Az)
0.6(True) IsSameObject Representation Space
0.45(False)
Figure The geometry of IsSameObject representation. Patch embeddings h; and hy are projected onto the IsSameObject subspace by Wi (query) and W2 (key), producing binding vectors, whose similarity is computed by a dot product. The asymmetry between query and key spaces (where W2 differs from W1 by sign- flipped rows) mirrors the asymmetric roles of query and key in self-attention.
4 Extracting the Binding Subspace of VIT Representations
4.1 Decomposing IsSameObject from features
Following the linear feature hypothesis [58], and similar (51), we assume that layer each token embedding decomposes into a "feature" part and "binding" part:
11 = (xr, + (x:),
where (I c) E Rd encodes all attributes of token 2, (texture, shape, etc.) given context (z1...., excluding any information about which other tokens it binds with, and (((a)))) E R encodes the binding information that determines which other tokens belong to the same object (i.e., the IsSameObject relation).
Consider two identical patches x and BB corresponding positions of identical objects A and B in the same image, and let their residual be A AB, It may be tempting to cancel the feature term directly, Indeed, without positional encoding (see proof in Appendix A.4.1 we have (x = (rr.). since for identical tokens the positional encoding the only signal that can differentiate their cross-token interactions.
We approximate (I,(,) (x since the two patches visually identical, appear in nearly the same context, and any positional difference can offloaded into the binding component. This yields:
AAB, h(x - h(*R.) [f(=A.) f(=n.)] - [b( b(xR)) b(xA.) (.))).
AAB, remains roughly consistent across patch pairs with the same index i, then b(xA,) ) and b( form linearly separable clusters, which thus this becomes problematic in natural images, where identical patches are rare.
object identity representations. However,
Instead, we take supervised approach to decoding the binding component. Our quadratic probe serves tool for separating binding from feature information within each token (Fig. ( Conceptu- ally. the quadratic probe can be viewed as projecting activation h into the IsSameOhject subspace, yielding (())rr((2) ()
(z) W1 and (2) (z) ()() (x) W2, and then measures the dot-product similarity between two projected vectors. Given that natural image datasets contain numerous objects where is the primary distinguishing factor, the probe should be optimized to discover a direction that
isolates b. With this strategy we can separate the binding signal from the rest of the representation.
The observation in [11 that binding vectors remain meaningful under linear combination, and become hard to discriminate when they close together, is consistent with this interpretation. In later ablation studies,
trained quadratic probe via ((9) (x) h(9) (x))) TW. W.
7

---

4.2 A Toy Experiment: distinguishing identical objects and similar looking objects
To probe the limits of object binding in ViTs, we construct test image with two identical red cars, a third red car of different brand, and red boat. This setup lets us track IsSameObject representations across layers by evaluating three distinctions: different object-class but similar appearance, same class with subtle differences, and exact duplicates. As expected, these distinctions become progressively harder. We chose natural objects rather than abstract shapes because both the ViT and our probe are trained on real-world images, which allows us to analyze binding in nontrivial setting.
Layer 0
Layer
Layer
Layer 18
Layer 23
1.0
Car
OsjemeObject
Card
Car
Car C
(a)
0.0
(b)
Figure 5: Liyer-wise visualization of Objectt predictions on the test image. We used three red cars and one red boat to make binding deliberately difficult. Early layers attend similar surface features (e.g., the red car or boat hull), mid-layers focus on local objects, and higher layers shift to grouping patches by object class.
To analyze where binding emerges, we plot the IsSameObject scores predicted by trained quadratic probe (Figure 5). We observe that, from early to mid-layers, the model increasingly discerns the local object (the one to which each patch belongs). Surprisingly, from mid-layers to later layers, the model shifts toward class-based grouping, increasingly treating all red cars as the same. Binding emerges in the middle of the network and is then progressively lost towards the top.
The Objeett representation low-dimensional. We use four identical red-car images and split each one into patches using exactly the same grid aligmment. We perform principal component analysis (PCA) on the residuals sets {A BA; ACA, A A where ÁBA = hB; hA, R bB, ba and visualize the first three components (see Figure 6 АAA. ACA: ADA fall into three linearly separable clusters the first three principal component space. The separation of these clusters in a very small number of principal directions demonstrates that IsSameObject lies in low-dimensional subspace: patches from the same object instance map to closely aligned binding vectors, and different instances are linearly separable with large margins.
Mid-layers capture local objects, and higher layers shift towards grouping patches by object class. A surprising observation is the sudden increase in the cross-object /sSemeOhject score (Fig
in the mid-layers of the DINOV2 model for instances the same class (Fig. 55. This is consistent
with prior work showing that VITs represent different types of information at different layers [59]. At the same time, token-position decodability drops in deeper layers (see Appendix [A.4.3). suggesting that the model is deliberately discarding positional information. Our interpretation is that the network initially relies on positional cues to support binding, since location necessary disambiguate tokens that share similar feature content. In later layers, the network removes positional signals once they are no longer useful and repurposes capacity for semantically relevant object structure. Our findings are consistent with experimental evidence from the ventral stream in the brain, showing that while the retinotopic organization of early ventral areas is necessary for perception and binding, global spatial information is instead processed and maintained by the dorsal stream (60-62).
8

---

Car
AAB, h(Xa) h(Om))
ROGA
b(tar) - b(km)
((xss)
Car
PCA([ABA, ACA,ADA)
Car
(a)
(25.08%)
2 (20.10%)
PC 3 (15.42%)
Projection Value
Ger B-Car
Cor
Car Deber
(G)
Figure 6: Identical objects form distinet object-level representations. The first principal compo- nents of the four identical cars, with the three linearly separable clusters denoting Á Aca; ADA. The percentage in parentheses indicates the variance explained by that principal component.
4.3 Attention weights (query-key similarity) correlate with IsSameOhject
In Section
3.2 we showed that IsSameObject is best decoded quadratically. Since self-attention is also a quadratic interaction, binding information the residual stream layer can principle guide how attention is allocated at layer f + 1, allowing the model to selectively route attention within the same object to build a coherent object-level representation.
To test this, we compute the Pearson correlation between attention weights and the IbJect scores (see Fig and Appendix A.5). In mid-level layers, observe a positive but modest correlation, indicating that the model does make use of the IsSameOhject signal when allocating attention. The modest strength of the effect is expected, because attention serves many roles beyond binding.
4.4 Ablation of OsSect hurts downstream performance and works against the minimization of the pretraining loss
Layer
Layer
Backgrourd car1
Background car
Car2
0.150
0.08
0.100
Backgrourd
Bockground
0.06
0.875
0.94
0.000
0.02
0.025
0.000
0.5
IdSemadbject
&Gamwottyect
Figure 7: Attention weights are correlated with IsSameObject. Dot size proportional to the Euclidean distance between patches. Attention weights correlate with IsSameObject in middle layers: (a) Pearson 0. 0.163, (b) Pearson r 0.201.

---

Table 2: Ablations demonstrate the functional role of IsSameOhject, Segmentation mloU, instance mloU, and DINO loss on layer 18 under uninformed (random shuffle) and informed (ground-truth injection) ablations.
Uninformed / ratio
Informed /
0
0.5
0.5
0
Segmentation mIoU (%)
44.14
41.03
39.20
44.14
44.91
43,59
Instance mloU (%)
35.14
31.39
28.19
35.14
36.37
37.02
DINO Loss
0.6182
0.6591
0.6749
0.6182
We conduct ablation studies and evaluate the impact on downstream segmentation performance and pretraining loss. Instead of directly subtracting the IsSameObject representation b(x,) from h(x;), we use less aggressive approaches:
Uninformed Ablation: Randomly shuffle b(x;) across patches in the image at specified ratio.
Informed Ablation (Injection): Using ground-truth instance masks, we inject the true IsSemeObject signal by linearly combining the mean object direction with each patch's 's binding vector by: by = (1 a) b EjEz bobject.J + a bbject,
We evaluate the semantic and instance segmentation performance with retrained segmentation heads on a subset of ADE20K under these variations. We also evaluate the teacher-student self-distillation loss employed in DINO (see Appendix A.6 for details).
Results show that uninformed ablation, which randomly shuffles the binding vector, reduces seg- mentation performance, whereas injecting the mean object direction improves accuracy. Ablating IsSameOhject with random shuffling leads to a noticeable gradual increase the DINO loss, suggest- ing that ablation of IsSameObject works against this pretraining loss.
5 Limitations
We assume the trained probe cleanly splits each patch embedding into "feature" and "binding" components, a simplification that would benefit from further empirical exploration. We do not establish a causal relationship between object binding and downstream task performance, and further analysis is needed to understand how different pretraining objectives induce object binding. Finally, our downstream evaluations focus only on segmentation, leaving open whether these emergent binding signals also benefit other vision tasks such as visual reasoning. More broadly, this paper studies object binding the patch level; more general forms of binding are not explored and are left for future work.
6 Conclusion
In this paper, we show that object binding naturally emerges large, pretrained vision transformers. especially in DINOv2, and this effect is consistent across multiple models. We also show that it an acquired rather than innate ability through comparisons across vision models. Objecc, whether two patches belong to the same object, is reliably decodable and lies in a low-dimensional latent space. Our results emergent object binding arises as a natural solution to self-supervised learning objectives. More broadly, our study bridges what psychologists identify object binding with emergent behavior in ViTs, challenges the belief that ViTs lack such ability.
Looking ahead, we suggest that addressing binding failures in vision models may not require explicit object-centric modules (e.g.. Slot Attention (T]), but could instead achieved by strengthening the intrinsic architectural modifications. Another important direction for future work is to study how bound object representations interact with one another, potentially through low-dimensional "object files" [63] Together, these efforts will deepen our understanding of how symbolic processing of objects can emerge in connectionist models.
mechanisms of ViTs through tailored training objectives or minimal
10

---

References
[1] Brian J Scholl. Objects and attention: The state of the art. Cognition, 80(1-2):1-40 2001.
[2] Anne Treisman. The binding problem. Current opinion in neurobiology, 6(2):171-178, 1996.
[3] Anne Treisman and Hilary Schmidt. Illusory conjunctions in the perception of objects. Cognitive psychology, 1471):107114,, 1982.
[4] Lynn Robertson, Anne Treisman, Stacia Friedman-Hiil and Marcia Grabowecky. The interac- tion of spatial and object pathways: Evidence from balint's syndrome. Journal of Cognitive Neuroscience, ((3):295-317) 1997.
[5] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208, 2020.
[6] Drew Linsley, Dan Shiebler, Sven Eberhardt, and Thomas Serre. Learning what and where to attend. arXiv preprint 1805.088 2018.
[7] Saeed Salehi, Jordan Lei, Ari S Benjamin, Glaus-Roberr Müller, and Konrad P Kording. Modeling attention and binding in the brain through bidirectional recurrent gating. bioRxiv, pages 2024-09, 2024.
[8] Tarun Khajuria, Braian Olmiro Dias, and Jaan Anu. How structured are the representations in transformer-ban vision encoders? an analysis of multi-object representations in vision- language models. arXiv preprint S.iv:2406.09006 2024.
[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supervuud vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[10] Paria Mehrani and John K Tsotsos. Self-attention in vision transformers performs perceptual grouping, not attention. Frontiers Computer Science, 5:1178450, 2023.
[11] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 33: 1525-11538, 2020.
[12] Benjamin Peters and Nikolaus Kriegeskorte. Capturing the objects of vision with neural networks. Nature human behaviour, 1(9):1127-1144, 2021.
[13] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/92. 2023.
[14] Christoph Von der Malsburg. The what and why of binding: the modeler's perspective. Neuron, 24(1):95-104, 1999.
[15] Jerome Feldman. The neural binding problem (s). Cognitive neurodynamics, 7:1-11, 2013.
[16] Semir M Zeki. Functional specialisation in the visual cortex of the rhesus monkey. Nature, 274 (5670):423-428, 1978.
[17] Margaret Livingstone and David Hubel. Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science, 240(4853):740-749 1988.
[18] Mortimer Mishkin, Leslie G Ungerleider, and Kathleen A Macko. Object vision and spatial vision: two cortical pathways. Trends in neurosciences, 6:414 417, 1983.
[19] Daniel Felleman and David C Van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 1991), 1(1):1-47, 1991.
[20] Kalanit Spill-Spector and Kevin Weiner. The functional architecture of the ventral temporal cortex and its role in categorization. Nature Reviews Neuroscience, 15(8)5536-548, 2014.

---

[21] Gaetano Kanizsa, Paolo Legrenzi, and Paolo Bozzi. Organization in vision: Essays on gestalt perception. (No Title), 1979.
[22] Stephen Palmer. Hierarchical structure in perceptual representation. Cognitive psychology, 9 (4):441-474, 1977.
[23] Irving Biederman. peeenition-b--coooooee a theory of human image understanding. Psychological review, 94(2):115, 1987.
[24] Christoph Von Der Malsburg. The correlation theory brain function. In Models of neural networks: Temporal aspects of coding and information processing in biological systems, pages 95-119. Springer, 1994.
[25] H Steven Scholte and Edward HF de Haan. Beyond binding: from modular to natural vision. Trends in Cognitive Sciences, 2025.
[26] Lynn C Robertson. Binding, spatial attention and perceptual awareness. Nature Reviews Neuroscience, 4(2):93 102, 2003.
[27] Adina L Roskies. The binding problem. Neuron, 24(1):7-9, 1999.
[28] Anne Treisman. Focused attention in the perception and retrieval of multidimensional stimuli. Perception & Psychophysics, 22: 22:1-11, 1977.
[29] John H Reynolds and Robert Desimone. The role of neural mechanisms of attention in solving the binding problem. Neuron, 24(1):19-29, 1999.
[30] Pieter R Roelfsema. Solving the binding problem: Assemblies form when neurons enhance their firing rate-they don't 't need oscillate or synchronize. Neuron, 11107):1033-011, 2023.
[31] Ruben van Bergen and Nikolaus Kriegeskorte. Going in circles is the way forward: the role of recurrence in visual inference. Current Opinion Neurobiology. 65:176-193, 2020.
2020.
[32] Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James DiCarlo. Evidence that recurrent circuits critical to the ventral stream's 's execution of core object recognition hehavior. Nature neuroscience, 22(6):974-99) 2019.
[33] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loie Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object repre- sentation learning with iterative variational inference. In International conference machine learning, pages 2424-2433 PMLR, 2019.
[34] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint arXiv:1901.11390 2019. https:/:://a wwrg//bs//9// 11390
[35] Thomas Kipf. Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprint arXiv:21 2021.
[36] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl- Johann Simon- Gabrie,, Tong He, Zheng Zhang. Bernhard Schölkopf. Thomas Brox, al. Bridging the gap to real-world object-centric learning. arXiv preprint arXiv:2209.14866 2022.
[37] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. Advances in Neural Information Processing Systems, 35: 18181-18196, 2022.
[38] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. arXiv preprint arXiv:2303. 10834, 2023.
[39] Whie Jung, Jaehoon Yoo, Sungjin Ahn, and Seunghoon Hong. Learning to compose: Improving object centric learning by injecting compositionality. arXiv preprint 4rXiv:2405.00666, 2024.
12

---

[40] Jinwoo Kim, Janghyuk Choi, Jaehyun Kang, Changyeon Lee, Ho-Jin Choi, and Seon Joo Kim. Leveraging image augmentation for object manipulation: Towards interpretable controllability in object-centric learning. arXiv preprint arXiv:23 01.22300..8990 10.00999, 2023.
[41] Alexander Rubinstein, Ameya Prabhu, Matthias Bethge, and Seong Joon Oh. Are we done with object-centric learning? arXiv preprint arXiv:2504.077092 2025.
[42] Wei Yuen Teh, Chern Hong Lim, Mei Kuan Lim, and Ian KT Tan. Towards discrete object representations in vision transformers with tensor products. In Proceedings of the 2023 7th International Conference Computer Science and Artificial Intelligence, pages 190-194, 2023.
[43] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017.
[44] Jianing Qian, Anastasios Panagopoulos, and Dinesh Jayaraman. Recasting generic pretrained vision transformers as object-centric scene encoders for manipulation policies. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 17544-17552. IEEE, 2024.
[45] Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James Crowley, and Dominique Vaufreydaz. Tokencut: Segmenting objects in images and videos with self- supervised transformer and normalized cut. IEEE transactions on pattern analysis and machine intelligence, 45(12):15790-1580 2023.
[46] Hossein Adeli, Seoyoung Ahn, Nikolaus Kriegeskorte, and Gregory Zelinsky, Affinity-based attention self-supervised transformers predicts dynamics of object grouping in humans. arXiv preprint arXiv:2306.00020 2023.
[47] Maria Mihaela Trusca, Wolf Nuyts, Jonathan Thomm, Robert Honig, Thomas Hofmann, Tinne Tuytelaars, and Marie- Franie-Fanncine Moens. Object-atribute binding in text-to- text-to-image generation: Evaluation and control. arXiv preprint arXiv:240..13766 2024.
[48] Taihang Hu, Linxuan Li, Joost van de Weijer, Hongcheng Gao, Fahad Shahbaz Khan, Jian Yang. Ming-Ming Cheng, Kai Wang, and Yaxing Wang. Token merging for training-free free semantic binding in text-to- sox--to-image synthesis. Advances in Neural Information Processing Systems, 37: 137646-137672, 2024.
[49] Boshi Wang and Huan Sun. Is the reversal curse a binding problem? uncovering limitations of transformers from a basic generalization failure. arXiv preprint arXiv:2504.01928, 2025.
[50] Declan Campbell, Sunayana Rane, Tyler Giallanza, Camillo Nicolò De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven Frankland, Tom Griffiths, Jonathan D Cohen, et al. Under- standing the limits of vision language models through the lens of the binding problem. Advances in Neural Information Processing Systems, 37:1 33436-1 13460, 2024.
[51] Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? arXiv preprint 1.X.i:2310.1719// 2023.
[52] Qin Dai, Benjamin Heinzerling, and Kentaro Inui. Representational analysis of binding in large language models. arXiv e-prints, pages arXiv-2409, 2024.
[53] Oriane Siméoni, Huy Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508. 10104, 2025.
[54] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[55] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv.1610.01644. 2016.
13

---

[56] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[57] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3)21112522 2015.
[58] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint urXiv:2311.03656 2023.
[59] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.058144, 2(3):4 2021.
[60] Michael Arcaro, Stephanie McMains, Benjamin Singer, and Sabine Kastner. Retinotopic organization of human ventral visual cortex. Journal of neuroscience, 29(34):00638-10622 2009.
[61] L. G. Ungerleider and L. Pessoa. What and where pathways. Scholarpedia, 3(11):5342, 2008. doi: 1044449schhoaapccdi.53.... revision #91940.
[62] Vladislav Ayzenberg and Marlene Behrmann. The dorsal visual pathway represents object- centered spatial relations for object recognition. Journal of Neuroscience, 42923)):693344710, 2022.
[63] Daniel Kahneman, Anne Treisman, and Brian Gibbs. The reviewing of object files: Object- specific integration of information. Cognitive psychology, 24(2):175-22) 1992.
[64] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
14

---

A Appendix
A.1 Experimental Setup
Following the DINOv2 standard setup, we use the ADE20K dataset with images resized and cropped to 512 512 pixels, then padded 518 x 518 pixels. We employ patch size of 14 x 14, resulting in total of 1,369 patches per image. All computations are performed using float32 precision on NVIDIA RTX 4090 GPU.
Dataset and Preprocessing.
Training Configuration. We use the Adam optimizer with a learning rate of 0.001 and step learning rate scheduler with step size of 8 epochs and gamma decay factor of 0.2.
All probes are trained for epochs with batch size of 256 or 128.
A.2 Probe Details
(8) LayerNorm (N + MulliHeadAAttention(((())))
(1)
((+1) LayerNorm ( (FN(s()))
(2)
Transformers propagate information across layers according to the above equations 12 where (() is the residual-stream output. We use (() as the patch embedding for probing at layer f.
For pairwise probes, we apply supervision over the pairwise similarity matrix between all patches in an image. In practice, to reduce computational cost, we randomly sample 64 patches per image in each epoch and apply supervision only to the resulting patch pairs. The probes are trained using binary cross-entropy loss.
For pointwise probes, we consider two tasks:
Semantic Segmentation: We use standard cross-entropy loss for pixel-level object class classification.
Following the DETR framework, we employ Hungarian matching for object assignment. The total loss is computed as:
Instance Segmentation:
Lisal Amask Adice
(3)
The hyperparameters follow the DETR configuration: mask_weight (Amask) = 5.0, dice_weight (Adise) = 5.0. Since only 64 patches are sampled per image in each epoch, we use a reduced number of object queries, mum__objec__qurieee 10.
A.2.1 Quadratic probe details
We enforce symmetry in W to reflect the property IsSameObjec((z, u) IsSameObjec((jse) All quadratic probes apply a 1/ /CVC normalization. In the full quadratic probe, lg weight decay encourages a low effective rank. The fixed-rank probe instead explicitly upper-bounds rank(W)) through its fac- torized construction. For analysis, we recover W1 and W2 from the expression a (a: W. W, b) in Equation 3.1 by computing the singular value decomposition (SVD) of the learned symmetric matrix W.
We do not directly parameterize the model using fixed-rank W1 together with sign matrix S that defines We by flipping rows W1, as such a discrete sign matrix is difficult to optimize in a fully differentiable manner.
Algorithm 1 Quadratic Probe (full rank)
Require: z. E RC
1: Learn A CCC b
2: W + { A T/VC
3: s+ Wy+h
4: return (88)
15

---

Algorithm 2 Quadratic Probe (with fixed rank r)
Require: 2, E RC
I: Learn U.V / R bER R
2: W < V + +VTU//V
3: Wy+b
4: return ((8)
A.3 Probe Performance
A.3.1 Baselines
We consider several baselines for the IsSameOhject task.
Majority baseline. As a trivial baseline, we always predict that patch pair belongs to different objects, reflecting the strong class imbalance the ADE20K dataset. This baseline achieves an accuracy of 72.6%.
Distance-based baseline. We also include distance-based baseline that captures dataset statistics. A single scalar threshold is learned the pairwise patch embedding distance: if the distance is below the threshold, the pair is predicted to belong to the same object; otherwise, it is predicted to be different. This baseline achieves accuracy of 77.16%.
Similarity- and attention-based probes. We further compare our IsSameOhject probes against methods motivated by prior work feature similarity and attention-based object selectivity. For all baselines in this category, only a scalar bias term is trained, while the underlying representations are kept fixed. Specifically, consider:
Cosine similarity probe: IsSameObject(z, v) = o
T Ta llull + 6):
Dot-product probe: IsSameObject(r,, y) = 00-x= Ty b);
Self-attention probe: IsSameObject(z, v) o(-x W query Wkey y b), where W Wquery and Wkey are the query and key projection matrices from the self-attention layer at layer + 1.
These results () demonstrate that quadratic probes capture IsSameObject structure beyond what can be explained by feature similarity or attention alone.
IsSameObject Class. Acc.
0.9
0.8
0.7
Quadratic (DINOv2)
Self-attention Dot product
Majority baseline
Casine similarity
Distance baseline
0.6
0
2
4
6
9
10
11
12
13
14 15 16 17
18
19
20
21
22
23
Layer Index
Figure 8: Layer-wise IsSameObject baseline accuracy in DINOv2-Large.
16

---

A.3.2 Cross-Model Comparison
We train and evaluate quadratic probes range of Vision Transformers (VīTs), including the full DINOv2 family (Small, Base, Large, and Giant), as well as CLIP, MAE, and an ImageNet-supprvised ViT-Large model. The evaluated backbones are listed in Table
Table 3: Vision Transformer backbones used for evaluation.
Model family
HuggingFace identifier
DINOv2-Smal
faceboaz/dinnov2smaal
DINOv2-Bil Base
facebook/ddhno2--base
DINOv2-Lan 2arge
faceboa///ino02-largg
DINOv2-Giant
facebbokk/dinor2-iiaant
CLIP ViT-L/14
openal//Lii-vtt--lagg--aaaa
MAE VIT-L
facebook/viit-mae-larg
ViT-L (INIK)
google/nit-larree-pachhh----
All DINOv2 models patch size of 14 x 14 pixels and operate raw inputs of size 518 518. To enable fair comparison across models with different native input resolutions and patch sizes, we standardize evaluation by matching DINOv2's per-patch spatial coverage. Specifically, inputs to all other models are resized such that their patch grids align with those of DINOv2, resulting identical spatial patch divisions and, consequently, the same majority baseline accuracy (72.6%).
For example, CLIP ViT-L/14 natively processes 224 224 images. To match DINOv2's patch grid, the input image is first resized so that its shortest edge is 518 pixels (as in DINOv2), then cropped into multiple 224 x 224 regions, each of which is patchified using 14 patches. This procedure ensures consistent spatial correspondence across models despite differences in architecture and pretraining.
Figure 9: IsSameObject predictions distinguish objects in complex scenes. Additional visualiza- tions of quadratic probe results for DINOv2-Larg at layer 18. The deer image is taken from the DINOv3 paper [53].
17

---

IsSameObject Class. Acc.
0.90
0.85
0.80
0.75
DINOV2
Supervised
CLIP
MAE
Baseling
0.70
5 7
9 11 13 Layer Index
15 17 19 21 23
Figure 10: Layer-wise IsSameOhject classification accuracy across ViT-Large under different pretraining objectives. DINOv2 achieves the highest performance, exceeding 90% accuracy in upper layers, while MAE remains close to the baseline, indicating little to no emergent object binding.
IsSameObject Class, Acc.
0.90
0.85
0.80
0.75
DINOv2-S DINOv2-B
DINOV2
DINOv2-G
Baseline
0.0
0.2
0.4
0.6
0.8
1.0
Normalized Layer Index
Figure 11: Liyer-wise IsSaneObject classification accuracy across the DINOv2 family as function of model size. DINOv2-Large achieves the strongest overall performance, while peak accuracy is comparable across model scales, reaching approximately 88%.
we observe that for ViT-Large models with different pretraining objectives, object-binding performance in DINOv2 begins emerge in the middle layers and peaks in the later layers, whereas ImageNet-supervised ViT and CLIP remain largely consistent across layers.
In Fig. 10
In Fig. observe that within the DINOv2 family, smaller models tend to develop strong object-binding representations at later normalized layers.
A.4 Positional Information
A.4.1 Positional Encoding Distinguishes Identical Objects
single transformer encoder layer can be viewed as comprising two complementary types of computations: token-wise (i.e., position-wise) operations, which act locally on individual tokens and can be executed in parallel to extract features and short-range interactions; and cross-token operations, implemented through scaled-dot-prodde attention, which enables long-range interactions by integrating contextual information across all tokens.
When inspecting the mathematical formulation, we can also show that for identical tokens (i.e., identical patches), positional encoding the only information that can guide the cross-token interac- tions. Here review the operations for transformer encoder with single head from [64] in order.
18

---

For simplicity, we also assume that query-key vectors have the same dimension as the model (i.e., d; = d). We use blue color for the token-wise operations and red color for cross-token operations. For sequence of input tokens t; E IR where l n-channels x patch-height patch width:
pre-processing:
embedding : c; t, WE
adding position embedding encoder layer:
+
Query-Key-Vaile : q: = We: k; = x, W,W, vi x, WV
Q(K)T V
self-attention U softmax
Vd
projection MLP: y = u, Wo
residual connection : Y; x; +
normalization 2, = ILayerNorm(y))
feed-forward network: z; ReLU Wi + bi) W2 + b2
residual connection 2; = 2; y;
where WE Rexd is the embedding layer, We Raxd WK ddd and WV Rdxd are the Query, Key, and Value layers, Wo [Rdd is the linear projection layer, and W1 gdxm W2 Rind are the feed-forward weights.
Proposition A.1: Positional Encoding Breaks Symmetry
For transformer encoder layer, if two input tokens satisfy t, t and p. = Pý- then their self-attention outputs are identical, i.e., = Uj. Therefore, for identical tokens, positional encoding is the only signal that differentiate their cross-token interactions.
Our goal to show that for two identical tokens (i.e., two patches with identical features), the transformer has to use the position tagging as cue for binding. Since most operations are boken-wise (position agnostic), we only need to show the results for the self-attention operation. We will show that if two input tokens are identical with no positional embedding (or with equal positional embedding), then due the symmetry of the attention mechanism, their output vectors after self-attention will be identical. Formally, if t2 i + and p. Py we want to show that Uj.
Assuming ty and p. Dj:
x; We E P: We + P, X;
x; = Xy then:
qi qj k; kj, Vj
Thus the attention score computed by qi and against all keys would be the same:
q kn = q. kn
Vn
So the attention weights (after softmax) for rows and j the same:
where a; softmax K
Oi,n aj.n
Vn
d
And since the values V are the same across all inputs for the same xn, the weighted sum of values will also be identical:
N
N
u;
ain Vn
ajn vn = U,
n=1
==1
19

---

A.4.2 Quantifying the Degree of Distinguishing Identical Objects
Here, use simplified form of our proposed toy example containing two identical cars and one red boat.
Layer
Layer
Layer 12
Layer 18
Layer 23
1.0
Boott
Car
AeSameObject Obbject
Boal
Car A
Car
(a)
Call
0.0
(b)
Figure 12:
Layer-wise visualization of IsSameObject predictions on the test image with two identical red cars and red boat.
We quantify the model's ability to distinguish identical objects by examining the kernel density estimation of IsSameOhject scores between patch pairs from the same object (Car A, Car B, Boat, Background) across layers in DINOv2-Large Figure T2
Ideally, patch pairs from the same object should achieve IsSameObject scores approaching 1.0. In early layers, the distributions cluster around 0.5, indicating the model cannot reliably distinguish same-object from different-object patch pairs. As processing progresses through later layers, these distributions shift toward 1.0. However, some same-object patch pairs continue to score near 0.0 even in deeper layers, representing indistinguishable token pairs.
We also analyzed patch pairs from different objects Figure 14 where we expect IsSameObject scores to approach 0.0. In layers before 12, the distributions correctly cluster near 0.0, showing the model can distinguish different objects. However, as the model learns to group patches within the same object (as shown in the previous analysis), it simultaneously loses its ability to tell the two identical cars apart. This trade-off visible the Carl-Car2 distribution, which gradually shifts upward through the layers and develops strong peak at 1.0 by the final layer.
A.4.3 Position Information Decay
We hypothesize that the transition from middle layers' capacity distinguish identical objects to later layers' failure stems from the gradual diffusion of precise positional information into more global, comantically-focssed representations. To test this hypothesis, trained linear probes decode the (x.y) coordinates of each patch from the model's 's internal representations (Figure 15). We observe a marked increase in probe RMSE layer 21, which supports our hypothesis.
A.5 Attention weights (query-key similarity) vs. IsSameObject
We investigate the relationship between attention mechanisms and object identity representations by comparing attention weights with IsSameObject scores. Attention weights are computed as:
QuKK Vdl.
Attention, = softmax
(4)
20

---

Layer
Layer
Layer
35
Background
25
17.5
20
15.0
25
Boat
Backgroand
Background
20
15
Car
10.0
Car
Car
Car
10
7.5
10
5.0
2.5
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
3.00
0.00
0.25
0.50
0.75
1,00
IsSameOllject
BSamecbject
IntameOlject
Layer 9
Layer 12
Layer
35
3.5
Beckground
Bachground
Car
3.0
3.0
car 1
Car
2.5
Car 2
Backgroundd
2.0
2.0
1.5
Car 2
Boat
1.5
0.5
0.5
0.0
0.0
0.00
0.50
0.25 0.50
KSSamotjjctt
#Samebbbjact
IntamaDbyect
Layer 18
Layer 21
Layer 23
Background
Backgroand
3.5
Background
2.5
Car
Car
Bost
2.0
Boat
2.5
Bowt
2.0
1.5
3.5
3.0
0.5
0.0
0.00
0.30 0.75
1.00
0.00
0.25
9.50
3,00
0.00
0.25
0.30
2.00
ISsaeebiect
8SameCbiect
IsSameOboect
Figure 13: Binding strengthens with depth for same-object patch pairs. Kernel density estimation of IsSameObject scores for patch pairs within the same object across different layers.
where Qi and K2 represent the query and key vectors for patches and j. respectively, and dj is the key dimension.
We then compute the Pearson correlation between attention weights at layer + and IsJamtOhject scores derived from the quadratic probe at layer t:
p = corr (Attentttiion (f+1) ))SaamObjeec(!))
(5)
Using the simplified two-car scenario from Figure 12 we examine how attention weights at layer (+ correlate with IsSaneObject scores at layer I. In early layers, observe minimal correlation between these two measures, which is likely because IsSameObject representation has not yet fully developed.
In deeper layers, certain patch pairs receive high attention weights despite having low IsSameOhject scores (indicating the model believes they belong to different objects). This phenomenon may be explained by background patches being repurposed for internal computational processes, as identified in prior work on DINO register tokens. Future research could further investigate how these specialized background patches contribute to object representation and their role in maintaining distinct "object files".
The Pearson correlations in Fig. 7 are statistically significant (p < 0,001 under permutation test).
A.6 Implementation of Ablation Studies.
We conduct ablation experiments at layer 18 of DINOv2-Large, where IsSSaeObjject representation achieves the best decodability. We apply both uninformed and informed ablation methods as described in Section 4.4
21

---

Layer
Layer
Layer
Background Car
Background Car
Background Car
-ara
Car
Boat
Boat
Background--oe Soat
Backggroudd-Bon fina
Background Boat
Dersity
Donsity
85
6
0
0.00
0.25
0.50
0.75 1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
edamentjjcct
aGamacijject
WCamiiitgert
Layer
Layer 12
Layer 15
Background-Cer
Background-Car
Beckground-Cor Car
Car 1 Car
Car 1 Car
Car C-r 2
Car Boat
Beot
Car Bear
Background Soot
Backigrund--Bon
Backirround Boot
Deriity
Dersit
Derstty
6
0
0
BSSosodejoct
BSameciijsct
I5Samobjjcrt
Layer 18
Layer 21
Layer 23
17.5
Background
Background
15.0
Boat
Beat
Boat
BackgrounddBBoo Boat
Background Roat
Backkround-Bee Roat
10.0
Qemiiy
Densiyy
7.5
2
5.0
2.5
1
0.0
0.00
0.25
0.30
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0,50
0.75
1.00
aSamechjjct
IisamiOteett
Figure 14: Identical objects collapse in representation at deeper layers. Kernel density estimation of IsSameObject scores for patch pairs from different objects across layers.
RMS
40
Baseino (19:3)
0.9
Suczales
Patch (x:9) RMSE (pixelss
35
Success Rafe
30
0.7
25
0.6
5
10
15
20
25
Layer Index
Figure 15: Positional information decays in later layers. Layer-wise decoding performance for
patch (x.y) coordinates, compared with the success rate of distinguishing patches from Car A and Car B.
Segmentation Evaluation. For both semantic and instance segmentation tasks, we retrain linear segmentation heads with ablated representations. The uninformed ablation randomly permutes binding vectors across patches, while the informed ablation injects object-averaged binding vectors using ground-truth masks. These linear heads use identical configurations to the pointwise probes described in Section [. which are effectively pointwise probes applied the final transformer layer.
DINO Loss Evaluation. assess the impact on the pretraining objective, we evaluate DINO loss using the pretrained model as both student and teacher networks. For computational simplicity,
22

---

Layer
Layer
0.175
Car1
Background
Car1
Background
0.10
0.10
Background-Car
BackgruundCrr
0.100
0.125
Car 1
fackground
Boat
0.08
Background@ort
Car Carl-Car2
Car: Boat
0.06
8.000
0.04
0.025
002
0.000
0.00
IsSameiiject
#Samecitject
0.6
0.0
(b)
Figure 16: Attention vs. Osjeec in early layers. Scatter plots comparing attention weights IsJame scores for patch pairs; correlation is still weak, indicating that binding has yet developed sufficiently to influence attention.
Layer 18
Layer 23
Background
0.7
0.4
Car
Car 2
0.3
Car 1 Car
Background-Car
Car
Background
Attention
Background Boat
0.4
Background-Car
0.3
Car 1- Car
Boat
Background isat
0.1
0.0
IsSameObijet
IsSameObject
(a)
(b)
Figure 17: Attention vs. Object in later layers. Attention is sometimes allocated to low- IsSameObject background tokens, suggesting these tokens might be repurposed for internal computa- tion.
we exclude the iBOT and KoLeo loss components from this analysis. Note that informed ablation cannot be evaluated under DINO loss, as the use of local crops alters the patch divisions, making object-averaged binding vectors undefined for the cropped regions.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction explicitly state the core contributions (i) demon- strating that object binding emerges naturally in large pretrained ViT', (ii) identifying its low-dimensional, and (iii) showing that this binding signal benefits both downstream tasks and the pretraining objective.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Section 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
24

---

Answer: [No]
Justification: Key conditions Section 4.1 for the linear decomposition hypothesis are described informally without rigorous specification.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper gives a complete description of the experimental setup in Sec- tions 3 and implementation details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We have released our code and scripts a public GitHub repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/publiccguu.eee CoodeSubeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper gives a complete description of the experimental setup in Sec- tions 3 4 and implementation details in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report Pearson values alongside their p- p-values (p <0.001) to convey statistical significance in the Appendix A.5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed compute specifications are provided the Appendix A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwuurr....ppaaa..aaccaaa
Answer: [Yes]
Justification: The study uses only publicly available datasets (ADE20K, ImageNet), involves no human subjects or sensitive personal data.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work focuses on internal representational analysis of pretrained ViTs and does not have direct societal impacts discuss.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not involve releasing new models or high-risk datasets, so no additional safeguards are required.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credit the DINO framework to Caron al. (MIT License), ADE20K dataset under its CC-BY-SA License, and the ViT models via Hugging Face (Apache 2.0).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All newly introduced assets documented in the Appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The study uses only pretrained models and public image datasets, with no human participants or crowdsourced annotation tasks involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper do not involve human subjects participant data.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Any use of LLMs was limited writing, editing, and formatting the manuscript; they did not contribute to the core methodology scientific results.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
30


Reviews:
[{"review_id": "THCLp7TXuu", "reviewer": "Reviewer_43Cy", "summary": "The authors study whether object binding - the ability of representations to bind low-level visual features into object level representations - appears in large self-supervised vision transformers. The authors propose a simpler definition of object binding, object features being grouped together, and propose a proxy task isSameObject to verify this definition in DINOv2 representations. In order to do so, the authors train various decoder probes on pairwise token features to classify whether two tokens (patches for vision transformers) encode features about the same object. The authors' experiments show that they are able to reliably perform this classifaction, especially with non-linear probes - which is an indicator that token level features encode object binding. They also show that early layers in ViTs bind object level information more while later layers bind class level information more. The authors also show that the isSameObject feature representations are low dimensional. Lastly, they show that isSameObject scores have a positive correlation with the attention weights, and hence are used in informing the model decisions.", "questions": "Please see weaknesses above", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n1. Solving object binding has been a key concern in representation learning, and learning object-part heirarchies is a core problem in computer vision. Past research has focussed on building architectural inductive biases focussed on learning these (e.g. Slot Attention mentioned by authors, Capsule Networks). But understanding whether training objective based inductive biases enable the learning of object binding is a very important question - as it will inform which direction to focus future research on learning such representations. Past research (e.g. Grounding inductive biases in natural images:invariance stems from variations in data, Bouchacourt et al; Do vision transformers see like convolutional neural networks?, Raghu et al) has already shown that learning based inductive biases often end up learning similar feature properties as architecture based biases. This paper does a good job at establishing this for the isSameObject property, which is relevant for assessing object binding in learned representations (although not a 1:1 proxy, see weaknesses below).\n\n\n# Weaknesses\n\n1. I think the major limitation of the paper is that the all experiments are conducted on vision transformers with one particular self-supervised training objective (i.e. DINOv2) whereas the scope of the paper based on the title *Large Pretrained Vision Transformers* is much larger. In this sense just looking at one pre-training loss objective is too limited of a scope to draw conclusions about large pretrained ViTs in general. Furthermore, it seems like all experiments were done on the ViT-Base model, and there is no comparison on effect of model size on object binding properties either. The paper would be much more robust if these additional factors were studied.   \n2. Why can object binding only occur through inter-token interactions? I am referring to this claim by the authors:  \n\n> [L132-135] Object binding can only occur through token-token interactions, so ultimately, activation patterns useful for binding must inform self-attention, which takes a quadratic, pairwise form. Therefore, if ViTs indeed perform any form of object binding, we expect to observe a pairwise token-level representation that indicates whether two patches belong to the same object, which we term IsSameObject\n\nEach token's features in successive layers goes through a multi-head attention where intra-token interactions are also happening (e.g. Key Value dot product, Scaling value with attention score). Saying only token-token interactions can lead to object binding is not entirely correct for ViT architectures. \n\n3. I do not agree with the central claim \"Since object binding is precisely the ability to group an object’s features together, decoding Is-SameObject reliably from ViT patch embeddings would provide direct evidence of object binding\". This is incorrect. According to Greff et al \"(Binding is) the capacity to acquire a compositional understanding of the world in terms of symbol-like entities (like objects)\". So, binding is the ability to both group an object's features together (concept level features), and to reliably decode them while disentangling other objects' features to perform downstream tasks (compositionality). This corresponds to the third hypotheses about how IsSameObject may be encoded in model activations - \"The signal is stored in a few specialized dimensions versus distributed across many dimensions\". In the isSameObject task, the distributed quadratic probe consistently outperforms the diagonal quadratic probe, and heavily outperforms the linear probe. To me this is an indicator that the second part of the binding condition is not being satisfied in the DINOv2 representations.", "was_revised": true, "final_justification": "The additional pre-training objectives studied in the rebuttal improve the strength of the paper's central claim, so I have increased my score from Borderline to Accept. But I think the authors should clearly make a note about the different aspects of the binding problem in the paper itself, and mention how compositionality was not specifically explored in this paper and is left for future work. I also went through the notes and rebuttals from the other reviews, and after reading them my original comments on the strengths of the paper's key problem and solution remain, while the additional results reported alleviate the issues around lack of experimental coverage. The problem, approach and results are enough to justify an acceptance.", "rebuttal": "# Response to reviewer 43Cy(1/1)\n\nDear Reviewer 43Cy,\n\nWe thank the reviewer for recognizing our work's contribution in showing that training objective–based inductive biases can drive object binding independent of architectural design. We agree with their feedback and are delighted that we were able to run the suggested experiments (across different objectives and models) and that they substantially improved our paper.\n\nBelow, we address their comments point by point.\n\n---\n\n### **[W1, Q1] Limited verification across pretraining objectives**\nWe have now extended our analysis beyond DINOv2 to include CLIP, MAE, and supervised-only trained ViTs. (Some limited experiments with CLIP and DINO model sizes were already included in the supplementary: Lines 471-478, Appendix Figure 7). To ensure fair comparisons, we now standardized the effective patch coverage by resizing inputs so that all models process the same spatial patch divisions. This ensures a shared baseline (72.6%) across all models (corresponding to the accuracy of always predicting \"different,\" which accounts for the class imbalance in the dataset). \n\nHere are our results: \n\n**Table: Probe accuracy on the *IsSameObject* task across ViTs**\n| **Model**                  | **Highest Probe Accuracy (%)** | **Improvement Over Baseline** | **Layer at Highest Accuracy (Normalized. 0=first layer, 1= last layer)** |\n|:-------------------------- |:-----------------------:|:----------------------------:|:--------------------------:|\n| **DINOv2‑Small**           | 86.7 %                  | + 14.1 pp                    | 1.00                       |\n| **DINOv2‑Base**            | 87.5 %                  | + 14.9 pp                    | 0.82                       |\n| **DINOv2‑Large**           | 90.2 %                  | + 17.6 pp                    | 0.78                       |\n| **DINOv2‑Giant**           | 88.8 %                  | + 16.2 pp                    | 0.77                       |\n| **CLIP (ViT‑Large)**       | 84.2 %                  | + 11.6 pp                    | 0.39                       |\n| **MAE (ViT‑Large)**        | 82.9 %                  | + 10.3 pp                    | 0.65                       |\n| **Supervised (ViT‑Large)** | 76.3 %                  | + 3.7 pp                     | 0.13                       |\nThis now allows us to produce much more interesting insights:\n- **Pretraining objectives**: CLIP, MAE, and DINO all demonstrate object binding capabilities, supporting our broader claims about *Large Pretrained Vision Transformers* beyond DINOv2 alone. We also show that supervised ImageNet classification yields poor object‑binding performance, indicating that binding is an acquired ability under specific pretraining objectives rather than being a universal property of all vision models.\n- **Model size**: Larger models maintain object binding effectiveness, but the optimal layer for object binding shifts earlier in the network (lower normalized layer index), with performance plateauing in later layers (see supplementary Lines 471-478, Appendix Figure 7). This suggests that increasing model scale makes object binding emerge earlier, and deeper layers become redundant for binding.\n\nWe offer explanations of why these training regimes work:\n- **DINO**: As noted in Lines 186–189, the DINO loss drives the model toward object-level features by enforcing consistency between augmented views, since each view contains the same objects and the model must align their representations; this objective has been shown to encourage class and object specific features (Caron et al. ICCV 2021), as the reviewer points out. \n- **CLIP**: By aligning images with text captions, CLIP effectively assigns each object a symbolic label (e.g., “the cat”), which can act like a pointer that pulls together all patches of that object. This supervision likely encourages patches from the same object to cluster in feature space. Future work could track object‑binding strength across CLIP training checkpoints to validate this hypothesis.\n- **MAE**: The masked autoencoder objective requires the model to reconstruct a missing patch from its surroundings. When the masked patch sits between multiple objects, correctly predicting its content forces the model to infer which object it belongs to, thus promoting the grouping of patches from the same object.\n\n**Our findings thus produce a much wider coverage of ViTs and we provide an understanding of potential reasons for why binding emerges.**\n\n---\n\n### **[W2] Object binding through intra-token interactions**\nAs the reviewer correctly points out, multi‐head attention involves both intra‑token operations (the linear projections that mix channels within each token) and inter‑token operations (the dot‑product and weighted summation that relate different tokens). In our work, we define object binding strictly at the level of patch tokens, i.e. whether separate tokens group together to form a single object. Under this definition, no binding can occur within a single token alone, so intra‑token projections cannot contribute to the effect we study. \n\nAn alternative notion–binding different attributes within one token, such as “red” and “square”, would require us to decompose a patch’s embedding into separate feature channels. We appreciate that investigating this within‑token compositionality is an important direction, but it falls outside the scope of our current study because it is likely to use rather different mechanisms (for example, dealing with a relatively small set of tokens with a large number of intra-token feature channels).\n\n---\n\n### **[W3] Lack of evidence of compositionality**\nWe appreciate the reviewer's comment on the importance of binding in functionally approaching human level generalization in neural networks. The reviewer references a seminal work (Greff et al. 2020), which defines binding as the ability to dynamically and flexibly bind information that is distributed throughout the network, they also investigate its *functional aspects*—segregation, representation, and composition—without implying that all of them are required by the definition itself. We agree these are useful perspectives, but note that the literature *does not universally require compositionality as part of the core definition of binding*.\n\nThe reviewer points out, while we clearly demonstrate (1) ViTs’ ability to group an object’s patches (i.e., segregation or parsing), we do not show (2) that those grouped features are disentangled into a truly compositional representation usable by downstream tasks. We appreciate this clearer framing and believe our results offer preliminary evidence of such compositional structure, yet we agree that a deeper exploration of compositionality lies ahead as future work. \n\nTo clarify,\n- The distributed quadratic probe’s superior performance demonstrates that the *IsSameObject* signal spans many channels rather than being confined to a handful of specialized ones. While the reviewer interprets this distribution as evidence against compositionality, a distributed code can still be fully compositional and accessible to downstream tasks. Consider a signal that is intrinsically one‑dimensional but oriented along a rotated axis: when expressed in the original coordinate system, its nonzero components appear across multiple dimensions, yet its underlying structure remains compact.\n- We distinguish between distributed representation and low intrinsic dimensionality of the *IsSameObject* space:\n    - **Distributed representation** means the model uses several activation dimensions (channels) to encode whether two patches belong to the same object.\n    - **Low intrinsic dimensionality** means that, despite this distribution, all those activations lie on a simple, low‑dimensional manifold or subspace—so the binding information itself is inherently compact. We demonstrate this low‑dimensional clustering in Lines 231–237.\n- We acknowledge that a deeper investigation of compositionality in downstream tasks remains an important direction for future work. The empirical evidence for the compositionality of the feature representations is accumulating (Amir et al., Deep ViT Features as Dense Visual Descriptors, ECCVW 2022; Darcet et al., Vision Transformers Need Registers, 2023).\n\nWe believe that our responses have substantially strengthened the contributions of our work and are thankful for the reviewer's feedback.\n\n**Best regards,**  \n*Authors*"}, {"review_id": "1sug92evNW", "reviewer": "Reviewer_x2fh", "summary": "This papers tackles the question of whether object binding emerges within large pre-trained vision transformers (focusing on DINO). The binding problem refers to how the brain is able to group disparate visual features with in a parallel distributed system to a single coherent whole to represent an object. This paper extends this problem to computer vision by trying to understand whether vision transformers exhibit any object binding mechanisms. The paper proposes training probes that can detect whether two patches belong to the same object and how that such probes can be successfully trained on patch embeddings. Furthermore, the paper presents results showing that ablating the ability of the model to detect isSameObject can be detrimental to performance.", "questions": "- Do the results apply to ViTs in general? If so, can you provide evidence that similar binding strategies hold in networks trained with other supervision?\n- Can you please comment on the choice of only studying binding within each layer while it was intended to measure how the brain binds information across different brain areas? \n- The paper states that there is a current view held by cognitive scientists that VITs lack object binding. Could you please provide evidence that this is a commonly held view in the cognitive science community? \n- Could you comment on the concerns raised about the experimental results and the conclusions in Sec 4.3 and 4.4.  \n- What is the baseline referred to in Fig 3? Is it just performance of the linear probe at the first layer?\n- Could you please explain how the binding vectors are extracted from the token embeddings? It seems clear that difference of embeddings (h_b - h_a) equates to difference of binding vectors (b_b - b_a). However, it's unclear how to decompose h_a into f_a and b_a which seem necessary to conduct the experiments in Sec 3.3.", "limitations": "While there is a limitations section that highlights two limitations of the work, it does not seem adequate to me. As noted in the weaknesses, the object binding problem is complex and mapping it to ViTs has to come with many assumptions that limit the claims that can be made about the work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The introduction and related work are very well written and very nicely set up the problem and contextualize it. \n- The binding problem is an interesting problem within connectionist systems, and I really appreciated the effort put to explain its significance as well as what it entails. As well as the delineation between efforts that try to solve it architecturally (eg, Slot Attention) as opposed to the question posed by this paper of trying to find its mechanisms within network. \n- I appreciate the care taken experimentally to differentiate between different hypothesis in Sec 3.1. \n\n----\n**Weaknesses**: I've tried to order the weaknesses in terms of importance/significance. \n- The title and framing of the paper does not match the experimental setup. The paper sets itself as measuring whether object binding emerges within \"pretrained vision transformers.\" This can be seen in both the title as well as the framing in the introduction (L50-71). However, the paper primarily focuses on DINO (with one experiment also reporting results on CLIP). This is a major confounder as DINO's loss results in class and object specific features; this was indeed one of the main claims in Caron et al (ICCV 2021). Without showing results that indicate that transformers learn this with different training regiments. \n- Object binding is concerned with how to relate perceptual representations that \"depend on distributed neural codes for relaying the parts and properties of objects.\" [Treisman, 1966]. On the other hand, this paper focuses on binding within each layer with little discussion of whether there are any binding mechanisms across layers (or a way to preserve the isSameObject across layers). This is important as some work has shown that ViTs represent different types of information at different layers (eg Amir et al. ECCVW 2022) which can be seen qualitatively in Figure 4b. \n- The abstract states that the findings \"challenge the view that ViTs lack object binding\" this is followed by similar claims in L51-53 with references 11, 12, 7. However, the references don't seem to provide evidence for this. Ref11 is a paper that proposes a clustering mechanism to improve performance on human pose estimation. Ref12 on the other hand doesn't make any mention of ViTs or self-attention, and the critique for computer vision approaches seems to focus on feedforward convolutional networks. It's unclear from the evidence provided that there's a strong view within the cognitive science community that ViTs lack this ability. \n- Some of the claims in Section 4 don't seem to be supported by the results:\n    - In Sec 4.3, the paper claims that attention weight correlate with isSameObject. The correlation plot shown in Figure 6 shows a strong spread even if it's exhibits a positive trend (as indicated by the r values of 0.163/0.201). Furthermore, isSameObject is a proxy estimate for whether two patches belong to the actual object. It would make sense to show that the attention is stronger between patches of the same object as opposed to patches belonging to different object of the same class (or different objects)? \n    - In Sec 4.4, the paper states that ablating isSameObject hurts downstream performance. First, it's unclear how this is ablated as Section 3.2 shows how the difference of features correlates to the difference of the binding parts. However, it doesn't explain how you can obtain the binding parts from the features. Second, the trends shown in Table 1 exhibit performance changes that (1) don't show monotonic improvement/regression as shown by segmentation accuracy for informed ablation and (2) seem like fairly small differences in performance without being supported by some baseline to calibrate expected model changes. \n- There are several experimental details that aren't clear from the paper. See questions. \n\n----\n**Minor concerns/comments**\n- There seems to be some latex/formatting issues in L141 (equations 1/2) and L157 (4.5-6). \n- I found the latter parts for section 3 harder to follow as they explain different experiments before showing the results sequentially in section 4. I would suggest that you consider interleaving the experimental results with the method which could improve clarity and readability.", "was_revised": true, "final_justification": "As noted in my initial review, the submission was already strong. There were some limitations regarding framing and experimental validation that were addressed in the rebuttal. I believe the resulting paper will be much stronger and will raise my rating to Accept.", "rebuttal": "# Response to reviewer x2fh (1/1)\n\nDear Reviewer x2fh,\n\nWe thank the reviewer for your careful reading, constructive feedback, and positive comments on our contextualization and emphasis of the research problem’s significance. We are delighted that we were able to run the requested experiments (across different objectives) and that they substantially improved our paper.\n\nBelow, we address your comments point by point.\n\n---\n\n### **[W1, Q1] Limited verification across pretraining objectives**\nWe have now extended our analysis beyond DINOv2 to include CLIP, MAE, and supervised-only trained ViTs (Some limited experiments with CLIP and DINO model sizes were already included in the supplementary: Lines 471-478, Appendix Figure 7). To ensure fair comparisons, we now standardized the effective patch coverage by resizing inputs so that all models process the same spatial patch divisions. This ensures a shared baseline (72.6%) across all models (corresponding to the accuracy of always predicting \"different,\" which accounts for the class imbalance in the dataset). \n\nHere are our results: \n\n**Table: Probe accuracy on the *IsSameObject* task across ViTs**\n| **Model**                  | **Highest Probe Accuracy (%)** | **Improvement Over Baseline** | **Layer at Highest Accuracy (Normalized. 0=first layer, 1= last layer)** |\n|:-------------------------- |:-----------------------:|:----------------------------:|:--------------------------:|\n| **DINOv2‑Small**           | 86.7 %                  | + 14.1 pp                    | 1.00                       |\n| **DINOv2‑Base**            | 87.5 %                  | + 14.9 pp                    | 0.82                       |\n| **DINOv2‑Large**           | 90.2 %                  | + 17.6 pp                    | 0.78                       |\n| **DINOv2‑Giant**           | 88.8 %                  | + 16.2 pp                    | 0.77                       |\n| **CLIP (ViT‑Large)**       | 84.2 %                  | + 11.6 pp                    | 0.39                       |\n| **MAE (ViT‑Large)**        | 82.9 %                  | + 10.3 pp                    | 0.65                       |\n| **Supervised (ViT‑Large)** | 76.3 %                  | + 3.7 pp                     | 0.13                       |\n\nThis now allows us to produce much more interesting insights: CLIP, MAE, and DINO all demonstrate object binding capabilities, supporting our broader claims about *Large Pretrained Vision Transformers* beyond DINOv2 alone. We also show that supervised ImageNet classification yields poor object‑binding performance, indicating that binding is an acquired ability under specific pretraining objectives rather than being a universal property of all vision models.\n\nWe offer explanations of why these training regimes work:\n- **DINO**: As noted in Lines 186–189, the DINO loss drives the model toward object-level features by enforcing consistency between augmented views, since each view contains the same objects and the model must align their representations; this objective has been shown to encourage class and object specific features (Caron et al. ICCV 2021), as the reviewer points out. \n- **CLIP**: By aligning images with text captions, CLIP effectively assigns each object a symbolic label (e.g., “the cat”), which can act like a pointer that pulls together all patches of that object. This supervision likely encourages patches from the same object to cluster in feature space. Future work could track object‑binding strength across CLIP training checkpoints to validate this hypothesis.\n- **MAE**: The masked autoencoder objective requires the model to reconstruct a missing patch from its surroundings. When the masked patch sits between multiple objects, correctly predicting its content forces the model to infer which object it belongs to, thus promoting the grouping of patches from the same object.\n\n**Our findings thus produce a much wider coverage of ViTs and we provide an understanding of potential reasons for why binding emerges.**\n\n---\n### **[W2, Q2] Cross-layer binding**\n\nWe thank the reviewer for highlighting cross‑layer interactions as a novel and necessary perspective on studying binding in ViTs; since brains bind across layers, we ask whether ViTs do too.\n\nCross-layer binding may actively happen under these conditions:\n- **Layer-specific information processing**: ViTs represent different types of information at different layers. Although the self-attention implements a \"full range\" interaction (all tokens to all tokens interaction), the model learns to limit the \"interaction\" between the correct tokens and integrate features of the object they belong to as we go deeper.\n- **Information retrieval from earlier layers**: ViTs may access information from earlier layers when current layers lack necessary binding cues. As shown in Appendix Figure 11 and in Amir et al. ECCVW 2022, positional information is gradually removed as the network deepens, and other low-level features may also be discarded. This loss may lead the model to retrieve binding cues from earlier layers instead of keeping them redundantly; evaluating this idea needs further work.\n\nCross-layer interactions in ViTs can occur via the residual skip connections, so we also train quadratic probes between non-adjacent layers. Specifically, we compute $\\phi(x,y) = x^TW_1^T W_2 y$, where $W_{1}$ and $W_{2}$ are the learned projection matrices from layer 15 and layer 18, respectively. The probe accuracies are:\n\n- Layer 15: 89.0%\n- Layer 18: 90.1%\n- Cross (15→18): 83.3%\n\nThe cross-layer probe shows decent cross-layer object binding, and a certain degree of object binding across the system. \n\n---\n### **[W3, Q3] Consensus on ViT object‑binding limitations**\n\nWe agree that our claim was not well articulated and could lead to misunderstanding. We should clarify that the cognitive- and neuro-scientists believe that binding requires a set of interacting mechanisms such as memory and recurrent processing trained on complex behavior (please see Roelfsema 2023, Solving the binding problem; Peters et al. 2021, Capturing the objects of vision with neural networks; Salehi et al. 2024, Modeling Attention and Binding in the Brain through Bidirectional Recurrent Gating). Therefore, our claim mentioned ViTs as part of the general set of artificial neural networks that do not meet these hypothetical requirements. Here we should emphasize that this hypothesis is now questioned by some cognitive neuroscientists (Scholte et al. 2025, Beyond binding: from modular to natural vision).\n\nWe would like to point out that this perspective is also shared by computer scientists (Greff et al. 2020): “In the case of neural networks, the binding problem is not just a gap in understanding but rather characterizes a limitation of existing neural networks.”\n\nFinally, We would remove ref11 from the references and include the work of Roelfsema as it is more consistent with our claim.\n\n**Our work on ViTs ties into a large and ongoing discussion about the capabilities of brains vs ANNs in object-based perception. We believe with the additional references we can make our relevance much clearer.**\n\n---\n\n### **[W4.1, Q4] Weak Attention–Binding Correlation**\nWe agree that the correlation in Figure 6 (r = 0.163/0.201) is modest, but it is consistently positive. However, such modest effects should be expected, after all attention does a lot more than binding. It is meaningful to plot attention weights against the estimated IsSameObject, as it reveals the model’s implicit belief about whether two patches belong to the same object.\n\nTo address the reviewer’s suggestion, we’ve extended our analysis to compare attention weights against the true patch‑pair labels, and find that attention is significantly higher for same‑object pairs than for any category of different‑object pairs.\n\n**Table 1**: Mean self‑attention weights for different patch‑pair categories\n| Pair Type                              | Mean Attention Weight |\n|----------------------------------------|-----------------------:|\n| Same‑object pairs                      |                  0.033 |\n| Different‑object pairs                 |                  0.014 |\n| Different cars (same class)           |                  0.013 |\n| Different classes (car – boat)         |                  0.018 |\n\n\n### **[W4.2, Q4] Ambiguous ablation method and marginal effects**\n\nWe’d like to clarify that “binding” part $b^l(x)$ is computed as $b^{(l)}(k) = h^{(l)}(x)^⊤W$, where W is the trained weight matrix of the quadratic probe. Our reasoning for this formulation is given in Lines 178–180.\nThe ablation yields only modest performance gains (or losses), reflecting a relatively small effect size; we will explicitly note this as a limitation in the camera‑ready version’s limitations section.\n\n---\n\n### **[Q5]  Baseline definition in Figure 3**\nThe baseline corresponds to the accuracy of always predicting \"different\", which accounts for the class imbalance in the dataset.\n\n---\n\n### **[Q6] Extraction of binding vectors from token embeddings**\nWe obtain the binding component $b_a$ by projecting $h_a$ onto the quadratic probe’s weight matrix $W$:\n$b_a = h_a^\\top W,\\quad f_a = h_a - b_a.$ Although this projection is an approximation, since the “true” binding vector was computed by subtracting embeddings of identical objects (Lines 231–237; Fig. 4), it performs well in practice, as the probe was trained on large, natural-image datasets rich in similar-looking objects that can only be distinguished by binding signals (Lines 178-180).\n\n---\n### **Minor concerns**\nWe thank the reviewer for their careful review. We will correct the LaTeX/formatting issues and restructure the section as suggested to improve clarity and flow.\n\n---\n\nWe believe that our responses have substantially strengthened the contributions of our work and are thankful for the reviewer's feedback.\n\n**Best regards,**  \n*Authors*"}, {"review_id": "0J1mC31N3Y", "reviewer": "Reviewer_gvWE", "summary": "his paper examines whether object binding emerges naturally in pre-trained ViTs, introducing the concept IsSameObject to indicate whether two patches belong to the same object. Using a similarity probe, the authors show that this signal is decodable, lies in a low-dimensional space, correlates with attention, and affects downstream performance. The study suggests that object binding can arise in ViTs without explicit modules.", "questions": "1. Generalization to Other ViTs: How does the IsSameObject signal behave in ViTs trained under different paradigms (e.g., MAE, CLIP)?\n\n2. Low-Dimensional Subspace: Why is it meaningful that IsSameObject lies in a low-dimensional subspace? Does this suggest compression, generalization, or modularity—and how could it be exploited in practice?\n\n3. Visualization and Interpretability: In Figure 4, the visualization of layer 23 does not support the conclusion that higher layers shift to grouping patches by object class. For example, feature from the boat also largely attends to the cars. Can the author provide more convincing visulizations?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nNovel Hypothesis and Probe Design: The introduction of IsSameObject as a testable signal is both conceptually interesting and grounded in the architecture of ViTs.\n\nEmpirical Rigor: Strong quantitative results showing high probe accuracy, supported by ablation studies and correlation with attention weights.\n\nTheoretical Insight: Bridges cognitive science (object binding) with deep learning, providing an interpretable lens through which to view ViT behavior.\n\nPractical Implications: Challenges the necessity of explicit binding modules and suggests more implicit, architecture-aligned approaches.\n\nWeaknesses:\n\nLimited Scope of Models: While DINOv2 is emphasized, it’s unclear how generalizable the findings are across a broader range of architectures or training paradigms.\n\nLimited Visual Support: The existing visualizations may not fully substantiate the claim that mid-layers capture local objects, and higher layers shift towards grouping patches by object class.\n\nUnclear Utility of Low-Dimensional Subspace: While IsSameObject is shown to lie in a low-dimensional subspace, the practical or theoretical implications of this finding are not well articulated. It's unclear why this matters or how it can be used.", "was_revised": true, "final_justification": "The rebuttal convincingly addressed major concerns by broadening experiments beyond DINOv2 to CLIP, MAE, and supervised ViTs with standardized baselines, adding cross-layer binding analysis, clarifying claims about ViT limitations with stronger neuroscientific context, and providing further attention–binding correlation evidence. Methodological details (baselines, projection methods, ablations) were clarified, improving reproducibility. Core strengths—novel framing, rigorous experiments, and relevance to both CV and neuroscience—remain strong. Minor concerns on efficiency profiling and modest ablation effect sizes are acknowledged but carry low weight. Overall, the work is technically sound, timely, and impactful, warranting acceptance.", "rebuttal": "# Response to reviewer gvWE (1/1)\n\nDear Reviewer gvWE,\n\nWe thank the reviewer for their careful reading and for recognizing our novel *IsSameObject* hypothesis, probe design, and robust quantitative results. We appreciate your feedback and are delighted that we were able to run experiments across different objectives and that they substantially improved our paper.\n\n---\n\n### **[W1, Q1] Limited verification across pretraining objectives**\nWe have now extended our analysis beyond DINOv2 to include CLIP, MAE, and supervised-only trained ViTs (Some limited experiments with CLIP and DINO model sizes were already included in the supplementary: Lines 471-478, Appendix Figure 7). To ensure fair comparisons, we now standardized the effective patch coverage by resizing inputs so that all models process the same spatial patch divisions. This ensures a shared baseline (72.6%) across all models (corresponding to the accuracy of always predicting \"different,\" which accounts for the class imbalance in the dataset). \n\nHere are our results: \n\n**Table: Probe accuracy on the *IsSameObject* task across ViTs**\n| **Model**                  | **Highest Probe Accuracy (%)** | **Improvement Over Baseline** | **Layer at Highest Accuracy (Normalized. 0=first layer, 1= last layer)** |\n|:-------------------------- |:-----------------------:|:----------------------------:|:--------------------------:|\n| **DINOv2‑Small**           | 86.7 %                  | + 14.1 pp                    | 1.00                       |\n| **DINOv2‑Base**            | 87.5 %                  | + 14.9 pp                    | 0.82                       |\n| **DINOv2‑Large**           | 90.2 %                  | + 17.6 pp                    | 0.78                       |\n| **DINOv2‑Giant**           | 88.8 %                  | + 16.2 pp                    | 0.77                       |\n| **CLIP (ViT‑Large)**       | 84.2 %                  | + 11.6 pp                    | 0.39                       |\n| **MAE (ViT‑Large)**        | 82.9 %                  | + 10.3 pp                    | 0.65                       |\n| **Supervised (ViT‑Large)** | 76.3 %                  | + 3.7 pp                     | 0.13                       |\n\n\nThis now allows us to produce much more interesting insights: CLIP, MAE, and DINO all demonstrate object binding capabilities, supporting our broader claims about *Large Pretrained Vision Transformers* beyond DINOv2 alone. We also show that supervised ImageNet classification yields poor object‑binding performance, indicating that binding is an acquired ability under specific pretraining objectives rather than being a universal property of all vision models.\n\n\nWe offer explanations of why these training regimes work:\n- **DINO**: As noted in Lines 186–189, the DINO loss drives the model toward object-level features by enforcing consistency between augmented views, since each view contains the same objects and the model must align their representations; this objective has been shown to encourage class and object specific features (Caron et al. ICCV 2021), as the reviewer points out. \n- **CLIP**: By aligning images with text captions, CLIP effectively assigns each object a symbolic label (e.g., “the cat”), which can act like a pointer that pulls together all patches of that object. This supervision likely encourages patches from the same object to cluster in feature space. Future work could track object‑binding strength across CLIP training checkpoints to validate this hypothesis.\n- **MAE**: The masked autoencoder objective requires the model to reconstruct a missing patch from its surroundings. When the masked patch sits between multiple objects, correctly predicting its content forces the model to infer which object it belongs to, thus promoting the grouping of patches from the same object.\n\n\n**Our findings thus produce a much wider coverage of ViTs and we provide an understanding of potential reasons for why binding emerges.**\n\n\n---\n### **[W2, Q3]  Limited visual support**\nAlthough we cannot include visualizations at this stage, we have validated our claim that mid‑layers capture local object structure and higher layers emphasize object‑class grouping by training new probes. The *IsSameObjectClass* probe’s accuracy increases in deeper layers as *IsSameObject* accuracy declines, directly supporting this shift. *IsSameObjectClass* accuracy drops at the final layer (23), likely because DINO’s self‑distillation loss targets only the [CLS] token’s projection‑head output—meaning the last‑layer patch‑token activations receive no direct training signal and can drift away from class‑level structure.\n\nWe also introduced *IsSameGreyscale* probe for low‑level perceptual features; its accuracy decreases rapidly with depth, suggesting a rapid shift from low-level perceptual features to high-level semantics.\n\n\n**Table: Mean probe accuracy (%) across layers for *IsSameObject*, *IsSameObjectClass*, and *IsSameGreyscale*.**\n| Layer | IsSameObject | IsSameObjectClass | IsSameGreyscale |\n|:-----:|-------------:|------------------:|------------:|\n| 0 | 40.4% | 40.2% | 91.2% | \n| 6 | 48.2% | 50.1% | 69.4% | \n| 12 | 75.2% | 74.3% | 49.7% | \n| 18 | 70.3% | 85.5% | 25.8% | \n| 23 | 50.1% | 53.1% | 20.9% |\n\n---\n\n### **[W3, Q2] Implications of low-dimensional *IsSameObject* space**\n\nA low‑dimensional IsSameObject subspace indicates that the binding information is concentrated in just a few directions. This makes learning of those directions which matters for downstream tasks and attention far easier. In this low-dimensional space, we also find that there are large margins between the clusters (Lines 231-237, Figure 5), which makes them easy to separate and guarantees generalization. This type of representation not only supports strong generalization (because large margins between clusters imply robustness to new inputs), but also enables efficient compression: by projecting full token embeddings onto the top binding dimensions, we can distill the binding signal into a much smaller code.\n\nWe believe that our responses have dramatically strengthened the contributions of our work and are thankful for the reviewer's feedback.\n\n**Best regards,**  \n*Authors*"}, {"review_id": "kfJu1ArTwI", "reviewer": "Reviewer_4Wjo", "summary": "This work explores whether Vision Transformers (ViTs) develop object binding, which is the ability to group visual features that belong to the same object. The authors propose a pairwise property to test this and show that ViTs do represent such object-based groupings. This representation emerges through layers, aligns with attention patterns, and supports the model’s performance, suggesting that symbolic structure can emerge in connectionist systems without explicit object-centric mechanisms.", "questions": "Figure 5 shows that even visually similar objects can be distinguished, which aligns with the linear decomposition hypothesis introduced in Section 3.2. However, in Figure 4, the probe fails to distinguish between different cars. Is this due to the ground truth used during probe training? Specifically, were different instances of the same object class labelled as \"same\"? If so, have you conducted an experiment using a more fine-grained ground truth, where same-class but different objects are treated as distinct? The results from such an experiment could help reconcile the findings in Figure 4 with the trends shown in Figure 3.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "$\\textbf{Strengths:}$\n\n1. This work contributes to a deeper theoretical understanding of Vision Transformers (ViTs), shedding light on how object-level representations may naturally emerge within their architecture. These insights can inform both our interpretation of ViTs and potential improvements to their design.\n\n2. The results are promising, both quantitatively and qualitatively.\n\n$\\textbf{Weaknesses:}$\n1. ViT is limited to DINO training. Other ViTs that are trained with different methods (MAE or supervised ViT) are not included. So, the claim of general ViT would be too strong. The tone should be lowered.\n\n2. L157: Object-class and object-identity probes may not work if there are multiple objects from the same class.\n\n3. Figure 4 contrasts with the claim that object binding emerges. For example, Layer 23 images show that four objects are the same (a boat and cars). However, this is acceptable because I would trust the number in Figure 3 more than just one sample.\n\n4. L253: It is unclear how IsSameObject scores are computed. How is correlation between attention weights and IsSameObject scores are computed?\n\n\n$\\textbf{Writing Issues:}$\n1. L143 $x_j^l$ should be consistent.\n\n2. The clarity of presentation could be improved.  L153-156 should clarify that W is a learnable parameter, and it should include the details on how to train it.\n\n3. Figure 4 does not have a value for the heatmap.\n\n4. L157: What is 4.5-6? Is it a typo?", "was_revised": true, "final_justification": "The authors already addressed all my concerns and answered a question. This is a good paper with a comprehensive evaluation. The paper aims to solve a highly relevant problem in both AI and vision science. I would recommend accepting this paper.", "rebuttal": "# Response to reviewer 4Wjo (1/1)\n\nDear Reviewer 4Wjo,\n\nWe thank the reviewer for their careful reading and for recognizing the significance of our research question and robust quantitative results. We appreciate your constructive feedback and are delighted that we were able to run more experiments (across different objectives) and that they substantially improved our paper.\n\n---\n\n### **[W1, Q1] Limited verification across pretraining objectives**\nWe have now extended our analysis beyond DINOv2 to include CLIP, MAE, and supervised-only trained ViTs (Some limited experiments with CLIP and DINO model sizes were already included in the supplementary: Lines 471-478, Appendix Figure 7). To ensure fair comparisons, we now standardized the effective patch coverage by resizing inputs so that all models process the same spatial patch divisions. This ensures a shared baseline (72.6%) across all models (corresponding to the accuracy of always predicting \"different,\" which accounts for the class imbalance in the dataset). \n\n\nHere are our results: \n\n**Table: Probe accuracy on the *IsSameObject* task across ViTs**\n| **Model**                  | **Highest Probe Accuracy (%)** | **Improvement Over Baseline** | **Layer at Highest Accuracy (Normalized. 0=first layer, 1= last layer)** |\n|:-------------------------- |:-----------------------:|:----------------------------:|:--------------------------:|\n| **DINOv2‑Small**           | 86.7 %                  | + 14.1 pp                    | 1.00                       |\n| **DINOv2‑Base**            | 87.5 %                  | + 14.9 pp                    | 0.82                       |\n| **DINOv2‑Large**           | 90.2 %                  | + 17.6 pp                    | 0.78                       |\n| **DINOv2‑Giant**           | 88.8 %                  | + 16.2 pp                    | 0.77                       |\n| **CLIP (ViT‑Large)**       | 84.2 %                  | + 11.6 pp                    | 0.39                       |\n| **MAE (ViT‑Large)**        | 82.9 %                  | + 10.3 pp                    | 0.65                       |\n| **Supervised (ViT‑Large)** | 76.3 %                  | + 3.7 pp                     | 0.13                       |\n\n\nThis now allows us to produce much more interesting insights: CLIP, MAE, and DINO all demonstrate object binding capabilities, supporting our broader claims about *Large Pretrained Vision Transformers* beyond DINOv2 alone. We also show that supervised ImageNet classification yields poor object‑binding performance, indicating that binding is an acquired ability under specific pretraining objectives rather than being a universal property of all vision models.\n\nWe offer explanations of why these training regimes work:\n- **DINO**: As noted in Lines 186–189, the DINO loss drives the model toward object-level features by enforcing consistency between augmented views, since each view contains the same objects and the model must align their representations; this objective has been shown to encourage class and object specific features (Caron et al. ICCV 2021), as the reviewer points out. \n- **CLIP**: By aligning images with text captions, CLIP effectively assigns each object a symbolic label (e.g., “the cat”), which can act like a pointer that pulls together all patches of that object. This supervision likely encourages patches from the same object to cluster in feature space. Future work could track object‑binding strength across CLIP training checkpoints to validate this hypothesis.\n- **MAE**: The masked autoencoder objective requires the model to reconstruct a missing patch from its surroundings. When the masked patch sits between multiple objects, correctly predicting its content forces the model to infer which object it belongs to, thus promoting the grouping of patches from the same object.\n\n\n**Our findings thus produce a much wider coverage of ViTs and we provide an understanding of potential reasons for why binding emerges.**\n\n---\n\n### **[W2] Probe failure with multiple same‑class objects**\nWe agree that object-class and object-identity probes could fail if there are multiple objects from the same class. We’d like to clarify that these probes are used only to test possible hypotheses about how binding works in ViTs: in this case, that the model determines “same‑objectness” by identifying each token’s class or identity and then comparing the token pair.\n\nTo evaluate this, we compared probe accuracies across designs: the quadratic probe consistently outperforms both object‑class and object‑identity probes in most layers (Figure 3), indicating that DINOv2‑Large employs a more sophisticated binding strategy than simply classifying tokens by object class or identity and comparing them.\n\n\n### **[W3] Limited generalizability shown in Figure 4**\nThe reviewer questions whether Figure 4 consistently supports our claim of emergent object binding. We’d like to point out that Figure 4 is intended as a stress test, which shows extreme cases with identical objects or same‑class objects in one image to highlight where ViTs may struggle (e.g., distinguishing identical instances or class mates). It is not meant to represent average performance; Figure 3 provides the overall evaluation of object binding across layers and models.\n\nWe also note that IsSameObject performance degrades at the final layer (23), likely because DINO’s self‑distillation loss trains only the [CLS] token’s projection‑head output, leaving last‑layer patch‑token activations without direct supervision and prone to drift.\n\n---\n\n### **[W4] Ambiguous calculation of IsSameObject scores and correlations**\nIsSameObject scores are the 0–1 outputs of our trained quadratic probe, reflecting the model’s assessed probability that two patches belong to the same object. Attention weights are detailed in Supplementary Lines 530–538. We compute the correlation as the Pearson correlation between all token‑pair IsSameObject scores and their corresponding attention weights.\n\n---\n### **[Q] Ground-truth label clarification**\nWe’d like to clarify that we always use instance‑level ground truth: different objects, even when they belong to the same class, are always labeled as “different” during probe training and evaluation. Therefore, the confusion seen in Figure 4 is not due to coarse labels but reflects genuine challenges in the model’s binding ability.\n\nFigure 4 evaluates object binding under extreme conditions where multiple identical or same‑class instances in a single image. It shows that, while ViTs do exhibit object binding, it is not flawless: identical objects can still be confused.\n\n---\n### **Minor concerns**\nWe will correct the listed typos and enhance both our methods' presentation and the rigor of our figures.\n\nWe believe that our responses have dramatically strengthened the contributions of our work and are thankful for the reviewer's feedback.\n\n**Best regards,**  \n*Authors*"}]

Author rebuttal:

Meta-review: This paper investigates whether object binding, the cognitive ability to group features into coherent object representations, emerges spontaneously in pretrained Vision Transformers (ViTs). Through a series of carefully designed probing experiments centered on a novel `IsSameObject` task, the authors provide compelling evidence that this capability is indeed present, develops through the model's hierarchy, is encoded in a low-dimensional subspace, and is functionally relevant to the model's pretraining objective. The paper received strong support from all reviewers, who converged on a clear consensus for acceptance after a highly effective rebuttal period.

The paper's primary strength lies in its novel and well-motivated research question, bridging the fields of cognitive science and deep learning interpretability. Reviewers commended the work for its clear setup, rigorous methodology (including probing, ablation studies, and correlation with attention), and the significance of its central inquiry. By moving beyond architectural solutions like slot-based attention, the paper provides valuable insights into how fundamental cognitive functions can arise from inductive biases in self-supervised learning objectives.

The most significant initial concern, shared unanimously by the reviewers, was the limited scope of the experiments. The paper's claims about "Large Pretrained Vision Transformers" were not fully supported by an analysis that focused almost exclusively on the DINOv2 pretraining objective. The authors comprehensively addressed this limitation during the rebuttal by conducting a substantial new set of experiments across a wider range of pretraining paradigms, including CLIP, MAE, and supervised-only ViTs. These new results not only generalized the original findings—showing that object binding emerges in multiple self-supervised frameworks—but also provided richer insights, notably demonstrating that standard supervised classification on ImageNet does *not* lead to strong binding capabilities. This experimental expansion was highly convincing and led all reviewers to confirm or raise their scores, solidifying the consensus for acceptance.

Beyond the expanded model scope, the authors also provided effective clarifications on several other points raised during the review, including the precise definition of binding versus compositionality (Reviewer 43Cy), the potential for cross-layer binding mechanisms (Reviewer x2fh), and the interpretation of specific figures and ablations (Reviewers gvWE, 4Wjo). The discussion was constructive and significantly improved the final manuscript.

In short, the resulting paper is a strong, well-supported, and insightful contribution. It addresses a fundamental question at the intersection of AI and cognitive science with a novel and effective experimental paradigm. The work is of high interest to researchers in model interpretability, self-supervised learning, and computational neuroscience. It presents a novel finding, supported by rigorous and now much broader evidence, and successfully navigated the peer-review process to become a significantly stronger manuscript.
Decision: Accept (spotlight)