Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects
Authors: Michael A. Lepori
Alexa R. Tartaglini
Wai Keen Vong
Thomas Serre
Brenden Lake
Ellie Pavlick
Venue: NeurIPS 2024 poster (2024)
Keywords: visual reasoning
mechanistic interpretability
transformers
cognitive science
Primary area: interpretability_and_explainability
TL;DR: We use methods from mechanistic interpretability to investigate how Vision Transformers perform an abstract visual reasoning task.

Abstract: Though vision transformers (ViTs) have achieved state-of-the-art performance in a variety of settings, they exhibit surprising failures when performing tasks involving visual relations. This begs the question: how do ViTs attempt to perform tasks that require computing visual relations between objects? Prior efforts to interpret ViTs tend to focus on characterizing relevant low-level visual features. In contrast, we adopt methods from mechanistic interpretability to study the higher-level visual algorithms that ViTs use to perform abstract visual reasoning. We present a case study of a fundamental, yet surprisingly difficult, relational reasoning task: judging whether two visual entities are the same or different. We find that pretrained ViTs fine-tuned on this task often exhibit two qualitatively different stages of processing despite having no obvious inductive biases to do so: 1) a perceptual stage wherein local object features are extracted and stored in a disentangled representation, and 2) a relational stage wherein object representations are compared. In the second stage, we find evidence that ViTs can learn to represent somewhat abstract visual relations, a capability that has long been considered out of reach for artificial neural networks. Finally, we demonstrate that failures at either stage can prevent a model from learning a generalizable solution to our fairly simple tasks. By understanding ViTs in terms of discrete processing stages, one can more precisely diagnose and rectify shortcomings of existing and future models.

Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects
Michael A. Lepori
Alexa Tartaglini
Wai Keen Vong"
Thomas Serre'
Brenden M. Lake
Ellie Pavlick?
Brown University New York University Stanfoord University
(michael leporii, thomas sernas__erreeeeeee se @brown edu
elexart@stanford edu
[waikeen. vong, brenden 8nyu. edu
Abstract
Though vision transformers (ViTs) have achieved state-of-the-art performance in a variety of settings, they exhibit surprising failures when performing tasks involving visual relations. This begs the question: how do ViTs attempt to perform tasks that require computing visual relations between objects? Prior efforts to interpret ViTs tend to focus on characterizing relevant low-level visual features. In contrast, we adopt methods from mechanistic interpretability study the higher-level visual algorithms that VIT's use to perform abstract visual reasoning. We present case study of fundamental, yet surprisingly difficult, relational reasoning task: judging whether two visual entities are the same different. We find that pretrained ViTs fine-tuned on this task often exhibit two qualitatively different stages of processing despite having no obvious inductive biases to do so: 1) perceptual stage wherein local object features are extracted and stored in a disentangled representation, and 2) relational stage wherein object representations are compared. In the second stage, we find evidence that ViTs can sometimes learn to represent abstract visual relations, a capability that has long been considered out of reach for artificial neural networks. Finally, we demonstrate that failures at either stage can prevent a model from learning generalizable solution ViTs in terms of discrete processing stages, one can more precisely diagnose and rectify shortcomings of existing and future models.
fairly simple tasks. By understanding
Introduction
Despite the well-established successes of transformer models (Vaswani et al. 2017) for variety of vision applications (ViTs: Dosovitskiy et al. (2020)) notably image generation and classification there has been comparatively little breakthrough progress complex tasks involving relations between visual entities, such as visual question answering (Schwenk et al. 2022) and image-text matching (Thrush et al. 2022 Liu et al. 2023 Yuksekgonul et al. [022]. One fundamental difference between these tasks is that the former is largely semantic - relying on pixel-level image features that correlate with learned class labels whereas the latter often involves syntactic operations those which are independent of pixel-level features (Ricci et al.) 2021 Hochmann et al. [202T). Though the ability compute over abstract visual relations is thought to be fundamental human visual intelligence (Ullman 1987 Hespos et al.) (02T), the ability of neural networks to perform such
syntactic operations has been the subject of intense debate (Fodor Pylyshyn 1988 Marcus 2003
Chalmers 1992: Quilty- -unnn et al. 2023; Lake et al. [0177] Davidson et al. 2024).
Much prior work has attempted to empirically resolve whether not vision networks can implement an abstract, relational operation, typically by behaviorally assessing the model's ability to generalize
Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to held-out stimuli (Fleuret et al. 2011 Zerroug et al. 2022 Kim et al. 2018 Puebla & Bowers 2022, Tartaglimi et al. 2023). However, strikingly different algorithms might produce the same model behavior, rendering difficult to characterize whether models do or do not possess an abstract operation (Lepori et al. 2023a). This problem is exacerbated when analyzing pretrained models, whose opaque training data renders it difficult to distinguish true generalization from memorization (McCoy et al. 2023). In this work, we employ newly-developed techniques from mechanistic interpretability to characterize the algorithms learned by ViTs. Analyzing the internal mechanisms of models enables us to more precisely understand how they attempt to implement relational operations, allowing to more clearly diagnose problems in current and future models when applied to complex visual tasks.
One of the most fundamental of these abstract operations identifying whether two objects are the same or different. This operation un- dergirds human visual and analogical reason-
(a) Discrimination
(b) RMTS
X
x
x
ing (Forbus& & Lovett, 2021 Cook & Wasser- man 2007]. and is crucial for answering wide variety of common visual questions, such as "How many plates are on the table?" (as each instance of the same object) or "Are Mary and Bob reading the same book?" (Ricci et al. 202I). Indeed, same- different judgments can be found across the animal kingdom, being successfully captured by bees (Giurfa et al., 200T), ducklings (Mart- inho III & Kacelnik 20166) primates (Thompson & Oden (000), and crows (Cook & Wasserman 2007).
x
Sume
Different
Same
Different
Figure 1: Two same-different tasks. (a) Discrim- ination: "same" images contain two objects with the same color and shape. Objects in "different" images differ in at least one of those properties- this case, both color and shape. (b) RMTS: "same" images contain pair of objects that exhibit the same relation as a display pair of objects in the top left corner. In the image the left, both pairs demonstrate a "different" relation, so the classi- fication is "same" (relation). "Different" images contain pairs exhibiting different relations.
plate must be identified
We analyze ViTs trained on two same-different tasks: an identity discrimination task, which con- stitutes the most basic instances of the abstract concept of same-different (and is most commonly studied in artificial systems) (Newport 2021). ) and a relational match-to-sane sample task, which requires explicitly representing and manipulating an abstract concept of "same" or "different" (Cook & Wasserman (007)) Geiger al. (023). See Figure for examples of each. We relate the algorithms that models adopt to their downstream behavior, including compositional and OOD generalization
Our main contributions are the following:
1. Inspired by the infant and animal abstract concept learning literature, we introduce synthetic visual relation match-to-se tample (RMTS) task, which assesses a model's ability to represent and compute over the abstract concept of "same" or "different".
2. We identify processing pipeline within the layers of several but not all pretrained ViTs, consisting of a "perceptual" stage followed by a more abstract "relational" stage. We characterize each stage individually, demonstrating that the perceptual stage produces disentangled object representations (Higgins et al., 2018). while the relational stage implements fairly abstract (i.e. invariant to perceptual properties of input images) relational computations an ability that has been intensely debated since the advent of neural networks (Fodor & Pylyshyn| 1988).
3. We demonstrate that deficiencies in either the perceptual or relational stage can completely prevent models from learning abstract relational operations. By rectifying either stage, models can solve simple relational operations. However, both stages must be intact in order to learn more complex operations.
2 Methods
Discrimination Task. The discrimination task tests the most basic instance of the same-different relation. This task is well studied in machine learning (Kim et al. 2018 Puebla & Bowers 2022
"Code is available here.

---

Tartaglini et al. 2023) and simply requires a single comparison between two objects, Stimuli in our discrimination task consist of images containing two simple objects (see Figure Tt). Each object may take on one of 16 different shapes and one of 16 different colors (see Appendix A for dataset details). Models are trained to classify whether these two objects are the "same" along both color and shape dimensions or "different" along at least one dimension. Crucially, our stimuli are patch-aligned. ViTs tokenize images into patches of N X N pixels (N = {14, 16, 32)). We generate datasets such that individual objects reside completely within the bounds of single patch (for N 32 models) or within exactly patches (for = 16 and N = 14 models). Patch alignment allows us to adopt techniques from NLP mechanistic interpretability, which typically assume meaningful discrete inputs (e.g. words). To increase the difficulty of the task, objects are randomly placed within patch boundaries, and Gaussian noise is sampled and added to the tokens (see Appendix A). Models are trained on 6, 400 images.
We evaluate these models on out-of distiiitittttooonn synthetic stimuli to ensure that the learned relations generalize. Finally, we evaluate rather extreme form of generalization by generating a "realistic" dataset of discrimination examples using Blender, which include various visual attributes such as lighting conditions, backgrounds, and depth of field (see Appendix AA.3.
Relational Match-to-Sample (RMTS) Task. Due to the simplicity of the discrimination task, we also analyze a more abstract (and thus more difficult) iteration of the sume-differen relation using a relational paatch-to-sampee (RMTS) design. In this task, the model must generate explicit represen- tations of "sameness" and "difference", and then operate over these representations (Martinho III & Kacelnik] 2016; Hochmann et al., 2017). Although many species can solve the discrimination task, animals (Penn et al., (008) and children younger than (Hochmann et al. 2017 Holyoak & Lu 2021) struggle to solve the RMTS task. Stimuli in the RMTS task contain 4 objects grouped in two pairs (Figure[b)) The "display" pair always occupies patches in the top left of the image. The "sample" pair can occupy any other position. The task is defined as follows: for each pair, produce same-different judgment (as the discrimination task). Then, compare these judgments both pairs exhibit the same intermediate judgment (i.e., both pairs exhibit "same" or both pairs exhibit "different"). ) then the label is "same". Otherwise, the label is "different." " Objects are identical to those in the discrimination task, and they are similarly aatch-aligned.
Models. Tartaglini et al. 2023) demonstrated that CLIP- CLIP--reetainne preerraaieed VITs (Radford et al. 2021 can
achieve high performance in generalizing the same-different relation to out-of-distribute stimuli when fine-tuned on a same-different discrimination task. Thus, we primarily focus our analysis on this mode(") 3 In later sections, we compare CLIP to additional ViT models pretrained using DINO (Caron et al., (021), DINOv2 (Oquab et al.), masked auto encoding (MAE: He et al. (2022)) and ImageNet classification (Russakovsky et al. 20155) Dosovitskiy et al. 2020 objectives. We also train a randomly-initializee ViT model on each task (From Scratch). All models are fine-tuned on either a discrimination or RMTS task for 200 epochs using the Adiam optimizer with learning rate of 1c-6. We perform a sweep over learning rate schedulers (Exponent ial with a decay rate of 0.95 and Reduce lateat eau with a patience of 40). Models selected by validation accuracy, and test accuracy is reported. Our results affirm and extend those presented in TTrtaglini et al. (2023) CLIP and DINOv2-pretrainee ViTs perform extremely well on both tasks, achieving 297% accuracy held-out test set. Appendix B presents results for all models.
3 Two-Stage Processing in ViTs
We now begin to characterize the internal mechanisms underlying the success of the CLIP and DINOv2 ViT models. In the following section, we cover how processing occurs in two distinct stages within the model: a perceptual stage, where the object tokens strongly attend other tokens within the same object, and a relational stage, where tokens in one object attend to tokens in another object (or pair of objects).
Methods - Attention Pattern Analysis. We explore the operations performed by the model's attention heads, which "read" from particular patches and "write" that information to other patches (Elhage et al. (202T). In particular, we are interested in the flow of information within individual objects, between the two objects, and (in the case of RMTS) between two pairs of objects. We refer to attention heads that consistently exhibit within-object patterns across images as local attention
Both B/16 and B/32 versions. Results for the B/32 variant presented in Appendix C
3

---

CLIP Discrimination
(b) From Scratch Discrimination
Global
0.75
WO
0.75
0.5
Attention Head Score
BG
3
5
7
9
10
11
0
6
10
0.5
(c) CLIP RMTS
(d) DINO RMTS
0.25
0.75
0.5
0.25
Local
WO
BP
BG
0
3
7
0
2
6
8
10
Model Layer
Figure 2: Attention Pattern Analysis. (a) CLIP Discrimination: The heatmap (top) shows the distribution of "local" (blue) vs. "global" (red) attention heads throughout CLIP ViT-B/16 model fine-tuned on discrimination (Figure[pa.. The r-axis is the model layer, while the y-axis is the head index. Local heads tend cluster in early layers and transition global heads around layer 6. For each layer, the line graph (boltom) plots the maximum proportion of attention across all 12 heads from object patches to image patches that are 1) within the same object (within-object= (bbject=WO),, 2) within the other object (within-paie Wa) or 3) in the background (BG). The stars mark the peak of each. WO attention peaks early layers, followed by WP, and finally BG. (b) From Scratch Discrimination: We repeat the analysis in (a). The model contains nearly zero local heads. (c) CLIP RMTS: We repeat the analysis for a CLIP model fine-tuned on RMTS (Figure[p). Top: Our results largely hold from (a). Bottom: We track fourth attention pattern attention between pairs of objects (between pair-BP). We find that WO peaks first, then WP, then BP, and finally BG. This accords with the hierarchical computations implied by the RMTS task. (d) DINO RMTS: We repeat the analysis in (c) for DINO model and find no such hierarchical pattern.
heads and heads that attend to other tokens global attention heads. To classify an attention head as local or global, we score the head from 0 to I, where values closer to indicate local operations and values closer to indicate global operations. To compute the score for an individual head, we collect its attention patterns on 500 randomly selected "same" and "different" images (1, 000 images total). Then, for each object in given image, compute the proportion of attention from the object's patches to any other patches that do not belong to the same object (excluding the CLS token)-this includes patches containing the other object(s) and non-object background tokens This procedure yields two proportions, one for each object in the image. The attention head's 's score for the image is the maximum of these two proportions. Finally, these scores averaged across the images to produce the final score.
Attention head scores for CLIP VIT-B/16 fine-tuned on the discrimination and RMTS tasks are displayed in the heatmaps of Figure 2 and [2. respectively. The first half of these models dominated by attention heads that most often perform local operations (blue cells). See Appendix F for examples of attention patterns. In the intermediate layers, attention heads begin to perform global operations reliably, and the deeper layers of the model are dominated by global heads. The prevalence of these two types of operations clearly demarcates two processing stages in CLIP ViTs: perceptual stage where within-object processing occurs, followed by relational stage where
Results.
"We include attention from object to non-object tokens because we observe that models often move object information to of background register tokens (Darcet al. 2023). See Appendix F

---

Counterfactual Interventions: CLIP B/16
Base
Source
Intervention Acc.
Disc. Shape
0.8
MO0 000000000000
Disc. Color
0.6
RMTS Shape
0.4
RMIS Color
0.2
9
10
Model Layer (b)
(a)
Figure 3: (a) Interchange interventions: The base image exhibits the "different" relation, as the two objects differ in either shape (top) or color (bottom). An interchange intervention extracts (shape, color color]inforne information from the intermediate representations generated by the same model run on a different image (source), then patches this information from the source image into the model's intermediate representations of the base image. successful, the intervened model will now return "same" when run on the base image. DAS is optimized to succeed at interchange interventions. (b) Disentanglement Results: We report the success of interchange interventions on shape and color across layers for CLIP ViT-B/16 fine-tuned on either the discrimination or RMTS task. We find that these properties are disentangled early in the model-one property can manipulated without interfering with the other. The background is colored according to the heatmap in Figure 2, where blue denotes local heads and red denotes global heads.
between-objed processing occurs. These stages are explored in further depth in Sections and [] respectively | We also find similar two-stage processing when evaluating discrimination dataset that employs realistic stimuli, suggesting that the patterns observed on our synthetic stimuli are robust and transferable (see Appendix
A.3).
The line charts in Figure show maximal scores of each attention head type (local and global) in each layer. Since values closer to 0 indicate local (i.c., within-object; WO in Figure 2 heads by construction, we plot these values subtracted from The global attention heads are further broken down into two subcategories for the discrimination task: within-pair attention heads, whereby the tokens of one object attend to tokens associated with the object it is being compared to (WP in Figure 2) and background attention heads, whereby object tokens attend to background tokens (BG in Figure 2 We add a third subcategory for RMTS: between- pair attention heads, which attend to tokens in the other pair of objects (e.g., display object attending a sample object; BP in Figure 2) For both tasks, objects strongly attend to themselves throughout the first six layers, with peak at layer 3. Throughout this perceptual stage, within-pair heads steadily increase in prominence and peak in layer 6 (discrimination) or 5 (RMTS). In RMTS models, this within-pair peak is followed by between-pair peak, recapitulating the expected sequence of steps that one might use solve RMTS. Notably, the within-pair (and between-pair) peaks occur precisely where an abrupt transition from perceptual operations relational operations occurs. Around layer 4, object attention set of background tokens begins to increase; after layer 6, object-to-backgroume backkgoudd attention accounts for nearly all outgoing attention from object tokens. This suggests that processing may have moved into a set of register tokens (Darcet et al. 2023).
Notably, this two-stage processing pipeline is not trivial to learn-several models, including randomly initialized model trained on the discrimination task and DINO model trained on RMTS (Figure[2p and d) fail exhibit any obvious transition from local to global operations (See Appendix E for results from other models). However, we do find this pipeline in DINOv2 and ImageNet pretrained models (See Appendix D and E). We note that this two-stage processing pipeline loosely recapitulates the processing sequence found in biological vision systems: image representations are first formed
during a feedforward sweep of the visual cortex, then feedback connections enable relational reasoning over these representations Kreiman & Serre (2020).
*We note that this attention pattern analysis is conceptually similar to Raghu et al. 2021 which demonstrated a general shift from local to global attention in ViTs. However, our analysis detines local vs. global heads in terms of objects rather than distances between patches.
5

---

Novel Representation Analysis: CLIP B/16 (Discrimination)
Added
Interpolated
Shape
Shape
Intervention Acc.
Color
Color
Base
?
Sampled
Random
0.8
Shape Color
Shape
06
Color
0.4
0.2
0
3
9
11
4
10
Model Layer
a
(b)
Figure 4: (a) Novel Representations Analysis: Using trained DAS interventions, we can inject any vector into model's shape or color subspaces, allowing us to test whether the same-different operation can be computed over arbitrary vectors. We intervene on a "different"
difffering only in its color property-by patching novel color (an interpolation of red and black) into both objects in order to flip the decision to "same". (b) Discrimination Results: We perform novel representations analysis using four methods for generating novel representations: 1) adding observed representations, 2) interpolating observed representations, 3) per-dimension sampling using a distribution derived from observed representations, and 4) sampling randomly from normal distribution (0, 1). The model's same-different operation generalizes well to vectors generated by adding (and generalizes somewhat interpolated vectors) in early layers but not sampled or random vectors. The background is colored according to the heatmap in Figure 2h (blue=local heads; red=global heads).
4
The Perceptual Stage
Attention between tokens is largely restricted to other tokens within the same object in the perceptual stage, but to what end? In the following section, we demonstrate that these layers produce disentangled local object representations which encode shape and color. These properties are represented in separate linear subspaces within the intermediate representations of CLIP and DINOv2-pretre prret--ree ViTs.
Methods - DAS. Distributed Alignment Search (DAS) (Geiger al. 2024 Wu al. 2024c)
is used to identify whether particular variables are causally implicated in model's 's computation Given neural network M. hypothesized high-level causal model C. and high-level variable v. DAS attempts to isolate a linear subspace s of the residual stream states generated by M that represents (i.e. takes on a value 81 to represent 11, 82 to represent U2, and so on). The success of DAS is measured by the success of counterfactual interventions. If C(e)) = y1 and C(uz) V2. and ((x) Y/1 for some input x, does replacing $1 with $2 change the model's 's decision to yz?
Concretely, M corresponds to our pretrained ViT, and a high-level causal model for the discrimination task can be summarized follows: 1) Extract shape, and color; from ob object, repeat for object;; 2) Compare shape; and shape2. compare color, and colorg; 3) Return same if both comparisons return same, otherwise return different. Similarly, we can define a slightly more complex causal model for the RMTS task. We use this method to understand better the object representations generated by the perceptual stage. In particular, we try to identify whether shape and color are disentangled (Higgins et al.) 2018) such that we could edit shape, _ shape's without interfering with either color property (See Figure BR.. For this work, we use a version of DAS where the subspace s is found by optimizing a differentiable binary mask and rotation matrix over model representations (Wu et al. 2024a). See Appendix G for technical details.
Results. We identify independent linear subspaces for color and shape in the intermediate repre- sentations produced in the early layers of cler-prerrained ViTs (Figure 3p). In other words,
"DAS has recently come under scrutiny for being too expressive (and thus not faithful to the model's intemal algorithms) when misused (Makelov et al. (2023). of. |Wu et al. (2024b). Following best practices, we deploy DAS on the residual stream. We the pyvene library (Wu al. 2024a) for all DAS experiments.

---

Abstract Representations of Same and Different: CLIP B/16
Probe Acc.
Intervention
Control
0.75
Accuracy
0.5
0.25
7
8
Model Layer
Figure 5: Linear probing and intervention results. We probe for the intermediate same-different judgments required perform the RMTS task (bluc). Probe performance reaches ceiling around layer 5 and maintains throughout the rest of the model. We use the directions defined by the linear probe to intervene model representations and flip intermediate judgment (green). This intervention succeeds reliably at layer 5 but not deeper. We add vector that is consistent with a pair's exhibited same-different relation control (yellow). This has little effect. The background is colored according to the heatmap in Figure 2c (blue=local heads; red=global heads).
can extract either color or shape information from one object and inject it into another object. This holds for both discrimination and RMTS tasks. One conclude that least one function of the perceptual stage to form disentangled local object representations, which are then used to solve name-different tasks. Notably, these local object representations formed in the first few layers and become increasingly irrelevant in deeper layers; intervening on the intermediate representations of object tokens layers and beyond results in chance intervention performance or worse. DINOv2- pretrained ViT' provide similar results (Appendix]) D). whereas other models exhibit these patterns less strongly (Appendix [ We present control experiment Appendix H. which further confirms our interpretation of these results.
5 The Relational Stage
We now characterize the relational stage, where tokens within one object largely attend to tokens in the other object. We hypothesize that this stage takes in the object representations formed in the perceptual stage and computes relational same-different operations over them. We find that the operations implemented by these relational layers are somewhat abstract that 1) they not rely on memorizing individual objects and 2) one can identify abstract same and different representations in RMTS task, which are constant even as the perceptual qualities of the object pairs vary.
Methods Patching Novel Representations. In Section 4 we identify independent linear subspaces encoding shape and color. Does the content of these subspaces matter the same-different computation? One can imagine an ideal same-different relation that is completely abstracted away from the particular properties of the objects being compared. this setting, model could accurately judge "same" vs. "different" for object representations where colors and shapes are represented by arbitrary vectors. To study this, we intervene on the linear subspaces for either shape or color for both objects in pair, replacing the content found therein with novel representations (see Figure 4p). To create "different" example, we start with a "same" image and replace the shape (or color) representations of both objects with two different novel representations; to create "same" example, we start with a "different" image and replace them with two identical novel representations. We then assess whether the model's decision changes accordingly. We generate novel representations using four methods: 1) we add the representations found within the linear subspaces corresponding to two randomly sampled objects in an IID validation set, 2) we interpolate between these representations, 3) we sample each dimension randomly from a distribution embeddings, and 4) we sample each dimension from OOD random distribution (a μ 0 normal). See Appendix | for technical details.
Results. The results of patching novel representations into a CLIP-pretrained ViT are presented in Figure 4p. Overall, we find the greatest success when patching in added representations, followed by interpolated representations. We observe limited success when patching sampled representations, and success patching random vectors. All interventions perform best in layers 2 and 3, towards
7

---

Disentanglement vs. Generalization: Disc.
Disentanglement vs. Generalization: RMTS
1.00
II0
IID
OOD
0.35
OOD
Compositional
Compositional
0.90
0.55
Generalization
Generalizane
0.70
0.65
0.3
0.4
0.5
0.5
0.7
0.8
0.9
0.60
0.55
0.70
0.75
0.80
0.85
0.90
Counterfactual Intervention
Counterfactual Intervention
Figure 6: We average the best counterfactual intervention accuracy for shape and color and plot against IID, OOD, and Compositional Test set performance for CLIP, DINO, DINOv2, ImageNet, MAE, and from-scratch B/16 models. We observe that increased disentanglement (i.e. higher counterfactual accuracy) correlates with downstream performance. The from from-scratch scratch model achieved only chance IID performance in RMTS, so we omitted from the analysis.
the end of the perceptual stage. Overall, this points to a limited form of abstraction in the relational stage. The same-different operation is somewhat abstract while it cannot operate completely arbitrary vectors, it can generalize to additions and interpolations of shape & color representations, indicating that it does not rely rotrely rote memorization of specific objects. Results for CLIP- -reP--prttrieed ViTs on RMTS and other models on both tasks are found in Appendix K Results for DINOv2 are found in Appendix D Other models largely produce similar results to CLIP in this analysis.
Methods Linear Interventions. The RMTS task allows us to further characterize the relational stage, as it requires first forming then comparing intermediate representations of same and different. Are these intermediate representations abstract (i.e. invariant to the perceptual qualities of the object pairs that underlie them)? We linearly probe for intermediate same or different judgments from the collection of tokens corresponding to object pairs. The probe consists of a linear transformation mapping the residual stream to dimensions representing same and different. Each row of this transformation can be viewed as a direction d in the residual stream corresponding to the value being probed for (e.g. d'ame is the linear direction representing same). We train probe for each layer on images from the model's train set and test on images from a test set. To understand whether the directions discovered by the probe are causally implicated in model behavior, we create a counterfactual intervention (Nanda et al., 2023). In order to change an intermediate judgment from same different, we add the direction daiff the intermediate representations of objects that exhibit the same relation. We then observe whether the model behaves as if this pair now exhibits the different relation | control intervention where we add the incorrect direction (e.g., we add d when the object pair is already "same"). This control intervention should not reliably flip the model's downstream decisions.
We run this intervention on images from the model's test set. We also run
Results. Probing and linear intervention results for CLIP-pretrained ViT are shown in Figure 5 We observe that linear probe performance peaks in the middle layers of the model (layer 5) and then remains high. However, our linear intervention accuracy peaks at layer 5 and then drops precipitously. Notably, layer 5 also corresponds the peak of within-pair attention (see Figure 2c). This indicates that-at least in layer 5-there exists single direction representing same and a single direction representing different. One can flip the intermediate same-different judgment by adding vector in one of these directions to the residual streams of any pair of objects. Finally, the control intervention completely fails throughout all layers, expected. Thus, CLIP ViT does in fact generate and operate over abstract representations of same and different in the RMTS task. We find similar results for DINOv2 pretrained model (see Appendix D). but not for others (see Appendix K).
Priior work normalizes the intervention directions and searches over a scaling parameter. In our setting, we find that simply adding the direction defined by probe weights without rescaling works well. See Appendix K for further exploration. We transf ooeerrermeeeeee for this intervention (Nanda & Bloom 2022).

---

Disentanglement Correlates with Generalization Performance
Object representations that disentangle perceptual properties may enable model to generalize to out- of-distribution stimuli. Specifically, disentangled visual representations may enable compositional generalization to unseen combinations of perceptual properties (Higgins et al. (2018): [Bnniio et al. (2013), cf. Locatello et al. )2019) & generalization, we fine-tune CLIP. ImageNet. DINO, DINOv2, MAE, and -iiitiiiiiiiiilizeeddd ViTs on new dataset where each shape only ever paired with two distinct colors. We then repeat our analyses in Section 4 to identify independent linear subspaces for shape and color We evaluate models in 3 settings: 1) on an IID test set consisting of observed shape-color combinations, 2) on a compositional generalization test set consisting of unobserved shape-color combinations (where each shape and each color have been individually observed), and 3) on an OOD test set consisting of completely novel shapes and colors. We plot the relationship between disentanglement (i.e. counterfactual intervention accuracy) and overall performance on these model evaluations. We find a consistent trend: more disentangled representations correlates with downstream model performance in all cases (See 6igure(6).
To investigate the relationship between disentanglement and
7 Failure Modes
Previous sections have argued that pretrained ViTs that achieve high performance when finetuned on same-different tasks implement a two-stage processing pipeline. In this section, we argue that both perceptual and relational stages can serve as failure points for models, impeding their ability to solve same-differees tasks. In practice, tasks that rely on relations between objects likely have perceptual and relational stages that are orders of magnitude more complex than those we study here. The results presented herein indicate that solutions targeting either the perceptual (Zeng et al.) 2022) or relational (Bugliarello et al. 2023) stages may be insufficient for producing the robust, abstract computations that we desire.
Perceptual and Relational Regularizers. We introduce two loss functions, designed induce disentangled object representations and multi-stage relational processing, respectively. When em- ploying the disentanglement loss, introduce token-level probes that are optimized predict shape information from one linear subspace (e.g., the first 384 dimensions) of the representations generated at an intermediate layer of the model and color information from the complementary linear subspace at that same layer (layer 3, in our experiments). These probes are optimized during training, and the probe loss is backpropagated through the model. This approach is motivated by classic work on disentangled representations (Fastwood & Williams (0088). The pipeline loss is designed to encourage discrete, specific stages of processing by regularizing the attention maps to maximize the attention pattern scores defined in Section [3] Specifically, early layers are encouraged to maximize attention within-object, then within-pair, and finally (in the case of RMTS stimuli) between-pair. See Appendix for technical details. Note that the disentanglement loss targets the perceptual stage of processing, whereas the pipeline loss targets both perceptual and relational stages.
Results. First, we note that models trained from scratch on the discrimination task do not clearly distinguish between perceptual and relational stages (Figure Thus, we might expect that a model trained on limited number of shape-color combinations would not learn a robust representation of the same-different relation. Indeed, Table [ confirms this. However, we see that either the disentanglement loss or the pipeline loss is sufficient for learning generalizable representation of this relation.
Similarly, we find that models trained from scratch on the RMTS task only achieve chance perfor- mance. However, in this case we must include both disentanglement and pipeline losses in order to induce a fairly general (though still far from perfect) hierarchical representation of same-different. This provides evidence that models may fail at either the perceptual or relational stages: they might
'Though Locatello et al. (2019) find that then-current measurements of of disentanglement fail to correlate with downstream performance in variational autoencoders, many of those models failed to produce disentangled representations in the first place.
"In order to train these interventions, we generate counterfactual images, some of which contain shape-color pairs that are not seen during model training. However, we emphasize that we only use these interventions to derive disentanglement metric, not train model weights.

---

Task
Disent. Loss
Pipeline Loss
Train Ace.
Test Acc.
Comp. Acc.
Disc.
77.3%
76.5%
76.0%
Disc.
97.3% (+20)
94.6% (+18.1)
86.1% (+10.1)
Disc.
95.6% (+18.3)
93.9% (+17.4)
92.3% (+16.4)
RMTS
49.2%
50.
50.0%
RMTS
V
53.9% (+4.7)
54.4% (+4.3)
54. 1% (+4.1)
RMTS
66.1% (+16.9)
50.1%
50. % (+0.1)
RMTS
95.1% (+45.9)
94.1% 1% (+44)
77.4% (+27.4)
Table 1: Performance of ViTs trained from scratch with auxiliary losses. Adding either disentanglement loss term to encourage disentangled object representations (Disent. Loss) or pipeline loss to encourage two-stage processing in the attention heads (Pipeline Loss) boosts test accuracy and compositional generalization (Comp. Acc.) for the discrimination task. Both auxiliary losses are required boost accuracy for the RMTS task.
fail to produce the correct types of object representations, and/or they might fail to execute relational operations over them. See Appendix M for further analysis.
Discussion
This work takes inspiration from the field of mechanistic interpretability, which seeks to characterize the algorithms that neural networks implement (Olah, 2022). Though many of these ideas originated the domain of NLP (Wang
Related Work.
2022
Hanna al. 2024 Feng & Steinhardt
2023 Wu et al. 2024c Merullo et al. 2023 Geva et al. 2022 Meng et al. 2022 and in toy settings
(Nanda et al. 2022 Elhage et al. 2022, Li et al.) 2022), 2, they are beginning to find applications in computer vision (Fel et al. [0233 Vilas et al. 2024 Palit et al. (023). These techniques augment an already-robust suite tools that visualize the features (rather than algorithms) that vision models use (Olah et al. 2017 Selvaraju et al. 20177: Simonyan et al. 2014). Finally, this study contributes to growing literature employing mechanistic interpretability to address debates within cognitive science
(Millière & Buckner 2024 Lepori et al. 2023a Kallini et al. 2024 Traylor et al. 2024).
Conclusion. The ability to compute abstract visual relations is fundamental aspect biological visual intelligence and a crucial stepping stone toward useful and robust artificial vision systems. In this work, we demonstrate that some fine-tuned ViTs adopt a two-stage processing pipeline to solve same-differee tasks tasks-despitt having no obvious inductive biases towards this algorithm. First, models produce disentangled object representations somewhat abstract version of the same-different computation in relational stage. Finally, we observe a correlation between disentanglement and generalization and note that models might fail to learn either the perceptual or relational operations necessary to solve task.
perceptual stage; models then compute a
Why do CLIP and JINOv2-pretraing ViTs perform favorably and adopt this two-stage algorithm so cleanly relative to other pretrained models? Raghu et al. (02T)) find that models pretrained on more data tend to learn local attention patterns early layers, followed by global patterns later layers. Thus, pretraining scale (rather than training objective) might enable these models to first form local object representations, which are then used in global relational operations. Future work might focus on pinning down the precise relationship between data scale and relational reasoning ability, potentially by studying the training dynamics of these models. Additionally, future work might focus on characterizing the precise mechanisms (e.g. the attention heads and MI.s)) used to implement the perceptual and relational stages, or generalize our findings to more complex relational tasks.
References
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1888 2013.
10

---

Bugliarello, E.. Nematzadeh, A., and Hendricks, L. A. Weakly-supervised learning of visual relations in multimodal pretraining. In The 2023 Conference on Empirical Methods Natural Language Processing, 2023.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in supe-vsee vision transformers. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 9650-9660, 2021.
Chalmers, D. J. Syntactic transformations on distributed representations. Connectionist Natural Language Processing: Readings from Connection Science, pp. 46-55, 1992.
Cook, R. G. and Wasserman, E. Learning and transfer of relational matching-to-saan by pigeons. Psychonomic Bulletin & Review, 14(6): 107-1114, 2007.
Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. Vision transformers need registers. arXiv
preprint arXiv:2309. 16588, 2023.
Davidson, G., Orhan, E., and Lake, M. Spatial relation categorization in infants and deep neural networks. Cognition, 245:105690, 2024.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 19929, 1929, 2020.
Eastwood, C. and Williams, K. A framework for the quantitative evaluation of disentangled representations. International conference on learning representations, 2018.
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen,
A., Conerly, T., et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1:1, 2021.
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., et al. Toymodelsof superposition. 2022.
Fel, T., Picard, A., Bethune, L., Boissin, T., Vigouroux, D., Colin, J., Cadène, R., and Serre, T. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 271 1-2721, 2023.
Feng, J. and Steinhardt, J. How do language models bind entities in context? arXiv preprint MxXiv23300177111, 2023.
Fleuret, F., Li, T., Dubout, C., Wampler, E. K., Yantis, S., and Geman, D. Comparing machines and humans on a visual categorization test. Proceedings of the National Academy of Sciences, 108(43);
17621-17625, 2011.
Fodor, J. A. and Pylyshyn, W. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3-71 1988.
Forbus, K. D. and Lovett, A. Same/different in visual reasoning. Current Opinion in Behavioral Sciences, 37:63-68 68, 2021.
Geiger, A., Carstensen, A., Frank, M. C.,, and Potts, C. Relational reasoning and generalization using
nonsymbolic neural networks. Psychological Review, 130(2):308, 2023.
Geiger, A., Wu, Z., Potts, C.,, Icard, T., and Goodman, N. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pp- 160-187. PMLR, 2024.
Geva, M., Caciularu, A., Wang, K., and Goldherg, Y. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2022.
Giurfa, M., Zhang, S., Jenett, A., Menzel, R., and Srinivasan, M. V. The concepts of 'sameness" and 'difference" in an insect. Nature, 11006831):930-9999 1):330-933, 2001.

---

Hanna, M., Liu, O., and Variengien, A. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, Advances Neural Information Processing Systems, 36, 2024.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 16000-16009, 2022.
Hespos, S., Gentner, D., Anderson, E., and Shivaram, A. The origins of same/different discrimination in human infants. Current Opinion in Behavioral Sciences, 37:69-74, 2021.
Higgins, I,, Amos, D., Pfau, D., Racaniere. S., Matthey, L., Rezende, D., and Lerchner, A. Towards a definition of disentangled representations. arXiv preprint arXiv: 1tr.iv:8112.022300 2018.
Hochmann, J..R., Tuerk, A. S., Sanborn, S., Zhu, R., Long, R., Dempster, M., and Carey, S. Children's representation of abstract relations in relational/array match-to-sample tasks. Cognitive psychology, 99:17-43, 2017.
Hochmann, J.-R., Wasserman, E., and Carey, S. Editorial overview: Same-different conceptualization. Current Opinion in Behavioral Sciences, 37:iiiv,, 2021.
Holyoak, K. J. and Lu, H. Emergence of relational reasoning. Current Opinion in Behavioral Sciences, 37:118-124, 2021.
Kallini, J., Papadimitriou. Futrell, R., Mahowald, K., and Potts, C. Mission: Impossible language models. arXiv preprint arXhv:2401.06416 2024.
Kim, J., Ricci, M., and Serre. T. Not-so-clevr; learning same sume-differen relations strains feedforward neural networks. Interface focus. 8(4) 2018
Kingma, D. and Ba, J. Adam: method for stochastic optimization. arXiv:1412.6980, 2014.
arXiv preprint
Kreiman, G. and Serre, T. Beyond the feedforward sweep: feedback computations in the visual cortex. Annals of the New York Academy of Sciences, 1464(1):222-241 2020.
Lake, B. M., Ullman, D., Tenenbaum, B., and Gershman, S. J. Building machines that learn and think like people. Behavioral and brain sciences, 40:c253,
Lepori, M., Serre, T., and Pavlick, E. Break it down: Evidence for structural compositionality in neural networks. Advances in Neural Information Processing Systems, 36:42623 36:42623-42660, 2023a.
Lepori, M. A., Pavlick, E., and Serre, T. Neurosurgeon: A toolkit for subnetwork analysis. arXiv preprint arXiv:2309.0022 2023b.
Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., and Wattenberg, M. Emergent world represen- tations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference Learning Representations, 2022.
Liu, F., Emerson, G., and Collier, N. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635-651, 2023.
Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 41 14244... PMLR, 2019.
Makelov, A., Lange, G., Geiger, A., and Nanda, N. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching. In The Twelfth International Conference on Learning Representations, 2023.
Marcus, G. F. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2003.
Martinho III, A. and Kacelnik, Ducklings imprint on the relational concept of "same or different". Science, 353(6296):286-2888 2016.
12

---

McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., and Celikyilmaz, A. How much do language models copy from their training data? evaluating linguistic novelty text generation using raven. Transactions of the Association for Computational Linguistics, 1 11:652-670, 2023.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 11117759-177777 2022.
Merullo, J., Eickhoff, C., and Pavlick, E. Circuit component reuse across tasks in transformer language models. arXiv preprint arXiv:2310.08744, 2023.
Millièree and Buckner, C. philosophical introduction to language models-part ii: The way forward. arXiv preprint arXiv:2405.03227, 2024.
Nanda, N. and Bloom, J. Transformerlens. httppp///gghhhh.../////////mmmmm
Trans fransformer.es Lens 2022.
Nanda, N., Chan, L., Lieberum, T., Smith. J., and Steinhardt, J. Progress measures for grokking via mechanistic interpretability. The Eleventh International Conference on Learning Representa- lions, 2022.
Nanda, N., Lee, A.. and Wattenberg, M. Emergent linear representations in world models of self- supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 16-30, 2023.
Newport, C. Abstract concept learning in fish. Current Opinion Behavioral Sciences, 37:56-62, 2021.
Olah, C. Mechanistic interpretability, variables, and the importance of interpretable bases. Trans- former Circuits Thread, 2022.
Olah, C., Mordvintsev, A., and Schubert, L. Feature visualization. Distill, 2(1 2(11)) 2017.
Oquab, M.. Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M.. Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research.
Palit, V., Pandey, R., Arora, A.. and Liang, P.P. Towards vision- languae mechanistic interpretability: A causal tracing tool for blip. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2856-28611 2023.
Penn, D. C., Holyoak, K. J., and Povinelli, J. Darwin's 's mistake: Explaining the discontinuity between human and nonhuman minds. Behavioral and brain sciences, (1)2)1109-13) 2008.
Puebla, G. and Bowers, J. S. Can deep convolutional neural networks support relational reasoning in the saff-eifferent task? Journal of Vision, 22(10):11-11, 2022.
Quilty-Dunn, J., Porot, N., and Mandelbaum, E. The best game town: The reemergence of the off-ouagg--o--huugll hypothesis across the cognitive sciences. Behavioral and Brain Sciences, 46: e261, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Míshkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8766 PMLR, 2021.
Raghu, M., Unterthiner, T.. Kornblith, S., Zhang, C., and Dosovitskiy, A. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34: 121188,,,,, 2021.
Ricci, M., Cadène, R., and Serre, Same-different conceptualizatton:: a machine vision perspective. Current Opinion in Behavioral Sciences. 37:47-55, 2021.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,, Karpathy, A., Khosla, A., Bernstein, M., et Imagenet large scale visual recognition challenge. International journiaal of computer vision, 115:211-22 2015.
13

---

Savarese, P., Silva, H., and Maire, M. Winning the lottery with continuous sparsification. Advances in neural information processing systems, 33:1 1380-11390. 2020.
Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R. A-okvqa: benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pp. 142.162. Springer, 2022.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE international conference on computer vision, pp. 618-628 2017.
Simonyan, K., Vedaldi, A., and Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR). ICLR, 2014.
Tartaglini, A. R., Feucht, S., Lepori, M. A., Vong, W. K., Lovering, C., Lake, B. M., and Pavlick, E. Deep neural networks can learn generalizable sume-different visual relations. arXiv preprint 10.06:2310009611. /2, 2023.
Thompson, R. K. and Oden, D. L. Categorical perception and conceptual judgments by nonhuman primates: The paleological monkey and the analogical ape. Cognitive Science, 2433)363.-36, 2000.
Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., and Ross, C. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2238-,248, 2022.
Traylor, A., Merullo, J., Frank, M. J., and Pavlick, E. Transformer mechanisms mimic frontostriatal gating operations when trained on human working memory tasks. arXiv preprini arXiv:2402.082111 2024.
Ullman, S. Visual routines. In Readings in computer vision, pp. 298-328. Elsevier, 1987.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, E.,, and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Vilas, G., Schaumlöffel, T., and Roig. G. Analyzing vision transformers for image classification in class embedding space. Advances in Neural Information Processing Systems, 36, 2024.
Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations, 2022.
Wu, Z., Geiger, A., Arora, A., Huang, J., Wang, Z., Goodman, N. D., Manning, C. D., and Potts, C. pyvene: A library for understanding and improving pytorch models via interventions. arXiv preprint X11002403.00809, 2024a.
Wu, Z., Geiger, A., Huang, J., Arora, A., Icard, T., Potts, C., and Goodman, A reply to makelov et al.(223)) s interpretability illusion" arguments. arXiv preprint (xXi:24401.1263. 2024b.
Wu, 7., Geiger, A., Icard, T., Potts, C., and Goodman, N. Interpretability at scale: Identifying causal mechanisms alpaca. Advances Neural Information Processing Systems, 36, 2024c.
Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, J. When and why vision- -aniunle models behave like bags-of-words, words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2022.
Zeng, Y., Zhang, X., and Li, H. Multi Multi-grained vision language pre-training: Aligning texts with visual concepts. In International Conference on Machine Learning, pp. 25994-26009. PMLR, 2022.
Zerroug, A., Vaishnav, M., Colin, J., Musslick, S., and Serre, T. A benchmark for compositional visual reasoning. Advances in neural information processing systems, 01129776--97888 2022.
14

---

A
Dataset Details
Figure 7: All 16 unique shapes and colors used to construct the Discrimination and RMTS tasks.
There thus 16 16 256 unique objects in our same-different datasets.
A.1 Constructing the Objects
Figure] 7 demonstrates single instance of the 16 shapes and 16 colors used in our datasets. Any shape can be paired with any color create one of 256 unique objects. Note that object colors are not uniform within a given object. Instead, each color defined by three different Gaussian distributions
for each RGB channel in the image-that that determine the value of each object pixel. For example, the color red is created by these three distributions: .N 233, a = 10) in the red channel, N(n 30, o 10) in the green channel, and ((, 90, 10) in the blue channel. All color distributions have a variance fixed at 10 to give them an equal degree of noise. Any sampled values that lie outside of the valid RGB range of [0, 255) are clipped to either 0 255. Object colors are re-randomized for every image, so no two objects have the same pixel values even if they are the same color. This was done prevent the models from learning simple heuristics like comparing single pixels in each object.
X
+
Different
*
Same
(a) Discrimination
(b) RMTS
Figure 8: More examples of stimuli for the discrimination and RMTS tasks. The top row shows "different" examples, while the bottom row shows "same" examples. Note that "different" pairs may differ one both dimensions (shape & color).
A.2 Constructing the Datasets
The train, validation, and test sets for both the discrimination and RMTS tasks each contain 6, 400 unique stimuli: 3, 200 "same" and 3, 200 "different." To construct a given dataset, we first generate all possible same and different pairs of the 256 unique objects (see Figure 7 We consider two objects to be the same if they match in both shape and color color- otherwise, they are different. Next, we randomly select subset of the possible object pairs to create the stimuli such that each unique object at least one pair. For the RMTS dataset, we repeat this process to select same and different pairs of pairs.
Fach object resized (from 224 x 244 pixel masks of the object's shape) such that it contained within single ViT patch for B/32 models four ViT patches for B/16 & B/14 models. For B/32 and
15

---

B/16 models, objects are roughly 28 x 28 pixels in size; for B/14 models (DINOv2 only), objects are roughly 21 21 pixels in size. These choices size mean that a single object can be placed in the center of 32 x 32 (or 28 x 28) pixel patch with radius of 4 pixels of extra space around it. This extra space allows us to randomly jitter object positions within the ViT patches.
To create stimulus, a pair of objects is placed over a 224 x 224 pixel white background in randomly selected, non-overlapping positions such that objects are aligned with ViT patches. For RMTS stimuli, the second "display" pair is always placed in the top left corner of the image. Each object's position (including the "display" objects for RMTS) is also randomly jittered within the ViT patches it occupies. We consider two objects in specific placement as one unique stimulus in other words, given pair of objects may appear multiple images but different positions. All object pairs appear the same number of times to ensure that each unique object is equally represented.
See Figure 8 for more examples of stimuli from each task.
Same Different
Figure 9: Examples of stimuli from our photorealistic same-different evaluation dataset. The top row contains "different" examples, while the bottom row contains "same" examples. Stimuli are constructed using 16 unique 3D models of objects placed on a table with a randomized texture; background textures are also randomized. Objects are randomly rotated and may placed at different distances from the camera or occlude each other.
A.3 Photorealistic Test Set
In order ensure the robustness of the two-stage processing observe in CLIP and DINOv2 artificial stimuli, we test models on a highly out-of-distrio df-triiiiiiiitiionn photorealistic discrimination task. The test dataset consists of 1, 022 photorealistic same-different stimuli that we generated (see Figure ( Each stimulus is 224x224 pixel image depicting a pair of same or different 3D objects arranged on the surface of a table in a sunlit room. We created these images in Blender, sophisticated 3D modeling tool, using a set of 16 unique 3D models of different objects that vary in shape, texture and color. To construct the dataset, we first generate all possible pairs of same different objects, then select a subset of the possible "different" pairs such that cach object appears in two pairs. This ensures that all objects are equally represented and that equal number of "same" and "different" stimuli are created. We create 32 unique stimuli for each pair of objects by placing them on the table in eight random configurations within the view of four different camera angles, allowing partial occlusions. Each individual object is also randomly rotated around its z-axis in each image because 11 of the objects lack rotational symmetry, these rotations provide an additional challenge, especially for "same" classifications.
We evaluate models that have been fine-tuned the diserimination task from the main body of the paper (e.g. Figure (h) in a zero-shot manner on the photorealistic dataset, meaning that there is no additional fine-tuning on the photorealistic dataset. We find that CLIP VIT attains a test accuracy of 93.9% on the photorealistic dataset, while all other models attain chance level accuracy (c.g. DINOv2 attains an accuracy of 48%). We also find that CLIP performs two-stage processing on the photorealistic stimuli (see Figure 10 and that the peaks in WO, WP. and BG attention all occur the same exact layers as the artificial stimuli (i.e. in Figure 2). DINOv2 also displays similar two-stage processing despite its poor performance on the photorealistic task (see Figure [0b). Note that BG attention for both models is higher overall during the perceptual stage when processing the photorealistic stimuli compared to the artificial stimuli; this is likely because the photorealistic stimuli contain detailed backgrounds, while the backgrounds in the artificial stimuli are blank. Overall, these findings generalize our results from the toy setting presented in the main body of the paper.
16

---

(a) CLIP: Realistic Stimuli
(b) DINOv2: Realistic Stimuli
0.75
WO
WP
0.5
0.25
0
2
3 4
7
8
9
10
10
11
Figure 10: Attention pattern analysis for CLIP and DINOv2 the photorealistic discrimination task. This figure follows the top row in Figure (a) CLIP: As in Figure WO peaks at layer WP peaks at layer 6, and BG peaks at layer 10. BG attention is higher throughout the perceptual stage, leading to lower perceptual score compared to the artificial discrimination task (i.e. fewer blue cells). (b) DINOv2: The attention pattern exhibits two stages, resembling the artificial setting (although the correspondence is somewhat looser than CLIPP', perhaps explaining DINOv2's poor zero-shot performance the photorealistic task).
B VIT B/16: All Model Behavioral Results
5 for behavioral results from all ViT-B/16 models trained on discrimination and RMTS tasks with either all 256 shape-color combinations only 32 shape-color combinations. The "Pretraining Scale" column denotes the number of images (in millions) in a given model's pretraining dataset. The models organized in descending order by pretraining scale. "Test Acc." refers to IID test accuracy. "Comp. Acc." refers to compositional generalization accuracy (for models trained on only 32 shape-color combinations). "Realistic Acc." (Table Taableooolyy only refers model's zero-shot accuracy on the photorealistic evaluation dataset. CLIP and DINOv2-the two models with pretraining scale on the order of 100 million images-atee attain near perfect accuracy on the RMTS task. However, only CLIP attains high performance on the photorealistic dataset.
See Tables
Pretrain
Pretraining Scale &
Train Ace.
Test Acc.
Realistic Ace.
CLIP
400M
100%
99.3%
93.9%
DINOv2
142M
100%
99.5%
48.0%
ImageNet
14.2M
100%
97.5%
53,0%
DINO
1.28M
100%
95.6%
50.9%
MAE
1.28M
100%
98.0%
52.4%
95.9%
80.5%
49.9%
Table 2: All behavioral results for ViT-B/16 models trained all 256 shape-color combinations on the discrimination task.
Pretrain
Pretraining Scale
Train Acc.
Test Acc.
Comp. Acc.
CLIP
400M
98.1%
98.5%
98.5%
DINOv2
142M
99.6%
98.5%
98.5%
ImageNet
14.2M
98.1% 18.1%
95.7%
95.7%
DINO
1.28M
98. %
92.3%
94.7%
MAE
1.28M
98. 1 %
94.9%
94.9%
77.3%
76.5%
76.0%
Table 3: All behavioral results for ViT-B/16 models trained the discrimination task.
32 shape-color combinations
17

---

Pretrain
Pretraining Scale
Train Acc.
Test Acc.
CLIP
400M
100%
98.3%
DINOv2
142M
100%
98.2%
ImageNet
14.2M
99.7%
89.3%
DINO
1.28M
100%
87.7%
MAE
1.28M
100%
93,4%
49.2%
50.1%
Table 4: All behavioral results for ViT-B/16 models trained on all 256 shape-color combinations the RMTS task.
Pretrain
Pretraining Scale +
Train Acc.
Test Acc.
Comp. Acc.
CLIP
400M
100%
98.0%
98.3%
DINOv2
142M
100%
96.4%
94.7%
ImageNet
14.2M
99.5%
92.3%
84.0%
DINO
1.28M
99.6%
94.7%
85.2%
MAE
1.28M
99.6%
85.3%
85.3%
49.2%
50.0%
50.0%
Table All behavioral results for ViT-B/16 models trained the RMTS task.
32 shape-color combinations on
C CLIP ViT-b32 Model Analyses
C.1 Attention Pattern Analysis
See Figure
C.2
Perceptual Stage Analysis
See Figure 12
(a) CLIP-B/32 Discrimination
(b) CLIP-B/32 RMTS
10
0.75
WO
WP
BG
0.25
WO
WP
BP
2
3
10
11
10
11
Figure 11: CLIP B/32 attention pattern analysis. See the caption of Figure 2 for figure and legend descriptions. The B/32 model follows the same stages of processing as CLIP VIT-B/16, and WO & WP peak at the same layers (3 and for discrimination respectively: and for RMTS respectively). However, WO attention remains high for longer than B/16 models.
18

---

Counterfactual Interventions: CLIP B/32
Disc. Shape
RMTS Shape
Intervention Acc.
0.8
Disc. Color
RMTS Color
0.6
0.4
0.2
7
8
9
10
11
Model Layer
Figure 12: CLIP B/32 DAS results.
Added
Interpolated
0.8
Shape Color
Shape
Intervention Acc.
0.6
Color
0.4
0.2
0
Sampled
Random
0.8
Shape
Shape
0.6
Color
Color
0.4
0.2
0
1
2
7
8
10
11
0
2
3
4
5
6
8
9
10
11
Model Layer
Figure 13: CLIP B/32 relational stage analysis: Novel Representations.
C.3
Relational Stage Analysis
See Figure 1 for novel representations analysis. See Figure 14 for linear intervention analysis. We find broadly similar results as CLIP B/16.
C.4
Generalization Results
We present CLIP-B/32 model results for models finetuned on all shape-color combinations, well as only 32 shape-color combinations, as Section 6. We present compositional generalization accuracy
Probe Acc.
Intervention
Control
0.75
Accuracy
0.5
0.25
10
Model Layer
Figure 14: CLIP B/32 relational stage analysis: Linear Intervention.
19

---

Task
# Combinations Seen
Train Ace.
IID Test Ace.
Comp. Gen. Ace.
OOD Acc.
Disc.
All
100%
99.6%
N/A
95.8%
Disc.
32
99.7%
98.5%
98.5%
98.0%
RMTS
All
100%
97.4%
N/A
90.5%
RMTS
32
100%
98.3%
98.3%
86.2%
Table 6: All behavioral results for CLIP-B/32 models.
(a) DINOv2 Discrimination
(b) DINOv2 RMTS
10
0.75
WO
0.5
WP
0.25
WO
WP
BP
BG
2
5
6 7 8
11 0
2
3
Model Layer
Figure 15: DINOv2 attention pattern analysis. See the caption of Figure 2 for figure and legend descriptions. Note that the stars in the line charts are placed differently in this figure compared other attention pattern analysis figures. Instead of marking the maximal values of each type of attention across all 12 layers, the stars mark the maximal value excluding the Oth layer. This is because all types of attention spike in DINOv2 in the Oth layer.
(when applicable) well OOD generalization accuracy. We find that all models perform quite well in-distribution and under compositional generalization. Accuracy drops somewhat for RMTS OOD stimuli. All results are in presented in Table [
D DINOv2 Analyses
D.1 Attention Map Analysis
See Figure 15 for DINOv2 attention pattern analyses. Like CLIP, DINOv2 displays two-stage processing (albeit somewhat less cleanly). One notable difference compared to CLIP is that all types of attention (WO. WP, BP, and BG) spike in the Oth layer. This might be related to DINOv2's positional encodings. Since the model was pretrained on images with a size of 518 x 518 pixels, the model's 's positional encodings interpolated to process artifact in the attention patterns the very beginning of the model. Disregarding this spike, the stages of processing follow CLIP. In the discrimination task (Figure 15 R), within-object attention peaks at layer (disregarding the initial peak), followed by within-pair and finally background attention. In the RMTS task (Figure 15p), within-object attention peaks at layer 3, followed by within-pair attention at layer 8, and finally between-pair attention in the final layer. Background attention remains relatively high throughout the model, indicating that DINOv2 might make greater use of register tokens to solve the RMTS task compared to other models.
224 224 stimuli; this might cause
D.2 Perceptual Stage Analysis
See Figure 16 for perceptual stage analysis of DINOVV-pretrained ViTs. Overall, we find highly disentangled object representations in these models.
20

---

Counterfactual Interventions: DINOv2 B/14
Disc. Shape
RMTS Shape
Intervention Acc.
0.8
Disc. Color
RMTS Color
0.6
0.4
0.2
10
11
Model Layer
Figure 16: DAS results for DINOv2 VIT-B/14.
Novel Representation Analysis: DINOv2 B/14 (Disc.)
Added
Interpolated
0.8
Shape
Shape
Intervention Acc.
Color
Color
0.4
0.2
Sampled
Random
0.8
Shape
Shape
0.6
Color
Color
0.4
0.2
3
4
10
11
2
3
4
10
11
Model Layer
Figure 17: Novel Representation Analysis for DINOv2 ViT-B/14 (Disc.).
D.3
Relational Stage Analysis
D.3.1
Novel Representation Analysis
See Figure 17 and T8 for novel representation analysis of DINOV2-pretrained ViTs for the discrimi- nation and RMTS tasks. These results replicate those found using CLIP-pretrained VITs.
D.3.2
Abstract Representations of Same and Different
See Figure 19 for linear probing and intervention results for DINOV2-pretrained VITs. We find that the intervention works extremely well for these models, replicating our results on CLIP -rer-prerr ViTs.
E Attention Pattern Analyses
See Figure attention pattern analyses for ImageNet, DINO, and MAE VIT on the Diserimination and RMTS tasks. ImageNet loosely demonstrates two-stage processing like CLIP and DINOv2. the other hand, DINO and MAE do not display two stage processing; instead, local and global processing appears to be mixed throughout the models. DINO and MAE also receive the smallest scale pretraining compared to the other models (see the Pretraining Scale column in Table this provides further support for our intuition that pretraining scale results in two-stage processing.
21

---

Novel Representation Analysis: DINOv2 B/14 (RMTS)
Added
Interpolated
0.8
Shape
Shape
Intervention Acc.
0.6
Color
Color
0.4
0.2
0
Sampled
Random
0.8
Shape
Shape
0.6
Color
Color
0.4
0.2
0
0
8
11
Model Layer
Figure 18: Novel Representation Analysis for DINOy2 ViT-B/14 (RMTS).
Abstract Representations of Same and Different: DINOv2 B/14
Probe Acc.
Intervention
Control
0.75
Accuracy
0.5
0.25
3
7
11
Model Layer
Figure Linear probe & intervention analysis for DINOv2 ViT-B/14.
F
Two Internal Algorithms Examined in Greater Detail
While the attention head analysis in Section shows that different models use qualitatively different internal algorithms to solve same-different tasks, the specific computations involved in these algo- rithms are less clear. What exactly is happening during CLIP's perceptual stage, for example? In this section, we seek to build intuitive picture of the algorithms learned by two models the discrimination task: CLIP VTT- and randomly initialized ViT-B/16 (From Scratch).
To do this, examine the attention patterns produced by individual attention heads throughout 2T displays attention patterns extracted from four randomly selected individual attention heads (black & white heatmaps) in response to the input image on the left. For CLIP, the examined heads are: layer 1, head (local head); layer 5, head 9 (local head); layer 6, head 11 (global head); and layer 10, head 6 (global head). For the from scratch model, the heads are: layer head 8; layer 5, head 11; layer 6, head 3; and layer 10, head 8. For visualization purposes, the attention patterns are truncated include only the indices of the two objects' tokens; since each object occupies four ViT patches, this results in an 8 x grid for each attention head. The sre axis (y-axis) in Figure 21 indicates the source token, while the dest axis (æ-axis) indicates the destination token (attention flows from sre ddec). The actual tokens in the input image are also visualized along these axes.
each model. F
Based on these attention patterns, we visualize how CLIP processes an image solve the discrim- ination task in Figure 22 1. Embedding: The model first tokenizes the image and embeds these
22

---

ImageNet Discrimination
(b) DINO Discrimination
(c) MAE Discrimination
Globol
0.75
0.75
Atfention Head Score
0.25
(d) Image. Net RMTS
(b) DINO RMTS
(1 MAE RMTS
0.5
0.25
0.75
0.25
WO
Model
Figure 20: ImageNet, DINO, and MAE VIT attention pattern analysis. See the caption of Figure for figure and legend descriptions. Like CLIP and DINOv2, ImageNet ViT displays two-stage processing on both the discrimination and RMTS tasks; however, performance of this model lags behind CLIP and DINOv2, possibly due to smaller pretraining scale (see the Pretraining Scale column in Table DINO and MAE do not display two-stage processing. These two models are also pretrained on the smallest amount of data, further supporting our intuition that pretraining scale rather than objective results two-stage processing.
tokens. Each object occupies four ViT-B/16 patches, so the objects are divided up into four tokens each. 2. Layer 1, Head 5: During the early perceptual stage, the local attention heads appear aid the formation of object representations by performing low-level comparisons within objects. For example, head 5 in layer compares object tokens from left to right within each object. Other attention heads in this layer perform such comparisons in other directions, such as right to left or top bottom. 3. Layer Head 9: Towards the end of the perceptual stage, all object tokens within single object attend to all other tokens within the same object. The four object tokens comprising each object have been pushed together in the latent space, and the model now "sees" single object as whole. 4. Layer 6, Head 11: The model switches from predominantly local to predominantly global attention in this layer, and within-pair (WP) attention peaks. The whole-object representations formed during the perceptual stage now attend to each other, indicating that the model is comparing them. 5. Layer 10, Head The model appears to utilize object tokens (and background tokens) to store information, possibly the classification decision).
23

---

CLIP Discrimination
Global
Head
Layer B.edd
Lave Head
Layer 6 Head 11
Layer Fled
Input
Attention Head Score
src
dest
b) From Scratch Discrimination
Head
Layer [aad
Layer
Layar' Fload
Layur Fladd
Input
Local
dest
Figure 21: Example attention head patterns for models trained on the discrimination task. (a) CLIP ViT-B/16: On the left is an example input image, which is fed into the model. The heatmap the same Figure 2 the x and yyaxes denote model layer and head index respectively, and the colors indicate the type of attention head as defined in Section 3 (local heads-blue, global heads=red). The attention patterns of four attention heads for this input image are displayed black & white heatmaps below; white indicates higher attention values. The src axis indicates the source token, which is visually marked-recee recall that each object occupies four tokens each. The dest axis indicates the destination token. Individual objects attend to themselves during the perceptual stage (layers 0-5); objects begin to attend to the other object during the relational stage (layer onwards). (b) From Scratch ViT-B/16: The analysis in (a) is repeated for from scratch model trained on discrimination. The attention patterns are less interpretable throughout.
G Distributed Alignment Search Technical Details
Approach We apply form of distributed alignment search (Geiger al. 2024) in order assess whether the object representations formed by the perceptual stage of ViTss are disentangled with respect to shape and color (the two of variation present in our dataset). For ViT B/32 models, each object is contained within the bounds single patch, making DAS straightforward to run: we train an intervention over model representations corresponding the patches containing individual objects that we wish to use as source and base tokens. For ViT B/16 models, each object is contained in four patches. Here, train a single intervention that is shared between all four patches comprising the base and source objects. Importantly, because we wish to isolate whole objecl representations, rather than patch representations, we randomly shuffle the patches comprising the source object before patching information into the base object. For example, the top-right patch of the base object might be injected with information from the bottom-left patch of the source object. This intervention should only succeed if the model contains disentangled representation of the whole object, and if this representation present in all four patches comprising that object. Given these stringent conditions, it is all the more surprising that DAS succeeds. During test, we intervene in patch-aligned manner: The vector patched into the top-right corner of the base image representation extracted from the top-right corner of the source image.
Data To train the DAS intervention, we must generate counterfactual datasets for every subspace that we wish to isolate. To generate a discrimination dataset that will be used to identify a color subspace, for example, we find examples in the model's training set where objects only differ along the color dimension (c.g., ob ject, expresses colori, ob ject2 expressed colorg. We randomly
24

---

Perceptual Stage
Relational Stage
1. Embedding
2. Layer 1, Head 5
3. Layer 5, Head 9
4. Layer 6, Head 11
5. Layer 10, Head 6
Figure 22: How CLIP ViT-B/16 processes an example from the discrimination task. Four attention heads are randomly selected from different stages in CLIP and analyzed on single input image (see Figure 2TD. Embedding: The model first tokenizes the input image. Each object occupies four VIT patches. 2. Layer 1, Head 5: During the perceptual stage, the model first performs low-level visual operations between tokens of individual objects. This particular attention head performs left-to-right attention within objects. 3. Layer 5, Head 9; Near the end of the perceptual stage, whole-object representations have been formed. 4. Layer 6, Head 11: During the relational stage, the whole-object representations are compared. 5. Layer 10, Head 6: Object and background tokens are used as registers store information cotimmatioo-rrsumaabl the classification.
select one object to intervene and generate a counterfactual image. WLOG consider intervening on ob jejett. Our counterfactual image contains one object (the counterfactual object) that expresses colorg. Our intervention is optimized to extract color information from the counterfactual object and inject it into object,, changing the model's 's overall discrimination judgment from different to same. Importantly, the counterfactual image label is also different. Thus, our intervention designed to work only if the intervention transfers color information. We follow similar procedure for to generate counterfactuals that can be used turn same image into a different image. this case, both base and counterfactual images are labelled same, but the counterfactual same image contains objects that are a different color than those the base image. The counterfactual color is patched into one of the objects in the base image, rendering the objects in the base image different along the color axis.
For the RMTS DAS dataset, we generate counterfactuals similarly to the discrimination dataset. We select pair of objects randomly (except either the display pair or sample pair). We then choose the source object in the other pair. We edit the color or shape property of just this source object, and use this as the counterfactual. For these datasets, the data balanced such that 50% of overall labels are changed from same to different, but also 50% of intermediate pair labels are changed from same to different. Note that flipping one intermediate label necessarily flips the hierarchical label. Thus, if the source object is in pair expressing the same relationship. then the counterfactual image will have the opposite label as the base image before intervention. could succeed by transferring the hierarchical image label, rather than by transferring particular color or shape properties from one object to another. However, this only occurs approximately 50% of the time. That is because it occurs in 100% of samples when both pairs exhibit same, which occurs 25% the time (half of the hierarchical same images), and 50% of the time when one pair exhibits same and the other exhibits different, which occurs 50% of the time (all of the hierarchical different images). However, this strategy provides exactly the incorrect incorrect result in the other 50% of cases. Nonetheless, this behavior might explain why RMTS DAS results maintain around 50% deeper into the model.
these cases, the intervention
For all datasets, we generate train counterfactual pairs from the model train split, validation pairs from the validation split, and test pairs from the model test split. We generate 2,000 counterfactual pairs for both splits. Note that in the case of models trained in the compositional generalization
25

---

(a) Discrimination: Color counterfactual pair. The brown color from the object on the right will be patched into the orange color in the object the left.
x +
H
H
(b) RMTS: Color counterfactual pair. The dark blue color from the display pair object on the right will be patched into one of the green sample objects on the left.
(c) Discrimination: Shape counterfactual pair. The two-circles shape from the object the right will be patched into cross shape in the object on left.
M
(d) RMTS: Shape counterfactual pair. The thin star shape from the sample pair object on the right will be patched into one of the orange objects on the left.
Figure 23: Counterfactual pairs used to train DAS interventions.
experiments (í.c. those found in Section 6), the counterfactual image may contain shape-color pairs that were observed during training. However, training our interventions has no bearing on the model's downstream performance on held-out data, though correlation between disentanglement and compositional generalization is thus extremely surprising. See Figure 23 for examples of counterfactual pairs used to train interventions.
Intervention Details DAS requires optimizing 1) rotation matrix over representations and 2) some means of identifying appropriate dimensions over which intervene (Geiger al. 2024). Prior work has largely heuristically selected contiguous subspaces over which to intervene (Geiger et al.)
26

---

Counterfactual Intervention Controls: CLIP B/16
Base
Source
Intervention Aec.
Disc. Shape
0.8
Disc. Color
0.6
RMIS Shape
0.4
RMIS Color
0.2
6 Model Layer
7
9
10
(a)
(b)
Figure 24: Controls for DAS Analysis (Section (a) We intervene on objects using the irrelevant source token in the counterfactual stimuli. For example, if the DAS interventions are meaningful, patching a bowtie shape into the base image in the top row should not change the model's decision to "same." (b) The controls fail to flip CLIP ViT's decisions at a rate above chance accuracy, indicating that the DAS results presented Section are indeed the result of meaningful interventions.
2024 Wu et al. 2024c). In this work, we relax this heuristic, identifying dimensions by optimizing binary mask over model representations as we optimize the rotation matrix (Wu et al. 2024a). We follow best practices from differentiable pruning methods like continuous sparsification (Savarese) ct al. 2020), annealing a sigmoid mask into a binary mask over the course of training, using an exponential temperature scheduler. We also introduce an Lo penalty encourage sparse masking. We use default parameters suggested by the pyvene library for Boundless DAS, another DAS method that optimizes the dimensions over which to intervne. Our rotation matrix learning rate is 0.001, mask learning rate is 0.01, and we train for 20 epochs for each subspace, independently for each model layer. We add scalar multiplier of 0.001 to our Lo loss term, which balances the magnitude of Lo loss with the normal cross entropy loss that we are computing to optimize the intervention. Our temperature is annealed down 0.005 over the course of training, and then snapped to binary during testing. Finally, we optimize our interventions using the Adam optimizer (Kingma & Ba 2014). These parameters reflect standard practice for differentiable masking for interpretability (Lepori et 2023b).
H Perceptual Stage Analysis Controls
As control for our DAS analysis presented in Section we attempt to intervene using the incorrect source token in the counterfactual image. this intervention fails, then it provides evidence that the information transferred in the standard DAS experiment is actually indicative of disentangled local object representations, rather than information that may be distributed across all objects. We note that this control could succeed at flipping same judgments to different, but will completely fail in the opposite direction. As shown in Figure 24] these controls do reliably fail achieve above-chance counterfactual intervention accuracy.
I Perceptual Stage Analysis: Other Models
26 27 and 28 for DINO, ImageNet, MAE, and From Scratch DAS results. We see See Figures 25
that models broadly exhibit less disentanglement than CLIP and DINOv2.
J Novel Representations Analysis Technical Details
During the novel representations analysis of Section 5 we patch in novel vectors into the subspaces identified using DAS. Notably, we must patch into the [color, shape) subspace for both objects that we wish to intervene on, rather than just one. This is because we want to analyze whether the same-different relation can generalize to novel representations. For example, if two objects in discrimination example share the same shape, but one is blue and one is red, we would like know whether intervene make the color property of each object an identical, novel vector, such
27

---

Counterfactual Interventions: DINO B/16
Disc. Shape
Intervention Acc.
0.8
Disc. Color
0.6
RMTS Shape
RMTS Color
0.4
0.2
2
3
6
7
8
9
10
11
Model Layer
Figure 25: DAS results for DINO ViT-B/16.
Counterfactual Interventions: ImageNet B/16
Disc. Shape
Intervention Acc.
0.8
Disc. Color
0.6
RMTS Shape
RMTS Color
0.4
0.2
2
3
4
6
7
8
9
10
11
Model Layer
Figure 26: DAS results for ImageNet ViT-B/16.
that the model's decision will flip from different to same. We run this analysis on the IID test set of the DAS data.
We create these vectors using four different methods. For these methods, we first save the embeddings found in the subspaces identified by DAS for all images in the DAS Validation
1. Addition: We sample two objects in the validation set, and add their subspace embeddings. For ViT-B/16, we patch the resulting vector in patch-aligned manner: The vector patched into the top-right corner of the base image representation is generated by adding the top-right corners of each embedding found within the subspace of the sampled validation images.
2. Interpolation: Same Method except vectors are averaged dimension-wise
3. Sampled: We form one Gaussian distribution per embedding dimension using our saved validation embeddings. We independently sample from these distributions generate vector that is patched into the base image. This single vector is patched into all four object patches for ViT-B/16.
4. Random Gaussian: We randomly sample from a normal distribution with mean 0 and standard deviation and patch that into the base image. This single vector patched into all four object patches for ViT-B/16.
K
Relational Stage Analysis: Further Results
K.1
CLIP B/16 RMTS Novel Representation Analysis
See Figure 29 for novel representation analysis on CLIP B/16, finetuned for the relational match to sample task.
28

---

Counterfactual Interventions: MAE B/16
Disc. Shape
RMTS Shape
Intervention Acc.
0.8
Disc. Color
RMTS Color
0.6
0.4
0.2
5
Model Layer
Figure 27: DAS results for MAE ViT-B/16.
Counterfactual Interventions: From Scratch B/16
1
Disc. Shape
Intervention Acc.
0.8
Disc. Color
0.6
0.4
10
11
Model Layer
Figure 28: DAS results for From Scratch ViT-B/16.
K.2 Novel Representation Analysis: Other Models
See Figures 30 31
34
36 for DINO Discrimination/MMTS., ImageNet Discrimina-
tion/RMTS, MAE Discrimination/RMTTT and From Scratch Discrimination Novel Representation Analysis results.
K.3 Abstract Representations of Same and Different
We run the linear probe and linear intervention analysis from Section on DINO B/16, ImageNet B/16, and MAE B/16. We find that the intervention works much less well these models than DINOv2 B/16, CLIP B/16 or CLIP B/32. This indicates that these models are not using one abstract representation of same and one representation of different that agnostic perceptual qualities of the input image.
Additionally, we try scale the directions that adding the intermediate representations by 0.5 and 2.0 for DINO and ImageNet pretrained models, and find that neither of these versions of the intervention work much better well for either model. See Figure 37 and 38 for DINO and ImageNet results. See Figure 39 for MAE model results.
L Pipeline Loss Technical Details
To instill two-stage processing in ViTs trained from scratch on discrimination, we shape the model's attention patterns in different ways at different layers. In particular, within-object attention is encouraged in layers 3, 4, and 5, while between-object attention is encouraged layers and (roughly following CLIP's two-stage processing; see Figure For models trained on RMTS, we additionally encourage between-pair attention layers and (see Figure 2b). Finally, whenever we add the disentanglement loss, it is computed in layer only.
29

---

Novel Representation Analysis: CLIP B/16 (RMTS)
Added
Interpolated
0.8
Shape
Shape
Intervention Acc.
0.6
Color
Color
0.2
0-
Sampled
Random
0.8
Shape
Shape
0.6
Color
Color
0.4
4
5
11
5
Model Layer
Figure 29: Novel Representation Analysis for CLIP ViT-B/16 (RMTS).
Novel Representation Analysis: DINO B/16 (Discrimination)
Added
Interpolated
0.8
Shape
Shape
Intervention Acc.
0.6
Color
Color
0.4
0.2
0
Sampled
Random
0.8
Shape
Shape
0.6
Color
Color
0.4
0.2
2
3
11
5
Model Layer
Figure 30: Novel Representation Analysis for DINO VIT-B/16 (Disc.).
30

---

Novel Representation Analysis: DINO B/16 (RMTS)
Added
Interpolated
0.8
Shape
Shape
Intervention Acc.
0.6
Color
Color
0.4
0.2
0
Sampled
Random
0.8
Shape
Shape
0.6
Color
Color
0.4
0.2
2
3
4
11
5
6
7
8
10
Model Layer
Figure 31: Novel Representation Analysis for DINO ViT-B/16 (RMTS).
Novel Representation Analysis: ImageNet B/16 (Disc.)
Added
Interpolated
0.8
Shape
Shape
Intervention Acc.
0.6
Color
Color
0.4
0.2
0
Sampled
Random
0.8
Shape
Shape
0.6
Color
Color
0.4
0.2
2
3
4
11
5
6
7
Model Layer
Figure 32: Novel Representation Analysis for ImageNet VIT-B/16 (Disc.).
31

---

Novel Representation Analysis: ImageNet B/16 (RMTS)
Added
Interpolated
0.8
Shape
Shape
Intervention Acc.
Color
Color
0.4
0.2
Sampled
Random
0.8
Shape
Shape
0.6
Color
Color
0.2
2
3
4
5
6
10
11
2
3
4
5
6
7
10
11
Model Layer
Figure 33: Novel Representation Analysis for ImageNet ViT-B/16 (RMTS).
Novel Representation Analysis: MAE B/16 (Disc.)
Added
Interpolated
0.8
Shape
Shape
Intervention Acc.
0.6
Color
Color
0.4
0.2
0
Sampled
Random
0.8
Shape
Shape
0.6
Color
Color
0.4
0.2
0
1
7
8
10
5
10
11
Model Layer
Figure 34: Novel Representation Analysis for MAE VIT-B/16 (Disc.).
To encourage particular type of attention pattern in given layer, we first compute the attention head scores (according to Section for a randomly selected subset either 4, 6, or 8 attention heads in that layer These scores then averaged across the layer. The average attention head score is subtracted from 1, which the maximum possible score for a given attention type (i.e. WO, WP, and BP following Figure 2). This difference averaged across model layers is the pipeline loss term. In the case of within-object attention, an average score of means that each attention head in the selected subset only attends between object tokens within the same object; other tokens attend to each other. Thus, using the difference between 1 and the current attention head scores as the loss signal encourages the attention heads assign stronger attention between tokens within objects and weaker attention between all other tokens. The same holds for WP and BP attention. However, the particular forms of the attention patterns WO attention score in given layer, models could learn to assign 100% of their attention between two object tokens only (instead of between all four tokens), or from a single object token itself. This
not constrained; for example, in order maximize the
Thiis selection is kept constant throughout training: i.e., the attention heads that receive the pipeline loss signal are randomly chosen before training but do not change throughout training.
32

---

Novel Representation Analysis: MAE B/16 (RMTS)
Added
Interpolated
0.8
Shape
Shape
Intervention Acc.
0.6
Color
Color
0.2
0
Sampled
Random
Shape
Shape
0.6
Color
Color
0.4
0.2
3
4
5
6
9
10
11
2
3
4
9 10 11
Model Layer
Figure 35: Novel Representation Analysis for MAE ViT-B/16 (RMTS).
Novel Representation Analysis: From Scratch B/16 (Disc.)
Added
Interpolated
0.8
Shape
Shape
Intervention Acc.
0.6
Color
Color
0.4
0.2
Sampled
Random
0.8
Shape
Shape
0.6
Color
Color
0.4
0.2
3
4
6
10
11
5
6
10
11
Model Layer
Figure 36: Novel Representation Analysis for From Seratch VIT-B/16 (Disc.).
33

---

Abstract Representations of Same and Different: DINO B/16
Probe Acc.
Intervention
Control
a 0.5
Accuracy
0.5
=
Accuracy
0.5
=
0.75
Accuracy
0.5
0.25
11
Model Layer
Figure 37: Scaled linear probe & intervention analysis for DINO VIT-B/16.
flexibility is inspired by the analysis in Figure which finds that within-object attention patterns can take many different configurations that might serve different purposes in the formation object representations. The same is true for WP and BP patterns.
M Auxiliary Loss Ablations
In this section. we present ablations of the different auxiliary loss functions presented in Section 7 Notably, the pipeline loss consists of These components correspond to the processing stages that they attempt to induce within-objec processing (WO), within-pair processing (WP), and between- setveen--airraa processing (BP), For discrimination. we find that either WO or WP losses confer a benefit, but that including both results in the best performance.
three modular components, depending on the task.
For RMTS, we find that including all loss functions once again confers the greatest performance benefit. Notably, find that ablating either the disentanglement loss or the WP loss completely destroys RMTS performance, whereas ablating WO loss results in a fairly minor drop in performance.
N
Compute Resources
employed compute resources at a large academic institution. We scheduled jobs with SLURM. Finetuning models on these relational reasoning tasks using geforce3090 GPUs required approx- imately 200 GPU-hours of model training. Running DAS over each layer in a model required approximately 250 GPU-hours. The remaining analyses took considerably less time, approximately
34

---

Abstract Representations Same and Different: ImageNet B/16
Probe Acc.
Intervention
Control
0.5
0.75
Accuracy
0.5
0.25
0
a =
0.75
Accuracy
0.5
0.25
a 2
0.75
Accuracy
0.25
2
10
Model Layer
Figure 38: Scaled linear probe & intervention analysis for ImageNet ViT-B/16.
Abstract Representations Same and Different: MAE B/16
Probe Acc.
Intervention
Control
0.75
Accuracy
0.5
0.25
2
5
8
10
11
Model Layer
Figure 39: Linear probe & intervention analysis for MAE VIT-B/16.
35

---

WO Loss
WP Loss
Train Acc.
Test Acc.
Comp. Acc.
77.3%
76.5%
75.9%
91.6%
88.8%
84.5%
93.0%
91.1%
87.5%
95.6%
(+18.3)
93.9%
(+17.4)
92.3% (+16.4)
Table 7: Performance of ViT-B/16 trained from scratch on the discrimination task with auxiliary losses.
Disent. Loss
WO Loss
WP Loss
BP Loss
Test Acc.
Comp. Acc.
50.1%
50.0%
49.4%
50.7%
49.3%
51.3%
50.0%
51.3%
48.9%
50.0%
85.6%
68.7%
50.0%
50.1%
51.0%
51.2%
61.3%
50.8%
83.2%
68.8%
49.8%
50.9%
49.9%
50.5%
51%
50.8%
87.7%
76.5%
50. %
50.1%
91.4%
77.4%
Table 8: Performance of VITs trained from scratch on RMTS with auxiliary losses.
50 GPU-hours in total. Preliminary analysis and hyperparameter tuning took considerably more time, approximately 2,000 GPU-hours in total. The full research project required approximately 2,500 GPU-hours.
36

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section D
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section g
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
37

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Sections
5 Appendices G
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
38

---

Answer: [Yes]
Justification: We link to an anonymized repository containing our code.
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https) //nips. publiclle i gu des CodeCooSuuaaa iss onPol for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Sections 4 5 Appendices G
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our results primarily concern analyses conducted on finetuned versions of pretrained models. We present model analyses separately as not to lose information through aggregation, at the expense of reporting statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
39

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
N
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://pw.eeeeee../o/////aacaaaa
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work centers around issues at the intersection of cognitive science and deep learning. This work does contribute toward practical algorithms or techniques that will see broad deployment.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
40

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The models used in this paper are standard vision transformers finetuned on simple synthetic task. There are no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papersswa.. www.osocittcceeeeeoo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
41

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
42


Reviews:
[{"review_id": "KHYUKKCirw", "reviewer": "Reviewer_UKTX", "summary": "This work studies ViTs' learning behavior with relational tasks by experimenting on 2 same-different tasks: discrimination and RMTS tasks.  And the authors propose a dataset to analyze. And discovers that there are 2 stages of attention processing of CLIP ViTs by attention scores from patches to other patches. They characterize 2 stages, perceptual stage and relational stage.", "questions": "1. ln 23-24, the motivation \"little breakthrough progress on complex tasks involving relations between visual entities...\" is that true? What about the recent breakthroughs or datasets of vision-language models, Llava, GPT4, etc. Please give more explanations.\n\n2. ln 41, please define what are \"algorithms\" learned by ViT, and are there algorithms learned by other architectures, CNNs, LSTMs, etc.?\n\n3. Are there no existing datasets that can study the relations of visual entities? I mean in general, not the RMTS, etc.\n\n3. The motivation of using CLIP ViTs for study is not clear. Since part of the motivation is from infant learning, why not use more human-like CNN like ConvNext? Also, CLIP ViT is different from ViT as well.\n\n4. Vaguely remember [1] discussed about using better disentangled model for generalization and color, are there new insights from Sec 6?\n\n5. With the 2 stage processing, what would you suggest the architecture to change to build a more \"robust\" version?\n---------\n[1] Better may not be fairer? [Chiu, et. al., 2023]", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper defined a novel avenue with inspiration from mechanistic interpretability to understand the working mechanism of CLIP ViTs. And proposed approaches to study them with attention analyses.", "weaknesses": "1. I believe readers in this track would generally not be experts in concepts of the proposed area of studies, the paper should include more literature review for context upfront.\n\n2. I find the writing and the flow of the paper hard to follow, maybe adding flow-chart would help improve readability.\n\n3. The dataset is not well-discussed.\n\n4. I think there is a slight mismatch between claimed study and experiments. The title suggest the study centers around ViTs but use CLIP ViTs, leaving me confused if the points made are the same on the original ViTs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and questions. Find point by point responses below:\n\n**Weaknesses**:\n1. “...Literature review for context…”: We completely agree. Upon acceptance, we will use the extra space to expand our literature review section to include more insights from both the (vast) cognitive science literature and the mechanistic interpretability literature (see global rebuttal #1 for specific topics).\n\n2. “Flow of the paper”: We are happy to add more signposting throughout the sections. Roughly, the paper is structured such that Section 3 motivates the investigation of two stages of processing, Sections 4 and 5 respectively investigate these stages in greater detail, and Sections 6 and 7 discuss the implications of some of our findings on downstream performance. We are happy to state the structure of the paper in the introduction!\n\n3. “Dataset is not well discussed”: We have written a vastly extended appendix detailing the dataset and its construction, which we will include in the final submission. We are also happy to answer any questions about the dataset during the discussion.\n\n4. “CLIP ViTs”: In this work, we investigate abstract relations in a variety of ViTs. In particular, we study ViTs that are pretrained with different kinds of supervision, e.g. CLIP, DINO, and ImageNet. However, as CLIP is the only type of pretraining that achieves near perfect test accuracy for our tasks, we focus on it for the majority of the paper. Since submission, we have added DINOv2 to our analyses, finding that it solves both discrimination and Relational-Match-to-Sample (RMTS) tasks with near perfect test accuracy (99.5% test accuracy on discrimination and 98.2% on RMTS); importantly, it also exhibits two distinct stages of processing similar to those we observe in CLIP. This generalizes our findings beyond CLIP and bolsters our intuition that pretraining data scale (rather than something particular to CLIP) provides the visual representations needed to solve these tasks. We will include these results in a new appendix to the paper and reference them throughout the main body. \n\n**Questions**:\n1. “Little progress”: Great point—we should be more specific. We mean that across a variety of tightly controlled benchmarks, vision models tend to struggle with tasks that involve relations [3, 4], especially compared to tasks that involve semantic computations. Additionally, the models that you mention have demonstrated some progress in processing visual relations, but there are still shortcomings to be addressed. We are happy to add these references to the final manuscript in an updated related work section.\n\n2. “Algorithms”: This terminology is borrowed partially from the mechanistic interpretability literature (i.e. in conceptualizing neural networks as implementing interpretable algorithms), and partially from the cognitive science literature (i.e. Marr’s levels of analysis). When we use the word “algorithm”, we are referring to the series of representations generated by a model to solve a particular task.\n\n3. “Existing datasets”: There are plenty of datasets used to study visual relations! For example: SVRT [1] and CVRT [5]. However, these datasets are not controlled enough to enable us to adapt state-of-the-art techniques from language model mechanistic interpretability for ViTs. Our dataset is explicitly constructed to enable these techniques by e.g. consistently aligning objects within the bounds of ViT patches. Our new dataset appendix details our design choices.\n\n4. “Motivation of using CLIP ViT”: As mentioned above (weakness #4), we use many different ViTs. Your point about CNNs being more human-like is well taken! We use ViTs because prior work has demonstrated that they can solve same-different tasks in a robust fashion [2]; they also enable us to use techniques from mechanistic interpretability that have previously been developed for Transformer language models (DAS, attention analysis, linear probing-based causal interventions, logit lens). Since CNNs lack tokens, it is not clear whether it is possible to apply these techniques to them. \n\n5. “Disentangled model for generalization”: We are happy to include this reference in the main body of the paper. Notably, our work demonstrates the benefits of disentangled representations for compositional generalization (rather than standard generalization) in a vastly different setting!\n\n6. “Build a more robust version”: We were very motivated by this question as well! Since submission, we have implemented a new auxiliary loss function derived from the attention head scores in Section 3 that encourages the model to adopt two-stage processing. We find that this helps a randomly-initialized ViT achieve significantly better downstream performance on the discrimination task (76.5% to 93.9% test accuracy; +17.4). It also significantly boosts compositional generalization accuracy for discrimination (75.9% to 92.3% accuracy; +16.4). We currently have promising preliminary results using this loss to improve performance on the RMTS task and will comment on them when they are ready. This loss is also very general and could possibly be used in the future to improve relational reasoning for a wide variety of visual relations. We will include these results and more details about the loss in the main body of the paper upon submission.\n\n**Citations**:\n1. Fleuret, F., et al. (2011). Comparing machines and humans on a visual categorization test. \n2. Tartaglini, A. R., et al.  (2023). Deep neural networks can learn generalizable same-different visual relations. \n3. Thrush, T. et al. (2022). Winoground: Probing vision and language models for visio-linguistic compositionality. \n4. Zeng, Y., et al. (2024). Investigating Compositional Challenges in Vision-Language Models for Visual Grounding. \n5. Zerroug, A., et al. (2022). A benchmark for compositional visual reasoning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "84E2ypVqo5", "reviewer": "Reviewer_DaaT", "summary": "The paper uses techniques from mechanistic interpretrability to analyze the algorithms implemented by pretrained ViTs to solve abstract visual reasoning tasks. The authors use two synthetic same-different tasks: discrimination and relational match-to-sample (RMTS), to analyze CLIP pretrained ViTs, DINO pretrained and Imagenet pretrained ViTs, finetuned on these tasks, as well as a ViT trained from scratch. Some pretrained ViTs, especially CLIP pretrained demonstrated a strong perceptual stage in the early layers, disentangling object representations, followed by a relational stage in the later layers, which implements somewhat abstract same-different relations. The authors also demonstrate that the formation of only the perceptual stage is enough to solve simple relational tasks like discrimination, but not enough for RMTS.", "questions": "1. According to Fig 2a,c deeper layers don’t seem to be dominated by global heads as WO attention score is higher than WP?\n\n2. Analysis of some models are missing in Fig 2 (e.g DINO Discrimination and Imagenet pretrained ViTs).\n\n\n3. Can the authors also perform the same analysis for ViTs pretrained using masked autoencoding objective [1]?\n\n4. Are the results in Fig 6 averaged for all models?\n\n[1] - He, K., Chen, X., Xie, S., Li, Y., Dollár, P. and Girshick, R., 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16000-16009).", "limitations": "The authors have discussed the limitations in the last section.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and easy to follow.\n2. Interesting use of mechanistic interpretrability to analyze how pretrained ViTs solve same-different tasks: discrimination and relational match-to-sample (RMTS).\n3. The paper shows that early layers of CLIP pretrained ViTs demonstrate local attention for within object processing (perceptual stage), whereas the later layers demonstrate global attention for between object processing (relational stage). The two stage processing doesn’t form strongly for the other pretrained models (DINO and ImageNet) especially the relational stage. \n4. The perceptual stage is characterized by disentangled object representations (color and shape). Models trained from scratch can be enforced to have disentangled representations using an auxiliary loss which is enough for generalization in simple relational tasks like discrimination but not for RMTS task.", "weaknesses": "1. Analysis is limited to simple tasks consisting of simple relations. It remains to be seen if the results would hold up for more tasks involving higher order relations and more complicated relations.\n\n2. The paper doesn’t give a clear and detailed intuition behind why CLIP pretrained ViTs implement the two stage processing more strongly compared to other pretrained ViT models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review! Find point-by-point responses below:\n\n**Weaknesses**:\n1. “Simple relations”: This is fair! With respect to “higher order relations”, we attempted to explore this using the Relational-Match-to-Sample (RMTS) task—an explicitly hierarchical version of the discrimination task. While this goes beyond much of the same-different literature (which focuses on variations of the discrimination task), we do limit our investigation to the same-different relation (rather than other relations) due to its particular significance in the study of abstract concepts in the cognitive sciences, as well as its conceptual simplicity for applying mechanistic interpretability techniques to. \n\n2. “Detailed intuition”: We agree that this description (pasted below) was somewhat speculative during the submission:\n\n> “Raghu et al. (2021) finds that models pretrained on more data tend to learn local attention patterns in early layers, followed by global patterns in later layers. This might give CLIP a particular incentive to create local object representations, which are then used in relational operations. Future work might test this hypothesis.”\n\n* Since submission, we have rerun our behavioral analysis and attention pattern analysis on a DINOv2 ViT, which is pretrained on about 142M images; this is close in size (in order of magnitude) to CLIP’s pretraining dataset (about 400M images). Despite having a vastly different pretraining objective to CLIP, the DINOv2 model matches CLIP’s performance on both discrimination and RMTS tasks (99.5% test accuracy on discrimination and 98.2% on RMTS). It also exhibits two-stage processing like CLIP. This bolsters our intuition that pretraining data scale (rather than the type of pretraining supervision) is the key. We will include all of these results in the camera ready version. See Figure 1 in the supplemental PDF for supporting figures.\n\n**Questions**:\n1. “WO attention score is higher than WP”: These attention scores refer to the maximum proportion of attention paid by any given attention head to either within-object, within-pair, between-pair, or background tokens. Though the max for WO is higher than the max for WP in later layers, it is more informative to look at the trend within a given head-type rather than between head types. Notably, we find that the peaks of these scores occur in the expected sequence. \n\n2. “Analysis of some models are missing”: Thank you for catching this! We will include these graphs in the appendix of the final submission. \n\n3. “MAE”: Can do! We fine-tune a pretrained ViT-MAE model on the discrimination and RMTS tasks and perform an attention pattern analysis (following Section 3). We find that ViT-MAE achieves very similar performance to ImageNet and DINO ViT. On discrimination, it achieves 98% test accuracy and 94.9% compositional generalization accuracy. On RMTS, it achieves 93.4% test accuracy (interestingly, somewhat higher than ImageNet and DINO) and 85.3% compositional generalization accuracy. Its attention patterns do not demonstrate two-stage processing like CLIP or DINOv2; instead, the local and global heads are mixed throughout the layers. We will add an attention pattern analysis on MAE to the appendix of the paper.\n\n4. “Figure 6”: No—each data point corresponds to a different model, which achieves a different maximum disentanglement score. We will revise the caption for clarity.\n\n**Citations**:\n1. Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., & Dosovitskiy, A. (2021). Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34, 12116-12128.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "acyGYxUVcm", "reviewer": "Reviewer_NSpf", "summary": "This paper examines how vision transformers (ViTs) process visual relations, focusing on same-different tasks. The paper finds that pretrained ViTs fine-tuned on these tasks often develop a two-stage processing pipeline: a perceptual stage that extracts object features into separate representations, followed by a relational stage that compares these representations. Using interpretability techniques, they show that the perceptual stage creates disentangled object representations, while the relational stage implements somewhat abstract comparisons. The paper demonstrates that failures in either stage can prevent models from learning generalizable solutions. By analyzing ViTs in terms of these processing stages, the authors suggest we can better understand and improve how models handle relational reasoning tasks.", "questions": "- What would the same analysis methods result when applied on real world images or more complex reasoning tasks such as ARC-AGI?\n- Can you compare the mechanistic interpretation result with other text-based interpretability methods such as logic lens?\n- How to improve relational reasoning capability in vision models? What are implications of such potential improvements?\n- How would success or failure in relational reasoning impact CLIP in terms of embedding's alignment with text descriptions?", "limitations": "The author doesn't discuss limitations (although the paper checklist claims that limitations are discussed in Section 8).", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Addresses an important drawback of supposedly generalist vision models\n- Clear demonstration and definition of tasks that the paper focus on (e.g. Figure 1)\n- Thorough analysis, both in terms of performance metrics and model mechanism, of results\n- Abundant visualization of results\n- Constructed dataset and tasks that can benefit future studies of this topic", "weaknesses": "- Focus on more toy settings. Lack evaluation on more advanced/real world tasks.\n- While the central claim of dividing ViT perception into perception and relation stage is inspired by infant and animal abstract concept (line 71), there's the risk of confirmation bias from this analogy. \n- Plots in the paper requires clearer annotations and explanations (e.g. Figure 2 has too many acronyms, and it's difficult to understand what conclusion is for the plot)\n- Lacks qualitative example of more intuitively describe how model processes relational examples \n- While the abstract claims that understanding the relational reasoning help rectify short comings of existing and futures models (line 19), the paper doesn't discuss much what can be done to improve model relational reasoning (especially empirical experiments for improvements).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! Find point-by-point replies below:\n\n**Weaknesses**:\n1. “Toy settings”: Fair point! To address this, we created a realistic same-different dataset using 3D models of objects and used it to evaluate our models, similarly to [1]. See Figure 2 in the supplemental PDF for examples of this dataset. We find that CLIP attains a zero-shot test accuracy of 93.9% on this dataset, while all other models attain chance performance. We also find that CLIP (and DINOv2, despite only achieving chance accuracy) exhibits similar two-stage processing on the realistic stimuli without additional fine-tuning on them (albeit with more attention paid to background tokens). This generalizes our results from the toy setting. See Figure 3 in the supplemental PDF for attention pattern analyses on these stimuli. We note that this task is not a perfect analogue to our synthetic setting due to more realistic perceptual variation (i.e. lighting, rotation, occlusion), but hopefully its inclusion can ameliorate concerns about real-world data/tasks!\n\n2. “Confirmation bias”: This is a reasonable concern. However, we try to let the data speak for themselves. Though our attention pattern analysis might be more qualitative and up for subjective interpretation, our analyses in sections 4 and 5 make clear that there is a marked difference between early and late processing along the lines of what we have described as a “two-stage processing pipeline.”\n\n3. “Clearer annotations”: We completely agree. Upon acceptance, we plan to use the extra space in the final submission to make this figure larger and include more detailed annotations. We are happy to clarify our interpretation of Figure 2 as well (especially in the caption).\n\n4. “Qualitative examples”: We have some qualitative examples in Figure 11 (in the appendix), but we have created a new figure that gives much more intuition and have also written a greatly expanded appendix outlining this and Figure 11. See Figure 4 in the supplemental PDF for the new figure.\n\n5. “Improve model relational reasoning”: In the submission, we attempted to induce relational reasoning by encouraging models to form local, object-level representations using an auxiliary loss in Section 7. However, your point is very well taken, and we had similar thoughts. In light of this, we have devised a new auxiliary loss derived from the attention head scores in Section 3 that explicitly encourages models to exhibit two-stage processing. We hypothesized that this would improve relational reasoning. We find that training randomly-initialized ViTs on the discrimination task with this additional loss term significantly boosts performance (76.5% to 93.9% test accuracy; +17.4). It also boosts the model’s compositional generalization (75.9% to 92.3% accuracy; +16.4). We have promising preliminary results using this loss to improve performance on the RMTS task and will comment on them when ready. This loss is also quite general, and we believe that it can be easily adapted to and possibly improve performance for a wide variety of relations. We will include these new empirical experiments and details about the loss in the main body of the paper upon acceptance. \n\n**Questions**:\n1. “Real world images”: See response to weakness #1.\n\n2. “Logit lens”: At your suggestion, we have implemented and run a logit lens analysis on several models, analyzing how different model components contribute to “same” and “different” classifications. The analysis reveals that CLIP seems to process examples in a qualitatively different way compared to the rest of the models, perhaps pointing towards a mechanism for its success on the realistic stimuli. In particular, CLIP appears to make greater use of register tokens in the background compared to other models; it also appears to use the CLS token in non-intuitive ways for intermediate computations (rather than just gradually storing the “same” or “different” decision in CLS across layers, as other models do). We will include this analysis in the appendix, though we leave further investigation to future work.\n\n3. “Improve relational reasoning”: See response to weakness #5.\n\n4. “Alignment with text descriptions”: In general, a vision model that cannot perform relational reasoning will not discriminate between closely matched sentences (i.e. “plants surrounding a lightbulb” vs. “a lightbulb surrounding plants”). Indeed, this has been born out in the literature in the form of Winoground [2] among other datasets.\n\n**Citations**:\n1. Tartaglini, A. R., Feucht, S., Lepori, M. A., Vong, W. K., Lovering, C., Lake, B. M., & Pavlick, E. (2023). Deep neural networks can learn generalizable same-different visual relations. arXiv preprint arXiv:2310.09612.\n2. Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., & Ross, C. (2022). Winoground: Probing vision and language models for visio-linguistic compositionality. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5238-5248).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T3sz7IRM3D", "reviewer": "Reviewer_c3Nf", "summary": "The paper aims to answer how Vision Transformers (ViTs) perform tasks requiring visual relational reasoning. The study focuses on two tasks: identity discrimination and relational match-to-sample (RMTS), showing that ViTs process information in two distinct stages: perceptual and relational. Further analyses using mechanistic interpretability approaches were used to validate this results. Overall, this work offered insights into same-different judgement which is intuitive to humans but proven hard for AI.", "questions": "I am curious about the potential impact of CLIP's text co-training, in contrast to DINO's image-only training, on their respective performances when fine-tuned for same-different tasks. Specifically, I wonder if CLIP's joint training on images and captions forces its embedding space (particularly in the later layers) to deviate from purely visual representations to accommodate textual information as well. Could this difference in training approaches be a significant factor in explaining their divergent behaviors on visual reasoning? In that case, the the perpetual/reasoning phases probably not a property of ViT (as title suggested) but a product of transformer combined with multi-modality.", "limitations": "- Related work section is very limited.\n- Claims on ViT seem to confined to CLIP, so probably a slight revision of the title and text makes the basis of the claim stronger.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Similarity judgement task has been long studied in humans and animals, offering insights into perception and cognition, that's why the main question addressed in this study is of high interest for both psychology and AI community\n- The analyses included support the main claims\n- The boundaries of claims, and the questions remained to be answered were clearly stated in the discussion", "weaknesses": "- Related work is poorly presented: Despite the vast literature on both the question and the methodology, only one paragraph was allocated to the related work. There are other works suggesting a two phase processing in transformers (see below for an example) which I think discussing them in the context of this work would help to put the relational reasoning in a broader context.\n- Maybe related to point one, but a theoretical explanation supporting the findings, that is, under what conditions these distinct phases appear would make the work stronger.\n\n@misc{cui2024,\n\n      title={A phase transition between positional and semantic learning in a solvable model of dot-product attention}, \n\n      author={Hugo Cui and Freya Behrens and Florent Krzakala and Lenka Zdeborová},\n\n      year={2024},\n\n      url={https://arxiv.org/abs/2402.03902}, \n}", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review! Find point-by-point replies below:\n\n**Weaknesses**:\n1. “Related work”: We completely agree! We will use the extra space in the final submission to expand our literature review section to include more insights from both the (vast) cognitive science literature and the mechanistic interpretability literature (see global rebuttal #1 for specific topics). We are also happy to include more related work on the processing pipeline in transformers (including the reference you mentioned).\n\n2. “Theoretical explanation”: We agree, and we are curious to find this out ourselves! While we believe a proper treatment of the learning dynamics that give rise to multi-stage processing in transformers is out of scope for this work, we are highly interested in pursuing this direction in the future. Our current understanding is quoted below and is supported by additional analyses on DINOv2, a vision-only ViT whose pretraining dataset is closer in scale to CLIP than the other models. Despite using a very different training objective, DINOv2 matches CLIP’s performance on discrimination and Relational-Match-to-Sample (RMTS) tasks (99.5% test accuracy on discrimination and 98.2% on RMTS); crucially, it also exhibits similar two-stage processing. We will include these results upon acceptance, though see Figure 1 in the supplemental PDF for an attention pattern analysis on DINOv2. \n\n> “Raghu et al. (2021) finds that models pretrained on more data tend to learn local attention patterns in early layers, followed by global patterns in later layers. This might give CLIP a particular incentive to create local object representations, which are then used in relational operations. Future work might test this hypothesis.”\n\n**Questions**:\n1. “Multi-modality”: We were very interested in this question as well. As mentioned earlier, we analyze DINOv2 to explicitly test this. While CLIP does show some differences from DINOv2 (see general rebuttal #2 and the response to reviewer NSpf, question #2), DINOv2 largely recapitulates the results from CLIP. This suggests that data scale drives the adoption of two-stage processing rather than multi-modality.\n\n**Citations**:\n1. Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., & Dosovitskiy, A. (2021). Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34, 12116-12128.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all of the reviewers for leaving thoughtful, high-quality comments and questions on our manuscript. Here, we address some common themes found in multiple reviews, and also list all of the additional analyses that we have performed (or are planning to perform) to address any outstanding concerns.

1. **Literature Review**: Several reviewers noted that our literature review was somewhat sparse. We agree, and we plan to greatly expand it using the additional page conferred upon acceptance. In particular, we will add an expanded discussion of prior work evaluating visual relations in computer vision models, a discussion of interpretability work identifying processing pipelines in transformers, and an expanded discussion of abstract visual reasoning in humans.

2. **Synthetic Task Concerns**: Several reviewers raise concerns about the relative simplicity of our datasets. We first note that our dataset is constructed to enable the use of state-of-the-art mechanistic interpretability techniques from NLP on a ViT (some for the first time, to our knowledge!). As in the language domain, these techniques require a high degree of dataset control, and they tend to trade task and data complexity for the ability to make precise interpretations. However, we understand these concerns and have run a behavioral evaluation on a realistic same-different dataset (made using 3D models of objects) to demonstrate the generalizability of our findings. We discover that CLIP attains a zero-shot test accuracy of 93.9% on the realistic dataset; all other models achieve only chance accuracy. Furthermore, CLIP exhibits the same two-stage processing on these stimuli without any additional training on them (albeit with greater attention paid to background tokens throughout the model). See Figure 2 in the supplemental PDF for example stimuli from the realistic dataset and Figure 3 for attention head scores for CLIP (and a new model, DINOv2; see reply #3 below) on these stimuli.

3. **CLIP ViTs vs. Other ViTs**: Several reviewers note that our analyses mainly focus on CLIP ViTs rather than ViTs in general. In the submission, this is simply because CLIP attains the best performance on our tasks. Thus, since CLIP yields the largest number of accurate responses, focusing on it simplified the application of the mechanistic interpretability techniques we use. However, this raises the question: is two-stage processing a result of pretraining data scale (as we suggest) or multimodal pretraining? To address this, we have since run additional analyses on DINOv2 [1], another ViT that is pretrained without linguistic supervision on a dataset of similar size (in order of magnitude) to CLIP (~142M images vs. CLIP’s ~400M). We find that DINOv2 performs as well as CLIP on both discrimination and RMTS tasks (99.5% test accuracy on discrimination and 98.2% on RMTS); it also demonstrates two-stage processing like CLIP. We will include these results in the paper upon acceptance; we have also included the attention pattern analysis for DINOv2 in Figure 1 in the supplemental PDF.

4. **How to Improve Models**: Several reviewers ask how our results might lead to models that achieve stronger relational reasoning skills. We had the same question! Since submission, we have derived a new auxiliary loss from our attention head scores in Section 3 that encourages models to exhibit the two-stage processing found in CLIP (and DINOv2). We find that introducing this loss during training significantly boosts the performance of a randomly-initialized ViT on the discrimination task (76.5% to 93.9% test accuracy; +17.4)—interestingly, it boosts the model’s compositional generalization as well (75.9% to 92.3% accuracy; +16.4). See Table 1 in the supplemental PDF for more results. We have also obtained promising preliminary results using this loss to improve performance on the Relational-Match-to-Sample (RMTS) task and will comment with those results when ready. The formulation of the loss is general (i.e. not specific to same vs. different), so we believe that it could potentially be useful for future work looking to improve performance on other visual relations. We plan to include these empirical results and more details about the loss in the final submission. 

**List of additional analyses**:
- DINOv2 behavioral results and attention analysis (will include Section 4, 5, & 6 analyses on DINOv2 with final submission); Figure 1 in supplemental PDF
- Realistic same-different evaluations; Figures 2 and 3 in supplemental PDF
- Experiments using an attention pattern loss to improve relational reasoning; Table 1 in supplemental PDF
- Logit lens implementation and analyses
- ViT-MAE behavioral and attention analysis

**Citations**:
1. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., ... & Bojanowski, P. (2023). Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193.

Meta-review: The paper investigates emerging capabilities of visual reasoning developed within Visual Transformers. The reviewers found the paper interesting, with novel approach towards mechanistic interpretability. They also appreciated the clarity of presentation. Several weaknesses should be addressed though before the paper is published, namely the extended literature review, confusion about using CLIP ViTs or others, as well as the intuitive explanations how model processes relational examples.
Decision: Accept (poster)