Visual Anagrams Reveal Hidden Differences in Holistic Shape Processing Across Vision Models
Authors: Fenil R. Doshi
Thomas Fel
Talia Konkle
George A. Alvarez
Venue: NeurIPS 2025 poster ()
Keywords: Deep Learning
Vision Models
Shape Processing
Holistic Shape
Primary area: neuroscience_and_cognitive_science

Abstract: Humans are able to recognize objects based on both local texture cues and the configuration of object parts, yet contemporary vision models primarily harvest local texture cues, yielding brittle, non-compositional features. Work on shape-vs-texture bias has pitted shape and texture representations in opposition, measuring shape relative to texture, ignoring the possibility that models (and humans) can simultaneously rely on both types of cues, and obscuring the absolute quality of both types of representation. We therefore recast shape evaluation as a matter of absolute configural competence, operationalized by the  Configural Shape Score (CSS), which (i) measures the ability to recognize both images in Object-Anagram pairs that preserve local texture while permuting global part arrangement to depict different object categories. Across 86 convolutional, transformer, and hybrid models, CSS (ii) uncovers a broad spectrum of configural sensitivity with fully self-supervised and language-aligned transformers -- exemplified by DINOv2, SigLIP2 and EVA-CLIP -- occupying the top end of the CSS spectrum. Mechanistic probes reveal that (iii) high-CSS networks depend on long-range interactions: radius-controlled attention masks abolish performance showing a distinctive U-shaped integration profile, and representational-similarity analyses expose a mid-depth transition from local to global coding. A BagNet control, whose receptive fields straddle patch seams, remains at chance (iv), ruling out any "border-hacking" strategies. Finally, (v) we show that configural shape score also predicts other shape-dependent evals (e.g., foreground bias, spectral and noise robustness). Overall, we propose that the path toward truly robust, generalizable, and human-like vision systems may not lie in forcing an artificial choice between shape and texture, but rather in architectural and learning frameworks that seamlessly integrate both local-texture and global configural shape.

Visual Anagrams Reveal Hidden Differences in Holistic Shape Processing Across Vision Models
Fenil Doshi
Thomas Fel Kempner Institute Harvard University tfeleg www.aarr.dd.... edu
Dept. of Psychology & Kempner Institute Harvard University
fenil_doshiiee harvard. edu
Talia Konkle
George A. Alvarez
Dept. of Psychology
Dept. of Psychology & Kempner Institute Harvard University
& Kempner Institute
Harvard University
sila_koonke@harr edu
al alvarez@wjh harvard. edu
Abstract
Humans are able to recognize objects based on both local texture cues and the configuration of object parts, yet contemporary vision models primarily harvest local texture cues, yielding brittle, non-compositional features. Work on shape-vs- texture bias has pitted shape and texture representations in opposition, measuring shape relative to texture, ignoring the possibility that models (and humans) can simultaneously rely both types of cues, and obscuring the absolute quality of both types of representation. We therefore recast shape evaluation as a matter of absolute configural competence, operationalized by the Configural Shape Score (CSS), which (i) measures the ability to recognize both images Object-Anagren pairs that preserve local texture while permuting global part arrangement to depict different object categories. Across 86 convolutional, transformer, and hybrid models, CSS fii) uncovers broad spectrum of configural sensitivity with fully self- supervised and language language-aligned aligned transformers exemplified by DINOv2, SigLIP2 and EVA-CLIP occupying the top end of the CSS spectrum. Mechanistic probes reveal that (iii) high-CSS networks depend on long-range interactions: radius- controlled attention masks abolish performance showing a distinctive U-shaped integration profile, and representational-siiiaarii analyses expose mid-depth transition from local to global coding. A BagNet control, whose receptive fields straddle patch scams, remains at chance (iv), ruling out any (border-hacking strategies. Finally, (v) we show that configural shape score also predicts other shape- dependent evals foregroud bias, spectral and noise robustness). Overall, we propose that the path toward truly robust, generalizable, and human-like vision systems may not lie in forcing artificial choice between shape and texture, but rather architectural and learning frameworks that seamlessly integrate both local-texture and global configural shape.
Introduction
Human object recognition is remarkably robust: we can effortlessly identify objects across dramatic variations in texture, scale, viewpoint, and context because we can focus aspects of global
Project httpp:://wwwe.eeeaaaaaaaaccooooa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Visual Anagram example
puzzale pleces configured look alophant
Object Anagram Dataset samples
Wolf
Bear
Figure I: Object-Anagres task: probe of configural shape perception. (A) visual-anagram example-an identical set of 16 square diffusion patches is spatially permuted to form two distinct objects, here wolf and elephant (one shared patch is outlined in red). (B) additional image pairs from the abject-anagram benchmark. each pair comprises globally different objects built from the same unordered patch multiset, forcing any successful classifier to rely solely the global arrangement of parts.
configuration that are stable across such local photometric quirks [1 -22
5 6 By contrast,
state-of-the-art vision networks still harvest local, high-frequency shortcuts 7
9 This strategy
achieves high ImageNet accuracy [10] but leaves models brittle under texture shifts, adversarial
noise, and compositional
13 14]. The failure arises because models often seize spurious yet linearly separable features when multiple predictive
stresses 11
available [T]] I7. These differences between models and humans are often studied using the shape-versus -exture bias diagnostic, which pits shape texture using cue-conflict stimuli (8), but this metric is inherently relative: scores rise whenever shape coding strengthens or when texture coding weakens, rendering the absolute fidelity of global shape ambiguous [18] 19 20]. Effective
vision systems should exploit both cues when helpful [21 12 23 motivating an absolute assessment of shape and texture processing A.I
close this gap by recasting shape evaluation as an absolute test of configural competence. Building "visual anagrams" [24], synthesize image pairs that share identical multiset of local diffusion patches yet differ in their permutations (Fig. [T]. Correctly classifying both views demands sensitivity to spatial relations alone. We formalise the task through the Configural Shape Score (CSS), a joint two-image criterion whose chance level below % and whose ceiling mandates perfect configural sensitivity. Our study benchmarks 86 convolutional, transformer, and hybrid checkpoints from BagNet (25 stylized X and adversarially robust CNNs (66 to fully self- supervised ViTs like DINOv2
27 28 and language-aligned models such as SigLIP (2 30 and
EVA-CLIP 31. Combining behavioral metrics with mechanistic probes yields five key findings:
The nine-category
dataset enforces a stringent falsification test for holistic vision by permuting the global arrangement of an invariant multiset of local patches; any success therefore hinges on configural integration. The Configural Shape Score over this dataset gives an absolute score of configural shape, rigorously decoupling genuine shape inference from the artefactual gains that cue-conflict paradigms can achieve through mere texture suppression.
Vision transformers optimized via self supervieed learning and language-alignment, notably DINOv2 (27) EVA-CLIP (31) and SigLIP2 (30], dominate the CSS spectrum; their global- consistency objectives appear uniquely effective at instilling holistic shape representations, whereas comparably accurate, purely supervised counterparts achieve lower configural shape
scores.
Mechanistic dissection reveals that high-CSS models leverage cross rrosh-patch communication span- ning long-range interactions: performance collapses under radius-clipped attention masks with a U-shaped integration profile indicating that intermediate layers perform the key configural processing; pntnesentational-smillaiiii analyses echo this profile, exposing mid-depth pivot from local to global coding that is predicitive of overall CSS score.

---

Architectures confined to local receptive fields, exemplified by BagNet, perform near chance, ruling out "border- hackder- and underscoring that authentic configural shape demands long-rann long-range integration.
Models with higher configural shape scores also score high on other haape-dependent evals like foregorund-ve -aackggooudd bias, robustness to noise, phase dependence and critical band masking.
By converting long-standing theoretical critique into falsifiable measurement and linking the resulting scores to identifiable computational mechanisms, this work advances the science of holistic shape perception and offers actionable design principles for future vision systems.
2 Related Work
Configural and Holistic Shape Processing in Human Vision: In humans, there is evidence that configural shape processing is multifaceted, and the term broadly encapsulates any computation where the precise arrangement of parts affects the representation of object appearance or identity [4 5) There is even evidence that the appearance and recognition of local parts can be influenced by long-range interactions with other distal parts [32, indicating that contextual modulation is an important component of configural processing. These forms of configural shape processing can be distinguished from texture-basc based representations, where items appear to have the same texture despite spatial shifts in local features or parts, as long as the key higher-order statistical properties are the same 33 34
Computational Approaches to shape sensitivity: Prior work has investigated shape representations and texture bias in vision models [8] 35
38 39], often framing these issues in terms of shorteut learning driven by spurious correlations in the training data [3] Building on this, more recent studies have begun to probe whether models are sensitive to the spatial configuration of object parts [40] 42 43). However, these efforts typically rely on synthetic datasets and/or require explicit fine-tuning, focusing
36 37
understanding whether an architecture is at all capable of supporting relational reasoning. rather than whether such sensitivities emerge naturally during training. A notable exception is work by Baker and Elder [9] who tested whether pretrained models could detect disruptions in object configuration. Their approach involved splitting silhouette images along the horizontal meridian, flipping the bottom half, and stitching the parts back together. This manipulation was intended to break global structure while preserving local part content. However, this manipulation has some limitations: it ineffective for symmetric shapes, and the use of fack-annd-hitt silhouettes can obscure subtle configural differences.
3 Object Anagram Dataset and Configural Shape Score (CSS)
Background and notation. Consider the classical supervised-leaming paradigm, where denotes the image space and y [C] the index set of C distinct categories. A classifier f X - y maps an input image æ to a predicted label y. Our goal is to probe configural shape acuity: the ability to parse global part arrangements while remaining invariant to permutations of local texture elements. To do so we fix grid of K = 16 equal-area square patches. Let GK be the symmetric group of the K! possible permutations, and write E G for an element thereof. Given an ordered multiset of patches P- = {px} k==1
} with Pą E xhhxxx3 we define the composition operator:
P=(1)
PT(4)
I(P,T)-
E X.
PP(13))
Pr(1
which re-assembles the permuted patches into a 256 x 256 canvas. The permutation æ therefore fully determines the global layout.
Object Anagram Dataset synthesis. The synthesis pipeline is directly adapted from 24 For every ordered label pair (11, 32) E we prepare text-layout tuple (c(u;), "T))) -;) where c(u;) encodes the prompt "high-quality painting of a well-shown Yj with simple black paint texture on a grey background" using a pretrained T5 encoder, and where क, = id while πт / т is drawn uniformly from k. Both tuples share a common Gaussian seed, ensuring identical low-level texture statistics, and are injected into the DeepFloyd-IF pipeling" To maintain texture consistency while
Available https: : //tg://ggtttbbb reittuu..cmmddeee www.e///eeeeeoooooeoeeeee floyd/IF.
3

---

supporting distinct global configurations, we use permutation operator II( π that rearranges according to л. At each reverse-diffusiig timestep t, we compute two denoising predictions: e! for the canonical arrangement (21) and e(2) for the permuted arrangement (y2). Formally:
Ez t, c(vi)), These are combined into symmetrized target
e (0([l(z), T2), t, c(v1)).
G c
II (c!22, T2),
where (-, inverses the permutation so that the two predictions align in the canonical frame. The reverse-diffusion update is then
1 I-cre V VI-cee 61) + ox Th: N/(, I),
II ae.
with cosine noise schedule a and variance of both category representations, using identical patch content but differing spatial arrangements, progressively refining shared image. In each timestep, we obtain 2 64 64 resolution and after T steps the resulting image seeds a second diffusion at 256 x 256 resolution. From the final image we extract the patch multiset by partitioning it into a x 4 grid, yielding sixteen patches that share texture but differ in arrangement across the two views:
Oz- This procedure jointly optimizes
x [[(P,m) T1),
x (2) ((P,T2).
with ground-truth labels (1("), (2))) The critical property of these image pairs is texture invariance: (x(1), (2) share exactly the same patch multiset, and therefore identical first- and many higher-ordee texture statistics (color distributions, edge patterns, local frequencies) while differing solely in global configuration. Consequently, local cues alone are insufficient for classification, making this dataset stringent test of a model's configural processing capabilities.
Configural Shape Score. Gathering N such pairs yields A = {xx (2)
: 1 Each
image is centre-cropped to 224x224, normalised by the training statisties of f. and forwarded through the network. Mapping the resulting ImageNet logits to the nine object-anagram categories (Appendix) (A.)) define
CSS(f) Σ
(((() vf ^
f(x y (2)
whose chance level is 1/C². Suppressing texture alone cannot raise the score; only a genuinely holistic integration of global layout yields high CSS values.
Vision Models
To dissect the computational determinants of configural shape sensitivity assembled suite of 86 pretrained models and four randomly initialized baselines that together span the principal of modern visual representation learning. Standard convolutional networks trained with cross- entropy on ImageNet: Res VGG-16(45), and AlexNet [46) establish a supervised point of comparison, while three targeted manipulations of this template probe whether amplifying shape- vs-texture bias alone suffices: Stylized models [8), Adversarially Robust models [26). and Top-k Sparse models [77. Architecturally bio-inspired models, including the CORnet family 48] 49). Long-Range Modulatory CNNs [50], and an Edge-AlexNet trained exclusively on edge statistics, test the hypothesis that neural plausibility intrinsically fosters holistic processing. The role of sheer data exposure is examined through the BiT checkpoints 52 together with SWSL-ResNed
and SeL-ResNet-500 trained on billion-ima -mage corpora (53). Recent architectural refinements are covered by ConvNeXt architectures [54] [55] as well as by ResNet-50, ResNet-101 and ViT-B/16 checkpoints trained with rigorous augmentation pipelines [56] 577. A second axis contrasts convolutional and transformer principles. To this end, we include supervised Vision Transformers (88 along with its self-supervised counterparts: BEiT and BEiTv2 ViTs (59 60), MAE-ViTs and Hiera-MAF- ViTs We also add version with language aligned encoders such as
[6 62). and DINOv2-ViTs 27 28

---

Within each ViT family, we included several variants (e.g. S/M/L variations). These comparisons isolate the contribution of transformer and multimodal objectives. Finally, BagNets 25 whose receptive fields never exceed local neighborhoods, serves as explicit local-only controls.
CLIP ViT' 66 64), SigLIP VIT's and SigLIP2 VITs 29 30), and EVA-CLIP ViTs 31 651.
Collectively, this curated but diverse cohort will allow us to identify which architectural, training, and data regimes are predictive of high Configural Shape Score (see A.11 for the complete list of all the evaluated models)..
5 Models differ substantially in reliance on configural information for recognition
A) Configural Shape Score for Vision Models
B)
Relationship with Recognition
approx. human capacity estimate
1 0.58
0.9
Top Imagenet Accuracy
Configura Shape
0.8
0.6
0.2
0.3
0.0
untrinal
Lhue
Pakk chel
contor
sevile cofy
Bisteer Chey
Ratuer ciee
daper Aup of?
flom Aud cies tantll
Bo pant carl
andd sang of?
snd ber yn
CLP to
uea Bait spaec
Ovred in
0.0
Configural Shape Score
0.4
0.8
Configural Shape Score and Shape-vs- Texture Bias
D)
Relationship with Shape Bias
0.41
1.0
Stylization
1.0
L2 Robustness
1.0
Top-k Sparsity
0.6
Resnet50
Resnet50
Alconet
Shape-ves ves Texture Blas
0.5
0.5
0.5
0.4
0.2
0.0
0.0
0.0 Shape-ve- (epe-----a- Texlure Bies
Shape-vg- Tuallure Bises
Configural Shape Score
Shape-vs- Texture Blas
Configural Shape Score
Configural Shape
0.0
Score
0.0
light dark colors inciicte increasing amount of intervention (e.g.
increasing sparsity)
Configural Shape Score
Figure 2: Configural Shape Score (CSS) reveals variation across vision models matched in recognition performance and dissociates from imagenet accuracy and shape-vs-texture bias. (A) CSS across 86 vision models, quantifying how accurately models recognize the distinct objects in each anagram pair. Human performance is shown as the dashed reference line. (B) Relationship between CSS and top-1 Imagenet Accuracy across models. (C) CSS compared to shape-vs-texture bias for models trained with stylization, adversarial robustness, and Top-K K sparsity. While these methods increase shape-vs-tcxture bias, they show modest-to-no gains in CSS. (D) Relationship between CSS and Shape-vs-Ten Texxttre bias across all models.
Configural Shape Score varies widely across the full suite of models tested (Fig 2 with the sccheit-scoring models approaching human-level scores (see A.4 for human experiment details), and the lowest scoring models demonstrating little-to-no configural shape sensitivity at all. Models that showed the highest CSS were either self-supervised (DTs(DIN or language-aligned ViTs (P and EVA-CLIP models). It is also notable that models with similar, generally high levels of ImageNet 1 accuracy vary markedly in their CSS scores. For example, a supervised ViT-B/16 (ImageNet top-1: 76.35%; 23.61 % CSS) and tanguage-alignedd SigLIP ViT-B/16(Imageh) 1: 74.96%; 77.78% CSS have near equivalent ImageNet top-1 accuracy, but achieve this through different reliance on configural shape information. Thus, achieving high accuracies on ImageNet is not enough to obtain a high Configural Shape Score, and ImageNet accuracy alone does not determine the Configural Shape Score (Fig. 2B)
top-
Configural Shape Score also dissociates from Shape-vs-Texture bias. As shown in Fig. 2C, CSS unaffected by three key strategies known to enhance shape-vs-textue bias: stylization-based training g adversarial training (26). and top-k activation pruning [47. stylization training, object textures are decorrelated from object identity during training, forcing models to rely more shape than the
5

---

object's s texture. Fig. 2F (left) shows that models trained with stylization (dark blue) have much higher shape-bias than models trained without stylization (light blue). but there's little to no effect of stylization on configural shape score. Adversarial training optimizes models for robustness to worst-case perturbations, varying strength of the adversarial attack with an epsilon parameter. Fig. 2c (center) shows that shape-bias increases with epsilon (purple), while CSS is unaffected by this manipulation. Finally, Top-k activation pruning restricts forward propagation to the highest-activating units within each layer, and increasing sparsity via this pruning (green bars) increases shape-bias but has no effect on configural shape score (2 right), Across these manipulations, we find that gains in CSS were modest-to-none compared to the substantial increases in shape bias. Finally, overall find that the correlation between CSS and shape-vs-texture textere bias is moderate (r=0.64) indicating that only about 41% of the variance in CSS is accounted for by shape-vs-texture bias and vice versa (2D).
Taken together, these results indicate that the Configural Shape Score varies widely across models and dissociates from both ImageNet accuracy and Shape-vs-Texturr bias. To gain deeper mechanistic insight into how models achieve high CSS, we next performed attentional ablation and representational similarity analyses.
6
Long-range Contextual Interactions lead to higher Configural Shape Scores in Vision Transformers
Attentional Ablation
DINOv2-B/14 (high CSS 61.11%)
CJ VIT-B/16 (low CSS 23.61%)
Window: 1 nearby patch
Effect on Cless Taken
Effect on Class Token reartyy patches
nearty
Window:
000 seiuurt
OOW UUN
35
0.2
Alavdd Ireite
Indde
Altend inakdo
Atlend dulside
Ao Tuanforme
Alkead Tharafore black
Abased Thandormm bookk
Ablinel Taaaaommrr Mlac
B 10
Window: nearby patches
Effect on Configural Shape Score
Elfect Configural Shape
Window neatryy patch
Wirchn:: ranityy palches
Windows meertty pricch
Windows martt parch
Ote quaa
Inaide
Altend Inside
Attend outside
shad Thandorer
Abared blodk
Srarsforss
10
Figure 3: Long-range Contextual Interactions leads to higher Configural Shape Score. (A) Ablating self-attention in DINOv2-B/14 by selectively restricting each patch to attend only inside (blue) outside (orange) local window.Ablations
applied over windows with I or 2 nearby patches. (B) Effect of attentional ablation on the class token representation and configural shape score for high CSS model (Dinov2-B/ Restricting attention to short-range interactions ("attend inside" condition- - blue line) changes class tokens and disrupts CSS, most strongly at intermediate blocks. This effect is minimal when restricting attention long-range interactions ("attend outside" condition - orange line). Dashed line shows CSS in unablated condition. (C) Effect of attentional ablation on the class token representation and configural shape score for low CSS model (ViT-B/16). Disruption for short-range interactions have reduced in this model.
Vision transformers provide a unique opportunity to examine the mechanisms of configural process- ing, because standard ViTs divide the image into a grid of patches and any configural processing (interactions between patch representations) must be performed via self-attention mechanisms. Thus, by targeting self-attention mechanisms with ablations, we can determine the relative impact of both short-range and long-range contextual interactions. Here we examined how intermediate attention mechanisms influence representational dynamics within DINOv2-BA -B144, a
ViT with 61.11% % CSS and 44.1% top-1 ImageNet recognition. DINOv2-B/Ls processes an input image of size 224x224 pixels by dividing it into a grid of 16x16 patches (each 14x14 pixels). For comparison, we also performed the ablation study on VÝT-B/16, which achieved high top-1 ImageNet accuracy (76.35%) but had low CSS (23.61%). We performed attentional ablations during inference at different intermediate stages of these models by selectively restricting each patch's 's attention within a targeted attention block.

---

To determine the relative impact of short-range and range attentional interactions, we defined two distinct attention masking conditions (Fig. (Fig (i) "attend inside," " where each query patch attends only to patches within a specified Manhattan radius, and (ii) "attend outside," where attention is restricted to patches outside this radius. The class token was always allowed unrestricted attention in both conditions. We measured the cosine similarity of the original class token (unmasked) to the class token in both ablation conditions, as well as the CSS scores. If the class token representation depends mostly short-range attention, then it should be unchanged for the "attend inside" masks, but should drop when "attending outside" (excluding critical short-range interactions). In contrast, if long-range interactions are most important, then the representation should be unchanged when attending outside, and should drop when attending inside (excluding the critical long-range interactions). We defined "short-rauge raaore attention interactions as those within radius of 1 or 2 patches.
The results of these ablations for the high-CSS DINOv2 are shown in Fig. 33. Attending only to very local neighborhoods (radius of or 2 patches) substantially disrupted both class token representations and CSS (blue line, "attend inside" drops dramatically in middle layers). In contrast, attending only to more distant patches (orange curves, "attend outside") resulted in little-to-no change in class-token representations or CSS. Thus, it appears that attention interactions beyond at least 2 patches are necessary and sufficient to determine the class token representation and CSS score. Fig. 3C shows that a vision transformer (ViT-B/16) with lower CSS shows a reduced dependence on long-range interactions, suggesting that long-range attentional interactions are crucial for obtaining high-CSS.
Finally, these results show a U-shaped trend, suggesting that long-range interactions are particularly important in intermediate layers, and less important at early and late layers. These results suggest that early layers process patches relatively locally, then intermediate layers reinterpret and modify these local patch representations based on context, and then later stages aggregate locally over these contextually-modulaa
patch representations en route to the final model output. This observation is broadly consistent with work on LLMs, which suggest that early layers process text at the local token-level, followed by syntatic and broader contextual processing, and then finally return to more token-specific processing focusing on task-specific predictions and output generation [66 67].
To further test the critical role of these intermediate layers, especially in deeper architectures where single-block ablations can be too subtle, we designed a more stringent cumulative ablation exper- iment on the full DINOv2 family. In this analysis, we measured the irrecoverable contribution of intermediate layers by systematically ablating long-range interactions in all blocks before or after a selected layer, and measuring it's impact on the class token's 's representation from the last block. The full methodology and results are detailed in A.5 The biggest drop was seen in the intermediate blocks across all model sizes, confirming that the mid-depth layers were indeed the locus the most critical computations for configural processing in these models.
7 Relational Positional Encodings Boost Configural Shape Scores
While long-range attention is neces- sary for configural processing, the un- derlying mechanism for encoding spa- tial positions is also critical. To iso-
Table I: Configural Shape Scores (CSS) for ViT-B/16 vari- ants.
Model Variant (ViT-B/16)
CSS (%)
late this factor, we evaluated five vari- ants of the supervised ViT-B/16 ar- chitecture, each with different po- sitional encoding (PE) scheme (Ta- ble D.. While the standard model us- ing learned absolute PEs achieved a CSS of 23.61 %, models equipped with relative PEs, such as Rotary Position Embeddings (RoPE), more than doubled this score to over %.. This result demonstrates that a model's ability to process the relational spatial structure of an image criticial for holistic shape perception, and that CSS rewards this capacity.
Relative Positional Embeddings
38.89
RoPE (Rotary Position Embedding) RoPE (Rotary Position Embedding) Absolute PE
51.38
51.38
Mixed RoPE
48.61
Mixed RoPE + Absolute PE
52.77
8 Disentangling the influence of object category and anagram puzzle pieces on configural shape score
The ablation experiments on vision transformers demonstrated the emergence of configural represen- tations in the intermediate model layers. However, our ablation method only applies models with
7

---

A) Control Pairs
B) Representational Similarity
Ct Quantifying influence of category and puzzle pieces fro last layer models
Analysis Control Pairs
dif cateepories
Cosiro Simvarrt
10
UM-CUP
component influence
puzzle
same
na
category Influence
[[
1-
1]
+[1
)-
]
2
02
consgories dilf pubtle pec
0.
Modal Layor
Mode: Resnetso
= 0.8
F -0.7
catepory intuvence
04
Cosiiae Stmsterty
component
0.
same ostegories puzzzl
1.
1
0.5
Modal Layer
Configural Shape Score
Configurd Shape Score
Figure 4: (A) Control pairs to tease apart category- and component-level influence in model representations. (B) Cosine similarity across layers for each control pair type in EVA-CLIP G/14 and ResNet50. (C) Quantifying influence of object category vs. puzzle component from final layer embeddings. Models with higher Configural Shape Score (CSS) show stronger category influence and weaker component influence
self-attention mechanisms. To examine this transition from local to configural representations more generally for all model classes, we conducted a representational similarity analysis using a subset of carefully controlled image pairs from the Object Anagram Dataset. The purpose of this analysis was to determine at which point, if ever, different models transition from locally-driven representations dominated by puzzle-piece similarity to globally-driven representations dominated by categorical sim- ilarity. To disentangle the contributions of tiizzle-picce similarity and configural-shape similarity, we measured the cosine-similarig, between representations for three types of image pairs (Fig. 4 (1)
Same- taanee-art/ddffreentt cfffeerenttcctegooyy: anagram pair composed of images sharing identical puzzle pieces but representing different global categories (e.g.. wolf vs. elephant); (2) pattrrenttpatss/iiffeeeettt with containing different puzzle pieces and an equivalent categorical difference as a matched anagram wwweeenttppatsssmmeeeeee categorr pairs composed of different
pair (e.g., wolf vs. elephant); and (3) puzzle pieces but the same category (e.g.. both wolves). We evaluated 60 image pairs of each type (180 total). If cosine-similarity depends shared puzzle pieces, we would expect higher correlation for the anagram pair (same-parts, differentt-ategorr)) than for either of the other pairs (which all have different parts). If cosine-similarity depends only on similarity in global-configuration (category), then it should be higher for the SaaeecttPaat//SameCCaaeee Category pairs and equally low for the the other pairs (which all have different category). Based on the ablation study, we expect high-CS: models to show configural effects emerging by middle-to-late model layers.
We quantified representational similarity using cosine similarity between image embeddings each intermediate layer of a model. Fig. HB shows layer-by-layer results for one selected high-CSS model (EVA-CLIP G/14 (SS==77.88%). and one-selected low-CSS model (Resnet50, CS8=16.67%) Focusing first on the high-CSS model (top), several patterns emerge that are consistent with the idea that configural shape representations emerge in intermediate layers and dominate the final output of the model. First, in early model layers, local-similarity dominates: image pairs with shared parts (green) are more similar to each other than image pairs with different parts (orange and blue). Second, just beyond the midpoint, the effect of category similarity emerges; images with the pateereent-pattssmme--cttt parts/different-attee pairs (blue), and by the later layers the same- cate-cry pairs actually show greater similarity than the anagram pair (green). Indeed, by the final layers, the green/blue lines have collapsed together, indicating that having the same puzzle pieces is irrelevant by that point, and only similarity matters. The results for the low-CSS Resnet50 are markedly different, and suggest that the ResNet50 model never shows a transition to more configuration-base processing. As shown in 4B (bottom), the ResNet50 model shows greater part-based similarity (green) across all layers, including the final layers, and there is only a slight increase in similarity for
(orange) begin show greater similarity than the different-
the configur waagggraaaatggrr-leeeee
8

---

the different-pardee patttsama--aateeeeeeeae category pairs (orange) relative to the iiffereet--partdifffeee------ pairs (blue) at the final layer.
We formalized these qualitative observations by computing two metrics (Fig. [CC Puzzle compo- nent influence and Category influence over the last and penultimate layer of all models. Puzzle component influence is the difference in cosine similarity between pairs with identical puzzle pieces (anagram pairs) and those with different puzzle pieces and the matched category differences (different- parts/different-cctegory Category Influence is the average similarity advantage for same-category pairs over different-catee category pairs, irrespective of local puzzle pieces. We then measure whether the configural shape scores can predict these metrics across all the models. Fig. 5C shows this relationship for the last layer. Higher Configural Shape Score corresponded to lower Puzzle Component Influence (negative correlations: r = -..7 at the last layer. r = -0.57 at the penultimate layer), suggesting that higher-CSS models focus less on details of the local part appearance. Conversely, Configural Shape Score correlated positively with Category Influence (r 0.80 at the last layer, 0.83 at the penultimate layer), indicating that high-CSS models encode representations that are shared between images within a category, while discriminating between categories. This trend is observed even when considering representations from DINOv2 backbones, which are fully self-supervised and have no pressure to form abstract category representations (r = 0.94 between CSS and Category Influence and r = -0.86 between CSS and Puzzle Component Influence).
Taken together with the results of the ablation study, these results suggest that long-range contextual interactions enable tigh-CSS models to transition from representations that are initially dominated by local parts, to representations that are dominated by a holistic view that depends on the configuration of parts - i.e., encodes the image as more than the sum of its parts, specifically in terms of relationships between those parts.
BagNets Provide Evidence Against a "Border Hacking" Solution
Can a local strategy be used to recognize both pairs of images in a visual anagram? When rearranging and rotating the puzzle pieces, what if features that emerge at the intersection between abutting pieces are sufficient to identify the global category of each image in a pair, yielding successful classification BagNets (25) provide some evidence against the viability of local solution for anagram recognition. These models have a ResNet-style architecture, except that the receptive field of the intermediate units is highly restricted (at max 9, 17, or 33 pixels throughout), making them very sensitive to local features and incapable of any kind of long- range spatial/contextuul interaction. While they are competent in ImageNet recognition (with topl accuracy: Bagnet9 at 41.38% Bagnet17 7 at 55.08%, Bagnet33 at 61. 61.28%), they all have very low CSS (Bagnet9 at 2.78%, B at 1. 1.38%, Bagnet33 5.5%). Overall, the near-chance CSS of Bagnets underscores that fine-scale junction statistics alone are insufficient for anagram disambiguation, strengthening the interpretation of CSS as a global-configuration probe.
through non-configural
10 From Configural Shape Score to Broad Shape-Dependent Performance
Robustness to Noise
Foreground-vs-
Phase Dependence
Critical Band Masking
Background Blas
Robustress Score
0.0
1.2
0.81
FG-gles
1.0
0.76
Phase-ale Phase--lligeed aligned
Accuracy
0.6
0.73
BNP
0.5
0.83
0.79
3.4
0.0
0.2
-0.4
0.5
0.0
0.0
0.4
0.5
0.0
0.0
0.4
0.9
Configural Shape Score
Configural Shape
Configural Shape Score
Configural Shape Score
Score
Figure 5: Configural Shape Score (CSS) predicts model performance across a range of bench- marks. CSS is positively correlated with www-ooundd----bkggnroee bias, robustness to noise, phase dependence and critical band masking bandwidth
To what degree does having a high CSS predict other representational benefits and qualities? As shown in Fig. 5 we found that Configural Shape Score was significantly correlated with scores from several evals, including: 1) Robustness to Noise (r=0.81), testing each model's performance across varying severity levels for five distinct noise types, as described in [I]. 2) Foreground-vs --BBackgroundd

---

Bias (r=0.76), tested using ImageNet-9 dataset from [68], quantifying the extent to which model relies primarily on the foreground object rather than background information for classification. 3) Phase Dependence (r=0.73), assessed by swapping phase information in Fourier space between images and then measuring top-1 accuracy, quantifying the model's 's reliance on phase information [69]. 4) Critical band masking strategy outlined by [14 (r=0.83), used to determine the bandwidth of spatial frequencies essential for accurate object recognition. In contrast, the shape-vs-texture bias score across these models showed weaker relationships to these evals (r=0.62 with Robustness to Noise; r0.32 with Bregroundd-s-Backgromm
Bias; r=0.52 with Phase Dependence and r=0.55 with Critical Band Masking). Statistical comparison using Williams's 's test confirmed that CSS was a significantly better predictor of these metrics than shape-vs-textur
bias (all p<0.001; see A.7 for test statistics and details). These results suggest that models with better configural shape scores also have other favorable and human-like pereeptual qualities. See A.6 for more information about these evaluations, and AA8 and A.9 for feature attributions qualitatively compare low- and high-CSS models on challenging stimuli from each benchmark.
11 Limitations and Discussion
Although the Object Anagram Dataset and the accompanying Configural Shape Score (CSS) provide a quantitative measure of holistic processing, several caveats warrant mention. First, the stimuli we generate are constrained by the priors of the diffusion model, and may explore only subspace of configural encoding relationships. Furthermore, we use a uniform "black paint texture" to ensure local cues are perfectly matched, though this raises the possibility of out-of-distribution effects. However, our analyses show that even models with poor CSS can classify single anagram images successfully, suggesting the primary failure is in configural processing, not in handling the texture itself. Second, this work targets whole-object configurations and therefore does not directly probe part-based compositionality, an orthogonal facet of shape reasoning that future work should address. Third, despite containing thousands of composites, the dataset is modest in scale compared with modern billion-image corpora, suggesting that larger or more ecologically varied stimuli could reveal subtler effects. To that end, we created a 20x larger set of 1440 anagram pairs to test the robustness of our main findings. Re-evaluating all 86 models confirmed that the CSS metric is highly stable: the relative model rankings were preserved, and the scores from the original and expanded datasets showed a correlation of r=0.99 (see Appendix A.
Within these bounds our contributions are threefold. First, we formalized configural shape sensitivity as an interpretable metric distinct from texture reliance. Second, by charting CSS across 86 pretrained networks, we showed that holistic competence is neither fully explained by ImageNet accuracy nor canonical shape-vs- ve-texture bias. Third, attention ablation and representational similarity analyses Long-range interactions enabling recognition based
revealed that CSS relies on intermediate-stage on the configuration of parts and contextual relations. Finally, we demonstrated that models with higher CSS also perform better across other shape-dependent evaluations. These findings highlight configural shape processing as a critical yet underexplored dimension of visual intelligence and invite future work in advancing vision models toward human-like holistic representations.
Our results suggest that high configural competence stems primarily from training objectives that enforce to-loba consistency. For instance, DINOv2's teacher-student objective, which trains a student network on local image crops to match the output distribution of a teacher network seeing global crops, directly rewards the integration of long-range information. This riew-consistency objec- tive leads to a more powerful signal than standard data augmentations alone or purely reconstructive approaches like MAE, whose local pixel-prediction task is insufficient on its own to produce the mid-layer "configural flip" characteristic of high-CSS models. Language supervision (BEiT-v2, EVA-CLIP, CLIP. SigLIP) would also enforce alignment between standard local-image crops and global text caption, and this form of local-globe on multiple distant concepts in the image. Together, these findings point to a clear roadmap for building more shape- aware systems, which should prioritize (1) global-local consistency objectives, paired with (2) architectures capable of long-range integration, and (3) evaluated with explicit shape metrics like CSS.
consistency objective could also place an emphasis
References
[1] Irving Biederman. Psychological review, 94(2):115, 1987.
a theory of human image understanding.
10

---

[2] Andrew W Young, Deborah Hellawell, and Dennis C Hay. Configurational information in face perception. Perception, 16(6):747-75, 1987.
[3] Barbara Landau, Linda B Smith, and Susan Jones. The importance of shape in early lexical learning. Cognitive development, 3(3):299-321, 1988.
[4] Daphne Maurer, Richard Le Grand, and Catherine J Mondloch. The many faces of configural processing. Trends in cognitive sciences, 6(6):255-260, 2002.
[5] Johan Wagemans, James H Elder, Michael Kubovy, Stephen Palmer, Mary A Peterson, Manish Singh, and Rüdiger Von der Heydt. A century of gestalt psychology in visual perception: I. perceptual grouping and figure-grovd ground organization. Psychological bulletin, 138(6): 1172,
2012.
[6] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34:23885-
23899, 2021.
[7] Nicholas Baker, Hongjing Lu, Gennady Erlikhman, and Philip J Kellman. Deep convolutional networks do not classify based on global object shape. PLoS computational biology, 14(12): e1006613, 2018.
[8] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. In International conference on learning representations, 2018.
[9] Nicholas Baker and James H Elder. Deep learning models fail to capture the configural nature of human shape perception. Iscience, 25(9), 2022.
[10] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 11:22112222, 2015.
journaal
[11] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 1903. 12261, 2019.
[12] Christian Szegedy, Wojciech Zaremba. llya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6199, 2013.
[13] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585. 2020.
[14] Ajay Subramanian, Elena Sizikova, Najib Majaj, and Denis Pelli. frequlcy channels, shape bias, and adversarial robustness. Advances in neural information processing systems, 36: 2023.
[15] Katherine L Hermann, Hossein Mobahi, Thomas Fel, and Michael C Mozer. On the foundations of shortcut learning, arXiv preprint arXiv:2310.1622 2023.
[16] Gaurav Malhotra, Marin Dujmovié, and Jeffrey S Bowers. Feature blindness: A challenge for understanding and modelling visual object recognition. PLOS Computational Biology, 18(5): c1009572, 2022.
[17] Thomas Fel, Louis Bethune, Andrew Kyle Lampinen, Thomas Serre, and Katherine Hermann. Understanding visual feature reliance through the lens of complexity. Advances in Neural Information Processing Systems (NeurIPS), 2024.
[18] Fenil R Doshi, Talia Konkle, and George A Alvarez. Quantifying the quality of shape and texture representations in deep neural network models. Journal of Vision, 2024.
11

---

[19] Ziqi Wen, Tianqin Li, Zhi Jing, and Tai Sing Lee. Does resistance to style-transfer equal global shape bias? measuring network sensitivity to global shape configuration. arXiv preprint arXiv:2310.07555, 2023.
[20] Christian Jarvers and Heiko Neumann. Shape-selective processing in deep networks: integrating the evidence on perceptual integration. Frontiers in Computer Science, 5:1 313609, 2023.
[21] Akshay V Jagadeesh and Justin L Gardner. Texture-like representation of objects in human visual cortex. Proceedings of the National Academy of Sciences, 119(()):c111330211 19, 2022.
[22] Jenelle Feather, Guillaume Leclere, Aleksander Madry, and Josh McDermott. Model metamers reveal divergent invariances between biological and artificial neural networks. Nature Neuroscience, 26(11):201722334 2023.
[23] Yue Chen, Xingyu Chen, Anpei Chen, Gerard Pons-Moll, and Yuliang Xiu. Feat2gs: Probing visual foundation models with gaussian splatting. arXiv preprint arXiv:2412.09606, 2024.
[24] Daniel Geng, Inbum Park, and Andrew Owens. Visual anagrams: Generating multi optical illusions with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2415424163, 2024.
[25] Wieland Brendel and Matthias Bethge. Approximating enns with bag-of-local-fettuee models works surprisingly well on imagenet. arXiv preprint 1rXiv:190..00700 2019.
[26] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017.
[27] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07993, 2023.
[28]
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16550 2023.
[29] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. Proceedings of the IEEF/CVF international conference on computer vision, pages 1975-11986, 2023.
[30] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14776, 2025.
[31] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2023.
[32] James W Tanaka and Joseph A Sengco. Features and their configuration in face recognition. Memory & cognition, 25(5):883-592 1997.
[33] Javier Portilla and Eero P Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision, 40:49-70, 2000.
[34] Benjamin Balas. Texture synthesis and perception: Using computational models to study texture representations in the human visual system. Vision research, 46(3):299-309, 2006.
[35] Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 0-223296-23008 2021.
[36] Priyank Jaini, Kevin Clark, and Robert Geirhos. Intriguing properties of generative classifiers. arXiv preprint 210X::2309.16779,,200
12

---

[37] Paul Gavrikov, Jovita Lukasik, Steffen Jung, Robert Geirhos, Muhammad Jehanzeb Mirza, Margret Keuper, and Janis Keuper. Can we talk models into seeing the world differently? In Thirteenth International Conference on Learning Representations. OpenReview. net, 2025.
[38] Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[39] Katherine Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[40] Hojin Jang, Pawan Sinha, and Xavier Boix. Configural processing as an optimized strategy for robust object recognition in neural networks. Communications Biology, 8(1):386, 2025.
[41] Fenil R Doshi, Talia Konkle, and George A Alvarez. A feedforward mechanism for human-likk contour integration. bioRxiv, pages 2024-06, 2024.
[42] Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. Advances neural information processing systems, 31, 2018.
[43] Valerio Biscione, Dong Yin, Gaurav Malhotra, Marin Dujmovic, Milton L Montero, Guillermo Puebla, Federico Adolfi, Rachel F Heaton, John E Hummel, Benjamin D Evans, et al. Mind- set: Vision. a toolbox for testing dnns on key psychological experiments. arXiv preprint arXiv:2404.05220. 2024.
[44] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[45] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. Workshop Proceedings of the International Conference on Learning Representations (ICLR), 2014.
[46] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2012.
[47] Tianqin Li, Ziqi Wen, Yangfan Li, and Tai Sing Lee. Emergence of shape bias in convolutional neural networks through activation sparsity. Advances in Neural Information P Processing Systems, 36:71755-71766. 2023.
[48] Jonas Kubilius, Martin Schrimpf, Aran Nayebi, Daniel Bear, Daniel LK Yamins, and James J DiCarlo. Cornet: Modeling the neural mechanisms of core object recognition. BioRxiv, page 408385, 2018
[49] Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najb Majaj, Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, et al. Brain-like object recognition with high-performing shallow recurrent anns. Advances in neural information processing systems, 32, 2019.
[50] Talia Konkle and George Alvarez. Cognitive steering in deep neural networks via long-range modulatory feedback connections. Advances in Neural Information Processing Systems, 36: 21613-21634, 2023.
[51] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pages 491-507. Springer, 2020.
[52] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 10925-10934,
2022.
13

---

[53] I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-seae scale semi-supervised learning for image classification. arXiv preprint arXiv:I905.0003 00546, 2019.
[54] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11996-11988 2022.
[55] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16133-16142, 2023.
[56] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.0047, 2021.
[57] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106. 10270,, 2021.
[58] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.I / 1929, 2020.
[59] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprint wxXv:2106.08254: 2021.
[60] Z. Peng, L. Dong, H Bao, Q Ye, and F Wei. Beit v2: masked image modeling with vector- quantized visual tokenizers. arxiv 2022. arXiv preprint arXiv:2208.06333 06366, 2, 2022.
[61] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 16000-16009, 2022.
[62] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhay Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al. Hiera: A hierarchical vision transformer without the bells-and-whistlee In International conference on machine learning, pages -944444. PMLR, 2023.
[63] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 2021.
pages
[64] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language imguagg--maae learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11118822,,91111 2023.
[65] Quan Sun. Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 15389, 2023.
[66] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020.
[67] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer ford-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203. /4680, 2022.
[68] Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.0999 2020.
[69] Thomas Garity, Thomas Fel, George A Alvarez, and Thomas Serre. The role of frequency in shaping features from artificial vision models. Cognitive Computational Neuroscience, 2024.
[70] Alexa Tartaglini, Wai Keen Vong, and Brenden Lake. develoomeentalt--inpiied examination of shape versus texture bias in machines. arXiv preprint a0XX::2200.08334000000
14

---

A Appendix
A.1 Shape-vs-Texture Bias: A Useful but Incomplete Metric for Assessing Shape
Representations
widely used benchmark for assessing the degree to which models rely on shape is shape bias, introduced by Geirhos et al. (2019). In this paradigm, each stimulus is hybrid image that contains the shape defined by one category (e.g., the shape of cat) with a conflicting texture from a different category (e.g., elephant skin). Humans show strong shape preference in this task, performing ceiling (9%%).I contrast, standard deep net models such as ResNet-50 typically exhibit a strong texture bias, favoring the incongruent texture 70 70-80% of trials. While this paradigm reflects the model's ss relative preference between two competing cues-shape or texture-it is ambiguous if a high score is attained by suppressing textural information enhancing shape representations. For all shape-vs- shape-vs-texture bias we follow the updated method used in [18] that adjusts for baseline shape accuracy, providing more principled measure of shape quality using the following equation:
Shape-vs- Texture Bias (accuracy-correctee)
#Correct Shape Decisions # Correct (Shape + Texture)
#Correct Shape Decisions Total Trials
A)
Standard
Object-masked Cue
Shape-vs-Testie Texture Bias (Accuracy Corrected)
Originl Image
Feature
Stylized
0ojes-mmatee Cue Contict
00
BA
2.5
68
1.5
1.0
1.0
Steppe-tee Tecture Bloos
(Accurscy Corrected)
G.E.
0.1
G.E
0.8
Paguarr
Pagale
Swbad
Payyy
Stentar (hgcc-maaled
Stender
Ubsect-radaad
Standard
Figure 6: (A) (Left) Cue-conflict stimuli illustrating hybrid images composed of shape from one category and texture from another. Object-masked cue-conflict stimuli has texture removed from the background (Right) Shape-vs- -exttue bias (accuracy corrected) across models using standard and cue conflict stimuli. (B) Shape-vs- -eeetrr bias (accuracy corrected) for original and stylized models. (C) Feature attribution on SesNee55--Sylizee reveals that model decisions are still driven by local texture-rich patches, not the full object extent, suggesting that stylization does not completely enhance shape processing.
In Fig. 6A we measure shape-vs-texture bias for a variant of the cue conflict dataset developed (70). in which the conflicting textures are masked out in the background, preserving only the object silhouette. If shape-vs-texture bias truly reflects shape representations, this manipulation should not significantly alter bias scores. However, across a broad range of well-trained deep networks (n = 86), we observed consistent and substantial increases in bias scores under the object-masked condition. To contextualize these findings, we compared the improvements achieved by background masking with those achieved through stylization-based training. As shown in Fig. 6B. stylized models evaluated on the standard cue conflict task showed much shape-vs-textuu as it would have shown when vanilla (non-stylized) architectures were tested on object-masked cue-conffict stimuli, suggesting that the shape bias measure is confounded not only by texture within
bias gains
15

---

the object but also by surrounding local image statistics that lie outside the object's boundary. In Fig. of. we applied ittribution-basee analyses on ResNet50- Stylized assess which parts of the image were most influential for the model's decision. The results show that model activations often remained highly localized-focuts focusing on small, texture-rich fragments-rather than spanning the full extent of the object, consistent with findings from [9]]. In other words, these results suggest that merely suppressing texture either during training (i.e. via stylization) or removing texture footprint in images via silhouette masking, all while keeping the shape information intact, can drive model's shape-vs- ve-texure bias scores up. Together, these findings suggest that while shape bias remains valuable comparative diagnostic for assessing model preference between competing shape and texture cues, it should not be interpreted as the only single evidence of good quality shape representation.
A.2 Compute Details
All models were analyzed on internal computer cluster with 24 cores, 384GB of system RAM, and a NVIDIA H100 GPU with upto 80GB memory. The object Anagram Dataset was generated using single NVIDIA A100 GPU with 40 GB memory.
A.3
Extracting 1000-way logits and mapping to 9 categories from the Object Anagram Dataset
For self-supervised models like BelTs, MAEs, CLIP, and EVA-CLIP models used the finetuned linear classifier head provided via the timm library and for DinoV2 we used the full-4 classifier head provided with the model backbone. For SigLIP models we analyzed zero-shot predictions extracted by probing the outputs of the vision encoder with embeddings from text encoder using the category prompts and the given image as inputs.
To map the 1000-way ImageNet logits to our nine target categories in the Object Anagram Dataset, used the category-to- -e-aggeNet class mapping used in [9] (see below). For each target category, collected logits corresponding to each category's ImageNet class indices and then took the maximum value from those indices. Once a logit value was computed for each target category, applied softmax to get a 9-way probability vector. The predicted label was set to the category with the highest probability.
Category
ImageNet Class Indices
bear
[294, 295, 296, 297]
bunny
[330, 331, 332]
cat
[281, 282, 283, 284, 285]
elephant
[101, 385, 386]
frog
[30, 32]
lizard
[38, 39, 40, 41, 42, 43, 44, 46, 47, [286, 287, 288, 289, 290, 291, 292, 293]
tiger
turtle
[33, 34, 35, 36, 37]
wolf
[269, 270, 271, 272, 273,274, 275]
Table 2: Mapping between 9 target categories on Object Anagram Dataset and their corresponding ImageNet class indices.
A.4 Human Configural Shape score Estimate
We measured human Configural Shape Score (CSS) using a behavioral experiment implemented in jsPsych. Participants first completed informed consent and viewed instructions explaining the task and rationale. Each trial began with a fixation display followed by an image from the dataset presented centrally for 750 milliseconds. Immediately afterward, noise mask consisting randomly generated grayscale pixels appeared for 500 milliseconds to disrupt visual persistence. Following the mask, participants selected the object's category from nine visually presented icons (bear, bunny, cat, elephant, frog, lizard, tiger, turtle, or wolf). Participants completed all 144 images (72 anagram pairs), presented in randomized order, with their category selections and response times recorded. The resulting human CSS served as approximate baseline for evaluating the configural shape sensitivity of the computational vision models. This study was approved by the IRB of the corresponding author's ss home institution.
16

---

A.5 Cumulative Ablation Analysis on DINOv2 Models
To more precisely characterize the role of intermediate layers for configural processing, we performed cumulative ablation study on the DINOv2 model family (S/14, B/14, 1/14, and G/14) and the ViT-B/16 baseline. We hypothesized that the single-block ablation presented in too subtle reveal the full processing dynamics in deeper models. To address this, we designed an ablation experiment with two conditions: (I) ablating all blocks up to block 'n' to isolate the contribution of the network up to that point, and (2) ablating
be
blocks after block 'n' to test for recovery by later layers. The ablation follows the same strategy of masking the attention in the targeted blocks using a mask window nearby patches for all the patches, hence blocking long- gongrrane interactions.
Fig. 7 shows the effect of these ablations the final class token representation. measured by its cosine similarity to the token from the original, unablated model. In the "Ablate all before" condition (blue line). the similarity starts high and drops steeply as intermediate blocks are removed, while the "Ablate all after" condition (orange line) shows the opposite trend, creating the characteristic crossover point in the intermediate layers of each model. The results demonstrate that across all DINO models tested, the drop in CSS is highest and most irrecoverable in the intermediate layers. This confirms that even for the large and giant DINO variants, the mid-depth layers are the critical locus for integrating local features into a global, configural shape representation.
Effect on Class Token from last block after masking attention with window: 2 nearby patches
VIT-B/16
DINOv2-S/14
DINOv2-B/14
1.0
0.0
0.3
10
12
10
12
1.
DINOV2-L/14
DINOv2-G/14
05
10
14
22
2
70 20 22 30
Logend
Ablate before
Ablate
Figure Effect of cumulative ablation on class token representation. The y-axis shows the cosine similarity between the final class token of the original model and the ablated model. The blue line (Ablate all before) shows the condition when ablating long-range interactions that 'accumulated" in all blocks before block n. The orange line (Ablate all after) shows the condition when ablating long-range interactions that could be 'recovered' in all blocks after block n.
A.6 Other Shape-dependent Evals
Evaluation
Stimuli
# Images
# Categories
Ref.
Robustness to Noise
Imagenette2
98,125
10
Hendrycks & Dietterich, 2019
(and fastai) [T]]
Foreground Bias
Imagenet9
4050
9
Xiao et al., 2020 6S
Shape-vs- Texture Bias
Cue Conflict
1200
16
Geirhos et al., 2018
[8
Critical Band Masking
Imagenet
1050
16
Subramanian et al., 2023 14
Phase- Denendeeee
Imagenet
50k
1000
Garity et al., 2024 (5
Table 3: Overview of evaluation metrics, stimuli, and dataset statistics.
17

---

Phase Dependence ImagoNat Dataset
Cure-corficl
Bephani
Gct acturacy basod prase prass alignid iabels Garity
Texture
Decisions scaled by
Overal Shape Acturacy
(Geitoo al. 2018: Doars et al. 2004)
Shape
Label x
Labely y
Labely
Foreground-ve Background Bias
Robustnees noise
Dacatel
Dompptions over Imagenatta Denaset
Dog
proportion FC Decisions scaled Overal FG Acoumacy
Foreground
Erior Rates classifirr
Hird Backgrounc
(Niso el al., 2000)
GRADO rebae brighowes
Get Rebustness using Comuption ostimates
Critical Band Masking
ImaceNett Dataset
BWN
(Hondrysks
Pscomrect each norse
dlea'
car dltion
[Suoraman
Figure 8: Schematic of other Shape-dependent Evals.
A.7 Statistical Comparison of Predictive Strength: CSS vs. Shape-vs-Texe -eeeturre Bias
To evaluate whether Configural Shape Score (CSS) better predicts other shape-dependent evals than the traditional Shape-vs-Texture Bias, we used Williams's 's test for dependent correlations with one variable in common (i.e., each eval score). The test compares two correlation coefficients (corr(CSS, Eval) and corr(Shape-vs-Terd Texture Bias, Eval)) that share a common outcome variable, accounting for the correlation between the two predictors. evals: Robustness to Noise, Boreground-vs-Baccgrooo
tested this for each of the four Bias, Phase Dependence, and Critical Band Masking Bandwidth. We used a one-tailed Williams test with n = models, reflecting the directional hypothesis that CSS should better predict eval performance than Shatu-vs-Texture Texture Bias. All tests were statistically significant at < 0.01, indicating that Configural Shape Score is significantly stronger predictor of these eval benchmarks than Shape-vs -exettuu-ee Texture Bias. Below are the results:
Eval Benchmark
r(CSS, Eval)
rSShape-vs-Tesee Texture Bias, Eval)
t-value
p-value
Robustness to Noise
0.81
0.62
3,4116
0.0005
Foreground-ve -as---s--B vs-Backgrunddd Bias
0.76
0.32
7,618
<000001
Phase Dependence
0.73
0.52
3.39
0.00053
Critical Band Masking
0.83
0.55
5.47
<0.0001
Table 4: Comparison of Configural Shape Score (CSS) and Shape-vs- Textere Bias.
18

---

A.8 Feature Attribution for Challenging stimuli
Resnet50 stylized
DINOv2-B/14
Figure 9: Feature Attribution Maps for challenging stimuli. Maps generated using Integrated Gradients for cue-confiict stimuli, phase swapped stimuli, and visual anagrams.
A.9 Feature Attributions Anagrams in High-CSS model (DINOv2-B/14) are Anagrams themselves
Feature Attribution
Reconfigure
Feature Attribution
Reconfigure
Figure 10: Applying permutation transformation on feature attribution maps of visual ana- grams.
19

---

A.10 Validation on Expanded Anagram Dataset
potential limitation of our initial study was the dataset size (72 anagram pairs). To address this, we expanded the dataset by 20x to include 1440 visual- -nagram pairs and re-evaluated all 86 vision models. The strong correlation (r=0.99) between the scores on the original and expanded datasets shown in Fig. demonstraaee that the Configural Shape Score is highly stable measurement and the conclusions drawn in the main paper are robust.
Expanded Anagram Set (1440 pairs)
1.0
Configural Shape Soores
0.8
n-0.99
Contigural
0.8
0.2
1440 Anagram Pairs
0.8
0.0
comuu taete
wetn mi
Original Anagram Set (72 pairs)
0.2
Contigunal Shapa Soote
0.0
0.0
0.2
0.4
0.6
0.8
0.4
Anagram
0.
the some dhr enee con
con
opt cuR
Figure 11: Validation of Configural Shape Scores (CSS) on an expanded dataset. (A) CSS scores for all models on the expanded 1440-pair anagram set. (B) The original CSS scores the 72-pair set. (C) A scatter plot between the scores from the two datasets.
A.11
Evaluated Models
Architecture
Notes
Type
AlexNet
Untrained baseline
Standard Convolutional Networks (Torch
VGG-16
Untrained baseline
Standard Convolutional Networks (Torch Vision)
ResNet-50
Untrained baseline
Standard Convolutional Networks (Torch Vision)
ResNet-101
Untrained baseline
Standard Convolutional Networks (Torch Vision)
AlexNet
Standard AlexNet
Standard Convolutional Networks (Torch Vision)
VGG-16
Standard VGG-16
Standard Convolutional Networks (Torch Vision)
R5
Supervised baseline
Standard Convolutional Networks (TorchVision
ResNet-101
Supervised baseline
Standard Convolutional Networks (Torch Vision)
VIT-B/16
Base
Standard supervised Vision Transformers (Torch Standard supervised Vision Transformers Vision)
VIT-I/116
Large
VYT-B/32
Base
Standard supervised Vision Transformers (TorchVision
ViT-L/32
Large
Standard supervised Vision Transformers (TorchVision
AlexNet
bhaped AlexNet
Stylized Models
Re
bhape- ResNet-50
Stylized Models
VGG-16
Shape-biased VGG-16 L2 0
Stylized Models
55
Adversarially Robust Models
5c
L2 E 0.01
Adversarially Robust Models
ResNet-50
L2 = 0.03
Adversarially Robust Models
ResNet-50
I. 0.05
Adversarially Robust Models
ResNet-50
L2 0.1
Adversarially Robust Models
ResNet-50
I.2 0.25
Adversarially Robust Models
ResNet-50
L2 0.5
Adversarially Robust Models
ResNet-50
L2 1.0
Adversarially Robust Models
50
L2 3.0
Adversarially Robust Models
ResNet-50
L.2 5 5.0
Adversarially Robust Models
Alexnet (top-k=80%)
Sparse activation variant
Top-k Sparse
20

---

Architecture
Notes
Type
Alexnet (top-k=00%)
Sparse activation variant
Top-k Sparse
Alexnet (top-k=40%)
Sparse activation variant
Top-k Sparse
VGG-16 (top-k=80%)
Sparse activation variant
Top-k Sparse
VGG-16 (lop-k=60%)
Sparse activation variant
Top-k Sparse
VGG-16 (top-k=0%%)
Sparse activation variant
Top-k Sparse
Resnet50-Bag/Net-99
Local Receptive field limited 9 pixels
BagNets
Resnet50-BagNet-17
Local Receptive field limited 17 pixels
BagNets
Resnet50-BagNeet33
Local Receptive field limited to 33 pixels
BagNets
ResNet-50
Instagram- IB + ImageNet
Data-scale ResNets (SWSL/ SSL)
ResNet-50
YFMC - ImageNet
Data-scale ResNets (SWSL// /SS//SSI,
Resnet-50v2
BiT
Data-scale ResNets (BiT)
Resnet-50v2
BiT-distilled (ResNet-101x1) BiT
Data-scale ResNets (BiT)
Resnet-101v2
Data-scale ResNets (BiT)
ResNet-50
Supervised with strong augmentation
Strong Augmentation Baselines (Timm)
ResNet-101
Supervised with strong augmentation Supervised with strong augmentation (Small)
Strong Augmentation Baselines (Timm)
ViT-S/16
Strong Augmentation Basclines (Timm)
ViT-B/16
Supervised with strong augmentation (Base)
Strong Augmentation Baselines (Timm)
ConyNeXt-Bass
Modern CNN
ConvNeXt
ComyNeXLV2-Base
FCMAE JMMAE->ImagNNe
ConvNeXt
Con-NeXtV2--Hage
PCMAE->ImageNee
ConvNeXt
EfficientFormer-L11
Efficient transformer, SnapDist
EfficientFormer
EfficientNet-BB BO
JFT Pretraining
EfficientNet
Edge-AlexNet
Trained on ddge-filtered images
Edge-trained Alexnet
CORnet-Z
CORnet Family
Bio- Bio-Inspired
CORnet-R
CORnet Family
Bio-Inspired
CORnet-RT
CORnet Family
Bio-Inspired
CORnet-S
SCORnet Family
Bio-Inspired
Alexnet-L RM-Passl
Dong-Range Modulatory CNNs
Bio-Inspired
Alexnet-LRMs RRM-Pass2
Long-Range Modulatory CNNs
Bio-Inspired
Alexnet-LRM PPass3
Long-Range Modulatory CNNs
Bio-Inspired
BEiT-B/16
ImageNet-22K + ImageNet
Self-supervised ViT BEiT
BEiT-L/16
ImageNet-22K + ImageNet
Self-supervised ViT BEiT
BEiTv2-B/16
ImageNet
Self-supervised ViT BEiTv2
BEiTv2-L/16
ImageNet
Self-supervised ViT BEiTv2
BEiTv2-B/16
ImageNet-22K + ImageNet
Self-supervised BEiTv2
BEiTv2-L/16
ImageNet-22K + ImageNet
Self-supervised ViT BEiTv2
Hiera-MAF-T
Hierarchical MAE (tiny)
Self-supervised ViT MAE
Hiera-MAE-S
Hierarchical MAE (small)
Self-supervised VIT MAE
Hiera-MAE-B
Hierarchical MAE (base)
Self-supervised VIT MAE
Hiera-MAE-L
Hierarchical MAE (large)
Self-supervised VIT MAE
Hicra-MAE-H
Hierarchical MAE (huge)
Self-supervised ViT MAE
DINOv2-VIT-S/14
DINO2 (small)
Self-supervised ViT DINOv2
DINOv2-ViT-B/14
DINO2 (base)
Solf-supervised ViT DINOv2
DINOv2-ViT-L/14
DINO2 (large)
Self-supervised DINOv2
DINOv2-ViT-G/14
DINO2 (giant)
Self-supervised ViT DINOv2
DINOy2-VIT-S/14
DINO2 (small) Registers
Self-supervised ViT DINOv2
DINOv2-VITE B/11
DINO2 (base) + Registers
Self-supervised VIT DINOv2
DINOV2-ViT-L/UE
DINO2 (large) 4 registers
Self-supervised VIT DINOv2
DINOv2-ViT VIT-G/14
DINOZ (giant) + Registers
Self-supervised VIT DINOv2
CLIP ViT-B/16
OpenAl CL.IP (base)
Language-aligned ViT CLIP I
CLIP VIT-B/32
OpenAl CL.P (base)
Language-aligned ViT CL.PP
CLIP ViT-L/14
OpenAl CL.IP (large)
language-aligned ViT CLIP
CLIP VIT-L/14
OpenAI CLIP (large) ImageNet-12K
Language-aligned VIT CLIP
CLIP VIT-H/14
LAION-2 2BB CLIP (huge)
Language-aligned ViT CLIP
CLIP VIT-H/14
LAION-2B 2B CLIP (huge) + ImageNet-124 12K
Language-aligned ViT CLIP
SigLIP VIT-B/16
SigLIP (base), Image size 224, zeroshot
Language-aligned ViT SigLIP
SigLIP VIT-B/16
LIP (base), Image size 256, zeroshot
Language-alignes ViT SigLIP
SigLIP ViT-L/16
LigLIP (large), Image size 256, zeroshot SigLIP2 (base), Image size 224, zeroshot SigLIP2 (base), Image size 256, zeroshot
Language-alignn ViT SigLIP
SigLIP2 VIT-B/16
Language-aligned ViT SigLIP2
SigLIP2 VIT-B/16
Language-aligned ViT SigL
SigLIP2 VIT-L/16
SigLIP2 (large), Image size 256, zeroshot EVA02 CLIP (giant)
Language-aligned ViT SigLIP2
Language-aligne VIT EVA-CLIP
21

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes]. [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper the relevant
information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made
in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It fine include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors encouraged to create separate "Limitations" section in their paper.
22

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If appplica, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof's in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reprodueibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (c.g., in the case of a large
Depending on the contribution, reproducibility
23

---

language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or way reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We plan to release the complete dataset and codebase, along with detailed reproduction instructions, upon acceptance and in time for the camera-ready submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurIPS code and data submission guidelines (https
to
liaips.cc//ublic/uudeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new
proposed method and basclines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to a level of detail
that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
24

---

Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eπrоrs is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Details Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster,
cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn' t make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics stttps//nneurs hc/ppublcc//hhhccssseeeeeleeeeee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require
deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration
due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [ÑA]
Justification: No direct positive/negative societal impacts are expected, so these are not mentioned in the pape
Guidelines:
The answer NA means that there no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets. patisss.pppsssssss thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
26

---

13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
We plan to make entire data and codebase publicly available with the camera-ready version
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We conducted small-scale in- behavioral experiment to estimate human performance on the Object Anagram task. All participants gave informed consent to participate in the study. Images were shown under controlled timing (750 ms stimulus, 500 ms noise mask), after which participants selected a label from nine options. No personal data was collected, and the study qualifies as exempt under the home institution's 's IRB guidelines. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: This study was approved by the IRB of the corresponding author's home institution.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Core method development this research does not involve LLMs
27

---

Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htt.l/////aarrrssscc////offfeeee LLMM) for what should or should not be described.
28


Reviews:
[{"review_id": "ySA9gUM66J", "reviewer": "Reviewer_Bepr", "summary": "This paper presents a new shape processing benchmark that aims to measure whether models focus on local cues or the global configuration of these local cues to recognize objects. They do this by generating pairs of images that share the same local cues but show a different class of object. They use earlier work to generate these visual anagrams where patches in one image are permuted to get the image of another object (e.g., wolf vs. elephant). They define what they call a configural shape score (CSS) using such pairs of images. They test many models ranging from supervised CNNs/ViTs to self-supervised vision and vision + language models and present the following findings.\n\n- DINOv2, SigLIP2, and EVA-CLIP has high CSS, while supervised CNNs and ViT have relatively low CSS.\n- Higher object recognition accuracy doesn't necessarily mean higher configural shape score.\n- CSS and shape-texture bias are unrelated. Different manipulations that lead to change in shape-texture bias does not affect CSS.\n- High CSS models rely more on long range interactions than low CSS models.\n  - They find that especially the intermediate layers are important in processing these long range interactions.\n- They show the same finding in a different way by looking at similarity between representations where either puzzle pieces or the overall category changes between images.\n-  They show that configural shape score also predicts other shape-dependent evals (e.g.,foreground bias, spectral and noise robustness).", "questions": "- Why not do the attentional ablation study in section 6 for all or a few more (attenion-based) models? It might be interesting to see if these results hold for other models as well.\n- When the images are fed to ViT based models, are you aligning the ViT grid with the anagram's grid? What happens if it is not aligned and one token in ViT actually straddles multiple patches in the anagram? I don't think this will make much of a difference but I was just curious.\n- line 298, typo: 2) -> 1)", "limitations": "None", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Presents a very interesting and useful metric for configural processing, which is an important aspect of shape processing\n- Evaluated on a wide range of models\n- Great analysis of how long range interactions are important and where these happen in a model\n\nWeaknesses:\n\n- I think overall the paper is pretty strong so these are only minor weaknesses.\n- It would be good to have more examples (currently 72 pairs) in the dataset to get a more robust score.\n- More discussion of the results, especially why certain models have higher CSS, would be helpful. Why do the authors think self-supervised models do better than supervised ones for example?\n- It would be good to have a full list of all models in the appendix.", "was_revised": true, "final_justification": "I think this is a pretty interesting strong paper that deserves to be accepted to the conference.", "rebuttal": "Thank you for the clear summary, the nice comments and targeted suggestions. We particularly appreciate your acknowledgement that the paper “presents a very interesting metric for configural processing. Below we address each of your points in turn.\n\n---\n\n> **“It would be good to have more examples (currently 72 pairs) in the dataset to get a more robust score.”**\n\nWe agree that a larger dataset is crucial for ensuring the robustness of our findings. In response, we have increased the corpus to 1440 visual‑anagram pairs ( a 20x increase in dataset size). We re-evaluated all models on this new, larger dataset. Preliminary results show that the scores for models are very stable and the relative CSS rankings of the models are preserved, the correlation between the CSS scores on the original and expanded datasets is r = 0.99. We will include the expanded set and report these results in the camera-ready version of the paper!\n\n---\n\n> **“Why not do the attentional ablation study in section 6 for all or a few more (attenion-based) models? It might be interesting to see if these results hold for other models as well.”**\n\nWe now expand this analysis to the remaining DINO models ( DINOv2-S/14,  DINOv2-L/14,  DINOv2-G/14).  When applying the same ablation paradigm, the effects in other models were less pronounced, particularly the deeper models.  We hypothesized the single-block ablation may be too subtle. To address this limitation, we designed a cumulative ablation experiment: First, we ablated all blocks up to block ‘n’ to isolate the contribution of the network up to that point and second we ablated all blocks after block ‘n’ to see if there is recovery by later layers. This analysis showed that, across all DINO models tested, the drop is the highest and most irrecoverable (even for the DINO large and giant variants) in the intermediate layers. We will include this cumulative ablation analysis in the camera ready version. Thank you for this suggestion.\n\n---\n\n> **“More discussion of the results, especially why certain models have higher CSS, would be helpful. Why do the authors think self-supervised models do better than supervised ones for example?”**\n\nWe were also genuinely wondering about this and we propose that two main factors contribute to high-CSS scores: The first is an objective that includes some form of global/local contrastive loss. For example the high-CSS DinoV2 models include a direct contrastive loss between global views (large crops), and local views (small crops) from the same image. Other high CSS models such as CLIP, SigLIP, SigLIP2, Beit2, Eva-CLIP are directly or indirectly language-aligned. For instance, EVA-CLIP uses masked image modeling but is distilled from a CLIP-trained vision encoder while BEiT-2 is trained to predict masked discrete visual tokens from a semantic CLIP tokenizer. To this end, language, either through direct contrastive learning or via knowledge distillation, provides a “global view” of the image. The pressure for local representations to align with global representations seems to be a key factor giving rise to high-CSS scores, and the ViT architecture (with global attention) seems particularly well-suited to developing high-CSS.\n\nThe second possible factor is training on a large, diverse dataset. For example, it seems that the high CSS models were also trained on larger datasets. So, disentangling what CSS benefits are confirmed by the training loss vs the dataset (or an interaction of the two, where benefits of the training objective work best with global attention and at a certain dataset scale and diversity) is important work to be done. We will expand our discussion of these results and our hypotheses about why self-supervised models seem to do better than supervised ones in the camera ready version. \n\n---\n\n > **“It would be good to have a full list of all models in the appendix.”**\n\nAbsolutely, we will add a detailed list in the appendix for all the evaluated models. \n\n---\n\n> **“When the images are fed to ViT based models, are you aligning the ViT grid with the anagram's grid? What happens if it is not aligned and one token in ViT actually straddles multiple patches in the anagram? I don't think this will make much of a difference but I was just curious.”**\n\nThe pieces of the anagram are not actually aligned with the VIT grid! In fact, the anagram pieces are actually shaped like puzzle pieces (not perfect squares, as was indicated in the caption–we have fixed this)--this means there is no way to align these anagram pieces within the ViT grid. Further, the anagram patches are from a coarse 4x4 grid of 64x64 px pieces, while the ViT16 patches are 16 x16px patches; thus, there are approximately 4 VIT patches per anagram puzzle piece. (To address worries about “border hacking”, in case new local features are created at the interface between puzzle pieces that could account for the performance instead of holistic processes, we have section 8 using bag net models to rule out this possibility.) Thank you for asking these questions; we will clarify these details. \n\n---\n\n**Summary:** We think with the 20x expansion of the number of anagrams images, and the extended ablation analyses, the paper will be much improved. Thanks for your feedback."}, {"review_id": "SreVjF1FUq", "reviewer": "Reviewer_L13o", "summary": "This paper defines a measure of image \"shape\" sensitivity present in trained deep neural networks by using images pairs that are \"visual anagrams\" of each other.  The method to generate the visual anagrams comes from a previous paper and uses a diffusion model.  The result is a set of image pairs, where each image is a permutation of patches of the other, so texture is locally preserved but the images have different global shape and therefore are associated with different Imagenet categories.  They then construct a score (Configural Shape Score, CSS) to measure the sensitivity of the deep network classification to the texture-preserving permutation by calculating the fraction of times both images are classified correctly, meaning they are both mapped to one of a set of imagenet categories (manually?) set to be correct.  The goal of CSS is to measure \"an absolute score of configural shape\" without simply suppressing texture, and the authors claim that this score measures a notion of \"genuine\" shape inference.  They then show that the CSS score varies over the modern spectrum of vision models, with vision transformers generally having higher scores.  They then perform a variety of experiments that study how the CSS score is correlated with the emergence of long-range interactions between patches in transformers, how the representational similarity of image anagrams is inversely related to the representational similarity of different images in the same category as a function of model layer in some models but not others, and how the CSS score is correlated with a selection of other vision model benchmarks.", "questions": "* There seems to be a relatively small number of image categories generated.  What is the limitation here?  Why only 9 categories of visual anagrams?  How do these results depend on this?\n* You keep using the world \"multiset\" to describe the set of patches that the visual anagrams share.  How is this a multiset?  Can patches occur more than once in the set?\n* You say that this work offers \"actionable design principles for future vision systems\".  Can you be more explicit about what these design principles are?\n* Are rotations of the patches allowed when generating the image anagrams?  This is suggested by figure 1 but I don't see it mentioned anywhere.\n* I wonder whether the low performance of some models is because the visual anagrams are out of the training distribution. In particular, the texture cues (e.g. asking for \"black paint texture\") may be different than images in the training set. Can the authors comment on this possibility? Does this introduce limitations in interpreting the results?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n* Overall this is a wonderful paper and I would expect it to be a contender for a spotlight talk if the authors can offer some further improvements during the rebuttal period.\n* The idea is clever and well-motivated. It is introduced through lucid writing and clear figures.\n* The paper makes a clear and convincing point about shape processing emergence in vision models. In addition to the main findings (which I consider to be the plots in Figure 2), \n* The subject matter is perfectly fit for ICLR\n\n**Weaknesses**\n\n* In places, the analyses are somewhat anecdotal. It would increase my enthusiasm for the paper if the authors could be more comprehensive and quantitative.\n    * There seems to be a relatively small number of image categories (nine) generated. Also how many image pairs were generated in total? I couldn't find an exact number.\n    * Figure 3 shows a suggestive comparison of two networks, but are the trends here consistent across the broader range of transformer architectures? What about transformer architectures that are *trained* with smaller context windows (rather than ablated afterwards)?\n* All of the visual anagrams seem to be prompted to have \"black paint texture\". While this controls that the two images have the same texture, it may mean the texture is out-of-distribution. This complicates my intuition and interpretation of the results (see question below).\n* (Minor) The authors use the term \"representational similarity analysis\" to describe their analysis in Figure 4. This is confusing because Representational Similarity Analysis (RSA; [Kriegeskorte et al. 2008](https://www.frontiersin.org/journals/systems-neuroscience/articles/10.3389/neuro.06.004.2008/full)) is a different technique that is closely related to the popular Centered Kernel Alignment method (CKA; [Kornblith et al. 2019](http://proceedings.mlr.press/v97/kornblith19a.html)) as described in [Williams, 2024](https://www.biorxiv.org/content/10.1101/2024.10.23.619871v1.abstract). These well-known methods focus on comparing representations across different neural systems. To avoid confusion, I suggest the authors come up with different language to describe their results in Figure 4... Maybe something like \"measuring representational divergence\"?\n\n**Minor/typos:**\n- line 63: typo \"predictive\"\n- line 141: grammar, possibly supposed to be \"we also added versions\"\n- Figure 3: \"cls\" undefined\n- Figure 5: caption missing \".\"", "was_revised": true, "final_justification": "All reviewers seem in agreement that this paper is worthwhile to accept. I think the basic idea is a very clever concept and will be of great interest.", "rebuttal": "We thank you for the generous evaluation and sharp comments. We particularly appreciate the recognition that the study is “clever and well‑motivated,” features “lucid writing and clear figures,” and delivers “a wonderful paper” that could be “a contender for a spotlight talk,” pending minor clarifications. Below, we address specific comments.\n\n---\n\n> **“There seems to be a relatively small number of image categories (nine) generated. Also how many image pairs were generated in total? I couldn't find an exact number.”**\n\nWe agree with the reviewer on the small number of anagrams. The benchmark initially contained 72 total visual anagram pairs, where the choice of the specific 9 categories was motivated by Baker & Elder, iScience, 2022. \nWe have now expanded the number of total stimuli by 20x, from 72 to 1440 visual‑anagram pairs. We re-evaluated all models on this new, larger dataset. Preliminary results show that the scores are very stable and the relative CSS rankings of the models are preserved, the correlation between the CSS scores on the original and expanded datasets is r = 0.99. We will include the expanded set and report these results in the camera-ready version of the paper!\n\n---\n\n> **“Figure 3 shows a suggestive comparison of two networks, but are the trends here consistent across the broader range of transformer architectures? What about transformer architectures that are trained with smaller context windows (rather than ablated afterwards)?”**\n\nFair point. Our original goal was to provide a clear, controlled comparison between a high-CSS and a low-CSS model with similar ImageNet performance, which is why we focused on DINOv2-B/14 and ViT-B/16. However, since then, we expanded this analysis to the remaining DINO models ( DINOv2-S/14,  DINOv2-L/14,  DINOv2-G/14). When applying the same ablation paradigm, the effects in other models were less pronounced, particularly the deeper models.  We hypothesized the single-block ablation may be too subtle. To address this limitation, we designed a cumulative ablation experiment: First, we ablated all blocks up to block ‘n’ to isolate the contribution of the network up to that point and second we ablated all blocks after block ‘n’ to see if there is recovery by later layers. This analysis showed that the drop is the highest and most irrecoverable (even for the DINO large and giant variants) in the intermediate layers. We will include this cumulative ablation analysis in the camera ready version. While this set is not the full range of transformer architectures, we hope the expanded analysis helps provide more support for the trends we observed. Thank you for this question.\n\n---\n\n> **“All of the visual anagrams seem to be prompted to have \"black paint texture\". While this controls that the two images have the same texture, it may mean the texture is out-of-distribution. This complicates my intuition and interpretation of the results (see question below).”**\n\nIndeed, we intentionally used a homogeneous \"black paint texture\" to control and match the local statistics, as this helps further ensure that any classification difference would stem from the global configuration. However, it’s a fair question to wonder if this unnatural texture is enough out-of-distribution (OOD) that holistic mechanisms observed here do not apply to more ‘natural’ images. To make the concern more concrete, is poor CSS performance actually due to a failure of OOD texture, rather than a failure of configural processing?\n\nThere are several results that address this concern. The main evidence comes from single-image accuracy for the anagram images – if the texture itself were the problem, models should fail to classify even individual anagram images. This is not the case. For example, a ResNet-50 classifies single anagram images with 52.7% accuracy (well above the 11% chance rate). Its CSS, which requires correctly identifying both images in a pair, plummets to 16.7%. Similarly Bagnet 33 is at 5% on CSS but at 39% for individual images. This shows the model can extract meaningful features from one image, but fails when it must distinguish two configurations built from the exact same local features.  \n\nFurther, we also know that poor CSS models are accurately representing the local texture:  the representational similarity analysis shows that these models show high correlations between same-part pairs (Figure 4, green lines), and low similarity between different-part pairs (blue-and orange) indicating they have strong representations of the local details. If all the “black paint textures” were treated the same on account of being out of distribution, none of these results would hold (all correlations would be high, and all three lines would be overlapping). \n\nThat said, we agree with you that employing a richer set of matched textures would improve ecological validity, and we will highlight this as an important direction for future work in our discussion.\n\n---\n\n> **You keep using the world \"multiset\" to describe the set of patches that the visual anagrams share. How is this a multiset? Can patches occur more than once in the set?**\n\nApologies for the confusion.  We will change this word to “set” in the places it occurs in the manuscript. (While we were initially trying to use it to clarify the difference between unordered and spatially ordered patches, we realized the sentences in the paper are actually clearer and still accurate when we use the word “set”).  \n\n---\n\n> **\"Are rotations of the patches allowed when generating the image…\"**\n\nYes. The transformation involves rotation of the patches. We will update Section 3 in the camera-ready version to explicitly state that the transformation involves both permutation and rotation, which provides the necessary flexibility to form two distinct and recognizable object shapes from the same collection of local parts.\n\n---\n\n> **You say that this work offers \"actionable design principles for future vision systems\". Can you be more explicit about what these design principles are?**\n\nFor sure, thank you for this question! The core actionable principles derived from our findings are:\n\n1. Prioritize Training Objectives that Enforce Global Structure. Our results, consistent with prior findings, also show that models trained with standard supervised classification often solve the task using local texture shortcuts. In contrast, self-supervised objectives that require matching global and local views (like in DINOv2) or multimodal objectives that align vision with language (like in EVA-CLIP and SigLIP2) are far more effective at inducing robust holistic processing. Choosing the right objective is critical. \n\n2. Ensure Architectures Can Perform Long-Range Spatial Integration. Our mechanistic analyses (Sec. 6 & 7) show that the ability to relate distant image patches in intermediate layers is the key computational mechanism behind high CSS performance. The complete failure of architecturally-local models like BagNet reinforces this: without the capacity for long-range communication, models cannot develop configural understanding. It is an open question if post- and pre-training methods that amplify these mechanisms in standard ViTs would unlock such a capacity without needing to train on large orders of data. \n\n3. Replace shape-vs-texture measure with an explicit shape measure. Our work shows that improving relative shape-vs-texture bias does not guarantee better configural processing. The principle is that these are distinct abilities. Rather than simply discouraging texture use, future work could also directly measure and optimize for absolute configural competence.\n\nWe will add a dedicated paragraph in our camera-ready version to clearly outline these principles for future model development.\n\n---\n\n> **“(Minor) The authors use the term \"representational similarity analysis\" to describe their analysis in Figure 4. This is confusing [...] To avoid confusion, I suggest the authors come up with different language”**\n\nWe see that this terminology is unfamiliar, and we modified the text to more carefully introduce the term. This terminology was originally introduced in the computational neuroscience community (Kriegeskorte, 2008), and is increasingly being adopted in the NeuroAI community (e.g., “Getting aligned on representational alignment”, Sucholutsky 2023), and in ML research focused on similarity of neural network representations (Kornblith et al., 2019). \n\n---\n\n> **Minor typos**\n\nThank you so much for pointing these out. We will correct it in the revised draft."}, {"review_id": "GJTYwbpjVW", "reviewer": "Reviewer_3trt", "summary": "This paper introduces the Configural Shape Score (CSS), an evaluation metric that assesses a vision model's sensitivity to global shape configuration by using visual anagram image pairs—images constructed from the same set of patches but rearranged to represent different object categories. The authors show that CSS correlates with multiple shape-related benchmarks and is highest in self-supervised and language-aligned vision transformers (e.g., DINOv2, EVA-CLIP, SigLIP). Mechanistic analyses (attention ablations and representational similarity) reveal that high-CSS models rely on long-range attention and develop mid-layer configural representations.", "questions": "- Given the strong correlations between CSS and prior shape-related metrics (e.g., robustness, shape bias, foreground-background bias), can you elaborate on what new diagnostic power CSS offers? What specific types of failure modes or model differences would CSS detect that previous methods would miss?\n- Can you evaluate CNN-based self-supervised models such as MoCo, SimCLR, BYOL, or SwAV to determine whether high CSS requires attention, or if it can arise from training objectives and augmentations alone?\n\n- Did you try training or fine-tuning DINOv2 (or a similar ViT) without spatial augmentations (e.g., removing random cropping or multi-view augmentations) to assess whether positional robustness is necessary for high CSS?\n\n- Given that your visual anagram patches are larger than ViT patch size (64×64 vs. 16×16), do you think CSS performance reflects models' ability to integrate subpatch relationships, or is it primarily a function of spatial layout processing?\n\n- Can you clarify whether the representational similarity analyses provide insight beyond the expected hierarchical abstraction of deep networks (i.e., early layers being input-like, later layers being task-driven)? \n\n- Have you tested models with different types of position encodings (e.g., relative vs. absolute, or no encoding) to better understand whether CSS relies on absolute spatial anchoring or on relational structure?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- **Clever, well-controlled benchmark**: The visual anagram paradigm is a creative way to leverage diffusion models to synthesize image sets and isolate configural shape sensitivity, controlling for texture completely.\n\n\n### Weaknesses\n\n- **Lack of novelty in insight**: CSS is strongly correlated with existing shape-sensitive metrics (e.g., robustness, foreground bias, shape-vs-texture bias), and the core mechanistic findings (need for long-range integration, mid-layer configural transitions) are consistent with prior literature and likely generalize to other shape-related evaluations.\n- **Better use of mechanistic analysis: pixel vs latent self-supervised**: While MAEs are included in the CSS benchmark and perform moderately, the paper does not explore *why* they underperform compared to models like DINOv2. This is a missed opportunity. MAEs are self-supervised but trained with a pixel-level reconstruction objective, which likely enforces precise spatial encoding and discourages position-invariant configural representations. A mechanistic analysis—such as attention ablation or representational similarity profiling—would help clarify whether MAEs fail to develop global shape sensitivity due to architectural limitations or due to the nature of the objective itself. Including such an analysis would strengthen the argument that not all self-supervision leads to configural competence.\n- **Augmentation not disentangled**: High CSS may be primarily driven by the use of strong spatial augmentations during training (e.g., cropping in DINOv2), but this is not tested directly. A simple ablation (e.g., DINOv2 w/o crop) would help isolate the contribution of augmentation vs. architecture.\n- **Unclear advantage of the mechanistic analyses** The mechanistic analyses (e.g., layer-wise representational similarity) show that high-CSS models transition from part-based to configural representations across layers. However, this finding may be somewhat tautological: given that CSS is a classification output metric, it's expected that later layers — which drive the prediction — will reflect configural coding, while early layers reflect lower-level inputs. This hierarchical progression is a general property of deep networks and does not, in itself, constitute a novel insight.\n\n- **No evaluation of non-transformer self-supervised models**: CNN-based self-supervised methods like MoCo, BYOL, or SimCLR—trained with similar global-invariance objectives—are not tested, leaving a gap in understanding whether attention is necessary for high CSS.\n- **Position encoding interpretation not fully addressed**: The task design (patch-based permutation) inherently relies on models using positional information. Those synthesized patches are aligned with the patches of most transformer-based models use (only bigger). So, in effect, transformer models have an advantage over CNN models and transformer models with proper augmentation in training to avoid reliance on the position of the patches should work better on CSS.", "was_revised": true, "final_justification": "I have read the authors' responses to reviews, and I appreciate the clarifications and the few experiments added to the supplementary. I am not however convinced to increase the score, because CSS while visually pleasing and intriguing doesn't provide any further insight into mechanisms of visual processing compared to the existing methods which it was compared to in the paper. However, this line of work is promising, and I am optimistic that next versions of the work could provide such mechanistic insights.", "rebuttal": "Thank you for your thoughtful and constructive review! We appreciate your recognition that this is a “creative way to [...]  isolate configural shape sensitivity, controlling for texture completely”. Below, we address specific comments.\n\n---\n\n> **“CSS is strongly correlated with other shape‑sensitive metrics, so the insight feels incremental.” and “What specific types of failure modes or model differences would CSS detect that previous methods would miss?”**\n\nWe respectfully disagree with the reviewer that the correlation with other shape metrics makes our contribution incremental. The shape-vs-texture bias measure is relative and thus can give a false sense of successful shape sensitivity. For example, shape-vs-texture bias can rise simply by suppressing texture (e.g., as in Stylised, Adversarial, Top-k CNNs), while their CSS hardly changes (Fig. 2C). In other words they are not more sensitive to holistic shape (which the CSS measure tells you clearly and directly). Moreover, CSS explains variance on four downstream, shape-dependent benchmarks far better than shape-vs-texture bias (Sec. 9, Williams test), revealing failure modes that earlier diagnostics would have missed. \nIn other words, we argue that CSS is a more targeted assay of the underlying construct of interest, which is about how models leverage holistic shape processing (beyond local cues). We will clarify this measure's advantage over existing ones in the Camera-Ready version.\n\n---\n\n> **“While MAEs are included in the CSS benchmark and perform moderately, the paper does not explore why they underperform compared to models like DINOv2.”**\n\nYou are correct and we agree that the moderate CSS performance of MAEs merits further explanation (with the page limit we had to make some choice). To address your concern we ran the representational similarity analysis on Hiera-MAEs and two other masked-image-models in our model suite - BEiT (token prediction) and BEiT-v2 (token prediction with CLIP-distilled targets), which share MAE’s mask-and-reconstruct recipe and differ in the semantic level of the prediction target - target either being 8096-D code-book indices from a vector-quantised VAE that was itself trained (without labels) to reconstruct pixels in BEiT or soft probability vectors distilled from a CLIP vision encoder. We find that only BEiT-v2 develops the mid-layer switch from part-based to category-based representation that we had observed in other high-CSS models (see Fig. 4B). The evidence suggests that just masked modeling (both, pixel or latent level reconstruction) is not sufficient in a purely self-supervised setting, instead DINO’s multi-view contrastive loss that enforces global consistency across large spatial crops appears to be critical to unclocking this configural capacity. When such global pressure is absent, comparable CSS can still be reached by injecting language-aligned targets, as seen in EVA-CLIP and BEiT-v2. Thus, not all self-supervision is equal. \n\nWe did not perform attention ablation studies on these MAEs, because probing them with the same radius-mask ablation used for global-attention ViTs is technically challenging without substantial redesign: Hiera-MAE starts with a 7 × 7 conv stride-4, then uses stage-wise local windows that shrink the receptive grid (56 → 28 → 14 → 7). A fixed “mask radius = k tokens” therefore corresponds to four different pixel extents across stages, making a layer-wise comparison impossible without redefining distance at each stage. In stages 0–2 each head already attends only within 8 × 8 or 4 × 4 windows, additional masking outside removes information the model never sees, probing a different phenomenon than our global-attention test. For clarity, we limited Fig. 4 to uniform-grid ViTs (DINO-B/14, ViT-B/16).\n\nWe appreciate this suggestion, which helped clarify which self-supervised objectives drive configural sensitivity. We will include this point in the camera-ready version.\n\n---\n \n> **“High CSS might just reflect heavy spatial augmentations like random cropping in DINOv2.”**\n\nFair point. We think heavy spatial augmentations alone does not explain high CSS.  Our model suite contains models trained with heavy augmentations like RandAugment (Cubuk et al., 2019), Mixup (Zhang et al., 2018) and Cutmix (Yun et al., 2019): \n\n1. Multiple variants of ResNets from Wightman et al., 2021. These models are trained with the A1 augmentation recipe with RandomResizedCrop (scale 0.08–1.0), RandAugment, Mixup/CutMix, color-jitter, and flips. \n\n2. ViT-S/16 AugReg and ViT-B/16 AugReg from Steiner et al., 2021 which applies RandomResizedCrop + RandAugment + Mixup augmentations. \n\nThese augmentations however are still fed to the network independently under a cross-entropy loss, no link is made between different views of the same object. The CSS scores for these models reach up to 36.11%. DINO, by contrast, pairs two spatially distant crops of the same image (local and global view) and forces their embeddings to coincide, so the model is directly rewarded for integrating long-range information that spans the global shape of the object in the image. So rather than random cropping which is also present in the DINO framework, we think that the high CSS scores for DINO models can be attributed to its objective that enforces a global-consistency constraint.\n\n---\n\n> **“Transformer models may benefit unfairly from explicit positional encodings matched to the anagram grid.” and “Have you compared models with different position encodings?”**\n\nReally good point, and we also thought so! We evaluated five new additional variants of the ViT-B/16 architecture from timm, each with a different positional encoding scheme. For context, the standard supervised ViT-B/16 from our paper (which uses learned absolute positional embeddings) has a CSS of 23.61%.\n\nNew Experimental Results:\n\n- ViT with Relative Positional Embeddings: 38.89% CSS\n- ViT with RoPE (Rotary Position Embedding): 51.38% CSS\n- ViT with RoPE (Rotary Position Embedding) + Absolute PE: 51.38% CSS\n- ViT with Mixed RoPE: 48.61% CSS\n- ViT with Mixed RoPE + Absolute PE: 52.77% CSS\n\nThe stronger performance of models with relative position encodings (like RoPE) demonstrates that CSS does reward models for processing relational spatial structure. We will add this new analysis to the appendix to provide a more complete picture of the role of positional encodings. Thank you for this valuable suggestion.\n\n---\n\n> **“Can you clarify whether the representational similarity analyses provide insight beyond the expected hierarchical abstraction of deep networks?\"**\n\nWe agree a simple transition from low-level to task-driven features is expected in deep networks. However, our analysis provides insight beyond this as we found qualitative difference in ‘how’ models make this transition, specifically two distinct patterns:\n\n1. The \"Stuck-on-Local\" Pattern: In low-CSS models, the representations remain dominated by local patch similarity across all layers. For the anagram pair (same patches, different objects), the final representations remain highly similar: the model cannot pull them apart. These models fail to make the full transition to a global representation when confronted with matching local evidence. \n\n2. The \"Configural Flip\" Pattern: In high-CSS models, we see a non-trivial \"representational flip\". While early layers are sensitive to the shared patches, the intermediate layers actively work to overcome this. By the final layers, the representations of the two different objects in an anagram pair are just as dissimilar as if they were built from entirely different patches. The model’s representation becomes fully determined by the global configuration, successfully overriding the misleading matched local cues.\n\nTherefore, the insight is not simply that a hierarchy exists. The novel insight is the existence of this \"configural flip\" as a hallmark of high-performing models and its absence in weaker ones. These high-CSS models are not just going from a local to a task-driven representation, but instead, have an intermediate transition of local enrichment due to long-range contextual interaction which ends up reflecting as the “configural flip” signature. This analysis provides a way to distinguish these models from those whose final output is still contaminated by local feature similarity.\n\n---\n\n> **“Can you evaluate CNN-based self-supervised models such as MoCo, SimCLR, BYOL, or SwAV to determine whether high CSS requires attention, or if it can arise from training objectives and augmentations alone?”**\n\nWe could not include MoCo, SimCLR, BYOL, or SwAV, as pretrained versions with 1000-way classification heads were not available. However, our suite includes SSL-ResNet-50 and SWSL-ResNet-50, which offer partial coverage. We agree this is a valuable direction and plan to test it systematically by holding architecture and data constant while varying the self-supervised objective.\n\n---\n\n> **“Does the 64×64 patch size of the anagrams mean CSS mostly measures layout rather than sub‑patch integration?”**\n \nOf course CSS requires both subpatch processing and global “layout” sensitivity, but it specifically targets the latter. Since both images in an anagram pair share identical parts, models must rely on configural information to succeed. This is why BagNets, which lack long-range integration, perform at chance. \n\n---\n\n>  **“Did you try training or fine-tuning DINOv2 (or a similar ViT) without spatial augmentations [...] to assess whether positional robustness is necessary for high CSS?”**\n\nWe did not retrain DINOv2 without multi-crop augmentations, as this would require full pretraining from scratch on large-scale data, which is beyond the scope of this paper. Our goal is to evaluate emergent properties of existing models. That said, we agree this is an important question, and a controlled comparison isolating the role of augmentations would be a valuable direction for future work."}, {"review_id": "9Is0BZoQ6N", "reviewer": "Reviewer_iP6z", "summary": "The authors introduce the Configural Shape Score (CSS), an absolute metric for holistic shape processing on “visual anagram” pairs. They show that transformer-based vision models reach near-human CSS via learning long-range interaction and that CSS predicts model robustness better than classic bias measures.", "questions": "- How many human participants were recruited for the study? This should be clearly stated.\n - Have the authors plotted “Top1 ImageNet Accuracy” vs. “Shape-vs-Texture Bias”? I believe a plot like this could be interesting if it is shown next to the “Top1 ImageNet Accuracy” vs. “CSS” plot in Figure 2. Would the R^2 be lower for Shape-vs-Texture Bias?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Quality (Good):**\n\nThis paper is technically sound, and the main results were well supported by additional experiments (ablation and disentanglement analyses). The experiments are extensive, although I do agree with the authors that they could have (perhaps marginally) benefited from creating a larger dataset.\n\n\n**Clarity (Fair):**\n\nThe authors did well in establishing the gap in the literature and how their work bridges this gap. Overall, the paper is easy to read. However, the abstract can be substantially improved by making it more succinct. Currently, the abstract (especially its latter part) is only understandable if you have read the entire manuscript, which defies the purpose of having an abstract.\n\n\n**Significance (Good):**\n\nI believe this dataset and CSS can/should be utilized by many vision researchers in the future. The utility of the authors’ framework is very well supported by the experimental results.\n\n\n**Originality (Fair):**\n\nApplying the vision-anagram technique for understanding shape vs texture processing is novel and useful.", "was_revised": true, "final_justification": "The authors resolved my concerns regarding clarity and small sample size, and I cannot think of any more flaws or shortcomings in the paper. This work is novel and well-written.", "rebuttal": "Thank you for your positive feedback on the quality and significance of our work. We truly appreciate your statement that “this dataset and CSS can/should be utilized by many vision researchers in the future,” and we wholeheartedly agree with this perspective.\n\n---\n\n>  **“The abstract can be substantially improved by making it more succinct.”**\n\nYes, we agree. We'll revise the abstract to make it more succinct and group the key findings more clearly. We will edit the abstract to be shorter and more clear in the camera ready version.\n\n---\n\n>  **“The experiments are extensive, although I do agree with the authors that they could have (perhaps marginally) benefited from creating a larger dataset.”**\n\nYou're absolutely right, and we agree that a larger dataset strengthens the findings. We've already expanded the corpus to 1440 visual anagram pairs, a 20x increase, and re-evaluated all models. The results are highly consistent: model scores remain stable, and the relative CSS rankings of the models are preserved, the correlation between the CSS scores on the original and expanded datasets is r = 0.99. These updated results and the full dataset will be included in the camera-ready version.\n\n---\n\n> **“How many human participants were recruited for the study? This should be clearly stated.”**\n\nOnly two participants were tested, as the goal of this study was only to confirm that the images in anagram pairs are easily recognizable. These images are not meant to to challenge human recognition, and the small sample already verify this. \t\nMoreover, the results are not intended for making direct comparisons between model and human behavior, which would require a larger-N (Alvarez & Konkle, 2024). We will clarify the role of these behavioral data in the manuscript.\n\n---\n\n>  **Have the authors plotted “Top1 ImageNet Accuracy” vs. “Shape-vs-Texture Bias”? I believe a plot like this could be interesting if it is shown next to the “Top1 ImageNet Accuracy” vs. “CSS” plot in Figure 2. Would the R^2 be lower for Shape-vs-Texture Bias?**\n\nWe found that adding an extra subplot to the Figure 2 disrupted the clarity a bit, but we instead will put a new figure in the Appendix, which also will show all pairwise relationships among CSS, Shape-vs-Texture Bias, Top-1 Accuracy, and related metrics.  Thank you for your suggestion."}]

Author rebuttal:

Meta-review: This paper introduces the Configural Shape Score (CSS), a novel benchmark using visual anagram pairs to assess holistic shape processing in vision models. It provides compelling evidence that high-performing self-supervised and multimodal transformers achieve stronger configural competence, driven by long-range interactions and mid-layer representational transitions. The authors’ dataset expansion (20× increase) and extended ablation analyses further strengthen the results.

Pros:
* Creative and well-motivated benchmark (visual anagrams) that isolates configural shape sensitivity.
* Extensive evaluation across 80+ models, including CNNs, ViTs, and multimodal systems.
* Strong mechanistic analyses linking CSS to long-range attention and representational “configural flip.”
* CSS correlates with robustness metrics and offers clearer diagnostic power than shape-vs-texture bias.
* Well-written, with thorough responses that add new experiments and clarifications.

Cons:
* Mechanistic insights overlap with known properties (hierarchical processing, long-range integration).
* Limited evaluation of non-transformer self-supervised CNNs (e.g., MoCo, BYOL) leaves some open questions.
* CSS remains correlated with existing shape-sensitive benchmarks, raising questions about incremental value.

All reviewers converge on acceptance, with two recommending spotlight consideration. Despite some concerns about incremental novelty, the benchmark is creative, the analyses are careful, and the expanded dataset makes it a valuable contribution. The work is likely to have significant impact in both neuroscience-inspired ML and vision research.
Decision: Accept (poster)