Does resistance to style-transfer equal Global Shape Bias? Measuring network sensitivity to global shape configuration

Authors: ziqi  wen
tianqin  li
zhi  jing
tai sing lee
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-01 08:15:16

Abstract: Deep learning models are known to exhibit a strong texture bias, while human tends to rely heavily on global shape structure for object recognition.  The current benchmark for evaluating a model's global shape bias is a set of style-transferred images with the assumption that resistance to the attack of style transfer is related to the development of global structure sensitivity in the model. In this work, we show that networks trained with style-transfer images indeed learn to ignore style, but its shape bias arises primarily from local detail. We provide a $\textbf{Disrupted Structure Testbench (DiST)}$ as a direct measurement of global structure sensitivity. Our test includes 2400 original images from ImageNet-1K, each of which is accompanied by two images with the global shapes of the original image distorted while preserving its texture via the texture synthesis program. We found that (1) models that performed well on the previous cue-conflict dataset do not fare well in the proposed DiST; (2) the supervised trained Vision Transformer (ViT) lose its global spatial information from positional embedding, leading to no significant advantages over Convolutional Neural Networks (CNNs) on DiST. While self-supervised learning methods, especially mask autoencoder significantly improves the global structure sensitivity of ViT. (3) Improving the global structure sensitivity is orthogonal to resistance to style-transfer, indicating that the relationship between global shape structure and local texture detail is not an either/or relationship. Training with DiST images and style-transferred images are complementary, and can be combined to train network together to enhance the global shape sensitivity and robustness of local features. Our code will be hosted in github: https://github.com/starsky77/DiST

Main paper: 
 introduction: 
 Deep learning models for object recognition are known to exhibit strong texture bias (Geirhos et al., 2018; Baker & Elder, 2022) . In solving problems, neural networks tend to discover easy shortcuts that might not generalize well (Ilyas et al., 2019; Drenkow et al., 2021) . Rather than learning a more structured representation of objects, i.e., the global configuration of the local components, a.k.a. global shape structure, or global structure, convolutional neural networks trained for classifying objects rely primarily on the statistical regularities of features discovered along the network hierarchy. Standard networks fumbled badly when the test images were subjected to style or texture transfer (Geirhos et al., 2018) , revealing their reliance on texture and local feature statistics, perhaps the easiest features, rather than global structure. Humans, on the other hand, are fairly robust against such style transfer manipulation in object recognition, indicative of our explicit utilization of global shape structure (Ayzenberg & Behrmann, 2022; Ayzenberg & Lourenco, 2022; Quinn et al., 2001a; b) . Such preferences for the global shape structure of the object are so-called global shape bias.Figure 1 : Left: Feature Attribution Analysis based SmoothGrad (Smilkov et al., 2017) on stylized augmentation trained models. Surprisingly, models that can resist style transfers still be primarily sensitive to local features, rather than the global shape configuration. Right: Illustration of our proposed Disrupted Structure Testbench (DiST). We hope machine would successfully distinct the images that have disrupted global structure from the original image, align with human that are using the global shape structure as a cue for object recognition To measure how well models understand the global shape structure, the style-transferred images, specifically, the image that transfers its texture into an image that belongs to a different class, known as the cue-conflict images, are commonly used as the benchmark of global shape bias (Vishniakov et al., 2024; Geirhos et al., 2021) . Various approaches have been developed to steer neural network learning towards shapes from texture (Brochu, 2019; Geirhos et al., 2021; Li et al., 2023) . Based on such a way of measurement, the most effective approach remains to be augmenting the training data with randomized style-transfer operation (Geirhos et al., 2018; 2021) .Despite the common usage of cue-conflict images, a crucial hypothesis underlaying the cue-conflict benchmark is that: if the models are not relying on the local texture, then it relies on the global structure. However, in this paper, we found that even if models are forced to be robust against styletransfer operation through such augmentations, models are still finding a shortcut that is not global shape structure. In Figure 1 , we can see that a stylized trained neural network becomes resistant to the style changes, however, its sensitivity map still shows heavy focus on the local features (the eye of the owl in this case), rather than the global structure (See results of Feature Attribution Results in Figure 1 (a), bright pixels in the middle column indicates the area is sensitive to the model perception, we refer the details to Section A.1). Our experiments show the relationship between global shape structure and local texture is not an either/or relationship, suggesting that showing the model is resistant to the change of texture does not necessarily means it understand the global shape structure.To remedy this problem, we developed evaluation dataset, called DiST (Disrupted Structure Testbench), to directly evaluate the sensitivity to the global shapes structure (Figure 1(b) ). In this dataset, images were transformed to disrupt their global structure while maintaining their texture statistics. We used DiST to perform an odd-man-out test on the various models and human subjects to measure their ability to distinguish the original image from its structure-disrupted variants as a metric of their global structure sensitivity. We found humans far superior to Style-transfer-trained networks in discriminating the differences in global forms. In fact, the Style-transfer network's performance is no better than the standard CNN that has not been subjected to augmented Style-transfer training.Besides, based on DiST, we are able to discover several surprising findings that has not been shown by cue-conflict benchmark. Vision transformers (ViTs) with supervised learning fared no better than standard CNN, contradicting the beliefs that ViTs had captured and utilized the global relationship of object parts in this task. While the self-supervised learning (SSL) method that uses masked autoencoder (MAE) significantly improve the global structure sensitivity of ViT.Along with the DiST dataset, we also demonstrate that networks trained with augmented DiST data also do well in discriminating the global structure of objects if using a carefully designed training approach (we name the method as DiSTinguish). Finally, we found that the DiSTinguish-trained network and Style-transfer-trained network are orthogonal and complementary, as one focuses on global structure, while the other tends to capture robust local feature. Thus, our paper provides a better and deeper understanding of the nature of global shape bias and texture bias within the networks. Moreover, through such evaluation method we are able to find self-attention and positional embedding itself does not necessarily provide perception of global structure, while how the model is trained is the thing that matters. 
related work: 
 Deep Neural Networks (DNNs) have been the cornerstone of the revolution in computer vision, delivering state-of-the-art performance on a wide array of tasks (Luo et al., 2021; Redmon et al., 2016; He et al., 2016; Brown et al., 2020) . However, understanding DNNs has been a vital topic to further advancement of these black box models (Drenkow et al., 2021; Petch et al., 2022; Gilpin et al., 2018) . One aspect of understanding DNNs in vision systems is identifying the biases they might have when classifying images. Two prominent visual cues are local texture detail and global shape structure (Garces et al., 2012; Janner et al., 2017) .Originally, it was believed that DNNs, especially those trained on large datasets like ImageNet, primarily learn shapes rather than textures, as visualization in convolutional neural networks shows clear hierarchical composition features of various levels of object shapes (Zeiler & Fergus, 2014) . This belief was also based on the intuitive understanding that shape structures are more semantically meaningful than textures for most object categories. However, Geirhos et al. (2018) challenged this belief and showed that DNNs trained on ImageNet have a strong bias towards texture. Our work re-examines their proposed style-transfer-based approach and further checks if the model has truly understood the global shape structure of the image. Geirhos et al. (2021) benchmarked various widely used models on the proposed Style-transfer datasets in Geirhos et al. (2018) . Among these, network architectures play a significant role in improving the models' global shape bias. Compared to the convolutional neural networks (CNNs), the newly proposed vision transformer family ViTs (Dosovitskiy et al., 2020) perform significantly better in terms of the Style-transfer based shape bias as well as other corruption-based robustness test measured by Paul & Chen (2022) . However, we observe that supervised trained ViTs yield no significant improvement on our proposed global structure disrupted test, the spatial information of patches is lost as the feedforward continues. While the ViT trained with self-supervised learning (SSL), (e.g. DINO (Caron et al., 2021) ) are significantly better, especially masked autoencoder (He et al., 2022) , which shows even better performance than human. Our Disrupted Structure Testbench (DiST) deliberately compares the representation before and after we disrupt the global structure of the image. One could imagine there could be many random global structure disrupted variants of an original image as the joint spatial configuration of local parts could be arbitrary. To get a quantitative measurement of the models' global structure sensitivity, DiST formulates the evaluation metrics as the accuracy of an oddity detection task. The subjects to DiST (models or humans) are asked to select a distinct image from a pool of choices, which consists of one original image where the global structure is intact, and N global structure disruption variants of the original image (each of which preserves the local patterns) . We pick N = 2 for all the DiST tests as we observe that increases N will not increase the difficulty of the task. 
methods: 
 Figure 3 : Mechanism of computing global structure disruption images. We implement approach proposed in Gatys et al. (2015) . Specifically, we optimize a randomly initialized image (yellow) so that when it passes through a pretrained VGG network, its intermediate layers' gram matrix match the targeted image (blue). This results in preserving the images' local features but randomizing the global structures.Structure Disruption via Texture Synthesis Program Texture Synthesis allows for the generation of images that retain the original texture details while randomizing the global structures. We utilize Gatys et al. (2015) in particular to construct the global structure disrupted images for the DiST oddity detection task. We illustrate the process of this texture synthesis in Figure 3 . For any given target image, I t ∈ R (3,H0,W0) (blue boundary image in Figure 3 ), we want to get I o ∈ R (3,H0,W0) that possess the same local features but disrupted global structure (as shown in yellow boundary images on the right). To achieve this, we initialize tensor I 1 ∈ R (3,H0,W0) using value independently sampled from an isotropic Gaussian distribution and complete the process of I 1 → I 2 → I 3 ... → I o through minimizing the L as Gram Matrix Matching Loss. Specifically,∂Loss ∂I i = ∂ ∂I i l ||Gram(A l (I t )) -Gram(A l (I i ))|| 2 (1)where A l (I) ∈ R (C l ,H l ,W l ) denotes the l-th layers activation tensor from which we destroy the global spatial information by computing the channel-wise dot products, i.e. Gram(A l (It )) ∈ R (C l ,C l ) .DiST Metric Formulation For each trial in the Disrupted Structure Testbench (DiST), two global structure disrupted variants are generated using distinct random seeds. Each image, denoted as I i , is then passed through the evaluation network F to obtain a feature vector F (I i ) from the final layer. The model identifies the image most dissimilar to the others by calculating the cosine distance between feature vectors. The procedure for this calculation is as follows:D i = j̸ =i (1 - F (I i ) • F (I j ) ∥F (I i )∥ 2 ∥F (I j )∥ 2 )/N (2)N represents the number of structure-disrupted images in each trial, in DiST it would be equal to 2. The dissimilarity of the image I i to the other 2 images is calculated as the average of the pairwise cosine distance of each two image pair. Cosine distance of vector u and v is calculated asD C (u, v) = 1 -S C (u, v), where S C (u, v) is the cosine similarity.The model will select the images a that is the most different from the other two images based onD i : a = arg max i exp(Di) j exp(Dj )). DiST is fundamentally different from the evaluation methods based on style transfer or other changes in texture details. Those methods apply style transfer operation to generate the evaluation data. The model trained with stylized augmentation could get an advantage in that evaluation due to the familiarity with different style domains. In contrast, DiST involves no style transfer operations. Instead, it directly assesses the representations learned by the model to show how sensitive it is due to the change in global structure. This approach eliminates any biases arising from familiarity with stylized images, offering an entirely new angle from which to evaluate global shape bias.Human vision is known to exhibit a strong bias towards shape. To quantify the gap between deep learning models and the human visual system, we conducted a psychophysical experiment with human subjects. To align this experiment closely with deep learning evaluations, participants were simply instructed to select the image they found to be "the most different," without receiving any additional hints or context. To mimic the feedforward processes in deep learning models, we displayed stimulus images for a limited time, thereby restricting additional reasoning. Furthermore, participants received no feedback on the correctness of their selections, eliminating the influence of supervised signals.In each trial, participants were simultaneously presented with three stimulus images for a duration of 800 ms. They then had an additional 1,200 ms, making a total of 2,000 ms, to make their selections. Any response given after the 2,000-millisecond window was considered invalid. To mitigate the effects of fatigue, participants were allowed breaks after completing 100 trials, which consisted of 100 sets of images. We accumulated data from 16,800 trials and 32 human subjects to calculate the overall performance of DiST to represent the human visual system. The final results, shown in Fig. 5 , represent the average performance across all participating human subjects. Further details of the psychophysical experiments and how the experiment is conducted can be found in the appendix.Deep learning models are excellent learners when we explicitly define the learning objectives. Stylized augmentation forces the model to learn style-agnostic representation, leading to its impressive performance in the stylized domain. Here we would like to directly force the model to distinguish between the original shape and the disrupted one. We propose DiSTinguish, as shown in Fig. 4 , a simple supervised training approach to explicitly enforce the constraint to guide the model to learn the global structure of the object. Rather than operating within the confines of an n-class classification task, we expand this to 2n classes. Where the loss of the network would be:L(θ) = - 2n i=1 y i log(p i ), where p i is the predicted probability of the sample belonging to class i and y is the one-hot code for the ground truth label of 2n classes. This expansion incorporates structure-disrupted versions of each original class as additional, separate classes. As it's not practical to produce the full structure-disrupted version of ImageNet1K, we applied an approximation here, the details of the approximation methods and their effectiveness are shown in the appendix. 2020)). As shown in Fig. 5 , where both human and model performances are ranked according to DiST Accuracy, we highlight the important discoveries as follows:Firstly, ResNet50-SIN, which is trained on stylized images (Geirhos et al. (2018) ), outperforms other models on the Cue-Conflict dataset but fails to surpass the performance of a normally trained ResNet50 on DiST (52.6% v.s. 69.4%). This suggests that its high performance on the Cue-Conflict dataset may not be attributed to a better understanding of the global structure but rather to some other learned "short-cut". Secondly, the DiST Accuracy of the models is inconsistent with their Cue-Conflict score. This suggests that not relying on texture detail (leads to high Cue-Conflict score) does not necessarily mean using the global structure information (leads to high DiST Accuracy).For the human evaluation result, we average human performance data collected from 16,800 trials to obtain the final human performance metric, which is 85.5%, outperforming almost all the deep learning models. The results consistently show that humans are robust shape-based learners, irrespective of the evaluation method used. 
the loss of global spatial information during supervised learning of vit.: 
 The core of vision transformer (ViT) (Dosovitskiy et al., 2020) : self-attention layer along with the positional embedding are believed to be more expressive than the convolution operation in terms of capturing global structure. However, as depicted in Fig. 5 , such an idea is challenged by the result that ViT models haven't exhibited a significant advantage over ResNet when evaluated on DiST. Notably, ViT-B even underperforms compared to ResNet50. Additionally, although the performance on the Cue-Conflict dataset suggests that a large model size is more robust against texture changes, a larger model size does not guarantee enhanced capability in perceiving global structures. The performance of various ViT sizes on DiST is incongruent with their parameter sizes.We further evaluate the DiST accuracy of the same ViT architecture with different training methods: supervised learning, compared with self-supervised learning (SSL): DINO (Caron et al., 2021) , and Masked Autoencoders (MAE) (He et al., 2022) . As shown in Table .1. ViT trained with SSL has shown significantly better performance on DiST compared with supervised learning. Surprisingly, with 93.7% accuracy, MAE is even better than human performance.To understand why supervised training failed to capture global structure as well as MAE, even if positional embedding should have provided information on the global spatial relationship of each The results indicate that as the layer goes deeper, it becomes harder for the simple network to decode the correct position of the image patches. Notably, it becomes significantly harder for supervised trained ViT to decode the correct position based on the patch embedding compared with ViT trained with SSL. Consist with the result shown by DiST, using the embedding trained with MAE, the simple network can easily decode the position of the patch, indicating that spatial information stored in positional embedding is still maintained as the layer goes deeper, while such information is easy to lose with supervised learning. The attention map of the final layer of ViT in Fig. 6b further shows that ViT trained with MAE tends to have more global attention that covers the whole object, even if the structure has been disrupted, such attention still works globally, while the head of supervised trained ViT focused on local part on both two versions of images. As augmented with style-transferred images force the model to be robust against texture changes, we follow a similar idea to force the model using global structure information. We employ DiSTinguish to explicitly train the model to differentiate between original and disrupted structures, and compare its effect with style-augmentation technique.We evaluate a ResNet50 model under four distinct training approaches. As shown in the Table .2, DiSTinguish significantly enhances performance on the DiST evaluation while maintaining comparable results on the original dataset. Importantly, the effectiveness of DiS-Tinguish is orthogonal to the effectiveness of style augmentation, as those two specially designed augmentation techniques do not significantly influence the performance on the benchmark created with a different technique. Moreover, DiSTinguish is fully compatible with stylized augmentation techniques, which allows for their combined use without any performance degradation in either the stylized domain or the DiST evaluations.We further examine the feature vectors from the last layer of a ResNet50 model trained under different conditions by visualizing them using t-SNE. As illustrated in Fig. 10 , both the baseline model and the one trained with Stylized Augmentation fail to effectively distinguish between original and disrupted structure. Their corresponding classes in the feature space show significant overlap. In contrast, the model trained with DiSTinguish clearly separates feature clusters corresponding to the original structure from those of their disrupted versions. And combining two methods together won't influence such separation.To investigate why Stylized Augmentation fails to significantly improve DiST and how DiSTinguish achieves better performance, we employ SmoothGrad (Smilkov et al. (2017) ) to generate sensitivity maps for a ResNet50 model trained using either DiSTinguish or Stylized Augmentation. Sensitivity maps reveal how responsive the model is to changes in pixel values. To clearly show which regions contribute most to the model's internal representation, we use a binary mask to mask out the pixels that have low sensitivity values.Across various stylized images, models trained with Stylized Augmentation tend to focus on specific local features that remain relatively invariant to changes in style. For instance, as depicted in the row (ii.), (iii.), and in Fig. 7 , a stylized augmented model may rely heavily on a single eye as the key feature for its decision-making. We hypothesize that the feature associated with the eye remains stable even when the style domain undergoes significant alterations. This enables the stylized augmented model to classify the image correctly despite changes to many texture details. However, this strategy fail in the DiST evaluation, where the global structure is altered but local features remain constant. As illustrated in row (i.) of Fig. 7 , the stylized augmented model fails to account for the global structure, concentrating solely on distinctive features like eyes and neglecting other regions.In contrast, models trained with DiSTinguish are compelled to make use of global features to effectively differentiate structure-disrupted images from original ones. Consequently, the model's sensitive regions are not confined to small, local areas; rather, they extend to larger, global structures. As shown in row (i.) of Fig. 7 , the model is highly responsive to most parts of the owl, even when they are spatially separated, thus enabling it to perceive changes in the global structure. This sensitivity to global features persists even in style-transferred images. Compared to stylized augmented models, which fixate on specific local features such as an eye, models trained with DiSTinguish are sensitive to the entire object. This suggests that the two training methodologies engender fundamentally different feature preferences in models. While DiSTinguish encourages models to focus on global structures to discern between disrupted and original structure, Stylized Augmentation prompts the model to rely on features that remain stable across various style domains as a defense against style transfer operations. Experiment on ImageNet10 In the small-scale experiment of ImageNet10, the class we select are exactly the same as the Imagenette dataset. The class label of the selected class are n01440764, n02102040, n02979186, n03000684, n03028079 n03394916, n03417042, n03425413, n03445777, n03888257. The model is trained by using SGD as the optimizer, with learning rate of 0.05, batch size of 256 on a single Tesla V100 GPU for 100 epochs. To eliminate the impact of data augmentation, no special augmentation is appiled during the experiment.The model we used in the experiment of ImageNet1K, except for the baseline model, which directly using the IMAGENET1K V1 weights from pytorch, are trained from scratch using ffcv (Leclerc et al. (2023) ). All the models are trained for 90 epochs with the start learning rate of 0.1 on a signle V100 GPU. Other configuration remains the same as the default configuration in ffcv for training ResNet50.Psychophysical experiments are conducted using a front-end web application developed in JavaScript. Subjects are instructed to "Find the image that is different from the other two" and can select their answers using keys '1', '2', or '3'. After making a selection, subjects press the spacebar to proceed to the next question.The trial procedure is illustrated in Fig. 8 . A set of images appears on the screen after a 300 ms delay and remains visible for 800 ms. In a standard trial, two structure-disrupted images and one original image are presented; the correct answer is the original image. Following the 800 ms display period, the images vanish, and subjects have an additional 1200 ms to make their selection, totaling 2 s for decision-making. If no selection is made within this time, the trial is marked as a timeout, and the response is considered invalid. Subjects are given the opportunity to take a break after every 100 images. To prevent the supervision signal, no feedback on answer correctness is provided during the test.To mitigate the risk of the "oddity pop-out" test devolving into a mere "detection task"-where subjects might focus solely on identifying the original image rather than the one that differs-we incorporate extra catch trials into the experiments, as illustrated in Fig. 9 .One catch trial is presented after every 10 standard trials. In each catch trial, two "original images" are displayed: one is a mirrored version of the other, accompanied by a structure-disrupted image.It is important to note that there is no overlap between the images used in catch trials and those used in standard trials. In these catch trials, the correct answer is actually the structure-disrupted image.The rationale for incorporating such catch trials is to compel subjects to focus on identifying the "different" image rather than the "original" one, thereby aligning the task more closely with how deep learning models behave during DiST evaluation. Results from the catch trials are not included in the final performance metric.In this section, we present the original sensitivity maps without binary mask for a ResNet50 model trained using the four different approaches examined in our ImageNet1K experiment. Sensitivity maps serve to illustrate the model's responsiveness to pixel-level changes; a lighter pixel suggests a more significant influence on the model's decision-making process.As shown in Fig. 12 , in line with the findings detailed in the main text, the sensitivity map of a model trained with both DiSTinguish and stylized augmentation qualitatively demonstrates the synergistic  
 
 Appendix:  



Reviews: review: title: title: Interesting new benchmark for shape bias but presentation needs improvement
main_review: main_review: - Clarity: The paper’s aims are clear, but the results and details of the methods are not always fully clear and do not always meet standard of reporting. The paper contains a lot of grammatical and spelling errors.
- Correctness: As far as I can tell the analyses and results are correct, though not many details of methods are easy to find (e.g. how many images are actually used in DIST evaluation and training).
- Novelty: The paper presents a new benchmark to evaluate global shape sensitivity in humans networks using a pre-existing method from Gatys et al., 2017. It presents a suite of new analyses showing how this new benchmark demonstrates that decreasing texture bias in neural networks does not automatically increase global shape bias. 
- Interest to the community: Given the general interest in texture bias as a well-known behavioral gap between DNNs and humans, the paper will be of some interest to this community. However, the paper focuses less on the human side (besides providing a novel accuracy benchmark to achieve, the human data is not further analyzed) and more on the DCNN modeling side, trying to understand why the models fail to represent global shape and what architectures and training objectives can be used to increase this representation.
strength_weakness: strength_weakness: - The paper focuses on the texture and shape bias debate which is a much-studied example in the literature on (mis)alignment of humans vs. deep networks.
- Most of the graphs are insightful and effectively illustrate why the most prominent technique to alleviate texture bias in deep networks (stylized augmentation) does not increase global shape representation.
title: title: Relevant dataset and empirical study on human and model robsutness towards global shape bias
main_review: main_review: The paper focuses on a relevant and interesting problem of deep learning architectures for computer vision, which is global shape bias. The paper contrasts global shape bias with resistance to style-transfer capabilities and thus is largely inspired by previous works [1,2]. I see its main contribution in the global shape dataset and its generation approach. Additionally, the human evaluation and model comparison of cue-conflict and DiST-Accuracy are interesting and relevant. I think the organization and clarity of the paper are good. I think the paper would benefit from a more in-depth discussion section to contextualize their findings and to gain more conclusive insights. There are a number of typos and grammatical errors that need to be fixed, e.g. on page 1: model understand -> models understand, that belong -> that belongs.

 [1] Robert Geirhos, et al.. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231, 2018.

 [2] Robert Geirhos, e al.,. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34:23885–23899, 2021.
strength_weakness: strength_weakness: I think the Inclusion of a human evaluation study to measure how model and human ratings of global shape compare is relevant for the workshop.

The author’s provide an interesting dataset DiST and its generation approach to the community, which may be useful for future studies to investigate model robustness.

The´ evaluation of DIST performance was done over a wide selection of models and offers a good starting point for future work.

The gram-based generation approach to break global shape is a relevant contribution.
title: title: Global structure sensitivity
main_review: main_review: This paper studies machine learning models’ understanding of global shape structure. The first contribution of the paper is a test bench (DiST) as a direct measurement of global structure sensitivity. The authors then conduct extensive experiments to evaluate various models on this benchmark, to examine vision transformer and masked autoencode’s ability to learn global structure, and to show that global structure sensitivity is orthogonal to resistance to style transfer. 

Several findings in this paper are original and significant, in particular the independence of a model’s reliance on local textures and on global structures, and the failure of ViTs on the proposed benchmark. The experiments are comprehensive. The psychophysical experiment is convincing in quantifying the gap between deep learning models and the human visual system. The writing is very clear.

I did not find significant weaknesses in this paper. However, below are some questions that might help clarifying a few details:
- Does the DiST metric satisfy the triangle inequality, which is commonly required for distance measures? 
- Why is learning global structures important? Is it correct to say that models that understand global structures well are closer to human reasoning?
- There is a repeat sentence in the conclusion. (Not relying on …)
strength_weakness: strength_weakness: Novel perspective and findings, as well as impressive experiment results.