SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The key idea of our framework (expanded from [46]) was to use neural features of an adversariably trained network as automatic 'concept' detectors, and further to use their activation maps as soft segmentations. One may argue then that the features discovered are biased to the underlying model used. However, we argue that the primary patterns that models rely on are generally the same. Note that the existence of patterns predictive of a class is determined by the data distribution and not the model. While models may learn to rely on different patterns and may also vary the degree to which they rely on the same patterns, we believe that the most predictive patterns will be detected and relied upon by nearly all models. In our experiments, we find that the class-wise spurious gaps are highly correlated for any two pairs of models, with average correlation of  $r^2 = 0.69$  (see figure 14). We also observe
Visual attribute
Highlighted visual attributes
(a) Visual attribute
Figure 18: Mechanical Turk study for discovering spurious features. This figure is from Singla and Feizi [46] and included here for completeness.
Object category name(s): admiral
Object supercategory: inverterbrates
Object definition: any of several brightly colored butterflies
Below you can find some images containing the object. Note that other possibly larger objects may also be present in these images:
(b) Main object
If you are still unsure about the object meaning, you may also consult the following Wikipedia page(s): https://en.wikipedia.org/wiki/Limenitis
Please answer the questions below
Q1. Which of the following best describes the focus of the red region in the Highlighted visual attributes?
- Amplified visual attributes can be useful in some cases when the red region overlaps over multiple objects such as in these examples: overlap between cougar and wires, overlap between groom and flowers.
- You can ignore Amplified visual attributes when the red region does not overlap over multiple objects.
The main object has the following name(s): snowplow, snowplough
The main object belongs to the following supercategory: vehicles
Focus is on the background of the main object.
Examples: greenery in the background of grasshopper, water in the background of albatross, flower in the background of bee, baseball pitch in the background of player.
Focus is on some other object (not a part of the main object).
Examples: baby is not a part of bassinet (baby's bed), legs are not a part of mailol (dress for dancers), tree branch is not a part of bird, graffiti is not a part of freight car, flowers are not a part of groom, wires are not a part of cougar.
Focus is on some part of the main object.
Examples: nose of a cougar, whiskers of a cat, legs of a lobster, structure of a steel bridge.
Q2. How confident are you about your answer to Q1?
(1) Not confident at all
(2) Slightly confident
(3) Somewhat confident
(4) Fairly confident
(5) Completely confident
Q3. Please justify your answer to Q1 by describing the highlighted visual attribute? This question is mandatory for completing this HIT.
Example answers: focus is on nose which is a part of cougar; focus is on wires that are not a part of cougar or its background. focus is on flower in the background of bee.
(c) Questionnaire
that the accuracy drops experienced due to targeted corruption of input regions where certain features reside for a ResNet and a vision transformer are highly correlated too ( $r^2 = 0.724$ ) (see figure 11). These results suggest that models are sensitive to the same data patterns and even rely on the different pieces of information (i.e. color, shape, texture) captured in the data patterns in the same ways.
(a) Heatmaps highlighting the visual attributes
(b) Questionnaire
Figure 19: Mechanical Turk study for validating heatmaps
To be perfectly clear, we do not claim to have captured all spurious (or core) dependencies, mainly because we only inspect the 5 most important robust neural features per class. We acknowledge that another reason for certain dependencies to be overlooked is because we use a single model to detect and measure the presence of concepts. However, this only occurs for patterns that other models are sensitive to but the adversarially trained model is not. The main patterns that fit this description are adversarial ones, which contain no semantic meaning and thus arguably should be overlooked for
our purposes. In other work, adversarially trained models were found to have increased reliance on spurious features [28]; this result was also corroborated in our spurious gap analysis. Thus, if anything, it seems like using an adversarially trained model may overlook core features, favoring spurious ones. Seeing as the goal of our analysis is to detect and assess spurious feature dependencies, we believe using robust neural features from an adversarially trained model is warranted, especially given the enhanced interpretability of this model, which allows for reliable heatmaps and feature attacks.
I.4 Our Contributions
We now detail the novel contributions we make to the Salient ImageNet framework and analysis from [46]. The most crucial contribution is the notion that to interpret and improve spurious correlation robustness, one can seek to simply reorganize the data they already have, as opposed to collecting more samples or additional instance-wise supervision (i.e. segmentations, manual annotations beyond class-label) that would require fundamental changes the training strategy. Using spuriosity rankings, we gain new and surprising insight into the nuanced ways that models rely on spurious features. Namely, we find that contrary to common belief, models can perform worse when spurious features are present than when they are absent (i.e. because of spurious feature collision, as some spurious features are depended on for multiple classes). Further, our spuriosity rankings naturally lends itself to a simple, effective, and efficient manner to both measure the biases caused by spurious feature reliance, and mitigate theses biases for any model. Finally, we make a crucial adjustment to the feature discovery framework of [46] that removes its most costly and at times prohibitive step (adversarial training). Observing that the framework can be applied by simply fitting a linear layer atop a frozen adversarially robust feature encoder vastly increases the usability of this framework, cutting the time and data required to perform feature discovery by huge margins.
We now take a closer look at our method for measuring bias caused by spurious feature reliance compared to the one proposed in [46], as we believe our method corrects crucial shortcomings of the prior work. Recall that the prior evaluation framework consisted of corrupting core or spurious regions by adding Gaussian noise multiplied by soft segmentations of core/spurious regions obtained via Neural Activation Maps of annotated robust neural features. The drop in accuracy due to these regional corruptions was used to measure the importance of the region. We enumerate the shortcomings of this prior framework, and explain how our method evades these concerns, below.
- Introduction of a synthetic distribution shift. The previous framework required taking images out of distribution by synthetically corrupting core and spurious regions with Gaussian noise. Model behavior on these non-natural images may not truly reflect how the model would behave on the more realistic distribution shift caused by breaking spurious correlations.
- Instability. Adding noise introduced hyperparameters, like the amplitude and norm of the noise, which we observe the metric to be undesirably sensitive to. In our method, the only hyperparameter is the number of images we include in the sets of highest and lowest spuriosity images. We verify our results our stable with respect to this hyperparameter.
- Inability to compare across diverse models. Because models have varying robustness to Gaussian noise (e.g. some more recent models, especially vision transformers which utilize heavier regularization to compensate for lacking inductive biases of convolutional networks, use noising as a data augmentation during training), drop in accuracy due to Gaussian noise cannot be reasonably compared as a means to measure how models use the corrupted image region. Our metric utilizes accuracy, which is standard and established for model comparisons. Moreover, we can appeal to the notion of effective robustness [27] to further control for varying accuracies of the models we consider.
Other contributions that are less novel but still impactful include massively expanding the scale of the feature annotation analysis and creating a web interface to view the discovered dependencies. Specifically, we annotate over  $4 \times$  more classes, yielding interpretation of feature dependencies for all 1000 classes in ImageNet. Given that this benchmark is ubiquitous, we believe a deep dissection of what one model perceives as important patterns in ImageNet can be extremely informative in better understanding how machine perception differs from human perception. We also perform the MTurk study to demonstrate the generalizability of our feature annotations for the top  $20^{th}$  percentile of images based on feature activation as opposed to the top  $5^{th}$  percentile, leading to reliable feature
Class: 350 (Alpine Ibex), Feature: 389
This feature is spurious for class Alpine Ibex. It is the third most important feature for class Alpine Ibex.
Below, we display the five Alpine Ibex images that most highly activate the feature, along with the Neural Activation Map (heatmap) and Feature Attack for each image. These are the same as the images shown to crowdworkers to annotate this class-feature pair.
We additionally provide the full responses of the five crowdworkers that annotated this pair, as well as links to other related pairs (i.e. same class or same feature).
MTURK WORKER ANNOTATIONS
We now show the responses from five crowdworkers used to annotate this class-feature pair as core or spurious. Workers first denote if the focus of the image is on the main object (core), or a separate or background object (spurious). They also mark their confidence on a scale of 1 to 5, and provide written explanations for their annotation.
Main Focus	Confidence	Explanation
Main Object	4	Focus is on some part of the main object ibex, capra ibex leg and head are focus
Background	3	The main object is capra ibex but the focus is on background.
Background	5	Focus is on mountain in the background of capra ibex
Main Object	3	Red focus is on ibex.
Separate Object	5	Focus is on mountain which is not part of ibex
Votes for core: 2, Votes for spurious: 3. Feature is thus annotated as spurious for the class.
RELATED CLASS-Feature PAIRS
IMPORTANT FEATURES FOR ALPINE IBEX
There are 3 features annotated as core and 2 features annotated as spurious for Alpine Ibox Core features include 462, 1207, 1526. Spurious features include 389, 820.
CLASSES WHERE FEATURE 389 IS AMONG MOST IMPORTANT (I.E. TOP-5)
Feature 389 is core for the following classes: Mountain, Valley, Volcano. Feature 389 is spurious for the following classes: Marmot, Alpine ibex, Ski.
Figure 20: Example of a landing page for one class-feature dependency. We present all visualizations shown to MTurk workers to annotate the feature as cor or spurious, as well as the annotations and explanations given by the MTurk workers. Further, we provide links to pages for other features relevant to the class, and other classes that also rely on the feature.
soft segmentations for many more images. Lastly, we create a website, which, while simply a user-interface, will greatly increase ease of viewing and transparency into our analysis, as we show all the visualizations used to perform our core/spurious neural feature annotations. The website can be found at https://salient-imagenet.cs.umd.edu. We present screencaps of pages from the website in Figure 20 and Figure 21. As a web-interface, our work is now accessible to people who do not code or lack the resources to generate our visualizations on their own devices.
COMPLETE LIST OF CLASS-Feature PAIRS
We catalog all 5000 annotated features below. Click any table cell to navigate to a page designated to the corresponding class-feature pair.
Classes are sorted by the number of spurious features. Spurious features are in cells with red backgrounds.
Alternatively, use ctrl-f to find a class directly.
Class	Class
Index	1st Feature	2nd Feature	3rd Feature	4th Feature	5th Feature
Dog Sled	537
Howler Monkey	379
Seat Belt	785
Ski	795
Sunglasses	837
Swimming Cap	433
Balance Beam	416
Gymnastic Horizontal Bar	602
Patio	706
Hockey Puck	746
Figure 21: Screencap of a main table from our web-interface for accessing all 5000 class-feature dependencies. One can simply search for a class by its name or ImageNet class index to see the five features deemed most important for (i.e. contributing the most to the logit of) the class. Each image then links to an individual landing page designated for the class-feature pair (see Figure 20 for an example).