Corner Cases: How Size and Position of Objects Challenge ImageNet-Trained Models
Authors: Mishal Fatima
Steffen Jung
Margret Keuper
Venue: Accepted by TMLR ()

Abstract: Backgrounds in images play a major role in contributing to spurious correlations among different data points. Owing to aesthetic preferences of humans capturing the images, datasets can exhibit positional (location of the object within a given frame) and size (region-of-interest to image ratio) biases for different classes. In this paper, we show that these biases can impact how much a model relies on spurious features in the background to make its predictions. To better illustrate our findings, we propose a synthetic dataset derived from ImageNet-1k, Hard-Spurious-ImageNet, which contains images with various backgrounds, object positions, and object sizes. By evaluating the dataset on different pretrained models, we find that most models rely heavily on spurious features in the background when the region-of-interest (ROI) to image ratio is small and the object is far from the center of the image. Moreover, we also show that current methods that aim to mitigate harmful spurious features, do not take into account these factors, hence fail to achieve considerable performance gains for worst-group accuracies when the size and location of core features in an image change. The dataset and implementation code are available at \url{https://github.com/Mishalfatima/Corner_Cases}.

Published in Transactions on Machine Learning Research (08/2025)
Corner Cases:
How Size and Position of Objects Challenge ImageNet- Trained Models
Mishal Fatima University of Monnheim
mishal.cccima@@uiimmmmiiii
Steffen Jung
sheffennjuung@uunimmmmmmiiiiie
Max Planck Institute for Informatics, Saarland Informatics Campus University of Mannheim
Margret Keuper
keuper@uun-momnteimn.ck
University of Mannheim
Mas. Planck Institute for Informatics, Saarland Informatics Campus
Reviewed on OpenReview: https: // openreview. net/ forum? id- Ygg2Bhafy2
Abstract
Backgrounds in images play a major role contributing to spurious correlations among different data points. Owing to aesthetic preferences of humans capturing the images, datasets can exhibit positional (location of the object within given frame) and size (region-of-interee to image ratio) biases for different classes. In this paper, we show that these biases can impact how much a model relies spurious features in the background to make its predictions. To better illustrate our findings, we propose synthetic dataset derived from ImageNet-1k, Hard-Spurious- lmagee,,,,,, which contains images with various backgrounds, object positions, and object sizes. By evaluating the dataset on different pretrained models, we find that most models rely heavily on spurious features in the background when the region-of-interes (ROI) to image ratio is small and the object is far from the center of the image. Moreover, we also show that current methods that aim to mitigale harmful spurious features, do not. take into account these factors, hence fail to achieve considerable performance gains for worst-group accuracies when the size and location of core features in image change, The dataset and implementation code are available at
com htt.thhww...oooiaaaaaaaaaaaaarna Cases
1 Introduction
Spurious features are defined as features that are predictive of the class label without being directly related to it. Such features are usually helpful for object recognition when the object is placed a perfect environment or context. An example of that would be a sea lion near body of water. This because most models learn associate water with sea lions and vice versa. On the contrary, spurious features can be extremely harmful when the object the "core" features background. This scenario can be fooled casily to predict the label from the spurious cues in the background without relying on "object' or "core" features in the image itself. Recently, a plethora of techniques have been proposed to mitigate the reliance on unnecessrr cues for image classification. (Sagawa et al. (2020) introduced a distributionally robust optimization tochnique which, coupled with strong regularization, helped in achieving high accuracies for data groups that have strong spurious feature reliance. Similarly, Kirichenko et al. (2023) address this problem by retraining the last. layer of DNN using equal data points from different groups with core and
observed in unusual environment against spurious happen when the model is deployed in the wild. Deep neural networks (DNNss)

---

Published in Transactions on Machine Learning Research (08/2025)
a) Tench
b) Zucchini
<) Goldfish
d) Sea Lion
Figure 1: Gradcam visualizations for Pre-trained ConvNext-Base. a) Model predicts core class Tench when the object is located in the center of the image, b) Spurious class Zucchini is predicted when the core class moves away from the center, c) Class Goldfish is predicted when the size of the core object is large (112 x 112), d) Spurious class Sea Lion predicted when size of core object reduces to x 84.
spurious backgrounds. These methods are helpful when the test set exhibits similar biases as the training data, yet they fail achieve similar performance gains when these biases are explicitly (see Figure 1).
Biases in datasets can hugely impact a deep neural network's performance. Earlier works have proven that convolutional neural networks not entirely translation invariant and have the capacity to learn location information about objects (Biscione & Bowers 2021). Some studies have found that models perform poorly on untrained locations [Biscione & Bowers 2020). Similarly, object size within an input frame can lead to models performing badly when the sizes differ at inference time. The deep learning community has tried to mitigate the effect of these biases by proposing different data augmentation techniques that ensure that models are robust to changes size and locations of the objects. However, the impact of the aforementioned factors in the presence of spurious features remains less explored.
In this work, we try to answer the questions: In the absence of position and object size biases, how much do pre-trained models rely on spurious backgrounds to make their predictions? Are current techniques that mitigate harmful spurious features enough to tackle this problem? Specifically, the contributions of our work are as follows:
We calculate centeredness and size scores of different classes in ImageNet- 1k (Deng et al. 2009 and analyze their relation with the level of spuriousity present in that class.
We derive a dataset from ImageNet-1k, called Hard-Spurious- ImageNet,, containing objects against spurious backgrounds with varying sizes and positions.
With the help of experimentation and ablation, we conclude that the size and location of the object should be taken into account when trying to mitigate harmful spurious correlations in the dataset.
2 Related Work
2.1 Spurious Features
Moayeri et al. 2022a show that adversarial training increases model reliance spurious features. They also show that increased spurious feature reliance occurs when the perturbations added to core features are too small to break spurious correlations. Murali et al. (2023) show that spurious features are related with a model's learning dynamics. Specifically, "easier" features learnt in the start of model training can hurt generalization, Neuhaus et al. (2023) proposed a method to identify spurious features in the ImageNet- 1k dataset and introduced a fix mitigate a modol's dependence on these features without requiring additional labels. While the proposed methods mitigate spurious feature reliance are helpful in many cases, their efficacy is less known when factors such size and location of core features in image change. For example
2

---

Published in Transactions on Machine Learning Research (08/2025)
Toyshop
Bookshop
Bolo
Classes
Aircraft Carric-
Dung Beetle
Ant
Bearskin
Horizontal Rar
Volleyball
Center
Beam
Size
0.0
0.2
0.4
0.6
0.8
1.0
Scores
Figure 2: Left: ImageNet-1k classes and their center and size scores. Toyshop has largest center and size scores, whereas Volleyball has smallest center score and Balance Beam has smallest size score. Other classes are sampled randomly for visualization. Right: Counts in log scale of relative centers of ground truth bounding boxes containing the object corresponding to the image class (ImageNet-Ik validation set). Most. object centers are concentrated around the image center, while some are present along the main axes. Objects of interest are rarely present in image corners.
Alshami et al. 2025 propose self-supervised feature masking to reduce model reliance spurious features. Prasse al. (2025) propose localized concepts in concept bottleneck models, making reliance on spurious features inspectable.
2.2 Existing Datasets
To study distribution shift robustness, Sagawa (2020) introduced the Waterbirds dataset, constructed by compositing bird images from CUB-220-2011 (Wah et al. 2011) ) onto backgrounds from Places (Zhou ct al. 2018 to induce a spurious correlation between bird species and scene type. Its test split includes counter-correlated "worst-group" examples that challenge models relying spurious background features. Xiao et al. (2021) investigate how strongly deep learning models depend on background cues, often spurious correlations, when classifying images in ImageNet-1k. Their central question whether models truly recognize the object itself or are misled by its surrounding context. To explore this, they introduce ImageNet-9, a reduced version ImageNet-1k containing nine broad classes (e.g., dog, bird, vehicle). Using bounding boxes, they separate foregrounds from backgrounds and then recombine them in various controlled ways to test model sensitivity. Our work builds these idcas but differs in scope: we use the full set of ImageNet-I1 1k classes and focus on how spurious background correlations interact with object size and spatial location.
Moayeri et al. (202bb) propose dataset derived from ImageNet-Ik 1k with segmentation masks for a subset of images. These masks label entire objects and various visual attributes. They name this dataset RIVAL10 and also lest different models' sensitivity noise in backgrounds and foregrounds. Moayeri et al. (2022c) propose dataset with segmentation masks for images 15 classes of ImageNet- These images have high spurious features. They attribute this objects being small and less centered in these images. Singla & Feizi 20022) label spurious and core features for ImageNet-lk samples. They achieve this by making use of activation maps soft masks. |Maayer et al. (023) rank images in ImageNet-1k dataset based on spurious cues present. They show that spurious feature reliance is influenced more by the data a model is trained on rather than how a model is trained. Lynch et. al.] (2025) propose a photo-realistic dataset with many-lo-many spurious correlations between different groups of spurious attributes and classes. One work closely related to ours is that of Yung et al. (2022). They do a fino-grained analysis of the robustness of different models by varying factors such as object size, location, and rotation. Our technical contributions differ from theirs because we take into account the spuriousity level of backgrounds and correlate with the above factors well.
2.3 Biases in Datasets
While capturing images through camera, humans often tend to place the region of interest in the center. Due to this, there often exists bias in classification datasets where objects are mostly located in the center
3

---

Published in Transactions on Machine Learning Research (08/2025)
Avg. Center Score 0.747
Sixe Soore 0.417
20
0
0.4
0.8
0.4
0.8
Centter Scores
Sizee Scores
Figure 3: Histograms showing distribution of scores in different classes of ImageNet- 1k validation dataset.
of images and away from the boundary the image. Exploiting the center bias ImageNet--lk resizing and center cropping has been usually usod for testing image classification models. Tacsiri al. (2024) show that there exists a strong center bias in out-of-disttrbuttion benchmarks such as ImageNet-A (Hendrycks al.) 2021) and ObjectNet (Barbu et al. 2019 by using resize and center crop operations only. They resize the image to multiple scales and patchify it, followed by a center crop operation at every patch. Doing this, they end up with different zoomed-in versions of the input images. The computed accuracy of the center crop is maximum showing the presence of a strong center bias in the dataset. In this paper, we do an in-depth analysis the presence of center and size bias in every class ImageNet-Ik 1k by computing distinct scores. The detailed explanations of these scores are given following sections.
3 Biases in ImageNet
In this section, quantitatively analyze positional and size biases present in ImageNet-Ik: To get better sense of these biases, propose centeredness and size scores. It is important to note that these scores are calculated by the provided ground truth bounding box of ImageNet-1k.
3.1 Centeredness Score
In the majority of images in ImageNet-1k, the objects of interest are located in the image's center. Hence, in this paper, we use "positional" and "center" as synonyms. To understand the extent of center bias prevalent in ImageNet-lk 1k, we propose a Center Score defined as
1 M N M
C.
E1
(114, - Oss.elloo):
(1)
f=1
where C. is the centeredness score for class M is total number of images in the class, N is total number of objects within a frame, is image center, and O is object center. The distance between image center and object center calculated by the foo norm. We choose this norm based on the fact that images are rectangular grids and the distances are measured in horizontal and vertical terms, not diagonally. It subtracted from to establish direct relationship between the score and center bias prevalent in the class
3.2
Size Score
To measure the average sizes of objects within images, define size score
1 N hywy =1 H.W,W'
S. M
(2)
where S. is the size score for class c, h and refer to the height and width of object j in image i. H and W the height and width of the image itself. Figure (left.) shows the center and size scores of different.
4

---

Published in Transactions on Machine Learning Research (08/2025)
Valdatoon Accunacy of Japateeee Imssgtfts
Vilidaatonn Amounry of Toopsteren TrcspooNs
Rendall's -0 0.206, Speacuan'' -0 -0.382
Kendall's y -0.270, Spearman's P -0.397
Rendall's -0.200, Speacmion' pmm -0.0000.00 -0.416
0.6
0.4
0.4
0.2
0.2
0.2
Center Boores
Htee Soccce
Stee Soccee Center Scoces
Figure 4: Correlation between the validation accuracy on inpainted ImageNet (as explained in Subsection 3.3 and, from left to right, center scores, size scores, and their product, respectively. Jointly considering center and size score shows strongest negative correlation with the accuracy.
classes, with Toyshop having the maximum center and size scores. The histograms in Figure 3 show the distribution of center and size scores of all the classes in the ImageNet-1k validation data. It can be seen that the majority of the classes in ImageNet-Ik are highly centered with objects of interest occupying half of the image pixels on average. These scores are calculated by using Ground Truth bounding boxes of ImageNet-Ike lk.
3.3 Relationship with the Level of Spuriosity
To establish a correlation between centeredness and size scores of every class to spurious feature reliance in ImageNet-1k, we first calculate the validation accuracies of different classes in ImageNet-Ik with object information removed. We achieve this by using Inpaint-Anything (Yu et al. 2023) with the goal of creating a more realistic effect when the region of interest is removed from the image. The imput to Inpaint Anything
are the object bounding boxes and it makes use of Segment Anything (SAM) (Kirillov et al. 2023) to predict masks for objects within these bounding boxes. These predicted masks are then input to the inpainting model LaMa (Suvorov et al. 2022) which fills the masked region predicted by SAM. Finally, we resize the inpainted images to 224 x 224. We use ConvNextt Base (Liu et al. 2022) pre-trained on ImageNet- 22k (Russakovsky et al. 2015 and fine-tuned on ImageNet-lk; to compute the validation accuracies for the inpainted dataset. Classes with higher validation accuracies indicate higher spurious feature reliance, since the model has learnt to associate the class label not just with the core object, but also with the background information. In order to assess the correlation present between center and size scores and the level of spuriousity present in different classes of ImageNet-1k, 1k, we use Kendall's 7 coefficient and Spearman's 's correlation coefficient. The negative correlation values (see Figure 4 depict that there is an inverse relationship between both inpainted data's 's accuracy and the different considered scores, which validates the hypothesis that a higher spurious feature reliance observed in case of non-centered large object sizes. The correlation is overall rather weak, which is to be expected since different classes are differently hard to classify, even from their core features.
4 Pard-SSurrouss-ImaeNee
Similar to the waterbirds dataset (Sagawa et al. 2020), we say that every datapoint (a, y) has an attribute a(x) E A which is spuriously correlated with label y. We conjecture that the strength of the correlation between attribute a(z) and label y is controlled by two factors: size and position p of the core features in the input image. To this end, we propose the problem of spurious feature reliance in the presence of varying object bounding box sizes, locations, and backgrounds. The prime motivation of creating the dataset is to have precise control over these factors and help the community build robust models against stronger spurious cues. We consider the image content within the provided ground truth object bounding boxes for ImageNet-1k as core features and the features outside the bounding box as the background. In ImageNet-lk, bounding boxes are available for all images in the validation data, yet only subset of images in training data are annotated. The images are annotated
set of synthetic datasets to illustrate
5

---

Published in Transactions on Machine Learning Research (08/2025)
CeO
CoO
CeR
CoR
CcO
CoO
CeR
CoR
CeO
CeR
CoR
56x56
84x84
112x112
Figure 5: Different samples from Hard-Spurious-Imageee Image size remains same in all images, i.e. 224 x 224, whereas object size changes. Label of every image is same as foreground object.
and verified through Amazon Mechanical Turk. These annotations provide an estimate of the location of core features in any image. As a first step, we want to disentangle core features from the rest of the image. We achieve this by cropping out the core objects from the images and inpainting the resulting image, as explained in the previous section. Next, we resize core object bounding boxes different sizes, and place them in two different locations against impainted backgrounds. The size and location of core objects and the kind of background chosen, gives rise to different groups in the (sce Figure 5p. To efficiently gauge the performance of these different groups, we calegorize them follows:
Group CeO: Core object in the Center of image against its Original inpainted background.
Group CoO: Core object in the top right Corner of image against its Original inpainted background.
Group CcR: Core object in the Center of image against Random inpainted background.
Group CoR: Core object in the top right Corner of image against Random inpainted background.
We consider three core object sizes: 56 x 56, 84 x 84, and 112 x 112. It is important to note that all the inpainted backgrounds have already been resized to 224 x 224, so the core object sizes mentioned above represent 4 th, az th, and 16th the whole image. We also experimented with object masks obtained from the Segment Anything (Kirillov al.) 2023) model rather than the provided bounding boxes as foreground objects (see Table in supplementary). We observed that the mask quality obtained from Segment Anything model for some objects was not good enough, hence, used provided bounding boxes for this work.
64
Furthermore, we also modify the test set by randomly positioning the object in the image (any random corner instead of top-right) and preserving the aspect ratio (AR) of the object by resizing the shortest size to 56, 84, and 112, while maintaining the AR. cases where the resized size excoods the image canvas (224 x 224), it is downsampled such that the longer size 224 and the shorter size is as close as possible the targel size. The above grouping for the AR preserving variant changes as follows.
Core object resized to preserve aspect ratio placed in the Center of image against its Original inpainted background.
Group CcO:
Group CoO: Core object resized to preserve aspect ratio placed in any random Corner of image against its Original inpainted background.
Group CeR: Core object resized to preserve aspect ratio placed in the Center of image against Random inpainted background.
Group CoR: Core object resized preserve aspect ratio placed in any random Corner of image against Random inpainted background.

---

Published in Transactions on Machine Learning Research (08/2025)
Hard- Spurious-Imaged
Hard- Spurious-ImageNet-D
Text Accuracles
100
56x56
84x84
112x112
56x56
84x84
112x112
Group CcD
80
Group CoO
(%)
60
Group CeR
40
Group CoR
Models
Conv
ResNet-50
CoAtNet
Figure 6: Benchmarking results of different models on Groups CeO (Core object in Center against its Original background), CoO (Core object in top right Corner against its Original background), CeR (Core object in Center against Random background), and group CoR (Core object in top right Corner against a Random background). Performance for our
the worst across all groups.
We report results on both test sets, The results for Hard-Spurios- ImageNNe and its AR-preserving variant. are given in [Tbll I We test the performance of the datasets on different pre- trained architectures: ConvNextt-Base (Liu et al., 2022), ResNet-50 (He et al., 2016), CoATNet (Dai et al. 2021), Hiera-Base with MAE (Ryali et al. 2023), and MViTv2-small (Li et al. 2022 All models are pretrained on ImageNet-I1 1k only. Across all models, the performance on Group CoR for size 56 x 56 is the worst.
4.1 Hard-Spurious-llmageNe---
Randomly chosen backgrounds have varying levels of spuriousity based on the classes they are taken from. We derive a variant of the proposed dataset where, instead of choosing backgrounds in a random fashion, they are chosen based on the level of spurious features present in them. To achieve this, we first analyze the level of spuriousity present in every class. We give inpainted images without the core objects, as input to the pretrained ConvNext-Base model, and record the accuracies of every class. The classes where accuracies are high indicate that the model has learnt to predict the class label without the presence of core objects. On the contrary, classes for which the accuracy is low are highly reliant core features to make predictions. We choose 10 classes that are highly spurious, namely: snorkel, bobsled, maypole, potter's wheel, gondola, bearskin, wolleyball, baskelball, canoe, geyser, and yellow lady' 's slipper as backgrounds. For foreground objects, we choose 10 classes with high reliance on core features such as: bluetick, box turtle, Chihuahua, Japanese spaniel, Maltese dog, Shih- Tzu, Blenheim sponiel, papillon, Rhodesian ridgeback, and basset. We combine the above-mentioned foregrounds and backgrounds to create dataset with 10 classes of foreground objects and highly spurious backgrounds. Similar to before, for every class, the chosen background class remains same for all images belonging to that class, but the backgrounds can differ from one image to another. Finally, we create four groups for the dataset and test on pre-trained models.
5 Experimental Results
We test the robustness of different models with (lard-Spurrouu-ImageNNe and its variants. The results with Hard-Spurious- Saadd-Spurions-Imageee--ll ImageNet-10 are given in the supplementary (see Table 8). Images are normalized with mean and standard deviation of the TmageNet-Ik dataset. We use HuggingFace PyTorc models to test the dataset.
Figure 6 shows test accuracies of the proposed data on three pretrained models. ConvNext- -ase performs best across all CNN architectures, whereas Hiera performs better compared with MViTv2 model. The difference in accuracy between groups CeR and CoR, when the core object size is 112 112 is less across all the models. This indicates that the core feature size is big cnough for the model to ignore changes in location. Moreover, 4th of the number of pixels in the image are occupied by core features in this case, so backgrounds are less exposed as compared to when the core object size is even less. Another interesting observation is that the impact of size change is far stronger on model performance than the location of core features. We also obscrve that in almost all the groups, there is significant drop in performance compared with clean accuracies on the standard validation dataset (see Table 1p. We also see that the pretrained models perform better aspect
7

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1:
Comparison of group test accuracies on Hard-Spurious- Seppurriuu--mmaeNeee between original and aspect-ratio- preeratii---rriinngggg test sets. Scores are shown as Original / AR- A.R-Preserving. Pretrained models consis- tently perform better on the prepervrg variant.
Clean Accuracy
Object Resolution
Group Accuracies (Original / AR Preserving)
Model
CeO
CoO
CeR
CoR
49.69 /
45.57 / 56.03
7.42 /
/ 18.42
Conv.NeXt-Base
84.43
84
53.24 / 67.24
62.17 / 70.34
4.27
19.85
22.38
40.36
112
73.20
78.35
71.21 / 77.91
32.27
50.34
36.53
59.79
56
47.17 / 57.77
42.39 / 54.45
15.15 / 31.11
11.11
29.20
ResNet-50
81.21
84
63.87
71.02
61.50 / 69.64
39.81
54.04
36.16
54.22
112
71.76 / 76.22
70.57 / 76.06
55.90
65.77
55.53
67.43
56
40.59 / 52.32
37.66 / 48.56
6.80
17.71
4.61 / 15.26
CoATNet
82.39
64.78 /
50.99 / 65.01
27.92
44.18
20.30
38.40
112
72.25
77.73
72.08 / 76.84
46.30
61.78
47.64
61.49
56
50.08
60.24
35.66 / 48.57
4.87
15.71
1.36
8.17
Hiera
84.49
84
68.10
/ 74.63
56.54 / 66.74
21.81
/ 41.25
12.49
30.58
112
74.49
80.14
69.89 / 76.92
47.68
64.54
38.05
56.16
42.10 / 54.35
32.07 / 45.83
5.37 / 16.97
1.54 /
MVitv2
83.77
84
67.84
/ 73.37
51.56 / 64.36
30.34
/ 44.98
14.18
32.78
112
70.53 / 78.10
65.15 / 74.93
47.83
63.58
37.72
56.51
ratio preserving variant of the dataset compared to the original dataset. This indicates that in the presence of resolution related artifacts, it becomes more difficult to classify the core object correctly.
Based on the above observations, we divide all the 12 groups consisting of different core feature sizes and locations into three distinct categories: Easy: This set consists of Groups CeO and CoO for larger core feature sizes, i.e. 84 84 and 112 112, these groups seem to be doing considerably better than the rest. Hard: Groups CeR and CoR are the worst performing across all architecture for core feature sizes 56 x 56 and 84 x 84. We categorize them as Hard group. The remaining groups, i.c. groups CeO and CoO for size 56 x 56, and groups CeR and CoR for size 112 x 112 seem to be performing moderately, we put them in Medium category.
Following the analysis done earlier (see Figure 3), we find that most of the images in ImageNet- 1k are centered with an estimatod size score of z 0.5, indicating that on average, the core features in an image occupy half the number of pixels of the entire image. Keeping this in mind, Hard-Spurious- Sww.SSuuuuuuuumeeeeeeee ImageNet consisting of majority and minority groups, where the number of images belonging to majority groups are far more than in minority groups. This is done to replicate the long-tailed distribution nature of the ImageNet- lk dataset in terms of hardness. For the training data, we consider 80 images per group in the Easy category and 10 images from groups in Medium and Hard categories. This brings the total to 400 images per class in the training data. Out of the 400 images, 320 images belong to the Easy group and 80 the Medium and Hard groups. For the validation set, we use a balanced dataset having equal data points from every group. We use 20 images per group, resulting in 240 images per class. Both training and validation set of Hard-Sppriouu-Imaagee is derived from the validation data. The test set is also balanced, comprising 50 images per group, tolaling 600 images in every class. The details of the dataset can be found in Table 11 in the supplementary.
create the training data of
are derived from training data of ImageNet-lk, whercas the test set
5.1 Vision Language Models
We do zero-shot classification with EVA-Giant (Sun et al.) 2023) and CLIP (Radford et al. 2021 models. EVA-CLIP was pretrained on LATON-400M (Schuhmann al. 2021] with CLIP and finc-tuned on ImageNet- 1k. The results (see Table 2) indicate improved performance across all groups, demonstrating the model's effectiveness. Similarly, also report results CLIP with ViT-B/31 backbone (trained on LATON-40MM)

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: Comparison group accuracies for CLIP and E'V--CLIP models under original and AR- AR-preserving variants. Each cell shows original AR value. AR-preserving improves performance across all groups.
Clean Accuracy
Object
Group Accuracies (Original / AR Preserving)
Model
Resolution
CeO
CoO
CeR
CoR
29.81 / 39.83
22.92
31.58
7.51
16.29
4.18
10.99
CLIP
59.25
84
43.06 / 52.02
35.68
/ 45.97
20.65
/ 32.14
14.63
/ 26.11
112
49.53
/ 57.35
45.57
54.42
30.47
42.06
25.95 / 38.23
T2.43
78.58
62.13
72.31
43.09
/ 57.61
26.79 / 45.70
EVA-CLIP
88.87
84
81.88
85.22
78.61
83.11
63.77
73.35
58.38
/ 68.71
112
85.54
87.27
84.03
86.27
76.65
80.33
72.32
77.45
Table 3: Group accuracies for Vision Transformer models under original and A AR-preserving variants. Each cell shows original / AR value. Performance increases both with model scale and with AR preservation.
Clean Accuracy
Object
Group Accuracies (Original / AR Preserving)
Model
Resolution
CeO
CoO
CeR
CoR
56
35.91 / 45.58
27.52
37.27
3.41
10.87
1.83 / 7.66
ViT-Tiny
75.44
84
53.10 / 61.94
46.40
/ 57.07
15.66
/ 31.13
12.68 / 27.54
112
62.98 / 69.74
60.04
/ 67.26
32.41
/ 47.94
30.96 / 46.17
56
46.59 / 57.51
36.25 / 48.36
8.49 / 21.93
4.63 / 15.67
ViT-Small
81.37
84
63.85 / 71.86
57.73
/ 67.60
30.61
/ 47.51
25.49 / 43.03
112
72.48 / 77.62
69.36 / 75.56
50.33
62.65
46.62 / 60.61
56
59.33 / 69.13
48.38
60.64
19.61
37.04
11.09 / 27.02
ViT-Large
85.83
84
74.44 / 80.04
69.49
/ 76.52
47.32
61.37
41.18 / 56.02
112
80.54
83.58
78.19
82.18
64.85
72.74
60.11 / 70.35
by giving the model an image and specifying all class names prompt. The results with EVA show that the performance is much better on hard groups, showing the strength of the model, although the overall trend still persists. Also, EVA is finc-tuned with ImageNet-1k whereas CLIP is not which also cxplains the gap in accuracy and sheds light on the importance of ImageNet pretaining for the proposed dataset. Performance across Al-preserv variant is better than the original dataset.
5.2 Scalability Analysis
In order to do a scalability analysis to check whether large-scale models perform better or not, we take a series models for VIT Dosoviiskiy et al. 2021 increasing in the order of scale (all trained on ImageNet-211 (Russakovsky al. 2015) and fine-tuned on ImageNet-11)) and evaluate the test of the proposed datasct and AR-preseriigg variant (see [Table 3). ViT-Tiny has 5.7M parameters, ViT-Small has 22M, and ViT-Large has 307M parameters. As expected, we see that large models perform better across all groups.
5.3 Training Strategy of Vision Transformer Models
In order to access which training strategy best suited for this task, we report results from two ViTs, all finc-tuned on ImageNet- 1k and roughly having the same paramcter count, i.c., 22M. We note that across images with the original background, the difference performance across both models is minimal, whereas across groups with random backgrounds, ViT-Small performs better, showing that knowledge distillation in DeiT Touvron et al. 2021 not making a huge difference in the classification (see Table 4).
The above results indicate that larger models perform better on the challenging benchmark. Training strategy like knowledge distillation in ViIs is not very useful. VMLs that have been fine-tuned on ImageNet- 1k (e.g. EVA) perform the best on the datasot, even for the challenging group CoR, compared to all other models. We also see that if not fintuned on ImageNet- 1k specifically (see Table 2). we see a sharp drop in performance, is evident in case of CL.PP We also there exists a sharp difference in performance between clean accuracies
9

---

Published in Transactions on Machine Learning Research (08/2025)
Table 4: Group accuracies for DeiT-Small and ViT-Small models under original and preseeveg variants. Each cell shows accuracy as original Baaaall/Apppeeeeeeeeeee AR-preserving. AR-preserving improves performance across all groups.
Clean Accuracy
Object
Group Accuracies (Original
AR Preserving)
Model
Resolution
CeO
CoO
CeR
CoR
56
41.78 / 52.81
28.86
42.00
3.53 / 13.72
1.08
7.52
DeiT-Small 79.85
84
62.73 / 69.33
47.31
60.98
28.08
42.30
10.16
27.89
112
68.02 / 75.09
67.60
73.12
44.12 60.29
38.85
/ 54.56
56
42.42 / 52.40
33.53
44.99
7.86 / 19.87
4.82
15.88
ViT-Small
78.84
84
59.13 / 67.46
54.31
63.84
27.57
43.81
25.34
42.39
112
68.81 / 74.21
66.35
72.30
47.44
59.33
46.19
58.84
Table 5: ResNet-50 baseline versus augmented models on original and aspect ratio (AR) preserving variants of the dataset. Values are shown as Original / AR Preserving within each model variant. Aspect ratio preserving improves performance across most groups, while data augmentation boosts overall accuracy.
Clean Accuracy
Object
Group Accuracies (Original
AR Preserving)
Model
Resolution
CeO
CoO
CeR
CoR
56
38.79 / 49.78
24.59
36.10
10.43
22.78
3.28
12.57
ResNet-50 Baseline
76.14
84
56.47
65.07
45.91
57.50
31.78
46.17
20.74
37.59
112
65.72
71.26
60.18
67.55
49.28
59.59
41.96
54.78
46.56
58.00
34.35 / 47.70
10.76
27.99
2.93
16.93
ResNet-50 Augmented
80.84
84
64.09
72.17
55.24
66.26
39.68
55.06
16.59
42.37
112
71.55
76.73
66.61
74.43
42.30
61.67
35.80
60.47
and group accuracies, indicating the challenging nature of the benchmark. Although the EVA-CLIP model minimizos this difference across many groups, the overall trend persists.
5.4 Effects of Data Augmentation
To measure the effect of data augmentations, we compared vanilla ResNet-50 trained without any augmenta- tions on ImageNet-1k 1k with an advanced training recipe involving auto-angment (Cubuk al.) 2019). random erase [Zhong et al. 2020), mixup (Zhang et al., 2018), and CutMix (Yun al. 2019). The results (shown in Table 5) indicate that while data augmentation increases accuracy across groups CcO, CoO, and CeR, the performance decreases in case of group CoR for all sizes. This indicates that standard data augmentation approaches do not take into account the presence of spurious features in the data while augmenting, hence, may end up highlighting them instead. Morcover, the gap in performance still persists across all four groups for a given core object size. This hints that mere data augmentation strategies are insufficient to deal with this problem. should be noted that clean accuracies differ here slightly because the training recipes in both models are slightly different than what is reported for benchmark.
5.5 Group Robustness Methods
Since the proposed original dataset is hardest of the two variants Periiial/AAR-preserving we measure the performance the original dataset using simple fine-tuning and two state-of-the-ar group robustness methods. Empirical Risk Minimization or ERM (Vapnik) 1991] is the standard paradigm for training machine learning models. Under ERM, models are optimized to minimize the average loss across the training distribution, typically leading high overall accuracy on datasets drawn from that distribution, yet it often fails in real-world settings where spurious correlations group imbalances exist. Deep Feature Reweighting (DFR) KKrichenno et al. 2023) is debiasing method that addresses spurious corrclations by retraining only the last. (classification layer of a pre-trained model. This retraining is done using a balanced subset of the training data, where each group (e.g., defined by some spurious attribute) is equally represented. The intuition is that carlier layers have already learned generalizable fcatures, and reweighting the last layer with graunebalanced data helps reduce reliance on spurious signals without requiring full retraining. Just Train Twice (JTT) [Liu al. 2021) is two-stage method that aims to mitigate bias by identifying and focusing
10

---

Published in Transactions on Machine Learning Research (08/2025)
60
Easy
Test Test Accuracy (%)
Test Accuracy (%)
70
59-
Medium
60
Average
57-
D3R
Epuchs
ERM
ERM
ERM
Figure 7: (left) The effect of training epochs of ERM model on the performance of DFR. ERM model trained with 20 epochs gives the highest performance for DFR. (right) ERMal narrows the gap between easy, medium, and hard groups.
on "hard" examples. First, a standard ERM model is trained. Then, misclassified examples from this model are upsampled in the training data by a factor Aup: under the assumption that these examples are more likely to belong to underrepresented or challenging groups. A second model is trained on this reweighted dataset, helping it better handle these minority or spurious-pront prone cases. We experiment with different variations of the above methods.
5.6 Implementation Details
We use pretrained ResNet-50 trained on ImageNet-lk for our experiments. The Base model (pretrained ResNet-50) is finc-tuned with batch size 256, constant lcarning rate of 0.001 for 20 epochs. The input images are randomly cropped with an aspect ratio the bounds (0.75,1.33) and finally resized to 224 x 224. Horizontal flipping applied afterward. momentum 0.9 and weight decay of 0.001 used. For DFR, we normalize the embeddings using mcan and standard deviation of validation data used to train the last layer, and use the same statistics to normalize embeddings of test data. We re-train the last layer for 1000 epochs, learning rate of 1, cosine learning rate scheduler and SGD optimizer with full-batch. We use l2 regularization with À set to 100. These hyperparamters are similar to the ones set by Kirichenko et al. (2023) for optimizing the last layer for ImageNet-9 dataset (Xiao et al. 2021). Since, the data distribution in the proposed datasct and ImageNet-9 is similar, we assumed the same hyperparameters. In case of JTT, models have the same hyperparameters as the ERM trained model. Aup set to 50.
5.7 Results
The results in Table 6 show that pretrained ImageNet models perform worst on the hard group. This could be attributed to the fact that the model has very little exposure to small core features against spurious backgrounds in the training data. The ERM model does better across easy, medium and hard groups, but there still exists a disparity performance among the three groups.
DFR is able to perform slightly better in the Hard group by sacrificing some accuracy in Easy and Medium groups. The average test accuracy is similar for ERM and DFR. The performance with JTT also decreases, which hints that the task of learning data has become difficult for the model in the presence of upsampled images. Since the embeddings in DFR are dependent on the ERM-trained model, we also analyze how the number of training epochs the ERM model is Irained for, impacts the DFR performance. The epochs for retraining the last layer remain fixed to 1000, all other hyperparameters also remain the same for DFR models trained with different ERM-trained embeddings. The left plot in Figure 7 indicates that, when the base model is fine-tuned for 20 epochs, the performance of DFR on the test set increases. As the training time increases for ERM, performance by DFR decreases, whereas the ERM model continues to improve.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Pre-trained
ERM Say
ERM
Pre-trained
ERM may
ERM al
Whiptail
Partridge
Stick Insect
Partridge
Bolete
Ice-Cream
Figure 8: Gradcam visualizations showing regions of the image the model pays attention to in order to make the classification decision. Labels in red show false predictions and labels in green indicate correct prediction.
Table Test Performance of different methods on Easy, Medium, Average accu- racy is the average test performance of all the groups combined. The model is ResNet-50. Clean represents the performance of the model clean ImageNet validation data. We see that there exists trade-off between the clean accuracies and the performance on newly created dataset.
Table Breakdown test accuracies with the ERMall model. The network architecture is ResNet-50. Compared to the overall trends shown in Figure 6 there is notable improvement in the CoR and CeR. groups, particularly at resolutions of 562 and 842.
and Hard categorics in HarddSppuriou--mmageeee
Methods Clean
Easy
Medium
Hard
Average
Size
CeO
CoO
CeR.
CoR
Pretrained
81.21
65.39
48.50
16.54
43.48
562
62.25
60.8
54.56
54.45
ERM
65.78
74.84
66.67
57.56
65.94
842
73.35
72.60
69.76
69.96
JTT
40.06
60.90
53.09
46.49
53.50
112a
77.19
77.13
75.34
75.48
DFR 69.25
72.47
65.65
59.79
65.97
In case of ERM, we also analyze the effect of the percentage of training data in minority groups, ie. easy and hard groups, on model's test performance. We refer to ERMeasy as the model that has been fine-tuned with data from the majority group only, i.e. 0% of data from medium and hard group. Conversely, we refer to ERM as the model that has been fine-tuned with equal data points from all the groups, and ERM as the standard training data consisting of 20% of data from minority groups.
The results depicted in the right plot in Figure 7
The x-axis shows the differently trained ERM models, iee ERMeasy: ERMa,, and standard ERM, and the y-axis shows performance across Easy, Medium, and Hard Groups. We see that training with the Easy group has worst performance on the Hard group. ERM, seems to narrow the gap between all groups. The accuracy of the Easy group remains similar across the models.
Table presents the subgroup accuracy breakdown for the ERM, model. Compared to the overall trends shown in Figure 6 we observe a notable improvement in the CoR and CeR groups, particularly at resolutions of 56 56 and 84 84. Among all subgroups, the highest alignment with the clean accuracy for ResNet-50 occurs in the CeO group at resolution of 112 112.
5.8 Analysing Classifications with Saliency Maps
We use Gradcam to visualize the predictions on the ResNet-50 model. Figure shows the visualizations the ImageNet-1k 1k pretrained model and two variations of ERM: ERMall which is fine-tuned with equal data points from all the groups and ERMaasy which is fine-tuned only with images from the Easy category, consisting of subgroups CeO and CoO for size 54 x 54 and 112 x 112 respectively.
The images on the left side of Figure & show a stick insect of size 56 x 56 placed in the center against an outdoor environment. The pro-trained and ERMeasy model make their predictions by picking up cues from the backgrounds and predicting class Whiptooi and Purtridge respectively. Upon inspection, we find
12

---

Published in Transactions on Machine Learning Research (08/2025)
that most of the images in these classes set in similar environments, hence the model has learnt to associate the given outdoor environment with these classes and are ignoring the core features. ERM"l, however, more robust to changes in environment. and makes the correct prediction of class Stick Insect.
56x56
112x112
The images the right show that, while the pre-trained model is confused by the spurious cues in the background, ERM makes the wrong predictions based on the cues in the core feal and the background logether. However, ERM makes the correct prediction by mostly relying on core features. Figure 9 highlights the effect of the size of core features on the ability of the ERM model to make correct predictions. Having smaller core feature size results in the model incorrectly predicting class Mushroom.
Mushroom
Goldfish
Figure 9: Effect of core feature size on model performance. Both the predictions are for the ERM model.
6 Challenges and Future Work
The dataset variants of Hard-Spurious- Tmpgrious-TmaceNettaa proposed to understand the extent of background reliance function of size and location of core features. One of the limitations of the dalasets is that they rely on ground truth bounding boxes of objects. In case of images where core features are not labeled by bounding boxes, no inpainting is performed on them, subsequently leading to core features in background and foreground occurring simultancously. Moreover, the presence of secondary objects and clutter in the background makes it difficult for the models to learn small core feature sizes. The lack of segmentation bounding boxes for all images in ImageNNtt-lk restricted us to using object bounding boxes instead of masks. Currently, we have only experimented with one location per core object. For future work, locations of core objects in the images and analyze the impact of using different network architectures with the dataset. Moreover, it would be interesting to extend this analysis to other datasets and models trained in different ways such as with contrastive learning, and various data augmentation techniques.
plan to experiment with different
7 Conclusion
In this paper, we propose a variant of ImageNet-lk, Hard-Spprroou--mmaehhe to help the deep learning community to better understand spurious feature reliance. We show that ImageNet-1k is center-biaser and exhibits a bias towards large object sizes. We also provide an analysis showing that there exists a negative correlation between size and location of core features in an image and the strength spurious cues in the background. To evaluate the robustness of models under these conditions, we experiment with suite of group robustness methods. While some methods show marginal gains, none fully resolve the issue, highlighting the limitations of current approaches in settings where spurious correlations arise from dataset biases.
Acknowledgments
This work was supported by the DFG Research Unit DFG-FOR. 5336 "Learning to Sense". We used computing resources from the University of Mannheim and received support from the state of Baden- Württemberg through bwForCluster Helix and the German Research Foundation (DFG) under grant INST 35/ 1597-1 FUGG. Additional computing time was provided on the high-performance computer HoreKa at the National High-Performance Computing Center at KIT (NHR@KIT).
13

---

Published in Transactions on Machine Learning Research (08/2025)
References
Eyad Alshami, Shashank Agnihotri, Bernt Schiele, and Margret Keuper. ATM: Amending Inherent Inter- pretability via Sell-Supervised Masking, 2025. URL
11502
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Valerio Biscione and Jeffrey Bowers. Learning Translation Invariance in CNNs. In NewrIPS Workshop SVRHM, 2020.
Valerio Biscione and Jeffrey S Bowers. Convolutional Neural Networks Are Not Invariant to Translation, but They Can Learn to Be. Journal of Machine Learning Research (JMLR), 2021.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. AutoAugment: Learning Angmentation Policies from Data. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Zihang Dai, Hanxiao Liu, Quoc Le, and Mingxing Tan. CoAtNet: Marrying Convolution and Attention for All Data Sizes. In Advances in Neural Information Processing Systems (NewrIPS), 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR), 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural Adversarial Examples. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Polina Kirichenko, Pavel Tæmailov, and Andrew Gordon Wilson. Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations. In International Conference on Learning Representations (ICLR), 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Yen Lo, et al. Segment Anything, In International Conference on Computer Vision (ICCV), 2023.
Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. MViTv2: Improved Multiscale Vision Transformers for Classification and Detection. In Conference on Computer Vision and Pattern Recognition caciiioonnC/ (CVPR), 2022.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsca Finn. Just Train Twice: Improving Group Robustness without Training Group Information. In International Conference on Machine Learning (ICML), 2021.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding DINO: Marrying DINO with Grounded Pre- Tre-Training for Open-Set Object Detection. In European Conference on Computer Vision (ECCV), 2024.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConyNet for the 2020s. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Aengus Lynch, Gbetondji Jean-Sebastien Dovonon, Jean Kaddour, and Ricardo Silva. Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases. In ICLR Workshop on Spurious Correlation and Shorteut Learning: Foundations and Solutions, 2025.
Mazda Moayeri, Kiarash Banihashem, and Soheil Feizi. Explicit Tradeoffs between Adversarial and Natural Distributional Robustness. In Advances in Newral Information Processing Systems (NewrIPS), 2022a.
Mazda Moayeri, Phillip Pope, Yogesh Balaji, and Soheil Feizi. A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Mazda Moayeri, Sahil Singla, and Soheil Feizi. Hard ImageNet: Segmentations for Objects with Strong Spurious Cues. In Advances in Neural Information Processing Systems (NeurIPS), 2022c.
Mazda Moayeri, Wenxiao Wang, Sahil Singla, and Soheil Feizi. Spuriosity Rankings: Sorting Data to Measure and Mitigate Biases. In Advances in Newral Information Processing Systems (NeurIPS), 2023.
Nihal Murali, Aahlad Puli, Ke Yu, Rajesh Ranganath, and Kayhan Batmanghelich. Beyond Distribution Shift: Spurious Features Through the Lens of Training Dynamics. Transactions on Machine Learning Research (TMLR), 2023.
Yannic Neuhaus, Maximilian Augustin, Valentyn Boreiko, and Matthias Hein. Spurious Features Everywhere Large-Scale Detection of Harmful Spurious Features ImageNet. In Conference on Computer Vision and Pottern Recognition (CVPR), 2023.
Katharina Prasse, Patrick Knab, Sascha Marton, Christian Bartelt, and Margret Keuper. DCBM: Data- Efficient Visual Concept Bottleneck Models. In International Conference on Machine Leuraing (ICML), 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML), 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
Chaitanya Ryali, Yuan-Ting Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Ya Pao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, el al. Hiera: Hierarchical Vision Transformer without the Bells-and- Bells-and-Whistlee Whistles, In International Conference on Machine Learning (ICML), 2023.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally Robust Neural Networks. In International Conference Learning Representations (ICLR), 2020.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LATON-100M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs, 2021. URL
Sahil Singla and Soheil Feizi. Salient ImageNet: How to discover spurious features in Deep Learning? In International Conference on Learning Representations (ICLR), 2022.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: Improved Training Techniques for CLIP at Scale, 2023. URL https://axxiv. wwtpxivv.org/eeee htgg/abs/23033 2203 15389
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Nacjin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution- Robust Large Mask Inpainting With Fourier Convolutions. In Winter Conference on Applications of Computer Vision (WACV), 2022.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Mohammad Reza Tacsiri, Giang Nguyen, Sarra Habchi, Cor-Paul Bezemer, and Anh Nguyen. ImageNet- Hard: The Hardest Images Remaining from Study of the Power of Zoom and Spatial Biases in Image Classification, In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficicnt image transformers & distillation through attention. In International Conference Muchine Learning (ICML). PMLR, 2021.
Vladimir Vapnik. Principles of Risk Minimization for Learning Theory. In Advances Neural Information Processing Systems (NewrIPS), 1991.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Datasel. Technical Report CNS-TR MA-S-TR-20110011 California Institute of Technology, 2011.
Kai Yuanqing Xiao, Logan Engstrom, Andrew Tlyas, and Aleksander Madry. Noise Signal: The Role of Image Backgrounds in Object Recognition. In International Conference on Learning Representations (ICLR), 2021.
Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint Anything: Segment Anything Meets Image Inpainting, 2023. URL https://arxiis orggabs/2304. 06790
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngioon Yoo. CutMix: Regularization Strategy Train Strong Classifiers with Localizable Features. In International Conference Computer Vision (ICCV), 2019.
Jessica Yung, Rob Romijnders, Alexander Kolesnikov, Lucas Beyer, Josip Djolonga, Neil Houlsby, Sylvain Gelly, Mario Lucic, and Xiaohua Zhai. S1-Score: An image dataset for fine-gre aue-grained analysis of robustness to object location, rotation and size. In ICML Shift Happens Workshop, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond Empirical Risk Minimization. In International Conference on Learning Representations (ICLR), 2018.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random Erasing Data Augmentation. In Associalion for the Advancement of Arlificial Intelligence (AAAT), 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: 10 Million Image Database for Scene Recognition. Transactions on Pattern Analysis and Machine Intelligence (TPAMI),
2018.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Table 8: Test Accuracies on Soar--Spurrous-lmageeeeee with highly spurious backgrounds.
Clean Accuracy
Object Resolution
Group Accuracies
Model
CcO
CoO
CeR
CoR
562
57.4
38.6
8.2
1.0
Convnext-Base
85.8
842
79.0
71.0
27.0
11.4
112
83.2
79.8
45.4
17.0
562
45.00
29.6
6.4
0.0
ResNet-50
82.2
842
70.8
58.4
17.0
9.8
1122
79.4
78.6
35.6
28.6
562
20.9
21.8
0.0
CoATNet
83.4
842
71.8
49.0
11.0
4.0
112
75.40
80.8
18.2
23.0
562
46.8
22.8
1.0
0.0
Hiera
85.8
842
75.6
55.6
4.0
1.8
1122
78.6
74.6
16.0
10.6
562
29.0
15.0
0.4
0.0
MVitv2
86.6
842
72.6
47.8
11.2
1.2
1122
72.4
66.0
18.4
12.6
A Benchmark Results
The results for Hard-Spurros- the datasets 5 different pre-trained architectures: ConvNext-Base (Liu et al. 2022). ResNet-50 (He el al. 2016), CoATNet (Dai et al. 2021), Hiera-Base with MAF (Ryali et al. 2023 and MVit2-small (Li al. 2022 All models pretrained ImageNetlk only. We see that Hard Maar-Spurious- ImageNet-I0 10 has far worse performance on groups CeR and CoR across all architectures and sizes. This indicates that the strength of spurious backgrounds far greater than that of core features when the size of core feat features starts to decrease.
are given in [ Similar to main paper. We test the performance of
Biases in ImageNet
Figure shows the distribution center and size scores for different classes in the training data of ImageNet. We calculate these scores using the available bounding boxes for ImageNet training data. Figure refers the distribution for the validation data.
C Inpaint Anything
The predicted masks from Segment Anything dilated by kernel size of 15 to avoid edge effects when the 'hole' is filled by LaMa. Some examples of the inpainted data are given in Figure 10
Eg: Spanicl
Core: Maliese Dog Potters Wheel
Corc: Shit: Tau
Figure 10: left: Original images with their resized inpainted versions. right: Despite inpainting, the background (Bg) consists of cues that help the model predict the background label.
17

---

Published in Transactions on Machine Learning Research (08/2025)
200
150
50
0.2
0.3
0.8
0.5
0.6
0.7
0.0
0.9
1.0
m100
Avg. Score 0.783
Arrg. 0.482
0.2
0.4
0.6
0.8
0.2
0.4 0.0
0.8
Figure 11: (top): Class-wise mIOU scores between Grounding DINO predictions and ImageNet annotations on the validation set. Averaged mIOU is 0.875. (bottom)::Hisograaan showing distribution of scores in different classes of train data in ImageNetik datasct. In the main paper, we show the distribution for validation data. This is the same plot for training data. The distributions very similar.
D
True Objects in Background
Ensuring that the backgrounds do not contain true objects depends on the fidelity of provided ImageNet annotations. We perform an additional analysis with a foundation model, Grounding DINO Liu et al. 2024), to extract bounding boxes from the images. We consider similarity scores between Grounding DINO predictions and the ImageNet annotations analyze the correctness of ImageNet annotations. For ImageNet validation data, we get an overall mIOU of 0.8675 across all classes between both sets of bounding boxes with 139 classes having mIOU value less than 0.8 (see Figure 11 for histogram by mIOU). This shows that the majority of the classes in ImageNet data have correct bounding boxes and the amount of objects from the foreground class in the background is negligible.
E hard-Spurious-ImageNett with SAM
We also experiment with using the Segment Anything (Krillov al. 2023) model to obtain masks for the objects inside bounding box and resize it to different sizes (56, 84, and 112). The resized masks are then placed in the center and corner of the inpainted image, similar to the setting described in the main paper. At the moment, we only consider one object per image. Since we have access to ImageNet-annotttee bounding boxes, we use them as prompts to be given to SAM. The results are shown in [able 9 Compared the results Table I the results with SAM are worse, mainly because the resized SAM object masks are entirely accurate cases where objects are small and thin, such as insects, etc. Hence, we prefer hunan-annotated ImageNet bounding boxes.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Table Test Accuracies on Hard-Spurious- Imauriee with SAM Masks.
Clean Accuracy
Object Resolution
Group Accuracies
Model
CcO
CoO
CeR
CoR
562
46.07
36.07
13.86
6.21
Convnext-Base
84.43
842
61.18
53.92
31.04
22.30
1122
67.78
64.69
42.91
13.84
562
29.33
24.34
6.68
4.36
ResNet-50
81.21
842
45.17
40.63
19.09
16.24
112
55.24
52.56
31.34
29.87
562
30.57
27.61
7.91
3.93
CoATNet
82.39
842
50.94
44.66
21.03
15.63
1122
60.60
56.73
33.00
29.30
562
37.94
25.88
9.08
2.92
MVit2
84.49
842
54.89
44.73
24.74
15.15
1122
63.73
57.94
36.80
30.60
562
39.88
27.06
10.34
3.198
Hiera
83.77
842
56.36
46.18
25.13
15.72
1122
66.14
60.38
39.15
31.63
F Group Robustness Methods
We use pretrained ResNet-50 trained on ImageNetlk for our experiments. The Base model is fine-tuned with batch size 256, constant learning rate of 0.001 for 20 epochs. The input images randomly cropped with aspect ratio in the bounds (0.75,1.33) and finally resized to 224 x 224. Horizontal flipping is applicd afterward. A momentum of 0.9 and weight decay of 0.001 is used. For DFR, we normalize the embeddings using mean and standard deviation of validation data used to train the last layer, and use the same statistics normalize embeddings of test data. We ro-train the last layer for 1000 epochs, learning rate of 1, cosine learning rate schoduler and SGD optimizer with full-batch. We l regularization with À set to 100. These hyperparamters are similar to the ones set by Kirichenko al. 2023 for optimizing the last layer for ImageNet-9 dataset (Xiao et al. 2021). Since, the data distribution in the proposed dataset and ImageNet-9 is similar, we assumed the same hyperparanteres. In case of JTT, models have the same hyperparameters as the ERM trained model. Aap is set to 50.
After extracting the embeddings from the pre-trained ERM model, the embeddings are normalized using www.pprooessinnntaaaaaaaaa
for val and test data, respectively. For the JTT model, the images are applied with random resized cropping followed by horizontal flipping. No additional data augmentation is applied afterward. We also experimented with Conv-ext-tiny pre-trained ImageNet-22kk and fine-tuned on ImageNetlk. We fine-tune the pre-trained model on the proposed data under various settings. ERM is trained by replicating the long-taide tnigt distribution of the data, while ERM is trained only with the easy group. ERM is trained with equal data points from all groups. DFR trained by extracting embeddings from ERM, and re-training the last layer only. The number train and test. images is similar to the data setting described the main paper.
fit. transform() and transform() functions
G Dataset Details
Table II provides details on the Hard Spurious ImageNet dataset such as the available number of train, validation and test images per resolution and group.
19

---

Published in Transactions on Machine Learning Research (08/2025)
Table 10: Test Performance of different methods Easy, Medium, and Hard categories in Hard--purious- ImageNet, Average accuracy is the average test performance of all the groups combined. The model is Convnext-tiny.
Methods
Easy
Medium
Hard
Average
Pretrained
71.14
54.93
29.21
51.75
ERM
76.91
70.63
63.48
70.34
ERMeasy
77.82
68.33
51.39
65.85
DFR
74.82
68.66
61.68
68.39
Table 11: Number Images in every group Hard Spurious ImageNet.
Set
Resolution
CeO
CoO
CeR
CoR
Total
56
10k
10k
10k
10k
400k
Train
84
80k
80k
10k
10k
112
80k
80k
10k
10k
56
50k
50k
50k
50k
600k
Test
84
50k
50k
50k
50k
112
50k
50k
50k
50k
56
20k
20k
20k
20k
240k
Val
84
20k
20k
20k
20k
112
20k
20k
20k
20k
20


Reviews:
[{"review_id": "A7V8fOyEFS", "reviewer": "Reviewer_J4sv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The manuscript revisits an important topic: model performance under spurious correlations and the impact of object size and position.\n- The division into groups CeO, CoO, CeR, CoR makes the analysis easier to parse when compared to the scatter plots in Figure 3.\n- The manuscript also studies the impact of group robustness methods like Deep Feature Reweighting and Just Train Twice in their setting.\n\n### Weaknesses\n There are three main weaknesses \n- The models explored are primarily ConvNet based whereas the computer vision community has wholesale converted to ViT based models. ViTs are analyzed only in a couple of tables (Table 2 and Table 7). A modern ViT needs to be included in all the Tables and plots for this manuscript to be relevant to the TMLR community. For example,EVA-02 from \"Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing\" which can be obtained here for pytorch https://huggingface.co/timm/eva02_base_patch14_224.mim_in22k could be used as the main model for all experiments.\n- Similar to the point above, a modern SoTA VLM such as ChatGPT / Gemini should also be evaluated. The prompt should list all the possible classes in ImageNet-1k and ask the chatbot to pick the right one for the image.\n- The dataset construction process, as highlighted in figure 14, has the issue that despite in-painting after masking out the foreground object, the foreground object cues are still present in the image (fish contours, ostrich shadows, pottery wheel parts and pot [see also figure 5 top left where the yellow fish outline is visible and a blurry fish is present in the background]). Furthermore, Section 6 states that  \"In case of images where core features are not labeled by bounding boxes,no inpainting is performed on them\". Thus the resulting dataset has lots of ambiguous images. Under the multi-class classification setting used in ImageNet, the ground truth class is not known.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript analysis the impact of spurious correlations between foreground and background regions on image classification, in particular in connection with varying object sizes and positions. The primary finding is that when size and position deviates from the dataset biases, spurious correlations become a bigger problem for image classification in ImageNet-1K.\n\nMethod: The authors remove foreground objects from images that have annotated ground truth bounding boxes. They use an inpainting technique for this. The inpainted background-only image is now used as a canvas and a (new) foreground object is pasted on top at different positions and scales. The model, if it is confused by the background and the spurious correlation between the background and the foreground, will misclassify such images. The study shows that this behavior is high especially for small objects not in the center of the image.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- To address the weaknesses a ViT and a VLM need to be included in most experiments in the manuscript.\n- To address the issue with ambiguous images in the dataset because of the inpainting procedure, it is unclear what could be changed, but this would require rerunning all the experiments in any case.\n\nOne remedy for the last point that could work for instruction tuned large VLMs is that, one could detail in the prompt how these images were constructed and what exactly the model needs to focus on and to also include a few ambiguous examples as few shot prompts to clarify in the instruction. But I don't see an easy fix like this for standalone backbones like EVA-02 and ResNet50.", "broader_impact_concerns": "The manuscript addresses broader impact concerns with existing models by highlighting their biases. The manuscript itself does not raise new ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "23KgtTbxWv", "reviewer": "Reviewer_1D6P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nShowing that the size of the object of interest strongly impacts classification accuracy, while impact from the object's location (center versus top right corner) is much less. Moreover, I personally like the inpainted ImageNet as a new idea/benchmark. \n\n## Weaknesses (specific remarks)\n\n1. The references are not correctly used: \\citep{ref} can be used when the reference is not part of the sentence and must thus be in parenthesis. This makes the text hard to read.\n2. A grammar spell check in the manuscript is highly advised.\n3. New scores and figure 3-4 locations: why are the figures before the explanation of the scores?  For instance figure 3 is expected to be located close to sec 3.3.\n4. “These scores are calculated by using Ground Truth bounding boxes of ImageNet.” This information must be stated in the beginning of Sec 3.1. As well as that the images are always resized to 224x224.\n5. Figure 2. Right: why are the ticks ~100x100 ? Are the authors resizing the images? Does that change the overall qualitative assessment? If not, one can remove the x/y ticks.\n6. Eq.1 : Why do the authors use the infinite norm?\n7. Figure 5: The depicted abbreviations are not defined in the same page or in the image’s captions and thus are quite confusing to the reader. \n8. Figure 3: what exactly is “inpainted ImageNet”? This could be solved by moving the image in its correct location and cross referencing in the main text where it is explained. At its current form it is particularly unclear what is happening. Right subfigure:The product of two scalar values between 0 and 1 may shift the distribution. Maybe averaging the two scores would have been a better alternative? In general, an intuitive interpretation is missing: for instance, classes with lower center/size scores are easier to classify while absent of semantic information.   \n9. What is the purpose of choosing ConvNext-Base pre-trained on ImageNet22k and fine-tuned on ImageNet1k? Why not pick a supervised or self-supervised model training only on ImageNet?  \n10. Figure 6: The figure can be improved by adding the clean accuracy (baseline as a horizontal line) or by adding the relative acc instead of having the separate table 1. Degradation on top of the bar. The coloring for Hard-Spurious-ImageNet versus the v2 can be replaced with a title on top of the 3 plots. The model names can be shown with a rotated text. The caption can be improved by explaining the abbreviations so that the reader does not need to see the manuscript. The position of the plot in the manuscript is expected to be close to sec 5. \n11. ConvNext-Base trained on ImageNet22k. If the model is pre-trained on Imagenet-22k why do the authors claim that the outperformance is attributed to the “data augmentation pipeline”? Why did the authors use the ConvNext-Base trained on Imagenet to have a fair comparison to the other weights?  \n12. Reference of Hiera-Base with Masked Autoencoder without reference or results is confusing. Either add the results on table 2 or remove it. Paper ref. Is missing so the reader doesn't know which model you are referring to. \n13. I don't understand what fig 7 is showing. Could the author provide an explanation?. The methods ERM, DFR, JTT need to be explained in more detail. I am not sure what ERM^{all} is supposed to mean. What does `conventional training to optimize average training accuracy` mean? If DFR trains only the last layer, why is it on the same plot as ERM in Fig. 7 , while the authors write `The Base model is fine-tuned with batch size 256, constant learning rate of 0.001 for 20 epochs`? What is ERM supposed to mean in the first place? Do the authors mean supervised fine-tuning the model on the new artificially superimposed training data? And if ERM is simply fine-tuning what is the point of presenting these alternative methods that do not show any significant improvement? And is the training stopped at 20 epochs since additional training is beneficial? This is all too opaque at the moment. I have no clue on what Figure 7 right is showing.\n14. Sec 5.4: “We refer to ERM-easy as the model that has been fine-tuned with data from the majority group only i.e. 0% of data from medium and hard group.” Why is this in the experimental results? The so-called ERM easy medium and hard should be explained in Sec. 4 not Sec. 5. \n15. Section 5 abruptly switches to a supervised trained resnet50 without any justification. Why did the authors choose the resnet50? \n16. “The results in Table 3 show that pre-trained ImageNet models perform worst on the hard group.” If the hard group was meant to be the most difficult to classify correctly, why is that surprising? \n17. The section “5.1 Effects of Data Augmentation and Self-Supervised Models” includes CLIP (i assume that’s what the authors mean by ViT-B in Tab. 2), which is not self-supervised. \n18. Sec. 5: The majority and minority groups are not clearly explained until much later in the text. \n19. Table 1 and Table 2: why are different feature extractors used? Why is the clean accuracy of the resnet50 not the same?\n20. “we propose Hard-Spurious-ImageNet, a synthetic dataset” ← since multiple benchmarks are introduced this is not a single dataset but a set of benchmarks. \n\n### Minor Weaknesses\n\n- Please see into the syntax and typos such as `... by computing distinct scores, The detailed explanation of these scores are given in following sections.` \n\n- Some sentences are hard to read: `We observed that the mask quality for some objects was not good enough, hence, we used provided bounding boxes for this work.`  Are you referring to the predicted masks from Segment Anything versus the ground truth? If yes, formulate it more clearly.  “we create four groups for the dataset as before” such formulations are unclear and make the manuscript hard to parse. “there is significant drop in performance compared with clean accuracies on standard validation dataset (see Figure 1).” ← this cross reference is probably wrong and table 1 is supposed to be referenced. “The Base model is fine-tuned” ← which model are you referring to?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new set of dataset that expose spurious correlations present in existing supervised ImageNet-trained Convnets. They first quantify a measure of how centered the data are based on the ground truth bounding box labels from the validation set of imagenet, as well as their bounding box ratio. Then, they create the so-called inpainted imagenet, a new dataset to show the bias toward object size and center using one model (ConvNext-Base [Liu et al. 2022] pre-trained on ImageNet22k). Based on that they introduce a series of new benchmarks with varying degrees of object’s ratio and position (CeO,CoO, CeR, CoR). Using inpainted imagenet, they measure the 10 classes with high acc thus spurious correlations and the 10 worst performing classes (“less-spurious”), where background features are not predictive of the sample’s accuracy. Then they create a dataset with 10 “less-spurious” classes of foreground objects and superimpose them with the 10 detected classes with the highly spurious backgrounds.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A. From all the above, it is apparent that the submitted manuscript is more a first draft than a polished manuscript. The manuscript requires a major revision and it is far below the acceptance threshold in my view. I suggest the other start building meaningful comparisons that isolate only one factor. For instance if one wants to compare the model architecture (ViT, Convnext, ResNet) all models need to be trained on the same data, with the same objective  (supervised learning, self-supervised, weakly supervised, etc) and roughly have a similar number of hyperparameters. Same for all other factors. Right now there are multiple changes taking place simultaneously and it’s impossible to make a fair comparison. In general the most interesting parts of the analysis should be presented in the main manuscript. \n\nB. The overall message of the manuscript gets too convoluted and it is hard to get a clear insight after reading it apart from “object size matters a lot to classification on supervised pre-trained convnets on imagenet”. It’s hard to discern anything else due to improper and unfair comparisons such as CLIP ViT, ConvNext and ResNets being pretrained on different datasets.\n\nC. Further grouping of the groups to easy, medium and hard is not well motivated and further complicates the message.\n\nD. A table summarizing the introduced datasets in terms of number of samples, object size , number of classes , would strongly ameliorate the paper’s readability.\n\nE. The analysis includes many different model weights alternating from one section to another. I would be more interested to see which i) architectures, ii) pre-training strategies, iii) fine-tuning robustness methods are performing best under the most challenging benchmarks. \n\nF. Gradcam visualizations are OK for convets, but if one wants to focus on interpretability the authors need to show how different models use regions of the image, including transformers (ViTs) using attention maps.  \n\n\nG. I personally like the inpainted ImageNet as a new idea/benchmark. Can the authors provide more comprehensive analysis on which models are the most sensitive? A scalability analysis could also be performed , i.e. start from CLIP Vit-B to ViT-G trained on LAION2B, do the larger scale models improve wrt spurious correlations? Similar experiments can be done with supervised models.\n\nH. The top right corner is chosen while in principle any corner can be used. Can the authors modify the data to have a random corner? Fine-tuning on images that are either object-centered or the object is on the top right corner might lead to shortcut learning: it is sufficient to know when something appears on the corner.\n\nI. I am strongly against the further grouping of easy, medium and hard. For instance, “Group CeO: Core object in the Center of image against its Original inpainted background” can directly be used as the easy benchmark without overcomplicated things, and CoR as the hard benchmark. I would include randomly positioning the object in one of the 4 corners as written above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tNx3llEOfT", "reviewer": "Reviewer_W8jg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The study of biases related to object positions and sizes represents a significant research direction.\n+ The paper is easy to follow.\n\nWeakness：\n- Regarding the object size bias study: The current methodology of simply resizing objects may confound size effects with resolution changes. How did the authors isolate and verify that the performance differences were indeed caused by size bias rather than resolution artifacts?\n- As shown in Figure 5, the authors simply cropped objects and placed them arbitrarily in images without proper background blending, resulting in unnatural compositions. Why not employ more advanced techniques like diffusion models or GANs for image generation? Furthermore, given the significant divergence from real-world scenes in Hard-Spurious-ImageNet, can such testing truly reflect model performance on authentic images?\n- The current evaluation appears limited in scope, employing only conventional models (e.g., ResNet50, ViT) and outdated debiasing approaches. We recommend incorporating more state-of-the-art architectures and mitigation methods to enhance the study's comprehensiveness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on biases caused by spurious correlations, particularly examining the impact of object size and position. Hard-Spurious-ImageNet is proposed, which contains images with various backgrounds, object positions, and object sizes. Experiments based on Hard-Spurious-ImageNet demonstrate that existing models exhibit significant biases toward object positions and sizes, and that ERM, DFR, and JTT methods fail to achieve considerable performance gains for worst-group accuracies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The object resizing approach fails to distinguish between genuine size bias effects and resolution-related artifacts, requiring clearer validation methods.\n2. The current image synthesis method produces unnatural compositions. Advanced generative techniques (e.g., diffusion/GANs) should be considered to improve realism.\n3. Expanding to state-of-the-art architectures and mitigation techniques would strengthen the findings.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)