In-distribution adversarial attacks on object recognition models using gradient-free search.
Authors: Spandan Madan
Tomotake Sasaki
Hanspeter Pfister
Tzu-Mao Li
Xavier Boix
Venue: Accepted by TMLR ()

Abstract: Neural networks are susceptible to small perturbations in the form of 2D rotations and shifts, image crops, and even changes in object colors. Past works attribute these errors to dataset bias, claiming that models fail on these perturbed samples as they do not belong to the training data distribution. Here, we challenge this claim and present evidence of the widespread existence of perturbed images within the training data distribution, which networks fail to classify. We train models on data sampled from parametric distributions, then search inside this data distribution to find such in-distribution adversarial examples. This is done using our gradient-free evolution strategies (ES) based approach which we call CMA-Search. Despite training with a large-scale (0.5 million images), unbiased dataset of camera and light variations, CMA-Search can find a failure inside the data distribution in over 71% cases by perturbing the camera position. With lighting changes, CMA-Search finds misclassifications in 42% cases. These findings also extend to natural images from ImageNet and Co3D datasets. This phenomenon of in-distribution images presents a highly worrisome problem for artificial intelligence---they bypass the need for a malicious agent to add engineered noise to induce an adversarial attack. All code, datasets, and demos are available at https://github.com/Spandan-Madan/in_distribution_adversarial_examples.

Published in Transactions on Machine Learning Research (01/2025)
In-distribution adversarial attacks on object recognition mod- els using gradient-free search.
Spandan Madan Horvard University
spandan-madan@ggadl
Tomotake Sasaki Fujitsu Limited
Hanspeter Pfister Horvard University
Sffster@sens.harvord.oom
Tzu-Mao Li UCSD
tali@rusdd edu
Xavier Boix Fujitsu Research America
aboiz@ffjjitsu.cm com
Reviewed on OpenReview: https: opwnrviiee......... net/ forum? id= CT
Abstract
Neural networks are susceptible to small perturbations in the form of 2D rotations and shifts, image crops, and even changes object colors. Past works attribute these errors to dataset bias, claiming that models fail on these perturbed samples as they do not belong to the training data distribution. Here, we challenge this claim and present evidence of the widespread existence of perturbed images within the training data distribution, which networks fail to classify. We train models on data sampled from parametric distributions, then scarch inside this data distribution to find such in-distribution adversarial examples. This is done using our gradient-[ree evolution strategies (ES) based approach which we call CMA-Search. Despite training with a large-scale (~ 0.5 million images), unbiased dataset of camera and light variations, CMA-Scarch can find failure inside the data distribution in over 71% cases by perturbing the camera position. With lighting changes, CMA-Search finds misclassificatins in 42% cases. These findings also extend natural images from ImageNet and Co3D datasets. This phenomenon of in-distribution images presents a highly worrisome problem for artificial intelligenc--they bypass the need for malicious agent to add engineered noise to induce an adversarial allack. All code, datasets, and demos are available at pestps://githine wtt.ggttt.....oppeeeeeeeeeeeeeee SoommmSpadan-aaaaae Maaann-MMannnnnaaa in distribution adversarial _xamplees
1 Introduction
Neural worksrks are highly susceptible to small perturbations 2D rotations and translations Engstrom et al. 2018). image crops Srivastava et al. (2019); Azulay & Weiss 2019 , and even changes in the color
space Mohapatra et al. (2020); Hosseini & Poovendran (2018) Shamsabadi et al. (020). Existing works have
claimed that these failures lie out of the training data distribution, and attribute these failures to dataset
bias
Myas et al. 2019 Lee et al. (2018): Grosse et al. (017 Karunanayake et al. 2024) Stutz et al. (2019)
Here, we put this hypothesis to test by training classificaiion models on datasets with explicitly controlled
train/test distributions, and scarching for adversarial examples within the training data distribution.
'Tomotake Sasaki currently affiliated with Tokyo Metropolitan, Chuo-Johaku Vocational Skills Development Center Japan Electronics College (current email address: 24e001 @ijec.ac.jp).

---

Published in Transactions on Machine Learning Research (01/2025)
(a)
(b)
Dut-of-Distribution failure (noise in pixel space)
Training image
z
Correctly classified test Image
FOV t1
CAMERA POSITION
Look At
(ety 22
x
(c)
Chair
Guitar
Bike
Pistol
Di-Diirribbuion failure Searched within training distribution.
Out-of of- Distribution failure (color change)
Training Data Distribution
Training image
X
Correctly classified test image
X
OOD failure
X
In-Distribution failure
Figure 1: In-distriuttion adversarial attacks. (a) The data distribution (depicted in black) refers to the space of all camera and light variations. Typical adversarial examples are created by adding noise to the image, which may result in images out of the data distribution. CMA-Search finds failures inside the data distribution. (b) 3D scene setup for our rendered images with camera parameters illustrated. (c) Example images with camera and light variations.
Our key finding is that there is a widespread presence of adversarial examples within the training distribution, illustrated in Fig, [a). Thus, networks are highly susceptible small perturbations not just of the training distribution previously known, but inside the training distribution as well. In practice, these in-distribution adversarial examples point to highly worrisome problem- these failures bypass the need for malicious agent to induce an error. These experiments are enabled by our gradient- freee, evolutionary strategics (ES) based approach for finding in-distribution adversarial examples, which we call CMA-Search.
We present results with CMA-Search across three levels of data complexity (i) parametric data sampled from disjoint per-category uniform distributions, (ii) parametric and controlled data of rendered images, and (iii) natural image data from ImageNet and Co3D datasets.
Across all datasets, models are highly susceptible to in-distribution adversarial attacks. CMA-Search can find in-distribution attacks for simplistic parametric data with 100% attack rate-there existed a failure in the vicinity of every single correctly classified lest point. For rendered data, CMA-Search found failures in the vicinity of 71% correctly classified images by perturbing the camera position, and for 42% images by perturbing lighting parameters. With natural images from the Common Objects in 3D (Co3D) dataset Reizenstein al. 202Th, CMA-Search found in-distribution adversarial examples for over 51% images. Finally, we also
employed CMA-Search in conjunction with novel view synthesis pipeline Tucker Snavely 2020 to find
in-distribution adversarial examples in the vicinity of ImageNet Deng et al. (2009).
2
Related Work
In efforts combat susceptibility to small transformations Engstrom et al. (018)) crops Srivastava al.
2019: Azulay Weiss (2019), and 2D rotations and translations Alcorn et al. 2019) alternative architectures
2

---

Published in Transactions on Machine Learning Research (01/2025)
have been proposed which are shift invariant. This includes anti-aliasing networks using the seminal signal processing trick of anti-aliasing Zhang (2019), and recently proposed truly shift invariant networks which use a new sampling methodology to guarantee a 100% consistency in classification under 2D shifts Charnn & Dokmanić (2021p. Unlike our work, these works have focused only on 2D transformations.
Recent work has also sought to generate adversarial perturbations which are human interpretable i.e. semantic adversarial examples. These works often rely on synthetic data, using differentiable rendering or other optimization methods to find adversarial images by modifying scene parameters Liu et al. (2019) Zeng
et al.
2019); Shetty et al. (2020); Jain et al. (2019] Xiao et al. (019); Joshi et al. 2019 Yao et al. 2020
These include a custom differentiable renderer to perturb the camera, lighting, or object mesh verticos, and using a neural renderer where light is represented by network activations.
They key differences between these works and ours is that our adversarial attacks are guaranteed to lie within the training distribution. While in-distribution attacks have been shown in thcoretical works and for toy data Gilmer et al. (2018); Pawzi al. 22016 2018a bb, this work provides the first evidence of such failures with real-world data to the best of our knowledge.
3 Datasets with explicitly controlled data distributions
Mathematically, sample x" is defined to be in-distribution w.r.t. dataset X == (xN):..EN) if xt and all points (#1...ZN)) are generated by sampling i.i.d. from the same generative distribution. Thus, x² E X. As an example, consider the Camera Position. Our dataset with camera and lighting variations was constructed by sampling rendering images with camera position uniformly sampled from [0.5, 8] units. Thus, any image of a scene with camera position within the range [0.5, 8] is in-distribution. Images with camera position not in this range are considered out of distribution. For all datasets, we sample uniformly across the support. This choice allows the support uniquely characterize the data distribution.
oc,
3.1 Generating simplistic parametrically controlled data
We created a binary classification task by sampling data from two N-dimensional uniform distributions confined to disjoint ranges (a, b) and (c, d), as described in the following:
Unif(a, b, N); ys = } Unif(c, d, N); y. = J
(1)
We set - 10, 20, 40 for experiments presented. However, observed that the exact choice of these parameters does not impact the findings. To generate an in-distribution test set, we simply sample new data points from the training distribution. This procedure is consistent with recent theoretical work on
the adversarial attacks Gilmer et al. 2018: Fawzi et al. (2018ah.
3.2 Generating an unbiased training dataset of camera and light variations
Large-scale datasets for computer vision have mostly been created by scraping pictures from the internet Deng et al.] (2009); Lin et al. (2014); Everingham et al.) Krause et al. (2013); Zhou et al. (0117). However, investigating in-distribution robustness requires sampling new points from regions of interest within the data distributtinn, which is not possible with these datasets. To address this issue, we use a computer graphics pipcline for generating and modifying images which ensures complete parametric control over the data distribution. We simply sample camera and lighting parameters from fixed, uniform distribution, and render subset of 3D models from ShapeNet Chang et al. (0055) objects with the sampled camera and lighting parameters.
All models were trained on 0.5 million rendered images across 11 categories, with 1000 images for every 3D model. Each image was constructed by rendering a frame from the 3D scene setup illustrated in Fig. [ one camera, one 3D model and 1-4 lights. Thus, every image parametrized by the camera and the light parameters. The camera parameters are 10 Dimensional, and each light is 11 dimensional. Multiple lights ensure that scenes contain complex mixed lighting, including self-shadows. There is one-to-one mapping
3

---

Published in Transactions on Machine Learning Research (01/2025)
between the pixel space (rendered images) and the (1in + 10) camera and light parameters, with n number of lights. Sample images are shown in Fig. I c) and Fig. S3 All camera and light parameters were sampled from uniform distributions with pre-specific ranges described in the supplement. For these parameters, and additional details please refer to Sec. ST
3.3 Natural image d ImageNet and Common Objects in 3D
As a real litmus test, we also ensure that our findings hold true for natural images. We present results on two popular natural image datasets ImageNet Russakovsky al. (2155) and the Common Objects in 3D (Co3D) Reizenstein et al. (222)) dataset.
Co3D: This dataset was originally created by users capturing short videos of fixed objects placed on a surface by a user moving mobile phone around the object with adjacent frames representing nearby 3D views. We utilize this to test in-distribution robustness. The training dataset was constructed by sampling uniformly across videos from 5 categories (car, chair, handbag, laptop, and teddy bear). This amounts to 187, 200 training images, or 38, 000 images per category which is 32 times the ImageNet training set on a per category basis. An in-distribution test set of 68, 854 images was generated by sampling the remaining frames from these categories. Thus, the test set represents interpolated viewpoints between training viewpoints. Once models are trained, our approach searched within 5 adjacent frames to find in-distribution failure. Additional details can be found in Sec. $2.2
Image.Net: Novel View Synthesis (NVS) Tucker & Snavely (2200) model was used to generate views in the vicinity of images. (See Sec. S2.1 for details). The NVS model takes as input an image and the (z, y, 2) offscts which describe camera movement along the X, Y and axes. Unlike our renderer, cannot introduce changes to the camera Look At, Up Vector, Field of View or lighting changes. CMA-Search optimizes these offset parameters of the NVS model to find perturbed image which is misclassified.
4 CMA-Search: Finding in-distribution failures by searching the vicinity
CMA-Search can be used to attack any parametric dataset. The methodology starts with the parametric representation of correctly classified input, and optimizes these parameters using Covariance Matrix Adaptation-Exolution Strategy (CMA-ES) Hansen & Ostermeier (1996); Hansen (016)) to find misclassified sample in the vicinity of the start point. Algorithm 1 provides an outline for the method which was implemented using pyema Hansen & Ostermeier (1996): : Hansen et al. (2019) We explain the methodology
with an example of finding in-distribution adversarial attacks within the distribution of camera paramcters.
The algorithm for searching adversarial altacks in the space of light parameters, and for attacking all other datasets analogous. For ease, the approach is also visualized as a flowchart in Fig. ST
Starting from the initial camera parameters of the scene, CMA-ES generates offspring by sampling from multivariate normal (MVN) distribution i.e. mutating the original parameters. These offspring are sorted based on the fitness function (1 p, where p denotes classification probability). The best offspring are used to modify the mean and covariance matrix of the MVN for the next generation. The mean represents the current best estimate of the solution i.e the maximum likelihood solution, while the covariance matrix dictates the direction in which the population should be directed in the next goneration. The search stopped either when a misclassification occurs, or after 15 iterations. At cach generation, 10 offspring were generated. For results presented on the simplistic parametrically controlled data, we checked for misclassification till 1500 iterations and 20 offspring were generated in each iteration. CMA-ES is unconstrained optimization procedure. Thus, we ensured that a misclassification counts as an in-distribution adversarial attack only if it met both criteria (1) it caused a misclassification, (2) it belongs to the training data distribution (i.e., inside the support of the underlying distribution).
Evaluating CMA-Scarch: For all models, we report the Attack Rate the percentage of correctly classified points for which CMA-Search successfully found a misclassification A model with no in-distribution failures would have an attack rate of O, making this natural goal for benchmarking studies using this metric.
4

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm CMA-Search over camera parameters to find in-distribution adversarial examples.
Tet x E 10 denote the camera parameters.
Let Render and Network denote the rendering pipeline and classification network respectively, function FITNESS( Render, Network)
image Render(x:
ceapttedd_atteory,, probability Network(image) return predicted category, probability
end function
Tinit: initial camera parameters, X: number offspring per generation (set to 20), yi image category, and Ry: Range of camera parameters in training data.
procedure CMA-SEARCH(2mntt X, D)
initialize μ Tinit, C
I denotes identity matrix.
iters 0
while iters < 20 do
for j = 1,..., \ do
multivariate Sattiitt___ommaap440000 C) I5 = saaple__mllivariittiee Py Pj FITNESS(21, Render, Nelwork)
Generate mutated offspring
> Calculate fitness offspring
if yソ # y then
> Classification fails for image with camera parameters æj
if xy E Rr then return True end if end if end for
> Ty is in in-Distribution failure
L <- +())) with s(j) arg)ort(p)) μ. C t fatatt__ppramctessss...... C) iters iters +1 end while
Pick best offspring
return False
end procedure

---

Published in Transactions on Machine Learning Research (01/2025)
For simplistic parametrically controlled data, the Attack Rate was measured by attacking 20, 000 correctly classified samples. Due to our use of a physically based renderer that accurately models the physics of light in the 3D scene, generating images in the vicinity of the correctly classified image is computational intensive process. For rendered data, it was measured by attacking 2, 000 correctly classified images for every architecture. For one model (ResNet18) we also measured the Allack Rale with 20, 000 images as an additional control. For the Co3D dataset, it was measured on 116, 850 images.
Visualizing the vicinity of an error: CMA-Search generates in-distribution error (2% )startnn from correctly classified point (z²). Using these two points, defined unit vector in the adversarial direction and set it as one basis vector (6i = xa -F2). For D dimensional data, we computed the remaining D orthonormal bases, and randomly selected one as the orthogonal direction (e5, such that e2 1 εī). Following past work Warde-Farley & Goodfellow (2016), we defined a grid of perturbations along the adversarial and the picked orthogonal direction. The classification model was then evaluated on noisy samples constructed by adding noise on the grid formed by these two basis vectors:
T Te aeT + B,2
(2)
Here, a, B E [0, 1] with intervals of 0.01. The church-window plot shows classification on these perturbed samples, with correct classifications shown in white, in-distribution adversarial examples in red, and out-of- distribution samples black.
5 Experimental Details
Below we provide the training details including model architectures, optimization strategies and other hyper-parametors used for the binary classification models trained on simplistic parametrically controlled data, and the object recognition models trained on our rendered images camera and light variations. All code to run these experiments can be found at /attps://githut: cwwmin- aadaddddiddeeeerallllls/iia httprrialiiinniiiiiiiiiioee wwwsiiiittiiiii adversarial examples
5.1 Training details for MLPs for classifying parametrically controlled uniform data
Let D denote data dimensionality, and N denote dataset size. A 5 layer multi-layer perceptron (MLP) with ReLU activations was used, with the output dimensionality of hidden layers set to 5D, D, D/5, D/5, and 2 respectively. However, we found that the number of MLP hidden layers and the number of neurons these layers had no significant impact trends of in-distribution robustness. For experiments with N << 64, 000 all data was passed in single batch. For experiments with more data points, each batch contained 64, 000 points. All models were trained for 100 epochs with stochastic gradient descent (SGD) with a learning rate of 0.0001. All experiments were conducted on a compute cluster consisting of 8 NVIDIA TeslaK80 GPUs, and models were trained on a single GPU at a time. Only models achieving a near perfect accuracy (> 0.99 on a held-out test set were attacked using CMA-Search.
5.2 Training details for Object recognition models for classifying images of real-world objects
All CNN models were trained with a batch size of 75 images, while transformers were trained with a batch size of 25. Models were trained for 50 epochs with an Adam optimizer with a fixed learning rate of 0.0003. Other learning rates including 0.0001, 0.001, 0.01 and 0.1 were tried but they performed either similarly well or worse. To get good generalization to unseen 3D models and stable learning, each image was normalized to zero mcan and unit standard deviation. As before, all experiments were conducted TeslaK80 GPUs, and each model was trained using single GPU at time.
cluster with
Except when dataset size=1000 and dimensions=1000 or 500. In these two case the training data was too small for high test. accuracy. These cases are still included for completion.

---

Published in Transactions on Machine Learning Research (01/2025)
6 Results
We report results on in-distribution adversarial attacks on classification models trained across four datascts (i) simplistic data sampled from disjoint per-category uniform distributions (Sec. 6. (ii) parametrically
controlled images of objects using our graphics pipeline (Sec. 6.2 (iii) Common Objects in 3D dataset Reizen
stein et al. (2021 (Sec. 0.3 and (iv) ImageNet Deng al. (2009) (Scc. 6.3 For each model, we report the
attack rate the percentage of correctly classified points for which we successfully found an in-distribution
failure using CMA-Search. Additional details the implementation and evaluation of CMA-Search are
reported in Sec
6.1 In-distribution adversarial attacks on uniformly distributed data
Fig. a(a) reports the allack rate for models the percentage of correctly classified points for which we successfully found an in-distribution failure using CMA-Search. Despite near perfect accuracy on a held-out test set, in-distribution adversarial examples can be identified in the vicinity of all correctly classified test points-the attack rate is 100% for models trained with 20, 100 and 500 dimensional data. Note that this simplistic dataset is easily separable by the simplest of models including a decision tree. However, DNNs trained on this dataset are plagued by in-distribution failures.
The attack rate start dips once critical dataset size is reached (Fig. a)). However, data complexity scales poorly with number of dimensions. As dimensionality grows from 20 to 100, the number of points required for robustness scales almost 100-fold. For 500 dimensions even 10 million training points were not sufficient. On average, only 51 iterations were needed to find a misclassification for 10 dimensional data. This dropped to 20 iterations for 100 dimensional data and 11 for 500 dimensional data.
Impact of dataset size:
Impact of robust training: We fine-tuned models on 20, 000 in-distribution adversarial examples found using CMA-Search for 100 dimensional data. The allack rale stayed at 100%, with improvement in model robustness against CMA-Search. This is expected, our identified adversarial examples lie within the training distribution. Thus, robust training in this case essentially amounts to marginal increase in the training datasel size which is already discussed above.
Fig. (b) reports the average distance between the (correctly classified) start point and the closest in- distribution adversarial example identified using CMA-Search. This distance increased with dataset size. At critical dataset sizes, adversarial examples are far enough from starting points that they are now not in-distribution. This results in the dip in the attack rate shown in Fig. 2 a)-
Visualizing failures: Fig. 2 c) shows the learned decision boundary using church window plots Warde-Farley
& Goodfellow (016) (see $3.3 for details). Intriguingly, there is clean transition from correctly classified points (white) to in-distribution adversarial examples near the decision boundary (red), beyond which points become out of the distribution (black). Thus, in-distribution adversarial examples are isolated to a region close to the calegory boundary, and in a contiguous fashion. This finding has been theorized Fawzi al. 2016 2018ab; Gilmer et al. (2018); Fawzi et al. (2018b). but the best of our knowledge this the first empirical evidence for this phenomenon.
6.2 Networks struggle to generalize across camera and light variations
Here, we present the first evidence of in-distribution adversarial attacks on visual recognition models. Despite 0.5 million images for 11 categories with over 1000 images for every 3D model, CMA-Scarch found small changes in 3D perspective and lighting which had catastrophic impact on network performance, as shown in Fig. a).
Table reports in-distribution adversarial attacks identified by CMA-Scarch using small changes in 3D perspective and lighting. For 71% images correctly classified by ResNet, there lies an in-distribution failure within 1.83% change in the camera position. For transformers, the impact is far worse with Attack
7

---

Published in Transactions on Machine Learning Research (01/2025)
CMA-Search attack rate
Distance to in-distribution failure
Attack Rate
1.00
L2 Distance
30
0.75
dims
20
0.50
20 dims
dims
0.25
100 dimm
10
500 dimm
0.00
1000
10000
100000
1000000
10000000
1000
10000
100000
1000000
10000000
Dataset Size (%)
Dataset Size (%)
(a)
(b)
Sample:
Sample: 2
Sample:
Sample:
125 25
orthogonal
Orthogunal
Orthogonal
Orthagonal
Adversarial
Adversarial
Adversarial
Sample:
Sample:
Sample:
Sample:
Ortheponal
Orthegonal
Orthoponal
Orthoponal
0.25 1a
Adversarial
Adversarial
Adversarial
Adversarial
In-distribution,
In-distribution
Out-of--istribun
correctly classified
adversarial
sample
(c)
Figure 2: In-distribution adversarial attacks parametric data sampled from high-dimensional, disjoint uniform distributions. (a) Attack rate measured using CMA-Search is 100% for all models there exists an in-distribution failure in the vicinity of every correctly classified sample. Models become robust beyond a critical dataset size, but the data needed scales poorly with dimensionality. (b) Average Euclidean distance between the starting point and the identified in-distribution adversarial sample incrcases as datasct size increases. (c) Church window plots depicting adversarial examples (red) located contiguously and in between the learned and ground-truth boundaries.
Rate of 85%. For lighting changes, CMA-Search can find a misclassification in 42% cases with just a 6.5% change. On average, iterations were needed to find an in-distribution failure with camera variations. For light variations, 3.5 iterations were required on average.
All architectures were most sensitive to changes in the Camera Position and the Camera Look At subtle, in-distribution 3D perspective changes. Shift-invariant architectures designed to improve robustness to 2D shifts performed better, they were still highly susceptible to 3D perspective changes (sce Table
Fig. 3 bb) shows the distribution of scene parameters for misclassified images. Errors are distributed across the space with no clear, strong patterns characterizing the camera and light conditions where networks struggle. This is stark contrast to human vision, which well-documented be significantly impacted by changes
in camera parameters in the form of canonical vs. non-canonical poses Gomez et al. (2008): Terhune et al.
2005); Blanz al. (199), and upside-down orientation
1960);
Lewis 2001
Thhompson (1980).

---

Published in Transactions on Machine Learning Research (01/2025)
Predicted: Bed
Predicted: Table
Predicted: Table
Predicted: Chair
Change in Camera Position: 0.91% Change in Camera Look At: 3.33%
Change in Camera Position: 1 1.25% Change in Camera Look At: 2.77%
(a)
Camera Positions (Incorrect Predictions)
Look At (Incorrect Predictions)
Up Vector (Incorrect Predictions)
Field View
(Incorrect Predictions)
25
8
20
15
-1
10
8
5
-8
0
D
0
40
100
0
1
(b)
Figure 3: In-distribution adversarial attacks in the camera parometer space. Sample in-distribution adversarial examples. Percentage of change in Camera Position and Camera Look At parameters needed to induce the misclassification are also reported. Attack rates are reported in Table [ (b) Distribution of camera parameters for in-distribution adversarial images. Unlike human vision, there were no clear patterns characterizing the camera and light conditions misclassified images.
CMA Cam
CMA Light
Model Architecture
Attack Rate (%)
Distance
Attack Rate (%)
Distance
(mean + std)
(mean = std)
Net
71
1.83 + 1.33
42
6.52 + 5.68
Anti-Aliased Networks
45
2.32 +'2 2.09
40
7.03 + 5.10
Truly Shift Invariant Network
53
2.22 + 2.16
25
6.72 + 5.41
ViT
85
1.34 1.16
65
4.63 + 3.49
DeIT
85
1.27 0.81
4.54 + 2.75
DelT Distilled
86
1.22
0.87
55
4.49 + 2.27
Table 1: Attack Rates for models attacked with CMA-Search over camera and light porameters. CMA-Search starts with correctly classified images, and searches the space camera and light parameters to find an in-distribution misclassifccation. The attack reports percentage of correctly classified images for which CMA-Search found failure. The change in parameter space (mean distance) required to induce error is extremcly small, highlighting the brittleness of these models.
among others. In the supplement we provide additional results reporting CMA-Search over camera and light parameter space (See Sec.

---

Published in Transactions on Machine Learning Research (01/2025)
Results with Co3D dataset. All models suffer from high attack rates, confirming the widespread presence of in-distribution failures for object recognition models.
Table 2:
Anti-A Aliased
ResNet
VIT
DelT
Networks
Test Accuracy
0.92
0.94
0.82
0.85
Attack Rate
0.51
0.39
0.72
0.72
Combined, these results confirm that object recognition models are plagued by in-distribution adversarial attacks.
6.3 Results Natural Image Data
2 reports the average accuracy and attack rate for models trained Co3D. Despite a high test accuracy of 92%, a ResNet model suffered from an attack rate of 51%. Thus, there were in-distribution adversarial examples within 1-5 frames of the correctly classified frame for over half the images. Sample failures are provided in Fig. a Transformers struggled even more, with ViT and DelT Touvron jtt al. (2011) having an attack rate near 72%. The shift invariant architecture was more robust, but attack rate was still high 39% (see Table 2 These trends are consistent with the results in Table
Results Co3D: Table
Results on ImageNet: We also confirmed that these results extend to ImageNet. We present empirical results for a ResNet18 model trained on ImageNet, and OpenAl's transformer-basee CLIP model Radford
et al. 2021 in Fig. lb). Additional ImageNet failures found using CMA-Search are provided in Fig. S5
Furthermore, Fig. SG presents results analyzing how the noise introduced by the NVS pipeline impacts
classification performance finer-grained detail.
Combined, these results across 4 datasets confirm that despite near-perfect test set accuracies and millions of training examples, classification models struggle with the widespread presence of in-distribution adversarial examples.
7 Conclusions
Susceptibilities of recognition models have often been attributed to biased training data. Here, we put this hypothesis test by training and testing with large-scale, unbiased dataset and propose a new search method for investigating the brittlencss of neural networks, which we call CMA-Scarch. We conducted experiments with datasets ranging from simple parametric data, to a rendered dataset of camera and light. variations, and finally natural image datasets ImageNet and Co3D.
Across all datascts our findings are consistent- while data augmentation, unbiased datasets, and specializod shift-invariant architectures are certainly helpful in improving model robustness, the real problem runs far deeper. Despite high test accuracies, networks are plagued by adversarial examples that lie within the training distribution as measured by the attack ratc.
8 Discussions
In practice, these in-distribution adversarial examples point to highly worrisome problem- these failures bypass the need for malicious agent to induce an error. Even when the model has near perfect test. accuracy, these examples lie hidden within the data distribution in plain sight.
This discovery in-distribution adversarial examples presents critical security challenges distinct from traditional adversarial attacks. While conventional attacks require engincered perturbations, these adversarial examples exist naturally within the expected data distribution, creating two severe vulnerabilities. First, they bypass traditional detection methods that rely identifying out-of-distribution characteristics unusual
10

---

Published in Transactions on Machine Learning Research (01/2025)
Macaque
Baboon
Rifle
Flute
Chair
Table
Tractor
Lawn Mower
(a)
Chair
Car
Car
Chair
(b)
Figure 4: In-distribution adversarial allccks on natural images. (a) Misclassífications ImageNet caused by CMA-Search + novel view synthesis. Examples are presented for ResNet model trained on ImageNet, and OpenAl's CLIP model. (b) Sample errors for the Co3D datasct scarched within 1 5 frames of a correctly classified image. Attack rates for Co3D are reported in Table 2
11

---

Published in Transactions on Machine Learning Research (01/2025)
perturbations Chen al. (2020 Lee al 20188 Roth al. (0019 Hendrycks & Gimpel 2016 Second,
and more concerning, they offer perfect plausible deniability to malicious actors. Unlike traditional attacks that leave forensic evidence of pixel manipulation, in-distribution adversarial examples are indistinguishable from legitimate data. Consider a self-driving car crash due to misclassifying a natural scene- not only does it bypass security systems as a perfectly natural image, but also becomes impossible to determine whether this scene was deliberately chosen to cause failure or was truly accidental. This fundamentally undermines security approaches that rely on detecting tampering or establishing malicious intent.
With CMA-Search, have presented tool to help futurre researchers search for such failures and evaluate such defense mechanisms. Naturally, model with no in-distribution failures would have an attack rate of O, making this a natural goal for benchmarking studies using this metric. Future research could also extend this to include
variants, such as Attack Rate at 1% or 5%, which would measure vulnerability within specific perturbation bounds.
Several promising mitigation strategies emerge from our analysis, which could guide future work in- distribution robustness. Firstly, our results in Fig. [(c) show these errors cluster near category boundaries. This suggests that targeted sampling strategies which densely sample training points in these regions could improve model robustness. Secondly, we also found that model initialization has profound impact on the attack rate, which warrants further research into identifying better initialization strategies which can lead to more robust models. Finally, drawing inspiration from biology, the saccadic movements of the human eye suggest that using multiple viewpoints of the same scene during test time could reduce vulnerability.
9 Acknowledgements
We are grateful to George Alvarez and Talia Konkle for their encouragement and for highlighting the relevance of our findings to the fields of Psychology and Vision Sciences. We also extend our thanks to Utkarsh Singhal, Pranav Misra, Fenil Doshi, and Elisa Pavarino for their insightful discussions and feedback. This research was partially supported by Fujitsu Limited (Contract No. 40009105), NSF grants CRCNS-2309041 IIS-2127544, and NIH grant R01HD104969.
12

---

Published in Transactions on Machine Learning Research (01/2025)
References
Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR)) pp. 4845-4854, 2019.
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transforma- tions? Journal of Machine Learning Research, 20(184):1-255 2019.
Volker Blanz, Michael J Tarr, and Heinrich H Bülthoff. What object attributes determine canonical views? Perception, 28(5):575 599, 1999.
Anadi Chaman and Ivan Dokmanić. Truly shift-invariant convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3773-3783, 2021.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An information 3D model repository. Technical Report arXiv:1512.03011 [cs.GR), Stanford University - Princeton University Toyota Technological Institute at Chicago, 2015.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Robust out-of-distribution detection via informative outlier mining. arxiv preprint ar arXi::2006. 15207, 1(2):7, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248 255, 2009.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. A rotation and translation suffice: Fooling CNNs with simple transformations. 2018.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Chal- lenge 2012 (VOC2012) Results. http://www.pasaalnetoortr. wwwp:ppaallllwtw...oooo /hwaalllnggee////oo/00000 htte://////wooooooo//oeo workshoop/indexx.tmn
Alhussein Fawxi, Seyed-Mohsen Moosavi-Dezzooli, and Pascal Frossard. Robustness of classifier:: from adversarial to random noise. In Advances in Newral Information Processing Systems, volume 29, 2016.
Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. Adversarial vulnerability for any classifier. In Advances in Neural Information Processing Systems, volume 31, 2018a.
Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Analysis of classifiers' robustness to adversarial perturbations. Machine learning, 107(3):411-508 2018b.
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow. The relationship between high-dimensional geometry and adversarial examples. arXiv preprint, arXiv:1801.0278 2018
Pablo Gomez, Jennifer Shutter, and Jeffrey N Rouder. Memory for objects in canonical and noncanonical viewpoints. Psychonomic Bulletin & Review, 15(5):940 944, 2008
Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. On the (statistical) detection of adversarial examples. Xr preprint 1rXxv:1702.000 06280, 2017.
Nikolaus Hansen. The CMA evolution strategy: tutorial. arXiv preprint, arXiv:1604 00772, 2016.
Nikolaus Hansen and Andreas Ostermeier. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Proceedings of IEEE International Conference on Evolutionary Computation, pp- 312 317, 1996.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolu- tionary Computation, 9(2):159-19) 195, 2001.
Nikolaus Hansen, Youhei Akimoto, and Petr Baudis. CMA-ES/pycmma on Github. Zenodo, DOL:10.5281/ www.do..2500e February 2019. URL. attps://dol: httpa///ddoo../1///// wttp///eo..///eeooo.ooo 2559634
13

---

Published in Transactions on Machine Learning Research (01/2025)
Radoslav Harman and Vladimir Lacko. On decompositionall algorithms for uniform sampling from n-spheres and n-balls. Journal of Mullivariate Analysis, 101(10):2297-2) 2244244)2977234 2010.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Dan Hendrycks and Kevin Gimpel. Early methods for detecting adversarial images. arXiv preprint ar arii::1088.0055 2016.
Hossein Hosseini and Radha Poovendran. Semantic adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1614-1619, 2018,
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they features. In Advances in Neural Information Processing Systems, 2019.
Lakshya Jain, Steven Chen, Wilson Wu, Uyeong Jang, Varun Chandrasekaran, Sanjit Seshia, and Somesh Jha. Generating semantic adversarial examples with differentiable rendering. http:////peneevieeeentttorooro SJJRFF4YYBB 2019.
Ameya Joshi, Amitangshu Mukherjee, Soumik Sarkar, and Chinmay Hegde. Semantic adversarial attacks: Parametric transformations that fool deep classifiers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4773 4783, 2019.
Naveen Karunanayake, Ravin Gunawardena, Suranga Seneviratne, and Sanjay Chawla. Out-of-distribation data: An acquaintance of adversarial examples-a survey. arXiv preprint arXiu:2044.055 05219 2024.
Wolfgang Köhler. Dynamics in psychology. WW Norton Company, 1960.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations fine-grained categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13), 2013.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Michael B Lewis. The lady's not for turning: Rotation of the Thatcher illusion. Perception, 30(6):769-774, 2001.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proceedings of the European Conference Compuler Vision (ECCV), pp. 740-755, 2014.
Hsueh-Ti Ti Derek Liu, Michael Tao, Chun-Liang Li, Derek Nowrouzezahrai, and Alec Jacobson. Beyond pixel norm-balle Parametric adversaries using an analytically differentiable renderer. In Proceedings of the International Conference Learning Represenlations (ICLR), 2019.
Jeet Mohapatra, Tsui-Wer Weng, Pin- Yu Chen, Sijia Liu, and Luca Daniel. Towards verifying robustness of neural networks against family of semantic perturbations. In Proceedings of the IEEE/CVF Conference Computer (CBcoonCCCPP (CVPR), pp. 244-252, 2020.
Vision and Pattern Recognition
Alec Radford, Jong Wook Kim, Chiis Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and 1lya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML). pp. 8748-878 8763, 2021.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novolay. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 10901-10911, 2021.
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test for detecting adversarial examples. In International Conference on Machine Learning, pp. 5498-5507. PMLR, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Saujeev Satheesh, Sean Ma, Zhiheng Huang, Audrej Karpathy, Adilya Khosla, Michael Berustein, el al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211225) 1522, 2015.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Ali Shahin Shamsabadi, Ricardo Sanchez-Matilla, and Audrea Cavallaro. Colorfool: Semantic adversarial colorization. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Paltern Recognition (CVPR), pp. 1151 1161-160,,
2020.
Rakshith Shetty, Mario Fritz, and Bernt Schiele. Towards automated testing and robustification by semantic adversarial data generation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 489 506, 2020.
Sanjana Srivastava, Guy Ben-Yosel, and Xavier Boix. Minimal images in deep neural networks: Fragile object recognition in natural images. In Proczedings of the International Conference on Learning Representations (ICLR), 2019.
David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generalization. Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 6976 6987, 2019.
Kyla P Terhune, Grant T Liu, Edward J Modestino, Atsushi Miki, Kevin N Sheth, Chia-Shang J Liu, Gabrielle R Bonhomme, and John C Haselgrove. Recognition of objects in non-canonical views: A functional MRI study. Journal of Oparhappttaam 25(4):273-25) 279, 2005.
Peter Thompson. Margaret Thatcher: a new illusion. Perception, 1980.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through atlention. In Procesdings of the 38th International Conference on Machine Learning (ICML), pp. 10347-10337, 2021.
Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, (CVPR), pp. 551 551-560, 2020.
David Warde-Farley and Ian Goodfellow. Adversarial perturbations of deep neural networks. In Tamir Hazan, George Papandreou, and Daniel Tarlow (eds.), Perturbations, Optimization, and Statistics, pp. 311-342. MIT Press, Cambridge, MA, USA, 2016.
Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from single image. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7467-7477, 2020.
Chaowei Xiao, Dawei Yang, Bo Li, Jia Deng, and Mingyan Liu. Meshadv: Adversarial meshes for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6898-6907, 2019.
Philip Yao, Andrew So, Tingting Chen, and Hao Ji. On multiview robustness of 3D adversarial attacks. In Practice and Experience in Advanced Research Computing, pp. 372-378. 2020.
Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from monocular camera. In Proceedings of the BEEE/CVE Conference on Compaler Vision and Pattern Recognition (CVPR), pp. 5336 5345, 2020.
Xiaohui Zeng, Chenxi Liu, Yu-Siang Wang, Weichao Qiu, Lingxi Xie, Yu- Wing Tai, Chi-Keung Tang, and Alan L Yuille. Adversarial attacks beyond the image space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognation. (CVPR), pp. 4302 1311, 2019.
Richard Zhang, Making convolutional networks shift-invariant again. In Proceedings of the International Conference on Machine Leurning (ICML), pp. 7324 7334, 2019.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 286 301, 2016.
15

---

Published in Transactions on Machine Learning Research (01/2025)
S1 Graphics pipeline to generate dataset of camera and lighting variations
S1.1 3D Scene Setup
Each scene contains one camera, one 3D model and 1-4 lights. To ensure no spuri- ous correlations with object texture [17], texture for all ShapeNet. objects was replaced with a simple diffuse material and the background was kept constant to ensure no spurious correlations between foreground and background. Thus, every scene is completely parametrized by the camera and the light parameters. As shown in Fig. | camera parameters are 10 Dimensional: one dimension for the FOV (field of view of camera lens), and three dimensions each for the Camera Position (coordi- nates of camera center), Look At (point on the canvas where the camera looks), and the UP vector (rotation of camera). Analogously, lights are represented by 11 dimensions two dimensions for the Light Size, and three each for Light Position, light Look At and RGB color intensity. Multiple lights ensure that scenes contain complex mixed lighting, including self-shadows. Thus, our scenes are (11n + 10) dimensional, where n is the number of lights. There is a one-to-one mapping between the pixel space (rendered images) and this low dimensional scene representation.
S1.2 Unbiased, uniform sampling of scene parameters
To ensure an unbiased distribution over different viewpoints, locations on the frame, perspective projections and colors, we ensured that scene parameters follow a uniform distribution. Concretely, camera and light positions were sampled from a uniform distribution on a spherical shell with a fixed minimum and maximum radius. The Up Vector was uniformly distributed across range of all possible camera rotations, and RGB light intensities were uniformly distributed across all possible colors. Camera and light Look At positions were uniformly distributed while ensuring the object stays in frame and is well-lit (frame size depends Camera Position and FOV). Finally, Light Size and camera FOV were uniformly sampled 2D and ID vectors. Hyper-panee parameters for rendering, along with the exact distribution for each scene parameter and the corresponding sampling technique used to sample from these distributions are reported in the supplement.
Below we specify the lyper-parameters for rendering, along with the exact distribution for each scene parameter and the corresponding sampling technique used to sample from these distributions.
For scene camera, first random radius re is sampled while ensuring r. Unif(0.5, 8). Then, the camera is placed on random point denoted (Ze,yer ze) on the spherical shell of radius re- To generate a random point on the sphere while ensuring an equal probability of all points, we rely on the method which sums three randomly sampled normal distributions Harman & Lacko (2010):
Camera Position:
X,Y, ~ N( 1),
(3)
== v 2),
(4)
v
(Ier Mos 20) = To * |ua||
(5)
Camera Look At: To ensure the object is shown at different locations within the camera frame, the camera Look At needs to be varied. However, range of values such that the object visible can be present across the entire range of the frame depends on the camera position. So, we sample camera Look At as le as follows:
1 ~ Unif(K * Hey ye: K * ze), where K= = 0.3.
(6)
The value K = 0.3 was found empirically. We found it helped ensure that objects show up across the whole frame while still being completely visible within the frame.
Camera Up Vector: Note that the camera Up Vector is implemented as the vector joining the camera center (0,0,0) to specified position. We sample this position and therefore the Up Vector u. as follows:
F. v, 2 Unif( 1,,
Ue (x, y. z).
(8)
16

---

Published in Transactions on Machine Learning Research (01/2025)
Camera Field of View (FOV): We sample the field of view fe while ensuring:
Se Uni K2).
(9)
Again, the values K1 35, K2 100 were found empirically to ensure objects are completely visible within the frame while not being too small.
For every scene we first sample the number of lights n between 1-4 with equal probability. For each light i, a random radius r; is sampled ensuring T2 Unif( R2), then the light is placed on a random point (æi, Vi, 2;) on the sphere of radius r;- R1 and R₂ were found empirically to ensure that the light is able to illuminate the 3D model appropriately.
Light Position:
Light Look At: To ensure that the light is visible on the canvas, light Look At is sampled as a function of the camera position:
4 ~ Unif(K Te: * Noy K ze); where = 0.3.
(10)
As the case of the Camera Look At parameter mentioned above, the value K = 0.3 was found empirically. Light Size: Every light in our setup is implemented as area light, and therefore requires height and width to specify the size. We generate the size s, for light i as:
h, w Uni L2),
(11)
8; (h, w).
(12)
L = 0.1, = 5 were found empirically to ensure the light illuminates the objects appropriately.
Light Intensity: This parameter specifies the RGB intensity of the light. For light i, RGB color intensity G was sampled as:
r. 9. b Unif(0,,)) 1), C: --(rgg,b) (r.g.b).
(13)
(14)
To ensure no spurious correlations bet ween object texture and category, all object textures were set to a single diffuse material. Specifically, the material is a linear blend between a Lambertian model and a microfacet model with Phong distribution, with Schilick's Fresnel approximation. Diffuse reflectance was set 1.0, and the material was to reflect on both sides.
Object Material:
S1.3 3D models used for generating two different test sets
Our dataset. contains 11 categories, with 40 3D models for every calegory chosen from ShapeNet Chang et al. (2015). Neural networks were evaluated on two test sets - one with the 3D models seen during training, and the second with new, unseen 3D models. The first test set was generated by simply repeating the same procedure described above. Thus, the (Geometry x Camera x Lighting) joint distribution matches exactly for the train set and this test set. The second test set was created by the exact same generation procedure, but with 10 new 3D models for every category chosen from ShapeNet. The motivation for this second test set was to ensure our models are not over-fitting to the 3D models used for training. Thus, the (Camera x Lighting) joint distribution matches exactly for this test set and the train set, but the Geometry is different in these two sets.
S2 Generating nearby views for Natural Image Datasets
S2.1 Views in the vicinity of ImageNet images
ImageNet contains only one viewpoint per object. While several variations of ImageNet have been proposed by adding noise in the form of corruptions and perturbations Hendrycks & Dietterich (209), these variations are
17

---

Published in Transactions on Machine Learning Research (01/2025)
Table S3: Performance of object recognition models on seen and new 3D models.
Anti- Aliased
Truly Shift Invariant
DeIT Distilled
Accuracy ResNet
VIT
DelT
Seen models 0.75
0.82
0.80
0.58
0.63
0.64
New models
0.70
0.74
0.72
0.59
0.64
0.65
designed to study the impact of dutrof-distribuutin shifts on object recognition models. Like these variations, our camera manipulations correspond to transforming input images to study its impact on object recognition models. However, the key difference is that our work focuses on in-distribution adversarial examples, due to which these datasets designed for out-of-distribution shifts cannot be repurposed for our experiments. Thus, major challenge in extending our results to ImageNet is generating natural images in the vicinity of a correctly classified image by slightly modifying the camera parameters. To do so for ImageNet is equivalent to novel view synthesis (NVS) from single images, which has been a long-standing challenging task in computer vision. However, recent advances in NVS enable us to extend our method to natural image datasets like
ImageNet Yoon et al. 2020 Zhou et al. 20166 Wiles et al. (2200] Tucker & Snavely 2020
To generate new views the vicinity of ImageNet images, we rely on a single-view synthesis model basod on multi-plane images (MPI) Tucker & Snavely (2020). The MPI model takes as input an image and the (z, y, 2) offsets which describe camera movement along the X, Y and Z axes. Note that unlike our renderer, it cannot introduce changes to the camera Look At, Up Vector, Field of View or lighting changes. An important limitation of this approach is that any noise added by the MPI model in image generation is a confounding variable which we cannot. account for. This further highlights the importance of our rendered and Co3D experiments as these experiments do not suffer from such noise.
S2.2 Views in the vicinity of Co3D images
As an additional control for any potential noise introduced by the novel view synthesis pipeline in gener- sting nearby views for ImageNet images, we present additional results on the large-scale, multi-viewpoint Co3D Reizenstein et al. (2021) dataset. Co3D was created by capturing short videos of fixed objects placed a surface by views in the vicinity of an image. We utilize this to test in-distribution robustness in the vicinity of correctly classified images. The classification datasel is created by picking 5 categories car, chair, handbag, laptop, and teddy bear. We created the training data by uniformly sampling frames across the whole video for all vidcos for these categorics amounting to 187, 200 training images. Note that this amounts to roughly 38, 000 images per category, which 32 times the ImageNet training set on per calegory basis. An in-distribution test set of 68, 854 images is generated by sampling the remaining frames to measure overall accuracy of the trained models. We then search for in-distribution failures in the vicinity (i.e., nearby frames) from the remaining frames from these videos in the Co3D dataset. Thus, novel view synthesis pipcline was used. Instead, pre-captured frames from the videos were used to search for in-distribution adversarial examples in the vicinity of viewpoints.
moving mobile phone around the object. Thus, nearby frames in the video represent.
S3
Additional details on CMA-Search
Below we provide details on the implementation and evaluation of our in-distribution adversarial search method-Cue CeM--errrh.
S3.1 Finding in-distribution adversarial examples by searching the vicinity of correctly classified image
CMA-Search can be used to lack any parametric datasel. To find an in-distribution failure our methodology requires a classification model, a correctly classified data point, and the parametric representation of this data point. CMA-Search optimizes those parameters using evolutionary strategies to find a sample which is misclassified by the model. We used a gradient-free optimization method Covariance Matrix Adaptation- Evolution Strategy (CMA-FIS) Hansen & Ostermeier (1966) Hansen 2016 CMA-ES has been found to
18

---

Published in Transactions on Machine Learning Research (01/2025)
Start
Initial Camera Fartmeterrs iters
itors 20
No
Generate Mutated Offpring. offeprings
Not Found
Check Misclasstiiatton
Misclasifiication?
Inside Distribution?
offsprings 20
Distribution Adversarial Attack
No
Pick Offspring, Update CMA Parameters
iters 1, affspringe-B
Figure S1: Flowchart describing CMA-Search step-by-step.
19

---

Published in Transactions on Machine Learning Research (01/2025)
Category: Piano, Network: Anti-Aliased
100
-B
40
40
02
60
200
(a)
(c)
(d)
Category: Guitar, Network: Anit-Aliased
20
10
(a)
(b)
(c)
(d)
Category: Pistol, Network: Truly Shift Invariant
60
8
20
8-8
40
(a)
(b)
(c)
(d)
Category: Airplane, Network: Truly Shift Invariant
10
30
8
(a)
(b)
(c)
(d)
Figure S2: Camera Parameters that lead to misclassifecations for multiple categories and architectures. (a)
Camera Position, (b) Camera Look At, (c) Up Vector, (d) Histogram of Lens Field of View.
work reliably well with non-smooth optimization problems and especially with local optimization Hansen Ostermeicr 2001h, which made it a perfect fit for our scarch stratogy.
Starting from the initial parameters, CMA-ES generates offspring by sampling from multivariate normal (MVN) distribution i.e. mutating the original parameters. These offspring then sorted based the fitness function (classification probability), and the best ones are used to modify the mean and covariance matrix of the MVN for the next generation. The mean represents the current best estimale of the solution i.e. the maximum likelihood solution, while the covariance matrix dictates the direction in which the population should be directed in the next generation. The search was stopped either when a misclassification occurred, or after 15 iterations over scene parameters. For the simplistic parametrically controlled data, we checked for misclassification till 1500 iterations.
20

---

Published in Transactions on Machine Learning Research (01/2025)
For ease, we present the algorithm for in-distribution errors in rendered images found by optimizing camera parameters. The methodology to attack all datasets is analogous. In this case, our method searches the vicinity of the camera parameters of a correct classified image to find an in-distribution error. Algorithm I provides an outline of using CMA-Search to find in-distribution adversarial examples by scarching the vicinity of camera parameters. The algorithm for searching for adversarial examples using light parameters in rendered data, and within parametrically controlled uniform data is analogous. Fig. (() presents examples of in-distribution adversarial examples found using CMA-Search over camera parameters. As shown, subtle changes in 3D perspective can lead to drastic errors in classification. We also report the subtle changes in camera position (in black) and camera Look At (in blue) between the correctly and incorrectly classified images in Fig. Cc).
This approach differs from existing work on adversarial viewpoints and lighting et al. 2019 ; Zeng et al. (2019); Jain et al. (2019) in two ways. First, unlike these works our approach finds in-distribution errors. Secondly, these methods rely on gradient descent and thus require high dimensional representations of the scene to work well. For instance, these works often use neural rendering where nelwork activations act
high dimensional representation of the scene Zeng et al. 20199 Joshi et al. (2019), or use up-sampling of moshes to increase dimensionality Liu et al. 2019). In contrast, our approach works well for as low as 3 dimensions.
S3.2 Evaluating CMA-Search and in-distribution robustness using the Attack Rate
The performance of CMA-Search was quantified using a new metric- the Attack Rate, which refers to the percentage of correctly classified points for which CMA-Search successfully found an in-distribution adversarial example. For simplistic parametrically controlled data, the Attack Rate was measured by attacking 20, 000 correctly classified samples using CMA-Search. Due to our use of a physically based renderer that accurately models the physics of light in the 3D scene, generating images in the vicinity of the correctly classified image is a computational intensive process. Thus, for rendered data, the Attack Rate is measured by attacking 2, 000 correctly classified images for every architecture, and these numbers are reported in Table D As an additional control, we also measured the Attack Rate for the ResNet18 architecture with 20, 000 images, and found the rate to be unchanged. For the Co3D dataset, Attack Rate is measured on 116, 850 images. As explained in Scc.3, we do not render or generate any new novel views for Co3D but simply scarch through natural images already provided in the dalaset.
S3.3 Visualizing in-distribution adversarial examples using Church-window plots
CMA-Search starts from a correctly classified point and provides an in-distribution adversarial example. We used these two points to define a unit vector in the adversarial direction, and fixed this as one of basis vectors for the space the data occupies. As data dimensionality was D, we calculated the remaining D-1 orthonormal bases. Following the same protocol past work Warde-Farley & Goodfellow (2066), randomly picked one of these orthonormal vectors as the orthogonal direction and defined grid of perturbations with fixed increments along the adversarial and the orthogonal directions. These perturbations were then added to the original sample and the model was evaluated at these perturbed samples. We plotted correct classifications in white, in-distribution adversarial examples in red, and dist---distiiittion samples in black.
S3.4 Computational efficiency of CMA-
CMA-Search operates iteratively, generating multiple offsprings in every iteration, and retaining the best in every iteration to calculate parameters for the next iteration. For simplistic parametrically controlled, CMA-Search was set to generate 20 offsprings in every itcration, and the search algorithm was set to stop when an in-distribution adversarial example is found, or if a maximum threshold of 1500 iterations were hit. On average, 51 iterations were needed to find an in-distribution adversarial example for 10 dimensional data. The average number of iterations needed dropped to 20 for 100 dimensional data. Note that as dimensionality increases, all steps become more computationally intensive, this includes training models, generating new offsprings using CMA-Search, and model inference to test offspring fitness. Thus, overall time required
21

---

Published in Transactions on Machine Learning Research (01/2025)
Figure S3: Sample Images from our rendered dataset.
to altack increases with dimensionality. However, computational elficiency of CMA-Search improves with dimensionality, lesser iterations are needed.
For rendered data, which is significantly higher dinensional, we found that CMA-Soarch is very efficient as extremely low number of iterations are needed to find an in-distribution failure. For both camera and light variation based attacks, CMA-Search was set to generate 10 offsprings in every iteration, and maximum iteration threshold was set to 15. On average, only itcrations were nceded to find an in-distribution failure with camera variations. For light variations, 3.5 iterations were required on average. This suggests that CM A-earrhh is more efficient at higher dimensions, despite working well at low dimensions.
22

---

Published in Transactions on Machine Learning Research (01/2025)
Category: Piano, Network: Anti-Aliased
100
-B
40
02
60
200
(a)
(c)
(d)
Category: Guitar, Network: Anit-Aliased
20
10
(a)
(b)
(c)
(d)
Category: Pistol, Network: Truly Shift Invariant
60
8
20
8-8
40
(a)
(b)
(c)
(d)
Category: Airplane, Network: Truly Shift Invariant
10
30
8
(a)
(b)
(c)
(d)
Figure S4: Camera Parameters that lead to misclassifeaation for multiple categories and architectures. (a)
Camera Position, (b) Camera Look At, (c) Up Vector, (d) Histogram of Lens Field of View.
S4 Additional Results with ImageNet
We present examples of in-distribution adversarial attacks with ImageNet Fig. S5
Furthermore, we
conducted comprehensive analysis on how noise levels impact robustness. Specifically, we evaluated the attack rate for ImageNet images under different levels noise introduced by NVS. A total of 27 noise levels were tested-thres levels each for the camera parameters controlling the X, Y, and axes (corresponding to horizontal translation, vertical translation, and zoom). These results presented in Fig. S6 While attack rates increased with noise across all axes, models showed particular susceptibility to Y-axis perturbations.
23

---

Published in Transactions on Machine Learning Research (01/2025)
Tank
Pickup Truck
Train
Street Car
Snake
Lizard
Train
Pickup Truck
Cup
Tennis Ball
Bear
Chimpanzee
Hamster
Sweet Pepper
Turtle
Beetle
Porcupine
Raccoon
Fish
Porcupine
Figure S5: ore exumples of misclassified ImageNel-like images discovered by CMA- Search combined with the single view MPT model.
24

---

Published in Transactions on Machine Learning Research (01/2025)
Attack Rate Across Camera Positions
Position: Near
0.8
Y Position: Center
Y Position: Far
0.8
0.7
0.7
0.7
0.14
0,22
0
0.46
0.86
0.84
0.6
0.6
0.6
Translation (X)
Translation
0.5
Attack Rate
0.5
Rate
Translation
0.5
Rate
0.30
0.20
0.30
1
0.48
0.48
0.46
0.80
0.82
0.80
0.4
0.4
0.4
0.3
0.3
2-
0.40
0.42
0.38
2-
0.52
0.50
0.56
2-
0.82
0.80
0.78
Zoom (Z)
Zoom (2)
-00
Zoom (Z)
Figure S6: Impact of NVS induced noise on ImageNet classification.
25


Reviews:
[{"review_id": "qeCrUpR4Z1", "reviewer": "Reviewer_GKQk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- This is an original and intelligent problem + method. The gradient-free search idea is clever and provokes thought about the mechanisms underlying these adversarial examples. It is very intuitive for this use-case.\n- Experimental setup is solid:\n  - Datasets are varied and well-chosen. It's good to have a mix of synthetic datasets that highlight the mechanism in question, and large natural datasets that show that it's a real mechanism. Datasets are highly varied and diverse. \n  - Good use of transformer-based architectures along with ResNets. Architecture has the potential to change the adversarial example landscape considerably, so it's helpful--even if intuitively expected--to see effects appear in different architectures. It also keeps this area of vision caught up with the state of the field.\n- Good figures! The different visualization approaches help a lot to build our understanding of the underlying nature of these adversarial inputs, which is important not only for security but interpretability.\n\n## Weaknesses:\n- The paper motivates itself by talking about how adversarial failures mainly occur out of distribution. I'm not sure this is entirely true, though I do think it's fair to say most work in adversarial examples focuses on synthetic out-of-distribution examples. \n- While the CMA-ES pseudocode is useful and important, it would also help to have a diagram of the method if possible. It's currently somewhat difficult to build intuition until rereading a few times. \n- More grounding on metrics such as \"attack rate\" would be helpful - what should we look for and benchmark against?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents CMA-Search, a gradient-free search method for finding in-distribution adversarial images. It is motivated by the claim that adversarial examples are seen and studied mostly as out-of-distribution inputs, but in-distribution inputs can be adversarial as well. \n\nIt quickly reaches its methods section, describing various datasets to test on. These include synthetic data generated to test adversarial robustness in a controlled manner. Next is rendered data from sim, giving more control over image parameters. And finally, natural images. It then presents the adversarial input-finding method itself: it uses covariance matrix adaptation-evolution strategy (CMA-ES) to find adversarial examples through perturbed camera and lighting parameters. It considers these within training distribution given that the adjusted parameter values are naturalistic. It samples and evaluates iteratively to find incorrectly labeled examples.\n\nThe paper then goes through its experiment details and results: various architectures including MLPs, CNNs, and transformers are tried. There is also standard variation of hyperparameters. The results show that in all datasets, adversarial images are found a lot of the time; in the simplest and most controlled dataset, they are found in every training setting. \n\nThe paper concludes by discussing the importance of understanding adversarial examples and what causes them.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Discuss not only the mechanism of determining adversarial images but also how to deal with them. This could be a future work, but discussing it is useful even in this paper because the solutions may tie directly to the identificatoin method itself. \n- For natural image experiments involving NVS, analyze how potential noise impacts results.", "broader_impact_concerns": "Adversarial examples are a security risk. This is not addressed int he paper. I don't think it needs extensive discussion, but some discussion on the implications of this project for threatening adversarial examples would be helpful.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4JRpBGCujn", "reviewer": "Reviewer_ioAP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n - Understanding failure cases of the model that are \"in-distribution\" is a problem of great practical importance as it is not only relevant in the presence of an adversary but also more broadly for any deployment scenario\n - The studied domains of controlling camera and light variations on ShapeNet (Section 3.2) and Novel View Synthesis on ImageNet are sufficiently closed to realistic problems to be of practical interest.\n - Figure 1 is a good illustration of the problem setting and makes the paper more accessible\n - Figure 3 provides relevant findings of in-distribution adversarial examples\n - the paper is overall well-written and easy to follow\n\nWeaknesses\n - ID vs OOD: in general, the notions of in-distribution vs. out-of-distribution are central for the paper but never clearly defined. Often, \"in-distribution\" is used in a way as if the region (support) of the distribution is meant, for instance when using formulations like \"search _inside_ this data distribution\" or drawing a region in Figure 1. However, even a simple 1D Normal distribution has the whole real numbers as support - so everything would be \"inside the data distribution\" for a Normal distribution. The authors need to be more explicit about what they understand as \"in-distribution\" without making the concept too broad and at the same time show that their method finds actual in-distribution adversarial examples under this definition\n - CMA-Search is a straightforward application of CMA-ES to adversarial example generation with practical limitations - for instance, how would this method deal with constraints on the parameter ranges?\n - It remains unclear to me how constraints of the problem/threat model are enforced - for instance, where do the authors enforce that adversarial examples on the Nd uniform distribution data from Section 3.1 actually stay within the prescribed intervals? \n - Please clarify how in-distribution and out-of-distribution are determined in Figure 2c - from  Section 3.1 my understanding would be that everything with a value between a=-10 and b=10 is in-distribution and belongs to class y_i=0\n - Related work is very short and shallow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the existence and generation of \"in-distribution adversarial examples\" for object recognition models. For this, the authors use a gradient-free adversarial attack that is based on the CMA-ES algorithm. The authors find interesting in-distribution adversarial examples when varying camera and light on 3d models from ShapeNet as well when generating novel views for ImageNet and Co3D datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the points listed under weaknesses. In particular, a term so central to the paper like \"in-distribution\" requires a clear definition.", "broader_impact_concerns": "no concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fEszRNKwQc", "reviewer": "Reviewer_mN3d", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n- They provide empirical evidence of the existence of in-distribution adversarial examples against deep neural networks trained on real-world data.\n- The main claim of this paper is adequately supported through experiments using multiple datasets, including real-world datasets.\n\n## Weakness\n- While this paper provides empirical evidence for the existence of in-distribution adversarial examples, it does not involve theoretical analysis. Therefore, it is unclear whether the existence of in-distribution adversarial samples is an unavoidable problem.\n- For the same reason, it is not clear whether there are any in-distribution adversarial samples for data from domains other than images.\n- Is there a mathematical definition of \"in-distribution\"? In the case of Co3D, I can understand the concept of in-distribution from the perspective of how the adversarial example is taken. However, I am not sure that the results of Novel View Synthesis for ImageNet are within the distribution of the training data.\n### minor\n- Reference on p.7, in the paragraph ``Visualizing failures'', is broken.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main contribution of this paper is to provide the evidence for the existence of in-distribution adversarial examples using real-world data. In-distribution adversarial example is the sample within the training data distribution, but causes severe degrade in model's performance.\nThe empirical evidence is provided through a series of experiments using various datasets from artificially generated toy data to real datasets including Co3D.\nThe experimental results show that we can construct a set of inputs that cause a significant decrease in accuracy from the training data distribution even for models with high accuracy on test data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weaknesses above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The action editor agrees with the reviewers on acceptance. This study of in-distribution attacks, or perturbations in the hull of the training data, shows that deep networks for vision are susceptible to minor alterations to their inputs even in the absence of dataset bias. Controlled synthetic studies and experiments with standard natural image datasets expose and analyze this issue for transformations like viewing angle, rotation, relighting, and cropping. These are informative results that can spur further research into robustness beyond attacks in the form of engineered and optimized adversarial noise.

Reviewers raised a number of points that were resolved by the author responses and revisions. Initial concerns about the clarity of the definitions (ioAP, mN3d), exposition (GKQk), and discussion (GKQk) have all been settled. Further methodological detail and discussion about the optimization for finding the perturbations presented in this work (ioAP, GKQk) has been included in addition to futher results on ImagNet (mN3d, GKQk). The decision recommendations by reviewers confirm that their concerns have been addressed and no obstacle to acceptance remains.

The TMLR process has improved the work due to the detailed comments by the reviewers and the thorough revisions by the authors. The action editor thanks everyone for engaging and congratulates the authors on their publication in TMLR!
Decision: Accept (poster)