SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Florian Stimberg  
Google DeepMind
Ayan Chakrabarti  
Google Research
Chun-Ta Lu Google Research
Hussein Hazimeh  
Google Research
Otilia Stretcu Google Research
Wei Qiao Google Ads Safety
Yintao Liu Google Ads Safety
Merve Kaya Google Research
Cyrus Rashtchian  
Google Research
Ariel Fuxman  
Google Research
Mehmet Tek Google Ads Safety
Sven Gowal Google DeepMind
Automated content filtering and moderation is an important tool that allows online platforms to build striving user communities that facilitate cooperation and prevent abuse. Unfortunately, resourceful actors try to bypass automated filters in a bid to post content that violate platform policies and codes of conduct. To reach this goal, these malicious actors may obfuscate policy violating images (e.g. overlay harmful images by carefully selected benign images or visual patterns) to prevent machine learning models from reaching the correct decision. In this paper, we invite researchers to tackle this specific issue and present a new image benchmark. This benchmark, based on IMAGENET, simulates the type of obfuscations created by malicious actors. It goes beyond IMAGENET-C and IMAGENET-  $\tilde{\mathbf{C}}$  by proposing general, drastic, adversarial modifications that preserve the original content intent. It aims to tackle a more common adversarial threat than the one considered by  $\ell_p$ -norm bounded adversaries. We evaluate 33 pretrained models on the benchmark and train models with different augmentations, architectures and training methods on subsets of the obfuscations to measure generalization. Our hope is that this benchmark will encourage researchers to test their models and methods and try to find new approaches that are more robust to these obfuscations.
Advances in in computer vision have lead to classifiers that nearly match human performance in many applications. However, while the human visual system is remarkably versatile in extracting semantic meaning out of even degraded and heavily obfuscated images, today's visual classifiers significantly lag behind in emulating the same robustness, and often yield incorrect outputs in the presence of natural and adversarial degradations. This is why evaluating robustness [1, 2] and improving the robustness of visual classifiers has been the subject of considerable research [3, 4], with multiple benchmarks looking at out of distribution examples [5, 6], distribution shift [7-9] or focusing on robustness to "natural degradations", like blur and noise, which are inherent in the imaging process [10, 11]. However, with more and more visual classifiers being deployed in real systems, robustness to adversarial perturbations—deliberate changes to an image introduced by an adversary to fool classifiers—has emerged as an important research direction [12-14].
Such work on adversarial robustness has largely focused on perturbations that are imperceptible to human observers—often cast as an explicit  $\ell_p$ -norm constraint between the original and perturbed
image [12, 15]. However, for many visual classifiers that are focused on keeping offensive, dangerous, pirated, or other policy-violating content off of online platforms, this limited definition does not suffice. In the real world, attackers who want to hide malicious content in images are not prevented by the fact that an image appears obviously perturbed to human observers, as long as the observer is also able to glean the underlying violating content [16]. This scenario is not theoretical—such malicious image manipulations are being used today by bad actors daily and at scale on modern online platforms [17].
Our work focuses on enabling research into making visual classifiers robust to such adversarial obfuscations—image transformations that can fool classifiers while leaving the underlying semantic content intelligible to human observers, but differing from prior adversarial attacks [12, 15] in allowing it to be obvious that the image is transformed. To this end, we introduce a benchmark to characterize the performance of classifiers on obfuscated images.
Naturally, we do not claim this benchmark to be exhaustive, since the space of adversarial obfuscations is limited only by the creativity of attackers and the resilience of the human visual system in inferring the underlying content despite significant manipulation. However, to provide a concrete starting point to make and measure progress in obfuscation robustness, we introduce a set of 22 transforms, illustrated in fig. 1, that (a) are compositions of various transformations available in image editing software—geometric and color transformations, image splicing and blending, style transfer, etc.; (b) are strong enough to fool most current classifiers while still leaving the underlying semantic content identifiable; and (c) are diverse enough to allow us to measure generalization of current and future robust training techniques, by measuring performance on obfuscations that are held out during training. Moreover, these transforms are similar to actual obfuscations that we have observed being used in the wild by attackers attempting to bypass Google's image policy classifiers.
Overall, our main contributions are as follows:
- We create, curate and tune a set of 22 strong, diverse, adversarial obfuscations. Compared to other benchmarks our obfuscations imitate methods by bad actors trying to fool content filter models and are allowed to drastically change the images. Our benchmark is set up to have training and hold-out obfuscations which allows to measure generalization to unknown obfuscations and monitor progress of leveraging known attacks.
- We evaluate 38 different pretrained models on our benchmark and train additional models to compare the effects of 7 different augmentation and 4 distribution shift algorithms. Our experiments show that scaling architectures, pretraining on larger datasets and choosing the right augmentations can make models more robust to unseen obfuscations.
- We train models on over 60 subsets of our training obfuscations to show relationships between them, which of them have the biggest effect on generalization and that we get diminishing returns when adding obfuscations to the training set.
- Finally, we show that training on our training obfuscations increases performance on all of 8 IMAGENET variants and that  $\ell_p$ -norm based adversarial training does not help robustness to our obfuscations.
We expect that our analysis, and particularly, this benchmark will stimulate research in the development of training techniques that make classifiers more robust to adversarial obfuscations. Such techniques promise to be of immediate practical value since they can be used by online platforms to strengthen their automated classification and detection models, thereby helping improve their users' online experience by keeping out unsafe and illegal content.
Datasets of natural distribution shifts. Characterizing model failures and empirically estimating their consequences often requires collecting and annotating new datasets. Hendrycks et al. [6] collected datasets of natural adversarial examples (IMAGENET-A and IMAGENET-O) to evaluate how model performance degrades when inputs have limited spurious cues. Hendrycks et al. [9] collected real-world datasets (including IMAGENET-R and DEEPFASHION REMIXED) to understand how models behave under large distribution shifts such as artistic renditions of various objects. Particular shortcomings can only be explored using synthetic datasets [18]. Hendrycks and Dietterich [19] introduced IMAGENET-C, a synthetic set of common corruptions.
Figure 1: Example images for obfuscations. From top left to bottom right: Clean, IconOverlay, Texturize, ColorPatternOverlay, LowContrastTriangles, PerspectiveComposition. See section appendix A.1 for examples of all obfuscations.
Recently, [11] proposed IMAGENET-  $\bar{\mathbf{C}}$  in a bid to understand whether progress on IMAGENET-C is truthful. They sample corruptions that are perceptually dissimilar from IMAGENET-C in feature space and observe that data augmentations may not generalize well.
Extending upon [19], Kar et al. [20] introduce corruptions that capture 3D information. They aim to guard against natural corruptions, such as camera rotation, camera focus change, motion blur.
All of the previously mentioned datasets either collect natural out-of-distribution examples or create variations that mimic natural corruptions that occur when capturing images. Our benchmark dataset goes beyond the realistic corruptions considered by above work and includes artificial corruptions that adversaries could produce using common image editing software.
Some examples of datasets that feature less natural transformations are Geirhos et al. [5], who study the propensity of Convolutional Neural Networks (CNNs) to over-emphasize texture cues, by evaluating such models on 4 types of obfuscations focused on texture and shape, as well as a dataset with texture-shape cue conflict. Xiao et al. [8], Sagawa et al. [21] investigate whether models are biased towards background cues by compositing foreground objects with various background images (IMAGENET-9, WATERBIRDS). While these not necessarily try to simulate natural processes, they are focused mostly on understanding how models deal with changes in specific aspects of images.
$\ell_{p}$ -norm adversarial robustness. In some cases, it is possible to discover failures via optimization or brute-force search.  $\ell_{p}$ -norm adversarial attacks introduce small, imperceptible perturbations to input examples, with the aim of causing misclassifications [2, 12]. While the majority of the attacks in the literature assume white-box access (i.e., all model weights are available to the attacker) [12, 2, 22-28], another line of work considers the more practical black-box setting where the attacker has no or limited knowledge about the model [29-31]. While there is a lot of merit in investigating white- or black-box adversarial robustness, most real world attacks on machine learning models are not of this form [16] which is why our benchmark tries to include obfuscations that are or realistically could be used by attackers.
Beyond  $\ell_p$  robustness. Given the practical implications of  $\ell_p$  robustness, there has been growing interest in studying broader threat models that allow for large, perceptible perturbations to images. Examples include robustness to spatial transformations [32, 33] and adversarial patches [34-38]. The majority of the work in this category considers local or simple transformations that retain most of original pixel content. Our benchmark considers a wider range of transformations, including ones that cause significant visual changes but do not change the image label.
3 Benchmark
3.1 Dataset
We choose to base our benchmark on the IMAGENET Large Scale Visual Recognition Challenge 2012 (ILSVRC2012)<sup>1</sup> dataset [39]. It has been established in the community as the main benchmark for image classification and there already exist several variations of it that aim to measure different aspects of robustness. This allows us to easily evaluate models trained on IMAGENET and its variants
<sup>1</sup>For simplicity we will refer to it as just "IMAGENET" for the remainder of the paper.
Figure 2: Accuracy on the hold-out obfuscations for multiple TensorFlow-Hub models trained on cleanImagenet.
on our benchmark and vice versa $^2$ . We apply each obfuscation to each image from the IMAGENET train and validation split. As the test split includes no labels, we use the validation split as our test set, similar to IMAGENET-C [10], and use 10k images of the original training set as a validation set.
The fine grained 1000-class task of the original IMAGENET set makes it hard to apply strong obfuscations as even for humans it is hard to, e.g., distinguish over 100 different breeds of dogs. To make the classification task easier, we use the 16 super-classes introduced by [40]. They encompass 207 of the 1000 IMAGENET classes (see appendix A.7 for a detailed listing). We only do this grouping at evaluation time, which allows us to compare models trained on the standard 1000-class IMAGENET scenario. As derived in the appendix of [40] we calculate the probability of an image to be part of a super-class by using the average probability of all member classes for each super class.
3.2 Obfuscations
Our benchmark includes 22 obfuscations in total: 19 training obfuscations and 3 hold-out obfuscations. These represent a wide range of strong and varied manipulations covering, color changes, transformations, compositions, overlays, machine-learning based obfuscations and combinations of them. To create a static benchmark, we do not include manipulations that require access to the model prediction.
The obfuscations have a number of hyperparameters, which each has an allowed range that we randomly draw from for each image. This makes the obfuscations more diverse and avoids overfitting. We tune these hyperparameter ranges manually to get strong obfuscations, that still keep the label intact. For each obfuscation we did a grid search to find the parameters that get the worst accuracy on a Big Transfer model [41] with an underlying ResNet 152x4 pretrained onImagenet-21K and fine-tuned onImagenet-1k. We then checked visually that out of 50 example images, a maximum of 2 are not classifiable as the right super-class. If more of the labels were not recognisable we checked parameters which resulted in higher accuracy until we found a fitting set. Next, we created the range of the possible parameters around this set to get variation in the applied obfuscation but also consistent performance.
Some of the obfuscations are very scale dependent. As the original IMAGENET images are of varying sizes, we do a central crop and resizing to  $224 \times 224$  before applying the obfuscations. None of
2The dataset and code to evaluate models can be found at https://github.com/deepmind/image_olfuscation_benchmark.
Figure 3: Worst case accuracy on hold-out obfuscations over GFLOPs (left, log scale) and number of parameters (right, log scale) for the models taken from TensorFlow-Hub. We did not include GFLOP numbers for the CLIP models as they are more than  $1000\mathrm{x}$  than the largest other models when doing 80 prompts for each of the 1000Imagenet classes.
the obfuscations change the input size, therefore all our training and evaluation data is  $224 \times 224$ . To stay consistent, we also do this to the clean training data, which will reduce the accuracy of the models trained for this paper as random cropping now operates on a smaller pixel space and needs to upsample.
We group the obfuscations into 5 categories: Color Changes, Transformations, Compositions, Overlays, Machine-Learning based Obfuscations and fig. 1 shows the hold-out obfuscations and a few examples of training obfuscations. For additional examples and a detailed description of each obfuscation, see appendix A.1 in the supplementary material.
When an obfuscation uses other images to overlay, merge or stylize the original images, we make sure that the chosen images do not introduce any objects which could be categorized into one of the 16 super-classes we evaluate on.
We chose ColorPatternOverlay, LowContrastTriangles and PerspectiveComposition as hold-out obfuscations because they cover multiple obfuscation categories (e.g. ColorPatternOverlay being both and overlay and a color change), they combine concepts from training obfuscations (PerspectiveComposition being a combination of PerspectiveTransform and PhotoComposition) and are among the hardest obfuscations for multiple model families (LowContrastTriangles is the strongest for ResNet models, while PerspectiveComposition is the strongest for Bit, ConvNext and ViT models).
3.3 Metric
Merging the classes into the 16 super-classes introduces a class imbalance in the evaluation data. We counteract this by weighting the accuracy by the inverse of the number of images of that super-class, thereby giving each super-class equal contribution.
As described earlier, we have 3 hold-out obfuscations that cannot be used during training. The accuracy on these is our main metric. To simulate adversaries trying out multiple obfuscations, we combine the obfuscations on a worst-case basis, i.e., to correctly classify an image, a model has to correctly classify it under all 3 hold-out obfuscations.
Since our super-classes are very unbalanced (see appendix A.7) calculating the average accuracy across all images would be mostly determined by a model's performance on a few super-classes. As an example: Just one super-class (Dog) would account for more than half of the overall accuracy, i.e. a classifier who perfectly classifies all dogs, but no other images, would get almost  $53\%$  accuracy, a classifier that perfectly classifies dogs and birds even over  $76\%$ . Therefore when we calculate the overall accuracy, we weight each image by the inverse of the super-class occurrence, i.e. we average the per-class accuracy.
Figure 4: Top 1 accuracy for the best models from the 5 pretrained model collections.
Figure 5: Accuracy on hold-out obfuscations and worst case accuracy for models trained with different augmentation schemes. All models are ResNet50 trained only on clean images. The augmentations used are color [42], AugMix [43], AutoAugment [44], RandAugment [45], random erasing [46], CutMix [47] and MixUp [48].
In summary we calculate our final metric by these steps:
1. Load the IMAGENET 2012 validation dataset
2. Filter out all images that do not share a class label with our 16 super classes
3. Obfuscate each image with the 3 hold-out obfuscations
4. For each obfuscated image evaluate the class probabilities for all 1000 IMAGENET classes
5. Calculate probabilities for each super class by averaging probabilities of all member classes
6. For each image check if the highest probability is for the correct super-class for all obfuscated version of that image
7. Calculate the final accuracy by averaging over all images weighted by the inverse of the super-class occurrence in the dataset
We start our experiments by evaluating a range of pretrainedImagenet classification models to analyse the robustness of different architectures to the obfuscations, and the effect of scaling models and pretraining datasets. We then compare 7 data augmentation schemes, and in section 4.3, train models on different subsets of training obfuscations to see their contribution to generalizing to the hold-out obfuscations. We then look at models trained on all training obfuscations, evaluate the effect of distribution shift algorithms, and at the end compare with results on other robustness benchmarks.
4.1 Evaluating PretrainedImagenet Models
All of the following models are evaluated on training and hold-out obfuscations as is, without any fine-tuning. We evaluate models from four different collections on TensorFlow-Hub<sup>3</sup>: Inception
<sup>3</sup>https://www.tensorflow.org/hub
Figure 6: Accuracy on hold-out obfuscations and worst case for models trained on clean images plus a single obfuscations.
and  $\mathrm{ResNet}^4$ , Big Transfer $^5$ , Vision Transformer $^6$  and ConvNext $^7$ . We also include several zero-shot models based on CLIP [54], with different image encoder architectures $^8$ . To obtain the best performance possible for CLIP, we used the best text-prompt configuration reported by [54] for ImageNet, consisting of a combination of 80 curated prompts. The accuracy of all models on the hold-out obfuscations and the worst case accuracy for all the models is plotted in fig. 2. We see that the ViT-B8 performs the best across all models, as it does on the standard IMAGENET dataset. The results for the Big Transfer models show the gain in accuracy by pretraining on the IMAGENET-21K dataset (BiT-S models are trained on standard IMAGENET, i.e. ILSVRC2012, while BiT-M models, are pretrained on IMAGENET-21K). This can be seen in more detail in fig. C.2 in the appendix, where we see accuracy for each individual obfuscation for all BiT models.
The performance of zero-shot models exhibits an interesting behavior. On some obfuscations such as ColorPattenOverlay and LowContrastTriangles the performance of the zero-shot CLIP model is slightly worse than that of the equivalent trained model with the same vision tower. However, on PerspectiveComposition it actually performs better than the trained model. This behavior is probably due to the distribution of the pretraining data distribution—PerspectiveComposition is a more "natural" looking type of obfuscation, which might have been present in CLIP's pretraining dataset.
Figure 3 shows that when comparing models of the same type, scaling them up in terms of parameters or computation leads to increased robustness to the obfuscations. It also shows that ConvNext and ViT models outperform BiT and ResNet models. This might be due to their patchified stem that has been shown to be more robust to  $\ell_p$ -norm adversarial attacks [55]. Additionally, both ViT and ConvNext models are all pretrained onImagenet-21K. In fig. 4, we look at the results for the best model from each category over all obfuscations. While the largest vision transformer model outperforms all other models on the worst-case hold-out accuracy, it does not get the best performance across all obfuscations, e.g. the largest ConvNext model performs better on 8 of the 22 obfuscations, in some cases by a large margin (e.g. HighContrastBorder by  $8\%$  and PhotoComposition by  $8.7\%$ ), indicating that different architectures are more robust to different obfuscations.
4.2 Comparing Augmentation Methods
One way to make models robust to out of distribution data is to use general augmentation schemes. In fig. 5, we see that all augmentations help the worst case accuracy, but only CutMix [47] and MixUp [48] improve accuracy across all 3 hold-out obfuscations, with MixUp giving by far the strongest boost overall. AugMix [43], which is one of the strongest methods on IMAGENET-C, gives a big boost on LowContrastTriangles but does not improve accuracy on PerspectiveComposition, which is very different from the natural corruptions in IMAGENET-C. This highlights that while many augmentations help with obfuscation robustness most are not universally helpful across all of them.
<sup>4</sup>https://tfhub.dev/google/collections/image based on [49] and [50].
<sup>5</sup>https://tfhub.dev/google/collections/bit/ based on [41].
<sup>6</sup>https://tfhub.dev/sayakpaul/collections/vision_transformer based on [51] and [52].
<sup>7</sup>https://tfhub.dev/sayakpaul/collections/convnext based on [53].
<sup>8</sup>https://github.com/OpenAI/CLIP
Figure 8: Top 1 accuracy for 3 different ResNet sizes when training on all training obfuscations.
4.3 Training on Obfuscations
In this section we train models on all or subsets of the 19 training obfuscations to analyse the effects and interactions. Unless specified otherwise, the models use a ResNet50 architecture and if error bars are plotted they represent the standard deviation from training 5 identical models with different random seeds. When we train on obfuscated images we always sample clean images with a weight of 0.5 and each obfuscation with equal weights summing up to 0.5.
4.3.1 Training on Subsets of Training Obfuscations
To see if there are interactions between different training obfuscations and how each of them helps with generalizing to the hold-out obfuscations, we train models only on one obfuscation (and clean data). The results can be seen in fig. 6. There are some clear connections between training and hold out obfuscations, e.g. Halftoning, LineShift, StyleTransfer, TextOverlay and Texturize all increasing the accuracy on ColorPatternOverlay significantly, while PerspectiveTransform and PhotoComposition lead to small improvements of PerspectiveComposition.
In appendix C.4 we also investigate the opposite, where we train models on all but one of the training obfuscations and observe similar behaviour for some obfuscations, e.g. that omitting LineShift significantly reduces the performance on ColorPatternOverlay but in other cases, like excluding StyleTransfer, this is compensated by the other training obfuscations.