SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 7: Accuracy on hold-out obfuscations (top) and worst case accuracy (bottom) when training on an increasing number of obfuscations.
To evaluate the effect of compounding training obfuscations, we proceed by sorting the training obfuscations by the mean accuracy on the training obfuscations when training only on images from that obfuscation. We then train models by an increasing number of obfuscations starting with the obfuscation that increases the mean accuracy the most. From the results in fig. 7 we can observe that the accuracy makes jumps when adding specific obfuscations that help with one of the hold-out obfuscations but there does not seem to be constant improvement from adding more and more obfuscations to the training data. We also observe that adding the obfuscated images to the training data does not reduce the clean accuracy of the models. We rather see a mild increase in clean accuracy when looking at fig. C.14.
4.3.2 Using All Training Obfuscations
In fig. 8 we see all models, even small ones, can achieve high accuracy on the training obfuscations when they see them during training. However, there is only limited generalization to the hold-out obfuscations even for the bigger models. When comparing results from fig. 2, we see that training a ResNet200 on all 19 training obfuscations is clearly outperformed by the larger BiT, ConvNext and
Figure 9: Comparison of a vision transformer model trained only on clean or clean and obfuscated images both with and without pretraining on the IMAGENET-21K dataset.
Figure 10: Comparing performance of different domain shift algorithms on the hold-out obfuscations, when training on all training obfuscations.
Figure 11: Hold-out accuracy of models with and without adversarial training both for training only on clean data and training on the training obfuscations. For adversarial training we used an  $L_{\infty}$  attack with  $\epsilon = 4 / 255$ .
Figure 12: Comparison of a ResNet50 trained with and without the training obfuscations on multipleImagenet variants.
ViT models despite them never encountering any of the obfuscations during training. This shows that there is still a lot of potential to better leverage the training obfuscations for generalization.
We also train a vision transformer model both only on clean, and obfuscated images. Similar to results in section 4.1, fig. 9 shows that this achieves high accuracy on the hold-out obfuscations even without seeing obfuscations during training. Interestingly, only ColorPatternOverlay and LowContrastTriangles receive a significant boost from the addition of obfuscations to training. Pretraining onImagenet-21K seems to not provide significant benefits in contrast to the effect we see for the BiT models in fig. 2.
Evaluating Algorithms Specialized for Distribution Shift To see if we can improve generalization, we employ several approaches that were proposed to help with distribution shifts. The algorithms we evaluated are standard training using cross-entropy loss, Invariant Risk Minimization (IRM) [4], DeepCORAL [56], Domain-Adversarial Neural Network (DANN) [57], and Just Train Twice (JTT) [58]. Additional to the image and the label, these algorithms (besides JTT) are given side information about the obfuscation that has been applied to each training image. Similar to the observations in [59], fig. 10 indicates that none of the algorithms give significant improvements over the baseline.
4.4 Comparing to other benchmarks
In section 2, we gave an overview over existing image robustness benchmarks, many also based on IMAGENET. In this section we investigate how performance on these relates to our image obfuscation benchmark. Not surprisingly, training on our obfuscations does not give any robustness to  $L_{p}$ -norm attacks but figure fig. 11 shows that adversarial training reduces the obfuscation robustness. This is in contrast to prior observations on the effect of adversarial training for robustness to common corruptions [60] but understandable because our obfuscations often change the images drastically.
We further investigate how models trained on our training observations do on other IMAGENET variants. fig. 12 shows results for IMAGENET-Real [61], IMAGENET-V2 [62], IMAGENET-A [6], IMAGENET-R [9], IMAGENET-Sketch [63], Conflict Stimuli [64], IMAGENET-C [10] and IMAGENET-C [11]. Training on the obfuscations improves accuracy across all of these variants, however, the improvements are relatively small, indicating that our dataset represents a significantly different distribution shift than existing variants. This can also be seen in the fact that the largest improvement is seen on IMAGENET-C, as its corruptions are more similar to our obfuscations compared to the other variants.
In this paper, we presented a new benchmark that evaluates the robustness of image classifiers to obfuscations. To our knowledge, this is the first benchmark that curates obfuscations similar to what bad actors use to circumvent content filter models. We show that when training on obfuscations, even smaller models can achieve high robustness to them, but this does not necessarily lead to strong generalization on similar but unseen obfuscations. In our experiments, we see that newer architectures, larger models, augmentation schemes and pretraining on bigger datasets all can make models more robust to obfuscations, even if they did not have access to any during training. But there is still a gap to fill to make models robust to unseen attacks and approach human perception. We have shown that models trained on our training obfuscations also achieve better performance across multiple other robustness benchmarks. On the other hand, adversarial training does not improve obfuscation robustness. We hope this benchmark gives practitioners guidance on robustifying their models, and can drive research towards finding ways to leverage known attacks into better generalization.
5.1 Limitations
Investigating adversarial obfuscations leads to a large scope of design decisions. In terms of transformations, we focus on obfuscations that are independent of the model and semantic image category. This allows precomputation of obfuscated images, but also limits the space of attacks. We could further develop obfuscation methods that adapt to the image content (e.g., human-centric images may be treated differently than object-centric). In terms of data, IMAGENET is far from perfect (see e.g. [65] for an evaluation of errors made by state-of-the-art models) and has the limitation of having a single label for each image. One could generate obfuscated versions of other datasets to check the robustness of models trained in a multi-class setting or on other vision tasks like segmentation or image retrieval. The main consideration for other datasets should be to ensure that the main target label (or other output) should be very unlikely to be altered by any of the introduced obfuscations.
5.2 Ethical Considerations
The specific obfuscations (as in fig. 1) that we use in our benchmark may have the potential to fool automatic filters and therefore increase the amount of harmful content on digital platforms. To reduce this risk, we decided against releasing the code to create the obfuscations systematically and instead only releasing the precomputed dataset. Furthermore, our obfuscations have been tuned specifically to IMAGENET images of a fixed size. Using the same obfuscations on other images would require a reimplementation and additional tuning. It is already known that cloud based image classifiers can be bypassed with widely available transformations [66, 67]. Additionally, the type of obfuscations that we cover in our benchmark are already available in standard image editing software and it is not hard to imagine that adversaries can already think of much more elaborate approaches than the ones we have presented here. Therefore, the benefits of creating a publicly available benchmark that can help discover new methods for training robust models and set an objective baseline for the evaluation of safety far outweigh the risks of presenting examples of obfuscations on IMAGENET.
We want to thank Chun-Sung Ferng, Dongjin Kwon, Sylvestre-Alvise Rebuffi and Olivia Wiles for their help with this work.
[61] L. Beyer, O. J. Henaff, A. Kolesnikov, X. Zhai, and A. van den Oord, “Are we done with imagenet?” CoRR, vol. abs/2006.07159, 2020. [Online]. Available: https://arxiv.org/abs/2006.07159
[62] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, “Do ImageNet classifiers generalize to ImageNet?” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 5389–5400.
[63] H. Wang, S. Ge, Z. Lipton, and E. P. Xing, “Learning robust global representations by penalizing local predictive power,” in Advances in Neural Information Processing Systems, 2019, pp. 10506–10518.
[64] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, "Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness," in ICLR. OpenReview.net, 2019.
[65] V. Vasudevan, B. Caine, R. Gontijo-Lopes, S. Fridovich-Keil, and R. Roelofs, “When does dough become a bagel? analyzing the remaining mistakes on imagenet,” 2022. [Online]. Available: https://arxiv.org/abs/2205.04596
[66] D. Goodman and T. Wei, “Cloud-based image classification service is not robust to simple transformations: A forgotten battlefield,” CoRR, vol. abs/1906.07997, 2019. [Online]. Available: http://arxiv.org/abs/1906.07997
[67] D. Goodman and X. Hao, “Attacking and defending machine learning applications of public cloud,” CoRR, vol. abs/2008.02076, 2020. [Online]. Available: https://arxiv.org/abs/2008.02076
[68] G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, and J. Shlens, “Exploring the structure of a real-time, arbitrary neural artistic stylization network,” ArXiv, vol. abs/1705.06830, 2017.
[69] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," in ICLR, 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7
[70] S.-A. Rebuffi, F. Croce, and S. Gowal, “Revisiting adapters with adversarial training,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=HPdxC1THU8T
[71] V. Vasudevan, B. Caine, R. Gontijo-Lopes, S. Fridovich-Keil, and R. Roelofs, “When does dough become a bagel? Analyzing the remaining mistakes on ImageNet,” arXiv preprint arXiv:2205.04596, 2022.
A Benchmark Details
A.1 Obfuscations
In the following section we give a description of all 18 training and 3 hold-out obfuscations in the benchmark. The hyperparameters that are drawn randomly drawn from predefined ranges for each image are emphasized. The obfuscations are grouped into 5 categories but these categorizations are very loose as some obfuscations, especially the hold-out obfuscations, can cover multiple concepts. Examples of the training and hold-out obfuscations can be seen in fig. A.1 and fig. A.2, respectively.
A.2 Color Changes
ColorNoiseBlocks The image is separated into blocks of a specific block size. Independently for each block, one of the 3 color changes is randomly selected and a uniform random value is added to that color channel. We ensure the value is between 0 and 1 by subtracting 1 from it if it goes above.
Halftoning (technique) We randomly apply one of 4 different halftoning techniques to an image. Each technique reproduces the image by separating the image into blocks of equal size and replacing them by some geometric object that has a property that is proportional to the average intensity in the block. This is done for each color channel independently. The 4 techniques are using circles or squares with their size being proportional to the intensity, using zigzag lines whose frequency is proportional to the intensity or using random pixels whose number is proportional to the intensity.
InvertLines We go along the image in either horizontal or vertical lines of a specific width. The lines are alternatively inverted in color or left unchanged.
LowContrastTriangles (hold-out) We divide the image into 3 different areas through triangles of size scale. In each of the 3 areas areas the contrast is reduced by a different factor.
A.3 Transformation
LineShift (horizontal, shift length, line thickness) Across either horizontal or vertical lines of a specific width we shift the pixels alternatively to the left and right (or top and bottom for vertical lines) by a specific length. We wrap around when the shift goes beyond the image borders.
PerspectiveTransform We choose a set of coordinates around the center of each of the quadrants. Then we apply a perspective transformation moving the corners of the image to their respective coordinate. The rest of the image is filled with black.
RotateBlocks We separate the image into blocks of a chosen size. Each of them is separated into 4 blocks of equal size again and their position inside the larger block is permuted by a specified number of rotations.
RotateImage We rotate the image by a specified angle. Corners will be cropped while empty space is filled with black.
SwirlWarp The image is transformed by a swirl warp with specified strength, radius and center coordinates.
WavyColorWarp We transform the image with a sine wave transformation of a certain wavelength and amplitude and modify the color of the image through a change in hue..
A.4 Composition
BackgroundBlurComposition We shrink the image by an independent width and height factor, not necessarily keeping the aspect ratio, and compose this image on top of the original image after applying a Gaussian blur of a certain strength on the latter.
(a) Clean
(c) BackgroundBlurComposition
(d) ColorNoiseBlocks
(b) AdversarialPatches
(e) Halftoning
(f) HighContrastBorder
(g) IconOverlay
(h) ImageOverlay
(i) Interleave
(j) InvertLines
(k) LineShift
(1) PerspectiveTransform
(m) PhotoComposition
(q) SwirlWarp
Figure A.1: Examples for the training obfuscations.
(n) RotateBlocks
(r) TextOverlay
(o) RotateImage
(s) Texturize
(p) StyleTransfer
(t) WavyColorWarp
(a) ColorPatternOverlay
(b) LowContrastTriangles
(c) PerspectiveComposition
Figure A.2: Examples for the hold-out obfuscations.
HighContrastBorder The image has its contrast reduced by a factor then it is resized to have a border of a certain size around it. The border is made by sampling each pixel uniform randomly.
PerspectiveComposition (hold-out) We compose the image into a specific location (fixed for each photo). The position also defines a perspective transform that is applied to the image. Small parts of the image can also be overlayed by the photo. We use 14 different photos and the position that the original image is composed into has semantic meaning, e.g. it is the screen of a phone or the glass of a window.
PhotoComposition The image is shrunk by a factor and composed on top of a photo at a random position. There are 10 different photos. They are the same photos that are used for ImageOverlay and Interleave. There is no overlap with the photos that are used for PerspectiveComposition.
A.5 Overlay
ColorPatternOverlay (hold-out) The image is turned into grey scale then overlayed by one of 9 patterns in one of 9 colors with a low grade of transparency.
IconOverlay A grid of a certain number of one of 10 icons is overlayed on the image with a grade of transparency.
ImageOverlay One of 10 different photos is overlayed over the image with a grade of transparency. The photos are the same that are used for ImageOverlay and Interleave. There is no overlap with the photos that are used for PerspectiveComposition.
Interleave The image is interleaved with one of 10 photos, either horizontally or vertically, in lines of a specific width. There is a low grade of transparency. The photos that are used are the same as for ImageOverlay and Interleave. There is no overlap with the photos that are used for PerspectiveComposition.
TextOverlay One of 13 text strings is overlayed onto the image repeatedly in one of 9 colors. The text size is varied.
A.6 Machine-Learning-Based Obfuscations
Adversarial Patches The image is shrunk and one of 3 different adversarial patches are overlayed onto the corners of the image. The patches are taken from [38].
Stylize We resize the image by a factor  $(>1)$  and then choose one of 7 classical paintings to stylize the image. We use a model based on  $[68]^{9}$  for the style transfer.
9Taken from https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2
Texturize We resize the image by a factor  $(>1)$  and then choose one of 10 texture photos to stylize the image. We use a model based on [68]footnote 9 for the style transfer.
A.7 Class reduction
To allow strong obfuscations we reduce the number of classes that need to be distinguished. We follow [40] by using 16 super-classes which encompass 207 of the original 1000 IMAGENET classes.
The 16 super-classes, their IMAGENET label id and the corresponding WordNet terms are listed below. The names given to the super-classes were chosen by us.
Airplane (1)
- 404 (airliner)
- Bear (4)
- 294 (brown bear, bruin, Ursus arctos), 295 (American black bear, black bear, Ursus americanus, Euarctos americanus), 296 (ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus), 297 (sloth bear, Melursus ursinus, Ursus ursinus)
- Bicycle (2)
- 444 (bicycle-built-for-two, tandem bicycle, tandem), 671 (mountain bike, all-terrain bike, off-roader)
- Bird (49)
- 8 (hen), 10 (brambling, Fringilla montifringilla), 11 (goldfinch, Carduelis carduelis), 12 (house finch, linnet, Carpodacus mexicanus), 13 (junco, snowbird), 14 (indigo bunting, indigo finch, indigo bird, Passerina cyanea), 15 (robin, American robin, Turdus migratorius), 16 (bulbul), 18 (magpie), 19 (chickadee), 20 (water ouzel, dipper), 22 (bald eagle, American eagle, Haliaetus leucocephalus), 23 (vulture), 24 (great grey owl, great gray owl, Strix nebulosa), 80 (black grouse), 81 (ptarmigan), 82 (ruffed grouse, partridge, Bonasa umbellus), 83 (prairie chicken, prairie grouse, prairie fowl), 87 (African grey, African gray, Psittacus erithacus), 88 (macaw), 89 (sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita), 90 (lorikeet), 91 (coucal), 92 (bee eater), 93 (hornbill), 94 (hummingbird), 95 (jacamar), 96 (toucan), 98 (red-breasted merganser, Mergus serrator), 99 (goose), 100 (black swan, Cygnus atratus), 127 (white stork, Ciconia ciconia), 128 (black stork, Ciconia nigra), 129 (spoonbill), 130 (flamingo), 131 (little blue heron, Egretta caerulea), 132 (American egret, great white heron, Egretta albus), 133 (bittern), 135 (limpkin, Aramus pictus), 136 (European gallinule, Porphyrio porphyrio), 137 (American coot, marsh hen, mud hen, water hen, Fulica americana), 138 (bustard), 139 (reddy turnstone, Arenaria interprets), 140 (red-backed sandpiper, dunlin, Erolia alpina), 141 (redshank, Tringa totanus), 142 (dowitcher), 143 (oystercatcher, oyster catcher), 144 (pelican), 145 (king penguin, Aptenodytes patagonica)
- Boat (5)
- 472 (canoe), 554 (fireboat), 625 (lifeboat), 814 (speedboat), 914 (yawl)
- Bottle / Jug (7)
- 440 (beer bottle), 720 (pill bottle), 737 (pop bottle, soda bottle), 898 (water bottle), 899 (water jug), 901 (whiskey jug), 907 (wine bottle)
- Car (3)
- 436 (beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon), 511 (convertible), 817 (sports car, sport car)
- Cat / Cougar (6)
- 281 (tabby, tabby cat), 282 (tiger cat), 283 (Persian cat), 284 (Siamese cat, Siamese), 285 (Egyptian cat), 286 (cougar, puma, catamount, mountain lion, painter, panther, Felis concolor)
Chair / Throne (4)
- 423 (barber chair), 559 (folding chair), 765 (rocking chair, rocker), 857 (throne)
- Clock (3)
- 409 (analog clock), 530 (digital clock), 892 (wall clock)
Dog (109)
- 152 (Japanese spaniel), 153 (Maltese dog, Maltese terrier, Maltese), 154 (Pekinese, Pekingese, Peke), 155 (Shih-Tzu), 156 (Blenheim spaniel), 157 (papillon), 158 (toy terrier), 159 (Rhodesian ridgeback), 160 (Afghan hound, Afghan), 161 (basset, basset hound), 162 (beagle), 163 (bloodhound, sleuthhound), 164 (bluetick), 165 (black-and-tan coonhound), 166 (Walker hound, Walker foxhound), 167 (English foxhound), 168 (redbone), 169 (borzoi, Russian wolfhound), 170 (Irish wolfhound), 171 (Italian greyhound), 172 (whippet), 173 (Ibizan hound, Ibizan Podenco), 174 (Norwegian elkhound, elkhound), 175 (otterhound, otter hound), 176 (Saluki, gazelle hound), 177 (Scottish deerhound, deerhound), 178 (Weimaraner), 179 (Staffordshire bullterrier, Staffordshire bull terrier), 180 (American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier), 181 (Bedlington terrier), 182 (Border terrier), 183 (Kerry blue terrier), 184 (Irish terrier), 185 (Norfolk terrier), 186 (Norwich terrier), 187 (Yorkshire terrier), 188 (wire-haired fox terrier), 189 (Lakeland terrier), 190 (Sealyham terrier, Sealyham), 191 (Airedale, Airedale terrier), 193 (Australian terrier), 194 (Dandie Dinmont, Dandie Dinmont terrier), 195 (Boston bull, Boston terrier), 196 (miniature Schnauzer), 197 (giant Schnauzer), 198 (standard Schnauzer), 199 (Scotch terrier, Scottish terrier, Scottie), 200 (Tibetan terrier, chrysanthemum dog), 201 (silky terrier, Sydney silky), 202 (soft-coated wheaten terrier), 203 (West Highland white terrier), 205 (flat-coated retriever), 206 (curly-coated retriever), 207 (golden retriever), 208 (Labrador retriever), 209 (Chesapeake Bay retriever), 210 (German short-haired pointer), 211 (vizsla, Hungarian pointer), 212 (English setter), 213 (Irish setter, red setter), 214 (Gordon setter), 215 (Brittany spaniel), 216 (clumber, clumber spaniel), 217 (English springer, English springer spaniel), 218 (Welsh springer spaniel), 219 (cocker spaniel, English cocker spaniel, cocker), 220 (Sussex spaniel), 221 (Irish water spaniel), 222 (kuvasz), 223 (schipperke), 224 (groenendael), 225 (malinois), 226 (briard), 228 (komondor), 229 (Old English sheepdog, bobtail), 230 (Shetland sheepdog, Shetland sheep dog, Shetland), 231 (collie), 232 (Border collie), 233 (Bouvier des Flandres, Bouviers des Flandres), 234 (Rottweiler), 235 (German shepherd, German shepherd dog, German police dog, alsatian), 236 (Doberman, Doberman pinscher), 237 (miniature pinscher), 238 (Greater Swiss Mountain dog), 239 (Bernese mountain dog), 240 (Appenzeller), 241 (EntleBucher), 243 (bull mastiff), 244 (Tibetan mastiff), 245 (French bulldog), 246 (Great Dane), 247 (Saint Bernard, St Bernard), 248 (Eskimo dog, husky), 249 (malamute, malemute, Alaskan malamute), 250 (Siberian husky), 252 (affenpinscher, monkey pinscher, monkey dog), 253 (basenji), 254 (pug, pug-dog), 255 (Leonberg), 256 (Newfoundland, Newfoundland dog), 257 (Great Pyrenees), 259 (Pomeranian), 261 (keeshond), 262 (Brabancon griffon), 263 (Pembroke, Pembroke Welsh corgi), 265 (toy poodle), 266 (miniature poodle), 267 (standard poodle), 268 (Mexican hairless)
- Elephant (2)
- 385 (Indian elephant, Elephas maximus), 386 (African elephant, Loxodonta africana)
- Keyboard / Typewriter (2)
- 508 (computer keyboard, keypad), 878 (typewriter keyboard)
- Cleaver (1)
- 499 (cleaver, meat cleaver, chopper)
- Rotisserie (1)
-766 (rotisserie)
Van / Truck (8)
- 555 (fire engine, fire truck), 569 (garbage truck, dustcart), 656 (minivan), 675 (moving van), 717 (pickup, pickup truck), 734 (police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria), 864 (tow truck, tow car, wrecker), 867 (trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi)
Training Details Unless other specified we trained a ResNet50 model for 300 IMAGENET equivalent epochs, with  $50\%$  of the training data clean (central cropped and resized to  $224 \times 224$ ) and
Figure C.1: Heatmap of top accuracy for all the obfuscations for multiple Inception and ResNet variants.
the rest being sampled with equal probability from the training obfuscations or a subset of them. As augmentations random crop and color augmentations as in [42] were used.