SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We trained models in two different frameworks with slightly different parameters. Experiments on the augmentations (fig. 5), adversarial training (fig. 11), the ImageNet variants (fig. 12) and the vision transformers (fig. 9) used framework 1. For ResNet we used SGD with momentum 0.9 and a cosine learning rate decay with base learning rate  $4e^{-4}$  after linear ramp-up of 10 epochs and a batch size of 4096.
For the ViTs we used a ViT-B16 model with a weight decay of 0.3, label smoothing of 0.1, an exponential moving average with momentum 0.9999. We used the AdamW optimizer [69] with momenta  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.95$  and base learning rate  $1e^{-4}$  with the same learning rate schedule as the ResNet50 above. For data augmentations we used random crops, as well as MixUp, CutMix and RandAugment, the latter using 2 layers, magnitude 9 and a random probability of 0.5.
Results on all the other experiments used framework 2 which used ADAM optimizer with learning rate  $1e^{-3}$  which we chose after comparing experiments with learning rate  $1e^{-4}$  and learning rate  $1e^{-2}$  and a batch size of 512.
Adversarial Training For results in fig. 11 we used adversarial training that used untargeted  $\ell_{\infty}$  attacks with  $\epsilon = 4 / 255$ . We follow [70] by creating the adversarial perturbations through 2 step projected gradient descent (PGD²) with a step size of  $\frac{5}{8}\epsilon \approx 0.01$ . We used early stopping to avoid robust overfitting. In case obfuscated images were used the attacks were done on both clean and obfuscated images.
To give more insights into the experiments we add more detailed figures, including both training and hold-out obfuscations, and new analysis for experiments in the main paper.
C.1 Evaluating Baseline Models Trained on Clean IMAGENET
In section 4.1 we showed results from models taken from TensorFlow hub that were trained on the standardImagenet dataset (and pretrained onImagenet-21K in some cases). Figure 2 showed the performance of all the models, but only on the hold-out obfuscations, while fig. 4 included the accuracy on all obfuscations, training and hold-out, but only for the best models from each type. In figs. C.1 to C.4 we show the results on all obfuscations for Inception and ResNet, BiT, ConvNext and
Figure C.2: Heatmap of the top 1 accuracy for all the obfuscations for multiple BiT variants.
Figure C.3: Heatmap of top 1 accuracy for all the obfuscations for multiple ConvNext variants.
Figure C.4: Heatmap of top 1 accuracy for all the obfuscations for multiple ViT variants.
Figure C.5: Heatmap of top 1 accuracy for all the obfuscations for multiple CLIP models using 80 prompts.
Figure C.6: Worst case accuracy on the hold-out obfuscations over accuracy on standard ImageNet (using all 1000 classes) for all the models taken from TensorFlow Hub.
Figure C.7: Heatmap of the top 1 accuracy for all the obfuscations for cleanly trained ResNet50 models with different augmentation schemes.
ViT models, respectively. It allows a more detailed look in how robustness to different obfuscations evolves when models are scaled up or are pretrained on IMAGENET-21K.
In fig. C.1 we can see that the different Inception and ResNet models get almost the exact same accuracy on some obfuscations like ImageOverlay or InvertLines, while other obfuscations, e.g. Halftoning or RotateImage, are easier to classify for the bigger models. Interestingly, this does not hold for the other model types as we see both ImageOverlay and InvertLines get improved performance from larger models in figs. C.2 to C.4.
We can also see in figs. C.2 and C.3 how pretraining onImagenet-21K helps make models more robust to the obfuscations. When comparing the BiT-S models to their corresponding BiT-M models we see an improvement on all obfuscations while for ConvNext-Base and ConvNext-Base-21k models this is more mixed, e.g. IconOverlay Interleave, LowContrastTriangles and PerspectiveComposition show no improvement, mirroring our results in fig. 9.
We mentioned in section 4.1 that the models that perform strongest on our hold-out obfuscations are also have the highest accuracy on the standard IMAGENET benchmark. This is visualised in fig. C.6 showing a clear correlation between the accuracy on the 1000-class accuracy on standard IMAGENET and the worst-case accuracy on our hold-out obfuscations.
In fig. C.7 we can see how the augmentation schemes that we evaluated in the main paper perform on all the obfuscations. As in fig. 5 we see that MixUp performs best, increasing the accuracy significantly for 15 of the 19 training obfuscations. You can also see clear connections between the augmentation methods and the obfuscations they do particularly well on. For example, MixUp does best on IconOverlay, ImageOverlay and TextOverlay, while CutMix does best on InvertLines, BackgroundBlurComposition and PhotoComposition.
C.2 Evaluating Upper Ceiling for Benchmark Metric
While we tried to carefully tune our obfuscations to be strong and keep the semantic content of the images intact, it is inevitable that there will be cases where the image is obfuscated beyond recognition. This is not unusual, even the clean IMAGENet dataset contains wrong labels and really hard images that even humans fail to label correctly [71]. To evaluate the upper ceiling of performance on our benchmark we create an oracle by combining multiple models in a best case way, i.e. we use the predictions of the ViT B8 model from fig. C.4 for ColorPatternOverlay and LowContrastTriangles and the predictions of the CLIP ViT L14 from fig. C.5 for PerspectiveComposition as these models get the best results on these hold-out obfuscations. Doing this we get a worst case hold out accuracy of  $69.82\%$  which is almost  $8\%$  higher than the best model (the ViT B8) gets. If we go further and combine both these models on an image-by-image basis, i.e. we give the correct prediction if at least one of the models predicts the correct super class. Doing this yields a worst case hold out accuracy of  $80.36\%$ . If we extend this to the best model from each of the 5 pretrained groups (Inception ResNet V2, BiT-M ResNet-152x4, ConvNext-XLarge-21k-1k, ViT B8 and CLIP ViT L14) we get to  $86.89\%$  and when using all 38 pretrained models we get  $95.12\%$  although the later might be discarded as too extreme as obviously we could get arbitrarily close to  $100\%$  by combining more and more random models. Nonetheless, we think this analysis shows that results over  $80\%$ , and therefore roughly  $20\%$  higher than what the best models we evaluated achieved, are realistic for the hold out worst case accuracy.
C.3 Evaluating Choice of Hold Out Obfuscations
As described in section 3.2 we chose ColorPatternOverlay, LowContrastTriangles and PerspectiveComposition as hold-out obfuscations because of their conceptual diversity and difficulty. To investigate how our choice compares we plot the histogram of the worst case hold out accuracy over all 1540 possible combinations to chose 3 out of 22 obfuscations as hold out. As you can see from figs. C.8 to C.12, our choice is always amongst the lowest in terms of the worst case hold out accuracy, which underlines that we chose one of the hardest sets of hold out obfuscations. The only exception for this seems to be the CLIP models, which is most likely because, as seen in fig. C.5, PerspectiveComposition is one of the easier transformations for it.
Figure C.8: Histogram of worst case hold out accuracy for all 1540 possible choices of 3 hold out obfuscations for the ResNet models. Our choice is marked with a vertical black line.
Figure C.9: Histogram of worst case hold out accuracy for all 1540 possible choices of 3 hold out obfuscations for the BiT models. Our choice is marked with a vertical black line.
Figure C.10: Histogram of worst case hold out accuracy for all 1540 possible choices of 3 hold out obfuscations for the ConvNext models. Our choice is marked with a vertical black line.
Figure C.11: Histogram of worst case hold out accuracy for all 1540 possible choices of 3 hold out obfuscations for the ViT models. Our choice is marked with a vertical black line.
C.4 Evaluating Models Trained on Subsets of Training Obfuscations
The results for all obfuscations when training on a single obfuscation (and clean images) are show in fig. C.13, corresponding to the hold-out results shown in fig. 6. The heatmap allows us to see relationships between training obfuscations, e.g. training on Interleave giving strong boosts to the accuracy on InvertLines and LineShift or training on RotateImage improving robustness to SwirlWarp. Sometimes this relationships is not symmetric e.g. while training on PhotoComposition gives a very large boost of over  $30\%$  on BackgroundBlurComposition the reverse only improves performance by less than  $10\%$ .
In fig. C.14 we see what happens to robustness to each obfuscation when adding more and more obfuscations to the training data. In fig. 7 in the main paper we saw the the hold-out performance has jumps when certain obfuscations are added but stays mostly flat otherwise. Here we can see are more nuanced picture where there is some clear generalization to other obfuscations initially but when adding obfuscations like TextOverlay later on there is no significant increase in the performance of other training obfuscations. This is not surprising as the obfuscations to add to the training set were ordered by their average performance across the training obfuscations in fig. C.13.
Figure C.12: Histogram of worst case hold out accuracy for all 1540 possible choices of 3 hold out obfuscations for the CLIP models. Our choice is marked with a vertical black line.
Figure C.13: Heatmap of top 1 accuracy for all the obfuscations for models trained on clean images plus a single obfuscations.
Figure C.14: Heatmap of top 1 accuracy for all the obfuscations for models trained on an increasing number of obfuscations.
Fig. C.15 shows accuracy across obfuscations when training on all but one of the training obfuscations. The results on the diagonal show which of the obfuscations can be generalized on from training on the others. Similar to the results on the hold-out obfuscations in fig. 8 we see that generalization to unseen obfuscations is varied and in some cases, like IconOverlay, extremely limited for ResNet models.
C.5 Comparing Other Models and Training Approaches
In fig. C.16 we see the results on all obfuscations when training with the different distribution shift algorithms from fig. 10. fig. C.17 shows the same heatmap for the ViT models we trained and whose hold-out accuracy we plotted in fig. 9. Similar to the results from the ViT models we took from Tensorflow Hub (fig. C.4) we can see that even when not training on any of the obfuscations, ViTs can achieve solid performance on most obfuscations. We also again see only small gains from pretraining on IMAGENET-21K, in contrast to the results we saw when comparing BiT and ConvNext models with and without pretraining. One possible explanation might be that pretraining only has a strong affect when performance is lower than what our ViT achieves already.
Another interesting observation is that the ViTs trained on all the training obfuscations achieve even higher performance than the ResNets we trained for fig. 8. All training obfuscations get an accuracy above  $90\%$  most even above  $95\%$ , showing that we were successful in tuning our obfuscations to keep the label intact for almost all images.
We see a more detailed view of the effects of adversarial training on the robustness to obfuscations in fig. C.18. This is based on the same experiments shown in fig. 11 but we can see that while adversarial training did not improve the accuracy on any of the hold-out obfuscations, it does offer significantly improved performance on some of the training obfuscations when training only on
Figure C.15: Heatmap of top 1 accuracy for all the obfuscations for models trained on all but one training obfuscation.
Figure C.16: Heatmap of top 1 accuracy for all the obfuscations for models trained on different distribution shift algorithms.
Figure C.17: Heatmap of top 1 accuracy for all the obfuscations for ViT-B16 model trained on clean images only or on the training obfuscations with and without pretraining onImagenet-21K
Figure C.18: Heatmap of top 1 accuracy for all the obfuscations for models trained on clean images only or on the training obfuscations with and without adversarial training.
clean images, namely Halftoning, IconOverlay, Interleave, LineShift RotateBlocks, TextOverlay and WavyColorWarp.
C.6 Comparing to Other Benchmarks
In fig. C.19 we plot the performance on the individual IMAGENET-C corruptions for the models trained on clean and obfuscated images from fig. 12. We see that the gains from training on obfuscations become bigger for the higher corruption strength (each IMAGENET-C corruption has 5 different strength levels) but that this seems to be limited to a few of the corruptions, mainly the ones that apply different kinds of noise to the image. This might be because some of our obfuscations force the models to classify images with reduced information similar to adding noise. That these strong performance gains are limited to the simpler corruptions further shows that our obfuscations are very different from the IMAGENET-C corruptions. Similar to what we saw in fig. C.13 it might also be the case that the generalization we can see is not reversible, i.e. that models that are trained with data augmented by the noise corruptions will not become significantly more robust to our obfuscations.
C.7 Top-K Accuracy
While the obfuscations drastically reduce the accuracy of the models it might be possible to counteract this by being more permissible in assigning a class label to an image if it is in the top- $k$  predicted (super-)classes. We show results for top-1 (same as fig. 4), top-3 and top-5 accuracy for the best model out of each pretrained category in fig. C.20. This leads to a significant increase in accuracy, up to  $87.6\%$  worst case accuracy for the CLIP ViT L14 model. This is still a significant reduction from the almost perfect accuracy that the models now achieve on clean images. Additionally using top-5 accuracy for 16 super-classes will lead to a significant increase of false positives if we e.g. assume these classes were different types of policy violations. Therefore using top-k accuracy for  $k > 1$  will only be a viable defense in cases where over-triggering of the classifier is not a concern. An interesting observation is that the ordering of the models changes in some cases, e.g. the CLIP ViT L14 model is the strongest model for top-3 and top-5 accuracy, while the ViT-B8 is stronger for top-1.
C.8 Results Without Class Reweighting
As described in section 3.3 we weight the accuracy per class by the inverse number of image per class as to not have the biggest classes dominate the overall accuracy. In fig. C.21 we show the results for the best pretrained models of each category without reweighting. If you compare the accuracies with the class reweighted ones in the top of fig. C.20 you see there are clear differences, e.g. the worst-case accuracy of the best CLIP model drops from  $55.5\%$  to just  $28.8\%$ , most likely because the clip model does not have good performance on one of the biggest classes. In general the results seem more noisy and extreme without reweighting and there is no clear trend of increase or decreased performance across either obfuscations or models. We can see this as well when looking at the standard deviation of the top 1 accuracy for 5 different random seeds with and without reweighting in fig. C.22.
(a) Corruption Strength 1
(b) Corruption Strength 2
(c) Corruption Strength 3
(d) Corruption Strength 4
(e) Corruption Strength 5
Figure C.19: Comparison of top 1 accuracy when training on only clean and training on the training obfuscations for individual IMAGENET-C corruptions.
C.9 Class Confusion
To investigate the mistakes models make through obfuscations we look at class confusion matrices for the ResNet50 v2 taken from Tensorflow Hub in section 4.1. figs. C.23 to C.26 show the percentage of images of the true class that were classified as the predicted class for clean images and all training and hold-out obfuscations. While there are some confusions between semantically similar classes like Dog and Bear or Car and Van / Truck that are simply amplified by the obfuscations we can see some interesting patterns. Some obfuscations lead to general misclassifications, i.e. classifying a large percentage of images as a specific class. Examples of this are PerspectiveTransform leading to many images being classified as Clock or Texturize leading to many images being classified as Elephant. The former might come from the fact the transformation on the black background makes the original image look like a clock on a wall, while the latter most likely comes from the fact that some of the textures are grey like elephant skin.
Figure C.20: Top 1 (top), top 3 (middle) and top 5 (bottom) accuracy for the best pretrained models of each category.
Figure C.21: Top 1 accuracy for the best pretrained models from each category without class reweighting.
Figure C.22: Standard Deviation of top 1 accuracy for 5 ResNet50 models with different random seeds trained only on clean images.
Figure C.23: Class confusion matrices for the different obfuscations
Figure C.24: Class confusion matrices for the different obfuscations
Figure C.25: Class confusion matrices for the different obfuscations
Figure C.26: Class confusion matrices for the different obfuscations