SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 12: Comparison of the different prompting methods with CVP for every CIFAR-10-C corruption type. The Standard model is WideResNet18. Number in bold shows the best performance.
Severity / Method	Standard	BN	Finetune	VP	CVP
Patch	Append	fixed (3*3) w/o update	fixed (3*3) w/ update	rand. (3*3) w/o update	rand. (3*3) w/ update
S1	59.68	52.48	43.28	59.94	59.76	65.17	66.00	65.98	68.07
S2	47.88	43.18	34.72	48.26	47.94	52.73	53.42	53.41	54.73
S3	40.31	36.44	29.08	40.67	40.32	44.22	44.87	44.51	45.96
S4	32.75	29.49	24.66	32.94	32.75	36.20	36.74	36.56	37.79
S5	28.20	25.16	21.63	28.46	28.25	30.42	30.64	30.68	31.61
Avg. Acc.	41.76	37.35	30.67	42.05	41.82	45.75	46.33	46.23	47.63
Avg. Error	58.24	62.65	69.33	57.95	58.18	54.25	53.67	53.77	52.37
Avg Diff.	-	4.41	11.09	-0.29	-0.06	-3.99	-4.57	-4.46	-5.87
- More evaluation on ViT-Base model architecture. As Table 14 shows, we evaluate on 15 types of corruption under severity level 1 for CIFAR-10-C. The test accuracy of ViT-Base clean CIFAR-10 is  $96.42\%$ . Compared to the accuracy  $(\%)$  before adaptation (standard) and using VP to do the adaptation, the CVP achieves better performance by up to 2 points.
Table 13: Comparison of the different adaptation baslines with CVP for every severity on CIFAR-10-C. The Standard model is WideResNet18. Number in bold shows the best performance.
ViT-Base	VP	CVP
Gaussian Noise	47.64	49.71	50.22
Shot Noise	41.92	45.67	46.03
Impulse Noise	80.77	80.39	82.13
Glass Blur	48.49	47.77	49.12
Defocus Blur	27.91	27.37	28.90
Zoom Blur	83.71	83.23	85.63
Motion Blur	58.34	56.22	58.87
Brightness	94.40	93.10	94.59
Snow	89.37	88.23	89.41
Frost	90.13	92.19	92.32
Fog	74.75	73.60	75.63
Contrast	90.14	91.34	92.34
Pixelate	66.06	66.37	63.53
Jpeg Compression	62.53	61.55	63.53
Elastic Transform	40.68	40.52	41.52
Averaged Acc.	66.45	66.48	67.77
Table 14: Performance on ViT-Base model.
- We show the detailed results for each corruption on ImageNet-C dataset in Table 15.
- In Table 16, we further compare with TTT [59], which is a test-time method with CVP. We combine our CVP on top of TTT, where the model weight will first adapt based on the rotation loss, and then the input will adapt by convolutional visual prompt.
Standard	Finetune	BN Adapt	VP	CVP
patch	append	fixed 3*3	rand 3*3	fixed 5*5	rand 5*5
Gaussian Noise	80.00	78.85	79.43	79.44	79.99	78.49	78.16	78.47	78.75
Shot Noise	82.00	80.80	81.57	81.56	81.97	80.45	80.00	80.10	80.81
Impulse Noise	83.00	81.80	82.72	82.72	83.00	80.80	79.82	80.88	81.40
Defocus Blur	73.58	75.49	75.32	77.27	73.56	74.13	73.73	74.29	75.10
Motion Blur	90.95	79.85	92.38	80.18	77.96	89.99	89.31	89.14	88.60
Glass Blur	76.32	87.16	76.86	90.41	79.96	75.98	75.47	75.99	75.45
Zoom Blur	80.00	79.84	80.52	82.07	88.98	79.87	79.67	79.72	79.45
Snow	43.86	45.10	45.82	47.88	47.95	44.24	44.27	44.60	44.91
Frost	79.88	81.04	82.22	83.78	74.99	80.12	80.19	79.69	80.05
Fog	74.38	75.74	76.80	78.06	64.38	74.53	74.91	74.36	74.85
Brightness	78.25	79.90	81.23	84.99	88.07	78.78	78.49	78.83	78.91
Contrast	71.00	73.27	74.14	76.59	70.98	71.46	70.83	71.31	71.79
Elastic Transform	87.58	87.96	88.89	96.50	87.62	87.93	87.85	87.42	88.15
Pixelate	74.72	74.22	75.75	78.30	74.68	67.05	63.98	67.29	64.15
Jpeg Compression	77.00	75.47	74.85	81.29	76.99	74.41	73.46	74.05	74.26
mCE	76.83	77.10	77.90	80.07	76.74	75.88	75.34	75.74	75.77
Diff.	0.27	1.07	3.24	-0.09	-0.95	-1.49	-1.09	-1.06
Table 15: ImageNet-C results. Number in bold shows the lowest mCE.
WideResNet18
Avg. Error (%)
Standard	58.24
VP (patch)	57.94 (-0.3)
CVP (rand. w/ update)	52.37 (-5.87)
TTT [63]	52.92 (-5.32)
TTT + CVP	53.07 (-5.17)
- Generalize to Cutout-and-Paste samples To justify that our method can be generalized to non-structured OOD, we do more experiments on other types of OOD samples, such as the Cutout-and-Paste samples. Here, we launch the experiment on the Waterbirds dataset, which is constructed by cropping out birds from images with "water" backgrounds in the Caltech-UCSD Birds-200-2011 (CUB) dataset [61] and transferring them onto backgrounds from the Places dataset [72]. We follow the GitHub repo* and choose the "Forest" as our new background to generate the samples. The training of SSL is based on the pre-trained ResNet34 backbone model for the original CUB dataset. The original CUB (200 classes) accuracy for the backbone ResNet34 is  $75.34\%$ . We compare our CVP with self-supervised VP and demonstrate that CVP is more effective on the Cutouted-CUB dataset. The following table shows our results. Our CVP improves the result upon Standard by 1.61 points and VP by 1.3 points.
Table 16: TTT [59] result for CIFAR-10-C
Cutouted-CUB (200)	Before Adapt	VP (patch)	CVP
Accuracy (%)	62.03	62.32	63.64
contrastive loss (Avg.)	2.78	2.71	2.52
Table 17: Performance on the Cutouted-CUB
*WaterBirds Datasethtps://github.com/kohpangwei/group_DRO
7.6 Distance measurement with SWD and SSIM
As the main paper mentioned, we measure the SWD and SSID on two input distributions: source domain distribution and target domain distribution (before/after adaptation). In Table 18 and Figure 9, we show the detailed result of SWD on every corruption type in CIFAR-10-C under severity 1. On average, CVP achieves lower SWD after adaptation, which means the target distribution is closer to the source one after adaptation. The average SWD reduced by  $0.7\%$  after prompting.
SWD (scale: 102) ↓	SSIM ↑
before	after	before	after
Gaussian Noise	5.90	4.71	0.7242	0.7849
Shot Noise	6.08	4.93	0.7124	0.7676
Impulse Noise	6.23	5.26	0.7463	0.7764
Glass Blur	8.85	9.19	0.5873	0.5865
Defocus Blur	13.52	11.82	0.6031	0.6013
Zoom Blur	4.13	3.09	0.8726	0.8703
Motion Blur	7.68	5.57	0.6491	0.6459
Brightness	2.48	3.94	0.9702	0.9692
Snow	5.18	6.07	0.8258	0.8275
Frost	7.61	7.72	0.8025	0.8012
Fog	13.49	9.99	0.5840	0.5785
Contrast	15.39	11.09	0.7049	0.6997
Pixelate	3.09	4.56	0.8603	0.8669
Jpeg Compression	2.58	3.65	0.8681	0.8710
Elastic Transform	5.62	5.75	0.5272	0.5789
Avg. Mean	7.19	6.49	0.7539	0.7884
Avg. Std	4.05	2.79	0.1294	0.7260
Table 18: Results of Sliced Wasserstein Distance and Structural Similarity Index Measure on CIFAR-10-C (Severity 1).
(a) Noise Group
(c) Weather Group
Figure 9: Violin Plot of SWD for different corruption groups on CIFAR-10-C. The left figure of each subplot shows the SWD before adapting, and the right shows the SWD after adaptation
(b) Blur Group
(d) Digital Group
- Distribution changes after applying the proposed CVP
In the main paper, Figure 2, we show the distribution changes in different corruption types and severity. Here, in Table 19, we show the distribution shifts after applying our CVP by calculating the average loss. In general, the distribution moves back to the original distribution. We show the SSL average loss (before adapt/after CVP adapt) for four corruption types on severity 1,3,5 for CIFAR10-C. The average SSL loss for the original CIFAR10 is 1.26. For every corruption we show here, the average SSL loss after adaptation is lower than the loss before adaptation.
severity	s1 Before /After	s3 Before /After	s5 Before /After
Gaussian noise	1.9 / 1.6	2.5 / 2.1	3.3 / 2.6
Defocus blur	3.2 / 2.9	3.4 / 2.8	3.7 / 3.1
Snow	3.1 / 3.0	3.8 / 3.3	3.9 / 3.5
Contrast	2.7 / 2.3	2.9 / 2.4	3.6 / 3.3
Table 19: Distribution changes on different corruption types.
7.7 The Effect of Different Prompt Designs
We do analysis on different prompting methods, including original visual prompts with different norm-bound  $(\ell_2,\ell_{\infty})$ , convolutional prompts, and their combinations  $(\ell_{2} + conv.,\ell_{\infty} + conv.)$ . We show the error rate on different numbers of adapt iters for every prompting method from 0, 1, 5, to 10. To compare the results, we set up other parameters such as the epsilon  $\epsilon$  as  $8 / 255$  for  $\ell_{\infty}$ , 1 for  $\ell_{2}$ . As Figure 10 shows the error rate for different prompting methods, the convolutional prompt conv. and its combination with  $\ell_{2}$  reduce the error rate, and the former one reduces more from  $40.32\%$  to  $36.08\%$  when increasing the adapt iters. However, other prompting methods increase the error rate after prompting. To understand the risk of over-fitting for different prompting methods, Figure 4b shows the SSL loss curve v.s. performance on different prompting methods.
7.8 Analysis on CVP and low-rank prompts
In Table 20, we show the detailed results of low-rank prompt (LVP) on different severity (from 1 to 5) for CIFAR-10-C. We set up the same rank as 3 for LVP and CVP. Our results show that the CVP is more effective than LVP when reversing natural corruption. In Figure 11, we further plot the averaged contrastive loss on different rank sizes for both LVP and CVP. On every corruption type, while increasing the rank size from 3 to 31, the loss curves of LVP consistently drop, which demonstrates the LVP is much more easier to overfit the contrastive loss.
Severity / Method	Standard	LVP	CVP
s1	40.32	37.05	31.93
s2	52.12	48.83	45.27
s3	59.69	56.05	54.04
s4	67.25	63.42	62.21
s5	71.80	68.89	68.39
Avg. Error	58.24	54.85	52.37
Diff.	-3.39	-5.87
Table 20
(a)
Figure 11: In subfigure a., We show the  $\ell_2$  distance between the ground-truth additive corruptions vector  $\Delta$  and our prompt reversing vectors  $\delta$ . Subfigure b shows the averaged contrastive loss of low-rank prompt and CVP on different rank sizes. For every corruption type, the loss curves of LVP consistently drop when increasing the rank to 31 while the  $\ell_2$  distances increase, demonstrating how LVP is more prone to overfitting the contrastive loss.
7.9 t-SNE analysis on different adaptation methods.
In addition to performing analysis on single sample, we further conduct the t-SNE visualization for whole sample distribution on different baseline. For each type of corruption data, we extract the 1-dimensional logit features in the last layer of model and calculate the distance between them with respect to the predicted class labels. We compare our method with standard, MEMO, and MEMO + Ours. As Figure 12 shows, the original feature embedding shows low separability between different classes. On the other hand, our approach clearly discriminates the embedding feature, which demonstrates its robustness against distribution shifts.
(a) Motion Blur
(b) Impulse Noise
Figure 12
7.10 Saliency map analysis on different corruption types
To better understand how self-supervised visual prompts adapt to the corrupted inputs, we visualize the saliency map of different types of corruption. As Figure 13 shows, from left to right, the first row is the original, corrupted, and adapted samples; the second row shows their corresponding Grad-CAM with respect to the predicted labels. The red region in Grad-CAM shows where the model focuses on for target input. We empirically discover the heap map defocus on the target object for corrupted samples. However, after prompting, the red region of the adapted sample's heap map is re-target on the similar region as original image, which demonstrates that the self-supervised visual prompts indeed improve the input adaptation and make the model refocus back on the correct regions.
Figure 13: Grad-CAM analysis on different types of corruption.