SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yun-Yun Tsai*
yunyuntsai@cs.columbia.edu
Chengzhi Mao*
mcz@cs.columbia.edu
Junfeng Yang
junfeng@cs.columbia.edu
Vision models are often vulnerable to out-of-distribution (OOD) samples, which often need adaptation to fix them. While visual prompts offer a lightweight method of input-space adaptation for large-scale vision models, they rely on a high-dimensional additive vector and labeled data. This leads to overfitting when adapting models in a self-supervised test-time setting without labels. We introduce convolutional visual prompts (CVP) for label-free test-time adaptation for robust visual perception. The structured nature of CVP demands fewer trainable parameters, less than  $1\%$  compared to standard visual prompts, combating overfitting. Extensive experiments and analysis on a wide variety of OOD visual perception tasks show that our approach is effective, improving robustness by up to  $5.87\%$  over several large-scale models.
Deep models surpass humans when tested on in-distribution data, yet their performance plummets when encountering unforeseen out-of-distribution (OOD) data at test time, such as unexpected corruptions and shiftings [20, 22, 21, 18, 44]. This vulnerability raises serious risks when these models are deployed, especially in safety-critical applications and high-stakes tasks [49]. Prior works have studied how to improve generalization to OOD data at training time [54, 38, 56, 12, 36, 35, 39, 31], yet little work has been done to adapt the model to OOD at test time, and most of them require to modify model weights [59, 71, 55, 33].
Visual prompting emerges as an efficient and lightweight method to adapt the model at test time without modifying the model [11, 60, 1] (previously also called adversarial reprogramming). In contrast to finetuning the whole model weights, prompting can modify the original task of the model by providing context in the input space. It requires fewer OOD samples and simplifies model version management for practical applications. However, the OOD samples must be labeled, preventing these methods from combating unforeseen distribution shifts.
Recent work [40] defends against unseen attacks at test time by repairing the adversarial inputs with "reversal vectors" - high-dimensional prompt vectors directly added to inputs. It generates the prompts by minimizing a self-supervised loss, requiring no prior labeled OOD samples. Unlike adversarial attacks that damage arbitrary pixels by arbitrary amounts (within a given bound), however, structured changes known as visual distribution shifts are not effectively addressed by unstructured high dimensional vector prompts. Given that a self-supervised objective often has a shortcut and trivial solutions [13], we empirically find prompts without the right structures often improve performance minimally. (Sec. 5.2).
This paper presents Convolutional Visual Prompt (CVP), which uses the convolutional structure as an inductive bias for adapting to visual OOD samples at test time. Prior work has demonstrated
* equal contributions
Figure 1: We demonstrate the self-supervised convolutional prompt that adapts model for robust perception. Clean image has a low self-supervision loss, denoted by the distance  $\mathcal{L}_s$  on the contrastive embedding. When an image is corrupted (e.g., a stop sign in foggy weather), the self-supervised loss generally increases. We then apply the convolutional visual prompts on the corrupted samples, which instruct the model to adapt and produce a small self-supervision loss.
that convolutional operations are effective for handling structured data with local motifs [30, 64]. Inspired by this, CVPs are convolutional kernels with only a small number of tunable parameters – less than  $1\%$  of the number of tunable parameters in typical unstructured visual prompts (see Figure 1 for illustration), making CVP extremely lightweight and efficient to adapt.
Our experiments show the importance of structured inductive bias for self-supervised adaptation: compared to high-dimensional free-form visual prompts, standard low-rank structured prompts improve robustness by  $3.38\%$ , and convolutional structure in prompts is significantly better than low-rank prompts by  $2.94\%$  on CIFAR-10-C. On ImageNet-Rendition, ImageNet-Sketch, and 15 types of unforeseen corruptions for both CIFAR-10 and ImageNet at five different severity levels, CVP improves robustness by  $5.87\%$  for popular large-scale visual models, including ResNet50, WideResNet18, and the state-of-the-art vision-language model CLIP [51]. Since our method modifies the input space, it complements established test-time model weight adaptation methods (e.g., TENT [62], BN [55], and MEMO [71]) and can be generalized to multiple self-supervised objectives, such as contrastive learning [4], rotation [14], and masked autoencoder (MAE) [15].
Domain Generalization. OOD data can lead to a severe drop in performance for machine learning models [20, 18, 53, 44, 39, 42]. Domain generalization (DG) aims at adapting the model with OOD samples without knowing the target domain data during training time. Adapting the model on OOD data [74, 10, 32, 73, 71, 55, 39, 42, 62, 59] also improves robustness.
Test-time adaptation is a new paradigm for robustness against distribution shift [40, 59, 71], mostly updating the weights of deep models. BN [55, 33] updates the model using batch normalization statistics, TENT [59] adapts the model weight by minimizing the conditional entropy on every batch. TTT [59] attempts to train the model with an auxiliary self-supervision model for rotation prediction and utilize the SSL loss to adapt the model. MEMO [71] augments a single sample and adapts the model with the marginal entropy of the augmented samples. Test time transformation ensembling (TTE) [50] proposes to augment the image with a fixed set of transformations and ensembles the outputs through averaging. The only two works that do not update the models is [40, 45], which modifies the pixels of adversarial, not OOD, samples to minimize the self-supervised objective.
Visual Prompting. Prompting was proposed in the natural language processing field to provide context to adapt the model for specific tasks [2]. Leveraging this idea, visual prompts [68, 26] adapt the model with a small number of trainable parameters in input space for vision tasks [9, 43] and foundation models [51]. Others proposed to prompt the samples with adversarial perturbations to repurpose the model for target classification tasks, known as adversarial reprogramming [11, 27, 60, 66], sharing the same idea with Visual Prompting. Black-box adversarial reprogramming [60] reprograms a black-box model for downstream classification tasks with limited data. V2S [66] reprograms speech recognition model for time-series data classification tasks. Robust visual prompts [41] are tuned at training time to improve the adversarial robustness of the model under attack. However, this work has not yet been applied to domain generalization where distribution is naturally shifted.
Self-supervised learning (SSL). SSL can learn effective representations from images without annotations [7, 6, 3, 22]. Prior works have shown that representations learned from different pretext tasks
Figure 2: We show the histogram of the contrastive loss distribution on different corruption types. The blue region represents the loss distribution of original sample. The yellow, green, and red regions represent the loss distribution of corrupted samples with different severity (1, 3, and 5). Our plot shows the great shifting in SSL loss distribution between original and corrupted samples.
(jigsaw puzzles [47], rotation prediction [14], image colorization [29] and deep clustering [25], etc.) can be leveraged for several downstream tasks such as image classification [4], object detection [8] and test-time domain adaptation [58]. Another well-known branch of SSL is contrastive learning, which aims at grouping associated features for a transformation of samples and distancing from other samples with dissimilar features in the dataset [5, 16, 48]. Some of the methods [22, 57, 70] use SSL for outlier detection, which aims to learn generalizable out-of-distribution features and rejects them during the testing time. In contrast to these methods which require information from the targeted domain distribution during the training phase for adaptation, our method can adapt the model without requiring any information on unforeseen domains during the testing phase.
3 Test-Time Convolutional Visual Prompting
3.1 Learning Intrinsic Structure with Self-Supervision Task
Standard ways to improve model robustness on OOD data is through making the training robust, where the training algorithm anticipates possible corruptions and distribution shifts at inference time and trains on them [18]. However, anticipating the test-time shifting is a strong assumption which is often unrealistic in the real world. Therefore, we improve robustness at test time by dynamically adapting to unforeseen corruptions and unknown shifts.
The ideal case of adapting the model at inference time is to know the ground truth label for the target task, yet this is impossible given that the test data is not labeled. To significantly improve performance on the downstream classification tasks for unlabeled data, we must choose the right self-supervision task that shares rich information with the target classification task.
There is a large body of work on good self-supervision tasks for representation learning at training time. For example, jigsaw puzzles [47], rotation prediction [14], image colorization [29] and deep clustering [25] can be applied to several downstream tasks such as image classification [4], object detection [8] and test-time domain adaptation [58].
For visual recognition, a popular self-supervised objective is contrastive learning, which learns a representation that maps the feature of the transformations of the same image into a nearby place. This is formally defined as:
$$
\mathcal {L} _ {s} (x) = - \mathbb {E} _ {i, j} \left[ y _ {i, j} ^ {s} \log \frac {\exp \left(\cos \left(z _ {i} , z _ {j}\right)\right) / \tau}{\sum_ {k} \exp \left(\cos \left(z _ {i} , z _ {k}\right)\right) / \tau} \right], \tag {1}
$$
where  $z$  are the contrastive features of  $x$  extracted from pre-trained backbone.  $y_{i,j}^{s}$  is a 0-1 vector for indicating the positive pairs and negative pairs. If  $y_{i,j}^{s}$  is 1, the  $i$ -th feature  $z_{i}$  and  $j$ -th feature  $z_{j}$  are both from the  $x$  sample. Otherwise, they are from different  $x$ . We denote  $\cos (\cdot ,\cdot)$  the cosine similarity,  $\tau$  as temperature scaling value. We optimize the model parameters for SSL model  $\mathcal{C}$  by using the contrastive loss. The objective function for training is defined as  $\min_{\theta_C}\mathbb{E}_{(x)\sim \mathcal{X}_s}\left[\mathcal{L}_s(\cdot)\right]$ , where  $\mathcal{X}_s$  is the source domain data for training. We train only on the clean samples drawn from the non-corrupted dataset. We compare this task to other self-supervision tasks in our ablation study.
Since we need to train the self-supervised task at training time, the self-supervised task will perform high on the test data that is the same distribution as the training one. Our results find that the
performance of the self-supervised task drops largely when the distribution is shifted at test time. See Figure 2. This suggests that the information that is useful for the self-supervised task is collaterally corrupted in addition to the classification performance drop.
Prior work demonstrated significant mutual information between self-supervised tasks and perception tasks, where pretraining with the self-supervised task improves the perception tasks' performance [7, 6, 3, 22]. We propose to adapt the model to minimize the self-supervised loss at inference time, where self-supervised task is a proxy that captures a large portion of information from the perception task. In recovering the information of the self-supervised tasks, we could recover the information of perception tasks that was corrupted due to distribution shifts.
3.2 Test-time Adaptation for Vision Models
Inference time adaptation allows the model to adapt to the unique characteristics of the new distribution online. However, the key challenge is that the SSL objective we use to adapt the model is a proxy, where there is often a trivial solution that reduces the loss but adapts the model in the wrong way. Several established methods exist to adapt the vision models, including foundation models.
Finetuning (FT) is a standard way to adapt the deep model. It often optimizes all the parameters of the deep models or partially, which is often heavy and requires large space to save the copy of the model to re-initialize and update. Prior work shows that this method is effective when it is finetuned on supervised tasks, yet adapting with the self-supervised task on a few examples remains under-explored. We discuss the finetuning method using the self-supervised task.
Partial Finetuning (PFT) is another way to adapt the model at inference time by only changing the statistics of the batch-normalization layer. This method assumes that the distribution drifts in the mean and standard deviation of the test data and can be recovered through test-time adaptation. The closest existing works are BN [55], Tent [62] and MEMO [71]. Tent updates the BN statistics but needs to continue training on the same distribution. MEMO only requires a single test data point, yet the algorithm is slow due to the whole model update and heavy augmentations. Here, we adapt batch normalization through our proposed contrastive learning-based self-supervised loss.
Visual Prompts (VP) have emerged as a lightweight way to adapt pre-trained models. There are two major ways to apply visual prompts to the vision model. Let the image be  $\mathbf{x}$ , it adds a vector  $\mathbf{v}$  to the input image:  $\mathbf{x} = \mathbf{x} + \mathbf{v}$ . For low-rank visual prompts [24, 67], we use a low-rank matrix in  $v$  during optimization. Most visual prompts are studied in training setup, yet they are under-explored at inference time. [40, 45] optimize an additive visual prompt on the image to repair the adversarial perturbation and improve the model robustness.
3.3 Adapting via Convolutional Visual Prompts
Adding the right structure is an effective way to avoid trivial solution to the self-supervised objective. We now introduce the convolution visual prompts (CVP), which instruct the deep model to adapt to the test distribution through convolution. Convolution has been proved to be a successful inductive bias for visual tasks [28], which is more sample efficient [34]. Our assumption is that a large family of distribution shifts in the image data are visually structured, which can be modeled by convolution. Our prompt is simple and is defined as:
$$
\mathbf {x} = \mathbf {x} + \lambda \operatorname {c o n v} (\mathbf {x}, \mathbf {k}) \tag {2}
$$
One major advantage of the convolution prompt is that the number of parameters in the prompt is significantly lower (1%) than other conventional prompts (e.g., patch prompt and padding prompt). Compared to adapting the whole model weights, our method is light weight and is fast by exploiting the structure of visual distribution shifts. In Appendix 7.2, we show the detailed algorithm of CVP. Our lightweight adjustment allows the model to quickly update to the novel data points without too much computation and memory overhead. As vision models are continuously deployed in edge devices, this is extremely important, given the limited computational resources.
This section demonstrates the detail of experiment settings and evaluates the performance of our method CVP, compared with conventional Visual Prompts (VP) and existing test-time approaches.
More analysis are shown in Section 5 and Appendix. We do a comprehensive study on the CVP, including different prompts design, kernel v.s. structure analysis, multiple SSL tasks, sensitivity analysis on the batch size and adapt iterations, GradCam visualization, and optimization cost of CVP.
4.1 Experiment Setting
Dataset. We evaluate our method on five kinds of OOD datasets, including CIFAR-10-C [21], ImageNet-C [46], ImageNet-R [19], ImageNet-Sketch [65], and ImageNet-A [23]. The following describes the details of all datasets.
- Synthetic OOD Data. The corruption data are synthesized with different types of transformations (e.g., snow, brightness, contrast) to simulate real-world corruption. The dataset contains CIFAR-10-C and ImageNet-C. Both of them are the corrupted versions of their original dataset, including 15 corruption types and 5 severity levels. A larger severity level means more corruption is added to the data. To well evaluate our method, We generate the corruption samples with five severities based on the official GitHub code* for each 15 corruption types.
- Natural OOD Data. The ImageNet-Rendition [18] contains 30000 images collected from Flickr with specific types of ImageNet's 200 object classes. ImageNet-Sketch [65] data set consists of 50000 sketch images. The ImageNet-Adversarial [23] is a natural adversarial shifting dataset contains 7500 images collected from the natural world.
Model. The backbone model architecture is pretrained on WideResNet18 [69] and ResNet26 [17] for CIFAR-10-C, ResNet50 [17] for ImageNet-C, Rendition, and Sketch. We extract the logit features before the fully connected layer of the backbone model for training the SSL model. The SSL model is a simple MLP with the final layer outputting the one-dimensional features for the contrastive learning task. We further extend our prompt method to the foundation model CLIP [51], where we only prompt the vision encoder.
Baseline Details We compare several test-time adaptation benchmarks with CVP.
- Standard: The baseline uses the pre-trained model without adaptation. For CIFAR-10-C, the standard is trained with 50000 clean CIFAR-10 train dataset on WideResNet18 and ResNet. For ImageNet1K-C, the standard is trained with  $\sim 1.2\mathrm{M}$  clean ImageNet train dataset on ResNet50.
- Finetune (FT): We adjust the whole model weight for every upcoming batch during the inference time with the self-supervised loss. In our experiments, after one-batch fine-tuning, the model will be restored to the initial weight status and receive a new type of corrupted samples.
- Partial Finetune (PFT): The partial fine-tune adapts batches of the samples to the model by only adjusting the batch normalization layers with self-supervised loss at every inference time. Same as Finetune baseline, the model will be restored to the initial weight status after the one-batch adaptation.
- SVP [40]: The prompting method to reverse the adversarial attacks by modifying adversarial samples with  $\ell_p$ -norm perturbations, where the perturbations are also optimized via contrastive loss. We extend this method with two different prompt settings: patch and padding. For the patch setup, we directly add a full-size patch of perturbation into the input. For the padding setup, we embed a frame of the perturbation outside the input. More baseline detailed are shown in the Appendix 7.3
Design of Convolutional Visual Prompts (CVP) We prompt the input samples by adding the convolutional kernels. Our kernels can be optimized under several different settings, including 1.) fixed or random kernel initialization 2.)  $3*3$  or  $5*5$  kernel sizes. We show a detailed evaluation of all kernel setups in the experimental results. For initialization, we can either random initialize the kernel size  $k$  in a uniform distribution or initialize with fixed values. For fixed initialization, we empirically found that starting from a sharpness kernel is effective. We optimize the kernel with 1 to 5 iterations of projected gradient descent. To preserve the original structure, we combine the residual of input and convolved output with learnable parameters  $\lambda$ . We jointly optimize the convolutional kernel  $k$  and  $\lambda$  with the self-supervised loss  $\mathcal{L}_s$ . The range of  $\lambda$  is predefined and empirically set up in a fixed
*We generate all types of corruption data based on the GitHub code: https://github.com/bethgelab/imagecorruptions
Method/Model	WideResNet18 Avg. Error (%)	ResNet26 Avg. Error (%)
Standard	58.24	62.12
FT	69.33 (+11.09)	63.07 (+0.95)
PFT	62.65. (+4.41)	62.47 (+0.34)
SVP (patch)	57.94 (-0.3)	62.82 (+0.7)
SVP (padding)	58.20 (-0.04)	62.18 (+0.06)
CVP-F3	53.67 (-4.57)	59.52 (-2.6)
CVP-R3	52.37 (-5.87)	59.32 (-2.80)
Table 1: Corruption benchmarks on CIFAR-10-C. We show the average error rate on 15 corruption types and 5 severities for CVP, compared with the baselines. For the CVP results, we denote the two initialization settings: fixed/random as F/R, and the number behind F/R means the kernel size (e.g., 3 or 5). We set up the batch size of adaption as 16 and the number of adapt iters as 5 for all methods.
ImageNet-C
mCE ↓	ImageNet-R
Error (%)	ImageNet-S
Error (%)	ImageNet-A
Error (%)
ResNet50	76.87	63.83	75.90	100.0
FT	77.10 (+0.26)	64.38(+0.55)	76.38 (+0.48)	99.95 (-0.05)
PFT	76.74 (-0.13)	69.63 (+5.8)	80.43 (+4.53)	99.89 (-0.11)
SVP (patch)	76.74 (-0.13)	68.86 (+5.03)	75.93 (+0.03)	99.94 (-0.06)
SVP (padding)	80.07 (+3.23)	63.84 (+0.01)	75.92 (+0.02)	99.91 (-0.01)
CVP-F3	75.88 (-0.95)	63.56 (-0.27)	75.32 (-0.58)	99.2 (-0.8)
CVP-R3	75.34 (-1.49)	63.49 (-0.34)	75.30 (-0.60)	99.2 (-0.8)
CVP-F5	75.74 (-1.09)	63.18 (-0.65)	75.35 (-0.55)	98.67 (-1.33)
CVP-R5	75.77(-1.06)	63.06 (-0.77)	75.33 (-0.57)	98.4 (-1.6)
Table 2: Comparison on CVP with other baselines on ImageNet-C, ImageNet-R, ImageNet-S, and ImageNet-A. The standard model is ResNet50 [17]. For ImageNet-C, we calculate the mean corruption error (mCE). Same as Table 1, we evaluate the CVP under multiple kernel setups and show the results. For ImageNet-C and Sketch, CVP-F3 achieves the best robustness, which reduces the error rate by  $1.49\%$  and  $0.6\%$ . Other datasets, such as ImageNet-R and ImageNet-A, outperform all the baselines when updating  $5^{*}5$  kernels with random initialization.
range. For CIFAR-10-C, we set up the range as [0.5, 3], and for ImageNet-C is set up as [0.5, 1]. We describe the detailed parameter settings and different prompt designs in the Appendix 7.4 7.7.
Table 1 presents the evaluation results of CVP on CIFAR-10-C. We compare CVP with five different baselines, including standard, VP (patch/padding), fine-tune (FT), and partially fine-tune (PFT) with SSL loss, using two model architectures. We also explore different kernel settings of CVP, including fixed/random initialization and different kernel size. Results show that for WideResNet18, CVP achieves the highest reduction in average error rate by  $5.87\%$  when updating the  $3*3$  kernel with random initialization. For ResNet-26, CVP consistently reduces the error rate by  $2.8\%$ .
Table 2 displays the results for ImageNet-C, Rendition, Sketch, and Adversarial. The standard baseline is pre-trained on ResNet50 [17]. For ImageNet-C, we report the mean corruption error (mCE), calculated with the performance rates on ResNet50 and standard AlexNet following the benchmark [21]. Due to the larger dimension of the ImageNet dataset (224), a larger kernel size effectively captures more information from the input. Thus, we set the kernel size as  $3^{*}3$  and  $5^{*}5$ . Our findings indicate that CVP achieves the highest error rate reduction for ImageNet-R and A by  $0.77\%$  and  $1.6\%$ , respectively, when updating with a  $5^{*}5$  kernel using random initialization. CVP consistently outperforms all baselines for ImageNet-C and Sketch. We observe that VP (patch) and VP (padding) degrade performance on most datasets, suggesting that unstructured perturbations with more trainable parameters are ineffective in adapting to natural OOD data with general shifts.
In Table 3, we further evaluate the performance of CVP on the large-scale foundation model, CLIP [52]. Compared to the baselines VP (padding) and VP (patch), CVP achieves the best performance on all datasets when updating under random initialization with  $5^{*}5$  kernel size.