Convolutional Visual Prompt for Robust Visual Perception
Authors: Yun-Yun Tsai
Chengzhi Mao
Junfeng Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: self-supervised learning
representation learning
visual prompts
domain generalization
input adaptation

Abstract: Vision models are often vulnerable to out-of-distribution (OOD) samples without adapting. While visual prompts offer a lightweight method of input-space adaptation for large-scale vision models, they rely on a high-dimensional additive vector and labeled data. This leads to overfitting when adapting models in a self-supervised test-time setting without labels. We introduce convolutional visual prompts (CVP) for label-free test-time adaptation for robust visual perception. The structured nature of CVP demands fewer trainable parameters, less than 1\% compared to standard visual prompts, combating overfitting. Extensive experiments and analysis on a wide variety of OOD visual perception tasks show that our approach is effective, improving robustness by up to 5.87\% over several large-scale models.

Convolutional Visual Prompt for Robust Visual Perception
Yun- Yun Tsai* Columbia University yunyuntsai@cs. www.woll..lc..oooooo edu
Chengzhi Mao"
Junfeng Yang
Columbia University mcz0es. columbia. edu
Columbia University
j unf eng0cs col columbia.com edu
Abstract
Vision models are often vulnerable to out-of-distribution (OOD) samples, which often need adaptation to fix them. While visual prompts offer a lightweight method of input-space adaptation for large-scale vision models, they rely high-dimensional additive vector and labeled data. This leads to overfitting when adapting models in self-supervised test-time setting without labels. We introduce convolutional visual prompts (CVP) for label-free test-time adaptation for robust visual perception. The structured nature of CVP demands fewer trainable param- eters, less than 1% compared to standard visual prompts, combating overfitting. Extensive experiments and analysis on wide variety of OOD visual perception tasks show that our approach effective, improving robustness by up to 5.87% several large-scale models.
Introduction
Deep models surpass humans when tested on in-distribution data, yet their performance plummets when encountering unforeseen out-of-distribution (OOD) data at test time, such as unexpected corruptions and shiftings [20 22 [4]. This vulnerability raises serious risks when these models
are deployed, especially
safety-critica applications and high-stakes tasks (49) Prior works have
studied how to improve generalization to OOD data at training time [54] 38 56 36
35 39 [3],
yet little work has been done to adapt the model to OOD at test time, and most of them require to
modify model weights (59 55 33).
Visual prompting emerges as an efficient and lightweight method to adapt the model at test time without modifying the model [1] 60 I (previously also called adversarial reprogranming)) In contrast finetuning the whole model weights, prompting can modify the original task of the model by providing context in the input space. It requires fewer OOD samples and simplifies model version management for practical applications. However, the OOD samples must be labeled, preventing these methods from combating unforeseen distribution shifts.
Recent work [40] defends against unseen attacks at time by repairing the adversarial inputs with "reversal vectors" high- high-dimensional prompt vectors directly added to inputs. It generates the prompts by minimizing a self-supervised loss, requiring no prior labeled OOD samples. Unlike adversarial attacks that damage arbitrary pixels by arbitrary amounts (within a given bound), however, structured changes known as visual distribution shifts are not effectively addressed by unstructured high dimensional vector prompts. Given that self-supervised objective often has shortcut and trivial solutions [33, we empirically find prompts without the right structures often improve performance minimally. (Sec.5)) 2).
This paper presents Convolutional Visual Prompt (CVP), which uses the convolutional structure as an inductive bias for adapting to visual OOD samples at test time. Prior work has demonstrated
equal contributions
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Clean Image
Self- Supe--eereiedd Convolutional Prompt
Figure 1: We demonstrate the self-supervised convolutional prompt that adapts model for ro- bust perception. Clean image has a low self- supervision loss, denoted by the distance Ls on the contrastive embedding. When an image is corrupted (e.g, stop sign foggy weather), the self-supervised loss generally increases. We then apply the convolutional visual prompts on the corrupted samples, which instruct the model to adapt and produce small self-supervision loss.
STOP
STO
STO
Contrastive Embeddingg
Corrupted Image
Corrupted Image
that convolutional operations are effective for handling structured data with local motifs 100 64
Inspired by this, CVPs are convolutional kernels with only small number of tunable parameters less than 1 % of the number of tunable parameters in typical unstructured visual prompts (see Figure for illustration), making CVP extremely lightweight and efficient to adapt.
Our experiments show the importance of structured inductive bias for self-supervised adaptation: compared to nigh-dimensional free-form visual prompts, standard low-rank structured prompts improve robustness by 3.38%, and convolutional structure in prompts is significantly better than low-rank prompts by 2.94% on CIFAR- 10-C. On SraggeeeetReedittie types of unforeseen corruptions for both CIFAR-10 10 and ImageNet at five different severity levels, CVP improves robustness by 5.87% for popular large-scale visual models, including ResNet50, WideResNet1 8, and the state-of-the-as vision- lasionnlanguage model CLIP (51). Since our method mod- ifies the imput space, it complements established test-time model weight adaptation methods (e.g.. TENT 662. BN (55), and MEMO (7l)) and can be generalized to multiple self-supervised objectives, such contrastive learning rotation [4] and masked autoencoder (MAE) (15
SaingeNet-Sketehl and 15
2 Related Work
Domain Generalization. models [20
OOD data can lead to severe drop in performance for machine learning the model with OOD
18
53
44
39
42) Domain generalization (DG) aims at adapting
samples without knowing the target domain data during training time. Adapting the model on OOD)
data 74 32
55
39
62 [9] also improves robustness.
Test-time adaptation is
paradigm for robustness against distribution shift 40
59
71,, mostly
updating the weights deep models. BN [55] 33 updates the model using batch normalization statistics, TENT [59] adapts the model weight by minimizing the conditional entropy on every batch. TTT [59] attempts to train the model with an auxiliary seff-superviiion model for rotation prediction and utilize the SSL loss to adapt the model. MEMO [71] augments single sample and adapts the model with the marginal entropy the augmented samples. Test time transformation ensembling
(TTE) [50] proposes augment the image with fixed set of transformations and ensembles the outputs through averaging. The only two works that do not update the models is 40 45), which modifies the pixels of adversarial, not OOD, samples minimize the self-supervised objective.
Visual Prompting. Prompting was proposed in the natural language processing field to provide con- text to adapt the model for specific tasks 22. Leveraging this idea, visual prompts [68]
26) adapt the
model with small number of trainable parameters in input space for vision tasks 9 4 and founda-
tion models [5T]. Others proposed to prompt the samples with adversarial perturbations to repurpose
the model for target classification tasks, known adversarial reprogramming [1I 27
60 66).
sharing the same idea with Visual Prompting. Black-box adversarial reprogramming (60 reprograms
black-box model for downstream classification tasks with limited data. V2S (66) reprograms speech recognition model for time-series data classification tasks. Robust visual prompts [4] are tuned at training time improve the adversarial robustness of the model under attack. However, this work has not yet been applied to domain generalization where distribution is naturally shifted.
Self-supervised learning (SSL). SSL can learn effective representations from images without anno- tations 7 6 3 22). Prior works have shown that representations learned from different pretext tasks

---

Gaussian Noise
Defecus Blur
Snow
Contrest
Origial
Corupeed Severity
102
202
Carupted Seserity)
Gar upteel Sameriggs
Number
63
Contrastiva Loss
Contractive Loss
Contrastive Loss
2
Loss
We show the histogram of the contrastive loss distribution on different corruption types. The blue region represents the loss distribution original sample. The yellow, green, and red regions represent the loss distribution of corrupted samples with different severity (1, 3, and 5). Our plot shows the great shifting in SSL loss distribution between original and corrupted samples.
Figure 2:
(jigsaw puzzles 47 rotation prediction 14 image colorization 29 and deep clustering 25 etc.)
can be leveraged for several downstream tasks such image classification 4 object detection g
and test-time domain adaptation [58]. Another well-known branch of SSL is contrastive learning,
which aims at grouping associated features for transformation of samples and distancing from other samples with dissimilar features in the dataset [6 48) Some of the methods [22] [57 70) use SSL for outlier detection, which aims learn generalizable out-of-distribution features and rejects them during the testing time. In contrast to these methods which require information from the targeted domain distribution during the training phase for adaptation, our method can adapt the model without requiring any information on unforeseen domains during the testing phase.
Tist-Time Convolutional Visual Prompting
3.1 Learning Intrinsic Structure with Self-Supervision Task
Standard ways to improve model robustness on OOD data is through making the training robust, where the training algorithm anticipates possible corruptions and distribution shifts at inference time and trains on them [18). However, anticipating the test-time shifting is a strong assumption which often unrealistic in the real world. Therefore, improve robustness test time by dynamically adapting to unforeseen corruptions and unknown shifts.
The ideal case of adapting the model at inference time is to know the ground truth label for the target task, yet this is impossible given that the test data is not labeled. To significantly improve performance the downstream classification tasks for unlabeled data, we must choose the right self-supervision task that shares rich information with the target classification task.
There is large body of work on good self suservueevviioon tasks for representation learning at training time. For example, jigsaw puzzles [47) rotation prediction [14), image colorization (29 and deep
clustering [25] can be applied to several downstream tasks such as image classification 4 object
detection and test-time domain adaptation (58).
For visual recognition, a popular self-supervised objective contrastive learning, which learns a representation that maps the feature of the transformations of the same image into nearby place. This is formally defined as:
(x( = -- vi.s log exp(cos(2z, 2/))
Σ. exp(coss22 (2,, =x))
where z are the contrastive features of x extracted from are-trained backbone. v(,; is 0-1 vector for indicating the positive pairs and negative pairs. If vi., is the i-th feature and j-th feature zy are both from the sample. Otherwise, they from different We denote cos(-.) -) the cosine similarity, 7 as temperature scaling value. We optimize the model parameters for SSL model by using the contrastive loss. The objective function for training is defined as ming; E(2)~x, [C)(.]. where X', is the source domain data for training. We train only on the clean samples drawn from the non-corrupted dataset. We compare this task to other self-supervision tasks in ablation study.
Since we need to train the self-supervised task at training time, the self-supervised task will perform high on the test data that is the same distribution the training one. Our results find that the

---

performance of the self- sepeerieeee task drops largely when the distribution is shifted at test time. See Figure This suggests that the information that useful for the self-supervised task is collaterally corrupted in addition to the classification performance drop.
Prior work demonstrated significant mutual information between self-supervised tasks and perception lasks. where pretraining with the self-supervised task improves the perception tasks' performance [7] We propose to adapt the model to minimize the self-supervised loss inference time, where self-supervised task is proxy that captures a large portion of information from the perception task. In recovering the information of the seff-supervieed tasks, we could recover the information of perception tasks that was corrupted due to distribution shifts.
6 3 [2].
3.2 Test-time Adaptation for Vision Models
Inference time adaptation allows the model to adapt the unique characteristics of the new distri- bution online. However, the key challenge that the SSL objective we use to adapt the model is a proxy, where there is often trivial solution that reduces the loss but adapts the model in the wrong way. Several established methods exist to adapt the vision models, including foundation models.
Finetuning (FT) is standard way to adapt the deep model. It often optimizes all the parameters of the deep models or partially, which often heavy and requires large space to save the copy of the model to re-initialize and update. Prior work shows that this method is effective when it is finetuned on supervised tasks, yet adapting with the self-supervised task on few examples remains under-explored. We discuss the finetuning method using the self-supervised task.
Partial Finetuning (PFT) is another way to adapt the model inference time by only changing the statistics of the normalaommaiiitiion layer. This method assumes that the distribution drifts in the mean and standard deviation of the test data and can be recovered through test-time adaptation. The closest existing works are BN [55], Tent [62] and MEMO (71). Tent updates the BN statistics but needs continue training on the same distribution. MEMO only requires single test data point, yet the algorithm is slow due to the whole model update and heavy augmentations. Here, we adapt batch normalization through our proposed contrastive cerning-basse self-super loss.
Visual Prompts (VP) have emerged as lightweight way to adapt pre-trained models. There are two major ways to apply visual prompts the vision model. Let the image be x. it adds vector v to the input image: x = x + v. For low-rank visual prompts (24] 67], we use a low-rank matrix in during optimization. Most visual prompts are studied in training setup, yet they are under-explored at optimize an additive visual prompt on the image to repair the adversarial perturbation and improve the model robustness.
inference time. 40 45
3.3 Adapting via Convolutional Visual Prompts
Adding the right structure is an effective way to avoid trivial solution to the self-supervised objective. We now introduce the convolution visual prompts (CVP), which instruct the deep model to adapt to the test distribution through convolution. Convolution has been proved to be a successful inductive bias for visual tasks (28), which is more sample efficient (34). Our assumption is that large family of distribution shifts in the image data are visually structured, which can be modeled by convolution. Our prompt simple and is defined as:
= Aconv(x, k)
(2)
One major advantage of the convolution prompt is that the number of parameters in the prompt is significantly lower (1%) than other conventional prompts (e.g., patch prompt and padding prompt). Compared to adapting the whole model weights, method is light weight and is fast by exploiting the structure of visual distribution shifts In Appendix 7.22 we show the detailed algorithm of CVP. Our lightweight adjustment allows the model to quickly update to the novel data points without too much computation and memory overhead. As vision models are continuously deployed in edge devices, this is extremely important, given the limited computational resources.
4 Experiment
This section demonstrates the detail of experiment settings and evaluates the performance of our method CVP, compared with conventional Visual Prompts (VP) and existing test-time approaches.

---

More analysis are shown in Section 5 and Appendix. We do comprehensive study on the CVP, including different prompts design, kernel v.s. structure analysis, multiple SSL tasks, sensitivity analysis on the batch size and adapt iterations, GradCam visualization, and optimization cost of CVP.
4.1 Experiment Setting
Dataset. We evaluate our method on five kinds of OOD datasets, including CIFAR-IO-C 21 ImageNet-C [46], ImageNet-R [19], ImageNet-Sketch [65] and ImageNet-A (23 The following describes the details of all datasets.
Synthetic OOD Data. The corruption data are synthesized with different types of transformations (e.g., snow, brightness, contrast) to simulate real-world corruption. The dataset contains CIFAR-IO-C and ImageNet-C. Both of them are the corrupted versions of their original dataset, including 15 corruption types and severity levels. A larger severity level means more corruption is added the data. To well evaluate our method, We generate the corruption samples with five severities based on the official GitHub code for each 15 corruption types.
Natural OOD Data. The ImageNet-RRndiing Renditton [18] contains 30000 images collected from Flickr with specific types of ImageNet's 200 object classes. ImageNet-Sketee [65] data set consists of 50000 sketch images. The ImageNet-Adversane images collected from the natural world.
[23] is a natural adversarial shifting dataset contains 7500
Model. The backbone model architecture is pretrained on WideResNet18 [69 and ResNet26 [7 for CIFAR-10-C, ResNet50 [77 for ImageNet-C, Rendition, and Sketch. We extract the logit features before the fully connected layer the backbone model for training the SSL model. The SSL model is simple MLP with the final layer outputting the one-dimensional features for the contrastive learning task. We further extend our prompt method to the foundation model CLIP [51), where we only prompt the vision encoder.
Baseline Details We compare several test-time adaptation benchmarks with CVP.
Standard: The baseline uses the pre-trained model without adaptation. For CIFAR-10-4 the standard is trained with 50000 clean CIFAR-10 train dataset on WideResNet18 8 and ResNet. For ImageNetlK-Ce the standard is trained with ~ 1.2M clean ImageNet train dataset on ResNet50.
Finetune (FT): We adjust the whole model weight for every upcoming batch during the inference time with the self-supervised loss. In our experiments, after one-batch fine-tuning, the model will be restored to the initial weight status and receive new type of corrupted samples.
Partial Finetune (PFT): The partial fine-tune adapts batches of the samples to the model by only adjusting the batch normalization layers with self-supervised loss at every inference time. Same as Finetune baseline, the model will restored to the initial weight status after the one-batch adaptation.
The prompting method to reverse the adversarial attacks by modifying adversarial samples with fy-norm perturbations, where the perturbations are also optimized via contrastive loss. We extend this method with two different prompt settings: patch and padding. For the patch setup, we directly add a full-size patch of perturbation into the input. For the padding setup, we embed frame of the perturbation outside the input. More bascline detailed are shown in the Appendix 7.3
SVP [40]:
Design of Convolutional Visual Prompts (CVP) We prompt the input samples by adding the convolutional kernels. Our kernels can be optimized under several different settings, including 1.) fixed or random kernel initialization 2.) 3*3 or 5%5 kernel sizes. We show detailed evaluation of all kernel setups in the experimental results. For initialization, we can either random initialize the kernel size k in uniform distribution or initialize with fixed values. For fixed initialization, found that starting from a sharpness kernel is effective. We optimize the kernel with to 5 iterations of projected gradient descent. To preserve the original structure, combine the residual of input and convolved output with learnable parameters X. We jointly optimize the convolutional kernel k and À with the self-supervised loss L.. The range of À predefined and empirically set up in fixed
empirically
We generate all types of corruption data based on the GitHub code: https://githut/ www.:/uww...oommmeeeeeebbaa.b
imagecorruptions
5

---

WedeResNet18
ResNet26
Table I: Corruption benchmarks on CIFAR-10- C. We show the average error rate on 15 cor- ruption types and 5 severities for CVP, com- pared with the baselines. For the CVP re- sults, we denote the two initialization settings: fixed/random as F/ R, and the number behind F R means the kernel size (e.g., 3 or 5). We set up the batch size of adaption as 16 and the number of adapt iters as for all methods.
Method/Model
Avg. Error (%)
Avg. Error (%)
Standard
58.24
62.12
FT
69.33 (+11.09)
63.07 (+0.95)
PFT
62.65. (+4,41)
62.47 (+0.34)
SVP (patch)
57.94 (-0.3)
62.82 (+0.7)
SVP (padding)
58.20 (-0.04)
62.18 (+0.06)
CVP-F3
53.67 (-4.57)
59.52 (-2.6)
CVP-R3
52.37 (-5.87)
59.32 (-2.80)
ImageNet-C
ImageNet-R Error (%)
ImageNet-S Error (%)
ImageNet-A Error (%)
mCE +.
ResNet50
76.87
63.83
75,90
100.0
FT
77.10 (+0.26)
64,38(+0,55)
76.38 (+0.48)
99.95 (-0.05)
PFT
76.74 (-0.13)
69.63 (+5.8)
80.43 (+4,53)
99.89 (-0.11)
SVP (patch)
76.74 (-0.13)
68.86 (+5.03)
75.93 (+0.03)
99.94 (-0.06)
SVP (padding)
80.07 (+3.23)
63.84 (+0.01)
75.92 (+0.02)
99.91 (-0.01)
CVP-F3
75.88 (-0.95)
63..6(--.27)
75.32 (-0.58)
99.2 (-0.8)
CVP-R3
75.34 (-1.49)
63,49 (-0.34)
75,30 (-0.60)
99.2 (-0.8)
CVP-F5
75.74 (-1.09)
63.18 (-0.65)
75.35 (-0.55)
98.67 (-1.33)
CVP-R5
75.77(-1.06)
63.06 (-0.77)
75.33 (-0.57)
98.4 (-1.6)
Table 2: Comparison on CVP with other baselines on ImageNet-C. ImageNet-R, ImageNet-S, and ImageNet-A. The standard model ResNet50 [17]. For ImageNet-C, we calculate the mean corruption error (mCE). Same as Table we evaluate the CVP under multiple kernel setups and show the results. For ImageNet-C and Sketch, CVP-F3 achieves the best robustness, which reduces the error rate by 1.49% and 0.6%. Other datasets, such as ImageNet-R and ImageNet-A, outperform all the baselines when updating 5*5 kernels with random initialization.
range. For 10-C, we set up the range as [0.5, 31, and for ImageNet-C set up as [0.5, 1]. We describe the detailed parameter settings and different prompt designs in the Appendix 7.477
4.2 Experimental Results
Table 1 presents the evaluation results of CVP on CIFAR-10-C. We compare CVP with five different baselines, including standard, VP (patch/padding), fine-tune (FT), and partially fine-tun tne-tune (PFT) with SSL loss, using two model architectures. We also explore different kernel settings of CVP, including fixed/random initialization and different kernel size. Results show that for RedeResNet18,, CVP achieves the highest reduction in average error rate by 5.87% when updating the 3*3 kernel with random initialization. For ResNet-26, CVP consistently reduces the error rate by 2.8%.
Table 2 displays the results for ImageNet-C, Rendition, Sketch, and Adversarial. The standard baseline is pre-trained ResNet50 [17]. For ImageNet-C, report the mean corruption error (mCE), calculated with the performance rates on ResNet50 and standard AlexNet following the benchmark (21). Due the larger dimension of the ImageNet dataset (224), a larger kernel size effectively captures more information from the input. Thus, we set the kernel size as 3*3 and 5*5. Our findings indicate that CVP achieves the highest error rate reduction for ImageNet-R and A by 0.77% and 1.6%, respectively, when updating with a 5 kernel using random initialization. CVP consistently outperforms all baselines for ImageNet-C and Sketch. We observe that VP (patch) and VP (padding) degrade performance on most datasets, suggesting that unstructured perturbations with more trainable parameters are ineffective in adapting natural OOD data with general shifts.
In Table 3 we further evaluate the performance of CVP on the large-scale foundation model, CLIP [52.. Compared to the baselines VP (padding) and VP (patch), CVP achieves the best perfor- mance on all datasets when updating under random initialization with 5*5 kernel size.
CVP Complements Other Test-Time Adaptation Methods Since our method modifies the input space and aligns the representation of adapted samples to the original manifold, it also complements established test-time adaptation approaches which adjusts the model weights. Thus, we combine the CVP with several existing methods, including MEMO 711, BN (55), and TENT (2). For fair comparison, we set up the batch size as 16 for all experiments. for the parameter settings, we set the kernel size for CIFAR-10-C as 3x3 and ImageNet-C, R, S, and A as 5x5, The adaptation iterations

---

CIFAR-IO-C
ImageNet-C
ImageNet-R Error (%)
ImageNet-S
Avg. Error (9))
mCE 4
Error (%)
CLIP(VIT/32)
58.39
77.93
32.09
60.55
SVP (patch)
58.43 (+0.04)
77.81 (-0.12)
32.12 (+0.03)
60.53 (-0.02)
CVP-F3
57.94 (-0.45)
77.43 (-0.50)
31.163))
59.47 (-1.08)
CVP-R3
57.91 (-0.48)
77.71 (-0.22)
31.31 (-0.78)
59.62 (-0.93)
CVP-F5
57.98(-0.41)
77.25 (-0.68)
31.14 (-0.95)
59.83 (-0.72)
CVP-R5
57.79 (-0.60)
76.67 (-1.26)
30.43 (-1.66)
59.43 (-1.12)
Table 3: Evaluation on the CLIP model. We compare CVP with other prompting baselines on CIFAR-IO-C, ImageNet-C, ImageNet-R, and ImageNet-S, Overall, the CVP-R5* achieves the best performance, which reduces the error rate by 1.16% on average.
CIFAR-IO-C
ImageNet-C mCE |
ImageNet-R Error (%)
ImageNet-S Error (%)
ImageNet-A
Avg. Error (%)
Error (%)
Standard
58.24
76.87
63.83
75.90
100.0
Mao et al. 400
57.94 (-0.3)
76.74 (-0.13)
68.86 (+5.03)
75.93 (+0.03)
99.94 (-0.06)
CVP (ours)
52.37 (-5.87)
75.43 (-1.49)
63.06 (-0.77)
75.30 (-0.6)
98.4 (-1.6)
MEMO 7
56.14
73.45
60.73
73.43
99.1
MEMO CVP
54.84 (-1.3)
72.02 (-1.33)
60.23 (-0.5)
72.67 (-0.76)
98.64 (-0.46)
BN [55
38.51
76,20
67.29
77.98
99.8
BN + CVP
37.39 (-1.12)
76.16 (-0.04)
67.21 (-0.08)
77.92 (-0.06)
98.67 (-1.13)
TENT [62]
38.52
70.45
58,45
73.88
99.7
TENT + CVP
36.69 (-1.83)
70.34 (-0.11)
58.42 (-0.03)
73.83 (-0.05)
98.54 (-1.16)
Table 4: Our prompt method complements other test-time adaptation approaches that update model weight, including MEMO, TENT, and BN. We show complements gain on every baseline when combined with CVP. Here, the Standard for CIFAR-10-C WideResNet18 and other dataset is ResNet50. For CIFAR-1O-C, on top of the TENT 62 method, we achieve the best gain on the performance, which reduces 1.83% error rate.
are all set as 5. In Table 4 we show the results on five benchmarks. For CIFAR-IO-C, CVP improves 1.83 points on top of TENT and reduces the eтrоr rate by 21.55% compared with the standard one. For the other datasets, CVP achieves the lowest еттоr rate on top of the TENT method. However, the BN method degrades the performance under the small batch setting. Due to the page limit, in Appendix 7.5 we show more evaluations on other benchmark, such as the Cutout-and-Pant data [6] and other baseline Test Time Training (TTT) 59
5
Ablation Study
Low-Rank Structure to Avoid Shorteut in SSL Objective. Since SSL objective is proxy for visual recognition task, sole minimizing SSL objective can produce a overfitted prompt without much improvement visual recognition. A typical way avoid this suboptimal solution is adding the right inductive bias. For the highly structured visual data, we study the best inductive bias we can add to prevent the shorteut during adaptation.
In addition to convolution, we also investigated the popular low-rank structure 24 67 We create visual prompts with low-rank structure (LVP) via singular value decomposition (SVD). and optimize the low-rank matrices using our algorithm (shown Appendix 7.2). We compare the effectiveness of low-dimensional prompts and convolution their ability to reverse natural corruptions. In Table 6 and Figure 3] for both LVP and CVP, increasing the prompt's 's rank leads to degraded performance in reverse the corruption to produce clean images, where the l norm distance between real shifting and the approximated shifting increase.
CIFAR-IO-C
ImageNet-C mCE
ImageNet-R Error
ImageNct-S
Table 5: Analysis on the shorteut of SSL. Error
Avg. Error
Standard
58.24
76.87
63.83
75.90
We compare the per- formance of SSL and supervised learning on
SSL-VP
57.95 (-0.29)
76.74 (-0.13)
68.86 (+5.03)
75.93 (+0.03)
SSL-CVP
52.37 (-5.87)
75.34 (-1.53)
63.06 (-0.77)
75.30 (-0.60)
Sup-VP VP
42.02 (-16.22)
73.54 (-3.33)
60.17 (-3.66)
73.30 (-2.60)
VP and CVP for four benchmarks
Sup-CVP
49.59 (-8.65)
74.10 (-2.77)
62.60 (-1.23)
74.14(-1.76)

---

2.90
3.60
3.30
2.80
3.69
+-
Cony.
3.20
3.30
2.70
Low Rank
2.50
3.10
3.67
3.20
3.00
3.67
31
19 Rank Value
31
19
15
Bank Volue
Rank
Rank Value
gaussian noise
defocus blur
The effect of structured prompt for reversing corruption. For the y-axis, we show the & distance between the ground-truth additive corruptions vector A and our prompt reversing vectors where a small distance indicates our reversal is successful. For the x-axis, we show the rank for the prompt matrices and the convolutional kernel. For convolutional kernel with size k k, its rank is k. We find a small rank help reverse the noise better. For the same rank, convolution is better than the additive matrix. This demonstrates the effectiveness of our convolution prompt over other low-rank structures in reversing natural corruptions.
Figure 3:
100
Bratura
(curs)
tours) (bounded)
Accuracy (96)
VP (porcht
CVP Jound
55
V
VP (bounded)
Contrastive ve Loss
3.0
60
Accuracy 45
50
2.0
20
10
20
30
20 Number of Iteration
10
30
Batch Size
Number of iteration
(b)
Figure 4: (a.) Analysis of the effect of batch size for different baselines on Cifar- 10-C.. We show the average performance on severity for every corruption type. When batch size is small, CVP has better performance on all corruption types, compared to fine-tune and two VP methods. (b.) We show the analysis for self-supervised loss v.s. performance on VP and CVP for gaussian corruption type in CIFAR-10-C, We setup the norm-bound for VP as 8/255 unbounded. The loss curve demonstrates that when increasing the iter. number from to 30, adapting with VP without norm bound hugely over-fits the supervupee loss. The loss of VP largely reduces, yet the performance does not improve. On the other hand, our CVP has a lower risk of over- wett-fitting as the loss curve can reduce smoothly with accuracy gradually increasing.
CVP in Supervised Learning. CVP is necessary for our test-time adaptation because SSL is proxy for our final task. There are solutions where SSL is minimized, but the final task performs suboptimal. We investigate the necessity of using CVP when optimized for objectives other than SSI., such as the supervised loss, where the training and evaluation objective matches. Hence, in 3 we show the VP and CVP under the supervised learning setting (Sup-VP and Sup-CVP) and SSL setting (SSL-VP and SSL-CVP). For all experiments, we use the same optimization setting and parameter settings Table We apply the prompts every batch for the supervised setting and update them with ground truth. Due to training on the ground-truth label, both VP and CVP achieve higher accuracy than SSL, VP performs higher than CVP. The results demonstrate that convolution inductive bias is not necessary when ground truth available. However, without the label, the structured convolutional prompt is necessary for improving performance.
Generalize CVP to other SSL tasks. We further show the CVP can generalized to other self- supervision tasks. In Table 7 we compare three SSL tasks, including contrastive learning, rotation prediction and masked autoencoder (MAE) [15]. We show our CVP is effective under all SSL tasks. On ImageNet-C, CVP improves robust accuracy for every severity level (from sl to s5) when optimized for all three SSL tasks. We find contrastive learning can perform better than rotation prediction, and MAE has the best performance.
The Effect of Batch Size and Number of Iteration for Adaptation. In Figure 4a we empirically demonstrate that the batch size affects the performance of different prompt methods. We set up the batch sizes from 2,4,8, 16 to 32 and compared the accuracy of CVP with Finetune, VP (padding), and VP (patch). When batch size is set as 2, the performance of fine-tune is only 36.95%, which is worse than CVP (71.53%). When increasing the batch size to and 8, the performance of fine-tune, VP slightly improves, yet still worse than CVP. Overall, CVP has ad advantage in adaptation under the

---

CIFAR-10-C Standard 58.24
LVP_R3
S1
S2
S3
S4
S5
54.86 (-3.38)
Standard
58.82
48.25
38.65
27.15
17.57
ImageNet-C 76.87
76.42 (-0.45)
Contrast
59.53
48.84
39.82
28.55
19.53
ImageNet-R 63.83
63.57 (-0.26)
Rotation
58.97
48.30
38.88
27.50
17.99
ImageNet-S 75.90
75.69 (-0.21)
MAE
60.94
51.10
41.79
30.11
19.92
Table 6: Performance of low-rank visual Table 7: Performance of CVP on three SSL tasks for ImageNet-C, including contrastive learning, rotation
prompt. We show the average emror rate on four benchmarks. For CIFAR-1O-C, we prediction, and MAF, We compare them with the WideResnet the standard model. For Standard ResNet50 and show the averaged accuracy ImageNer-C, R, and S, we use the ResNer50. for 15 corruption types every severity,
Original
Shifted
Ours
Original
Shifted
Ours
Original
Shifted
Ours
Image
Sallency
Map
Figure Visualization. From left to right we show three kinds corruption types, including contrast, fog, and frost, ImageNet examples. By applying convolutional prompt on the corrupted images, our method can partially remove the corruptions and make the image casier to recognize. In addition, the saliency map calculated from Grad-Cam also shows that our approach instructs the model to look at a similar region as the original one. This highlights why our convolutional can adapt the input for robustness.
small batch setting. In Figure 4b we show the performance on different numbers of iterations for adaption. We empirically found that when increasing the iterations, CVP has lower risk overfit on the self-supervised objective. More ablation studies, such as different prompt designs can be found in Appendix 7.7
Visualization of Saliency Map To better understand how CVP adapts to the corrupted inputs, we visualize the saliency map of different types of corruption. As Figure Sshows, shows, from left to right, the first row is the original, corrupted, and adapted samples; the second row shows their corresponding Grad-CAM with respect to the predicted labels. The red region in Grad-CAM highlights where the model focuses on target input. We empirically discover the heap map defocuses on the target object for corrupted samples. After CVP, the red region of the adapted sample's 's heap map is re-targeted on similar part as the original image, demonstrating that the self-supervised visual prompts indeed improve the input adaptation and make the model refocus back on the correct areas. We provide more visualization in Appendix 77.00
Training Cost v.s. Different Kernel Size In Table we evaluate different kernel sizes for CVP and empirically discover that increasing the kernel to a proper size can improve the performance slightly. We choose one corruption-type impulse noise and show the results. When increasing the kernel size, the optimization cost increases. For impulse noise, kernel size 787 achieves the best robust accuracy. yet the optimization is much higher.
Kernel Size
Accuracy (%)
Training Cost/Batch
3+3
16.22
0.67s
5*5
16.3
0.68s
7+7
16.62
1.24s
13*13
16.61
1.28s
21*21
16.52
1.29s
25+25
15.4
1.32s
Table 8: Training Cost v.s. Different Kernel Size
Training Time v.s. Number of Adapt Iteration Figure 4(b), we have shown the CVP trained under different adapt iterations v.s. their performance. When increasing the number of adapt iterations, the training time increases. The following Table shows the result of CIFAR-10- 10-C on gaussian noise type with severity 1. We compare the accuracy and per batch training time on several numbers of

---

adapt iters (from 0 to 20). We empirically found that CVP has a larger performance gain than VP while adapting with few epochs (epoch number 1).
# of Adapt lters
10
15
20
Cost/Batch
0.00s
0.17s
0.67s
1.29s
1,.92%
2.57s
CVP (cc.
39.51
51.09
56.1
58.76
59.30
59.58
Table Training Time v.s. Number of Adapt Iteration
Does CVP Reverse Corrupted Images Back to Normal One? We do the quantitative measurement on the distribution distance via Sliced Wasserstein Distance (SWD) and structural similarity index measure (SSIM). We measure the distance between two input distributions: source domain distribution and target domain distribution (before/after CVP adaptation). To calculate the distance between two input distributions via the Sliced Wasserstein Distance, distributions from a high dimensional probability distribution via the linear projection, then calculate the p- p-Wasserstein Distance for those marginal distributions. Table [0 and Figure shows the result of SWD on CIFAR-IO-C with severity 1. On average, CVP achieves lower SWD after adaptation, which means the target distribution is closer the source one after adaptation. The average SWD reduce by 0.7% after prompting. In Appendix Table 18 and Figure
first obtain group of marginal
9
we show the more detailed analysis.
SWD (scale: 10")
SSIM 1
before
after
before
after
Avg. Mean
7.19
6.49
0.7539
0.7884
Avg. Std
4.05
2.79
0.1294
0.7260
Table 10: Results of SWD and SSIM on CIFAR-10-C (Severity 1).
11.9
12.3
pestaalont mode
Impuinee meise
What relse
Seitalaot mojs
Imsstng noise
shst roise
moem May
making Blut
Wooae Bkur
menorn
plass
delecus
plas
delania
(a) Noise Group
(b) Blur Group
Figure 6: Violin Plot of SWD for different corruption groups CIFAR- (severity 1). The left figure of each subplot shows the SWD before adapting, and the right shows the SWD after CVP adaptation. CVP brings the corrupted image distribution back to the clean image distribution.
Conclusion
Self-supervised convolutional visual prompt (CVP) is novel method for test-time adaptation of OOD samples. In contrast to prior works training the visual prompts with the label, CVP is label- free and lightweight. It reduces the trainable parameters to less than 1% parameters of previous visual prompts and avoids the risk of overfitting when adapting for self-supervised objectives at test time. Results on five state-of-the tethhe-rr art benchmarks show that CVP improves model robustness by 5.87% and complements existing weight-adaptation methods. Extensive ablation studies suggest that distribution shifts are actually structured; therefore, CVP can capture the structures better than VP during the adaptation, which provides new insight into the frontier of visual prompting techniques and test-time adaptation. Future work includes interpreting convolutional prompts and prompting with multi-modality in large-scale foundation models.
Acknowledgement
This research supported in part by a GE/DARPA grant, a CAIT grant, and gifts from Google and Accenture. We thank suggestions from Yi Zhang, Yow-Kuan Lin, and Raphael Jedidiah Sofaer.
10

---

References
[1] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Madifying pixel space to adapt pre-trained models, arXiv preprint arXiv:2203. 2222203.777.,, 17274, 2022.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are fow-shot leamerrs. Advances in neural information processing systems, 33:1877-1901, 2020.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, vol- ume 33, pages 9912-9924. Curran Associates, Inc., 2020.
[4] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[5] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020.
[6] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 20t.:20000004297., 2020.
[7] Virginia R de Sa. Learning classification with unlabeled data. Advances in neural information processing systems. pages 112-112, 1994.
[8] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pages 1422-1430. 2015.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 119.99 2020.
[10] Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, Ben Glocker. Domain generalization via model-agnostic learning of semantic features. Advances in Neural Information Processing Systems, 32, 2019.
[11] Gamaleldin F Elsayed, Ian Goodfellow, and Jascha Sohi--ickstein.. Adversarial reprogramming of neural networks. arXiv preprint arXiv:1806.1 1146, 2018
[12] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pages 1 1180-1189 PMIR, 2015.
[13] Robert Geirhos, Jörn-Henria Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-67) 673, 2020.
[14] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1rXiv:1803.07723. 2018
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:2 NXXX::2/11.06377/ 2021.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie. and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9738,9738, 2020.
[17] Kaiming He,, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 770-778 2016.
[18] Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of distriibiittion generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8349, 2021.
11

---

[19] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song. Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis out-of-distribution generalization. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2021.
[20] Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint A903: /2261, 2019.
[21] Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
[22] Dan Hendrycks, Mantas Mazcika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in Neural Information Processing Systems, 32, 2019.
[23] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021.
[24] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models, arXiv preprint arXiv:2106. 09685, 2021.
[25] Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9865-9874, 2019.
[26] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.12119 2022.
[27] Eliska Kloberdanz, Jin Tian, and Wei Le. An improved (adversarial) reprogramming technique for neu- ral networks. In Igor Farkavs, Paolo Masulli, Sebastian Otte, and Stefan Wermter, editors, Artificial Neural Networks and Machine Learning ICANN 2021, pages 3 15, Cham, 2021. Springer International Publishing.
[28] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[29] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In European conference on computer vision, pages 577-593. Springer, 2016.
[30] Yann LeCun, Koray Kavukcuoglu, and Clément Farabet. Convolutional networks and applications in vision. In Proceedings of 2010 IEEE international symposium on circuits and systems, pages 253-256. IEEE, 2010.
[31] Bo Li, Yezhen Wang, Shanghang Zhang, Dongsheng Li, Kurt Keutzer, Trevor Darrell, and Han Zhao. Learning invariant representations and risks for semi-supervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 104-1113, 2021.
[32] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In Shcory-Second AAAI Conference on Artificial Intelligence, 2018.
[33] Yanghao Li, Naiyan Wang. Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation, 2016.
[34] Zhiyuan Li. Yi Zhang, and Sanjeev Arora. Why are convolutional nets more sample-efficient than fully-connected nets? arXiv preprint arXiv:2010.0851/ 2020.
[35] Ziwei Liu, Zhongqi Miao, Xingang Pan, Xiaohang Zhan, Dahua Lin, Stella X Yu, and Boqing Gong. Open compound domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12406-12415, 2020.
[36] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97 105. PMLR, 2015.
[37] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
[38] Zhihe Lu, Yongxin Yang. Xiatian Zhu, Cong Liu, Yi-Zhe Song. and Tao Xiang. Stochastic classifiers for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9111-9120, 2020.
12

---

[39] Chengzhi Mao, Augustine Cha, Amogh Gupta, Hao Wang. Junfeng Yang, and Carl Vondrick. Generative interventions for causal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3947-3956, 2021.
[40] Chengzhi Mao, Mia Chiquier. Hao Wang, Junfeng Yang. and Carl Vondrick. Adversarial attacks are reversible with natural supervision. arXiv preprint arXiv: 103. 14222, 2021.
[41] Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models, arXiv preprint arXtv:22 2022.
[42] Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, and Irfan Essa. Discrete representations strengthen vision transformer robustness. arXiv preprint arXiv:2111. 2021.
[43] Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang, Xin Wang, and Carl Vondrick. Doubly right object recognition: A why prompt for visual rationales. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2722-2732 June 2023.
[44] Chengzhi Mao, Kevin Xia, James Wang. Hao Wang, Junfeng Yang, Elias Bareinboim, and Carl Vondrick. Causal transportability for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages
2022.
[45] Chengzhi Mao, Lingyu Zhang, Abhishek Joshi, Junfeng Yang. Hao Wang, and Carl Vondrick. Robust perception through equivariance. arXiv preprint 27X.02221 2022.
[46] Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Mathhis Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint 1907:01977.07484 2019.
[47] Mehdi Noroozi and Paolo Favaro. Unsupervised learning visual representations by solving jigsaw puzzles. In European conference on computer vision, pages 69-84. Springer. 2016.
[48] Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Yan Zhu Contrastive learning for unpaired image-to-image translation. In European Conference Computer Vision, pages 319-345. Springer, 2020.
[49] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. Deepxplore: Automated whitebox testing of deep learning systems. In proceedings of the 26th Symposium on Operating Systems Principles, pages 1-18, 2017.
[50] Juan C Pérez, Motasem Alfarra, Guillaume Jeanneret, Laura Rueda, Ali Thabet, Bernard Ghanem, and Pablo Arbelácz. Enhancing adversarial robustness via test-time transformation ensembling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 81 91, 2021.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Cabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[52] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[53] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR. 2019.
[54] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference computer vision, pages 213-226. Springer, 2010.
[55] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2019.
[56] Kuniaki Saito, Kohci Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3723-3732, 2018
[57] Vikash Sehwag, Mung Chiang. and Prateek Mittal, Ssd: A unified framework for self- supervised outlier detection. arXiv preprint arXiv:2103.12051, 2021.
13

---

[58] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In Hal Daumé III and Aarti Singh, editors. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9229 9248. PMLR, 13-18 18 Jul 2020.
[59] Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts, 2019.
[60] Yun- Yun Tsai, Pin- Yu Chen, and Tsung-Y Yi Ho. Transfer learning without knowing: Reprogramming black- box machine learning models with scarce data and limited resources. In International Conference on Machine Learning, pages 9614 9624. PMLR, 2020.
[61] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[62] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization, 2020.
[63] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[64] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc. E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[65] Haohan Wang, Songwei Ge. Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, pages 105061 2019.
[66] Chao-Har Huck Yang, Yun-Yor Yun Tsai, and Pin- Yu Chen. Voice2serics: Reprogramming acoustic models for time series classification, 2021.
[67] Yuzhe Yang, Guo Zhang, Dina Katabi, and Zhi Xu. Me-net: Towards effective adversarial robustness with matrix estimation. arXiv preprint arXiv:1905. /1971, 2019.
[68] Yuan Yao, Zhang. Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun. Cpt: Colorful prompt tuning for pre-trained languaga models. arXiv preprint #xXiv:2109111997 2021.
[69] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. urXiv preprint arXiv:1605.07146. 2016.
[70] Zhiyuan Zeng. Keging He, Yuanmeng Yan, Hong Xu, Weiran Xu. Adversarial self-supervised learning for out-of-domain detection. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5631-5639, 2021. 2021.
[71] M. Zhang, S. Levine, and C. Finn. MEMO: Test time robustness via adaptation and augmentation. 2021.
[72] Bolci Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEFE Transactions on Pattern Analysis and Machine Intelligence, PP:1-1. 07 2017.
[73] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain-adversarial image generation for domain generalisation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13025 13032, 2020.
[74] Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. arXiv preprint Xiv:2104.02008, 2021.
14

---

7 Appendix
7.1 Flow of CVP
We illustrate the whole flow of CVP in Figure 7
There are two phases our proposed method: (1.) Offline one-time training SSL model: The SSL is simple MLP model, where it takes the
logit features from pre-trained backbone as input and trains the model with contrastive learning task.
(2.) CVP adaptation for any OOD benchmark during test time: We leverage the SSL loss from well-trained SSL model to optimize the convolutional kernel prompts for every upcoming OOD sample.
Testing Phase
Before Prompling
Conusded
Training Phase
Feothrn Extractor
Classttier
Predicaal Class Solder Moniey X
Frence Traiobla
Clean Image
Hobiin
Model
Chocsiaas
Cieedtration
Prompting with CVP
Comptte
OVF
Feature Extrcdoo
Cloosller
SSL Modil
Ocskessivv 1oss
Oprinteed the paomeeens
Figure 7
7.2
Algorithm
7.3 Baseline Details
Here, show the detail of the baselines that compare with.
Standard: The baseline uses pre-trained model without adaptation. For CIFAR-IO-C, the standard is trained with 50000 clean CIFAR-10 train dataset WideResNet18 8 and ResNet. For ImageNet1 C, the standard is trained with ~ 1.2M clean ImageNet train dataset on ResNet50.
SVP: The self-supervised visual prompts attempt to reverse the adversarial attacks by modifying
the input pixels with fy-norm perturbations, where the perturbations are optimized via contrastive loss [40]. For the patch setting, we setup the shape of VP as 32*32*3 for CIFAR-C and 224 for all ImageNet OOD datasets. For the padding setting, we set the padding size as 1 for CIFAR- 10-C and 15 for ImageNet OOD dataset. Take CIFAR data example, we first initialize mask with all zeros value with the shape 30*30*3 and set the pad value as 1 with padding size so that the mask after padding is as the same shape of CIFAR data (32#32#3). Then, we multiply the mask with the VP to preserve only the VP located at the position we just pad with value. We can further optimize the VP with mask by adding it with the corrupted samples.
B5(55): The model adaptation method aims to adjust the BN statistics for every input batch during the test-time. It requires to adapt with single corruption type in every batch.
The test-time training trains the model with auxiliary SSL rotation task and leverages the rotation loss for model adaptation during the testing time. In TTT method, instead of adapting the whole model, they only adapt the last few layers of the model and freeze the parameters in the front layers.
TTT [59]:
The model adaptation method proposed in 71 alters single data point with different augmentations (ie., rotation, cropping, and color jitter,...etc), and the model parameters are adapted by minimizing the entropy of the model's marginal output distribution across those augmented samples.
MEMO:
15

---

Algorithm 1: Convolutional Visual Prompt
Input:
Pretrained classifier F(.), OOD images z, Self Self-supervised objective function L,(-),
Convolutional operator Conw(.), Convolutional kernel Learning rate 7) Number of iteration T
Output: Class prediction ý for adapted sample of æ
Inference
# Initialize the kernel parameters
k ~Uf(a, 3)
# Calculate initial SSL loss
loss (2)
for t E [1.....T) do
# Generate adapted samples
x' = + A* Comv(x k*)
# Calculate SSL. loss with adapted samples
10
loss* = L.(x*)
# Update kernel parameters
12
11+1 R2t + ag
Blass'
# Get optimal kernel parameters
k-
ET
if losss - > loss then
# Use the initial kernel parameters
k: k0
18 # Get final adapted samples
x²= x + Conu(x, k*)
return 1 + F(**)
Algorithm 2: Low-Rank Visual Prompt
Input: Pretrained classifier F(-). OOD images X, Self-supervised objective function C-(-), Rank Number r, Learning rate a, Number of iteration
Output: Class prediction ý for adapted sample of Inference
# Initialize the matrices parameters
SVD on input X(N C' W) > get pairs of initial UEVT (3 channel)
for / E (1...,KK
K} do # Get N inverse matrices
Mc. + U, x Σ, x V/T'
# Apply low-rank SVD on M, with rank
UK, E/, VLT" SV Dlow-rank (M,)
# Generate adapted samples with low-rank inverse matrices
= 2 V'T
10
X; X; + M'
12
Calculate SSL loss with adapted samples
13
loss' Lx(X))
14
# Update the three matrices
U211 U, Moss'
Ex+1 E,
Maass' DE,
VHL VT.
Mlass 0V
3U.
16 Get optimal low rank matrix
17 M +- UK x Σg x VT
18 # Get final adapted samples
19 X= X
M
20 return y - F(x)
16

---

The method adapts the model by minimizing the conditional entropy on batches. In our experiment, we evaluate TENT in episodic mode, which means the model parameter is reset to the initial state after every batch adaptation.
TENT 633
7.4 Implementation details
For the training part of SSL model, we set the training parameters with batch size as 64, training epoch as 200, and the learning rate (Ir) as 0.001. The Ir is decayed with a cosine annealing for each batch (37]. The transformations for contrastive learning are predefined. We augment the inputs with random resize crop, random flip, and random rotation in degree [-90, 90] for positive/negative pairs generation in every batch. The number of transformations for one sample
For the test-time adaptation part, we set the range of parameter á for VP. For the f2-norm perturbations, the is (8/255,8/255) and the step size is 2/255. We the iteration number i either means each component has or steps during PGD. For the update iterations, as the Table 17. show larger number of iterations has better performance. However, the training cost becomes higher.
5, which
The choice of
setting on ranges and kernel
For the À parameter, it controls the magnitude of convolved output when combined with the residual input. We the range to be [0.5, 3] and run test-time optimization to automatically find the optimal solution, which does not require validation set.
We use 3x3 for cifar-10 and 5x5 for ImageNet. In general, small kernel size is used to avoid overfitting. We increase the kernel size for large images, such as ImageNet.
When adapting, the kernel should be initialized with fix/random initialization. We use sharpness kernel as an initial point for the fixed initialization setting, which is a n matrix. It starts from specific values, which can control the edge enhancement effect by extracting the frequency information of inputs with a high pass filter. When the kernel size is 3, we set up the sharpness kernel as [[0,-1, O], [-1, -1,. [0. -1, O]] Similar the convolutional kernel prompt. sharpness transforms an image using a convolutional 2D filter. simultancously enlarges the figh-frequency part of images and then filters the low frequencies part.
CIFAR-IO-C ImageNet-CR.S.AA
Kernel Size
3#3
3 3
[0.5, 1]
[0.5,
Update iters.
1, 5 (default), and 10
Initialization
fixed / random
Table 11: parameter setting
Number of Trainable Parameters: We compare the trainable parameters v.s. accuracy for different
prompting methods. As Figure & shows, CVP contains less than 0.2% number of trainable parameters, compared to VP(patch).
50.0
CVP (ours)
47.5
VP (padding)
(patch)
45.0
Accuracy (%)
Partiel Finetune
42.5
Finetune
40.0
35.0
32.5
30.0
0.009
3.07 17.9
36500
Number of Parameters (Unit: k)
Figure 8
17

---

7.5 More Evaluation
We show more detailed results for CIFAR-IO-C in Table 12 and
Standard
BN
Finetune
VP
CVP
fixed w/o update
fixed
rand (3*3)
rand
patch
append
w/ update
w/o update
w/ update
Gaussian Noise
19,90
24.51
19.26
20.07
19.92
23.11
23.50
24.59
26.27
Shot Noise
20.37
24.25
19.43
20.56
20.40
22.95
23.32
24.29
25.26
Impulse Noise
27.44
26.14
19.19
27.54
27.44
30.76
30.98
31.13
31.08
Defocus Blur
13.18
Motion Blur
23.26
19.95
27.67
30.50
23.29
29.21
30.03
30.37
31,89
Glass Blur
25.97
33.30
18.61
19.70
26.01
36.14
37.54
37,42
40.51
Zoom Blur
71.08
50,46
42.42
71.07
71.06
86.76
87.65
85.76
88.19
Brightness
89.38
71.47
61.03
89.39
89.40
89.37
89.39
89.25
89.31
Snow
71.21
49.73
39.95
71.52
71.22
71.23
71.44
71.17
71.52
Frost
74.83
38.40
46.35
74,93
74.84
74.79
74.81
74.69
74,90
Fog
45.69
50.08
Contrast
58.36
49.87
39.22
57.95
58.47
67.74
68.99
69.05
70.21
Elastic Transorm
17.54
24.01
19.95
17.72
17.56
17.39
17.62
18.07
19.66
Pixelate
23.45
39,80
34.26
24.10
23.50
25.91
26.47
28.39
30.58
Jeeg Compression
45.06
31.37
26.42
45.68
45.17
43.99
44.44
40.74
43.43
Avg.
41.82
Avg. Emor
58.24
62.65
69.33
57.95
58.
54.25
$3.67
53.77
52.37
Avg Diff.
11.09
-0.29
-0.06
-3.99
-4.57
-4.46
Table 12: Comparison of the different prompting methods with CVP for every CIFAR-1O-C corruptionn
type. The Standard model is WideResNet18. Number bold shows the best performance.
Severity Methed
CVP
fixed (3+3)
fixed (3*3)
rand. (3*3)
rand. (3*3)
Patch
Append
update
update
update
update
52.48
43.28
59,76
65.17
66.00
63.98
68.07
$2
47.88
43.18
34.72
48.26
47.94
52.73
53.42
53.41
54.73
44.22
$4
32.75
29.49
24.66
32.94
32.75
36.20
36.74
36.56
37.79
25.16
21.63
30.42
30.64
30.68
Avg. Acc.
41.76
37.35
30.67
42.05
41,82
45.75
46.33
46.23
47.63
Avg. Erro
58.24
62.65
69.33
57.95
58.18
54,25
53.67
53.77
52.37
Avg
Table 13: Comparison of the different adaptation baslines with CVP for every severity on CIFAR- C. The Standard model is WedeResNet18. Number in bold shows the best performance.
More evaluation ViT-Base model architecture. As Table 14 shows, evaluate 15 types of corruption under severity level I for CIFAR-IO- IO-C. The test accuracy of ViT-Base clean CIFAR-10 is 96.42%. Compared the accuracy (%) before adaptation (standard) and using VP to do the adaptation, the CVP achieves better performance by up to points.
Base
VP
CVP
Gaussian
49.71
50.22
Shot Noise
41.92
45.67
46.03
Impulse Noise
80.77
80.39
82.13
Glass Blur
48.49
47.77
49.12
Defocus Blur
27.91
27.37
28.90
Zoom Blur
83.71
83.23
85.63
Motion Blur
58.34
56.22
58.87
Brightness
93.10
94.59
Snow
89.37
88.23
89.41
Frost
90.13
92.19
92.32
Fog
74.75
73.60
75.63
Contrast
90. 14
91.34
92.34
Pixelate
66.37
63.53
Jpeg Compression
62.53
61.55
63.53
Elastic Transform
40.68
40.52
41.52
Averaged Acc.
66.45
66.48
67.77
Table Performance on ViT-Base model.
We show the detailed results for each corruption on ImageNet-C dataset Table 15
In Table 16
we further compare with TTT 59 which is test-time method with CVP. We combine
our CVP on top of TTT, where the model weight will first adapt based on the rotation loss, and then the input will adapt by convolutional visual prompt.
18

---

Standard
Fineture
BN Adapt
VP
CVP
palcc
append
Geussian Noise
80.00
78.85
79.43
79.44
79.99
78.49
78.16
78.47
78.75
Shot Noise
82.00
80.80
81.57
81.36
81.97
80.45
80.00
80.10
80.81
Impulse Noise
82.72
82.72
83.00
80.80
79.82
80.88
Defocus Blur
73.58
75.49
75.32
77.27
73.56
74.13
73.73
74.29
75.10
Motion
89.31
Glass Blur
76.32
87.16
76.86
90.41
79.96
75,
75.47
75.99
75.45
Zoom Blur
80.00
79.84
80.52
82.07
79.35 79.87
79.67
79.72
79,45
Snow
43.86
45.10
45.82
47.888
47.95
44.24
44.27
44,50
44,91
Frost
79.88
81.04
82 22
83.78
74,99
80 12
80.19
79.69
80.05
76.80
74.91
Brighiness
78.25
79.90
81.23
84,99
88.07
78.78
78.49
78.83
78,91
76.59
71.31
71.79
Elastic Transform
87.58
87.96
88
96,50
87.62
87.93
87.85
87.42
88.15
Pixelate
74.72
74.22
75.75
78.30
74.68
67.05
63,98
67.29
64.15
Compression
77.00
75.47
74.85
81.29
76.99
74.41
73.46
74.05
74.26
mCE
76.83
77.10
77.90
80.07
76.74
75.88
75.34
75.74
75.77
1.07
Table 15: ImageNet-C results. Number in bold shows the lowest mCE.
WideResNet18
Avg. Error (%)
Standard
58.24
VP (patch)
57.94 (-0.3)
CVP (rand. w/ update)
52.37 (-5.87)
TTT 63
$2.92 (-5.32)
TTT CVP
53.07 (-5.17)
Table 16: TTT [59] result for CIFAR-10-C
Generalize to Cutout-and-Passe samples To justify that our method can be generalized to non non-structured OOD, we do more experiments on other types of OOD samples, such as the Cutout- and-Paste samples. Here, we launch the experiment the Waterbirds dataset, which is constructed by cropping out birds from images with "water" backgrounds in the Caltech-UCSS Birds-200-2011 (CUB) dataset [6]] and transferring them onto backgrounds from the Places dataset [72]. We follow the GitHub repo and choose the "Forest" as our new background to generate the samples. The training of SSL based the pre-trained ResNet34 backbone model for the original CUB dataset. The original CUB (200 classes) accuracy for the backbone ResNet34 is 75.34%. We compare our CVP with self-supervised VP and demonstrate that CVP more effective on the Cutouted-CUU CUB dataset. The following table shows our results. Our CVP improves the result upon Standard by 1.61 points and VP by 1.3 points.
Cutouted-CUR (200)
Before Adapt
VP (patch) CVP
Accuracy (%) contrastive loss (Avg.)
62.03
62.32
63.64
2.78
2.71
2.52
Table 17: Performance on the Cutouted-CUB
'WaterBirds Datase http:////g.aattoooo////a///ggggp
19

---

7.6 Distance measurement with SWD and SSIM
As the main paper mentioned, we measure the SWD and SSID on two imput distributions: source domain distribution and target domain distribution (before/after adaptation). In Table an Figure | we show the detailed result of SWD on every corruption type in CIFAR-10- under severity 1. On average, CVP achieves lower SWD after adaptation, which means the target distribution is closer to the source one after adaptation. The average SWD) reduced by 0.7% after prompting.
SWD (scale: 10a)
SSIM f
before
after
before
after
Gaussian Noise
5.90
4.71
0.7242
0.7849
Shot Noise
6.08
4.93
0.7124
0.7676
Impulse Noise
6.23
5.26
0.7463
0.7764
Glass Blur
8.85
9.19
0.5873
0.5865
Defocus Blur
13.52
11.82
0,6031
0.6013
Zoom Blur
4.13
3.09
0,8726
0.8703
Motion Blur
7.68
5.57
0,6491
0.6459
Brightness
2.48
3.94
0.9702
0.9692
Snow
5.18
6.07
0,8258
0.8275
Frost
7.61
7.72
0,8025
0,8012
Fog
13.49
9.99
0.5840
0.5785
Contrast
15.39
11.09
0.7049
0.6997
Pixelate
3.09
4.56
0.8603
0.8669
Jpeg Compression
2.58
3.65
0.8681
0.8710
Elastic Transform
5.62
5.75
0,5272
0.5789
Avg. Mean
7.19
6.49
0.7539
0.7884
Avg. Std
4.05
2.79
0.1294
0.7260
Table 18: Results of Sliced Wasserstein Distance and Structural Similarity Index Measure on CIFAR- 10-C (Severity 1).
Betore CVP
After CVP
Betore CVF
After
500
Paumaion Poite
Impulise Noue
Most, puice
pesanian Aooii
Impulist plone
Mhit rolie
mestem Alur
placs blur
dedocis plur
montant blir
plass blar
detacus Mos
taam
(a) Noise Group
(b) Blur Group
200
23.5
5003 ocatte 2011
12:5
immatt
transform
Diesiate
Wag compressiion
elasti tramfom
Besiaate
pagg conpression
bogiines
Intue
fostt
Inow
led
(c) Weather Group
Digital Group
Figure 9: Violin Plot of SWD for different corruption groups on CIFAR-10-C. The left figure of each subplot shows the SWD before adapting, and the right shows the SWD after adaptation
20

---

Distribution changes after applying the propased CVP
In the main paper, Figure 2, show the distribution changes in different corruption types and severity. Here, in Table we show the distribution shifts after applying our CVP by calculating the average loss. In general, the distribution moves back the original distribution. We show the SSL average loss (before adapt/after CVP adapt) for four corruption types on severity 1,3,5 for CIFARIO-C. The average SSL loss for the original CIFAR 10 1.26. For every corruption we show here, the average SSL loss after adaptation is lower than the loss before adaptation.
severity
sl
s3
s5 Before /After
Before /After
Before /After
Gaussian noise
1.971.6
2.572.1
3.372.6
Defocus blur
3.2 2.9
3.4/ 2.8
3.7 3.1
Snow
3.1 3.0
3.8 3.3
/.5
Contrast
2.7/
2.9
3.6/3.3
Table 19: Distribution changes on different corruption types.
21

---

7.7 The Effect of Different Prompt Designs
We do analysis on different prompting methods, including original visual prompts with different norm-bound (62, ),o), convolutional prompts, and their combinations (f2 + conv., Es + conv.). We show the error rate different numbers of adapt iters for every prompting method from 1,5, to 10. To compare the results, we set other parameters such as the epsilon as 8/255 for foc. 1 for lz. As Figure Thoshows the error rate for different prompting methods, the convolutional prompt comv. and its combination with f2 reduce the error rate, and the former one reduces more from 40.32% to 36.08% when increasing the adapt iters. However, other prompting methods increase the error rate after prompting. To understand the risk of over-fitting for different prompting methods, Figure 4b shows the SSL loss curve v.s. performance on different prompting methods.
50
Patch (12)
48
Potch (linf)
Patch (12) Conv.
46
Patch [linf) Conv.
Error (%)
44
Conv
42
40
38
36
0
1
10
Adapt Iters
Figure 10
22

---

7.8 Analysis on CVP and low-rank prompts
In Table 20 we show the detailed results of low-rank prompt (LVP) on different severity (from to 5) for CIFAR-10-C. We set the same rank as 3 for LVP and CVP. Our results show that the CVP is more effective than LVP when reversing natural corruption. In Figure [T] we further plot the averaged contrastive loss on different rank sizes for both L.P and CVP. On every corruption type, while increasing the rank size from to 31, the loss curves of LVP consistently drop, which demonstrates the LVP is much more easier to overfit the contrastive loss.
Severity Method
Standard
LVP
CVP
s1
40.32
37.05
31.93
s2
52.12
48.83
45.27
s3
59.69
56.05
54.04
67.25
63.42
62.21
s5
71.80
68.89
68.39
Avg. Error
58.24
54.85
52.37
Diff.
-3.39
-5.87
Table 20
3.09
3.40
3.30
88
3.09
3.20
3.30
2.70
- Low Rank
3.68
3.20
3,00
3 7
11
15
18
23
27
31
7
11
15
19
23
27
31
11
15
13
23
27 21
11
15
19 Value
23
27
Bank Vill
Bank Value
Rank Vilue
Bank
gaussian_noise
defocus blur
snow
contrast
(a)
0.75
Contrasive Less
1.00
0.80
0.50
0.50
0.90
Corw:
0.60
0.25
0.25
0.60
Low Bank
0.40
0.00
0.00
0.40
.0.25
-025
27
27
23
11
23
Kernel
Kernel
Kernel
Kernel
gaussian noise
defocus
Snow
contrast
In subfigure a., We show the b distance between the ground-truth additive corruptions vector A and our prompt reversing vectors d. Subfigure b shows the averaged contrastive loss of low-rank prompt and CVP on different rank sizes. For every corruption type, the loss curves of LVP consistently drop when increasing the rank 31 while the & distances increase, demonstrating how LVP is more prone to overfitting the contrastive loss.
Figure 11:
23

---

7.9 t-SNE analysis on different adaptation methods.
In addition to performing analysis on single sample, we further conduct the t-SNE visualization for whole sample distribution on different baseline. For each type of corruption data, we extract the 1-dimensional logit features in the last layer of model and calculate the distance between them with respect to the predicted class labels. We compare our method with standard, MEMO, and MEMO + Ours. As Figure 12 shows, the original feature embedding shows low separability between different classes. On the other hand, our approach clearly discriminates the embedding feature, which demonstrates its robustness against distribution shifts.
original
ours
memo-only
memo + ours
10
20
10
70
-10
10
-32
(a) Motion Blur
original
ours
memo-only
memo ours
-10
-30
10
-10
-10
10
(b) Impulsc Noisc
Figure 12
7.10 Saliency map analysis on different corruption types
To better understand how self-supervised visual prompts adapt to the corrupted inputs, we visualize the saliency map of different types of corruption. As Figure 13 shows, from left to right, the first row is the original, corrupted, and adapted samples; the second row shows their corresponding Grad-CAM with respect to the predicted labels. The red region in Grad-CAM shows where the model focuses on for target input. We empirically discover the heap map defocus on the target object for corrupted samples. However, after prompting, the red region of the adapted sample's 's heap map is re- re-target on the similar region as original image, which demonstrates that the self-supervised visual prompts indeed improve the input adaptation and make the model refocus back on the correct regions.
24

---

Original
Corrupted
Adapted
Original
Corrupted
Adapted
Input
Input
Grad CAM
Grad CAM
(a) Contrast
(h) Gaussian Noise
Original
Corrupted
Adapted
Original
Corrupted
Adapted
Input
Input
Grad CAM
Grad-CAM
(c) Fog
(d) Impulse noise
Original
Corrupted
Adapted
Original
Corrupted
Adapted
Input
Input
Grad-CAM CAM
Grad-CAM
(c) Brightness
(f) Pixclate
Figure 13: Grad-CAM analysis on different types of corruption.
25


Reviews:
[{"review_id": "9GJa653x6k", "reviewer": "Reviewer_uZKT", "summary": "- The paper introduces Convolutional Visual Prompts (CVP), a novel methodology designed to increase the model robustness when faced with Out-of-Distribution (OOD) data during test time.\n- The primary innovation of CVP is the use of convolutional structures as inductive biases for adapting to visual OOD instances during test time.\n- The experimental results show that these convolutional structures are efficient and effective at dealing with OOD corruptions.", "questions": "- What motivated the choice of convolutional structures over other possible structures for the CVP?\n- Could you discuss the wider usage of CVP when applied to settings other than test-time adaptation?\n- Could you also discuss the extendability of CVP model, like the possibility/effectiveness of adding more kernels? I am not saying to develop a new variant during the rebuttal, but to indicate how the followers can use the method to the best extend would enhance the significance of the work.", "limitations": "The paper seems not discuss the limitations, but when the authors try to answer the weakness and questions above, they might come up with more discussion on the limitation of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- *Originality*: The paper proposes a novel method, the use of convolutional structures as inductive biases for visual prompt tuning to OOD data. This work would interests the visual prompt community.\n- *Quality*: Extensive experiments have been conducted across a variety of OOD recognition tasks, providing empirical support for the effectiveness of CVP.\n- *Clarity*: The paper is well-written and comprehensible, offering clear explanations of the problem, the proposed solution, and the obtained results.", "weaknesses": "The major concern of this work is its Motivation behind the choice of convolutional structures: The paper does not provide a clear motivation or justification for choosing convolutional structures as the basis for the Convolutional Visual Prompts (CVP). While the paper mentions the effectiveness of convolutional operations for handling structured data with local motifs, a more explicit link between this principle and the task of OOD adaptation would be beneficial. For instance, the authors could hypothesize that the learned convolutional kernel might act as the de-corruption operator for specific corruptions, offering a possible reason for their choice (the point was raised by the visual evidence like Figure 11, which seems indicate the reason behind why CVP works).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Motivation of using convolutional visual prompt for test-time adaptation**\n\nThank you for your question. We agree with the reviewer that our CVP can be viewed as a de-corruption operator for natural images. Our paper studies how to be robust under major natural distribution shifts, such as style, smoothness, or lightening changes. Those shifts are often locally structured, which motivates us to use convolution that handles locally structured data. Empirically, our convolution directly undo the corruption, as shown in Figure 11.\nMoreover, our empirical results show that our proposed convolutional visual prompts outperform standard visual prompts and low-rank visual prompts. We will make the input de-corruption operation point more clear in our revision. It will be interesting to explore how well CVP adapts to more global changes, such as shape change, which we leave for future work.\n\n**Discuss the wider usage of CVP when applied to settings other than test-time adaptation**\n\nWe thank the reviewer for pointing out this. We will add it in the discussion part for the later revision. We list a few additional usages of CVP below:\n\n1. Beyond unsupervised test-time adaptation, we can also supervisedly optimize our convolutional visual prompt when a few labeled data are available on the target domain. Since our convolutional visual prompt has fewer parameters, applying it to this setting can reduce the training cost. \n\n2. Continual learning can benefit from our lightweight CVP. Since the environment often changes incrementally, applying the structured CVP adaptation can potentially help continual adaptation without forgetting.\n\n3. The model deployed on edge devices can also benefit from CVP. As it has limited memory and requires handling multiple types of corruption, the efficiency of the parameter of CVP allows us to maintain individual prompts for each scenario without using too much memory. \n\n**Discuss the extendability of CVP model**\n\nThank you for your question. Yes, our method can be extended by stacking deeper convolutional prompts. In addition, future work can study adding CVP to the latent representations and study if the hyper network can be used to generate the parameters for the CVP for meta-learning. In terms of application, CVP can be used to adapt not only classification models but also generative models such as diffusion. \n\n"}, {"review_id": "8UsLxDfM8b", "reviewer": "Reviewer_zs9s", "summary": "The paper proposes a variant of Visual Prompt Tuning (VPT) where the prompt is applied to a penultimate layer of the encoder network and is a result of a convolution with a 3x3 or 5x5 kernel (essentially it is an added residual block or residual adapter if you will). The added adapter is trained at test time by applying a contrastive loss on a test batch (so, in fact, it seems to be a variant of transductive learning, requiring a batch of test images - to form a negative set for the contrastive loss, so not truly online and does not seem to be able to operate on a single test sample). The authors test their approach on a collection of OOD benchmarks, some synthetic, and some real - these ones are in fact UDA / UDG benchmarks, however, UDG baselines are not compared. Small improvements are observed over naive baselines, though prior methods like [53] and [60] seem to deliver much better performance (Table 4) and the authors only show the benefit of their approach via ensembling / combining with the other methods.", "questions": "- PromptTuning is a weaker variant of the broader family of PEFT methods, usually applied on the input level (penultimate layer in this case) it does not have enough representation power to model more complex deviations, as opposed to more popular LoRA or prefix tuning methods. Would the authors' approach be applicable to the more powerful PEFT methods?", "limitations": "limitations not discussed", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- test time adaptation, VPT, and handling domain shifts are important topics\n- some improvement is observed over simple baselines", "weaknesses": "- novelty: adding a convolutional residual adapter + contrastive loss for transductive learning (test time adaptation on a batch) is hardly novel\n- comparisons to prior works:\n* table 4 shows [53] and [60] attain significantly better results than the proposed approach\n* relevant baselines / benchmarks seem to be missing (e.g. https://arxiv.org/pdf/2210.04831.pdf reports better results on ImageNet-C + DomainNet benchmark could be added)\n* I would expect comparison to / on top of UDG baselines that solve the same problem\n* combining methods as in table 4 is fine, but other methods could also be combined, would it yield even higher results?\n- Supervised variant of the approach explored in the ablation needs to be compared to few-shot and in particular transductive few-shot methods, expecting much higher results there.\n- what about the online test-time tuning (from one sample) seems not possible due to the use of the contrastive loss?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Novelty of our work**\n- Our paper provides deep insights on what is a good visual prompt design for test-time adaptation, which is an important problem. The key novelty of our work is to propose this simple and effective convolutional visual prompt to address the overfitting challenge at test time.\nPrior work like convolutional residual adapter adapts the model architecture with lots of parameters, which is designed for training time optimization but can lead to overfitting on test-time adaptation. Our method only adds a small convolutional prompt on the inputs during the inference time, which is both lightweight and avoids test-time overfitting.\n- Moreover, the other Reviewers XDjm and HEpT find our method to be novel and well-motivated. Reviewer uZKT also mentions this research direction would interest the visual prompt community.\n\n**Performance compared with other methods (BN [53] and TENT [60])**\n\n- Our method is orthogonal to BN [53], TENT [60] and can be applied on top of them, improving those methods even more. Since our method updates only the input prompt without modifying the model weights, it works with frozen models and avoids the high cost of fine-tuning (especially on edge devices like smartphones). Our work provides a new direction for the robustness field by visual prompting at test time, which we think is worth publishing.\n\n**Compare with relevant baselines such as [A]**\n\n- We thank the reviewer for suggesting this relevant paper, which we will cite. \n\n- We highlight the difference between our work and this paper as follows. 1.) Our CVP creates during the adaptation phase, and we don't need any source domain sample to tune the prompt first. 2.)  In contrast to [1] tuning a collection of visual prompts and the classification head, we adapt every batch sample using CVP, which is just a single convolutional kernel. Our method is simple, lightweight, and easy to use.\n\n- [A] Gao, Yunhe, et al. \"Visual prompt tuning for test-time domain adaptation.\" arXiv preprint arXiv:2210.04831 (2022).\n\n**Comparison to / on top of UDG baselines that solve the same problem**\n\n- Standard UDG methods require training on multiple source domains and testing generalization on a different domain, which is not the ImageNet benchmark setup we evaluate on. \n- Our UDG scenario has only a single training domain, like Imagenet, and tests the model on multiple OOD domains. We thus only compare UDG methods that conduct single-domain training, such as BN, TENT, and MEMO. Experimenting with UDG with multiple training domains is out of the scope of our paper. \n- We would like to include more baselines if the reviewer can point out other single training domains UDG work that we are missing.\n\n**Combining other methods**\n- Thank you for your question. We agree that since our method applied a prompt for the input, it can be easily combined with other methods that change the model weights. We believe our method can further improve the robustness of future work that adapts model weights.\n\n**Compare the supervised variant of the approach with transductive few-shot**\n- We want to clarify that results from the supervised variant are only for training the best case and thresholding the upper bounds of our method.  \n- Our goal is to focus on test-time adaptation without the annotations, where we indeed compare with the transductive few-shot method (For example: MEMO) under the self-supervised setting (see Table 4.). The supervised variant of the approach represents the baseline which is not our main contribution. \n\n**The online test-time tuning with one sample**\n\n- Our convolutional visual prompt is general and can be applied to other SSL tasks that use one sample. In Table 7, we show our method also works with rotation prediction and MAE, which only needs one sample and achieves up to 3.14\\% improvement in robustness. \n\n\n**Can CVP complement other parameter-efficient tuning (PEFT) methods?**\n\n- Yes, they are complementary. The PEFT methods often update the model weights, while our CVP updates the input. Thus our method is orthogonal to PEFT and can be directly applied to PEFT to improve the model adaptation performance further. We will add this in our future work.\n\n\n"}, {"review_id": "t3tk8ppHE3", "reviewer": "Reviewer_HEpT", "summary": ".The paper proposes a new method for test-time adaptation (TTA) to structured distribution shifts. The proposed method learns convolutional visual prompts to prevent the model from overfitting to SSL objectives due to high dimensional prompts. The results show that the proposed approach consistently results in performance improvements for TTA to OOD variants of CIFAR-10 and ImageNet.", "questions": "I am happy to update my rating if the concerns around comparisons to prior work and hyperparameter selection are adequately addressed. But I request the authors to provide the missing details.\n\n1. In Table 8, multiple choices are provided for kernel size and update iters, how are the chosen hyperparameters validated?\n2. What are the hyperparameters used in Table 4/5/7, where the kernel size is not specified?\n3. How is CVP applied to CLIP-ViTs? Is the image first convolved and then tokenized?\n4. Which CLIP-ViT/32 model and can was used for the results in Table 3? [A] does report numbers on CLIP-ViT-base/16.\n5. What architecture is used for the results in Table 7? \n6. For the MAE baseline in Table 7: is the contrastive objective of CVP directly replaced by MAE objective? But this introduces additional decoder parameters as well right? It would be good to see a comparison of the number of parameters that are optimized in different SSL methods.\n7. L9: “1% when compared to standard visual prompts”. How does this compare to the shallow prompts used in [26].\n8. Tables 9 and 12 show that CVP does not outperform baselines for fog/frost/snow corruptions. Also, Figure 3 shows CVP does not outperform low-rank prompts for snow corruption. Any intuition on why CVP does not perform well when weather conditions are changed?\n9. In Table 15, are the reported losses on samples that were used to tune prompts? Reporting the loss on unseen samples may give a better idea of the extent of overfitting.\n10. Since [A] is a prior TTA work for CLIP, would there be any benefits of dual text-visual prompt tuning (like [C]) with: CVP (your method) + [A] (NeurIPS’22)? \n\n\n[A] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models. Shu et al. (NeurIPS'22)\n\n[B] Test-Time Training with Masked Autoencoders. Yossi et al. (NeurIPS'22)\n\n[C] Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model. Xing et al.", "limitations": "The paper may need to add a limitation (and future work) section if the above concerns (applicability to uni-modal ViTs and hyperparameter tuning) can not be addressed in the current submission.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "- The idea of using convolutional prompts to avoid SSL overfitting is novel and well-motivated (Table 5 shows how SSL results in overfitting and Figure 3 shows the need for some form of inductive bias).\n- The results indicate consistent results when CVP are added to baselines and other prior TTA works.\n- The method tunes very few parameters. The parameter efficiency allows it to be adopted in edge computing devices working with low memory and compute.", "weaknesses": "1. The paper compares their method with only two types of visual prompts (patch and padding based [40]). The paper does report results with CLIP ViT-B (Table 3), but does not compare to other prompts used for adapting ViTs [26] and text-only prompts for CLIP-pretrained ViTs. [A]\n2. The paper mentions on L63: “The only one that does not update the model (for Test time adaptation) is proposed by [40]”. However the following uncited works do not update the model as well [A, B].\n3. The paper does not show results with unimodal ViT architectures (eg. pretrained models from TIMM). Similar gains over the baselines (and [26]) in the context of ViTs can boost the applicability of the approach.\n4. The paper sets different hyperparameters for different datasets (range of $\\lambda$, kernel sizes), without providing guidelines on how these are to be determined for a new dataset (or corruption type).\n5. Missing details on evaluation and implementation (covered under Questions). Minor writing: $\\lambda$ is introduced much later in text, while being introduced in 3.3.\n\n[A] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models. Shu et al.\n\n[B] Test-Time Training with Masked Autoencoders. Yossi et al.\n\n[C] Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model. Xing et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate the reviewer’s comments and suggestions. Thanks to the reviewer for recognizing the novelty of our work. We have answered and addressed the questions.**\n\n**Comparison with other prompts**\n\n- Thank you for your suggestion of comparing with other prompts. We have followed your suggestion and included an additional comparison with the shallow prompt [26] in the Table below.\n- We mention that [A] uses text prompts tailored to vision language model adaptation. Our method is more general and can be applied to any visual perception model.\n- We run the experiment on CIFAR10-C and show robust accuracy. The shallow prompt doesn't improve under our setting because it requires more learnable parameters in the prompt and leads to overfitting for adaptation.\n- |                | s1    | s2    | s3    | s4    | s5    |\n|----------------|-------|-------|-------|-------|-------|\n| CLIP-ViT-b/32  | 58.58 | 48.45 | 40.12 | 33.38 | 27.51 |\n| Shallow prompt [26] | 56.87 | 47.22 | 38.72 | 30.49 | 24.10 |  \n| CVP            | 59.11 | 49.09 | 40.76 | 33.51 | 27.80 |\n\n**Other test-time adaption method**\n\nThanks to the reviewer for pointing out those works [A] [B]. We will definitely cite them in our revision.\n\n[A] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models. Shu et al.\n[B] Test-Time Training with Masked Autoencoders. Yossi et al.\n\n**ViT Results**\n- Thanks to the reviewer for suggesting this. We ran the experiment for CVP on the ViT-Base model with results showing a similar gain in performance.\n- The following table shows our results of the ViT-Base model for 15 types of corruption under severity level 1 on CIFAR-10-C. The test accuracy of ViT-Base on CIFAR10 is 96.42\\%. Compared to the accuracy (\\%) before adaptation (standard) and using VP to do the adaptation, CVP achieves up to 2% better performance. More detail of ViT results can be found in the PDF Table 1. in general responses.\n\n- |                   |          | ViT-Base/32 |       |\n  |-------------------|----------|:--------:|-------|\n  |                   | standard |    VP    |  CVP  |\n  | Averaged Acc. | 66.45 | 66.48 | **67.77** |\n\n\n**The choice of hyper-parameter setting on kernel size, $\\lambda$ range, and update iterations**\n- For the $\\lambda$ parameter, it controls the magnitude of convolved output when combined with the residual input. We set the $\\lambda$ range to be [0.5, 3] and run test-time optimization to automatically find the optimal solution, which does not require a validation set. \n\n- We use 3x3 for cifar-10 and 5x5 for ImageNet. In general, small kernel size is used to avoid overfitting. We increase the kernel size for large images, such as ImageNet.\n\n- For the update iterations, as the Table 17. show a larger number of iterations has better performance. However, the training cost becomes higher.\n\n**Missing details on evaluation and implementation**\n- Thanks for mentioning it. We will add all of these details in our later revision.\n1. How are the chosen hyperparameters validated in Table 8?\n    - See the above section, \"The choice of hyper-parameter setting.\"\n2. What hyperparameters are used in Table 4/5/7, where the kernel size is not specified\n    - For Table 4, we set the kernel size for CIFAR-10-C as 3x3 and ImageNet-C, R, S, and A as 5x5. The adaptation iterations are all set as 5.\n    - For Table 5, same as the setting above, the CIFAR-10-C uses the 3x3 and others use 5x5. The adaptation iterations are all set as 5.\n    - For Table 7, we evaluate the performance for three SSL tasks on ImageNet-C. The standard model is ResNet50. For the CVP kernel, we use the same setting as above, the 5x5, which is the optimal choice. \n3. How is CVP applied to CLIP-ViTs? Is the image first convolved and then tokenized?\n   - Sorry for missing details. Yes, our CVP is applied before tokenized. The CVP is applied to CLIP-ViTs by adding a convolutional kernel on the input sample and then getting an adapted sample as a new input iteratively. In our later revision, we will add more detail on how CVP applies to CLIP. \n4. Which CLIP-ViT/32 model was used for the results in Table 3?\n    - Thanks for the careful check. The CLIP model we use for evaluation is the CLIP-ViT-base/32. In our later revision, We will clarify our model CLIP-ViT/32 as CLIP-ViT-base/32.\n5. What architecture is used for the results in Table 7?\n    - The architecture we use for Table 7 is the ResNet50 pre-trained with original ImageNet-C.\n6. Clarification for MAE baseline\n    - Yes,  the contrastive objective of CVP is directly replaced by the MAE objective. We replace the contrastive loss with reconstruction loss.\nSince the decoder for MAE is fixed, we do not optimize additional parameters at test-time adaptation.\n7. L9: 1\\% when compared to standard visual prompts”. How does this compare to the shallow prompts used in?\n   - Thank you for your question. We compared to shallow prompts, which is also less than 1\\%.\n8. Why does CVP not outperform baseline on weather conditions\n   - While our CVP doesn't outperform the VP baseline on weather corruption, it outperforms the standard baseline results. \n   - In Figure 2., we show deep insight into the contrastive loss distribution and empirically find that the weather corruption (for example, snow) leads to a huge shift from the source domain even in the low severity. Therefore, we conjecture that the weather corruption types over-damaged the intrinsic structure and need more learnable parameters to restore the structure.\n9. In Table 15, are the reported losses on samples used to tune prompts?\n    - Yes, we report the losses for samples already tuning with the prompt. \n    -  We run the loss analysis on unseen test samples and show CVP can avoid overfitting. (shown in PDF Fig. 3 in general responses).\n10. Dual text-visual prompt tuning\n    - Thank you for these suggestions. We feel that combining text prompts and CVP is an interesting idea. We will add it in future work.\n\n\n"}, {"review_id": "bzaHE0jAdH", "reviewer": "Reviewer_XDjm", "summary": "This paper proposes a novel label-free approach CVP for test-time adaptation on out-of-distribution data. The main idea is to use a convolutional kernal as the visual prompt. It captures the structure of data distribution shift, and reduces the trainable parameters. Experiments show that CVP improves model robustness and complements existing weight-adaptation methods.", "questions": "1. In Ln 206, \"starting from a sharpness kernel is effective\". Could the authors elaborate on what 'sharpness kernel' is and why it is effective?\n\n2. The convolutional kernel applies on the whole image. Sometimes, the corruption may happen in some local regions, e.g., motion blur on foreground objects only. How do the authors comment on such situations?", "limitations": "The authors have not discussed limitations. But I feel that the method is simple, and experimental analyze are extensive enough.\n\nIt is helpful to discuss the effectiveness under other OOD types like open-class data, high-level style changes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Prompt tuning has become an important technique to adapt large visual models. This paper provides a simple solution and may be useful in various applications.\n\n2. The paper is well motivated. Using convolutional structure to inject the inductive bias is simple yet effective. It reduces the required parameters, and shows significant better performances than low-rank prompts.\n\n3. The method does not require labeled data.\n\n4. The method can be combined with other weight adaptation methods. It can also be generalized to multiple self-supervised objectives. The authors provide corresponding experiments with encouraging results. \n\n5. The paper is well structured and nicely written. Empirical results are quite extensive. The attention visualization provides a clear insight.\n\n", "weaknesses": "1. The corruption types considered in this paper seem to be restricted to low-level transformations. This explains why the convolutional kernel is suitable. I wonder if CVP would still work well for high-level distribution shift, such as style changes? \n\n\nTypo:\nReference missing in Ln 175.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for endorsing and recognizing our work as realistic, interesting, and well-studied.**\n\n**CVP would still work well for high-level distribution shift, such as style changes?**\n\n- It's a very good point. We do have experiments on those style changes benchmark such as the ImageNet-Rendition, Sketch, and Adversarial. Our results show that CVP can improve robust accuracy on such a high-level distribution shift.\n\n**In L206., \"starting from a sharpness kernel is effective\". Could the authors elaborate on what 'sharpness kernel' is and why it is effective?**\n\n- When adapting, the kernel should be initialized with fix/random initialization. We use a sharpness kernel as an initial point for the fixed initialization setting, which is a n*n matrix. It starts from specific values, which can control the edge enhancement effect by extracting the frequency information of inputs with a high pass filter. When the kernel size is 3, we set up the sharpness kernel as [[0,-1, 0], [-1, 5, -1], [0, -1, 0]]\n- Similar to the convolutional kernel prompt, sharpness transforms an image using a convolutional 2D filter. It simultaneously enlarges the high-frequency part of images and then filters the low frequencies part.\n\n**Can convolutional kernel applies on the partial region**\n- This is an interesting point. We only apply it to the whole image. We will study this in future work."}, {"review_id": "PU8i4LBkcJ", "reviewer": "Reviewer_QKSZ", "summary": "The paper addresses test time adaptation by employing learnable convolution operation named CVP on out-of-distribution (OOD) images. The motivation is to take advantage of the inductive bias imparted by convolution operations with the added advantage of learning fewer parameters compared to existing approaches. CVP in conjunction with other sota approaches improve the performance on benchmark datasets.", "questions": "Questions are all detailed with the associated weaknesses raised above (in ‘weaknesses’ section). Here, in addition, some minor typos are mentioned.\n- Line 156: Between ‘shifts’ and ‘In’, a stop is missing.\n- Line 174: ‘… method, We …’ -> ‘method, we’\n- Line 175: Appendix number is missing\n- Line 178: ‘contains’ -> ‘containing’\n- Line 200: ‘More baseline detailed are shown’ -> ‘More details about the baselines are provided’\n- Line 204-205: ‘random initialize the kernel size k’ -> ‘randomly initialize the kernel of size k’", "limitations": "Limitations are not explicitly described. It may be useful to discuss the scope of the claims regarding the convolution operation in CVP handling structural bias in OOD.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Using small convolution operations at the start for OOD test time adaptation to take advantage of inherent inductive bias seems to be new.\n2. The combination of selfsupervision and convolution makes it both parameter as well as label efficient.\n3. Main and ablation experiments are largescale", "weaknesses": "1. Eqn. (1): When $y_{i,j}^s$ is 0, $i$ and $j$ are from different samples (negative pair). So, in the loss calculation negative pairs are not considered. Then how is it a contrastive formulation?\n2. Line 111: It is said that $\\mathcal{X}_s$ is the source domain data for training. If source domain data is used then it is not a test time adaptation framework and comparing it with such works would not be fair.\n3. Line 150: One of the important assumptions made in this paper tells the distribution shift in images is often visually structured. Why is this assumption right? What tells that the distribution shift is 'visually structured'?\n4. Line 180+: When describing the proposed model, it is told that the logits before the fully connected layer of the backbone model are extracted for training the SSL model. This is making me confused. Does it mean the conv operation is applied on the logits and not on the image? I think having a figure describing the proposed approach would have been useful here. I don’t see such a figure and thus I have to presume things. Also, it is said that a MLP gets trained in addition. Is it used during inference? Also, is this taken into account in the learnable parameter count, especially when compared with other related approaches like TENT or other traditional prompting methods? In this regard, the figure 6 in the supplementary is incomplete. What is the unit in x-axis? Millions, billions, K?\n5. Line 184: When extending the proposed approach to CLIP, why only vision encoder is used for prompting? Why not text encoder only? Is there any ablation that is done?\n6. Section 4.1: FT and PFT Baseline: Why is this restoring to initial weights required? Is it standard in literature? Any reference?\n7. Line 206: What is a sharpness kernel?\n8. Section 4.2, CVP Complements Other Test-Time Adaptation Methods: More details are required on what is meant by 'combining' CVP with these methods. Also, the results of this experiment (Table 4) are not convincing. Improvement is very marginal compared to TENT, in all three imagenet variants. Compared to standalone CVP, CVP+x provides huge error rate drop - then this means the major workforce behind the performance is x and not CVP.\n9. Ablation study – Low rank structure: More details are needed on how the low rank prompts are created? Are they directly applied on the input? According to the original LoRA paper [24], low rank adaptation is shown to be effective in transformer architectures. How is it adapted to convolutional architectures? As two low rank matrices are involved (I'm supposing so, as details are missing), this evidently involves more learnable parameters and thus susceptible to overfitting. So, the comparison to conv kernels is not fair. The corresponding figure in Appendix (Fig. 9), does not have any descriptive captions. It’s better not to give such figures if everything about the figure has to be inferred by the reader.\n10. Line 266-271 and Table 7: What is 'standard' SSL task? Is it employing eqn. (1)? Its difference with 'contrastive' variation needs to be detailed. While rotation or MAE were shown to perform worse compared to the standard SSL task (in Table 7), there is no detail of these two other SSL approaches. How many variations of rotation prediction were used? How much percentage of the images were masked in MAE? Is there any study by varying the mask percentage? How long are these other SSL tasks trained? Generally Contrastive and MAE tasks require a long training for better learning. Was this good practice followed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Clarification for contrastive formulation**\n\nThe loss indeed considers the negative pair, which is in the denominator. If $x_{i}$ and $x_{j}$ are positive pair, the $y_{i,j}$ is 1; then we maximize the similarity between the pos on the nominator and minimize the denominator, which is similarity between all negative pairs $x_{i}$ and $x_{k}$, where $j \\neq k$. When $y_{i,j}$ is zero, the whole term is 0, and does not impact the loss.\n\n**Clarification for test-time adaptation framework**\n\nAs the paper [A] defined, Test-time adaptation is running optimization at test time for target-domain data, which does not prohibit a model pre-trained with source data. Test-time adaptation methods, such as TENT, TTT, and MEMO, all train their model using the source data. \n\n[A] Gao, Yunhe, et al. \"Visual prompt tuning for test-time domain adaptation.\" arXiv preprint arXiv:2210.04831 (2022).\n\n**What tells that the distribution shift is 'visually structured'?**\n\nOur visual world is known to be structured, and the changes are not random. For example, the shift introduced by blur is highly structured: you can tell there is blur in the image. This assumption works well according to our empirical results. We agree that our method may not handle arbitrary shifts that are not structured, but this is out of the scope of our paper and we leave this for future work.\n\n**Clarification Line 180+**\n\n- The conv. operation is applied to the image and only during the adaptation phase. We draw the flow of our proposed method in PDF Figure 1 in general responses.\n-  Yes, the MLP is used during the inference, yet the weight of it is fixed and not counted as learnable parameters at inference. When compared with other approaches, Fig. 6 shows only the number of learnable parameters at the inference time. \n- For Fig. 6, we redraw it and show it in the PDF Fig. 4. The x-axis is in unit K.\n\n**Line 184: When extending the proposed approach to CLIP**\n- We focus on the convolutional prompt only applicable to the vision encoder. Combining text prompts with CVP will be future work.\n\n**Clarification: Section 4.1 FT and PFT Baseline**\n\nYes. Standard test-time adaptation initials or reset weights after a one-time adaptation, such as MEMO. Because the samples that arrived in batches can be uncorrelated to each other at inference, using the updated weights from prior batches will hurt the performance.\n\n**Line 206: What is a sharpness kernel?**\n- Sharpness is a standard $nxn$ matrix that enhances edges with a high pass filter. For example, [[0,-1, 0], [-1, 5, -1], [0, -1, 0]].\n\n**Compared to other TTA baselines**\n\nOur method is orthogonal to BN [53], TENT [60] and can be applied on top of them, improving those methods even more. Since our method updates only the input prompt without modifying the model weights, it works with frozen models and avoids the high cost of fine-tuning. \n\n**Details about low-rank prompts in ablation studies**\n\n1. Pseudo algorithm for LVP. We will add this to the supplementary.\n - SVD on input $X$ →  get initial $ U \\Sigma V^T$  \n - Set rank number $r$ (ex: 3)\n - For every adapt iteration $t$:\n    - Get inverse matrix $M_t = U_t * \\Sigma_t * V^T _t$ \n    - Apply low rank SVD on $M_t$ with rank number $r$ →  get low rank $U'_t$, $\\Sigma'_t$, $V'^T _t$\n    - Get new inverse matrix $M’_t = U’_t * \\Sigma’_t * V’^T _t$\n    - Get low rank X  → $X’ = X + M’_t$ \n    - Calculate contrastive loss $L_s$ on $X’$ by inference the  SSL model\n    - Get gradient for $U'_t$,  $\\Sigma'_t$, $V'^T _t$ and Update the three matrices via SGD \n\n2. Are they directly applied to the input?\n    - Yes, the low-rank matrices are directly applied on input X\n\n3. Clarification: How is it adapted to convolutional architectures?\n    - Since our low-rank prompt is applied on the input, not the intermediate layers of the model, it can be applied to convolutional neural networks following prior literature [B].\n    - [B] Yang, Yuzhe, et al. \"Me-net: Towards effective adversarial robustness with matrix estimation.\" arXiv preprint arXiv:1905.11971 (2019).\n\n4. Compare low-rank prompts with CVP\n   - We agree that low-rank prompts induce more learnable parameters due to the decomposition of matrices. However, compared with VP, CVP, and low-rank prompts require only 1\\% and 9\\% of the parameters, respectively, making both methods lightweight. Therefore, we want to highlight that both of these lightweight prompts have significant performance improvements.\n\n5. We are sorry for the missing details. We already added a detailed explanation and full caption for Fig. 9 in our PDF Fig. 2.\n\n**Line 266-271 and Table 7, Questions about SSL tasks**\n\n1. Sorry for the confusion. The 'Standard' means the backbone model without adding SSL. In Table 7, the standard model is the ResNet50 pre-trained on ImageNet. We compare the adaptation performance of three SSL tasks with this standard model.\n2. Sorry for the typo. In Table 7, the reported results are the average robust accuracy, not the average error rate. As the subparagraph in the Ablation study mentioned, our CVP improves robust accuracy for every severity level on ImageNet-C when optimized with all three SSL tasks. We will add the details of SSL tasks in our later revision.\n4. We have four degrees for our rotation prediction tasks, including 0, 90, 180, and 270.\n5. We set up 75 \\% of the images to be masked in MAE, following the default setting in [C] for setting up the mask percentage. We will add more studies on the varying mask percentages for MAE. \n    - [C] He, et al. \"Masked autoencoders are scalable vision learners.\" CVPR'22.\n6. As our MLP model is small, the SSL tasks take only 1~2 hours for training. We train SSL only once before making the adaptation. Once the SSL has been trained, it is fixed, and we don't need to tune it at test time. SSL module introduces little inference overhead in exchange for a large gain in robustness.\n\n**Thanks for mentioning the minor typo, we will fix it in our later revision**\n\n"}]

Author rebuttal:
- We thank the reviewers for the constructive feedback and insightful questions. We are delighted that most reviewers like the well-motivated novel method of CVP and think it would interest the visual prompt community, the extensive experiments conducted across various OOD recognition at large scale, and the clear writing. We address two common questions here.


 **Q1. Motivation of convolutional visual prompt (CVP)**
- Our paper systematically studies how to be robust under major natural distribution shifts, such as style, blurring, or lighting changes. Those shifts are often locally structured, which motivates us to use the convolutional kernel to handle locally structured data. Besides, the traditional visual prompts (VP) often induce too many learnable parameters, which causes the overfitting problem. Therefore, we propose a new structured prompt, the CVP. Our extensive experimental results show that CVP is better than traditional VP, which is more efficient and lightweight.

**Q2. Performance comparison with other test-time adaptation baselines**
- The traditional TTA methods usually assume the model weights can be adjusted during the test time. However, we want to claim that our method is orthogonal to those traditional TTA methods, such as TENT, MEMO, and BN, where we consider a more challenging scenario and assume the model weights are frozen during inference time. We thus demonstrate how our method can be combined with traditional TTA. Combining them shows that our method complements theirs and has improved on several OOD benchmarks.

We address the questions in individual responses and include additional experiments and figures to support our responses. We will add all the details to our later revision.  The figures and tables can be found in the PDF file. 

Thank you again for your efficient handling of our submission.

Meta-review: The final rating for this paper were mixed with one accept, two weak accepts, one borderline reject and one reject. The reviewers had a thorough rebuttal/discussion with the authors that resulted in many questions being resolved and one reviewer (HEpT) raising their score to weak accept.  The key strengths of the work are the focus on an important problem with a paper that is clearly written an extensive experiments. There was disagreement among the reviewers about whether the proposed method which introduces a convolutional visual prompt for test time training has sufficient novelty to warrant acceptance. After considering all reviewer and author comments / discussion the concerns about limited novelty are overshadowed by the fact that this work introduces a simple and potentially useful approach for test time adaptation with extensive experiments in a well written form factor.
Decision: Accept (poster)