Efficient Open Set Single Image Test Time Adaptation of Vision Language Models
Authors: Manogna Sreenivas
Soma Biswas
Venue: Accepted by TMLR ()

Abstract: Adapting models to dynamic, real-world environments characterized by shifting data distributions and unseen test scenarios is a critical challenge in deep learning. In this paper, we consider a realistic and challenging Test-Time Adaptation setting, where a model must continuously adapt to test samples that arrive sequentially, one at a time, while distinguishing between known and unknown classes. Current Test-Time Adaptation methods operate under closed-set assumptions or batch processing, differing from the real-world open-set scenarios. We address this limitation by establishing a comprehensive benchmark for Open-set Single-image Test-Time Adaptation using Vision-Language Models. Furthermore, we propose ROSITA, a novel framework that leverages dynamically updated feature banks to identify reliable test samples and employs a contrastive learning objective to improve the separation between known and unknown classes. Our approach effectively adapts models to domain shifts for known classes while rejecting unfamiliar samples. Extensive experiments across diverse real-world benchmarks demonstrate that ROSITA sets a new state-of-the-art in open-set TTA, achieving both strong performance and computational efficiency for real-time deployment. The code is released at https://github.com/manogna-s/ROSITA.git.

Published in Transactions on Machine Learning Research (05/2025)
Efficient Open Set Single Image Test Time Adaptation of Vision Language Models
Manogna Sreenivas
manognas@iisc.ac.it
Indian Institute of Science, Bengaluru
Soma Biswas
somabisuas @risc. a.c.
Indion Institute of Science, Bengaluru
Reviewed OpenReview: https: //
https://openrevieau openreview. net/ pet/formm? id= 72YVobBEr!'
Abstract
Adapting models to dynamic, real-world enviromments characterized by shifting data dis- tributions and unseen test scenarios is a critical challenge in deep learning. In this paper, we consider a realistic and challenging Test-Time Adaptation setting, where a model must continuously adapt to test samples that arrive sequentially, one a time, while distinguishing between known and unknown classes. Current Test-Time Adaptation methods operate under closed-set assumptions
batch processing, differing from the real-world open-set scenar- ios. We address this limitation by establishing comprehensive benchmark for Open-sel Single-image Test- Time Adaptation using Languuge Models. Furthermore, we propose ROSITA, a novel framework that leverages dynamically updated feature banks to identify reliable test samples and employs a contrastive learning objective to improve the separation between known and unknown classes. Our approach effectively adapts models domain shifts for known classes while rejecting unfamiliar samples. Extensive experiments across diverse real-world benchmarks demonstrate that ROSITA sets open-set TTA, achieving both strong performance and computational elficiency for real-time deployment. The code is released at httpps:/giithub.cmm manommaasnnaaaaaa /ROSITA.gtt
state-of-the-art in
1 Introduction
Over the past decade, deep learning has revolutionized computer vision tasks such as image classification, object detection, and segmentation (Deng et al.) 2009; Ron et al., 2015 He et al.] 2017). However, these advancements are predominantly realized on the assumption that the training and Lest data follow the same distribution. In contrast, the real world is dynamic and ever-changing, making such assumptions often untenable. Distribution gaps between training and test data manifest in diverse forms HHendryeks & Dictterich 22019), including domain shifts and semantic shifts. Domain shifts could emerge from variations in lighting, weather, camera specifications, or geographical locations between the train and test datasets. Semantic shifts occur when model, initially trained on a specific set of classes, encounters previously unseen classes during testing. Hence, navigating deep learning models through these dynamic test environments is imperative.
Test-Time Adaptation (TTA) addresses this challenge by enabling models adapt during inference without access the training data (Wang et al. 2021 Schneider al. 2020 Döbler al. 2023) TTA is characterized by three defining features: (1) no access to source data during adaptation, (2) the absence of ground truth labels for test data, and (3) an online adaptation scenario where test samples encountered sequentially and accessible only once. These constraints reflect the dynamic and streaming nature of real-world applications. Existing TTA methods have predominantly focused on closed-set scenarios, assuming all test data belongs to known classes, hence falling short in real-world settings, where models are exposed to unseen categories beyond their training distribution. For example, an autonomous driving system trained to recognize urban vehicles like car, truck, motoreycle may incorrectly classify bicycle as a motoreycle when deployed in rural setting. In such scenarios, the model must not only adapt to domain shifts within known categories but also

---

Published in Transactions on Machine Learning Research (05/2025)
identify unfamiliar elements as unknown rather than incorrectly classifying them part of the known set. This highlights the critical need for Open-set adaptation in TTA. In addition, most TTA methods assume access to batches of test samples (Wang et al. 2021 Sreenivas et al., 2024), which are processed collectively for adaptation. However, this assumption is often impractical in real-time scenarios, where test samples arrive sequentially, one at a time, necessitating elficient Single image Test-Time Adaptation methods.
Large-scale Vision-Language Models (VLMs), such as CLIP (Radford et al. 2021 have demonstrated exceptional generalization capabilities across diverse domains, making them promising candidates for TTA. Recent works like TPT (Shu et al. 2022 and PromptAligu (/aamadh et al.) 2023 have explored prompt-tuning based adaptation of VLMs at the level of individual test samples, achieving improved zero-shot performance. TDA (Karmanov et al. 2024) employs a Training-fres Dynamic Adapter to adapt VLMs during test-time. However, these methods are restricted to closed-set scenarios and do not address the challenges of more realistic open-set scenarios. Conversely, open-set TTA methods (Li et al.; 2023 Lee et al., 2023) focus on adapting vision-only backbones (CNNs) trained on specific domains but require batch-wise processing of test. samples, making them unsuitable for scenarios where test samples arrive one at time. The combined challenges of open-set recognition and single-image adaptation remain largely unexplored.
To bridge these gaps, we establish benchmark for Open-set Single-image Test-Time Adaptation (OSTTA) using VLMs, addressing both open-set recognition and single-image adaptation. We refer to the classes of interest (say CIFAR-10) as desired classes and the rest as undesired classes (say CIEAR-100). To identify whether a test sample belongs to a desired or undesired class, we employ a Linear Discriminant Analysis (LDA) Fisher (1936): Li et al. (2023) based class identifier. Samples identified as belonging to the desired classes are then classified accordingly into one of the desired classes. We equip closed-set single image TTA methods with this class identifier to handle open-set scenarios.
We propose a novel framework termed ROSITA designed for OSTTA using VLMs. At the core of this framework is the ReDUCe loss, which effectively leverages Reliable samples to enhance the separability between Desired and Undesired classes through a Contrastive loss. Additionally, moving beyond existing prompt-tuning approaches, we analyze the optimal set of parameters for adapting VLMSs during test-time and identify that adapting LayerNorm parameters offers a lightweight yet effective solution for continuous adaptation. ROSITA dynamically updates these LayerNorm parameters using the RcDUCe loss, enabling it to adapt in open-sel environments by accurately identifying unseen classes as "unknown" while maintaining the performance of VLMs on known categories. Our contributions are summarized as follows:
To the best of our knowledge, we are the first to explore the capability of VLMs in addressing the challenging and realistic problem of Open-set Single-image Test- Time Adaptation (OSTTA), establishing a comprehensive benchmark for this setting.
Our framework, ROSITA, introduces the ReDUCe loss to enhance separability between desired and undesired class samples, enabling reliable recognition of desired samples under domain shifts while effectively rejecting unfamiliar ones saying "I don't know".
We conduct a systematic analysis of parameter selection for VLM adaptation during test-time and identify LayerNorm parameters as the optimal choice for lightweight, continuous adaptation of VLMs.
We demonstrate the effectiveness of ROSITA through extensive experiments across diverse domain adaptation benchmarks, simulating real-world lest environments with single-domain shifts, continuous and frequent domain shifts, and varying proportions of desired and undesired class samples.
2 O sen Single-image Time Adaptation
2.1 Problem Setup
Test stream. The model encounters a single test sample i, at time t, sampled from D. = DaUD, comprising of: (i) Desired class samples: Da = {a+: V. E C.), with domain shift and belonging to one of the Ca desired classes, for example, Ca = {car, bus,... motorcycle); (ii) Undesired class samples: D. = {ae yr C.), which have semantic shift (irrelevant classes) such that Ca O C. d.
2

---

Published in Transactions on Machine Learning Research (05/2025)
Goal. Given a test sample 2, arriving at time t, the goal is to first recognize if it belongs to desired class or not, constituting a binary classification task. Tf is identified as a desired class sample, subsequent [Cal-way classification is performed. Else, the prediction is "I don't know". In essence, the overall process can be viewed as a |Cal +I 1 way classification problem.
OSTTA scenarios. We simulate several test scenarios inspired by the real world to evaluate the effectiveness of our method. (1) Single domain: We extend the standard TTA scenario where the test samples come from unseen domain Da (say snow corruption of CIFAR-10C) by incorporating undesired samples D, (say CIFAR-100C), (2) Continuously changing domains: Here, Dr changes with time as (D) U D.) + (D)UD.) + (Dğ Du), where D2 is the 1th domain encountered. (3) Frequently changing domains: Here, we significantly reduce the number of samples per domain in continuous open-set TTA. The fewer the samples per domain, the more frequently the test domain changes, simulating very dynamic open-set test scenarios. (4) Varying sample ratio: The proportion of samples from C3 and C2 in the test stream is varied.
2.2 Benchmark for OSTTA using VLMs
Here, we describe our motivation for using VL.Ss for the OSTTA problem and further describe how we establish a benchmark for the same.
CNNs vs VLMs for OSTTA, Test-Time Adaptation (TTA) traditionally focuses on CNNs, which are vision-ouly backbones trained specific datasets. The goal is to adapt these CNNs to mitigate performance degradation when encountering unscen environments such as noisy or weather-affccted conditions. These models usually require specific retraining for each dataset or desired classes. On the other hand, Vision- Language Models (VLMs) like CLIP (Radford et al.) 2021 are pretrained on diverse image-text pairs from the web. These models demonstrate strong zero-shot generalization capabilities across diverse domains without any specific retraining. This makes VLMs promising candidate for TTA scenarios. However, defining unseen classes or domains in the context of VLMs is non-trivial due to their exposure to diverse visual data. Although CLIP performs well in zero-shot classification for clean datasets, its performance on corrupted or stylc-shifted datasets like ImageNet-C/R remains suboptimal (Shu et al.) 2022), making TTA still relevant. Moreover, CLIP can only classify an image by making a choice from the given set of desired classes. It lacks the ability to explicitly say "I don't know" when presented with a sample that does not belong to the set of desired classes, highlighting the need for open-set recognition. Noting these differences and advantages of VLMs over CNNs, we ask these questions: 1) How well con VLMs perform in open-set scenarios? 2) Can they be effectively adapted in a continuous manner? 3) How do we equip VLMS to handle domain shifts within desired classes while accurately rejecting unfamiliar samples? ? To address these rescarch questions, we est establish a new benchmark and propose a framework termed ROSITA using VLMs.
Classification using VLMs. We evaluate our approach using Vision-Language Models (VLMs) such as CLIP Radford et al. 2021 and MaPLe (Khattak et al. 2023) as backbones. CLIP consists of a Vision (Fv) and Text (Fr) encoder trained via contrastive learning on image-lexi pairs. The MaPLe backbone extends CLIP by incorporating multimodal prompts, enhancing its adaptability for downstream tasks. Given a test image 2, and set of desired classes C2 = (ci, ...,..................... cN ,N. text prompts. Each class name is prepended with the prompt "A "A photo of a", creating class-specific text inputs (Pr, a... These inputs are passed through the text encoder to generate text embeddings f. = Fr((P:; ci)) for each G E Ca. The resulting classifier consists of text embeddings [t,, t2,..., J. Class prediction for a sample Fe is made by identifying the text embedding t, with the highest similarity to the image feature fr extracted from the vision encoder.
we construct text-based classifiers using predefined
Baseline Methods. To establish a strong benchmark, we adapt several existing TTA methods designed for closed-set settings to the open-set Single Image TTA scenario. These include ZSEval (Radford et al. 2021 TPT (Shu et al. 2022) PAlign (Samadh et al.) 2023), TDA (Karmanov et al. 2024h and DPE (Zhang et al. 2024). Furthermore, we extend TPT and PAlign to support continuous model updates by adapting prompts, referring to these variants as TPT-C and P PAlign-C, respectively. We also adapt open-set TTA approaches originally designed for CNNs, such as (K+1)PC (Li et al. 2023) and UniEnt (Gao et al. 2024), to work with VLMs. These methods are described in detail in Appendix B For a fair comparison, all bascline methods are equipped with a simple and efficient class identification mechanism based on the LDA objective (Li et al. 2023) to handle the open-set nature of the test stream.
3

---

Published in Transactions on Machine Learning Research (05/2025)
Desired vs Undesired Class Identifier. In an open-set TTA setting, it is crucial for the model to distinguish between samples belonging to desired classes (Ca) and undesired classes (Ca) and appropriately reject samples from Cu. This problem can be viewed as binary classification problem between desired and undesired samples based on the score St (Equation 1 To achiove this, we equip all bascline mothods with a
parameter-free classifier based on Linear Discriminant Analysis (LDA) (Fisher 1936 Li el al. 2023). This
classifier uses the similarity score 8t, defined as the maximum cosine similarity between the image embodding fa of the test sample 2y and the text embeddings i of the desired classes Ca:
8t max sim tx)
(1)
Rather than relying on fixed threshold, which can be challenging to define in streaming TTA scenario, we dynamically determine an optimal threshold using a continuously updated score bank S. This bank stores the most recent |S| similarity scores, capturing the evolving distribution of the test stream. The optimal threshold 7 is computed using 1D LDA to minimize intra-class variance, as follows:
c arg mill,
1 [aal
Σ SESA
(s- pa)²
Σ (s- mw)² SES.
Su
where Sa {silss > 7, 8; E S) and S, {silss < T, ES represent the scores of samples classified desired and undesired, respectively, and pa and Yu are their means. Using this threshold, the test sample xy. is classified as:
desired
st 2 Ti
(3)
undesired if 8t < 7?
This simple yet effective approach equips the model handle open-set scenarios by explicitly rejecting undesired samples, thereby ensuring robust performance during adaptation.
We equip all the above described baseline methods with this LDA objective based Desired Undesired class identifier for a fair comparison in the Open-set Single Image TTA setting. In Appendix C.4 we demonstrate the superior performance of this method compared to naive confidence thresholding. With this comprchensive benchmark established, we now describe our proposed framework, ROSITA, which sets a new standard for Open-set single-image TTA.
3 Proposed ROSITA Framework
Given a single test sample 2, at time I, to ensure effective TTA, we first characterize the sample based on the class and the quality of samples: (1) Desired vs. Undesired Classes: These refer to the ground truth class groups, where desired classes belong to Ca and undesired classes belong to C.. (2) Reliable us. Unreliable Samples: A test sample Ia is considered reliable if its score st confidently places it in either the desired or undesired class distributions, estimated using LDA statistics (Ha,Hu). We leverage Reliable samples to differentiate Desired vs Undesired class samples through a Contrastive (ReDUCe) Loss for Open-set. Single-image Test-time Adaptation (Figure D.. For this, we categorize the test samples as follows:
1. Reliable Desired Class Sample
Ti < Hd < St.
2. Unreliable Desired Class Sample
: t < st S pd.
3. Reliable Undesired Class Sample
8t ph. < T
4. Unreliable Undesired Class Sample
Pm. S S T/
ReDUCe Loss. A contrastive objective typically needs positive and negative features, the goal being to maximize the similarity between a sample and its positives (could be augmentations (Chen el al. 2020) or nearest neighbours (Dwibedi et al. 2021)), ), while minimizing its similarity with the negatives. Such objectives (Chen et al. 2020 He et al. 2020 Khosla et al., 22020] Dwibedi et al., (202)) have been extensively used self-supervised manner. While self-supervised learning assumes access to abundant data in an offline manner, giving the freedom to carefully choose positives and negatives, this
learn good image representations
4

---

Published in Transactions on Machine Learning Research (05/2025)
Ca or C
ReDUCe Loss
car bike
Reliable sample C 4 ? Md
Text Encoder
photo of
f2
Kc
Rd + L
bicycle
Mu
KNN +
Mu
Vision Encoder
M,
KNN
mix
Score bank
Reliable sample from C, < Myy
X
Pseudo label
L h 0 (*)
8
Test Data Stream
The test stream with samples from Ca and C2 arrive one at a time. An input image Hę is recognized as sample from Ca and Cu through an LDA based class identifier. Further, if a test sample is reliable, the respective feature banks are updated and the proposed ReDUCe loss is optimized to update the LayerNorm parameters of the Vision Encoder.
Figure 1: ROSITA framework:
problem is set in an online scenario. Here, the test samples arrive one at time and are accessible only at that instant. This challenging setting makes it non-trivial to use objectives like (Dwibedi et al. 2021). To circumvent this issue of lack of abundant test data, we propose to store two dynamically updated feature banks Ma and Ma of sizes Na and Nu, to store the features of reliable samples from Ca and C, respectively. We propose ReDUCc loss contrast reliable sample from Ca by choosing its positives and nogatives as the K nearest neighbours from Ma and M,, respectively and vice versa for a reliable sample from C.. The bulfer size for Ma is set as |Cal x K, where |Cal is the number of desired classes and K is the number of neighbours retricved. The feature banks Ma or Ma are updated with feature fü if it is detected as a reliable sample from Ca or C.,
We fetch the K nearest neighbours reliable test sample It from each feature bank as follows.
Qd kNN(fe:Ma: Qu = kNN(f(;
(4)
Case 1: Reliable sample from Ca. If test sample is identified as reliable sample from Ca, we use reliable pseudo-label loss the sample I, and its augmentation i, as follows:
LR = CCE(TI, 6.) + Cce(#, te); de argmax, sim (an(ff.tt) fq)
(5)
where sim represents cosine similarity. Further, we also propose to use a contrastive objective to enhance the clustering desired class samples while pushing them apart from the undesired class samples.
As we aim to correctly classify the desired class samples, we select positives 2 from Qa if its prediction yt matches with d-- The features Qu consisting of its kNN from Ma act its negatives. The following is the ReDUCe loss for reliable sample from Ca:
exp (sim (f1,2+) /T) Σ exp(sim(fee22))1 //T) EQ.
Lo = K+
Σ 2 STEQa
10y+ = log
(6)
where n+ = =+== 1 = -t). 0t), is the number of neighbours positively matched with ÿt-
Case 2: Reliable sample from C.. If test sample identified as a reliable sample from Cn, we use the following contrastive objective by selecting positives = from Qu and negatives 2 from Qa:
Cu = # EE
1 K
Σ
log
exp (sim 2T) exp(sim (f, x )/T)
(7)
EQu
5

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Results with Image.Net-C/B as desired class data Da. MNIST and SVHN for Du.
IN-C/MNIST
IN-C/SVHN
IN-R/MNIST
IN-R/SVHN
Method
AUC 1
1
AUC
AUC1
AUC:
1
ZS-Eval
93.39
55.52
41.43
85.89
72.91
40.83
91.27
91.09
71.50
90.43
75.04
71.00
TPT
93.12
58.01
42.21
85.43
74.47
40.95
91.25
91.23
71.98
90.43
74.98
72.36
TPT-C
56.37
99.12
6.19
11.38
100.00
7:24
82.81
85.79
68.25
80.94
80.03
69.18
(K+1) PC
95.76
10.43
12.95
87.75
26.23
38.50
97.
11.78
81.51
97.55
11.17
80.39
Unilint
94.19
46.98
41.53
87.56
67.03
41.10
91.64
86.67
71.73
90.86
71.53
71.96
TDA
90.54
76.23
43.66
86.76
75.45
43.07
91.79
87.85
71.56
90.67
75.41
71.48
DPR
87.92
81.09
73.30
70.64
ROSITA
99.52
4.06
48.53
98.34
10.21
46.82
99-44
4.29
83.58
98.62
9.08
80.75
+-6133
15 141.46
7,10
H-12455
62.70
5.44
+8..77
86.80
122003
+ +-8.19
+-65.96
+99099
ZS-Eval
81.49
92.95
41.70
83.26
71.15
42.77
90.15
83.54
74.42
92.74
65.70
75.71
TPT
81.38
93.17
39.92
83.18
71.52
40.93
90.14
83.58
74.00
92.74
65.68
75.23
TPT-C
83.25
42.81
83.18
70.60
42.86
90.35
81.49
74.73
92.79
75.59
MAPLE
PAlign
81.38
93.17
41.32
83.18
71.52
42.30
90.14
83.58
74.66
92.74
65.68
75.93
PAlign-C
71:22
86,32
27.14
32.17
94.32
15.44
92.21
59.70
75.23
933.51
54.59
75.67
(K+1)PC
98.38
3.35
69
77.17
39.74
38. 10
3.16
84.23
95.11
13.77
80.16
UnilEnt
81.53
93.45
41.50
83.41
70.84
42.78
90.14
83.49
74.48
90.14
$3.49
74.48
TDA
76.79
99.02
42.98
82.46
91.75
44.63
90.43
86.56
73.66
92.92
64.63
74.16
DPE
73.97
99.39
11.39
80.06
87.10
41.05
90.11
78.77
72.67
93.48
55.74
76.71
ROSITA
99.56
1.66
51.30
98.68
5.09
99.39
2.95
84.70
97.85
12.98
83.07
+18.07
-91.29
-9.60
15.42
+ 66.06
+7.90
+9.24
+-80.59
+11.28
+.111
4-52.72
+7.36
The LayerNorm parameters of the Vision Encoder are updated to minimize the following test-time objective to adapt the model one sample at time in an online manner:
{ [RR + CD
if
3t >
C RRDUUC =
{ Luu
(8)
if 8t Pu.
This objective improves the proximity between the test sample and its positives, suitably chosen based on its score while also pushing apart the test sample and its negatives. This collectively encourages the model to adapt such that each of the desired classes and undesired classes are clustered and farther apart from each other, improving the overall classification performance of Ca and C.. We now perform Gradient Analysis on the loss function and theoretically justify how the proposed ReDUCe loss helps in enhancing the discriminability between desired and undesired class samples.
3.1 Gradient Analysis of the proposed REDUCE Loss
The key to understanding the behavior of the contrastive loss is to analyze its gradient. The softmax term in the denominator encourages ft to have lower similarity with negative samples, and the numerator encourages S to have higher similarity with positive samples. We compute the gradient of the loss components L.D and Le of the ReDUCe loss with respect to ft (Appendix A).
OLD Of
1
Σ
1
01)
z Σ 2(=))=- EQu
K+ ztEQu
(9)
dLy
1
x Σ p2=))2² EQu
x
Qf.
K
EtEQu
where p (i the softmax probability of the negative samples defined as
exp (sim (fe, 2 /TT p(>)== Eveg- exp (sim (fr, x²) /T)
(10)
where Q is Qu for LD and Qu for Ly. The gradient of these contrastive loss formulations drives the following behavior in this context:

---

Published in Transactions on Machine Learning Research (05/2025)
1. Attraction to positive neighbors. In the gradient of LD; the first term pulls the test feature fi towards its positives 2+ E Qd, representing ing the attraction force that encourages samples from desired classes to form |Cal tight clusters as the positives are chosen such that Dr = =yt Similarly, in the gradient of Lv, the first term pulls ft towards its positives 2+ E Qu, encouraging all samples from Cv to cluster together.
2. Repulsion from negative neighbors. The second term p (=-)== in the gradient pushes the test feature f away from its negatives x E EQ (Q- is Qu for LD and Qa for Lv). The strength of the repulsion is controlled by the softmax probability p(²"), where more similar negatives exert stronger repulsive force on fi increasing the separation between samples from Ca and C.. As the negatives selected are its K nearest neighbours of the opposite type, they are, in fact, hard negatives. Further, the contrastive objective inherently models the degree of hardness through the means of this probability p(=-). The closer the hard negative, the stronger the repulsion force.
We now present our analysis on the paramcter choices for continuous adaptation of VLMs.
4 Analysis on parameters for Continuous Adaptation of VLMs
Dost-Timeaar adaptation methods using CNNs (Wang et al. 2021 Schneider et al. 2020 Liang et al. 2020 Chen
et al. 2022) successfully leverage test domain data arriving in an online manner (in batches) to continuously update the model. In this work, we study TTA of VLMs like CLIP, which has only been explored very recently (Shu et al.) 2022 Karmanov et al., 2024 Zhang et al. 2024 by adapting prompts independently for cach image. While these methods show promisc for on-the-fly adaptation in a zero-shot framework, it is not clear whether they can leverage the online data stream to continuously update the model parameters. Based on the evidence in prior TTA works (Wang et al. 2021 Chen et al.) 2022). we analyze two aspects of VLMs for the TTA task: (1) Here, we question if VLMs can be continuously adapted in a similar manner, but using only a single test image at a time; (ii) If so, are prompts (Shu et al. 2022) the best parameters to continuously update?
Experiment. We choose six different parameter groups: (1) Prompts, (2) LayerNorm parameters (Zhao ct al. 2023), , (3) Full network, (4) First Attention Block of ViT, (5) Last Attention Block of ViT, (6) 7) ToRA Adapters (Imam et al. 2024). We perform single image TTA in a closed set scenario on CIFAR-10C, by continuously adapting each of these parameter groups of CLIP, using reliable
Prompts +
entropy loss, LTTA= = 1(s; > (z)) which is commonly used in several TTA methods (Wang et al. 2021
Niu et al. 2022) and VLM based prompt tuning methods like TPT, PAlign. Here, H₁ and st refer to the test
sample and its confidence, respectively. is the confidence threshold used to select reliable samples Niu
et al.
2022 for the model update, which we set to 0.7 in this analysis.
Observations. We find that contimuous model adap- tation can indoed improve VLMs performance based on our empirical analysis (Figure 2) (1) Using a high learning rate of 10-2 for any parameter group results in a severe drop in accuracy compared to the zero- shot performance of CLIP in this extreme setting of continuous single image model update. (2) The other extreme of low learning rate of 10-6 performs at par with ZSEval for all parameter groups, suggesting the model has not sufficiently changed. (3) Updating the Full Network results in an accuracy of about 10% across all learning rates, suggesting that giving the highest flexibility can cause the model to lose the inherent generalization ability of the VLM. (4) We also explore updating carly attention layers and LoRA adapters. We observe these can be polential parameter choices on carefully choosing the learning rate (Appendix However, LoRA introduces additional parameters, scaling with the adapter rank. Unlike training-time scenarios where LoRA weights can be mergod post-finctuning, continuous TTA cannot benefit from such merging, thereby increasing deployment complexity. Moreover, performance varies considerably with the choice of rank (Table 15). adding another hyper-parameter to be
100
+ 2S-Eval
Prompts
80
+- LN
++ Full
First block
Acmuscy
60
+. Lost block
Prompta-+
40
LORA
20
10-4
10-5
10-4
10
10
learning
Figure 2: Accuracy on fine-tuning different parameter groups for single image TTA.
7

---

Published in Transactions on Machine Learning Research (05/2025)
tuned. (5) We find that LayerNorm tuning remains stable across optimizers, requires less hyper-parannn parameter tuning, adds no additional model parameters, making it a more reliable and lightweight choice for our setting.
Adapting Image encoder vs Text classifiers: Most existing TTA approaches (Schneider et al. 2020; Wang et al. 2021 Chen et al. 2022) focus on adjusting image representations for domain shifts during test-time while keeping the classifiers fixed. This strategy helps retain class discriminative information. Tn contrast, in TPT and PAlign, the text- eext-based classifiers that depend on learnable prompts are updated based single images. While this does not impact zero-shot evaluation as the model weights are reset after each image, it can be detrimental during continuous updates.
Based on this analysis, we freeze the text-based classifiers and modify only the image representations using LayerNorm affine parameters. The rationale behind this approach is that text represenlations can be inherently more robust across domains. Text embeddings, often derived from a wide range of linguistic contexts, capture semantic mcanings that are less susceptible to variations in visual data. Therefore, adapting the image encoder allows for more effective handling of domain shifts while retaining the class-level discriminative information from the text modality. This ensures that the model can be updated continuously without the need for resets, ultimately enhancing its performance in dynamic, real open-set environments.
5 Experiments
Datasets. We experiment with diverse set of datasets to choose desired class data Da and un- desired class data Du. For Da, we use CIFAR- 10C (Hendrycks & Dietterich 2019), CIFAR-
Table 2: ACCни on VisDA dataset and Clipart, Paint- ing, Sketch domains from DomainNet as Da and MNIST as Du.
100C (Hendrycks & Dietterich| 2019). ImageNet-
Method
VisDA
Clipart
Painting
Sketch
C. (Hendrycks & Dietterich, 2019), CCC Press et al.
ZSEval
78.28
50.22
47.81
48.59
(2023) from the corruption category and ImageNet-
TPT
78.42
57.71
49.73
54.67
R (Heudryeks et al., 2021), VisDA (Peng et al.]
TPT-C
75.35
57.57
49.31
54.41
2017) and the Clipart, Painting, Sketch domains from DomainNet (Peng et al. 2019) as style transfer datasets. We introduce samples from MNIST [LCun et al.] 1998), SVHN (Netzer et al. p011p, CIFAR- 10/ 10/100C (Hendrycks & Dietterich 2019) and Tiny- ImageNet {Le & Yang, 2015) datasets as D, in the test. stream. We describe the datasets in detail in Appendix
(K+1)PC
90.35
71.21
70.61
67.21
UniEnt
78.09
57.88
49.75
54.76
TDA
76.85
61.04
51.20
55.26
DPE
53.67
54.52
47.91
32.18
90.64
71.40
70.89
67.35
ROSITA
+12.36
+21.18
+ 23.08
+ +18.7
B.3
Implementation Details. We use CLIP and MaPLe backbones with VIT-B16 architecture. For ROSITA, we use SGD optimizer with a learning rate of 0.001 to update the LayerNorm parameters of the Vision encoder. We set size of the score bank to 512, number of neighbours K to 5. The size of fcature bank Ma is set K x Ca and that of M, to 64. Implementation details for all the baseline methods are presented in Appendix B.4 We equip all methods with the same Ca Cn class identifier described in Section 2.2 All experiments are done single NVIDIA A6000 GPU.
Evaluation Metrics. We employ standard metrics, namely Area Under the Receiver Operating Charac- teristic Curve (AUROC) and False Positive Rate at a True Positive Rate of 95% (FPR95), from the OOD detection literature (Lee et al. 2023 Li et al., 2023 Wang et al. 2023). Additionally, we compute the classification accuracy for desired class samples ( (Accp) and the binary classification accuracy for correctly recognizing samples from C. (Accy) as defined below. To gauge the overall performance, we compute AсChм (HM), representing the harmonic mean of ACeD and Accu, which serves comprchensive metric capturing the trade-off between Accp and Accy. Here, we summarily report AUROC (AUC), FPR95 (FPR) and AcCHa (HM) for all the datasets.
Accp E(eEWNED, 1 (y ū.) (y) E Ca). E(D.WEDD 1 (y Ca)
Accu E(EwnEE, 1 1 C.) 1 (1s E Cu)
Eimmee 1 (y C.))
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table Results with CIFAR-10C/100C as desired class data Da and four other datasets as Du.
MNIST
SVHN
Viny-InageNe
CHFAR- 100I--000//1-CC
Method
4
HM
AUC
4
1
HM
ZS-Eval
91.91
85.04
75.57
82.57
68.8
TPT
91.89
85.35
75.81
89.93
61.11
71.36
91.31
27.23
75.17
82.57
68.06
69.17
TPT-C
81.64
67.53
74.86
58.48
71.72
48.26
74.08
61.45
49.88
61.45
94.30
46. 10
CLIP
(K+1)PP
98.05
12.50
83.27
50.33
70.10
87.09
73.96
62.55
56.46
UniEnt
91 .998
85.2
75.62
80.97
04.38
74.18
91.40
26.96
74.73
82.59
68.14
66.98
TDA
92.91
71.11
77.06
92.02
52.08
76.61
91. 08
25.37
75.91
83.54
66.06
70.13
CIFAR- -OC
DPE
46.97
90.10
27.60
84.15
85.24
68.52
89.92
31.30
69.90
79.18
75.06
62.34
99.10
7.63
84.17
94.79
32.59
78.80
96,43
12.10
80.06
82.99
62.89
69.56
ROSITA
+7.19
+77.41
+8860
+ 1-4.86
+ +-1.66
+-4.72
++5.10
++1...9
+5.44
+0.42
+5.003
+-0.6
ZS-Eval
98.18
3.77
83.63
98.34
7.86
83.57
90.86
27.54
76.01
86. 86.14
52.08
71.76
TPT
98.15
5.67
62.73
90:86
75.46
70.94
TPT-C
98.56
3.74
83.51
98.32
8.18
83.17
91.18
26.93
76.31
86.50
50.56
71.07
MAPLE
PAlign
98.15
5.67
82.24
98.34
7.90
83.51
90.86
27.60
75.98
86.15
52.18
71.52
PAlign-C
(K+ (K+1)B I)C
98.34
9.63
86.52
71.01
78.78
68.70
71.20
85.81
68.29
62.35
88.44
61.89
Unilfut
98.17
5.49
82.64
98.35
7.85
83.65
900.00
27
76.08
86.16
51.91
71.72
TDA
98.12
4.13
81.97
76.81
86.72
72.61
DPE
83.82
92.73
55,52
97.42
12.95
79.41
89.10
31.13
74.32
73.57
73.67
53.64
ROSITA
99.34
5.22
87.63
97.80
13.15
84.17
91.67
25.31
77.67
86.82
50.33
73.15
4-0.86
-1.45
4-4.00
+-0.5
-5.29
+-0.60
4-0.81
+-2.28
4-1.63
4-0.68
4-1.75
4-1.99
ZS-Eval
77.78
99.93
48.39
64.70
98.68
15.85
67.31
T3.89
15.80
63.
93.25
41.01
TPT
77.76
90.94
48.33
64.71
98.63
415.8
67.26
73.82
45.93
63.26
93.20
44.02
TPT-C
51.57
100.00
27.04
9.40
99.98
5.74
59.71
79.70 79.76
18.11
55.86
86.35
13.64
CLIP
(KC+T)PC
96.89
09.72
43.73
41.84
31.83
54.02
93.93
32.00
Unitint
77.94
99.93
48.32
64.78
98.61
45.84
67.40
73.77
45.88
63,
93.18
44.04
TDA
80.33
90.57
46.52
71.77
96.11
46.01
70.70
69.63
47.52
66.07
91.90
45.79
DPE
42.54
CIFAR-100M 1000
ROSITA
96.07
83.55
68.54
+18.29
4-80.65
+-8.95
417.39
+344.04
+-2.32
+11..24 1.6.4
4-23.13
+-10.08
4-5.26
+-3.5
+-3.94
78-Exal
87.43
64.19
54.97
92.98
40.51
56.42
68.80
74.35
48.24
66.93
87.94
46.06
TPT
87.42
64.09
53.09
92.97
40.44
54.37
68.80
74.20
46.97
66.93
87.95
44.38
TPT-C
87.65
63.08
55.14
93.09
40.30
56.31
68.85
74.71
48.53
66.97
87.94
46.30
MAPLE
PAlign
87.42
64.11
53.98
92.97
40.48
55.37
68.80
74.23
47.68
66.93
87.93
45.16
PAlign-C
88.25
37.31
55.69
93:45
39.39
57.39
08.76
78.12
18.15
66.82
87.80
47.01
[K++1)PC
96.19
9.42
62.97
65.73
78.63
32.66
42.91
99.95
27.52
53.488
91.26
31.70
UnlEnt
87.40
64.02
54.86
92.99
40.36
56.42
68. 8-1
74.26
48.41
66. 66.93
87.90
46.09
52:24
95.04
30.76
59.51
72.05
71.83
49.19
49.06
DPE 39.05
98.88
33.66
84.29
76.13
52.20
03.74
82.75
45.74
65.61
90.67
46.36
97.04
11.01
62.06
96.26
20.99
59.25
70.37
77.00
48.68
69.57
83.61
48.80
ROSITA
+-9.8
+ +53.18
+-7.09
+3.28
+19.52
+-2.83
+1.57
-2.85
+-0.44
+-2.64
+-4.33
+2.74
6 Research Questions
1) How does ROSITA perform in comparison with prior methods OSTTA setting?
We observe, from Table ! 3 that TPT and PAlign perform similar ZSEval in most datasets, as the prompts reset after every single image update. On continuously updating prompts in TPT-C and PAlign-C, we observe a reduction in HM compared to 7S-Fval. The effect is more severe with CLIP when compared to MaPLe, as only the text prompts are updated keeping the vision encoder fixed (as also observed in Section A. (K-1)PC and UniEnt, where LayerNorm tuning is done, perform better than prompt tuning methods. However, ROSITA, being equipped with a carefully designed objective to better discriminate between samples from Ca and C, samples (Figure 3), results in overall better metrics general.
2) How does ROSITA perform in different real-world inspired OSTTA scenarios?
(a) Continuously changing domains: We sequentially present 15 corruptions from CIFAR-10C, which form the domain Da, alongside samples from four other datascts D.,- We also experiment with Contimuously Changing Corruptions (CCC) (Press et al. 2023) benchmark where gradual domain changes are synthesized across 15 corruptions of ImageNet-C as Da and MNIST as D, and report the detailed results in Appendix D.1 (b) Frequently changing domains: To further simulate more dynamic test environments, for CIFAR- 10C/MNIST, we reduce the number of samples per corruption 100, 250, 500, and 1000 in the continuously changing domain open-set TTA scenario. Reducing the sample count per corruption causes more frequent.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Table 5: Performance in different Open-set TTA scenarios.
(a) Continuously changing domains
(b) Frequently changing domains
(e) Varying ratio of Ca/Cu
Method
CIFAR- -0C
CCC
No. of samples per corruption
Ratio
SVHN
MNIST
Tiny
C-100C
MNIST
100
200
500
1000
0.2
0.4
0.6
0.8
ZSExal
64.33
64.04
66.50
58.49
31.01
61.41
61.87
61.42
63.30
75.56
75.59
75.57
75.56
TPT
64.26
64.03
66.50
58.47
31.04
61.33
62.32
61.59
63.24
75.67
76.75
75.81
75.83
TPT-C
33.05
46.44
59.38
37.24
13.56
60.62
61.30
57.16
34.88
72.70
74.31
74.79
75.16
(K+1)PC
65.13
62.52
66.93
57.46
33.84
60.90
60.76
61.40
63.26
62.31
68.85
81.70
82.90
TDA
66.02
66.44
67. 77.64
59.44
33.87
60.17
61.43
63.22
64.82
72.45
75.04
77.54
77.91
DPE
23.36
50.12
58.96
35.56
31.16
47.48
46.22
39.83
46.52
65.67
66.12
56.38
29.98
ROSITA
66.86
65.26
68.89
59.16
34.84
61.64
66.82
67.97
73.24
82.96
83.97
84.51
8.1.37
domain changes, increasing the challenge for adaptation. (c) Varying ratio of samples belonging to classes Ca vs Cy: We simulate real-world scenarios using the CIFAR-1CC/MNIST dataset by varying the ratio of samples from the known classes Ca versus unknown classes C2 in the test stream by varying this ratio as 0.2, 0.4, 0.6, and 0.8. From results in Table 5 we observe that ROSITA demonstrates consistent. superiority across all three open-set TTA scenarios, showcasing its capability to adapt effectively to both continuously and frequently changing domains, well as varying class distributions.
3) What is the importance of each loss component proposed in ROSITA?
From Table 4 we observe that only using Lan or LD improves the metrics for CIFAR-10C dataset. For ImageNet-R (IN-R) as Da. using Re or LD is observed to increase FPR and decrease HM. IN-R. has 200 classes making it a more challenging and confusing task compared to CIFAR-10C 1OC. This de- crease in performance for IN-R can be attributed to the misclassificatio ion of some samples from C, as re- liable desired class samples, increasing the confusion between Ca and C, classes. Using Ly significantly reduces the confusion between samples from Cd and Cu, shown by the significant drop in FPR compared to 7SEval. The contrastive objectives LD and Lu to scparate the two types of samples, in conjunction with Lnte which aids to improve the |Cal-way classification of desired class samples, gives the overall best results.
Table 4: Ablation study on loss components.
CIFAR- DEAA--OC/MMIIST
IN-R/MNIST
Cuv
Co
by
AUC
FPR
HM f
AUC 1
FPR 4
HM
X 91.91
85.04
75.57
91.27
91.09
71.5
95.29
30.82
80.97
81.07
99.02
61.32
95.23
28.91
79.71
87-73
94.67
67.28
X
98.61
12.73
79.84
99.39
4.81
$0.82
V
96.23
22.73
79.24
76.78
99.22
62.54
X
98.69
12.06
82.98
99.34
4.67
82.98
V
99.27
4.15
80.69
99.48
4.40
81.92
V
99.10
7.63
84.17
99.44
4.29
83.53
4) What is the role of using reliable samples for OSTTA in ROSITA?
To understand the role of selecting reliable samples for TTA, we do a simple experiment where we only use the threshold Tr to distinguish between Ca
Table 6: Noed for Reliable samples.
Thresholds
Da: MNIST IN-C
Ta/T/Taa
C-10C
C-100C
IN-R
VisDA
and Cu samples. For all the samples with 8 > To identified to belong to Ca, we perform TTA using
T////T
84.99
55.16
14.05
83.28
91.24
Ln + LD (Equation 6). Similarly, we use Lu ( Equation 7) for all samples identified to belong to Cu based on the criterion s, < Tr. From the results in Table 6. we see that, for CIFAR-10C and VisDA, this case performs slightly better than our caso(last row Table where TTA is performed only on reliable samples. CIFAR-10C and VisDA dataset have 10 and 12 classes of interest respectively. The zero shot performance of these datasets being good, the class confusion is less, using all samples for TTA can be helpful. On the other hand, the classification in CIFAR-1008 ImageNet-C and ImageNet-R. is harder, due the inherent confusion arising due to the large number of classes. Using non reliable test samples, with scores in the range Pa < 8t < μa can adversely affect the adaptation process. Hence, using only reliable samples for TTA performs better for these datasets as seen in Table 6). In a real world test-time adaptation scenario, where we have no prior information about the difficulty of the classification task, terms of severity of domain shift and class confusion, is desirable to only use reliable samples for model updates.
Ha/t//ua
84.17
57.34
48.53
83.53
90.64
10

---

Published in Transactions on Machine Learning Research (05/2025)
3.00
2,000
2100
1.30
(a) ZSEval
(b) ROSITA
(c) scores with time
[d] Accuracy with time
Figure 3: Histograms of the scores st for ZS-Eval (a) and ROSITA (b) on CIFAR-ICC/MNIN dataset. (c)
Change in scores for Ca and C. class samples, the best threshold Tı; (d) Accuracy metrics on samples seen until time t. Samples from Ca and C. separate better and the accuracy metrics improve with time.
5) How do the scores and the performance of ROSITA vary with time?
We plot the scores 8, of samples from Ca and C₁ over time, along with the threshold in Figure 3c using ROSITA. Initially, the scores of Ca and C, overlap significantly (f < 2500), leading to unstable performance as shown Figure 3d During this phase, the threshold T tends to classify most Cn samples correctly, resulting in high Accy but low Acep, as many desired class samples are incorrectly rejected. However, as the ReDUCe loss progressively improves class separability, т adapts to the evolving score distribution, enhancing discrimination between Ca and Cu. This refinement stabilizes the model's performance, yielding steady improvements in Acep and Асснм for > 2500. The instability observed for t < 1500 is attributed to the initial learning process and the small sample size, as accuracy is measured on the cumulative number of samples seen up to time t, which is exactly t in single image TTA.
6) How does ROSITA fare in terms of memory required?
Prompt tuning methods like TPT, PAlign do not require any memory buffer. TDA requires memory buffer of size (|Ca) x (3+2)) F to store 3 features per desired class in the positive cache and 2 features per class in the negative cache. DPE requires a memory buffer of size (|Cal x 3) x F to store features per desired class. ROSITA requires a memory buffer of size (|Ca) K + |Mal)x for the two feature banks.
Table 7: Memory overhead ROSITA.
Dataset
[Cal
No. of features
Memory (in MB)
CIFAR-10C
10
5x10+64
0.758
VisDA
12
5x12464
0.778
CIFAR-100C
100
5x100+64
1.679
ImageNet-R
200
5x200+61
2.703
ImageNet-C
1000
5x1000+64
10.89
For ViT-B16 (F 512) model with ImageNet-C C (|Cal 1000), the required memory buffer size is 5 1000 x 512 + 64 x 512 (10.89MB) from Table 7
The memory to store these fealures and compulation required to compute feature similarity is as lightweight as performing a forward pass through a simple linear layer, demonstrating the memory and computational efficiency of ROSITA for real time applications.
7) How does ROSITA fare in terms of the GPU memory required and inference time?
The GPU memory and time taken (eece)mage) for prompt tuning methods TPT scales with the mum- ber of classes, as more memory is required to store the intermediate activations and gradients to back- ward pass through the text encoder. On the other hand, ROSITA requires two forward passes and one backward pass through the vision encoder for re- liable test samples. Figure 4 compares the GPU memory and time complexity of ZS-Eval, TPT, and ROSITA representative of training- free methods (ZS- Eval, TDA), prompt-tuning (TPT/-C, PAlign/-C), and LayerNormm-tuingg(( (K+1)PPC UniEnt, ROSITA) based methods in Figure 4 For e.g., for ImageNet-C datasct with 1000 classes, ZSEval, TPT and ROSITA Figure 4: Complexity Analysis of different methods.
ZSEval
TPT
ROSITA
IN-C
IN-0
IN-R
18-R
10000
1:000
VinnA
V&DA
0.2
GPL Momory
(imun flans)
require 5.71 GB, 23.24 GB and 5.73 GB GPU memory perform a single image based model update. Thus, ROSITA achieves computational efficiency comparable to training-free hodh while being far more efficient.
11

---

Published in Transactions on Machine Learning Research (05/2025)
than prompt-tuning approaches. Despite its minimal computational overhead, ROSITA offers substantial performance gains, providing balanced trade-off between efficiency and effectiveness for OSTTA scenarios.
8) What are the key factors distinguishing ROSITA from prior works?
1. Enhunced use of LDA Statislies to identify reliable somples: Apart from the threshold T,, ROSITA leverages the score statistics pa and Yu provided by the LDA class identifier, combined with the novel ReDUCe loss function, adapt the model. This synergy enhances the discriminability between desired (Ca) and undesired (Ca) class samples, offering a clear advantage over basclines that use the same LDA identifier but fail to exploit this additional information (Figure 3
2. Bridging CNN and VLM-Based TTA insights: ROSITA integrates key insights from CNN-based TTA
methods such as normalization layer updates with vision-language models (VLMs) (Section 4.. While simple in hindsight, this baseline was overlooked in prior VLMMbased TTA works (Shu et al., 2022 Karmanov et al.
2024; Zhang et al.] 2024). In this work, we attempt to highlight how these learnings can translate effectively to VLMs, underscoring their utility as foundational approach for TTA.
3. Holistic design for Open-set TTA:: ROSITA introduces the ReDUCe loss to distinctly separate desired (Ca) and undesired (C,.) class samples using compact feature banks. Although it is inspired by contrastive learning frameworks (Chen et al. 2020 2022), it is specifically designed for open-set TTA: (i) Reliable samples from C. use nearest Cv samples as negatives, and vice versa (ii) Unlike the Ca+l-way classification in (Li et al.) 2023), ROSITA forces C2 features to form distinct clusters and pushes C, features away. (iii) The feature banks are populated only with reliable samples, ensuring robust updates during adaptation. This approach specifically mitigates the significant overlap of scores 8, between Ca and C₂ in vision-language models, hence reducing misclassification and boosting discriminability.
9) What are the limitations of ROSITA which can be addressed in future?
Although we follow the datasct choices in (K+1)PPC Li ct al. (2023), the first work on open-sct TTA, we acknowledge that using MNIST as undesired class samples may seem unrealistic for natural image scenarios. However, even in this seemingly simple open-set scenario, our observations (Tables 2 3
show that CLIP struggles to reject MNIST samples, with high FPR for most prior methods and significant overlap in similarity scores between desired and undesired classes (Figure 3a We have also conducted experiments using CIFAR- 10C as desired classes and CIFAR-100C as undesired ones, which consist of corrupted but semantically similar images. While gains are less pronounced than in simpler settings like MNIST, our findings underscore the core challenge of enabling VLMs to reliably say "I don't know.'. Additionally, ROSITA was not benchmarkod in standard TTA setting where there are no undesired class samples in the test stream. Future work could explore a unified TTA framework that adapts effectively regardless of the nature of test-time samples.
Appendix. We present more detailed experimental analysis in the Appendix: bars, Analysis of parameter K, of different Co vs C. Class identifiers for Open-sct TTA, C.5 Extensive analysis on parameter choice for innuu adaptation of VLMs, D Additional experiments demonstrating the generalizability of ROSITA to more datasets, scenarios and VLM backbones, E Failure case analysis,
C.I Analysis on error
Detailed analysis of ReDUCe Loss components,
Comparison
F Broader impact concerns.
7 Conclusion
In this work, we address the challenging problem of Open-set Single-image Test-Time Adaptation (OSTTA), where models must adapt continuously to shifting data distributions and distinguish between known and unknown classes, all while processing test samples one at a time. To advance research in this area, establish comprehensive benchmark for OSTTA using Vision-Languag Models (VLMs), bridging the gap between open-set recognition and sequential adaptation in dynamic environments. We propose ROSITA, novel framework specifically designed for OSTTA, overcoming the limitations of prior methods that assume closed-set conditions or batch-wise Lest processing. ROSITA leverages two dynamically updated feature banks to differentiate between desired and undesired class samples. At its core, the proposed ReDUCe loss facilitates effective model adaptation by leveraging reliable samples while mitigating the negative influence of undesired class samples. Extensive experiments across diverse domain adaptation benchmarks demonstrate that ROSITA consistently outperforms prior methods, achieving good accuracy with computational efficiency.
12

---

Published in Transactions on Machine Learning Research (05/2025)
References
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR, 2022.
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICML, 2020.
Mehdi Cherti, Romain Beaumont, Ross Wighlman, Mitchell Wortsman, Gabriel Ilbarco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive Snguage-image learning. In CVPR, 2023.
J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. Imagenct: database. In CVPR, 2009.
large-scale hicrarchical image
Mario Döbler, Robert A Marsden, and Bin Yang. Robust mean teacher for continual and gradual test-time adaptation. In CVPR, 2023.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighbor contrastive learning of visual reprosentations. In ICCV, 2021.
Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, 7(2):179 188, 1936.
Zhengqing Gao, Yao Zhang, and Cheng-Lin Liu. Unified entropy optimization for open-set test-time adaptation. In CVPR, 2024.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 1at.:993.11000 12261, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of oull-of-distribttion generalization. In CVPR, 2021.
Raza Imam, Hanan Gani, Muhammad Huzaifa, and Karthik Nandakumar. Test-time low rank adapta- tion via confidence maximization for zero-shot generalization of vision-language models. arXio preprint arXiv:2407.15013 2024.
Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 141771 2024.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, 2023.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NewrIPS, 2020.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yann LeCun, Léon Botton, Yoshua Bengio, and Patrick Haffner. Gradient-based lcarning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Jungsoo Lee, Debasmit Das, Jacgul Choo, and Sungha Choi. Towards open-set test-time adaptation utilizing the wisdom of crowds in entropy mininization. In ICCV, 2023.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Yushu Li, Xun Xu, Yongyi Su, and Kui Jin. On the robustness of open- world test-time training: Self Self-training with dynamic prototype expansion. In TCCV, 2023.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupercised feature learning, volume 2011, pp. 7. Granada, Spain, 2011.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient lest-time model adaptation without lorgetting. In ICML, 2022.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprinl 1xXii:17100000 2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV, pp. 1406 1415, 2019.
Ori Press, Steffen Schneider, Matthias Kümmerer, and Matthias Bethge. Rdumb: A simple approach that questions our progress in continual test-time adaptation. NeurIPS, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML. PMLR, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NewrIPS, 2015.
Jameel Hassan Abdul Samadh, Hanan Gani, Noor Hazim Hussein, Muhammad Uzair Khattak, Muzammal Naseer, Fahad Khan, and Salman Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. In NewrIPS, 2023.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In NewrIPS, 2020.
Manli Shu, Weili Nic, Dc-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS, 2022.
Manogna Sreenivas, Goirik Chakrabarty, and Soma Biswas. pstare: Pseudo source guided target clustering for fully test-time adaptation. In WACV, 2024.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. Tn JCLR, 2021.
Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. Tn ICCV, 2023.
Ce Zhang, Simon Stepputtis, Katia P Sycara, and Yaqi Xie. Dual prototype evolving for test-time generalization of vision-language models. In NewrIPS, 2024.
Bingchen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie. Tuning layernorm in attention: Towards efficient multi-modal Ilm finetuning, arXio preprint arXiv:2312. 11420, 2023.
14

---

Published in Transactions on Machine Learning Research (05/2025)
APPENDIX
A Gradient Analysis of the ReDUCe Loss
Here, we delve deeper into the ReDUCe loss function in ROSITA, breaking down its key components and mathematically retonstrate why the proposed objective improves the separation of Ca and C. samples. We'll focus on contrastive loss components Lp and Ly which are designed to improve discriminability.
ReDUCe loss in a nutshell. test sample 2; arrives at time with feature representation ft- Two feature banks, MM and Ms store reliable sample features from Ca and C. respectively. ReDUCe loss aims to pull the test sample's feature fe towards its positive samples 2* which are its K ncarest neighbors Qa kNN Ma) if it is reliable Ca sample or Qu = KNN (Je: M.) if it is reliable C, sample. The feature fe is pushed away from its negative samples 2 which are the K nearest neighbors from the undesired feature bank M, if it is a reliable C2 sample or from the desired feature bank Ma if it is a reliable C. sample. The features f,,2t= are all unit norm vectors. The key to understanding the behavior of the proposed loss is to analyze gradient.
Gradient of LD with respect to fú:
The contrastive loss for desired class samples Ln is defined as:
1 CD
Σ 10y = (t) log
exp (sim ++) /T)
K+
Σ -s-ea EQ. exp(sim fe, 2- )/T)
StEQa
(11)
8LD
1
0 log exp (sim (fr, x //T Σx-eq. exp(sim(ft, z )/T)
1 01) Of.
0f
K
The loss is of the log-softmax structure. Consider gradient of the following term:
i exp (sim (Jz, [1,=+)/T] af
(sim (Jes (11xzt))
log
log Σ
exp(sim(ft, )//)
Σ exp(sim ft, 2 T/T] Of.
af
= EQ
=-
The gradients of the two terms involved
0
sim (fe, 2
z
of T
Σ
a cxp(sim(fe.27)//
0
log Σ
exp(sim(Se,2 2/1
EQ
as.
0f.
Σ (xp(sim(Jt5==)) )/T)
2-
EQ
(12)
Σ exp(sim(ffs /TT
1 EQ
5 Σ exp(sim(ft, x )/T)²
EQ
1 Σ p(=T)=
EQ
The final gradient of the log-softmax term is
i
log
exp exp (fr,z))/5
x - -
Of
Σ cxp(sim(ft,,- 2-
EQ
x EQ
where p (2- Vis the softmax probability of the negative samples defined
15

---

Published in Transactions on Machine Learning Research (05/2025)
exp (sim (fe, /f,,>>//1)
p(=))
Σ exp (sim (fr, 22) /T)
E
Substituting Equation 12 in Equation 11 respect fe as
we get the gradient of the desired sample contrastive loss Lp with
8Ln
Σ ztEQu
1
0z) x -- Σ =
(13)
Of.
K+
EQ
Gradient of LD with respect to fú: The contrastive loss for desired class samples Lp is defined as:
exp (fe, (fe,2*)/7)
LU
Σ
log
K
Σ exp(sim St; )/T)
2 EQu
EQQ
(14)
0Lv
eximsim (fr, z²+) /T)
log
Of.
K
f
Σ exp(sim ft, 2 =/T))
EQ
EQa
Substituting Equation 12 in Equation 14
we get:
ULu
Σ
z - x())=
(15)
05,
K+
=tEQu
EQa
Interpretation of the Gradients:
Both the gradient terms Equations 13 and 15 have two components: Positive term x and Negative lerm ((= =T The positives and negatives are suitably chosen from the desired and undesired feature banks.
Positive term z+: The term = pulls the test feature Se closer feature vectors z+. This term represents the attraction force that encourages Ca samples to cluster together in Lp and C. samples to cluster together in Ly.
Negative term 2== The negative samples 2 exert a repulsive force, pushing fs away from them. The strength of this repulsion is controlled by the softmax probabilities p (2"), where higher similarity between fe and = incereasss the repulsion force. This inherently models the degree of hard negatives from the negative feature bank.
The overall gradient update encourages fe to move closer to its positives while moving away from its negatives, enhancing the separation between samples from Ca and C2 classes.
16

---

Published in Transactions on Machine Learning Research (05/2025)
B Baselines
B.1 Vision Language Models
CLIP (Radford al. 2021) is a multimodal VLM consisting of two modules: Vision encoder and Text, encoder denoted as Fv and Fr respectively. During pre-training, the two modules are jointly trained in a contrastive self-supervised fashion to align massive amounts of web scrapped image-text pairs. CLIP has demonstrated impressive zero-shot performance across wide variety of datasets.
MaPLe (Khattak et al, 2023 is a multimodal prompt learner model that simultaneously adapts both vision and lext. encoders while fine-tuning CLIP for downstream tasks. They use learnable text prompts pr and bridge the two modalities using visual prompts obtained as py = Proj(pr). Learnable tokens are also introduced in deeper layers of both image and text encoders, cnabling progressive feature adaptation.
B.2 Methods
ZSEval (Radford et al., 2021): Given lest image It, the image feature is extracted from the vision encoder as ft FF (24). For C-class classification problem, the classifier is obtained by prepending predefined text prompt pr="A photo of a', with the class names {ei, c.. to form class specific text inputs (pr, 4 for i E (1,...CC C}. These texts are then embedded through the text encoder as 4 = Fr(PPr;c) to get the text classifiers {4,, 12....cc The class prediction is made by identifying the text feature t; which has the highest similarity with the image feature ft.
TPT (Shu et al., 2022) aims to improve the zero shot generalization ability of CLIP by providing custom adaptable context for each image. This is done by prepending learnable text prompts pr to the class names. The text classifiers t = FF(((r;e))), { C'} are now function of these learnable prompts, which are specially adapted for each test image using an entropy minimization objective as arg minp, Cent The entropy is obtained using the average score vector of the filtered augmented views.
Prompt Alggn (PAlign) (Samadh et al. 2023) leverages multimodal prompt learner model MaPLe Khat- tak et al.] 2023) to facilitate the adaptation of both vision and language encoders for cach test sample. They align the token distributions of source and target domains, considering ImageNet as a proxy for the source datasct of CLIP. The vision and language prompts of MaPLe are optimized with the objective arg min(pvvpr)) pr) Lent + Latign for each sample
TPT-C (Shu et al., 2022) PAAlinn (Samadh et al., 2023): We adapt TPT and PAlign for continuous model update, which we refer as TPT-C and PAlign-C respectively. The prompts {pr} and and PAlign are continuously updated with the test stream with their respective test objectives.
in TPT
(K+1)PC (Li et al., 2023):: This was the first work exploring open world TTA, however it was done in the context of CNNs and not VLMs. Also, the test samples come in batchos, while we perform single image TTA. We adapt this method for problem setting as follows: As use VLMs, we use the text prototypes (instead of the source prototypes). The prototype pool is dynamically updated by adding features of reliable test samples recognized to belong to undesired classes. The vision encoder is updated using a (K+1) way prototypical cross entropy loss.
TDA (Karmanov et al., 2024): : TDA is a training-free dynamic adapter for TTA lision-language models, utilizing lightweight key-value cache for efficient pseudo label refinement without backpropagation.
DPE (Zhang et al.) 2024): DPE accumulates task-specific knowledge by dynamically evolving two sets of prototypos, textual and visual, during test time. These prototypes are refined to capture increasingly accurate multi-modal representations for target classes. To ensure consistency between modalities, DPE incorporates learnable residuals for each test sample, aligning textual and visual prototypes for improved representation alignment.
UniEnt (Gao al.) 2024): This is a very recent work addressing open-set TTA in the context of CNNs. They use a Distribution Aware Filter (DAF) based on Gaussian Mixture Modeling of the scores to distinguish
17

---

Published in Transactions on Machine Learning Research (05/2025)
between desired and undesired class samples. They employ entropy minimization and entropy maximization objectives for desired and undesired class samples respectively.
We equip all the baselines with the same LDA based Desired vs Undesired class identifier described in Section 2.2 for fair comparison of the TTA methods for this problem.
B.3 Datasets
We experiment with diverse datasets encompassing corruption, style transfer and other common datasets.
CIFAR10-C (Hendrycks & Dietterich 2019 a small-scale corruption dataset of 10 classes with 15 common corruption types. consists of 10,000 images for each corruption.
CIFAR-100C (Hendrycks & Dictterich 2019 also corruption dataset with 100 classes and 15 corruption types. It also consists of 10,000 images for each corruption.
ImageNet-C (Hendrycks & Dietterich 2019 is large-scale corruption dataset spanning 1000 categories with a total of 50,000 images. 15 types of corruption images are synthesized from these 50,000 images.
ImageNet-R. (Hendrycks et al. 2021 is realistic style transfer dataset encompassing interpretations of 200 ImageNet classes, amounting to a total of 30,000 images.
VisDA Peng et al. 2017] is synthetic-to-real large-scale datasct, comprising of 152,397 synthetic training images and 55,388 real lesting images across 12 calegories.
DomainNet (Peng al. 2019 is large-scale domain adaptation dataset. We use the Clipart, Painting and Sketch domains with 345 categories from the DomainNet dataset for our experiments.
CCC Press et al. 2023 is a benchmark designed to assess long-term continual TTTA behavior in a changing world, covering scenarios such as weather changing from foggy to rainy, day to night.
MNIST [Leuun al. 1998 is dataset of handwritten images consisting of 60,000 training and 10,000 testing images.
SVHN (Netzer et al. 2011 is also digits dataset with house numbers captured from real streets. It consists of 50,000 training images and 10,000 testing images.
We perform experiments on eight domains Da for desired class samples. The corrosponding Da are chosen such that there is no overlap between the classes Ca and C, as described in Table g The 15 corruptions of CIFAR-10C/1100 10OC/ and ImageNet-CC C fall into four categories: synthetic weather effects, per-pixel noise, blurring, and digital transforms. snow corruption is a synthesized weather effect on which all the main experiments of CIFAR-10C, CIFAR-100C and ImageNet-C are done. To evaluate the robustness of our method across different, corruption types, we do additional experiments with impulse noise, motion blur and ipeg compression corruptions from the categories per-pixel noise, blurring and digital transforms respectively and report the results in Section D.4
Table 8: Details of desired and undesired class dataset combinations
Datasets
images
Da
Du
Da
Du
Total
CIFAR-10C COFO
MNIST, SVHN Tiny ImageNet, CIFAR-100C
10000
10000
20000
MNIST, SVHN, Tiny ImageNel, CIFAR-10C
10000
10000
20000
ImageNet-C
MNIST, SVHN
50000
50000
100000
ImagoNet-R
MNIST, SVHN
30000
30000
60000
VisDA
MNIST, SVHN
50000
50000
100000
Clipart
MNIST, SVHN
29208
29208
58416
Painting
MNIST, SVHN
43700
43700
87400
Sketch
MNIST, SVHN
41832
41832
83664
18

---

Published in Transactions on Machine Learning Research (05/2025)
B.4 Implementation Details
Here, we describe the parameters chosen for all the baseline methods and our proposed method.
TPT (Shu et al. )022): The prompt is initialized with the default A photo of a text. The corresponding tokens in the input text embodding space are optimized for each test image. The prompt is reset after each updale. single lest image augmeented 63 times using random resized crops to create a batch of 64 images. The confident samples with 10% lowest entropy are selected. The test time loss is the entropy of the averaged prodiction of the selected confident samples. AdamW optimizer with learning rate of 5e -4- is used, following (Shu et al. 2022).
PAlign (Samadh et al. 2023): : Following Promp-
tAlign (Samadh et al. 2023). MaPLe (Khattak et al.)
2023 model trained on ImageNet using 16-shot train- ing data with prompt tokens for a depth of layers is used. The prompts on both the text and vision en- coders are optimized on single test image. Similar to TPT, 10% of 64 angmentations are selected to compute the entropy loss. The token distribution loss to align the token statistics of lest. with that of source data is computed for all 64 images. AdamW optimizer with a learning rate of 5 update the prompts for each image, following (Samadh et al. 2023). The prompts are reset to the ImageNet trained prompts after each update.
100
TPT-C (AdonW)
PAlignOG (AdoaW)
TTT-C (SGD)
PAllgn-C (SGD)
75
50
25
0
TPT-C (Shu et al. 2022)/ / PAlign-C (Samadh et al., 2023): We create the continuous prompt up- date versions of TPT and PAlign as TPT-C and PAlign- C respectively. The only difference is that the prompts are continuously updated using the test stream of sam- ples. If sample is detected as reliable Ca sample, the respective test time objectives are used to update the prompts. For this purpose, vary the learning rate and optimizer to select the best optimizer for continuous prompt update. On performing experiments on CLFAR-10C/MNIS' data, from Figure 5 we observe that SGD optimizer with learning rate 10 works the best for continuous prompt update and honce we use this for all the experiments of TPT-C and PAlign-C.
10-5
10-4
10-3
10-2
learning rate
Figure 5: Performance of TPT-C and PAlign-C for CIFAR-10C/MNTE with AdamW and SGD opti- mizer on varying learning rates.
(K+1)PC (Li et al.) 2023): The vision encoder updated using a (K+1) way prototypical cross entropy loss The prototypes are updated using the test stream of samples. The learning rate is set to 0.001.
TDA (Karmanov et al.,
2024): : We use 7, from the LDA based Co vs C. identifier to recognise the desired and undesired class samples. Following (Karmanov et al., 2024,, we set the shot capacity to 3 and the number of key-value caches is Cd as we use the adapter only for desired class samples.
DPE (Zhang et al.) 2024): : We use the same LDA based Ca vs C. identifier to recognise the desired and undesired class samples. We use the same hyperparameters presented in (Zhang al.) 2024). A priority queue storing visual features per class used. The text and visual prototype residuals are updated with a learning rate of 0.0006 using AdamW optimizer.
UniEnt (Gao et al. 2024): We use the UniEnt objective in combination with LDA based class indentifier. The entropy minimization and maximization objectives are used for desired and undesired class samples respectively. The Nore parameters are updated with a learning rate of 0.001 using SGD optimizer.
We use SGD optimizer with a learning rate of 0.001 to update the LayerNorm affine parameters of the Vision encoder. We set the size of score bank S to 512, number of neighbours K to and the size of M, is set to to 64.
ROSITA:
19

---

Published in Transactions on Machine Learning Research (05/2025)
C Additional Analysis
In this section, in addition to the analysis done in Section 6 we study the robustness of the proposed method ROSITA more extensively, in the terms of (1) Error bars on different test data streams, (2) Role of the parameter K, the number of neighbours, (3) Analysis of the scores on using different combinations of the proposed loss components, (4) Comparison of different Cd vs Cv Class identifiers for Open-set TTA.
C.1 Analysis on error bars
To study the robustness of our method for differently ordered test streams, we run ROSITA with five random seeds and report the Mean and Standard deviation of the Aссни in Table g for CIFAR-1OC/100C as Da and MNIST, SVHN, Tiny ImageNet, CIFAR-100C/106 as D. (corresponding to our results in Table 3 in the main paper). We observe that the variance in the performance of ROSITA is very low, reinforcing the robustness of the proposed method for different shuffled datasets and augmentations created.
Table 9: Performance (Mean and Standard deviation of AсCни) of ROSITA across 5 random seeds for CIFAR-10/100C as Da with 4 other datasets as Du-
Da
MNIST
SVHN
Tiny
CIFAR-100/10C
CIFAR-10C 84.07 + 0.023
78.90 + 0.038
80.10 + 0.014
69.44 + 0.018 48.10 + 0.024
CIFAR-100C
57.09 + 0.041
47.90 + 0.047
55.95 + 0.051
C.2 Analysis on parameter K
Table 10: Performance (Accна)) on varying K with MNIST as D.-
K
Da
CCa
0
1
3
5
7
9
CTFAR-10C
10
80.97
83.9
84.32
84.17
84.10
84.02
ImageNet-R
200
64.32
83.65
83.87
83.53
83.39
83.42
ImageNet-C
1000
42.05
48.35
47.17
48.53
48.37
47.73
We vary the hyperparameter K which represents the number of positives and negatives chosen in Equation 6 and and report the results (Aceим) in Table 10 The size of the feature bank Ma is set as Na = x Ca- Na increases with the number of classes as well as the number of neighbours K. We set K to be in all main results reported, which corresponds to feature bank size Na of 50, 1000, 5000 respectively for the datasets CIFAR-10C, ImageNet-R and ImageNet--C respectively. In Table I0] use the notation K 0 to correspond to the case where only the reliable pseudo label loss Lne is used. The results show that even with K = 1, there is a significant improvement in AсCHи when compared to the case where LD, Ly is not used (K = 0). On further increasing K, we observe improvement only for the CIFAR-10C as Da, but the performance is similar for ImageNet-R and ImageNet-C for higher values of K as well. Further, we investigate this observation that the performance of ROSITA is similar on significantly varying K or the feature bank size. For K 5, we check the average number of positives actually selected for Lp in Equation for each of these datasets. We find this to be 4.1, 2.5 and 1.5 for CIFAR-10C, ImageNet-R. and ImageNet-CC respectively. This agrees with the results in Table 10] where K of 3, 5 works better compared to more neighbours have common pscudo label, aiding the clustering of classes of interest. For CIFAR-10C and ImageNet-R, using K 5 suffices and for ImageNet-C as only 1-2 neighbours are matched for majority of reliable desired class samples, setting K = 1 suffices. For practical purposes, this observation suggests that the buffer size for Ma can indoed be reduced basod on storage budget available depending on the application and device the model is deployed on. For e.g., if the memory budget available can store only upto 1000 features, K can be set flexibly depending on the number of classes of interest. For Image with 1000 classes, K can be set to 1.
20

---

Published in Transactions on Machine Learning Research (05/2025)
C.3 Detailed analysis of ReDUCe Loss components
We provide detailed results of Table including all the five metrics in Table 11 Additionally, visualise the histograms of the scores šr on using different combinations of the loss components of ReDUCe Loss in the Figures justifying their role better discrimination of samples from Ca and C..
Table 11: Detailed performance metrics analysing the ReDUCE Loss components.
CIFAR-IOC/MNIST
ImageNet-R/MMi RNNIST
Re
LD
Lv
AUC
FPR
Acep
Accy
АсCHи
AUC
FPR
Acep
Accu
AсCHи
X
X
X
91.91
85.04
60.82
99.77
75.57
91.27
91.09
55.67
99.90
71.50
X
X
95.29
30.82
68.36
99.30
80.97
81.07
99.02
48.42
95.76
64.32
X
V
X
95.23
28.91
66.93
98.52
79.71
87.73
94.67
51.13
98.34
67.28
X
X
98.61
12.73
66.60
99.68
79.84
99.39
4.81
67.81
99.99
80.82
X
V
99.27
4.15
67.76
99.73
80.69
99.48
4.40
69.38
99.98
81.92
V
V
99.10
7.63
72.81
99.74
84.17
99.44
4.29
71.73
99.98
83.53
3,000
3,000
1000
01
0.2
0.3
04
0.1
0.5
633
041
0.1
03
03
0.1
0.3
03
(a) ZSEval
(b) Li
(c) Lo + Lu
(d) LR + LD
Lu
Figure Histograms of Ca and C. class scores for ZS-Eval and on using different loss components of the proposed ReDUCe loss on CIFAR-1OC/MNIST dataset with CLIP.
3,000
5,00
8,00
3,00
0/90
6.000
61101
2000
2,000
2000
2100
0.3
01
01
164
0
055
(a. ZSEval
(b) LR
(c) Lp + Lu
(d) L + Lp + Lu
Figure Histograms of Ca and Cu class scores for ZS-Eval and on using different loss components of the proposed ReDUCe loss on EmageNet-R/MMeN dataset with CLIP.
From Figure and we observe that, on using just LRE, the scores Cd and C2 classes still sufficiently overlap, similar to the case of 7.Svval. The performance purely depends on the quality of pseudo labels of the detected reliable desired class samples. In CIFAR-1OC, as there are only 10 classes and given that the performance of ZSEval in CIFAR-10C is fairly good, ensures good quality pseudo-labels, hence resulting in overall better metrics even using LRe shown in Table II ImageNet-R dataset inherently has more confusion as it is a 200-way classification problem. This naturally could result in lower quality pscudo-labels, in turn degrading the performance compared to ZSEval. In addition, using LRe for desired class samples that are misclassified as undesired class samples increases the FPR and results in a decrease in overall metrics compared to ZSEval. However, using Lp and Ly scparates the scores 8, of the samples from C2 and C.,. resulting in two distinct peaks as seen in Figure and 7 which in turn results in significantly low FPR reported in Table 11 Hence, the best results (Table 11 are obtained using the proposed ReDUCe loss, where
21

---

Published in Transactions on Machine Learning Research (05/2025)
all the loss components help each other to better discriminate the desired classes Ca from Cu (measured by AUC, FPR) and also improving the Ca-way accuracy (Acep) on desired classes.
Effect of ReDUCe loss in representation space: Contrastive learning through ReDUCe loss influences the representation space by enhancing feature separability, improving the discriminability of desired vs. undesired classes. In addition to the accuracy metrics presented, here we compute the inter-feature distance between desired (Ca) and undesired (C.) class features through the following distance metrics:
(a) Mean Pairwise Distance is simple but effective way is to compute the pairwise Euclidean or cosine distance between all Ca and Cn samples and then take the mean:
dEuclidean
Σ Σ IIfa - full2; dcosine
fa fo
(16)
Fal
Fall
faEFa f.E.. [Aal
FuEFF.
where Fa and F. are the feature banks for desired and undesired class samples, respectively.
(b) Wasserstein Distance (Optimal Transport) mcasures the optimal transport cost between the two feature distributions, making it ideal for comparing feature distributions of desired (Ca) and undesired (Cw)
classes.
dassersteein FElI(FaFF))
inf
B(fafa))>lllf - full
(17)
Table 12: Inter-feature distance measures for ZSEval and ROSITA using CIFAR-10C as Da-
Du
Method Euclidean (t)
Cosine (t)
Wasserstein (t)
ZSEval 0.5831
0.1731
0.5553
MNIST
ROSITA
1.2970
0.8623
1.2708
ZSEval 0.6963
0.2456
0.6535
SVHN
ROSITA
1.0785
0.5918
1.0265
ZSEval
0.7522
0,2860
0.6583
TinylmageNet
ROSITA
0.9939
0.4994
0.8852
ZSEval 0.5587
0.1598
0.4265
CIFAR-100C
ROSITA
0.8194
0.3411
0.6594
The results clearly show that ROSITA improves feature separability across all metrics compared to ZSEval, reinforcing its effectiveness refining the representation space for open-set recognition. The degree of separation varies with task difficulty for MNIST, the distances increase significantly, while for CIFAR-10CC, the improvement is smaller. This aligns with intuition, as distinguishing between CIFAR-10C and CIFAR- 100C is inherently more challenging than separating CIFAR-10C from MNIST. The quantitative results validate that contrastive adaptation enhances representation learning, but the extent of improvement depends the complexity of the undesired dataset.
C.4 Comparison of different C. C. Class identifiers for Open-set TTA
To study the role of the Ca C, class identifiers in Open-set Single Image TTA, we experiment with three class identifiers, five datasets Da with MNIST as D, using CLIP backbone.
(1) Simple thresholding based on Maximum Softmax Probability (MSP): We set fixed thresholds Ta: Ta to identify reliable samples from Ca and Cn classes respectively and т. to distinguish between Ca and C, samples. We combine this class identifier with the ReDUCe loss the proposed ROSITA framework.
(2) Distribution Aware Filter (DAF) (Gao et al., 2024) : We adopt the Distribution Aware Filter proposed in UniEnt (Gao et. al. 2024), very recent. method open-set TTA using CNNs, where they
22

---

Published in Transactions on Machine Learning Research (05/2025)
model the scores 8, (similarity between image feature and source prototype) as Gaussian Mixture Model for each batch. In our case, as we do single image TTA, use score bank described in Section 2.2 as a proxy for the batch of samples, to estimate the parameters of the GMM. As it is a 2-component GMM, we identify a sample as a dcsired class sample if the probability x(x)) of the sample belonging to the dcsired classes(componee) with higher mean estimated) greater than 0.5 or vice versa. The GMM based class identifier defined as follows:
Ca
if Tx)) 2 0.5
i = |=
(18)
E Co C. if =(; 0.5
We combine this class identifier with the Unified entropy objective and ReDUCe loss proposed by UniEnt Gao et al.
2024 and our proposed ROSITA framework respectively.
(2) Linear Discriminant Analysis (LDA) based (Li et al., 2023 As described in Section 2.2 set Ta to Ma and Tu to Pa to identify reliable Cu and C samples to perform TTA. We set Ty to M4 to distinguish between Ca and Cn samples. The thresholds are estimated in an online manner using the score bank S. The LDA based class identifier is defined as follows:
E Cd
if st
T if <
ú =
(19)
E Cx
We combine this class identifier with the Unified entropy objective and ReDUCe loss proposed by UniEnt Gao et al. 2024) and our proposed ROSITA framework respectively. The three thresholds for ReDUCe loss Table 13 correspond to Tu/Te/Te where Tu and Ta is used to identify reliable test samples and 7: used to distinguish between Ca and C₂ samples. In the case of DAF with ReDUCe loss, use the means pá and p. for the two gaussian mixture components to identify reliable samples.
Table 13: Comparison of Ca Cu class identifiers: MSP DAF vs LDA. The three thresholds for ReDUCe loss correspond to Tu/Ti/Ta where Tu and тą is used to identify reliable test samples and is used to distinguish between Ca and Cu samples. In the case of DAF with ReDUCe loss, we use the estimated means μa and p. of the two Gaussian mixture components to identify reliable samples.
Test-time objective
D.: MNIST
Ca VS C2
Threshold
C-10C
C-100C
IN-C
IN-R
VisDA
0.4/0.6/0.8
43.44
34.42
1.20
77.12
88.49
MSP
0.3/0.5/0.7
ReDUCe
33.70
32.60
1.74
80.29
50.87
0.5/0.5/0.5
22.82
37.41
1.91
30.90
32.31
LDA
8t > Tt.
75.62
48.31
41.53
71.73
78.09
UniEnt
DAF
Txx,) > 0.5
79.43
50.12
46.52
79.30
86.79
LDA
Pu/Te/ ////0.///
84.17
57.34
48.53
83.53
90.64
ReDUCe
DAF
83.56
55.37
48.33
83.32
90.97
Our key observations based on the results in Table 13 are as follows:
The performance of both, DAF and LDA basod class identifier is significantly better than the simple thresholding case adaptation using ReDUCe loss. The thresholds estimated in online manner using the score bank S are more reliable than fixed thresholds. The DAF and L.DA-based class identifier is able to better discriminate between Ca and Cn samples, resulting in better performance.
Fixed vs Dynamic Thresholds:
The performance on using ReDUCe loss (with either DAF or LDA class identifier) is significantly better than using the Unified entropy objective proposed in UniEnt (Gao al. (024).. The ReDUCe loss components aid each other to better discriminate the desired classes Ca from C, (measured by AUC, FPR) and also improve the Ca-way accuracy (Accp) desired classes.
UniEnt vs ReDUCe loss:
23

---

Published in Transactions on Machine Learning Research (05/2025)
The performance of LDA and DAF based class identifier perform very similarly when used in combination with ReDUCe loss. This suggests that ReDUCe loss in ROSITA is robust. to the choice of a dynamically updating class identifier.
LDA vs DAF with ReDUCe loss:
Why is ReDUCe loss better than Unified entropy objective for Open-set TTA of VLMs?
Both LDA (Li el al. 2023 and DAF (aa el al. 2024 were proposed for CNN based open-set TTA where a source model is trained on say clean data and is adapted to new domains, with the observation that the featuro-prototype similarity scores 8, can distinguish desired and undesired class samples. In the case of VLMs, the source model is trained on a large scale dataset and is adapted to potentially unseen/ unseen/corrupee /corrrptee//ooarriiie covaraattaattte data. The prior that the feature-prolotype similarity scores 9, can distinguish desired and undesired class samples does not translate to VLMs as the scores overlap signéficantly, observed in ZSEval histogram plots in Figures 6 and 7
In the case of CNNs, where the the initial scores are well separated and model has access to a batch of lest samples at a time, UniEnt leverages this to further aid the separation of desired and undesired class samples the batch through the UniEnt objective. In the case of VLMs, the scores are not well separated initially. This results in the means pa and Pa in the case of LDA to be very close leading to misclassification of Ca and C. class samples using the estimated threshold 76. Similarly, in the case of DAF, the two components of GMM would not be very distinctive to well distinguish desired and undesired class samples. This misclassification can result in entropy minimization being applied on C. samples and entropy maximization on Ca samples, which is undesirable. Employing UniEnt objective with several misclassified samples may not actually separate desired and undesired classes, as also empirically observed in Tables 2 (UniEnt has high FPR rate in general). Entropy maximization of C. samples does not explicitly enforce the separation of desired and undesired class samples in the feature space.
The Lo and Ly loss components of ReDUCe loss explicitly enforce the separation desired and undesired class samples in the common VL latent space, while the LRee loss aims to only align the desirod class samples to align with the text prototypes. With time, the model adapted such that undesired class samples are away from the desired class samples and also the text prototypes. This ReDUCe loss addresses the challenges in single image open-set TTA in a holistic manner, resulting in better performance.
On adopting UniEnt objective to single-image TTA, either entropy minimization or maximization loss would be active based on whether a test sample is identified desired or undesired class sample, which is limitation, the objective cannot enforce distinction between the two types of feat
In the case of CNNs, where the the initial scores are well separated and model has access to a batch of test samples at a time, UniEnt leverages this to further aid the separation of desired and undesired class samples in the batch through the UniEnt objective. In the case of VLMs, the scores are not well separated initially, hence the ReDUCe loss components (with the help of feature banks) acts the driving force to better separate the desired and undesired class samples in the common latent space, resulting in lower FPR rates consequence.
C.5 Extensive analysis on parameter choice for continuous adaptation of VLMs
Our initial experiments showed that updating LayerNorm parameters with simple entropy objective can effectively improve closed-set TTA performance. We illustrate this in Section 4 on CIFAR-10C dataset. Further, to justify our choice of updating LayerNorm parameters, we present the detailed experiments we conducted based on the following choices: (a) Learnable paramcters: (1) Prompts, (2) Full network, (3)
Lo)A Adapters [Imam et al. 2024), (6) LayerNorm parameters (Zhao et all 2023) (b) Datasets: In addition to CIFAR-10C (Section 4], we experiment with ImageNet-R, a relatively large scale datasct consisting of 30,000 images from 200 classes. (c) Optimizer: Along with SGD, we experiment with AdamW optimizer also used in [1), with varying learning rates both CIFAR-10C and ImageNet-R dataset.
First Attention Block of ViT, (4) Last. Atlention Block of ViT, (5)
24

---

Published in Transactions on Machine Learning Research (05/2025)
Table 14: Accuracy on updating different parameter groups on CIFAR-10C and ImageNet-Rt datasets.
CIFAR-106
ImageNet-R
Optimizer
Parameters
le
le
le
le-3
le
le
le
1e
le
lc 1e-2
Prompts
73.40
31.04
12.53
11.18
10.19
73.97
74.17
74.71
25.68
10.63
Full
10.48
10.44
9.99
10.00
10.01
14.18
7.19
0.65
0.65
0.42
First Block
75.1
76.12
78.27
13.07
10.01
73.84
74.31
74.91
8.76
0.32
SGD
Last Block
73.45
72.42
59.44
10.17
10.02
75.95
77.93
24.82
0.52
0.67
Prompts+ +LN
73.82
46.77
24.71
10.24
10.18
73.76
75.09
76.35
28.72
11.74
LoRA Adapters
73.86
73.90
75.42
83.15
12.58
73.51
73.57
74.22
77.39
34.83
LayerNorm
74.35
76.61
80.41
84.58
11.69
74.13
74.35
75.23
76.92
33.07
Prompts
72.40
18.6
12.83
10.04
10.08
74.4
75.17
27.93
6.82
4.37
Full
10.32
10.03
10.00
10.00
9.97
14.83
0.95
0.28
0.52
0.66
First Block
79.05
24.70
10.84
10.00
10.00
74.6
74.8
5.68
0.26
0.15
AdamW
Last Block
59.23
10.84
10.49
10.00
10.01
77.44
10.67
0.51
0.25
0.33
Prompts+LN
75.01
72.10
21.92
13.33
10.01
74.52
76.45
12.99
8.87
5.55
LoRA Adapters
77.64
81.35
14.01
10.25
10.02
74.34
76.14
18.63
2.26
0.62
LayerNorm
76.10
81.57
85.9
85.27
10.03
73.96
75.64
78.28
78.81
31.47
LoRA Adapters for Model adaptation:
Following Imam al. (024)), we employ LoRA adapters with rank 16 for multi-head self-altention layers in
the vision encoder. We also vary its rank to study its sensitivity (Table 15
Table 15: Change in HM, number of parameters for varying rank of LoRA adapters vs. LayerNorm.
LoRA rank
2
4
8
16
LayerNorm
HM (f)
27.49
34.08
79.24
83.15
84.58
Additional params (1)
110,592
221,184
442,368
884,736
Learnable params (1)
110,592
221,184
442,368
884,736
39,936
(a) Sensitivity to Learning Rate: As observed in Tables 14 and 15
LoRA-based adaptation can be effective when appropriate learning rate is chosen. For instance, it performs well at le-3 with SGD and 1e-5 with AdamW. While it may require some tuning to achieve optimal performance, LoRA remains a viable option for test-time adaptation under the right hyperparameter settings.
(b) Increased Model Complexity: Unlike LayerNorm tuning, which does not introduce additional parameters, ToRA requires the addition of adapter modules during deployment. As shown in Table 15 the number of additional parameters scales with the rank of the LoRA adapter, increasing from 110K (rank-2) to 884K (rank=16). This added complexity especially in resource-const rained environments. A key distinction between using LoRA for training- time finetuning and test-time adaptation is that in the former, LoRA weights can be merged with the base model post-training, effectively eliminating the additional parameter overhead. However, in TTA, adaptation is a continuous process, meaning that the additional parameters cannot be merged with the backbone model. This makes LoRA-based adaptation more computationally and memory-intensive compared LayerNorm tuning, which operates within the existing model structure.
be concern for lightweight real-time adaptation settings,
(c) Sensitivity to Rank Selection: As seen in Table 18 above, the performance of LoRA is highly dependent the rank selection. Lower-rank configurations (c.g., rank-2, 4) lead to poor adaptation, whereas higher-rank settings (e.g. rank=16) perform significantly better but at the cost of increased parameter overhead. This trade-off int roduces another hyperparameter that must be carefully tuned.
While LoRA-based adaptation has been explored recent works, our findings suggest that LayerNorm tuning remains more efficient and robust choice for OSTTA due its stability across learning rates, no additional parameter overhead, and suitability for continuous adaptation without requiring explicit rank selection.
25

---

Published in Transactions on Machine Learning Research (05/2025)
D Additional Experiments
In addition to the results presented in the main paper, we perform additional experiments supporting the claims made and for more comprehensive understanding of the analysis presented in Section
D.1 Open Set Single Image CTTA Experiments
In addition to the Open-set CTTA experiments done on CIFAR-10C as desired class dataset, here we also experiment with CIFAR-100C as desired class dataset. We present the 15 corruptions of CIEAR-IOC/CIES /CIFAR- 100C sequentially as Da, one sample at time along with different datasets for C. samples, namely MNIST, SVHN, Tiny ImageNet, CIFAR-10C CIFAR-100C and report the results in Table 16 We observe that the improvement in performance of ROSITA agnostic open-set scenarios including different. combinations of Da (continuously changing domains) and D, datasets.
Table 16: Open-set CTTA performance for CIFAR-10C and CIFAR-100C as desired dataasess.
CIFAR-10C
CIFAR-100C
Method
SVHN
MNIST
Tiny
C-100C
MNIST
SVHN
Tiny
C-10C
ZSEval
64.33
64.04
66.50
58.49
39.00
36.29
38.41
35.04
TPT
64.26
64.03
66.50
58.47
39.00
36.24
38.38
34.45
(K+1)PC
65.13
62.52
66.93
57.46
40.64
37.05
38.23
34.55
TDA
66.02
66.44
67.64
59.44
40.49
40.35
39.92
35.42
DPE
23.36
50.12
58.96
35.56
30.75
19.23
35.85
27.62
ROSITA
66.86
65.26
68.89
59.16
41.64
38.02
40.44
36.05
D.1.1 Performance on Open-set Continuously Changing Corruptions Benchmark
CCC benchmark (Press et al., 2023 was specifically introduced to assess the long-term continual adapta- tion behavior of TTA methods in changing world, covering scenarios such as weather changing from foggy to rainy, day to night. We experiment with the CCC dataset where gradual domain changes are synthesized across 15 corruptions of ImageNet-C as Da and MNIST as D, for a sequence length of 300k samples. From Table 17 observe that ROSITA consistently outperforms prior methods even in this challenging CCC dataset.
Table 17: Results on open-set CCC benchmark.
CCC/MNIST
Method
AUC 1
FPR
Accp 1
Accy t
Aсcнaи
Fv-lval
88.45
99.53
(K+1)PP
95.82
20.16
20.37
99.97
33.84
TDA
87.42
85.83
20.42
99.35
33.87
UniEnt
89,90
82.86
18.99
99.62
31.90
DPF
84.68
87.56
18.47
99.40
31.16
ROSITA
96.02
19.96
21.14
99.11
34.84
D.2 Varying OOD ratio
In addition to the results presented in Table perform experiments varying the OOD ratio using ImageNet-R as desired class dataset which is rela- tively large scale dataset with 50,000 images from 200 classes. We use MNIST as undesired class dataset and vary the ratio of number samples belonging to ImageNet-Bi and MNIST 0.2, 0.4, 0.6, 0.8. From observe consistent improve-
Table 18: Varying ratio for
Method
0.2
0.4
0.6
0.8
ZS-Eval
65.46
67.13
69.25
70.77
TPT
65.67
67.73
70.12
71.54
TPT-C
64.83
64.55
48.97
63.86
(K+1)PC
78.09
81.08
81.35
82.61
TDA 67.90
71.33
71.54
71.47
Table 5 and Table 18 ments of ROSITA across datasets compared to the prior methods.
DPE
66.89
68.47
69.72
70.87
ROSITA
82.22
83.32
83.59
83.84
26

---

Published in Transactions on Machine Learning Research (05/2025)
D.3 Performance of ROSITA large Vision Language backbones
Here, in addition CLIP ViT-B/16 (Radford et al. 2021 and MAPLE (Khattak
backbones, perform experiments using large-scale Vision language backbones including CLIP ViT-I 14 by Ope-
2023
nAI (Radford et al., 2021 and Open-CLIP ViT-L/14 (Cherti et al. 2023 with CIFAR- 10C/ 100C as Da and MNIST, SVHN, Tiny-ImageNel and CIFAR-100C/100 as D.. From Table 19
we observe that ROSITA
consistently outperforms even very recent baselines like (K+1)PC (Li et al. 2023), TDA (Karmanov et al, 2024 suggesting that the performance of ROSITA is agnostic to the choice of VL backbone.
Table 19: Comparison of ROSITA with prior methods on large scale Vision Language backbones.
CIFAR-10C
CIFAR-100C
VL Backbone Method
MNIST
SVHN
Tiny
C-100C
MNIST
SVHN
Tiny
C-10C
ZSEval
83.94
74.54
80.16
72.32
56.29
52.35
53.25
49.89
CLIP VIT-L/14
(K+1)PC
85.43
80.60
81.65
71.90
64.14
55.18
54.53
47.90
TDA
84.91
76.87
81.07
74.23
59.11
55.25
55.44
52.48
ROSITA
89.46
83.42
83.61
75.63
65.41
60.31
57.55
54.66
7SEval
80.64
76.90
84.10
75.40
62.96
59.38
61.10
59.57
Open-CLit CLIP ViT-L/14
(K+1)PC
85.84
82.42
84.99
75.70
70.14
63.36
60.56
59.43
TDA
80.57
77.92
84.60
75.79
64.90
60.70
62.01
61.20
ROSITA
89.04
82.98
85.55
76.62
70.54
63.84
62.57
61.84
D.4 Experiments using different corruption types
To evaluate the robustness of our method across different domains, we do additional experiments with impulse noise, motion blur and ipeg compression corruptions from the corruption categories per-pixel noise, blurring and digital transforms respectively and report the results here. From Table 20 Table 21 and Table 22 we observe that ROSITA either outperforms or at par with prior methods in most cases even using the same set of byperparameters. This demonstrates its robustness across a variety of corruption types.
Table 20: Results on CIFAR-10C/100C (Impulsc Noisc) as Da with other D.-
MNIST
SVIIN
Tiny-ImageNer
CIFAR-100/11--
Method
AUC
FPR 4
HM f
AUC t
FPR 4
HM
AUC
FPR 4
HM f
AUC 1
FPR.
HM
-- 100C (Impulse noise) [C-10C (Impulse noise)
ZS-Eval
86.34
97.77
57.67
84.40
79.43
56.80
88.97
31.86
61.11
78.61
67.88
54.40
CLIP
TPT
86.35
97.83
59.80
84.43
79.52
58.97
88.96
31.99
64.48
78.60
68.24
56.38
TPT-C
62.34
87.66
39.90
59.71
83.29
35.42
81.30
38.59
37.02
66.22
89.92
30.86
ROSITA
98.87
9.43
71.31
82.85
56.82
61.03
93.36
21.47
64.47
78.69
69.45
57.87
MAPLE
ZS-Eval
91.10
76.09
64.01
92.98
45.28
63.66
83.77
44.44
60.93
79.22
65.26
$7.49
PAlign
65.76
45.18
65.28
44.42
62.75
79.22
65.24
58.80
PAlign-C
92.43
63.39
63.61
92.92
45.86
641.0
83.36
45.74
60.83
79.30
64.47
57.00
ROSITA
98.80
6.10
71.79
95.39
28.06
72.13
84.92
45.35
65.30
80.49
65.57
61.63
ZS-Eval
70.48
99.17
25.
51.12
96.44
25.69
59.90
67.18
27.72
53.51
94.97
25.16
CLIP
TPT
70.56
99.17
25.26
51.21
96.38
26.26
59.91
67.09
28.36
53.53
94.94
25.63
TPT-C
57.65
93.07
8.71
79.28
57.07
2.74
90.40
22.60
5.71
50.26
95.34
3.26
ROSITA
36.47
99.96
20.98
24.17
99.77
18.99
53.57
79.85
26.27
58.07
94.15
29.75
MАPLE
2S-Eval
69.29
89.49
33.66
$1.03
73.94
34.99
49.57
84.71
26.09
57.84
94.44
29.34
PAlign
69.31
89.54
33.74
81.05
73.98
34.96
49.60
25.81
57.84
94.48
29.53
PAlign-C C
71.14
34.38
82.08
68.24
47.27
25.95
57.79
93.54
30.73
ROSITA
95.38
8.80
433.06
80.25
41.21
34.88
42.77
97.15
19.70
49.73
96.72
12.62
27

---

Published in Transactions on Machine Learning Research (05/2025)
Table 21: Results on CUPAR-10C/110C(MMoii0 blur) as Da with other Dw.
MNIST
SVIIN
Tiny-ImageNet
CIPAR-100//104
Method
AUC
FPR
HM 1
AUC
FPR
HM
AUC
FPR 4
HM f
AUC
FPR
HM
C-10C (Motion blur)
CLIP
7S-Eval
TPT
97.73
97.72
2.75
2.08
74.15
73.69
96.40
96.39
18.16
18.34
73.82
74.42
95.23
95.25
15.75
15.72
75.03
74.27
79.37
79.56
70.08
69.86
62.86
63.25
TPT-C
80.73
86.28
63.74
62.09
62.52
42.19
80.76
51.66
48.04
55.66
97.04
37.53
ROSITA
99.90
0.04
81.87
96.50
21.55
77.47
96.58
13.65
77.44
82.03
65.95
66.96
MAPLE
2S-Eval
96.52
18.33
78.68
97.08
14.78
78.15
88.45
33.15
71.19
84.00
57.94
66.93
PAlign
96.51
18.37
78.92
97.08
14.82
78.38
88.45
33.13
71.73
83.99
57.99
67.15
PAlign-C
97.17
13.447
78.49
96.89
15.87
78.09
88.80
32.94
72.09
84.29
56.80
67.40
ROSITA
98.49
10.01
83.26
92.61
44.87
78.93
87.48
38.2
73.24
84.27
57.60
70.67
CC-100C (Motion blur)
ZS-Eval
93.08
58.92
48.17
83.63
81.33
46.04
79.34
53.56
48.53
64.03
91.54
11.63
CLIP
TPT
93.06
39.87
48.18
83.61
81.56
45.54
79.29
53.76
48.26
64.02
91.63
41.25
TPT-C
66.77
98.77
19.96
29.69
99.94
11:39
69.25
02.87
17.10
53.22
94.57
13.59
ROSITA
98.93
6.79
55.49
89.39
37.86
48.50
90.20
31.61
55.05
65.30
91.59
42.54
MAPLE
7S-Eval
81,21
80.28
45.66
89.04
60.73
46.98
60.84
80.63
40.60
64.01
90.18
42.30
PAlign
81.20
80.52
44.52
89.03
61.01
45.76
60.84
80.64
40.03
64.01
90.26
41.26
PAlign-C
82.72
68.08
49.92
90.48
53.83
51.87
62.00
82.85
41.66
64.47
89.05
43.58
ROSITA
57.30
85.13
80.20
62.55
94.62
41.54
Table 22: Results on CIFAR-10C/100(((PEG Compression) as Da with other Du-
MNIST
SVHN
Tiny-ImageNot
CIFAR- IIPAR-100C/10-C
Method
AUC
FPR
HM f
AUC t
FPR
HM
AUC
FPR 4
HM f
AUC
FPR.
HM
ZS-Eval
08.16
100.00
53.92
67.01
99.93
55.69
79.44
65.02
59.66
73.65
85.60
56.30
CLIP
TPT
68.07
100.00
54.16
66.97
99.93
56.06
79.37
65.11
60.09
73.64
85.58
56.87
C-10C (JPEG)
TPT-C
68.28
99.37
53.12
54.76
95.97
35.64
66.70
72.20
39.02
59.82
94.78
32.78
ROSITA
81.83
58.81
60.34
82.85
61.38
61.87
95.06
15.84
67.87
71.19
86.62
51.98
MAPLE
ZS-Eval
95.15
33.39
69.72
95.96
22.02
69.73
86.64
36.79
65.68
79.26
68.19
60.10
PAlign
95.13
33.57
69.62
95.95
22.01
69.31
86.63
36.82
65.62
79.26
68.18
59.86
PAlign-C
96.53
20.14
70.50
95.91
21.51
70.01
87.38
35.07
66.42
79.85
66.17
61.11
ROSITA
99.28
5.71
76.74
95.54
29.06
72.86
89.88
31.12
68.78
80.69
61.64
62.23
CIFAR- 1000 (JPEG)
ZS- Eval
50.88
100.00
32.27
39.25
100.00
26.41
48.65
95.60
29.92
53.51
95.59
32.48
CLIP
TPT
50.78
100.00
32.38
39.18
100.00
26.48
48.55
95.60
29.86
53.49
95.57
32.70
TPT-C
12.11
100.00
3.32
10.05
99.98
2.45
63.07
90.01
9.49
52.23
95.05
6.33
ROSITA
29.10
100.00
22.83
35.58
99.
23.50
50.76
94.76
31.64
53.96
96.18
30.39
MAPLE
ZS-Eval
78,86
80.60
37.60
87.72
61.14
39.18
58.31
80.75
34.03
54.50
95.49
34.02
PAlign
78.82
80.92
36.62
87.69
61.37
38.01
58.29
80.79
33.17
54.49
95.52
32.96
PAlign-C
81.85
63.37
40.87
89.96
49.09
41.89
59.33
81.48
33.84
53.82
95.17
33.28
ROSITA
97.68
46.51
92.14
34.44
51.33
96.68
25.44
E
Failure Case Analysis
Our findings indicate that while ROSITA outperforms baselines, struggles in cases where undesired classes are highly similar to desired ones (e.g., CIFAR-10C CIFAR-100C), leading to higher false positive rate (FPR). Harder datasets like ImageNet-C pose additional challenges as the Zero-shot classification accuracy of CLIP itself is poor due to severe domain shift and increased number of desired classes. Towards the goal of studying dataset specific challenges, we study the two major components of ReDUCe loss by measuring the following metrics:
(1) KNN Retrieval accuracy: We compute the average number of correctly matched neighbors K+ in Cp (Equation 6), where the pseudo-label yt of the retrieved neighbors matches the test sample's pseudo-label y1-
28

---

Published in Transactions on Machine Learning Research (05/2025)
(2)Pseudo label accuracy of Reliable samples: We evaluate the pseudo-label accuracy of the reliable samples, as only these are stored in the feature bank for kNN retrieval and model adaptation. Higher accuracy of these reliable samples directly benefits adaptation.
Table 23: Analysis on quality of Reliable samples.
Metric
CIFAR-10C
ImageNet-R
ImageNet-C
Average K+ in Lp
4.1
2.5
1.5
Accuracy reliable desired class samples (%) Accuracy reliable undesired class samples (%)
88.7
86.40
43.74
99.75
100
99.95
For LD, the average number of correctly matched neighbors aligns with datasel difficulty; CIFAR-10C has the highest K+ (4.1), while ImageNet-C has the lowest (1.5), reflecting the increased complexity of retrieval in more challenging datasets. Additionally, the high accuracy of reliable desired (2 86% for CIFAR-100C and ImageNet-R) and undesired (nearly 100%) class samples ensures that the retrieved neighbors serve strong supervisory signals for adaptation. Since LD relies only on correctly matched neighbors, the adaptation process remains robust, effectively leveraging reliable samples regardless of dataset difficulty. However, there is still significant scope of improvement desired class accuracy in harder datasets like ImageNet-C, where CLIP accuracy itself still poor, given the severe domain shift and the 1000-way classification task, making it very challenging. The proposed ReDUCc loss in ROSITA primarily focuses on separating desired and undesired class samples. We use simple reliable pseudo label loss Lre and clustering objective Ln that aims towards improving desired class accuracy AccD. More sophisticated methods can be employed in addition to this to improve Accp-
F Broader Impact Concerns
Our work introduces Open-set Single-image Test-Time Adaptation (OSTTA) for Vision-Language Models (VLMs), novel and realistic problem studied for the first time. While VLMs are powerful for open-world recognition, real-world deployment benefits from the ability to recognize unknown objects and adapt effcctively. Our study provides a strong baseline for evaluating their robustness under distribution shifts. While this step forward, some considerations are to be aware of before practical deployment. healthcare, careful adaptation is crucial avoid misinterpretations; in surveillance, fairness needs attention; and in autonomous systems, ensuring reliability is key. That said, our method is designed with privacy in mind it does not store images, only extracted features, minimizing risks. Future work could explore safeguards against potential adversarial attacs but overall, we see this as positive step toward making VLMs more adaptable and reliable in diverse settings.
29


Reviews:
[{"review_id": "FegLNaD6jM", "reviewer": "Reviewer_cGsT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n- The method consistently improves performance while effectively distinguishing undesired classes in open-set recognition situations.\n- Beyond overall performance metrics, Table 1 effectively demonstrates the method's capability in discriminating between desired and undesired classes.\n\n**Weaknesses**\n\nMethodology\n- Limited Methodological Innovation:\n  - The proposed method primarily combines existing techniques rather than introducing innovative components.\n  - The Desired vs Undesired class identifier employs pre-existing 1D Linear Discriminant Analysis (LDA).\n  - Test-time adaptation using kNN and contrastive learning has been previously explored in existing TTA methods [1, 2].\n  - LayerNorm tuning cannot be considered a methodological component as it has been treated as a hyperparameter in prior work including TPT. A more innovative approach or stronger theoretical justification is needed.\n\n- Technical Concerns:\n  - The retrieval of K nearest neighbors using image features (Equation 4) may be unreliable, as recent literature indicates that the CLIP image encoder can ambiguously encode semantically distinct images with high cosine similarity.\n  - Why does LayerNorm tuning outperform other parameter tuning approaches?\n  - How does contrastive learning affect the representation space? Comprehensive analysis and justification are needed.\n\nExperiments\n- In Tables 1 and Table 2, the performance improvements are marginal. The superiority of the proposed method over baselines is not convincing. In particular:\n  - The source of significant performance gain compared to ZSEval requires more explanation.\n  - Performance improvement compared to (K + 1)PC appears only marginal.\n- Table 3 shows occasional drops in FPR under certain settings but does not analyze these variations. The authors should provide potential explanations for these inconsistencies.\n- The choice of MNIST as undesired samples is highly unrealistic for natural image scenarios, as handwritten digits rarely appear among natural images in real-world applications. Using corrupted but semantically similar images would provide a more realistic evaluation scenario.\n- The ablation study in Table 4 is incomplete, focusing solely on AUC/FPR/HM metrics while omitting crucial analysis of how each component affects desired class accuracy.\n- Experiments in Table 5 would benefit from a broader evaluation using various backbones and datasets beyond CIFAR-10C to demonstrate generalizability.\n- Table 7 and Figure 4 require more comprehensive baseline comparisons for memory/time assessment, as relative efficiency metrics are crucial for practical deployment.\n\n[1] Liu, Yuejiang, et al. \"TTT++: When Does Self-Supervised Test-Time Training Fail or Thrive?.\" NeurIPS, 2021.\n\n[2] Chen, Dian, et al. \"Contrastive Test-Time Adaptation.\" CVPR, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This research explores **Open-set Single image Test Time Adaptation (OSTTA)** for Vision-Language Models, addressing the complex task of continuous model adaptation to individual test samples while maintaining the ability to identify unknown classes—a critical challenge in real-world applications where test samples arrive sequentially and may contain unseen categories. \n\n- The proposed solution, **ROSITA**, combines a contrastive learning-based **ReDUCe** loss with optimized LayerNorm parameter adaptation, enabling efficient separation between desired and undesired classes without requiring batch processing or closed-set assumptions that limit existing approaches. The framework not only establishes new benchmarks for OSTTA but also shows remarkable performance in managing domain shifts and unknown class recognition across diverse adaptation scenarios, demonstrating its practical utility in dynamic real-world environments.\n\n- Through extensive experiments, the authors validate ROSITA's effectiveness in both adapting to domain shifts within known categories and accurately identifying unseen classes as \"unknown,\" while maintaining computational efficiency suitable for real-time deployment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Questions**\n- What is the conceptual justification for Equation 7?\n- What insights does the gradient analysis provide, and how does it contribute to understanding the method's effectiveness?\n\n**Writing Suggestions**\n- Equation-related issues:\n  - Always include \",\" or \".\" after equations by considering each equation is part of a complete sentence.\n  - Fix missing equal signs \"=\" in Equations 17 and 18 (Page 22).\n- Notation standardization:\n  - $Q^d$, $Q^u$ and $Q_d$, $Q_u$ are used interchangeably throughout the paper - maintain consistency with one format.\n  - Fix inconsistent set notation: Change \"Cd = {c1, c2, . . . cN} / {t1, t2, . . . tN }\" to \"Cd = {c1, c2, . . ., cN} / {t1, t2, . . ., tN }.\"\n- Terminology consistency:\n  - \"open-set\" vs \"open set\"\n  - \"test-time\" vs \"test time\"\n  - Recommend using hyphenated versions (\"open-set\" and \"test-time\") consistently throughout\n- Citation formatting:\n  - Use \\citep instead of \\cite for most cases (e.g., in TPTShu et al. (2022))\n\n- Specific page corrections:\n  - Page 8: \"Appendix B.4 We\" should be \"Appendix B.4. We.\"\n  - Page 10: \"Similarly, we use LU ( Equation 7)\" should be \"Similarly, we use LU (Equation 7).\"\n  - Page 22: Add missing equal signs in Equations 17 and 18.", "broader_impact_concerns": "There are no concerns in this aspect.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "08cLclYC0f", "reviewer": "Reviewer_63vr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n1. Innovative Approach to Open-set TTA: The paper introduces a novel approach for Open-set Single Image Test Time Adaptation (OSTTA) using Vision-Language Models (VLMs). By distinguishing between \"desired\" and \"undesired\" classes, it addresses a real-world scenario where models must not only adapt to domain shifts within known classes but also reject unfamiliar samples, providing significant improvements over previous methods.\n2. Efficient Adaptation: The framework emphasizes computational efficiency, especially through the identification of LayerNorm parameters as optimal for continuous adaptation of VLMs. The paper provides a comprehensive benchmark for OSTTA using VLMs and compares ROSITA with a variety of existing methods, demonstrating clear superiority in terms of accuracy and efficiency. \n3. Theoretical Justification and Gradient Analysis: The gradient analysis for the ReDUCe loss adds strong theoretical grounding to the method, making the approach more rigorous and robust. This detailed analysis helps clarify how contrastive loss works to improve the separation between classes.\n4. Realistic Test Scenarios: The authors simulate several real-world-inspired test scenarios (e.g., changing domains, varying sample ratios), which provides a practical context for evaluating the method’s effectiveness. \n### Weakness\n1. The robustness in real-world application: The tuning of VLM is highly sensitive to the learning rate, which can lead to unstable performance, and the experiments are based on relatively small-scale datasets, limiting the method's ability to handle larger, more complex data distributions in practical deployment scenarios. These factors should be addressed to improve its reliability in dynamic, real-world environments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the challenge of Open-set Single Image Test Time Adaptation (OSTTA) in dynamic real-world environments, where models must adapt to sequential test samples and distinguish between known and unknown classes. The authors introduce ROSITA, a novel framework that leverages dynamically updated feature banks and a contrastive learning objective to improve the separation between desired and undesired classes. The key contributions include the establishment of a benchmark for OSTTA using Vision-Language Models (VLMs), the introduction of the ReDUCe loss to enhance class separability, and the identification of LayerNorm parameters as an efficient adaptation strategy for VLMs in open-set scenarios. Extensive experiments show that ROSITA achieves state-of-the-art performance, offering strong accuracy and computational efficiency for real-time deployment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide More Detailed Explanations of Symbols and Terminology: The paper would benefit from more detailed explanations of the various symbols used, particularly in the loss function. For instance, it would be helpful to specify what each positive and negative sample refers to in the context of the loss. Clarifying these elements would enhance the paper's readability and ensure that readers can more easily follow the methodology.\n2. Emphasize the Difference Between \"Desirable\" and \"Reliable\" Samples: Given the importance of both \"desirable\" and \"reliable\" samples in the proposed method, a clearer distinction between these two concepts is needed. These terms have a conceptual overlap, and further discussion of their differences and mechanisms would help avoid confusion. Expanding on this distinction and providing a more in-depth explanation of their roles in the method would improve both the clarity and depth of the paper.", "broader_impact_concerns": "The paper does not address the ethical implications of the proposed method, especially regarding real-world applications like healthcare, surveillance, or autonomous systems.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PQTtfHjk9b", "reviewer": "Reviewer_hu7J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The formulation of single-image based TTA adaptation with open-set constraints is a valid problem setting and closely mirrors the realistic environment that foundational models would face when deployed in real-world settings.\n- The idea of utilizing a two-step inference procedure and the use of contrastive loss in the proposed ROSITA setup is motivating.\n- The study regarding VLM fine-tuning strategies at test time is valuable and can serve as a clear reference for anyone pursuing fine-tuning of VLMs at test time.\n- Overall, extensive experiments have been provided, and the proposed solution leads to improved results compared to baselines and existing leading methods.\n- Important ablation studies have been conducted in a timely manner, and main design choices are properly motivated using analysis experiments.\n\n**Weaknesses** \n- My major concern is the use of memory banks in the proposed adaptation framework. Memory banks indirectly mean storing the exact samples (albeit in latent space). This raises many questions, e.g., why not directly use batch samples? Therefore, this framework might violate the regulation that it requires only a single test image for adaptation. This is a very critical point, and this work must highlight/resolve this aspect in the framework.\n- As the method is continuous, it is still not clear how the performance varies for the initial test samples when the memory features are not yet updated (or not present) since no samples have been seen at the beginning.\n- Additionally, the method strongly relies on the quality/difficulty of the initial test samples. In my understanding, if initial samples are more tricky/difficult, then the resulting feature banks will have noisy estimates and may even degrade performance for future samples. Similarly, if initial test samples are easy to predict correctly, the method will be more effective for future test image predictions. Unfortunately, these important investigations are not studied/highlighted in the current manuscript. It would be very important to study the impact of such cases for the proposed framework.\n- Ablation studies on the memory banks module should be included in the manuscript.\n- Although this work has presented an in-depth study for evaluating various fine-tuning strategies, I believe it is important to also discuss the effect of LoRA adaptation strategies for test-time adaptation. I understand it is not the most important part, but as a contribution listed in your manuscript, it would be good to fully cover all aspects of this direction. Lastly, there are recent works that explore TTA using LoRA [1], which can be a good reference for discussion.\n\n- Based on the results, it would be good to have an analysis of failure cases and qualitative results, especially on challenging datasets that contain many classes. This would be a good resource for follow-up works building on the proposed method.\n\n[1] Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models\n\n\n\n**Minor comments**\n- There are a few spacing errors in the writing. E.g., TPT citation on page 2. \n- As referenced by this work, there exists open-set TTA methods (vision-only) and benchmarks but they might not conform to the single-image case. Does the proposed benchmark use the same datasets and benchmarks as these previous open-set works? Actually, it would be good to have a discussion on the benchmarks, and if they can be unified together (because only exception is the use of single image) for easy benchmarking process.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work explores and identifies a special case of test-time adaptation (TTA), called open-set TTA, using a single image sample. Existing variants of TTA do not cater to realistic setups where only a single test image is available during inference. To provide a standardized setup, this work proposes a benchmark for this problem setting, called OSTTA. Secondly, a new learning-based OSTTA framework for VLMs is proposed, called ROSITA. The main formulation of ROSITA utilizes desired samples (from known classes) and undesired samples (from unknown classes) to adapt VLMs at test time using the proposed ReduCE loss. ReduCE utilizes contrastive learning and pseudo-label-based objectives to continuously adapt the model for test-time images. Specifically, to gather the positive and negative samples for the contrastive loss, separate memory feature banks are used, which are updated/collected in an online fashion during the inference lifecycle. Finally, it is shown that tuning the layer-norm leads to improved TTA performance compared to other fine-tuning configurations. Detailed experiments are conducted, and the performance of the proposed framework is thoroughly compared with previous leading methods. The paper concludes with extensive ablation studies to motivate the design choices of the proposed framework.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have added my comments in the weaknesses section of the review. It is requested that the authors provide a clear response that addresses these issues, which is crucial for the overall positive review decision. \n\nI am looking forward to the response from the authors!", "broader_impact_concerns": "Discussion on the broader impact concerns is not provided in the manuscript.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes ROSITA, a test-time adaptation framework composed of a new loss function suitable for open-set test-time adaptation, paired with a study of the optimal parameters to fine-tune at test.

I would like to follow the recommendation of all reviewers to accept the paper, however minor revisions are required to incorporate key results of the author-reviewer discussion phase more prominently into the main paper. Three important reasons for revisions I see are:

- The results from the rebuttal ([see here](https://openreview.net/forum?id=72YVabBErN&noteId=2Fctne0uAa)) are currently not fully worked into the paper and right now only briefly mentioned in page 12, “Appendix”. It would be good if the authors extended their discussion section and commented on the key insights generated during the review/discussion phase.
- Limitations of the benchmark (e.g. using MNIST prominently as distractor samples) and the reply the authors gave on this point during the rebuttal phase should be worked into the main paper. It is also a limitation that the authors did not benchmark the performance of the method in full absence of distractor samples (i.e., in standard TTA settings), which could be made transparent.
- The repository should be updated to reflect the state of the experiments post-rebuttal, to meet the claim of “establishing a comprehensive benchmark for Open-set Single image Test-Time Adaptation”, or the claim should be tuned down.
Decision: Accept (poster)