Title: OOD-Barrier: Build a Middle-Barrier for Open-Set Single-Image Test Time Adaptation via Vision Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Open Set Learning

OOD Detection

Vision Language Models
Abstract: In real-world environments, a well-designed model must be capable of handling dynamically evolving distributions, where both in-distribution (ID) and out-of-distribution (OOD) samples appear unpredictably and individually, making real-time adaptation particularly challenging. While open-set test-time adaptation has demonstrated effectiveness in adjusting to distribution shifts, existing methods often rely on batch processing and struggle to manage single-sample data stream in open-set environments. To address this limitation, we propose Open-IRT, a novel open-set Intermediate-Representation-based Test-time adaptation framework tailored for single-image test-time adaptation with vision-language models. Open-IRT comprises two key modules designed for dynamic, single-sample adaptation in open-set scenarios. The first is Polarity-aware Prompt-based OOD Filter module, which fully constructs the ID-OOD distribution, considering both the absolute semantic alignment and relative semantic polarity. The second module, Intermediate Domain-based Test-time Adaptation module, constructs an intermediate domain and indirectly decomposes the ID-OOD distributional discrepancy to refine the separation boundary during the test-time. Extensive experiments on a range of domain adaptation benchmarks demonstrate the superiority of Open-IRT. Compared to previous state-of-the-art methods, it achieves significant improvements on representative benchmarks, such as CIFAR-100C and SVHN — with gains of +8.45\% in accuracy, -10.80\% in FPR95, and +11.04\% in AUROC.

OOD-Barrier: Build a Middle-Barrier for Open-Set Single-Image Test Time Adaptation via Vision Language Models

Abstract

In real-world environments, a well-designed model must be capable of handling dynamically evolving distributions, where both in-distribution (ID) and out-ofdistribution (OOD) samples appear unpredictably and individually, making realtime adaptation particularly challenging. While open-set test-time adaptation has demonstrated effectiveness in adjusting to distribution shifts, existing methods often rely on batch processing and struggle to manage single-sample data stream in open-set environments. To address this limitation, we propose Open-IRT, a novel open-set Intermediate-Representation-based Test-time adaptation framework tailored for single-image test-time adaptation with vision-language models. OpenIRT comprises two key modules designed for dynamic, single-sample adaptation in open-set scenarios. The first is Polarity-aware Prompt-based OOD Filter module, which fully constructs the ID-OOD distribution, considering both the absolute semantic alignment and relative semantic polarity. The second module, Intermediate Domain-based Test-time Adaptation module, constructs an intermediate domain and indirectly decomposes the ID-OOD distributional discrepancy to refine the separation boundary during the test-time. Extensive experiments on a range of domain adaptation benchmarks demonstrate the superiority of Open-IRT. Compared to previous state-of-the-art methods, it achieves significant improvements on representative benchmarks, such as CIFAR-100C and SVHN - with gains of +8.45% in accuracy, -10.80% in FPR95, and +11.04% in AUROC.

1 Introduction

In real-world environments, the model's ability to perform real-time adaptation is particularly crucial for handling the emergence of an unknown category or distributional shifts. This capability is essential in safety-critical applications such as autonomous driving, where failure to adapt can have serious consequences. Despite these demands, most approaches in computer vision [1-3] assume a closed-set paradigm, where training and testing data come from the same distribution. In contrast, real-world scenarios frequently encounter open-set environments [4-6], where models must handle unknown distributions and unseen categories, as shown in Fig. 1. This fundamental deviation from the closed-set assumption introduces fundamental challenges in maintaining model performance.

This transition from closed-world to open-set learning necessitates requiring innovative strategies to effectively filter, classify, and adapt to both distributional and semantic shifts without explicit supervision [7]. However, traditional approaches based on predefined categories are not equipped

Test-time Parameter Updating

Separate ID/OOD Samples

OOD Barrier Method to handle the emergence of unknown categories in open-set environments, particularly in real-time inference and single-sample adaptation constraints. As a result, accurate and efficient OOD detection has emerged as a critical research focus [7-9] in dynamic, open-set scenarios. Update OOD Sample Features ID Sample Features

With the advent of visionlanguage models (VLMs), recent studies [10, 11] have leveraged their strong generalization for OOD detection tasks. Models such as CLIP [10] facilitate adaptation to unseen categories in openset situation by learning rich cross-modal representations. Furthermore, recent research [12] have demonstrated that VLMs can perform zero-shot reasoning based on imagetext associations and further adapt to single image inference, which enhances both their zero-shot generalization and OOD detection performance.

Confusion in Overlap Space

Barrier and Separate

ID

ID

OOD

OOD

…

ID

OOD

ID

Generalization Area

Overlap

Open World Single-Image Stream

(a) Original Feature Space

ID

OOD

(b) Feature Space after Open-IRT

OOD

ID

MID

OOD

ID

MID

Build a Middle Domain

Separate ID and OOD

X

In-Distribution OOD Distribution Test-time adaptation [13-16] offers a promising approach for adapting models to incoming data streams during inference, without relying on labeled data. Recent developments have extended this paradigm to single-sample test-time adaptation [17, 18], which is particularly valuable in domains such as security surveillance and industrial inspection, where real-time adaptation to dynamic environments is critical. However, most existing single-image test-time adaptation methods mainly focus on closed-set assumptions, whereas open-set batch processing strategies struggle to effectively adapt to changes in individual samples, particularly in real-time scenarios where only a single image is available for adaptation, as illustrated in Fig. 1.

To overcome this limitation, we focus on Open-Set Single-Image Test-time adaptation setting, where the model dynamically adapts to each input sample during inference without relying on batch data. Specifically, we propose a novel Open -set I ntermediateR epresentation-based T est-time adaptation ( Open-IRT ) framework for single-sample test-time adaptation in OOD detection. The overall motivation is illustrated in Fig. 1. As illustrated in Fig. 1b, Open-IRT establishes a structured separation between ID and OOD samples in feature space. The framework consists of two key modules. First, we introduce a P olarity-aware P rompt-based OOD F ilter ( PPF ) module in Fig. 2a. Here, the term 'polarity' refers to our utilization of the disparity between positive and negative prompts. PPF leverages the rich cross-modal information in vision language models to fully construct the ID-OOD distribution from both positive and negative prompts. It is guided by Semantic Contrast Hypothesis 1, which considers the absolute semantic alignment and relative semantic polarity. Next, we introduce I ntermediate D omain-based T est-time adaptation ( IDT ) module in Fig. 2b based on Intermediate-Domain Hypothesis 2, which indirectly decomposes the distributional discrepancy between ID and OOD representations by modeling an intermediate domain that bridges the gap between these two. As shown in Fig.4, the IDT explicitly models this middle domain, and the learning objective enforces divergence of both ID and OOD samples from the intermediate domain. Therefore, this strategy effectively enlarging the distance between ID and OOD distributions indirectly. In addition, IDT uses a dynamic threshold strategy to generate bidirectional pseudo-labels, encouraging the model to reinforce positive feature representations and suppress intra-class noise.

The main contributions are summarized as follows: (i) We propose the PPF, an effective OOD filtering mechanism based on Semantic Contrast Hypothesis 1. The PPF captures both absolute semantic alignment and relative semantic polarity between an input and its paired prompts, enabling effective ID-OOD separation. (ii) We introduce the Intermediate-Domain Hypothesis 2, which leads to the development of IDT module. This module construct a intermediate domain strategy to establish a real-time two-way repulsion constraint between ID and OOD feature distributions, enhancing ID-OOD separation indirectly. (iii) Open-IRT consistently outperforms prior state-of-the-art methods

on standard benchmarks, including ImageNet-C [19], Tiny-ImageNet [20], VisDA [21], CIFAR10C/100C [19], and digit datasets [22-24]. For instance, on CIFAR-100C and SVHN, it achieves +8.45% accuracy, -10.80% FPR95, and +11.04% AUROC, supporting our motivation and hypothesis.

2 Related Works

2.1 Out-of-Distribution Detection

Out-of-distribution (OOD) detection aims to determine whether a given sample originates from the training distribution or from an unseen distribution. The OOD detection approaches can be broadly categorized into classification-based methods [7, 25, 26] and density-based methods [8, 27]. Classification-based OOD methods relied on a maximum softmax probability to distinguish between ID and OOD samples. These include post hoc techniques such as ODIN [28], which utilizes temperature scaling and input perturbation. In addition, there are classic approaches such as JointEnergy scores [26], Mahalanobis distance [29], and activation space-based techniques [30, 31] that enhance the separability of ID and OOD samples without altering the training process. On the other hand, density-based methods use probabilistic models, such as class-conditional Gaussian distributions [29] and flow-based models [32, 9], to identify OOD samples based on their likelihood. To address high likelihoods challenges of OOD, techniques such as likelihood ratio [33], likelihood regret [34], and SEM scores [35] have been proposed. However, traditional methods are predominantly designed for the training phase, limiting their ability to adapt to distribution and semantic shifts in real-time.

2.2 OODDetection with Vision Language Models

Vision-language models have gained significant attention in recent years for their ability to integrate visual and textual information. Renowned vision-language models, such as CLIP [10] and MaPLe [11], achieve impressive results by training on large-scale image-text pairs. To adapt vision-language models for downstream tasks (e.g., OOD detection), additional lightweight modules have been introduced, including prompt learners [36, 37], vision adapters [38, 39], and LoRA [40, 41].

Recent research in OOD detection has begun to utilize vision-language models as auxiliary tools, starting with CLIP [42], which aims to distinguish samples that do not belong to any ID class text provided by the user [43]. These approaches often employ techniques such as OOD label retrieval [44], generation [45], or alignment [46]. Training-free methods such as MCM [47] detect OOD using only ID labels, while auxiliary training-based methods, such as CLIPN [48] leverage additional pre-training to enhance OOD detection. From the perspective of specific training methodologies, some approaches implement a specialized handling of prompt words [49, 50], while NegPrompt [51] further explores the use of negative prompt techniques. Moreover, ROSITA [52] consider this task in test-time, yet it remains constrained by its reliance on direct feature alignment. Unlike previous methods, Open-IRT is fundamentally guided by an intermediate domain located near the boundary of the feature space, which indirectly decomposes the distributional discrepancy between ID and OOD representations. It introduces a bidirectional repulsion constraint, combining semantic alignment with relative semantic polarity, to increase the distributional separation.

2.3 Test-Time Adaptation

Test-time adaptation, originating from domain adaptation [53-55], adapts pre-trained models to test data with distribution shifts, without requiring training data. Test-time adaptation is essential for realworld applications, such as autonomous driving in diverse weather conditions. Several methods have been proposed, such as adjusting partial model weights [13, 56] or normalization statistics [14, 15]. Specifically, TENT [15] adapts batch normalization layers by entropy minimization, TTT [56] updates classification layers during testing, and T3A [13] introduces an optimization-free classifier adjustment. As a technology for changing environments, test-time adaptation can be integrated with OOD detection (e.g., RTL [57], UniEnt [58]). However, they typically address the task using batch processing methods, which limits their ability to handle dynamic scenarios effectively.

To reduce reliance on multiple test samples, some approaches prioritize single-sample adaptation. MEMO[59] enforces consistency through augmentations of the same test sample, while TPT [12] fine-tunes prompts for vision-language models during testing. DiffTPT [60] enhances this by using pre-trained diffusion models to augment test data. TDA [61] addresses computational efficiency

gv

Density

Score

μ

2

μ

1

Open World Single-Image Stream

ID Feature Bank

Positive Prototypes

Negative Prototypes

KNN

KNN

Lmid

ID

ID

OOD

OOD

…

gt

ID

Text Encoder

A photo of a

<ID class>

A photo of a no

<ID class>

if ID

-LCE

if OOD

Middle domain

Freeze

Update

(b) IDT:Intermediate Domain-based TTA

(a1) PPF:Score Calculation

Simpos

Simneg

Simdiff

Simpos

Simneg

Simpos +

α

Simdiff

Vision Encoder

OOD

Score Bank

GMM

(a2) PPF:OOD Filtering

ŷ

μ

ID

μ

OOD

σ

OOD

σ

ID

Feature

μ

ID

σ

ID

Feature

μ

OOD

σ

OOD

+LCE

Conf >

μ

I

Conf <

μ

O

OOD Sample Features

ID Sample Features

Scores

Embed

OOD Feature Bank

with a cache-based model. While these methods assume closed-set settings, Open-IRT enables single-image Test-time adaptation in open-set environments. Unlike continual test-time adaptation, it further considers semantic shift in addition to distribution shift and refines the ID-OOD boundary in terms of the intermediate representation.

3 Methodology

In open-world environments with real-time perception demands, models frequently encounter singleimage data streams that contain OOD samples. To address this challenge, we propose Open-set Intermediate-Representation-based Test-time Adaptation (Open-IRT) in Fig. 2. Open-IRT is an open-set, single-image test-time adaptation strategy composed of two key modules: Polarity-aware Prompt based OOD Filter module and Intermediate Domain-based Test-time adaptation module.

3.1 PPF: Polarity-aware Prompt-based OOD Filtering Mechanism

As shown in Fig. 2a, we introduce the Polarity-aware Prompt-based OOD Filter (PPF) module, which improves ID-OOD separability by exploiting the dual-polarity semantics of vision-language models. Specifically, we design positive prompts p c (e.g., 'a photo of a [CLS]') to represent ID prototypes, and negative prompts p ′ c (e.g., 'a photo of no [CLS]') to encode inverse semantics [48]. Given a vision feature f = g v ( x ) after L2 normalization, we compute its cosine similarity with both the positive

Hypothesis 1 (Semantic Contrast Hypothesis) . For ID samples ∀ c ∈ [ C ] , f ∼ P I and OOD f ∼ P O:

where thresholds satisfy τ I > τ O and ∆ I > ∆ O.

First, we propose the Semantic Contrast Hypothesis. For ID samples, the alignment with positive prompts is strong, i.e., sim( f, p c ) is high, and the polarity gap | sim( f, p c ) -sim( f, p ′ c ) | is also large, leading to a confidently larger overall sum. For OOD samples, the alignment with positive prompts is weaker, and the polarity gap is less marked due to semantic ambiguity, leading to suppressed overall sum. The proposed scoring function S ( f ) jointly captures absolute semantic alignment and

relative semantic polarity between an input and its paired prompts. This promotes effective ID-OOD separability by maximizing the distributional contrast between them.

where ϕ is a min-max normalization operator to ensure the score lies within the range [0 , 1] , and α controls the contrast intensity. As shown in Fig. 3, while individual sample have unique semantics, the resulting aggregate score distribution exhibits a clear bimodal pattern, demonstrating consistent statistical regularity across large-scale data. Furthermore, the introduction of the polarity gap term significantly enhances ID-OOD separation, thereby validating the Hypothesis 1. This dual-polarity design-leveraging alignment with positive prompts and contrast with negative prompts-offers an effective mechanism for ID-OOD distinction. Detailed theoretical analyses are provided in appendix.

Then, we introduce the score bank B s and feature bank B f , both updated using a sliding window strategy, where it functions equivalently to a FIFO queue. The score bank B s stores the scores of individual samples and is divided into two components: ID ( B s I ) and OOD ( B s O ) score banks, based on the Gaussian Mixture Model as described in Eq. 4.

Here, π ( x ) denotes the probability that S ( x ) belongs to the ID class, and µ s I/O , σ s I/O 2 are the mean and variance of the ID/OOD components. The probability π ( x ) is computed using the Expectation-Maximization algorithm.

Upon the arrival of a new sample x , its score S ( x ) is appended to both B s I and B s O , and Eq. 4 is used to classify it as either ID ( ˆ b = 1 ) or OOD ( ˆ b = 0 ). To mitigate the impact of limited samples during the early stages of learning, we employ bootstrapped resampling, which helps prevent overfitting to sparse observations. The classified features are then inserted into the corresponding feature banks B f I/O for use in the subsequent module.



























































3.2 IDT: Intermediate Domain-based Test-time adaptation

We now introduce the Intermediate Domain-based Test-time adaptation (IDT) module, which leverages the feature banks B f I and B f O to further refine adaptation. To address the challenge in Fig. 1, we first propose Hypothesis 2 that models the intermediate domain ( F M ).

Hypothesis 2 (Intermediate Domain Characterization) . Let F I ≜ { f | f ∼ P I } and F O ≜ { f | f ∼ P O } denote the ID/OOD feature distributions respectively. There exists a measurable transformation T : F I ∪ F O → F M such that for the induced intermediate feature space F M that satisfies the following approximate equality:

where d H represents the Hilbert-Schmidt independence criterion-based dissimilarity.

Rather than directly maximizing the ID-OOD distance ( F I , F O ) , which is challenging due to limited knowledge of OOD distribution, we instead construct an intermediate domain F M to act as a bridge. By encouraging both ID and OOD samples to move away from this intermediate domain (i.e., increasing d H ( F I , F M ) and d H ( F O , F M ) ), we indirectly enhance the overall discrepancy d H ( F I , F O ) . This aligns with the intuition of margin-based separation in representation space.

Contrastive Learning-based Middle Domain Loss. We first model the intermediate domain feature f m as per Hypothesis 2. To normalize the sample features f s , we compute their mean and variance

based on augmented features, which is derived the f s as Eq. 6. The method for calculating f s has been validated for its rationality in [62], where it is computed across spatial dimensions independently for each feature channel. The feasibility of this approach lies in preserving channel-specific style information. The variance for each channel emphasizes fine-grained structural characteristics in that feature dimension, such as color, texture intensity, etc., without mixing with other channels, thereby achieving refined style representation and alignment.

Then, we extract the style features stored in the feature memory banks B f I/O to compute µ f I/O and σ f I/O . The design of intermediate style feature f m is theoretically motivated by domain adaptation principles [62], which suggest that constructing an intermediate feature space through style-transferlike transformations can effectively position the new domain between source and target domains, enabling more controlled feature interpolation. f m is then re-assigned as follows.

The confidence Conf(x) is defined in Eq. 11. To further enhance interclass discriminability and suppress intra-class noise, we introduce two thresholds, µ I and µ O . They represent the mean confidence scores of the accumulated 512 ID/OOD Conf ( x ) , respectively, updated by a sliding window mechanism. If Conf ( x ) > µ I , we treat the input as a confident ID sample. If Conf(x) < µ O , it is more likely to be a noisy sample or an OOD sample6. The effectiveness of f m construction is visually confirmed in Fig. 4, where the T-SNE visualization shows a clear separation between ID

T-SNEwithID,OOD and MiddleDomainFeatures

1.0

ID

OOD

0.8

MID

0.6

0.4

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0

and OOD features, with a distinct intermediate cluster. This indicates that the constructed intermediate representations indeed satisfy the desired relational constraints in the feature space, thereby validating the theoretical motivation of Hypothesis 2. Then, we select the K nearest neighbors N I and N O , for the sample feature f from the feature banks B f I and B f O , respectively. When the sample is confidently distinguished as ID ( ˆ b = 1 and Conf ( x ) > µ I ), the loss L I based on InfoNCE loss [63] is constructed as follows.

where K + = ∑ z + ∈N I 1 ( y + = ˆ y p ) denotes the number of positively matched neighbors with the pseudo label ˆ y p . For confident ID samples, the objective of L I is to align with ID while minimizing similarity to OOD and distancing from the intermediate feature f m .

For confident OOD samples ( ˆ b = 0 and Conf ( x ) < µ O ), the objective of L O is to improve sensitivity to OOD samples by pulling OOD features away from both ID and the intermediate domain feature.

The differences between Eq. 8 and Eq. 9 arise from the distinction between ID and OOD categories in the standard OOD detection task. For ID data, the model needs to learn fine-grained semantic consistency and achieve precision for each class, which is why we set 1 ( y + = ˆ y p ) in Eq. 8.

Threshold-based Bidirectional Pseudo Loss. To facilitate positive knowledge while mitigating intra-class noise, we design an effective pseudo-label strategy. First, we calculate the confidence Conf ( x ) using the maximum cosine similarity between feature g v ( x ) and classifier weights C :

We analyze Conf ( x ) from two perspectives: For high-confidence scenarios, we align predictions with ˆ y p . For low-confidence scenarios, ˆ y p may introduce noise, so we stop relying on it and use reverse optimization to push predictions away from it, mitigating the risk of misclassification.

If Conf ( x ) > µ I , we treat the input as an confident sample and optimize the model to align predictions with ˆ y p using L CE , enhancing the inter-class discriminability. If Conf ( x ) < µ O , we deviate the prediction results from the low-quality pseudo-label ˆ y p by -L CE , suppressing the intra-class noise.

Total Test-time Adaptation Loss. The total loss function L TTA combines the pseudo-label loss L psd and the intermediate domain loss L mid . L psd aims to enhance inter-class discriminability and suppress intra-class noise, while L mid indirectly enlarge the ID-OOD distance by increasing both d H ( F I , F M ) and d H ( F O , F M ) in Hypothesis 2. This combination enables dynamic adjustment of the model's adaptation strategy.

During test time, each incoming sample is used to update the model parameters via backpropagation with the test-time adaptation loss L TTA . Subsequently, the same sample is used to compute the evaluation metrics. This online update-then-evaluate approach simulates a realistic scenario where the model continuously adapts to distributional shifts without access to ground-truth labels.

4 Experiments

4.1 Implementation Details

Datasets. We utilize representative ID and OOD datasets to ensure a complete evaluation. For ID datasets, we leverage CIFAR-10C/100C [19], ImageNet-C [19], and VisDA [21]. The OOD datasets include MNIST [22], MNIST-M [23], SVHN [24], Tiny-ImageNet [20], and CIFAR-10C/100C [19].

Metrics. We utilize the Area Under the Receiver Operating Characteristic Curve (AUROC), the False Positive Rate at 95% True Positive Rate (FPR95) and Acc HM as main metrics. Here, Acc HM denotes the harmonic mean of Acc I and Acc O , where Acc O is the precision of ID-OOD binary classification, and Acc I is the general accuracy to correctly identify ID categoriess.

Baselines. We utilize the CLIP [10] and MaPLe [11] as models, which are based on the ViT-B16/ViTB32 [64] architectures. We adopt ZS-Eval [52], TPT/TPT-C [12], PAlign/PAlign-C [46], TDA [61], DPE [65], UniEnt [58], OWTTT [66] and ROSITA [52] as baselines.

Details. All experiments are reproduced based on publicly available code, with ImageNet-C experiments conducted on an NVIDIA A6000 GPU and all other experiments on an NVIDIA 3090 GPU. In main experiments, the test size for both ID and OOD datasets is 10,000, except for the VisDA in

4.2 Main Result

Cifar Benchmark. The cifar benchmark leverages CIFAR-10C/100C [19] as ID datasets, and MNIST [22], SVHN [24], and Tiny-ImageNet [20] as OOD datasets. CIFAR-100C/10C is also treated as an OOD case with small distribution shifts, applying label offsets to distinguish from ID data. As shown in Table 1, Open-IRT has demonstrated significant performance advantages. For example, in the CIFAR-10C → SVHN case with CLIP, Open-IRT achieved notable improvements by +1.58% AUROC, -12.88% FPR95, and +1.50% Acc HM . Additionally, for the CIFAR-100C → SVHN case with CLIP, Open-IRT achieves an +8.45% in Acc HM , -10.80% in FPR95, and +11.04% in AUROC, as detailed in appendix. OpenIRT also achieves better results in MaPLe, a multi-modal fine-tuning framework for CLIP, due to its adaptability across fine-tuning methods through direct feature space operation. Furthermore, Open-IRT exhibits obvious improvements in the FPR95 (up to -12.88%), highlighting its effectiveness in reducing the misclassification of OOD samples. This can be attributed to Open-IRT's ability to model the ID-OOD distribution effectively (see Fig. 3).























































































































(a) Analyze of Open-IRT with GMM and LDA.

(b) Backbone ViT-B/16, ViT-B/32 and ResNet50.

VisDA and ImageNet Benchmarks. The VisDA [21] and ImageNet-C [19] serve as ID datasets, while MNIST [22], MNIST-M [23], and SVHN [24] serve as OOD datasets. The complexity of VisDA and ImageNet-C introduces additional challenges for OOD detection. In Table 2, Open-IRT achieves notable improvements over existing baselines. For example, in ImageNet-C → MNIST experiment with CLIP, Open-IRT achieves gains of +0.47% AUROC, -7.49% FPR95, and +3.75% Acc HM . Similarly, in Table 3, Open-IRT achieves gains of +1.58% AUROC, -7.46% FPR95, and +1.09% Acc HM in VisDA → MNIST-M with CLIP. Moreover, compared to Table 1, when the ID dataset becomes more complex (e.g., ImageNet-C), the model's performance in OOD detection tasks does not fully saturate, resulting in a relatively lower Acc HM . In such cases, Open-IRT's improvement in Acc HM is more obvious, such as a +5.12% increase in the ImageNet-C → SVHN experiment with CLIP. Additionally, Open-IRT achieves gains of -16.86% FPR95 and +6.55% AUROC. Since Open-IRT operates primarily in the feature space and enhances the separation of the ID-OOD boundary (see Fig. 4), its effectiveness becomes increasingly apparent as the feature distribution grows more complex.

4.3 Experiment Analysis

Score Ablation in PPF. We assess the effectiveness of our PPF mechanism in Fig. 5a, comparing PPF with GMM and LDA [67]. The results indicate that the PPF with GMM performs better results, such as -2.55% FPR95 improvement in CIFAR-10C → Tiny-ImageNet task. This performance gain can be attributed to GMM's ability to model complex and non-linearly separable feature distributions, which more accurately capture the characteristics of real-world domain shifts. In contrast, LDA relies on the assumption of linear decision boundaries, which limits its capacity to handle such complexities.

Loss Ablation in IDT. We analyze the effectiveness of each component of IDT in Eq.14, as shown in Table 4, with the first row representing the zero-shot evaluation scenario. The results demonstrate that L mid significantly improves FPR95 and AUROC, enhancing ID-OOD binary classification by increasing the separation between ID and OOD samples in feature space. On complex datasets like VisDA, L mid outperforms L psd by +1.99% and +3.62% in Acc HM , respectively. However, on simpler datasets like CIFAR-10C, L psd performs better in Acc HM as it helps reduce intra-class noise, which is more beneficial for simpler features. Furthermore, setting λ = 0 to evaluate the intermediate-domain strategy shows that introducing the intermediate-domain improves performance by up to -5.94% in FPR95, supporting our Hypothesis 2.

Analyze Backbones. In Fig. 5b, we utilize ViT-B/16, ViT-B/32 and ResNet50 as the backbones in CIFAR-100C → MNIST task with CLIP. The results demonstrate that Open-IRT achieves the highest Acc HM (e.g., 59.56% in ViT-B/16, 58.62% in ViT-B/32) and the lowest FPR95 (e.g., 11.29% in ViT-B/16, 3.74% in ViT-B/32). These results highlight the robustness of Open-IRT, as it directly optimizes the feature space, independent of the backbone type, demonstrating its broad applicability.

Varying OOD Ratios. To examine the robustness of Open-IRT, we conduct CIFAR10C → MNISTexperiments on CLIP under different OOD ratios with additional experiments are in appendix. As shown in Table 5, Open-IRT consistently achieves better results across all OOD ratios with Acc HM fluctuate by only 1.01%. This stability suggests that Open-IRT is better suited to handle the uncertainties associated with fluctuating OOD proportions.

T-SNE after TPT

T-SNE after ROSITA

T-SNE after Open-IRT

1.0

1.0

1.0

1.0

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

T-SNE Visualization. We conduct T-SNE experiments of CIFAR-10C → MNIST. The T-SNE visualization in Fig. 6 reveals that Open-IRT achieves better separation and more compact clustering within the ID classes, while also establishing a clearer boundary between ID and OOD classes.

5 Conclusion

In this paper, we address the challenges of real-time adaptation in open-set environments with singlesample stream and propose the Open-set Intermediate-Representation-based Test-time adaptation (Open-IRT) framework. Its Polarity-aware Prompt-based OOD Filter module leverages the rich crossmodal information of vision language models, corporating both absolute semantic alignment and relative semantic polarity. The Intermediate Domain-based Test-time adaptation module constructs an intermediate domain and indirectly decomposes and enlarge the ID-OOD distributional discrepancy in real-time. Experiments across various benchmarks underscores the Open-IRT's potential to enhance the robustness and adaptability in dynamic real-world. Future work can focus on extending its application to real-world object detection or semantic segmentation. Moreover, a tension exists between the method's requirement for a sizable memory bank and its premise of single-image testtime adaptation. Future work should aim to reconcile this, achieving robust performance under a strictly single-image setting.

Acknowledgements : This work was supported by the National Key Research and Development Program of China (No. 2024YFE0211000), in part by the National Natural Science Foundation of China (No. 62372329), in part by the Shanghai Scientific Innovation Foundation (No. 23DZ1203400), in part by the China Postdoctoral Science Foundation (No. BX20250383, GZB20250385, 2025M771530, 2025M771539), in part by Tongji-Qomolo Autonomous Driving Commercial Vehicle Joint Lab Project, and in part by Xiaomi Young Talents Program.

References

[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016. 1

[2] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , pages 10012-10022, 2021.

[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , pages 213-229, 2020. 1

[4] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In CVPR , pages 5830-5840, 2021. 1

[5] Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE TPAMI , 43(10):3614-3631, 2020.

[6] Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE TPAMI , 35(7):1757-1772, 2012. 1

[7] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. IJCV , pages 1-28, 2024. 1, 2, 3

[8] Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022 , 2021. 3

[9] Ev Zisselman and Aviv Tamar. Deep residual flow for out of distribution detection. In CVPR , pages 13994-14003, 2020. 2, 3

[10] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763, 2021. 2, 3, 7

[11] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR , pages 19113-19122, 2023. 2, 3, 7

[12] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. NeurIPS , 35:14274-14289, 2022. 2, 3, 7, 8, 9, 10

[13] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. NeurIPS , 34:2427-2440, 2021. 2, 3

[14] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. NeurIPS , 33:11539-11551, 2020. 3

[15] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726 , 2020. 3

[16] Chentao Cao, Zhun Zhong, Zhanke Zhou, Tongliang Liu, Yang Liu, Kun Zhang, and Bo Han. Noisy test-time adaptation in vision-language models. ICLR , 2025. 2

[17] Haoyu Dong, Nicholas Konz, Hanxue Gu, and Maciej A Mazurowski. Medical image segmentation with intent: Integrated entropy weighting for single image test-time adaptation. In CVPR , pages 5046-5055, 2024. 2

[18] Klara Janouskova, Tamir Shor, Chaim Baskin, and Jiri Matas. Single image test-time adaptation for segmentation. arXiv preprint arXiv:2309.14052 , 2023. 2

[19] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 , 2019. 3, 5, 6, 7, 8, 9

[20] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015. 3, 7, 8

[21] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 , 2017. 3, 7, 9

[22] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. 3, 5, 6, 7, 8, 9

[23] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR , 17(59):1-35, 2016. 7, 9

[24] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop , volume 2011, page 4, 2011. 3, 7, 8, 9

[25] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. NeurIPS , 33:21464-21475, 2020. 3

[26] Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't know? NeurIPS , 34:29074-29087, 2021. 3

[27] Alireza Zaeemzadeh, Niccolo Bisagno, Zeno Sambugaro, Nicola Conci, Nazanin Rahnavard, and Mubarak Shah. Out-of-distribution detection using union of 1-dimensional subspaces. In CVPR , pages 9452-9461, 2021. 3

[28] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690 , 2017. 3

[29] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NeurIPS , 31, 2018. 3

[30] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. NeurIPS , 34:144-157, 2021. 3

[31] Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In ECCV , pages 691-708, 2022. 3

[32] Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE TPAMI , 43(11):3964-3979, 2020. 3

Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. NeurIPS , 32, 2019. 3

Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. NeurIPS , 33:20685-20696, 2020. 3

Jingkang Yang, Kaiyang Zhou, and Ziwei Liu. Full-spectrum out-of-distribution detection. IJCV , 131(10): 2607-2622, 2023. 3

[36] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR , pages 16816-16825, 2022. 3

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV , 130(9):2337-2348, 2022. 3

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. IJCV , 132(2):581-595, 2024. 3

Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In ECCV , pages 493-510, 2022. 3

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021. 3

[41] Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In CVPR , pages 1593-1603, 2024. 3

Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. NeurIPS , 34:7068-7081, 2021. 3

[43] Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Yueqian Lin, Qing Yu, Go Irie, Shafiq Joty, Yixuan Li, Hai Li, et al. Generalized out-of-distribution detection and beyond in vision language model era: A survey. arXiv preprint arXiv:2407.21794 , 2024. 3

[44] Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided ood detection with pretrained vision-language models. arXiv preprint arXiv:2403.20078 , 2024. 3

[45] Chentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu, Tongliang Liu, and Bo Han. Envisioning outlier exposure by large language models for out-of-distribution detection. arXiv preprint arXiv:2406.00806 , 2024. 3

[46] Jameel Abdul Samadh, Mohammad Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muhammad Muzammal Naseer, Fahad Shahbaz Khan, and Salman H Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. NeurIPS , 36, 2024. 3, 7, 8, 9

[47] Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of-distribution detection with vision-language representations. NeurIPS , 35:35087-35102, 2022. 3

[48] Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In ICCV , pages 1802-1812, 2023. 3, 4

[49] Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Locoop: Few-shot out-of-distribution detection via prompt learning. NeurIPS , 36, 2024. 3

[50] Yichen Bai, Zongbo Han, Bing Cao, Xiaoheng Jiang, Qinghua Hu, and Changqing Zhang. Id-like prompt learning for few-shot out-of-distribution detection. In CVPR , pages 17480-17489, 2024. 3

[51] Tianqi Li, Guansong Pang, Xiao Bai, Wenjun Miao, and Jin Zheng. Learning transferable negative prompts for out-of-distribution detection. In CVPR , pages 17584-17594, 2024. 3

[52] Manogna Sreenivas and Soma Biswas. Effectiveness of vision language models for open-world single image test time adaptation. arXiv preprint arXiv:2406.00481 , 2024. 3, 7, 8, 9, 10

[53] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML , pages 1180-1189, 2015. 3

[54] Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing , 312:135-153, 2018.

[55] Boyang Peng, Sanqing Qu, Yong Wu, Tianpei Zou, Lianghua He, Alois Knoll, Guang Chen, and Changjun Jiang. Map: Mask-pruning for source-free model intellectual property protection. In CVPR , pages 23585-23594, 2024. 3

[56] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML , pages 9229-9248, 2020. 3

[57] Ke Fan, Tong Liu, Xingyu Qiu, Yikai Wang, Lian Huai, Zeyu Shangguan, Shuang Gou, Fengjian Liu, Yuqian Fu, Yanwei Fu, et al. Test-time linear out-of-distribution detection. In CVPR , pages 23752-23761, 2024. 3

[58] Zhengqing Gao, Xu-Yao Zhang, and Cheng-Lin Liu. Unified entropy optimization for open-set test-time adaptation. In CVPR , pages 23975-23984, 2024. 3, 7, 8

[59] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. NeurIPS , 35:38629-38642, 2022. 3

[60] Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In ICCV , pages 2704-2714, 2023. 3

[61] Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In CVPR , pages 14162-14171, 2024. 3, 7, 8, 10

[62] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. ICCV , 2017. 6

[63] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018. 6

[64] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020. 7

[65] Ce Zhang, Simon Stepputtis, Katia Sycara, and Yaqi Xie. Dual prototype evolving for test-time generalization of vision-language models. NeurIPS , 37:32111-32136, 2025. 7, 8, 10

[66] Yushu Li, Xun Xu, Yongyi Su, and Kui Jia. On the robustness of open-world test-time training: Self-training with dynamic prototype expansion. In ICCV , pages 11836-11846, 2023. 7, 8, 10

[67] Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics , 7(2): 179-188, 1936. 9

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In both our abstract and introduction, we explicitly define the scope of our study as addressing open-world single-sample test-time domain adaptation. Our contribution lies in proposing a state-of-the-art strategy that effectively addresses the aforementioned challenges, supported by a solid theoretical foundation.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the appendix, future work will focus on extending our approach to realworld tasks such as object detection and semantic segmentation. These tasks present more significant distribution estimation challenges due to the shift from image-level understanding to fine-grained spatial localization and semantic parsing.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The theoretical framework and corresponding proofs are comprehensively elaborated in the Methodology section (Section 3) of the main text, with supplementary derivations and extended demonstrations provided in the Appendix section.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper elaborates on the technical details in the experimental section (Section 4), while supplementary materials provide additional experimental investigations along with fully reproducible code implementation.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The source code has been archived in the supplementary materials, with all experimental datasets being publicly available benchmark datasets that are comprehensively documented in the main manuscript and supplementary materials, and are publicly accessible through their original repositories.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper meticulously documents experimental details in the Section 4, including hardware specifications (e.g., GPU models, memory capacity) and hyperparameter settings (learning rate, batch size, etc.). Additional analyses, such as robustness checks, are provided in the Supplementary Materials to ensure reproducibility and thorough validation.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our experimental results strictly follow the standardized benchmarking protocols to ensure fair and reproducible performance assessment. Furthermore, all reported results are averaged over multiple runs to account for randomness and enhance the statistical reliability of our findings.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In the experimental section, we specify that our computational resources are based on NVIDIA A6000 GPUs and NVIDIA 3090 GPUs. For detailed configurations, please refer to Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our submitted code fully conforms to the official conference standards, including strict adherence to the anonymity policy.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In the introduction (Section 1) and Methodology (Section 3) and appendix, we provide an in-depth discussion of the societal impact of our method, particularly emphasizing that the model's ability to perform real-time adaptation in open environments is crucial for addressing the emergence of unknown categories or distributional shifts. Such capability is essential for ensuring safety in high-stakes applications like autonomous driving.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: As stated in the experimental section (Section 4), all datasets used in our study are publicly available and legally obtained, and all models are based on open-source implementations. Therefore, there is no foreseeable risk of misuse associated with our work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: As indicated in the experimental section (Section 4) and supplementary materials, our method is properly cited and fully complies with the authors' licensing terms.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The assets associated with our paper, including the code, are submitted collectively in a compressed archive.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: Only used for improve writing.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Open-IRT, a single-image open-set Test-Time Adaptation framework for VLMs. The method proposes to use semantical text understanding to build a statistical barrier between ID and OOD samples. The method comprises different procedures, including preserving a memory of scores and a memory of features, the introduction of an intermediate domain based on confidence thresholds, and the employment of InfoNCE to contrast each image with respect to this intermediate domain. The method is evaluated on a series of popular TTA benchmarks, and compared to previous adaptation methods applied on CLIP and MaPLe, showing competitive results in harmonic accuracy and OOD detection metrics.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n\n-The ideas are novel and tackle important issues in the field of (open-set) TTA: single image adaptation, modelling of the ID-OOD distributions without using batches of images, and explicitly enforcing separability between ID and OOD samples in the feature space (which is not often done, with exceptions like UniEnt).\n-Experiments include different popular benchmarks and a significant number of baselines on two different VLMs, showing competitive results.\n-Using InfoNCE to create and contrast an intermediate domain is sound and clever.\n\nWeaknesses:\n\n-The paper is well written and generally clear, but information is very condensed and feels crowded by the end of Section 3; a better narrative could be possible. \n-Most of the most recent open-set TTA methods are evaluated against Open-IRT, however, the performance of other closed-set strategies is overlooked, specially in VLMs. For instance, methods such as WATT, TENT or OSTTA (pioneer in open-set) could be present for completeness.\n-Although several benchmarks are included, using the standard Imagenet (or any other representative of natural images) would further demonstrate to what extent this method works because of its open-set detection capabilities, an not only due to the fact that some images are corrupted and the OOD datasets are not. Similarly, having corrupted versions of MNIS, SVHN, etc., would aid to the completness of the study.\n\nQUESTIONS:\n1. Harmonic accuracy is used to cover both ID accuracy and detection accuracy, but what is the traditional ID accuracy for the experiments). Open-set TTA is a combination of two problems: a) OOD detection, and b) ID classification. This harmonic accuracy does not allow us to see at which of both does Open-IRT excel at, or if it is actually competitive in both. \n2. How can we be sure that the good detection performance (problem a)) is not due to the fact that the ID images are corrupted (or synthetic in the case of VisDA-C), and the OOD images are not? Using corrupted versions of the OOD datasets would be beneficial. In the opposite sense, what happens natural images are used as ID datasets? Think of Imagenet, CIFAR-10, CIFAR-100, etc. \n3. Is the final ID/OOD score the one computed from Eq. 3? This part is not very clear. As I mentioned before, although the writing is clear zooming in, the narrative can get confused when we zoom out Section 3. \n4. What's the performance of other closed-set methods? TENT for instance, can be seamlessly applied on VLMs by just doing entropy minimization on the softmax cosine similarity logits. Moreover, other recent methods have been proposed: WATT(Neurips '24, model ensembling), OSTTA (ICCV '23, confidence-based detection), which are not included.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel open-set test-time adaptation method designed to handle single-image inputs. The approach first classifies each test sample as in-distribution (ID) or out-of-distribution (OOD) using the Polarity-aware Prompt-based OOD Filtering (PPF) module, which computes similarity scores with both positive and negative prompts. These scores are modeled with a Gaussian Mixture Model (GMM) to distinguish ID from OOD samples. Once classified, the model performs adaptation via an Intermediate Domain-based Test-time adaptation (IDT) module, which constructs a middle domain between ID and OOD representations. Adaptation is guided by a contrastive loss and a bidirectional pseudo-labeling strategy to refine the feature space and improve ID-OOD separation over time.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well written and clearly structured, making the proposed method easy to understand and potentially straightforward to reproduce.\n- The approach is extensively compared against a wide range of state-of-the-art baselines across multiple datasets, consistently showing strong performance.\n- The authors conduct several relevant ablation studies, including the impact of different score modeling methods (e.g., GMM vs. LDA) and the adaptation loss components.\n- The supplementary material is thorough and provides additional insights, which strengthens the overall experimental validation.\n\nWeaknesses:\n- Although the method targets the single-image test-time adaptation setting, it still relies on feature and score banks that accumulate information over multiple samples. Therefore, the title and claims of true single-image operation may be slightly overstated. In addition, the supplementary study on batch size does not explore values below B=64; evaluating lower values would be important to fully support the single-image claim, even if performance degrades.\n- The method employs negative prompts such as “a photo of no [CLS]” to distinguish OOD samples. However, recent works (e.g., Learning Transferable Negative Prompts for OOD Detection, 2024; CLIPN for Zero-Shot OOD Detection, 2023) suggest that CLIP often struggles with such formulations. A more detailed discussion on prompt engineering choices and their limitations would strengthen the paper and help clarify whether this design choice generalizes well.\n- While the authors introduce bootstrapped resampling to stabilize GMM estimation with few samples, there is limited quantitative or qualitative evidence that this improves performance. It would be useful to understand how sensitive the method is to GMM misestimation early in adaptation.\n\nQUESTIONS:\n1. The supplementary material discusses performance across different batch sizes, but does not report results for very small batches (e.g., B=1 or B=8). Could the authors include results in that regime, or comment on how the method degrades with decreasing sample support?\n\n2. The method relies on negative prompts (e.g., \"a photo of no [CLS]\") for OOD detection. However, prior work (e.g., CLIPN, NegPrompt) reports that such prompts can be unreliable with CLIP. Could the authors provide more insight into how their specific formulation avoids these pitfalls, or whether other prompt designs were tested?\n\n3. Could the authors elaborate on the design and selection of prompts (positive and negative)? For example, how sensitive is the method to the phrasing of prompts? Did they explore alternatives such as learned or templated prompt tuning?\n\n4. During early test-time steps when only a few samples are available, how stable is the GMM estimation? Could the authors elaborate on how bootstrapped resampling helps here, and whether other techniques were considered?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of open-set single-image test-time adaptation (TTA), where in-distribution (ID) and out-of-distribution (OOD) samples appear unpredictably. The proposed framework comprises two main components: (1) Polarity-aware Prompt-based OOD Filter (PPF) and (2) Intermediate Domain-based Test-time adaptation (IDT). The PPF computes a sample-wise score by combining absolute semantic alignment (between image features and positive prompts) and relative semantic polarity (between image features and corresponding positive/negative prompts). This score is then modeled via a Gaussian Mixture Model (GMM) to classify samples as either ID or OOD. The IDT constructs intermediate domain features and uses contrastive learning (via InfoNCE loss) to encourage separation between ID and OOD features from the intermediate representations.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths\n1. The paper introduces a novel and practically relevant setting: open-set single-image test-time adaptation.\n2. The proposed method performs competitively on standard benchmarks.\n\n* Weaknesses\n1. Technical clarity and presentation\n\n1-1. The introduction lacks intuitive motivation. For example, the rationale behind using intermediate domain features to distinguish ID and OOD samples is not clearly justified. The term “polarity” is also introduced abruptly and is only clarified later in (3).\n\n1-2. Lines 144–148 just provide a description of the method, but the underlying motivation and the importance of the proposed design are not sufficiently justified.\n\n1-3. There are mismatches between terminology used in figures and text. For instance, the terms 'Sim_pos’, 'Sim_neg’, and 'Sim_diff’ in Figure 2 are not consistently reflected in the main text. `Prototypes’ in Figure 2 are mentioned as prompts in the main text.\n\n1-4. The sliding window strategy is not clearly defined. Is it equivalent to a FIFO queue? This should be clarified.\n\n1-5. $f_s$ is a multi-dimensional vector, so applying variance directly is questionable; covariance may be more appropriate unless the computation is done per-channel. This needs clarification.\n\n1-6. While (8) uses samples that satisfy ($y^+ = y_p$), this constraint is not applied to (9). The motivation behind this inconsistency should be discussed.\n\n1-7. What happens when the sample does not belong to the two cases in (7), (10), and (13), i.e., $\\mu_O <= Conf(x) <= \\mu_I$? The current decision rule does not account for such cases.\n\n\n2. In (13), the use of $-L_{CE}$ for OOD samples aims to suppress intra-class noise, but this may unintentionally distort OOD representations.\n\n3. The method depends heavily on the threshold parameters $\\mu_I$ and $\\mu_O$. Empirical analysis of sensitivity to these hyperparameters should be provided.\n\n4. L202: While the proposed method attempts to distance both ID and OOD samples from the intermediate domain representations, it is possible that ID and OOD samples may still be drawn closer to each other (e.g., if vectors $F_I - F_M$ and $F_O - F_M$ are aligned). This could undermine the objective of separation and should be theoretically or empirically addressed.\n\n* Overall evaluation\n\nOverall, this paper tackles a challenging and underexplored problem in open-set single-image TTA and provides an reasonable approach leveraging vision-language models. However, the current version suffers from unclear technical presentation and insufficient explanation of key design choices.\n\nQUESTIONS:\nRefer to weaknesses above.\n\nAdditional question: How are the positive and negative prompts learned?\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper addresses the challenge of open-set single-image test-time adaptation and introduces Open-IRT that incorporates a Polarity-aware Prompt-based OOD Filter (PPF) with an Intermediate Domain-based Test-time Adaptation (IDT) module. Instead of directly maximizing the distance between ID and OOD feature representations, this module constructs a "middle domain" in the feature space and optimizes a loss function that pushes both ID and OOD features away from this intermediate barrier. Experiments show that Open-IRT outperforms strong baselines on existing benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths
1. The paper addresses the Open-Set Single-Image Test Time Adaptation problem with Vision Language Models, which is a challenging and practical scenario.
2. The paper is well written and easy to follow.
3. The paper provides extensive experiments, showing the effectiveness and versatility of the proposed method.
4. The idea of constructing middle domain is interesting.

Weaknesses
1. Lack of important comparision with most recent baseline [1].
2. The improvement is limited compare to the best baseline (Table 1 and 3), and is even worse than the best baseline in some cases.
3. The proposed Open-IRT framework is quite complex, involving multiple modules, feature/score banks, GMMs, KNN searches, and several loss terms. 

[1] Noisy Test-Time Adaptation in Vision-Language Models, ICLR 2025

QUESTIONS:
1. What are the performances compared with the most recent baseline [1]?
2. There are too many hyperparameters (bank size B, K, alpha, lambda, as well as mu_I and mu_O in eq13). How are they selected?
3. Detailed analysis of the computational latency per sample?

[1] Noisy Test-Time Adaptation in Vision-Language Models, ICLR 2025

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02