Title: Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Computer Vision

Segmentation

Test Time Adaptation

Vision Language Models
Abstract: Recently, test-time adaptation has attracted wide interest in the context of vision-language models for image classification. However, to the best of our knowledge, the problem is completely overlooked in dense prediction tasks such as Open-Vocabulary Semantic Segmentation (OVSS). In response, we propose a novel TTA method tailored to adapting VLMs for segmentation during test time. Unlike TTA methods for image classification, our Multi-Level and Multi-Prompt (MLMP) entropy minimization integrates features from intermediate vision-encoder layers and is performed with different text-prompt templates at both the global CLS token and local pixel-wise levels. 
Our approach could be used as plug-and-play for any segmentation network, does not require additional training data or labels, and remains effective even with a single test sample. Furthermore, we introduce a comprehensive OVSS TTA benchmark suite, which integrates a rigorous evaluation protocol, nine segmentation datasets, 15 common synthetic corruptions, and additional real and rendered domain shifts, with a total of 87 distinct test scenarios, establishing a standardized and comprehensive testbed for future TTA research in open-vocabulary segmentation. Our experiments on this suite demonstrate that our segmentation-tailored method consistently delivers significant gains over direct adoption of TTA classification baselines. Code and data are available at https://github.com/dosowiechi/MLMP.

Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation

Abstract

Recently, test-time adaptation has attracted wide interest in the context of visionlanguage models for image classification. However, to the best of our knowledge, the problem is completely overlooked in dense prediction tasks such as OpenVocabulary Semantic Segmentation (OVSS). In response, we propose a novel TTA method tailored to adapting VLMs for segmentation during test time. Unlike TTA methods for image classification, our Multi-Level and Multi-Prompt (MLMP) entropy minimization integrates features from intermediate vision-encoder layers and is performed with different text-prompt templates at both the global CLS token and local pixel-wise levels. Our approach could be used as plug-and-play for any segmentation network, does not require additional training data or labels, and remains effective even with a single test sample. Furthermore, we introduce a comprehensive OVSS TTA benchmark suite, which integrates a rigorous evaluation protocol, nine segmentation datasets, 15 common synthetic corruptions, and additional real and rendered domain shifts, with a total of 87 distinct test scenarios , establishing a standardized and comprehensive testbed for future TTA research in open-vocabulary segmentation. Our experiments on this suite demonstrate that our segmentation-tailored method consistently delivers significant gains over direct adoption of TTA classification baselines. Code and data are available at https://github.com/dosowiechi/MLMP .

1 Introduction

Contrastive Vision-Language Models (VLMs) such as CLIP [1] have demonstrated remarkable generalization capabilities by aligning vision and language modalities through large-scale pre-training. This versatility has positioned VLMs as powerful foundation models for numerous downstream tasks [2, 3, 4]. A promising direction for leveraging VLMs beyond classification is Open-Vocabulary Semantic Segmentation (OVSS), where models aim to segment objects beyond a pre-defined set of categories, via VLMs' zero-shot recognition capabilities. Unlike traditional segmentation methods that require pixel-wise supervision, OVSS enables generalization to unseen object categories through language-driven representations.

Although existing OVSS methods have made significant progress, they remain vulnerable to domain shifts at test time, such as environmental changes or image corruptions, which may dramatically degrade segmentation quality. In the absence of a mechanism enabling them to adapt to unseen test-time distributions, these models might lose their generalization capabilities, which limits their reliability in real-world applications. Consequently, there is an unresolved gap for the Test-Time

L

7

L

8

L

9

L

10

L

11

L

12

L

13

L

14

L

15

L

16

L

17

L

18

L

19

L

20

L

21

L

22

L

23

L

24

CLS

1.0

1.5

2.0

2.5

Entropy (mean ± std)

over 7 templates (V20-C)

+154.0%

+216.3%

+202.9%

+99.0% +95.2%

+49.0%

+31.7%

+58.7%

+2.9%

+16.3%

+20.2%

+34.6%

-5.8%

+70.2%

+66.3%

+39.4%

+20.2%

+51.0%

CLS token

Spatial tokens

V20

V20 +

V20 +

Defocus

V20 +

Snow

V20 +

JPEG

V20-C

Average All

60

65

70

75

80

85

90

mIoU

+7.9

+8.2

+7.8

+8.0

+9.5

+8.6

Original

Corruptions

No adapt

Adaptation (TTA) of OVSS models, which would enable models to dynamically adjust both to the task shift of VLM-based segmentation and to the domain shifts encountered during inference.

To close this gap, we present a novel M ultiL evel M ultiP rompt (MLMP) test time adaptation strategy, the first fully test-time adaptation framework that could be plugged into any OVSS model , to the best of our knowledge. MLMP is lightweight and plug-and-play, boosting performance on the fly without access to labels. Its power comes from two key ideas: ( i ) adaptively integrating intermediate vision-encoder layers to harvest complementary, shift-resilient features, and ( ii ) a multi-prompt optimization that exploits VLMs' template sensitivity to provide a robust adaptation signal across diverse text-template conditions.

The core requirement for test-time adaptation is a reliable signal that faithfully reflects the current input distribution-even under severe domain shifts or corruptions. To meet this need, MLMP begins by adaptively integrating intermediate layers of the vision encoder: earlier layers preserve fine-grained edges and textures, while deeper blocks encode semantic context, and each layer reacts differently when the data distribution changes. By aggregating these multi-level features into the adaptation process and weighting them by their confidence, MLMP harvests the most trustworthy signals for each input sample.

Beyond multi-level fusion, MLMP leverages VLMs' prompt sensitivity to model uncertainty. Prior work [5, 6] shows that changing a prompt template, e.g., from ''a photo of a {class}' to ''an origami of a {class}', could drastically change the classification performance. In segmentation, we show that this effect is even more extreme: per-pixel predictions under different prompts diverge far more than the single CLS token used for classification (Figure 1a: CLS token vs. last-layer spatial tokens). This sensitivity effect is even more pronounced in intermediate feature maps. The intermediate layers that we fuse for reliability exhibit even stronger template-specific shifts (Figure 1a: intermediatelayer tokens) . Instead of viewing this inconsistency as a weakness, MLMP models it directly by incorporating multi-prompt, multi-level predictions into its adaptation objective function. This multiprompt approach not only smooths out template-specific noise, thereby reducing gradient variance and preventing degenerate collapse, but also ensures that the model yields segmentations that are consistent across diverse linguistic formulations. In this way, MLMP transforms prompt sensitivity into a powerful adaptation signal that complements its multi-level feature integration. As illustrated in Figure 1b, our MLMP method consistently outperforms the non-adapted baseline, achieving about 8-9 absolute mIoU improvements on domain-shifted inputs, while also boosting performance on original (non-corrupted) images. This demonstrates the benefit of our joint multi-level, multi-prompt adaptation.

We outline our key contributions as follows:

Plug-and-Play TTA Framework for OVSS: We introduce MLMP, which is, to the best of our knowledge, the first fully test-time adaptation method that could be easily applied to any OVSS backbone.

Adaptive Multi-Level Fusion: MLMP integrates features from intermediate vision-encoder layers to capture complementary, shift-resilient cues. To further enhance robustness, we propose an uncertainty-aware strategy that re-weights features from individual layers based on their prediction entropy.

Multi-Prompt Local-Global Test-Time Optimization: MLMP turns prompt sensitivity into signal by directly minimizing entropy across different text prompt templates at both the global CLS token and local pixel-wise levels. This optimization naturally complements our multi-level feature fusion by enforcing consistency across linguistic perspectives and feature depths.

Comprehensive OVSS TTA Benchmark Suite: We curate a rigorous evaluation protocol spanning nine mainstream segmentation datasets and 15 common synthetic corruptions, and additional real and rendered domain shifts, with a total of 87 distinct test scenarios , establishing a standardized and comprehensive testbed for future TTA research in open-vocabulary segmentation. Our experiments on this suite demonstrate that MLMP consistently delivers significant gains over baselines across all scenarios.

2 Related Work

Test-time adaptation (TTA) for open-vocabulary semantic segmentation (OVSS) remains unexplored-existing TTA methods focus on classification or single-modality segmentation, while OVSS approaches use VLMs without any online adaptation. We bridge this gap with our proposed method MLMP, a plug-and-play TTA framework that can be applied to any OVSS method.

Test-Time Adaptation. TTA addresses domain shifts by adapting pre-trained models to unlabeled target data without source samples. Methods like PTBN [7] and TENT [8] update batch statistics and affine parameters via entropy minimization but rely on large batches or augmentations. MEMO [9] simplifies this with single-sample augmentations, LAME [10] clusters features via Laplacian smoothing, and SAR [11] stabilizes adaptation using batch-agnostic normalization and sharpness-aware entropy minimization.

Test-Time Adaptation on Segmentation. TTA enhances segmentation robustness against domain shifts without source data. Methods include self-supervised adaptation via entropy minimization or contrastive learning [8, 12], single-image adaptation optimizing per-image predictions [13], and continual TTA that leverages clustering to prevent forgetting [14]. Multi-modal adaptation uses cross-modal self-supervision [15], while active TTA integrates minimal human feedback for guided refinement [16]. These approaches assume a fixed, vision-only label space and rely on spatial or surrogate tasks, making them ill-suited for zero-shot, text-driven OVSS. Consequently, none have been applied to VLMs.

Open-Vocabulary Semantic Segmentation. OVSS enables segmentation of unseen categories using vision-language models like CLIP. Approaches fall into fully-supervised, weakly-supervised, and training-free categories. Fully-supervised methods use pixel-wise annotations [17, 18, 19], while weakly-supervised ones leverage image-text pairs [20, 21, 22, 23]. Training-free OVSS avoids adaptation data but may rely on auxiliary pre-trained models [24, 25, 26]. Training-free OVSS approaches aim to enhance segmentation without additional training data. Some methods, such as SCLIP [27], adjust self-attention mechanisms to improve feature localization, while others, like MaskCLIP [28], refine feature extraction from CLIP's visual backbone. GEM [29] introduces additional optimization techniques to extract better dense features without fine-tuning. Among these, NACLIP [30] enhances CLIP's dense prediction capabilities by introducing neighborhood attention, which ensures that image patches focus on nearby regions, and by refining similarity measures to improve spatial consistency.

To the best of our knowledge, this is the first work to address TTA for OVSS models, filling a previously unexplored intersection between these fields.

To better situate this contribution within the broader landscape of general adaptation methods, Table 1 summarizes the key distinctions among zero-shot inference, domain generalization (DG) [31, 32], few-shot segmentation (FSS) [33], test-time training (TTT) [34, 35, 36], and our fully unsupervised test-time adaptation (TTA). This comparison highlights that MLMP addresses the most challenging

and realistic scenario, adapting models entirely from unlabeled test data without any source access or annotated support samples.

3 Methodology

We first revisit the contrastive vision-language model (VLM) for open-vocabulary semantic segmentation (OVSS), and then present our Multi-Level Multi-Prompt (MLMP) adaptation strategy.

3.1 OVSS with VLMs

Given an input image X ∈ R H × W × 3 and a set of concepts C k ∈ C expressed in natural language, OVSS seeks a semantic mask y ∈{ 1 , . . . , K } H × W that assigns one concept to every pixel.

Following recent approaches for OVSS [30, 27], we employ a transformer-based VLM to extract visual and text features from the image and concepts in natural language. Specifically, we feed the image X into the ViT-based vision encoder to extract a visual token matrix F = [ f [ cls ] , f 1 , . . . , f N ] with each f i ∈ R D , where N = ⌊ H/s ⌋×⌊ W/s ⌋ is the number of patches of size s × s in the image and [ cls ] is the CLS token for classification. We define Q = [ q [ cls ] , q 1 , . . . , q N ] , with each q i ∈ R D ′ , the output features before the projection layer: F = proj ( Q ) . At the same time, the text encoder is employed to extract text features t k ∈ R D for each concept C k ∈ C . This is achieved by combining C k with a text prompt template, for instance ' A photo of a [ C k ] ' or ' An image of a [ C k ] ' where C k is an arbitrary text description like ' white horse '.

The standard approach for classifying images with a contrastive VLM such as CLIP [1] computes the cosine between the CLS token features and text embeddings of classes, and assigns the image to the class with highest similarity:

For extending this approach to segmentation, we instead compute the similarity between patch embeddings f i and text embeddings t k and assign a class/concept to each patch.

3.2 MLMP: Proposed Method

Figure 2 illustrates our full test-time adaptation pipeline, MLMP . MLMP integrates three complementary ideas: uncertainty-aware multi-level fusion , image-level entropy minimization and multi-prompt adaptation .

We begin by modifying the entropy minimization objective of TENT [8] from image classification to work with spatial tokens . More specifically, for a batch of B images, each containing N tokens, the probability that token i belongs to concept k is

where τ is a softmax temperature scaling parameter, T = [ t 1 , . . . , t |C| ] , and norm( · ) denotes a function normalizing the columns of its input matrix to unit length. Also, let P denote a matrix

ǁ

a photo of the

small {

𝑪𝒌

}

an origami {

𝑪𝒌

}

…

Text Encoder

Vision Encoder

…

𝑙 1

𝑙 2

𝑙 3

𝑙

𝐿

Projection

Head

𝑓 1,1

𝑓 1,2

…

𝑓 1,𝑊

𝑓 2,1

𝑓 2,2

…

𝑓 2,𝑊

…

…

…

…

𝑓 𝐻,1

𝑓 𝐻,2

…

𝑓 𝐻,𝑊

𝑓 [𝑐𝑙𝑠]

…

…

Similarity Scores

…

…

…

…

…

…

Patch-Level Features

Global Features

Patch-level

Features

CLS Token

Text Features

𝑡

1

1

𝑡

2

1

…

𝑡 𝐶

1

𝑡

1

2

𝑡

2

2

…

𝑡 𝐶

2

…

…

…

…

𝑡

1

𝑇

𝑡

2

𝑇

…

𝑡

𝐶

𝑇

Adaptation Phase

Evaluation Phase

Adapted Vision Encoder

…

Weighted Averaging of Features

𝒒 𝟏

𝒒 𝟐

𝒒 𝟑

𝒒 𝑳

Feature Averaging

𝒒 𝟏

𝒒 𝟐

𝒒 𝟑

𝒒 𝑳

𝜶 𝟏

𝑙 1

𝜶 𝟐

𝜶 𝟑

𝑙 3

𝜶 𝒍

𝑙

𝐿

𝑙 2

𝑓 1,1

𝑓 1,2

…

𝑓 1,𝑊

𝑓 2,1

𝑓 2,2

…

𝑓 2,𝑊

…

…

…

…

𝑓 𝐻,1

𝑓 𝐻,2

…

𝑓 𝐻,𝑊

Similarity Scores

…

…

Patch-Level Features

ArgMax &

Bilinear

Upsampling

Average

Weighted

Patch-level

Features

𝑡 1

𝑡 2

…

ǁ 𝑡𝐶

Calculating the entropy of the Intermediate layers

Projection

Head

Projection

Head

𝑳𝒇𝒊𝒏𝒂𝒍 𝜽

=

𝟏

𝑻 ෍

𝒕=𝟏

𝑻

𝑳𝑼𝑴𝑨𝑳(𝑻𝒕) + 𝑳𝑰𝑳𝑬(𝑻𝒕)

At

tus

ǁ

containing the probabilities in (2), which could be expressed more compactly as follows:

The batch-wise entropy, which is minimized for adaptation, is then defined as follows:

Following [5, 37], we keep the entire text encoder frozen and update only the LayerNorm parameters of the vision encoder during adaptation. Freezing the text encoder greatly reduces computational overhead, since text embeddings can be precomputed and reused across all test samples.

Uncertainty-Aware Multi-Level Fusion. In VLM-based classification, the CLS token in the last layer of the visual encoder is typically used to compute the class label probabilities. This approach relies on the idea that the relevant information for classification lies at the end of the ViT and that intermediate layers serve to transform features. In segmentation, however, features from intermediate layers are often used to capture complementary information at different scales [38]. This hypothesis is validated in Table 2 as well as Figure 3, showing that a higher segmentation mIoU is obtained when combining the features from different intermediate layers.

Inspired by this result, and leveraging the useful property of ViTs that the output of each layer has the same shape, we extend the entropy-based loss described above to use features from multiple layers . Denoting as q ℓ i the visual features of patch i obtained at layer ℓ , we seek to aggregate the multi-level features into a single vector q i for segmentation prediction. A simple approach for doing this is to compute q i by averaging q ℓ i across all layers ℓ . However, this approach ignores the relative contribution and confidence of each layer in the final segmentation. To address this limitation, we estimate a confidence weight α ℓ for each layer ℓ based on its prediction entropy. First, we get visual features F ℓ i = proj ( Q ℓ i ) using the same projection head as for the final segmentation. Following the same approach as before, we then compute the batch-wise entropy of layer ℓ as

Finally, the confidence weight of the layer is obtained using a softmax as follows:

Here, β is a parameter controlling the 'sharpness' of the weight distribution. During adaptation, we set β = 0 to promote a uniform contribution from all layers in the prediction. During inference, we sharpen the distribution with a value of β = 1 , emphasizing the more confident layers in the final prediction.

With these confidence weights, we can now obtain our uncertainty-aware multi-level (UAML) features as

giving the following entropy-based loss to minimize:

Image-Level Entropy Minimization. Since the CLS token is not directly linked to individual patch predictions but rather captures a more global representation of the input, we also include an image-level entropy (ILE) minimization term specifically for this token. As illustrated in Figure 1, the CLS token demonstrates increased robustness and reliability. This term, which encourages the model to produce more confident global predictions is expressed as:

Here, p [ cls ] b,k denotes the predicted probability for concept C k obtained using the CLS token in the last layer for the b -th sample.

Multi-Prompt Adaptation. Prior work on TTA for classification [5] has shown the usefulness of leveraging multiple prompt templates in VLM to encode class labels, based on the idea that the templates capture complementary information about these classes. As shown in Figure 1a, the sensitivity to the choice of prompt templates is even more pronounced in segmentation tasks, where fine-grained spatial predictions are required. Using multiple templates acts as cross-modal regularization, encouraging more stable and generalized learning signals. While different from image augmentation, it can be seen as a strong, safe, and lightweight text-space augmentation. Rather than averaging the weights adapted from different prompt templates as in [5]-a computationally expensive approach for dense prediction tasks such as segmentation-our method minimizes our proposed UAML and ILE losses across these templates. Let T t be the text features obtained using the t -th template. Our final adaptation loss is defined as:

Theoretical Justification. Each template t contributes its own adaptation loss, as we optimize their average in Eq. (10). By optimizing the adaptation loss of each prompt directly, we force the model to correct for the unique wording and visual cue of each template, rather than 'averaging' these differences in the text embedding space. This loss-level integration treats each template as an independent critic, translating diverse linguistic perspectives into separate gradient signals. Averaging those signals produces an unbiased descent direction whose variance decays as 1 /T , enabling each adaptation step to represent the full prompt ensemble while being stable under noisy shifts.

Proposition 1 (Unbiasedness and Variance Bound). Assume that each per-template gradient g t ( θ ) = ∇ θ [ L UAML ( T t ) + L ILE ( T t ) ] has variance bounded by σ 2 , then the ensemble gradient, defined by ∇ θ L final = 1 T ∑ T t =1 g t ( θ ) , is unbiased and satisfies the following variance bound:

Proof. A proof of Prop. 1 is provided in the Appendix.

The 1 /T reduction in gradient variance, as stated in Prop. 1, explains the improved stability we observe. Table 4 confirms that this loss-level ensemble outperforms alternative fusion strategies.

L

24

(last)

L

23

-

24

(last two)

L

22

-

24

(last three)

L

19

-

24

(last 25%)

L

13

-

24

(last 50%)

L

7

-

24

(last 75%)

L

1

-

24

(all layers)

V20 (Original)

77.00

±

0.04

77.65

±

0.02

77.66

±

0.09

80.61

±

0.05

80.50

±

0.03

81.67

±

0.04

78.79

±

0.02

63.02

±

0.06

64.41

±

0.05

65.39

±

0.13

66.88

±

0.18

66.88

±

0.02

67.82

±

0.01

63.06

±

0.09

Defocus Blur

72.06

±

0.12

72.93

±

0.19

72.84

±

0.02

76.10

±

0.16

76.37

±

0.05

78.78

±

0.02

77.56

±

0.09

Snow

71.04

±

0.05

72.09

±

0.04

72.56

±

0.02

74.47

±

0.12

74.41

±

0.01

76.39

±

0.02

73.72

±

0.07

JPEG Compression

71.84

±

0.15

73.88

±

0.11

74.40

±

0.07

76.96

±

0.02

77.67

±

0.03

78.73

±

0.08

75.87

±

0.19

V20-C Average

69.33

70.33

70.78

72.89

73.45

74.90

72.02

L

7

L

8

L

9

L

10

L

11

L

12

L

13

L

14

L

15

L

16

L

17

L

18

L

19

L

20

L

21

L

22

L

23

L

24

Layer

0.00

0.05

0.10

0.15

0.20

0.25

Weight

Pascal VOC 20

Original

Defocus blur

Snow

JPEG compression

L

7

L

8

L

9

L

10

L

11

L

12

L

13

L

14

L

15

L

16

L

17

L

18

L

19

L

20

L

21

L

22

L

23

L

24

Layer

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Weight

Cityscapes

Original

Defocus blur

Snow

JPEG compression

L

7

L

8

L

9

L

10

L

11

L

12

L

13

L

14

L

15

L

16

L

17

L

18

L

19

L

20

L

21

L

22

L

23

L

24

Layer

0.00

0.02

0.04

0.06

0.08

0.10

Weight

CocoStuff

Original

4 Experimental Settings

Experimental Setup. Following prior work on TTA in classification [37, 5], we restrict updates to the normalization layers within the vision encoder. The adaptation process is carried out over 10 iterations using the Adam optimizer with a constant learning rate of 10 -3 across all datasets. We use a batch size of 2 images during adaptation across all datasets. For each new batch, the model undergoes a reset, restoring it to its initial weights before adaptation is applied.

Datasets. In traditional TTA for segmentation, two datasets are commonly employed to simulate domain shifts-one for model training and another for adaptation during inference (e.g., GTAV and Cityscapes)-as both must share the same semantic label space. In our study, as this is the first exploration of TTA for VLMs in segmentation tasks and given that VLMs are pre-trained, we draw inspiration from ImageNet-C [39] to introduce 15 synthetic corruptions on segmentation datasets. Our experiments are conducted on Pascal VOC 20 (v20), Pascal VOC 21 (v21) [40], Pascal Context 59 (P59), Pascal Context 60 (P60) [41], and Cityscapes [42], incorporating both original version (clean) and the synthetic 15 corruptions (denoted with a '-C' suffix). For COCO-Stuff [43] and COCO-Object [44], we use only the original versions. To further evaluate robustness under real and rendered distributional shifts, we additionally include ACDC [45]-capturing real-world adverse conditions such as fog, night, rain, and snow-and GTA-V [46], which provides photorealistic, game-rendered urban scenes. This extended setup results in 87 distinct test scenarios encompassing synthetic, real, and rendered shifts, enabling a comprehensive evaluation of MLMP across diverse conditions.

Benchmarking. While MLMP is compatible with any OVSS framework, we incorporate NACLIP [30] with ViT-L/14 as our baseline OVSS model, which leverages neighborhood attention to enhance spatial consistency in a training-free manner. The compared methods include TENT [8], which serves as a baseline and minimizes entropy during adaptation; CLIPArTT [37], which employs pseudo-labels generated via conformal learning; WATT [5], which averages learnable parameters across multiple parallel branches; and TPT [6], which performs prompt tuning to adapt VLMs at test time. For a fair comparison, we modified all methods for the segmentation setting by processing all spatial tokens extracted from the VLM, rather than relying solely on the CLS token.

5 Results

5.1 Ablation studies

Effect of Intermediate Layers. To analyze the impact of layer selection in our uncertainty-aware multi-level adaptation strategy, we use different ranges of intermediate layers. As shown in Table 2,

1

3

7

20

80

62.5

65.0

67.5

70.0

72.5

75.0

77.5

80.0

mIoU

Original

Defocus blur

Snow

JPEG compression

V20-C Average

performance varies notably with the fusion range. While using only the final layers yields moderate improvements, incorporating the last 75% of the layers consistently achieves the best performance across both clean and corrupted inputs. This highlights that multi-level fusion is a key driver of adaptation performance: earlier layers, although less semantically abstract, contribute valuable lowlevel features-such as texture and edge cues-that enhance robustness to distribution shifts. In this ablation, we isolate the effect of multi-level fusion by applying only the first term in Eq. 10, using a single prompt template and omitting the L ILE term.

Effect of Uncertainty-Aware Layer Fusion. We investigate strategies for aggregating multi-level features by comparing uniform averaging ( β = 0 ) and uncertainty-aware fusion ( β = 1 ) during evaluation, using the same 75% layer range identified in the previous ablation. As shown in Table 3, incorporating entropy-based weighting improves performance by 4.29% on V20 and 3.31% on V20-C. This highlights the importance of leveraging layer-wise confidence when aggregating features.

Visualization of Layer-Wise Confidence Weights. We visualize the mean and standard deviation of the learned layer weights to better understand the behavior of our uncertainty-aware fusion strategy. As shown in Figure 3, deeper layers tend to receive higher confidence, though earlier layers also contribute, especially under corrupted conditions. This variation is most pronounced in the Cityscapes dataset, where the distribution fluctuates more across layers and corruption types. In contrast, the COCO-Stuff dataset shows a flatter distribution, where the final layer is not consistently the most influential. These results underscore the core strength of our fusion mechanism: its ability to adaptively reweight layers based on input conditions, assigning greater importance to those that remain more reliable under distribution shifts and corruption. Please refer to the Appendix for additional results on other datasets.

Effect of Global Image-Level Adaptation Term. The image-level entropy term, L ILE complements our patch-level adaptation by encouraging consistent global predictions through the CLS token. While the multi-level loss targets fine-grained spatial predictions, the ILE term introduces global context that helps stabilize adaptation. As shown in Table 3, when added in isolation, L ILE improves performance by 1.74% and 2.66% on V20 and V20-C, respectively, demonstrating the benefit of incorporating global context under distribution shift.

Effect of Number of Prompt Templates. To isolate this effect, we evaluate performance using different numbers of prompt templates while disabling both the multi-level fusion and the image-level entropy (ILE) term in Eq.10. As shown in Figure4, increasing the number of templates improves performance up to 7, after which the gains begin to saturate or slightly decline. This trend holds across

both clean and corrupted settings, indicating that a moderate number of diverse prompt templates is sufficient for MLMP. We therefore use 7 templates by default in our main experiments. Please refer to the Appendix for template details.

Where to Integrate Multi-Prompt Information. Here, we empirically compare strategies for integrating multi-prompt information into the adaptation process. Specifically, we evaluate: (1) text-level averaging, where prompt embeddings are averaged before computing logits-a technique commonly used in zero-shot learning [1]; (2) a learnable parameter averaging baseline (Params) inspired by WATT [5]; and (3) our proposed method (Loss), which incorporates all prompt templates directly into the adaptation loss (Eq.10). To isolate the effect of prompt integration, this analysis excludes other components such as multi-level fusion and the image-level entropy (ILE) term. As shown in Table 4, our loss-level formulation consistently outperforms the alternatives across both clean and corrupted settings.

Full Component Analysis. Table 3 presents an extensive ablation evaluating the contribution of each component in our MLMP strategy. Each proposed element yields consistent gains when added independently, but it is their combination that delivers the highest overall performance across both clean and corrupted settings, highlighting their strong complementary effects within a unified framework. Additionally, we provide further ablations in the Appendix, including alternative OVSS backbones, different VLMs, ViT architecture variants, computational complexity analysis, and MLMP segmentation map visualizations, as well as discussions on the effect of longer prompts, effect of adaptation iterations, and episodic vs. online adaptation.

5.2 Final Comparison with Alternative Adaptation Methods

Performance on Clean Data (No Distributional Shift). We begin by evaluating MLMP on clean test data (original), where no distributional shift is present. This setting is crucial, as TTA methods must avoid degrading performance when adaptation is unnecessary. As shown in Table 5, MLMP achieves strong mIoU gains of +7.85, +5.66, +3.72, and +3.04 on V20, V21, P59, and P60, and

+5.04/+2.91 on challenging datasets COCOObject and COCOStuff. In contrast, most alternative adaptation methods fail to improve performance in this setting. These gains highlight the robustness and generalization of our method, even when no explicit domain shift is present.

Performance Under Distributional Shift. Under distributional shifts, the advantages of MLMP become even more apparent. As shown in Table 5, MLMP consistently outperforms both the zero-shot baseline and existing adaptation methods, achieving mIoU gains of +8.60, +5.50, +3.15, and +2.68 on V20-C, V21-C, P59-C, and P60-C, respectively. Beyond standard corruptions, we further evaluate on the Cityscapes dataset, which presents natural domain shifts such as environmental variation, weather conditions, and resolution differences. Despite its challenging nature and low zero-shot performance, MLMP improves mIoU by +3.86, demonstrating its real-world adaptability. To push this further, we apply 15 corruption types to create Cityscapes-C, where MLMP still yields a +1.39 gain. While TENT provides modest improvements, most other adaptation methods-including ClipArTT, TPT, and WATT-either fail to improve or degrade performance. These results highlight that naive strategies like pseudo-labeling, prompt tuning, or weight averaging are insufficient for openvocabulary segmentation, and emphasize the need for segmentation-specific adaptation techniques such as MLMP. Detailed results can be found in the Appendix.

While Cityscapes already embodies natural distributional shifts caused by variations in lighting, camera viewpoint, and urban layouts, we further assess MLMP on datasets explicitly designed to capture targeted domain shifts. Specifically, we evaluate on ACDC [45], which contains realworld adverse conditions (Fog, Night, Rain, and Snow), and GTA-V [46], a photorealistic, gamerendered dataset that introduces a distinct synthetic distribution shift relative to real imagery seen during CLIP pre-training. As summarized in Table 6, MLMP achieves consistent improvements over both the non-adapted baseline and TENT across all ACDC domains, yielding on average +6 mIoU gains under real-world conditions. Similarly, on the GTA-V dataset, MLMP improves by +3.8 mIoU over the baseline, further confirming its robustness across both realistic and rendered distribution shifts.

6 Conclusion

We presented MLMP, a plug-and-play test-time adaptation framework for open-vocabulary semantic segmentation that can be integrated with any OVSS method. By combining uncertainty-aware fusion of intermediate ViT features with a novel loss-level integration of multiple prompt templates, MLMP consistently enhances performance across both clean and shifted domains-including common corruptions and natural distributional shifts. Our comprehensive OVSS-TTA benchmark-covering nine datasets and 87 distinct test scenarios-demonstrates MLMP's broad applicability and establishes a rigorous evaluation protocol for future work in adaptive, language-aware segmentation. While MLMP demonstrates strong, consistent gains, there remain opportunities to further refine its components. In particular, our current layer-weighting mechanism relies on entropy estimates from a shared projection head, which may not fully reflect each layer's unique characteristics. Future work could investigate more flexible architectures-such as lightweight adapters or dedicated projection modules per layer-to more accurately assess and fuse intermediate features.

Acknowledgments

We appreciate the computational resources and support provided by Compute Canada and the Digital Research Alliance of Canada.

Table 6: mIoU comparison on realistic (ACDC) and rendered (GTA-V) domain shifts. Full ACDC results (including reference/clean views) are provided in the Appendix.

References

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard De Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In European Conference on Computer Vision , pages 388-404. Springer, 2022.

Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 976-980. IEEE, 2022.

Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmentation and tumor detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 21152-21164, 2023.

David Osowiechi, Mehrdad Noori, Gustavo Vargas Hakim, Moslem Yazdanpanah, Ali Bahri, Milad Cheraghalikhani, Sahar Dastani, Farzad Beizaee, Ismail Ayed, and Christian Desrosiers. Watt: Weight average test time adaptation of clip. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems , volume 37, pages 48015-48044. Curran Associates, Inc., 2024.

Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems , 35:14274-14289, 2022.

Zachary Nado, Shreyas Padhy, D. Sculley, Alexander D'Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv:2006.10963 [cs, stat] , January 2021. arXiv: 2006.10963.

Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726 , 2020.

Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems , 35:38629-38642, 2022.

Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8344-8353, 2022.

Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world, 2023.

Hong Liu, Guoliang Kang, Shan You, Jiancheng Bao, Mingkui Li, and Yao Zhang. Source-free domain adaptation for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Jiacheng He, Zhilu Zhang, Zhen Wang, and Yan Huang. Autoencoder based test-time adaptation for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 998-1007, 2021.

Charanpal D Mummadi, Matthias Arens, and Thomas Brox. Test-time adaptation for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11828-11837, 2021.

Gabriele Valvano, Andrea Leo, Kuniaki Saito, and Tatiana Tommasi. Learning multi-modal self-supervision for test-time adaptation. In NeurIPS , 2023.

Zhi Wang, Yue Zhang, Kun Yu, and Hao Zhang. Active test-time adaptation for semantic segmentation. arXiv preprint arXiv:2312.01835 , 2023.

Davide Barsellotti, Roberto Amoroso, and Barbara Caputo. Enhancing open-vocabulary semantic segmentation with prototype retrieval. In Pattern Recognition. ICPR International Workshops and Challenges , pages 243-258. Springer, 2023.

Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Scaling open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision , pages 343-360. Springer, 2022.

Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7061-7070, 2023.

Kaixin Cai, Pengzhen Ren, Yi Zhu, Hang Xu, Jianzhuang Liu, Changlin Li, Guangrun Wang, and Xiaodan Liang. Mixreorg: Cross-modal mixed patch reorganization is a good mask learner for open-world semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1196-1205, 2023.

Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11165-11174, 2023.

Yuxin Chen, Yujie Li, Zhaoxiang Zhang, and Yunchao Liu. Exploring open-vocabulary semantic segmentation from clip vision encoder distillation only. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10010-10019, 2023.

Jiarui Xu, Shalini De Mello, Sifei Liu, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18134-18144, 2022.

Davide Barsellotti, Roberto Amoroso, and Barbara Caputo. Fossil: Free open-vocabulary semantic segmentation through synthetic references retrieval. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1234-1243, 2024.

Andrea Corradini, Federico Bianchi, Debora Nozza, and Dirk Hovy. Freeseg-diff: Training-free open-vocabulary segmentation with diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 567-576, 2024.

Lukas Karazija, Wafaa Bousselham, Andrei Bursuc, Thiemo Alldieck, and Patrick Pérez. Diffusion models for zero-shot open-vocabulary segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1010-1020, 2023.

Feng Wang, Jieru Mei, and Alan Yuille. SCLIP: Rethinking self-attention for dense visionlanguage inference. In European Conference on Computer Vision , pages 315-332. Springer, 2025.

Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision , pages 288-304. Springer, 2022.

Wafaa Bousselham, Andrei Bursuc, Thiemo Alldieck, and Patrick Pérez. Grounding everything: Emerging localization properties in vision-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5678-5687, 2024.

Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2025.

Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Gustavo A Vargas Hakim, David Osowiechi, Ismail Ben Ayed, and Christian Desrosiers. Tfs-vit: Token-level feature stylization for domain generalization. Pattern Recognition , 149:110213, 2024.

Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Gustavo A Vargas Hakim, David Osowiechi, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. Fds: Feedback-guided domain synthesis with multi-source conditional diffusion models for domain generalization. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV) , pages 8493-8503, February 2025.

Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In proceedings of the IEEE/CVF international conference on computer vision , pages 9197-9206, 2019.

David Osowiechi, Gustavo A. Vargas Hakim, Mehrdad Noori, Milad Cheraghalikhani, Ismail Ayed, and Christian Desrosiers. Tttflow: Unsupervised test-time training with normalizing flow. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 2125-2126, Los Alamitos, CA, USA, jan 2023. IEEE Computer Society.

Gustavo A Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Ismail Ben Ayed, and Christian Desrosiers. Clust3: Information invariant test-time training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6136-6145, 2023.

David Osowiechi, Gustavo A. Vargas Hakim, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. Nc-ttt: A noise constrastive approach for test-time training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6078-6086, June 2024.

Gustavo Adolfo Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. Clipartt: Light-weight adaptation of clip to new domains at test time. arXiv preprint arXiv:2405.00754 , 2024.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages 234-241. Springer, 2015.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. 2019.

Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. , 88(2):303-338, 2010.

Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2014.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Holger Caesar, Jasper R. R. Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. CoRR , abs/1612.03716, 2016.

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR , abs/1405.0312, 2014.

Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10765-10775, 2021.

Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision , pages 102-118. Springer, 2016.

Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786 , 2025.

Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European conference on computer vision , pages 310-325. Springer, 2024.

Ivona Najdenkoska, Mohammad Mahdi Derakhshani, Yuki M Asano, Nanne Van Noord, Marcel Worring, and Cees GM Snoek. Tulip: Token-length upgraded clip. arXiv preprint arXiv:2410.10034 , 2024.

Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin. On pitfalls of test-time adaptation. arXiv preprint arXiv:2306.03536 , 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract, we claim that we introduce a new method to adapt VLMs for segmentation. In the paper, we present this method and prove its performance compared to state-of-art.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We mentioned limitations in the last part of the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All our assumptions are included in the Introduction and the Methodology .

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In the Section 4, we present all our different settings which are based on previous methods for TTA. We also provide our code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Everything is detailed in our github link provided in the Abstract .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Everything is detailed in the section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments have been conducted at least three times to ensure statistical robustness.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Everything is detailed in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have considered all potential harms caused by the research process, societal impacts, and potential harmful consequences, as described in the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper addresses a research field that is not currently utilized for direct practical applications, thereby limiting its immediate social impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In our paper, we cite all sources that inspired our work, and we provide links in our git repository to the codes from which we drew inspiration.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We document all aspects of our research and provide a link to our implementation, along with step-by-step explanations for reproducing the results.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer:[NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [No]

Justification: This work does not use LLMs in the core methodology, scientific rigorousness, or originality of the research.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation - Appendix

A Proof of Proposition 1

Unbiasedness and Variance Bound Proposition 1 (Restated): Assume each per-template gradient g t ( θ ) = ∇ θ [ L UAML ( T t ) + L ILE ( T t )] has variance bounded by σ 2 . Then, the ensemble gradient, defined by ∇ θ L final = 1 T ∑ T t =1 g t ( θ ) , is unbiased and satisfies the following variance bound:

Step 1: Unbiasedness. By linearity of expectation, we have:

Hence, averaging the gradients from all T templates gives an unbiased estimate of the true gradient of the final loss.

Step 2: Variance Bound. Assuming independence with Var( g t ) ≤ σ 2 , we get

Thus, the variance bound holds:

This completes the proof of Proposition 1.

B Implementation Details

Unless otherwise noted, all experiments utilize NACLIP [30] with ViT-L/14 as the OVSS backbone. We adapt only the LayerNorm parameters within the vision encoder, amounting to approximately 0.02% of the model's total parameters. Our adaptation setup follows prior work in classification [5, 37], using the Adam optimizer with a fixed learning rate of 0 . 001 and 10 adaptation steps (iterations) across all experiments. Additionally we use batch of 2 images during adaptation. After each batch, model weights are explicitly reset to their original pre-adaptation values to ensure that each batch is adapted independently, without leveraging information from previously processed data. Following standard settings from [1], we use the default softmax temperature value of 100 in all experiments. All images are resized to 224 × 224 pixels. Due to the high resolution of images in the Cityscapes dataset, we split them into overlapping patches of size 224 × 224 pixels with an overlap of 112 pixels between patches. The segmentation predictions from these patches are aggregated to reconstruct the final, full-resolution segmentation maps. No image augmentation techniques are applied during either the adaptation or evaluation phases.

All experiments are conducted on NVIDIA V100 GPUs equipped with 32GB memory. We implement our approach using the PyTorch deep learning framework. To ensure statistical robustness and fairness in our comparisons, we repeat each experiment three times, reporting average performance along with standard deviation. We have provided detailed instructions and step-by-step scripts in our repository, clearly demonstrating how to generate datasets, perform the described test-time adaptations, and reproduce our reported results.

We adapt other baseline methods (TENT [8], TPT [6], WATT [5], CLIPArTT [37]) to work with spatial tokens. General adaptation hyperparameters (optimizer, learning rate, adaptation steps, and

Table 7: List of the seven prompt templates used in our MLMP method. These general-purpose templates, originally proposed by the CLIP authors, serve as diverse textual views of each class and are not tailored to specific datasets or domains.

batch size) remain consistent with our setup. Method-specific hyperparameters or components are retained as reported in their original implementations. Specifically, for WATT, we used the sequential version (WATT-S) with default values of l = 2 and m = 5 , for CLIPArTT we used the default k = 3 , and for TPT we used the 4 learnable tokens. Adapting these baseline methods to the segmentation task allowed us to systematically evaluate how various adaptation strategies-such as prompt tuning (TPT), pseudo-labeling (WATT, CLIPArTT), prompt refinement (CLIPArTT), and weight averaging (WATT)-perform in the context of open-vocabulary segmentation.

C Template Details

Table 7 lists the seven prompt templates used in our MLMP method. These templates were selected by the original CLIP authors 2 and are general-purpose, not tailored to any specific image content. The CLIP repository also provides a full set of 80 prompt templates, which we use for larger-scale ablations.

More specifically, for the ablation in Figure 4 of the main paper, we vary the number of templates T as follows:

T = 1 : the default CLIP prompt ' a photo of a {class} '

T = 3 : the first three templates { T 1 , T 2 , T 3 } in Table 7

T = 7 : all seven templates { T 1 , . . . , T 7 } in Table 7

T = 20 : the first 20 templates from the 80-template pool

T = 80 : the complete set of 80 CLIP templates

D Dataset Details

This section details the datasets used in our experiments, along with the synthetic, real, and rendered domain shifts applied to evaluate robustness under distributional changes.

More specifically, we conduct experiments on a diverse set of segmentation benchmarks:

Pascal VOC (V20/V21) [40]: Contains 20 foreground object classes (v20) with a background class (v21), widely used for benchmarking semantic segmentation tasks.

Pascal Context (P59/P60) [41]: An extension of the Pascal VOC 2010 dataset, providing pixel-level annotations for more than 400 classes. Due to sparsity, a frequently used subset includes 59 object classes plus a background class, totaling 60 categories.

Cityscapes [42]: A large-scale dataset for semantic segmentation of urban street scenes. It comprises around 5,000 finely annotated images from 50 cities, recorded under various daylight conditions, featuring dynamic objects, varying layouts, and changing backgrounds, capturing significant natural domain shifts.

COCO-Stuff [43]: Extends the original COCO dataset by adding annotations for background categories ("stuff"), resulting in 171 classes-80 objects, and 91 stuff categories.

COCO-Object [44]: A subset of the original COCO dataset, consisting exclusively of 80 object categories without background annotations.

ACDC [45]: Includes pixel-level annotations for 19 semantic classes, covering diverse driving scenes under fog, night, rain and snow conditions.

GTA-V [46]: Contains 24,966 synthetic images with pixel-accurate semantic labels; one version aligns to 34/19 classes compatible with real-world segmentation benchmarks.

In addition to the original versions of each dataset-some of which already reflect natural domain shifts (e.g., Cityscapes)-we further assess the robustness of all methods by applying synthetic corruptions. Inspired by ImageNet-C [39], we apply 15 synthetic corruptions to evaluate the robustness of segmentation models under various perturbations. These corruptions include Gaussian noise, shot noise, impulse noise, defocus blur, glass blur, motion blur, zoom blur, snow, frost, fog, brightness variations, contrast variations, elastic transformations, pixelation, and JPEG compression. Each corruption is applied at severity level 5, representing the most challenging scenario.

We resize images from all datasets to 224 × 224 pixels. Due to the high resolution of Cityscapes and ACDCimages, we process them as overlapping patches of size 224 × 224 pixels (with overlaps of 112 pixels). Predictions for these patches are subsequently aggregated to produce the final segmentation maps.

This extended benchmark comprises a total of 87 distinct test scenarios , encompassing synthetic, real, and rendered domain shifts. It provides a rigorous and comprehensive evaluation protocol that captures variations in resolution, scene diversity, object size, and semantic granularity.

E Computational Complexity

In this section, we provide an exhaustive analysis of each test-time adaptation method's resource footprint by measuring: (i) latency, (ii) floating-point operations (FLOPs), (iii) peak GPU memory usage, and (iv) number of learnable parameters. For a fair comparison, all measurements were performed on a single test sample using the same NVIDIA V100 (32 GB) GPU. The results are summarized in Table 8.

We compare TENT, TPT, CLIPArTT, WATT, and our MLMP across all four complexity metrics. In TENT, WATT, CLIPArTT, and MLMP, only the LayerNorm parameters of the vision encoder are updated during adaptation, whereas TPT introduces additional learnable tokens at the input of the text encoder. Additionally TENT, WATT, and MLMP use a fixed sets of text features without a need for recalculating them during adaptation/evaluation, so all prompt templates can be encoded once ( ✓ in the 'One-time Text Encoder' column) and then reused throughout both adaptation and evaluation phases. In contrast, TPT and CLIPArTT modify prompt embeddings or refine text templates during adaptation, requiring multiple forward passes through the text encoder.

In terms of latency, MLMP completes both adaptation and evaluation in just 0.582 ms, which is only marginally slower than TENT (0.480 ms) but substantially faster than WATT (5.215 ms) and CLIPArTT (3.525 ms). Despite leveraging multi-level fusion and multiple prompt templates, MLMP maintains a lightweight computational profile with only 82.4 GFLOPs and 82.9 GFLOPs for adaptation and evaluation, respectively-comparable to TENT and significantly lower than all other baselines. Furthermore, MLMP's peak memory usage is among the lowest at 2,093.9 MB, and like TENT, it updates only 0.02% of the model parameters. These results highlight MLMP's efficiency: it delivers rich representational capacity through multi-level and multi-prompt integration by boosting the results significantly (as shown in Table 5 of the main paper), yet remains almost as lightweight as the simplest baseline.

F Performance with a Single Test Sample

As shown in Table 9, while TENT yields improvements on the original dataset, it leads to a performance drop on V20-C. In contrast, our method, MLMP, consistently improves performance over the

no adaptation baseline, with gains of 8.77% on the otiginal data and 9.40% on the average across corruptions.

G Generalization Across Model Variants

To evaluate the robustness and generality of our method, we conduct a series of experiments using different model configurations within the segmentation pipeline. Specifically, we assess how MLMP performs when (i) changing the vision transformer backbone, (ii) switching between different openvocabulary semantic segmentation (OVSS) formulations, and (iii) adopting an entirely different vision-language model. These experiments demonstrate that our method maintains consistent improvements across a wide range of architectural and algorithmic configurations, confirming its flexibility and transferability.

G.1 Comparison Across ViT Backbones

To evaluate whether our method generalizes across different vision transformer backbones, we replicate the main experiments using ViT-B/16 and ViT-B/32 in place of the default ViT-L/14 model. These backbones represent lighter configurations, with fewer parameters and larger patch (32 and 16). As shown in Tables 10 and 11, MLMP continues to provide substantial improvements over all

baselines across both configurations, on both clean data (V20 Original) and under severe synthetic corruptions. These results confirm that the benefits of our multi-level and multi-prompt adaptation strategy are not tied to model scale or specific architectural configurations, and remain effective even in lower-resolution settings.

G.2 Comparison Across OVSS Methods

Our method is designed to be flexible and agnostic to the underlying open-vocabulary semantic segmentation (OVSS) formulation. While all main experiments in the paper use NaCLIP as the OVSS baseline-paired with No Adapt., TENT, TPT, CLIPArTT, WATT, and our MLMP-we further evaluate the generality of our approach by applying MLMP to two alternative OVSS methods.

Specifically, we consider the original CLIP [1], adapted for pixel-wise segmentation via patch-level similarity, and SCLIP [27], which incorporates spatial priors into the vision-language matching process. As shown in Table 12, applying MLMP on top of these OVSS baselines yields consistent improvements across both clean and corrupted settings.

G.3 Generalization to Emerging VLMs

To further assess the generality of our approach, we evaluate MLMP on SigLIP v2 [47], one of the most recently introduced vision-language models. As shown in Table 13, MLMP continues to deliver consistent improvements across both natural and corrupted datasets, indicating that the core components of our method-multi-level and multi-prompt aggregation-generalize well beyond CLIP-based architectures.

H Effect of Longer Prompts

Recent studies such as Long-CLIP [48] and TULIP [49] have explored extending the text length capability of vision-language models, suggesting that richer linguistic descriptions may improve alignment. Motivated by this, we investigated whether incorporating longer and more descriptive prompts could further enhance MLMP's performance.

To this end, we generated extended templates derived from our original seven class-agnostic prompts using ChatGPT, together with extended class names where each category was expressed in full natural language. For example, 'a photo of the large {class}' becomes 'a photograph of a very large {class}, where the size of the object dominates the frame or is shown in contrast to smaller elements,' while 'aeroplane' becomes 'a powered flying vehicle with fixed wings and engines, designed to transport people or cargo through the air over long distances.'

As summarized in Table 14, all experiments were conducted using the Long-CLIP [48] backbone. We evaluate three text variants: (1) the default short templates used throughout our main experiments, (2) Extended Templates , where each template is replaced with a longer and more descriptive sentence, and (3) Extended Classes , where category names are expanded into full natural-language descriptions.

For each variant, we report results with and without our MLMP adaptation ( + MLMP ). All scores correspond to mIoU.

Overall, these results indicate that MLMP's adaptation mechanism effectively handles both concise and extended textual inputs without overfitting to prompt verbosity. We include this analysis to provide a clearer picture of MLMP's behavior under longer or more descriptive prompts.

I Effect of Adaptation Iterations

The number of adaptation iterations in TTA varies across the literature. While some studies evaluate performance under a single adaptation step, others perform multiple iterations to improve convergence. In our main experiments, we followed the common 10-iteration setup adopted in prior TTA works [8, 5, 37], as it provides a consistent protocol for comparing vision-language adaptation methods such as CLIPArTT and WATT. For fairness, we used identical hyperparameters across all methods.

To further examine the influence of the iteration count, we also evaluated MLMP under a stricter one-iteration protocol, where only a single adaptation step is performed. As summarized in Table 15, MLMP still improves over the baseline and outperforms TENT even with just one iteration, which demonstrates the effectiveness of our method even under a stricter setting.

J Episodic vs. Online Adaptation

We further investigate the difference between episodic (reset-based) and online adaptation settings in test-time adaptation (TTA). In our main experiments, all methods-including MLMP and baselines-were evaluated under the episodic setup, where the model is reset to its initial weights after each batch. This protocol avoids cumulative error propagation and is widely used in prior TTA works [8, 37, 5].

To analyze the impact of continuous updates, we also tested an online version of MLMP, where model parameters are updated sequentially without resets and carried over to the next batch. We used identical hyperparameters as in the episodic setting, and further experimented with lower learning rates and fewer adaptation steps for stability.

As shown in Table 16, we observe that while the online variant improves over baselines, it still underperforms compared to the reset-based (episodic) setting. We believe this is primarily due to error accumulation and model drift, as documented in prior work showing that some recent classification TTA methods exhibit performance degradation in an online setting-even when the distribution is static [50]. This issue is particularly pronounced in semantic segmentation, where dense, spatial predictions make the model more sensitive to incorrect updates, and entropy minimization can amplify early misclassifications. Over time, this leads the model to drift away from its well-initialized source parameters, reducing its ability to generalize across the target domain.

Episodic (reset) adaptation avoids the runaway effects of carrying over errors, which is why it tends to be safer [2]. In contrast, fully online adaptation-though attractive for its potential to continuously refine the model-must confront these challenges.

We believe that online adaptation in segmentation can be improved through several future directions: (1) introducing occasional resets or using exponential moving average (EMA) of model weights to limit drift and reduce accumulated errors; (2) filtering high-entropy samples or gradients, which is particularly important in segmentation due to its spatial granularity and sensitivity to local noise; and (3) incorporating an auxiliary self-supervised objective, such as rotation prediction [1], to provide a more reliable adaptation signal-though this may come with increased computational cost. Notably, the fact that our method works effectively with very few test samples and does not rely on accumulating state makes it efficient for real-world applications.

K Visualization of Layer-Wise Confidence Weights

In the main paper (Figure 3), we presented the layer-wise confidence weights for the V20, Cityscapes, and COCO-Stuff datasets. Here, we extend the analysis by visualizing the weights for four additional datasets: V21, P59, P60, and COCO-Object in Figure 5. As with the earlier results, we plot the mean and standard deviation of the learned weights across layers under various corruption types.

L

7

L

8

L

9

L

10

L

11

L

12

L

13

L

14

L

15

L

16

L

17

L

18

L

19

L

20

L

21

L

22

L

23

L

24

Layer

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Weight

Pascal Context 59

Original

Defocus blur

Snow

JPEG compression

L

7

L

8

L

9

L

10

L

11

L

12

L

13

L

14

L

15

L

16

L

17

L

18

L

19

L

20

L

21

L

22

L

23

L

24

Layer

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Weight

Pascal Context 60

Original

Defocus blur

Snow

JPEG compression

L

7

L

8

L

9

L

10

L

11

L

12

L

13

L

14

L

15

L

16

L

17

L

18

L

19

L

20

L

21

L

22

L

23

L

24

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Weight

Pascal VOC 21

Original

Defocus blur

Snow

JPEG compression

L

7

L

8

L

9

L

10

L

11

L

12

L

13

L

14

L

15

L

16

L

17

L

18

L

19

L

20

L

21

L

22

L

23

L

24

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

Weight

CocoObject

Original

Layer

Layer

The observed trends closely mirror those reported in the main paper. In particular, deeper layers consistently receive higher confidence scores, while lower and mid-level layers still contribute under corrupted conditions. This reweighting behavior reflects the adaptive nature of our fusion strategy, which dynamically emphasizes the most reliable representations depending on the dataset and corruption type. These results further support the robustness and generality of our layer-wise uncertainty-aware fusion mechanism across diverse segmentation benchmarks.

L Visualization of Segmentation Maps

Figure 6 presents qualitative segmentation results on the v20 dataset, comparing predictions from the non-adaptive baseline, TENT, and our MLMP method. MLMP demonstrates stronger spatial consistency and fewer semantic errors, particularly in challenging regions where both the baseline and TENT struggle. The integration of intermediate-layer supervision appears especially beneficial for refining small object boundaries and correcting fine-grained details.

In Figures 7-15, we provide additional qualitative examples across a range of corruption types, using NaCLIP as the our OVSS backbone. These include both successful and failure cases under Gaussian noise, defocus blur, snow, and JPEG compression. Overall, the results illustrate the robustness of MLMP in mitigating noise-induced artifacts and improving prediction confidence, while also highlighting failure modes that warrant further investigation.

Image

Ground Truth

No Adaptation

TENT

Image

GroundTruth

NoAdaptation

TENT

Image

GroundTruth

NoAdaptation

TENT

Image

Ground Truth

No Adaptation

TENT

Image

GroundTruth

NoAdaptation

TENT

Image

GroundTruth

NoAdaptation

TENT

Image

GroundTruth

NoAdaptation

TENT

Image

GroundTruth

NoAdaptation

TENT

真

真

春

Image

GroundTruth

NoAdaptation

TENT

Image

GroundTruth

NoAdaptation

TENT

M Detailed Results of the Main Paper

This section provides complete versions of the experimental results that were summarized or partially reported in the main paper. It includes full tables for ablation studies and detailed comparisons with baseline adaptation methods across various datasets and evaluation settings. These results offer a more comprehensive view of the effectiveness and robustness of our proposed approach.

M.1 Ablation Studies

We provide here the detailed versions of the tables referenced in the ablation section of the main paper. Table 17 shows the detailed results of using different layer ranges in the proposed multi-level adaptation. Table 18 reports results for different strategies to integrate different prompt templates. Table 19 presents a detailed analysis of the impact of the number of templates. Finally, Table 20 presents results for the combination of all proposed components, showing individual and combined contributions.

M.2 Comparison with Alternative Adaptation Methods

This section presents comprehensive tables with the full experimental results corresponding to those summarized in the main paper. Table 21 reports detailed results for the V21 dataset, Table 22 for P59, Table 23 for P60, and Table 24 for the Cityscapes dataset. In addition to the datasets reported above, we also include detailed results on the ACDC [45] dataset, which is specifically designed to capture real-world adverse conditions such as fog, night, rain, and snow. It is worth mentioning that in ACDC, approximately half of the adverse-condition images have corresponding reference (clean) views captured at nearly the same locations. While these reference images are captured at approximately the same location as the shifted ones, the scene content may differ (e.g., different vehicles or objects present) due to real-world variability. Following our main protocol, we perform test-time adaptation independently on both reference and adverse views, and report the mIoU for each condition in Table 25. The results show that MLMP consistently improves over TENT and the non-adapted baseline across all conditions, confirming its robustness to real, non-synthetic distribution shifts.

Multi-Level Fusion

✗

✓

✓

✗

✗

✓

✓

✓

✓

✗

✓

✓

Multi-Prompt Loss

✗

✗

✗

✓

✗

✓

✓

✗

✗

✓

✓

✓

Image-Level Entropy

✗

✗

✗

✗

✓

✗

✗

✓

✓

✓

✓

✓

Uncertainty-Aware Weight.

✗

✗

✓

✗

✗

✗

✓

✗

✓

✗

✗

✓

Original

77.00

±

0.04

77.38

±

0.01

81.67

±

0.04

79.70

±

0.06

78.74

±

0.08

78.97

±

0.03

83.00

±

0.03

77.69

±

0.01

82.70

±

0.01

81.15

±

0.02

79.13

±

0.00

83.76

±

0.00

63.02

±

0.06

65.42

±

0.04

67.82

±

0.01

66.75

±

0.01

65.66

±

0.04

65.96

±

0.04

69.13

±

0.07

66.17

±

0.04

69.00

±

0.05

69.62

±

0.01

67.35

±

0.09

71.13

±

0.09

Shot noise

65.88

±

0.06

67.49

±

0.01

70.52

±

0.12

70.03

±

0.04

68.97

±

0.03

69.05

±

0.03

72.31

±

0.01

69.50

±

0.06

73.22

±

0.03

72.89

±

0.02

71.02

±

0.05

75.02

±

0.03

Impulse noise

64.17

±

0.04

65.11

±

0.17

68.10

±

0.01

67.88

±

0.07

66.35

±

0.12

65.39

±

0.12

68.86

±

0.13

65.16

±

0.10

68.77

±

0.15

70.31

±

0.09

67.17

±

0.09

71.34

±

0.11

Defocus blur

72.06

±

0.12

76.65

±

0.23

78.78

±

0.02

74.31

±

0.09

75.00

±

0.07

76.46

±

0.15

78.78

±

0.10

77.29

±

0.10

79.78

±

0.05

77.14

±

0.05

77.79

±

0.19

80.36

±

0.06

70.74

±

0.07

75.24

±

0.05

77.66

±

0.05

74.38

±

0.36

73.54

±

0.23

75.46

±

0.09

77.48

±

0.01

75.71

±

0.10

78.09

±

0.04

76.17

±

0.08

76.87

±

0.07

78.84

±

0.05

Motion blur

73.50

±

0.10

77.16

±

0.13

79.39

±

0.16

75.72

±

0.04

76.09

±

0.09

77.72

±

0.09

79.97

±

0.04

78.81

±

0.03

81.49

±

0.02

78.25

±

0.05

79.00

±

0.07

81.41

±

0.05

Zoom blur

61.36

±

0.07

64.22

±

0.12

66.50

±

0.16

64.61

±

0.01

64.04

±

0.02

66.38

±

0.15

68.41

±

0.13

65.12

±

0.16

67.69

±

0.08

68.32

±

0.17

67.61

±

0.05

69.41

±

0.12

Snow

71.04

±

0.05

72.64

±

0.07

76.39

±

0.02

74.66

±

0.01

74.16

±

0.02

73.25

±

0.12

77.31

±

0.11

74.05

±

0.08

78.50

±

0.16

77.20

±

0.02

74.94

±

0.06

79.53

±

0.05

Frost

67.01

±

0.02

67.30

±

0.03

71.17

±

0.00

69.50

±

0.01

69.31

±

0.00

67.57

±

0.19

71.73

±

0.21

68.31

±

0.12

72.81

±

0.08

71.34

±

0.02

68.69

±

0.11

73.20

±

0.07

Fog

70.54

±

0.07

72.73

±

0.03

76.55

±

0.06

73.33

±

0.03

73.41

±

0.00

75.06

±

0.08

78.38

±

0.03

73.48

±

0.13

77.62

±

0.05

75.98

±

0.02

75.81

±

0.02

79.81

±

0.06

Brightness

75.61

±

0.02

77.23

±

0.07

81.34

±

0.04

78.69

±

0.02

77.34

±

0.01

77.69

±

0.02

82.00

±

0.03

77.20

±

0.05

82.16

±

0.03

80.63

±

0.05

78.27

±

0.06

83.51

±

0.01

Contrast

70.51

±

0.04

73.36

±

0.08

76.87

±

0.06

74.09

±

0.01

74.14

±

0.14

73.57

±

0.05

77.42

±

0.09

74.09

±

0.11

78.09

±

0.08

76.97

±

0.01

74.75

±

0.06

79.06

±

0.16

Elastic transform

65.78

±

0.05

65.38

±

0.12

70.59

±

0.14

69.14

±

0.05

67.92

±

0.06

68.76

±

0.04

72.96

±

0.02

66.78

±

0.03

71.80

±

0.02

71.53

±

0.03

69.77

±

0.07

74.03

±

0.01

Pixelate

76.95

±

0.12

79.53

±

0.03

83.02

±

0.04

80.09

±

0.03

79.09

±

0.03

80.77

±

0.05

83.93

±

0.06

79.85

±

0.08

83.90

±

0.00

81.92

±

0.12

81.34

±

0.07

84.97

±

0.04

JPEG compression

71.84

±

0.15

74.38

±

0.12

78.73

±

0.08

75.56

±

0.02

74.77

±

0.11

76.77

±

0.00

80.81

±

0.09

74.61

±

0.06

79.79

±

0.02

77.98

±

0.02

77.94

±

0.07

82.06

±

0.01

V20-C Average

69.33

71.59

74.90

72.58

71.99

72.66

75.97

72.41

76.18

75.08

73.89

77.58

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method for test-time adaptation of contrastive VLMs for open-vocabulary semantic segmentation. The method can be summarized in two steps:\n\n1. Update/adaptation: the authors tune the layer norm modules by minimizing the entropy of the distribution resulting from a (softmaxed) dot-product between prompt embeddings and image features. For this, they use (a) features from the last 75% of ViT blocks, averaged together, and (b) 7 prompt templates. They then calculate loss terms for each template and average them.\n2. Inference: the authors predict the segmentation map using the same formulation as CLIP zero-shot for image classification but on a spatial token basis. They use the same seven prompts and features as in the adaptation step, but they re-weight the visual features based on their batch-wise entropy. \n\nThe authors demonstrate that their method significantly improves performance compared to the non-adapted baseline on seven datasets, both in its \"clean\" version and when corrupted using the same image corruptions as those in ImageNet-C.\n\nTheir method also performs significantly better than other test-time adaptation strategies originally conceived for image classification but adapted to open-vocabulary semantic segmentation.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The paper is reasonably well-written and easy to follow. The method is novel and tackles a previously understudied area.\n2. The method significantly outperforms non-adapted backbones and existing baselines adapted for open-vocabulary semantic segmentation. These improvements are validated across various datasets, corruptions, model sizes, and model families, the latter two explored in the supplementary materials.\n3. The methodological choices are explained through extensive ablations.\n\n---\n\n### Weaknesses\n\n1. Some parts of the paper could be further polished and are a bit unclear.\n\t1. In Figure 1, I would recommend adding an x-axis label for the figure on the left and specifying in the caption what backbone is being used and what V20 means (it's not immediately clear that this refers to Pascal VOC). Furthermore, it's not clear how the variance in the left plot was computed. Is it computed over a single example? Or multiple?\n2. The paper claims that using multiple prompts prevents degenerate collapse and ensures that the model yields segmentations that are consistent across diverse linguistic formulations but never shows a case of degenerate collapse with a single prompt. I am also skeptical of the second claim, as to my understanding, the authors always use the same set of prompts for both adaptation and inference across the paper's experiments.\n3. In Figure 4, it could also be interesting to see the performance with no adaptation to see how much a single prompt improves over the baseline.\n4. On line 212, you mention the GTAV dataset but do not cite it.\n5. On line 155, the paper states that the hypothesis that layers contain complementary information is validated in Figure 1b. However, this figure only compares the overall performance of MLMP to a non-adapted baseline and does not provide insights into this specific claim.\n\nQUESTIONS:\n1. The adaptation/inference loop is not fully clear to me. My understanding is that: you pick two test samples, perform 10 steps of adaptation, predict the masks of both samples, and then restore the model. Is this correct? If yes, why was a batch size of two chosen? Intuitively, I would expect that adapting for a single sample would work better.\n\t1. Would a larger (or smaller) batch size lead to better results?\n\t2. It could be interesting to explore an online version of the method, i.e., rather than discarding the updates, use them as a starting point for the next batch or sample. Would the performance degrade or improve significantly, as in [1]?\n2. WATT and CLIPArTT perform significantly worse than a non-adapted baseline. Do you have any insights on why this happens?\n3. Since the multi-prompt loss alone improves performance by a good margin, it could be interesting to see if further improvements can be achieved with longer prompts using either Long-CLIP [2] or TULIP [3].\n\n---\n\n- [1] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts, Yu Sun et al., ICML 2020.\n- [2] Long-CLIP: Unlocking the Long-Text Capability of CLIP, Beichen Zhang et al., ECCV 2024\n- [3] TULIP: Token-length Upgraded Clip, Ivona Najdenkoska, Mohammad Mahdi Derakhshani et al., ICLR 2025\n\nLIMITATIONS:\nThe authors appropriately discussed the method's limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel test-time adaptation (TTA) method, MLMP, for adapting vision-language models (VLMs) to segmentation tasks during inference. MLMP performs multi-level and multi-prompt (MLMP) entropy minimization by integrating features from intermediate layers of the vision encoder and optimizing both global CLS tokens and local pixel-level representations using different textual prompt templates. MLMP is a plug-and-play module that can be applied to any segmentation network without requiring additional training data or labels, and it remains effective even when only a single test sample is available. A comprehensive benchmark suite for open-vocabulary semantic segmentation (OVSS) TTA is constructed, covering seven segmentation datasets, 15 common corruptions, and 82 diverse testing scenarios, providing a standardized and thorough platform for future OVSS TTA research.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:  \n1、The focus on test-time adaptation for VLMs is of high practical relevance.  \n2、Theoretical analysis demonstrates that using multiple prompt templates helps reduce the variance of the loss gradient, laying a foundation for future research.  \n3、The experiments are extensive and thorough, supporting the proposed method from multiple perspectives.  \n4、The proposed method achieves significant performance improvements across multiple datasets.  \n\n**Weaknesses**:\n\n1、Feature fusion across layers and the use of image-level entropy as a loss are relatively common techniques, making this part of the method less innovative.  \n2、In the Experimental Setup section, it is stated that the adaptation process is conducted over 10 iterations with resets. However, as far as I know, TTA is typically evaluated under a one-epoch protocol to test effectiveness. How does the proposed method perform under that setting?  \n3、How exactly does the method support open-vocabulary settings? Line 120 mentions that a set of concepts C_k in C must be provided during inference. Does this imply that the model can only segment categories within the predefined set C? It is suggested to provide more detailed clarification to highlight the method's capability in handling open-vocabulary segmentation.  \n4、The method appears sensitive to the choice of prompt templates. Although multi-prompt optimization helps mitigate this issue to some extent, selecting more effective prompt templates to further improve performance remains an open challenge that warrants deeper investigation.\n\nQUESTIONS:\nPlease answer the questions in Weakness and refute the weaknesses I pointed out about Feature Fusion and Prompt Templates.\n\nLIMITATIONS:\nThe authors discussed their limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MLMP (Multi-Level Multi-Prompt), a test-time adaptation (TTA) framework for open-vocabulary semantic segmentation (OVSS). MLMP focuses on dense prediction tasks by:\n- Integrating features from intermediate vision-encoder layers (e.g., ViT layers) to capture complementary, shift-resilient cues, with uncertainty-aware weighting based on entropy.\n- Minimizing entropy across multiple text-prompt templates at both global and local levels to leverage prompt sensitivity as a robust adaptation signal.\nThe authors also establish an OVSS TTA benchmark to demonstrate its effectiveness in adapting to domain shifts during inference.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The writing and architecture figure is clear and easy to follow.\n- The observation revealed in Fig 1.(a) is interesting. And the ablation accordingly (Fig 3) is vivid.\n\nWeakness:\n- Weak Motivation:\nI question the necessity of the OVSS TTA task. Most domain shifts in the authors' benchmark (e.g., blur, noise) are artificially created and lack practical significance. Such image shifts can be easily resolved through training-time data augmentation or post-processing. Modern segmentation frameworks no longer emphasize these augmentations because these shifts are already considered solved problems. Thus, re-investigating adaptation for them is of limited value. The authors should identify more compelling TTA tasks or scenarios.\n\n- Missing Comparison:\nBeyond zero-shot (ovss are considered as zss), the authors should compare with classical few-shot segmentation (fss) methods. These solutions also address domain shifts by adapting with a few samples without requiring training. The lack of comparison and discussion with such methods undermines the justification for OVSS TTA's necessity and validity.\n\n\n- Overclaim of generalization ability:\nThe claim that MLMP applies to \"any OVSS framework\" is incorrect. It clearly only works for pixel-to-text similarity matching frameworks, not query-based ones (e.g., SAM, SEEM) that use decoders for segmentation. Additionally, validation is limited to NACLIP, no other CLIP-based or non-CLIP frameworks are tested or mentioned. The \"any OVSS framework\" claim is obviously an overstatement.\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes a novel TTA method—Multi-Level and Multi-Prompt (MLMP) entropy minimization—that adapts VLMs for segmentation using intermediate vision-encoder features and diverse text prompts at both global and pixel levels. The proposed method is plug-and-play and requires no extra data or labels. It further  introduces a comprehensive OVSS TTA benchmark with 7 datasets, 15 corruptions, and 82 test cases. Experiments show that the proposed method consistently outperforms classification-based TTA baselines in this setting.\n\nSTRENGTHS AND WEAKNESSES:\nStrength -\n\n1. Exhaustive ablations and comparison: The work showcases exhaustive ablation and comparison showing that the proposed method works well.  \n\n2. It proposes new benchmark - test time adaptation for Open vocabulary segmentation model.  \n\n3. The work is nicely written and well motivated.  \n\n\nWeaknesses - \n\n[A] Novelty: The technical contribution comprises of MLMP - that performs uncertainity aware entropy minimization of prediction probability across vision features, optimizing normalisation layers statistics. This has been the core idea of TENT work. Further, using multiple prompts[eg. OVSeg[1] have been used quite well for open-vocabulary segmentation. The technical novelty seems a bit limited.\n\n[1] Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP, CVPR 23\n\n[B] Comparison dataset - Although the comparison is quite exhaustive, but mostly focuses on synthetic corruptions (gaussian noise, zoom, motion blur etc) and doesn’t focus much on real world datasets. Prior test time adaptation works [2,3,4] primarily show results on adaptation from Cityscapes -> ACDC [2,3], realistic but synthetic SHIFT[2]. Would be good to add results atleast on realistic ACDC dataset.\n\n[2] Continual Test-Time Domain Adaptation, CVPR 22\n[3] Distribution-Aware Continual Test-Time Adaptation for Semantic Segmentation, ICRA 24\n\n[C] Selection of prompt: The work uses 7 prompt templates out of 80 CLIP standard prompts. However it doesnt seem straightforward to select the best performing set. How was this done? Further in these 7 prompts, there are prompts like - ‘a origami of <class_name>’, ‘a <class_name> in video-game’. How are such prompts helpful? They seem to out of domain for common corruption based images.\n\n[D] Minor: Any reason why last layer does not have higher confidence for COCOStuff dataset, as such is seen for PASCAL and Cityscapes dataset (Fig 3). Generally one would assume, last layer features to have more weightage in segmentation task as shown for PASCAL and Cityscapes datasets.\n\nQUESTIONS:\nMy main concern is \n\n1. how does the work select 7 prompts out of 80? Different selection might impact the performance.\n2. Comparison should be shown on realistic ACDC dataset.\n3. Technical novelty should be clarified.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}