Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation
Authors: Mehrdad Noori
David OSOWIECHI
Gustavo Adolfo Vargas Hakim
Ali Bahri
Moslem Yazdanpanah
Sahar Dastani
Farzad Beizaee
Ismail Ben Ayed
Christian Desrosiers
Venue: NeurIPS 2025 poster ()
Keywords: Computer Vision
Segmentation
Test Time Adaptation
Vision Language Models
Primary area: deep_learning
TL;DR: Proposed a test-time adaptation method for Vision-Language Models for the first time to improve segmentation performance across unseen domains during test time thanks to a multi-template multi-level loss.

Abstract: Recently, test-time adaptation has attracted wide interest in the context of vision-language models for image classification. However, to the best of our knowledge, the problem is completely overlooked in dense prediction tasks such as Open-Vocabulary Semantic Segmentation (OVSS). In response, we propose a novel TTA method tailored to adapting VLMs for segmentation during test time. Unlike TTA methods for image classification, our Multi-Level and Multi-Prompt (MLMP) entropy minimization integrates features from intermediate vision-encoder layers and is performed with different text-prompt templates at both the global CLS token and local pixel-wise levels. 
Our approach could be used as plug-and-play for any segmentation network, does not require additional training data or labels, and remains effective even with a single test sample. Furthermore, we introduce a comprehensive OVSS TTA benchmark suite, which integrates a rigorous evaluation protocol, nine segmentation datasets, 15 common synthetic corruptions, and additional real and rendered domain shifts, with a total of 87 distinct test scenarios, establishing a standardized and comprehensive testbed for future TTA research in open-vocabulary segmentation. Our experiments on this suite demonstrate that our segmentation-tailored method consistently delivers significant gains over direct adoption of TTA classification baselines. Code and data are available at https://github.com/dosowiechi/MLMP.

Test-Time Adaptation of Vision-Language Models for Open-Vocabulany Semantic Segmentation
Mehrdad Noori*
David Osowiechi*
Gustavo A. Vargas Hakim
Ali Bahri
Moslem Yazdanpanah
Sahar Dastani
Farzad Beizaee
Ismail Ben Ayed
Christian Desrosiers
LIVIA, ÉTS Montréal, Canada International Laboratory Learning Systems (ILLS)
Abstract
Recently, test-time adaptation has attracted wide interest in the context of vision- language models for image classification. However, to the best of our knowledge, the problem is completely overlooked in dense prediction tasks such as Open- Vocabulary Semantic Segmentation (OVSS). In response, we propose a novel TTA method tailored to adapting VI.Ms for segmentation during test time. Unlike TTA methods for image classification, our Multi-Level and Multi-Prompt (MLMP) entropy minimization integrates features from intermediate vision-encoder layers and is performed with different text-prompt templates at both the global CLS token and local pixel-wise levels. Our approach could be used plug-and-plas for any segmentation network, does not require additional training data or labels, and remains effective even with a single test sample. Furthermore, we introduce a comprehensive OVSS TTA benchmark suite, which integrates rigorous eval- uation protocol, nine segmentation datasets, 15 common synthetic corruptions, and additional real and rendered domain shifts, with a total of 87 distinct test scenarios, establishing standardized and comprehensive testbed for future TTA research in open- open-vocabulary segmentation. Our experiments on this suite demon- strate that segmentationn-tilored method consistently delivers significant gains over direct adoption of TTA classification baselines. Code and data are available at /MLMP
Introduction
Contrastive Vision-Languag Models (VLMs) such as CLIP 1 have demonstrated remarkable generalization capabilities by aligning vision and language modalities through large-scale pre-training. This versatility has positioned VLMs powerful foundation models for numerous downstream tasks 3 4 A promising direction for leveraging VLMs beyond elassification Open-Vone Vocabulary Semantic Segmentation (OVSS), where models aim segment objects beyond pre-defined set of categories, via VLMs' zero-shot recognition capabilities. Unlike traditional segmentation methods that require pixel-wise supervision, OVSS enables generalization to unseen object categories through language-driven representations.
Although existing OVSS methods have made significant progress, they remain vulnerable to domain shifts at test time, such as environmental changes or image corruptions, which may dramatically degrade segmentation quality. In the absence of mechanism enabling them to adapt to unseen test-time distributions, these models might lose their generalization capabilities, which limits their reliability in real-world applications. Consequently, there is an unresolved gap for the Test- Time
Eqqual contribution.
Correspondence to mehrdad.noorii
noori. Settoorr.@eesseeeeee @enssetsmtl.caa and david.osowiechi. www. @ens.etsmtl.ca
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Eskepy thean (120-C)
-23.25
-4.m%
Spadal
V00
Figure 1: Motivation. (a) Left: Mean + std entropy across seven text templates for the CLS token and the spatial tokens of the final and intermediate vision layers. Even the final-layer spatial tokens exhibit higher entropy and variability than CLS, and this sensitivity grows further in intermediate layers (numbers show % std increase relative to CLS). These patterns highlight pronounced prompt- induced uncertainty multiple depths and motivate both multi-level and multi prompt adaptation. (b) Right: mloU of the baseline vs. MLMP on clean and corrupted data, showing consistent absolute improvements and underscoring the effectiveness of our joint adaptation strategies. Here, V20 denotes the Pascal VOC 20 dataset, and V20-C represents the average performance over its 15 synthetic corruption types. The variance in (a) computed across all samples and all corruptions.
Adaptation (TTA) of OVSS models, which would enable models to dynamically adjust both to the task shift of VLM- segmentation and to the domain shifts encountered during inference.
To close this gap, we present novel Multi-Level Multi- Prompt (MLMP) test time adaptation strategy, the first fully test-time adaptation framework that could be plugged into any OVSS model, to the best of our knowledge, MLMP is lightweight and plug-and-play, boosting performance on the fly without access to labels. Its power comes from two key ideas: (i) adaptively integrating intermediate vision-encoder layers to harvest complementary, shift-resilient features, and (ii) a optimization that exploits VLMs' template sensitivity to provide robust adaptation signal across diverse text-template conditions.
The core requirement for test-time adaptation is reliable signal that faithfully reflects the current input distribution- even even under severe domain shifts or corruptions. To meet this need, MLMP begins by adaptively integrating intermediate layers of the vision encoder: earlier layers preserve fine-grained edges and textures, while deeper blocks encode semantic context, and each layer reacts differently when the data distribution changes. By aggregating these multi-level features into the adaptation process and weighting them by their confidence, MLMP harvests the most trustworthy signals for each input sample.
Beyond multi-level fusion, MLMP leverages VLMs' prompt sensitivity to model uncertainty. Prior work SS 6 shows that changing a prompt template, e.g., from "a photo of {class)" 1" to "an origami segmentation, we show that this effect is even more extreme: per-pixel predictions under different prompts diverge far more than the single CLS token used for classification (Figure CLS token vs. last-layer spatial tokens). This sensitivity effect is even more pronounced in intermediate feature maps. The intermediate layers that we fuse for reliability exhibit even stronger semplate-specifff shifts (Figure Zy: intermediate- layer tokens). Instead of viewing this inconsistency as weakness, MLMP models it directly by incorporating multi-prompt multi-level predictions into its adaptation objective function. This multi- prompt approach not only smooths out cemplate-speciffe noise, thereby reducing gradient variance and preventing degenerate collapse, but also ensures that the model yields segmentations that are consistent across diverse linguistic formulations. In this way, MLMP transforms prompt sensitivity into a powerful adaptation signal that complements its multi-level feature integration. As illustrated in Figure Ip. our MLMP method consistently outperforms the non-adapted baseline, achieving about 8-9 absolute mloU improvements on domain-shifee inputs, while also boosting performance on original (non-corrppted) images. This demonstrates the benefit of our joint multi-level, multi-promph adaptation.
of a (class)", could drastically change the classification performance.
We outline our key contributions follows:

---

Plug-and-Play TTA Framework for OVSS: We introduce MLMP, which is, to the best of our knowledge, the first fully test-time adaptation method that could be easily applied to any OVSS backbone.
Adaptive Multi-Level Fusion: MLMP integrates features from intermediate vision-encoder layers to capture complementary, shift-resilient cues. To further enhance robustness, we propose an uncertainty-avee aware strategy that re-weights features from individual layers based on their prediction entropy.
Multi- Prommt Local-Global Test-Time Optimization: MLMP turns prompt sensitivity into signal by directly minimizing entropy across different text prompt templates at both the global CLS token and local pixel-wise levels. This optimization naturally complements our multi-level feature fusion by enforeing consistency across linguistic perspectives and feature depths.
Comprehensive OVSS TTA Benchmark Suite: We curate rigorous evaluation protocol spanning nine mainstream segmentation datasets and 15 common synthetic corruptions, and additional real and rendered domain shifts, with total of 87 distinct test scenarios, estab- lishing a standardized and comprehensive testbed for future TTA research in open-vocabulary segmentation. Our experiments on this suite demonstrate that MLMP consistently delivers significant gains over baselines across all scenarios.
2
Related Work
Test-time adaptation (TTA) for open-vocabulary semantic segmentation (OVSS) remains unex- plored-existing TTA methods focus on classification or single-modality segmentation, while OVSS approaches use VLMS without any online adaptation. We bridge this gap with our proposed method MLMP, a plug-and-ples TTA framework that can be applied to any OVSS method.
Test-Time Adaptation. TTA addresses domain shifts by adapting pre-trained models to unlabeled target data without source samples. Methods like PTBN (7] and TENT [8] update batch statistics and affine parameters via entropy minimization but rely on large batches or augmentations. MEMO 9 simplifies this with single-sample augmentations, LAME [10] clusters features via Laplacian smooth- ing, and SAR 11 stabilizes adaptation using batch-agnostic normalization and sharpness-aware entropy minimization.
Test-Time Adaptation on Segmentation. TTA enhances segmentation robustness against domain shifts without source data. Methods include self-supervised adaptation via entropy minimization or contrastive learning 8 [2]. single-image adaptation optimizing per-image predictions [13, and continual TTA that leverages clustering prevent forgetting [14]. Multi-modal adaptation uses cooss-modal self-supervision [15], while active TTA integrates minimal human feedback for guided refinement [66. These approaches assume fixed, vision-only label space and rely spatial or surrogate tasks, making them ill-suited for zero-shot, text-driven OVSS. Consequently, none have
been applied to VLMs.
Open-Vocabulary Semantic Segmentation. OVSS enables segmentation of unseen categories using dision-languag models like CLIP. Approaches fall into fully-supervised, seakly-supervised. and training-free categories. Fully-supervised methods use pixel-wise annotations [17 18 [9],
while weakly-supervised ones leverage image-text pairs 20 21
22
23
Training-free OVSS avoids 26). Training-free OVSS
adaptation data but may rely on auxiliary pre-trained models 24
25
approaches aim to enhance segmentation without additional training data. Some methods, such as SCLIP 27], adjust self-attention mechanisms to improve feature localization, while others, like MaskCLIP [28]. refine feature extraction from CLIP's visual backbone. GEM (29 introduces additional optimization techniques to extract better dense features without fine-tuning. Among these, NACLIP 30] enhances CLIP's dense prediction capabilities by introducing neighborhood attention, which ensures that image patches focus on nearby regions, and by refining similarity measures to improve spatial consistency.
To the best of our knowledge, this is the first work to address TTA for OVSS models, filling a previously unexplored intersection between these fields.
To better situate this contribution within the broader landscape of general adaptation methods. Table summarizes the key distinctions among zero-shot inference, domain generalization (DG) R1 32
few-shot segmentation (FSS) [K), test-time training (TTT) 34 35 36), and our fully unsupervised test-time adaptation (TTA). This comparison highlights that MLMP addresses the most challenging
3

---

Table I: Comparison of general learning and adaptation paradigms. Here, x²; denote labeled source samples and x²; y" denote target (test) samples and labels. Domain Generalization trains on labeled multi-domain source data improve robustness unseen targets, while few-shot and test-time training methods rely on labeled or source data during or after training. Our approach (Fully TTA) adapts solely using unlabeled test samples, requiring neither supervision nor source access.
Setting
Target
Train
Zero-Shot Inference
Domain Geveralization (DG)
x², e"" (often multi-domain))
C(x" w²) (DG objectives)
X
ShotShot Learning (FSS)
(le))
C(x', v/) (FSS objectives)
Test-Time Training (TTT)
D(x" v/) Loax(x")
Lacar (2f Exnaup (2e))
Fully Tis-Time Adaptation (TTA, Ours)
and realistic scenario, adapting models entirely from unlabeled test data without any source access or annotated support samples.
3
Methodology
We first revisit the contrastive vision-language model (VLM) for open-vocabulary semantic segmen- tation (OVSS), and then present Multi-Level Multi-Prompt (MLMP) adaptation strategy.
3.1 OVSS with VLMs
Given an input image X #xx33 and set of concepts Cj C expressed in natural language, OVSS seeks semantic mask e [1..... ........K. K] HxW that assigns one concept to every pixel.
Following recent approaches for OVSS 30
employ transformer-based VLM extract visual and text features from the image and concepts in natural language. Specifically, we feed the image into the ViT-based vision encoder extract a visual token matrix F [[ccs]) fi..... fx] with each f, R.P. where = H/s} [W/s] is the number of patches of size sxs in the image and [cls] the CLS token for classification. We define Q = [q(cas] qı..... qNp with each q, E RP', the output features before the projection layer: F projí Q). At the same time, the text encoder is employed to extract text features t E _D for each concept C2 C. This is achieved by combining Cs with text prompt template, for instance "A photo of a [C,]" or "An image of a [C²]* where C is an arbitrary text description like "white horse".
The standard approach for classifying images with contrastive VLM such as CLIP [ computes the cosine between the CLS token features and text embeddings of classes, and assigns the image to the class with highest similarity:
arg max sim (fica=), tu), where sim (x,
y | ||
For extending this approach segmentation, instead compute the similarity between patch embeddings f; and text embeddings f; and assign a class/concept to each patch.
3.2 MLMP: Proposed Method
Figure 2 illustrates our full test-time adaptation pipeline, MLMP. MLMP integrates three com- plementary ideas: uncertainty-awarr multi-level fusion, image-level entropy minimization and multi-prompi adaptation.
We begin by modifying the entropy minimization objective of TENT g from image classification to work with spatial tokens. More specifically, for batch of B images, each containing N tokens, the probability that token belongs to concept
exp (sim (f,, tx)/T)
Pik
[0| K= exp (sim (f:, tsr)/T)
where T a softmax temperature scaling parameter, function normalizing the columns of its input matrix to unit length. Also, let P denote matrix
[t,,..., , tcc]- and norm(.) denotes

---

Sinturity Scores
Global Fasture
CLS Token
fix
Featir lneengiig
Feutures
Adaptation
Vition
Palch-Lavl
Bancngg
Evoluation Phose
Calcubitin the setragy of enorradii byort
1g
Arttaa Upsurpping
Festures
Patch-Lead Features
Sieviiiit Scarns
Figure 2: Overview of our MLMP method. In the Adaptation Phase, the model is adapted by leveraging multiple prompt templates alongside various intermediate feature layers, as well as the global feature. During the Evaluation Phase, the model computes weights based on the entropy of the intermediate features perform weighted averaging. These averaged features, combined with the different templates, are then used to generate the final segmentation map.
containing the probabilities in which could be expressed more compactly as follows:
P softmax (norm(F)) normm(T) /T).
The batch-wise entropy, which is minimized for adaptation, is then defined as follows:
[CI
H(P)
Pik log Pik-
B.N
Following 5 37 we keep the entire encoder frozen and update only the LayerNorm parameters of the vision encoder during adaptation. Freezing the text encoder greatly reduces computational overhead, since text embeddings be precomputed and reused across all test samples.
Uncertainty-Aware Multi-Level Fusion. VLM-based classification, the CLS token in the last layer of the visual encoder is typically used to compute the class label probabilities. This approach relies on the idea that the relevant information for classification lies at the end of the ViT and that intermediate layers serve to transform features. In segmentation, however, features from intermediate layers are often used to capture complementary information at different scales (38]. This hypothesis
is validated in Table as well as Figure 3] showing that a higher segmentation mloU is obtained when combining the teatures from different intermediate layers.
Inspired by this result, and leveraging the useful property of ViTs that the output of each layer has the same shape, extend the entropy-based loss described above layers. Denoting as of the visual features of patch obtained at layer E, we seek to aggregate the multi-level features into single vector q for segmentation prediction. A simple approach for doing this is to compute q by averaging o across all layers f. However, this approach ignores the relative contribution and confidence of each layer in the final segmentation. To address this limitation, estimate confidence weight a' for each layer based on its prediction entropy. First, we get visual features F proj( Q using the same projection head as for the final segmentation. Following the same approach as before, we then compute the batch-wise entropy of layer & as
features from multiple
ht H(P)), with p softmax (norm(F') norm(T) /T). Finally, the confidence weight of the layer obtained using a softmax as follows:
(5)
exp( exp(-9-h')
a = El=1 cxp(-x-Pt
(6)
5

---

Here, 3 is parameter controlling the "sharpness" of the weight distribution. During adaptation, we set / = 0 to promote a uniform contribution from all layers the prediction. During inference, we sharpen the distribution with value of f = 1, emphasizing the more confident layers in the final prediction.
With these confidence weights, we can now obtain our uncertainty-awarr multi-eveel (UAMl (UAML) features
proj(Q): Q), with Q
Ea'
(7)
giving the following satropy-based loss to minimize:
LUAML (T) H(P), with P (eltmax(normm)
FF) norm (T)
(8)
Image-Level Entropy Minimization. Since the CLS token is not directly linked to individual patch predictions but rather captures more global representation of the input, we also include an image-level entropy (ILE) minimization term specifically for this token. As illustrated in Figure [ the CLS token demonstrates increased robustness and reliability. This term, which encourages the model produce more confident global predictions is expressed as:
Ic
1
|cl=]
[cla] log Po,k
(TLE(T)
Po,k
B
b=
Here, |els] denotes the predicted probability for concept C₂ obtained using the CLS token in the last layer for the b-th sample.
Multi-Prompt Adaptation. Prior work TTA for classification [5] has shown the usefulness of leveraging multiple prompt templates in VLM to encode class labels, based on the idea that the templates capture complementary information about these classes. As shown in Figure Th, the sensitivity to the choice of prompt templates is even more pronounced in segmentation tasks, where fine-grained spatial predictions are required. Using multiple templates acts as cross-modal regularization, encouraging more stable and generalized learning signals. While different from image augmentation, it can be seen as a strong, safe, and lightweight rext-space augmentation. Rather than averaging the weights adapted from different prompt templates as in [5] computationally expensive approach for dense prediction tasks such as segmentation our method minimizes our proposed UAML and ILE losses across these templates. Let T, be the text features obtained using the 1-th template. Our final adaptation loss is defined as:
1 T'
Cana(()) (0) = T
Σ (CuAML(T.) + (TLE(TT)).
(10)
T
Theoretical Justification. Each template contributes its own adaptation loss, we optimize their average in Eq. (TO). By optimizing the adaptation loss of each prompt directly, we force the model to correct for the unique wording and visual cue of each template, rather than 'averaging' these differences in the text embedding space. This loss-level integration treats each template as an independent critic, translating diverse linguistic perspectives into separate gradient signals. Averaging those signals produces an unbiased descent direction whose variance decays as 1/T, enabling each adaptation step to represent the full prompt ensemble while being stable under noisy shifts.
Proposition (Unbiasedness and Variance Bound). Assume that each per-template gradient (0) Vo [CUAMI + LILE has variance bounded by o², then the ensemble gradient, defined by Vo Lanal 1 E Se (0), is unbiased and satisfies the following variance bound:
+² T
VaLanal
= E (0)]
Var VeLinal) =
Var (g.(0)) S
(11)
T
Proof. A proof of Prop. 1 is provided in the Appendix.
The 1/T reduction in gradient variance, as stated in Prop. 1, explains the improved stability we observe. Table 4 confirms that this loss-level ensemble outperforms alternative fusion strategies.

---

Table 2: mloU performance when using different layer ranges in the proposed multi-level adaptation.
L2s-01 (last two)
Loa-ai (last three)
Lig-21 (last 25%)
L15-21 (last 50%)
Ly-te (last 75%%)
L1-21
VIT-L/14 Layer Runge
(last)
(all layers)
V20 (Original)
77.00 gnoi
77.65 2002
77.66 2018
80.61 2005
80,50 2003
81.67
:::004
78.79 26.02
Gaussian Noise
63.02 20.6
64.41 #005
65,39 20.13
66.88 +0.18
66.88 =0:2
67.82 =::01
63.06 #0,09
Defocus Blur
72.06
72.93
72.84 same
76.10 10.1s
76.37 2005
78.78 ::002
77.56 :0.09
Snow
71.04 200s
72.09 2004
72.56 20m
74.47 1012
74.41 a0m
76.39
73.72 20.07
JPEG Compression
71.84
73.88
74.40
76.96
77.67
78.73
75.87
V20-C Average
69.33
70.33
70.78
72.89
73.45
74.90
72.02
0.5
Chrocape
OhocSturt
Oripmal
Organa
Sanskar Nove
Coao Nobo
Deficu
compossin
0.125
0.120
0.30
6025
6030
0.69
0.00
6025
0.0
00030
Figure 3: Mean and standard deviation of layer-wise confidence weights of MLMP across datasets. The fusion mechanism adaptively emphasizes more reliable layers based input conditions.
Experimental Settings
Following prior work on TTA in classification [37 5) we restrict updates to the normalization layers within the vision encoder. The adaptation process is carried out over 10 iterations using the Adam optimizer with a constant learning rate of 10-3 across all datasets. We use batch size of 2 images during adaptation across all datasets. For each new batch, the model undergoes a reset, restoring it to its initial weights before adaptation is applied.
Experimental Setup.
Datasets, In traditional TTA for segmentation, two datasets are commonly employed to simulate domain shifts shifts-one for model training and another for adaptation during inference (c.g.. GTAV and Cityscapes) as both must share the same semantic label space. In our study, as this is the first exploration of TTA for VLMs in segmentation tasks and given that VLMs are pre-trained, inspiration from ImageNet-C [3 to introduce 15 synthetic corruptions on segmentation datasets. Our experiments
draw
conducted on Pascal VOC 20 (v20), Pascal VOC 21 (v21) 40 Pascal Context 59 (P59), Pascal Context 60 (P60) [41]. and Cityscapes [42]. incorporating both original version (clean) and the synthetic 15 corruptions (denoted with "-C" suffix). For COCO-Stuff 4 and COCO-Object [44). we use only the original versions. To further evaluate robustness under real and rendered distributional shifts, we additionally include ACDC [45] cappttiin real-world adverse conditions such as fog, night, rain, and snow-and and GTA-V [46]. which provides photorcalistie, game-rendered urban scenes. This extended setup results in 87 distinct test scenarios encompassing synthetic, real, and rendered shifts, enabling a comprehensive evaluation of MLMP across diverse conditions.
Benchmarking. While MLMP is compatible with any OVSS framework, we incorporate NA- CLIP (30) with ViT-L/14 as our baseline OVSS model, which leverages neighborhood attention to enhance spatial consistency in a training-free manner. The compared methods include TENT (8] which serves as baseline and minimizes entropy during adaptation; CLIPArIT (37], which employs pseudo-labete labels generated via conformal learning: WATT [5]. which averages learnable parameters across multiple parallel branches; and TPT 6 which performs prompt tuning to adapt VLMs at test time. For a fair comparison, we modified all methods for the segmentation setting by processing all spatial tokens extracted from the VLM, rather than relying solely on the CLS token.
5 Results
5.1
Ablation studies
Effect of Intermediate Layers. To analyze the impact of layer selection in our uncertainty-aware multi-level adaptation strategy, we use different ranges of intermediate layers. As shown in Table 2
7

---

Table 3: mloU comparison of MLMP components, showing individual and combined contributions.
Multi- Multi-Level Fusion
x
x
Multi-Prompt Loss
x
Image-Level Entropy
K
Uncertainty-AWoe Awaree Weighting
X
X
X
V20 (Original)
77.00
77.38
81.67
79.70
78,74
78.97
83,00
77.69
82.70
81.15
79.13
83.76
Gaussian Noise
63.02
65.42
67.82
75
65.60 66
65. 96
69.13
66.17
69.00
69.62
67.35
71.13
Defocus
72.06
76.65
78.78
74.31
75.00
78.78
77.29
79.78
77.14
77.79
80.36
Snow
71.04
72.64
76.39
74,66
74.16
73.25
77.31
74.05
78.50
77.20
74.94
79.53
JPEG Compression
71.84 14
74.38
78.73 78.73
75.56
74. 77
76.77
80.81
74.61
79.79
77
77.55
82.06
V20-C Average
69.33
71.59
74.90
72.58
71.99
72.66
75.97
72.41
76.18
75.08
73.89
77.58
402
Table 4: mloU performance for prompt-integration strategies (Text, Params, Loss) clean and corrupted data.
710
794
724
Dataset: V20
Text
Params
Loss
700
Original
78.91 +0.07
74.46
=0221
79.70 0.06
473
08.
Gaussian Noise
66.27 10000
62.83
1004
66.75 10001
625
carage
Snow
Defocus Blur
73.78 #0002
74.05 #0.10
70.10
70.28
=0.30
=0.16
74.31
74,66
00009
0001
Figure 4: mloU performance of our method for different numbers of templates.
JPEG Compression
74.98 ::0.5
70.55
=0.11
75.56
0.02
V20-C Average
71.92
68.44
72.58
performance varies notably with the fusion range. While using only the final layers yields moderate improvements, incorporating the last 75% of the layers consistently achieves the best performance across both clean and corrupted inputs. This highlights that multi-level fusion is key driver of adaptation performance: earlier layers, although less semantically abstract, contribute valuable low- level features- such texture and edge cues whes-that enhance robustness to distribution shifts. In this ablation, we isolate the effect of multi-level fusion by applying only the first term in Eq. I0 using single prompt template and omitting the LILE term.
Effect of Uncertainty-Awarc Layer Fusion. We investigate strategies for aggregating multi-level features by comparing uniform averaging (B = 0) and uncertainty- aware fusion (3 = 1) during evaluation, using the same 75% layer range identified in the previous ablation. As shown in Table 3 incorporating entropy-based weighting improves performance by 4.29% on V20 and 3.31 % on V20-C. This highlights the importance of leveraging layer-wise confidence when aggregating features.
Visualization of Layer-Wise Confidence Weights. We visualize the mean and standard deviation of the learned layer weights to better understand the behavior of our uncertainty- fusion strategy. As shown Figure deeper layers tend to receive higher confidence, though earlier layers also contribute, especially under corrupted conditions. This variation is most pronounced in the Cityscapes dataset, where the distribution fluctuates more across layers and corruption types. In contrast, the COCO-Stufl dataset shows flatter distribution, where the final layer is not consistently the most influential. These results underscore the core strength of our fusion mechanism: its ability to adaptively reweight layers based on input conditions, assigning greater importance to those that remain more reliable under distribution shifts and corruption. Please refer to the Appendix for additional results on other datasets.
Effect of Global Image-Level Adaptation Term. The image-level entropy term, LILE complements our patch-level adaptation by encouraging consistent global predictions through the CLS token. While the levti- loss targets fine-grained spatial predictions, the IIE term introduces global context that helps stabilize adaptation. As shown in Table when added in isolation, I.JLE improves performance by 1.74% and 2.66% V20 and V20-C, respectively, demonstrating the benefit of incorporating global context under distribution shift.
Effect of Number Prompt Templates. To isolate this effect, we evaluate performance using different numbers of prompt templates while disabling both the multi-level fusion and the image-level entropy (IIE) term Eq TO As shown in Figure4 increasing the number of templates improves performance up to 7, after which the gains begin to saturate slightly decline. This trend holds across
8

---

Table 5: mloU comparison of MLMP and baselines several datasets. CLIPArTT could not be run for a few cases owing to GPU memory shortages. Full per-dataset results are in the Appendix.
OVSS Backbone: NACLIP
Adaptation Method
Dataset
No Adapt.
TENT
TPT
WATT
CLIPATTT
MLMP
V20 (Original)
75.91
77.00 1034
75.93 1001
57.73 10:05
72.77
83.76
Gaussian Noise
62.89
63,02 #006
62.98 #001
36.44
$3.36
71.13
Shot noise
66.26
65,88 10.06
66.33 1002
40,95 10,05
58.15 2:
75.02 10,03
Impulse Noise
63.
64.17 +0.4
63.12
63.12+#00 #001
34,90 +0:06
54.83 #:003
71.34 1:0.11
Defocus blur
72.59
72.06 10.12
72.55 1000
52.43 10:03
65.39 10:45
80.36 10.06
Gilss blur
71.44
70.74 2007
71.40::00
49.96 10:05
64,62
78.84
Motion blur
73.10
73.50 +0.10
73.16 16 +0002
53.35 +0:06
67.48 #0.17
81.41 +H1A5
Zoom blur
59.03
61.36 +0.07
59.00 =001
41.39 +0.05
52.37 1:
69.41 #0.12
V20-C
Snow
71.49
71.04
71.44 +:001
51.18
66.97
79.53
Frost
65,38
67.01 10.02
65.46 1000
45,75 10:05
60.48 1.0.08
73.20 10,07
Fog
70.69
70.54 +0007
70.70 1001 0001
52.96 00.04
67.85 =:00
79.81 #0.06
Brightness
74.95
75.61 10000
1:00
10:05
71.52
10.01
Contrast
71.51
70.51 +004
71.49 +0
50.74 +0.05
66.01 #0006
79.06
Elastic transform
62.86
65.78 20,05
62.95 1:001
45.45 10.34
60.41 1:0.10
74.03 10,01
Pixelate
77.28
76.95 10.12
77.31 1001
59,76 10.05
73.14 10.17
84,97 10.04
JPEG compression
72.59
71.84
72.56
$3,44:200
68.21
82,06 :0.01
Average
69.01
69.33
69.03
48.30
63.39
77.58
(Original)
45.65 =00:
:001
20:05
=:00
50.78
V21-C Average
40.75
40.95
40.77
24.12
34.16
46.25
P59 (Original)
28.23
28.73 +000
28.26 :001
16.55 20.44
24.60 =003
31.95 +10022
P59-C Average
23.88
23.88
23.88
13.37
19.72
27.03
P60 (Original) P60-C Average
24.95
25.29 10.01
24.98 1001
14.77 10,03
21.88 10.03
27.99 10,03
21.39
21.25
21.49
12.08
17.79
24.07
Scapes (Original)
29.49
30.54 +0.4
29.57 +000
20.77 +0:06
33.35 +:10
Scapess Average
21.63
21.64
21.60
13.45
23.02
COCOObject (Original)
23.80
24.88 +0011
23.84 +001
14.14 +0.0
21.34 #003
28.84 +0.00
COCOStulf (Original)
18.76
1001
1001
+0.01
both clean and corrupted settings, indicating that moderate number of diverse prompt templates is sufficient for MLMP. We therefore use 7 templates by default in our main experiments. Please refer the Appendix for template details.
Where to Integrate Multi-Prompi Information. Here, we empirically compare strategies for integrating multi-prompt information into the adaptation process. Specifically, we evaluate: (1) text-level averaging, where prompt embeddings are averaged before computing logits - technique commonly used in zero-shot learning [I; (2) learnable parameter averaging baseline (Params) inspired by WATT [) and (3) our proposed method (Loss). which incorporates all prompt templates directly into the adaptation loss (Eq [O). To isolate the effect of prompt integration, this analysis excludes other components such as multi-leve fusion and the image-level entropy (ILE) term. As shown in Table our loss-level formulation consistently outperforms the alternatives across both clean and corrupted settings.
Full Component Analysis. Table presents an extensive ablation evaluating the contribution of each component in our MLMP strategy. Each proposed element yields consistent gains when added independently, but is their combination that delivers the highest overall performance across both clean and corrupted settings, highlighting their strong complementary effects within unified framework. Additionally, we provide further ablations in the Appendix, including alternative OVSS backbones, different VLMs, ViT architecture variants, computational complexity analysis, and MLMP segmentation map visualizations, as well as discussions on the effect longer prompts, effect of adaptation iterations, and episodic vs. online adaptation.
5.2 Final Comparison with Alternative Adaptation Methods
Performance on Clean Data (No Distributional Shift). We begin by evaluating MLMP on clean test data (original), where no distributional shift is present. This setting is crucial, as TTA methods must avoid degrading performance when adaptation unnecessary. As shown in Table MLMP achieves strong mloU gains of +7.85, +5.66, +3.72, and +3.04 V20, V21, P59, and P60, and

---

+5.04/42.91 on challenging datasets COCOObject and COCOStuff. In contrast. most alternative adaptation methods fail improve performance in this setting. These gains highlight the robustness and generalization of our method, even when no explicit domain shift is present.
Performance Under Distributional Shift. Under distributional shifts, the advantages of MLMP become even more apparent. As shown in Table[5] MLMP consistently outperforms both the zero-shot baseline and existing adaptation methods, achieving mloU gains of +8.60, +5.50, +3.15, and +2.68 on V20-C, V2I-C, P59-C, and P60-C, respectively. Beyond standard corruptions, we further evaluate on the Cityscapes dataset, which presents natural domain shifts such as environmental variation, weather conditions, and resolution differences. Despite its challenging nature and low zero-shot performance, MLMP improves mIoU by +3.86, demonstrating its real-world adaptability. To push this further, we apply 15 corruption types to create Cityscapes-C, where MLMP still yields a + +1.39 gain. While TENT provides modest improvements, most other adaptation wrathods-ineluding ClipArTT, TPT, and WATT- fail to improve or degrade performance. These results highlight that naive strategies like pseudo-
prompt tuning, weight averaging are insufficient for open- vocabulary segmentation, and emphasize the need for iigmentation-speciine adaptation techniques such MLMP. Detailed results can be found in the Appendix.
While Cityscapes already embodies natural dis- tributional shifts caused by variations in lighting, camera viewpoint, and urban layouts, we further assess MLMP on datasets explicitly designed to capture targeted domain shifts. Specifically, evaluate ACDC 45,, which contains real- world adverse conditions (Fog, Night, Rain, and Snow), and GTA-V 4 photorealistic, game- rendered dataset that introduces a distinct syn- thetic distribution shift relative to real imagery seen during CLJP pre-training. As summarized in Table 6 MLMP achieves consistent improve- ments over both the non-adapted baseline and TENT across all ACDC domains, yielding on average +6 mloU gains under real-world condi- tions. Similarly, on the GTA-V dataset, MLMP improves by +3.8 mloU over the baseline, fur- ther confirming its robustness across both realis- tic and rendered distribution shifts.
Table 6: mIoU comparison on realistic (ACDC) and rendered (GTA-V) domain shifts. Full ACDC results (including reference/clean views) are pro- vided in the Appendix.
OVSS: NACLIP
Adaptation Method
Dataset
No Adapt.
TENT
MI.MP
Fog
23.88
26.89 =004
33.33
10.04
Night
22.12
24.17 10.0
24.76
1.003
Rain
23.86
26.84
1004
32.44
10.04
Snow
23.54
27.25
2005
2008
30.59 10.03
Average
23.35
26.29
30.28
GTA-V
25.09
26.62 =0.01
28.84 10002
6 Conclusion
We presented MLMP, plug-and-pla test-time adaptation framework for open-vocabulary semantic segmentation that can be integrated with any OVSS method. By combining uncertainty-awan fusion of intermediate VIT features with a novel loss-level integration of multiple prompt templates, MLMP consistently enhances performance across both clean and shifted domains-includinn common cor- ruptions and natural distributional shifts. Our comprehensive OVSS-TTA datasets and 87 distinct test rigorous evaluation protocol for future work in adaptive, minguage-award segmentation. While MLMP demonstrates strong, consistent gains, there remain opportunities to further refine its components. In particular, our current
nine MLMP's broad applicability and establishes
mechanism relies on entropy estimates from a shared projection head, which may not fully reflect each layer's 's unique characteristics. Future work could investigate more flexible architectures such as lightweight adapters or dedicated projection modules per layer-to more accurately assess and fuse intermediate features.
Acknowledgments
We appreciate the computational resources and support provided by Compute Canada and the Digital Research Alliance of Canada.
10

---

References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PMLR, 2021.
[2] Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard De Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In European Conference on Computer Vision, pages 388-404. Springer, 2022.
[3] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976-980. IEEE, 2022.
[4] Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmentation and tumor detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21152221164 2023.
[5] David Osowiechi, Mehrdad Noori, Gustavo Vargas Hakim, Moslem Yazdanpanah, Ali Bahri, Milad Cheraghalikhani, Sahar Dastani, Farzad Beizaee, Ismail Ayed, and Christian Desrosiers. Watt: Weight average test time adaptation of clip. In A. Globerson, L.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Process- Curran Associates, Inc., 2024.
ing Systems, volume 37, pages
[6] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in rision-language models. Advances in Neural Information Processing Systems, 35:14274-14289, 2022.
[7] Zachary Nado, Shreyas Padhy, D. Sculley, Alexander Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv:2006. 10963 [cs, stat), January 2021, arXiv: 2006.10963
[8] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
[9] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems, 35:38629-3866 -36622, 2022.
[10] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto, Parameter-free online test-time adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8344-8353, 2022.
[11] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation dynamic wild world, 2023.
[12] Hong Liu, Guoliang Kang, Shan You, Jiancheng Bao, Mingkui Li, and Yao Zhang. Source-free domain adaptation for semantic segmentation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2021.
[13] Jiacheng He, Zhilu Zhang, Zhen Wang, and Yan Huang. Autoencoder based test-time adaptation for semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 998-1007, 2021.
[14] Charanpal D Mummadi, Matthias Arens, and Thomas Brox. Test-time adaptation for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11828-11837, 2021.
[15] Gabriele Valvano, Andrea Leo, Kuniaki Saito, and Tatiana Tommasi. Learning multi-modal self-supervision for test-time adaptation. In NeurlPS, 2023.
[16] Zhi Wang, Yue Zhang, Kun Yu, and Hao Zhang. Active test-time adaptation for semantic segmentation. arXiv preprint arXiv:3312.01835, 2023.

---

[17] Davide Barsellotti, Roberto Amoroso, and Barbara Caputo. Enhancing open- vece--obyylaay se- mantic segmentation with prototype retrieval. In Pattern Recognition. ICPR International Workshops and Challenges, pages 243-258. Springer, 2023.
[18] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoe V Le, and Barret Zoph. Scaling open-vocauulary image segmentation with image-level labels. In European Conference on Computer Vision, pages 343-360. Springer, 2022.
[19] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open- veea-vocabulary semantic segmentation with mask -daated clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061-7070, 2023.
[20] Kaixin Cai, Pengzhen Ren. Yi Zhu, Hang Xu, Jianzhuang Liu, Changlin Li, Guangrun Wang, and Xiaodan Liang. Mixreorg: Cross-moda mixed patch reorganization is a good mask learner for open-world semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11 1196-1205, 2023.
[21] Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11165-11174 2023.
[22] Yuxin Chen, Yujie Li, Zhaoxiang Zhang, and Yunchao Liu. Exploring open-vocabulary semantic segmentation from clip vision encoder distillation only. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10010-10019, 2023.
[23] Jiarui Xu, Shalini De Mello, Sifei Liu, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144, 2022.
[24] Davide Barsellotti, Roberto Amoroso, and Barbara Caputo. Fossil: Free open-vocabulary semantic segmentation through synthetic references retrieval. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1234-1243, 2024.
[25] Andrea Corradini, Federico Bianchi, Debora Nozza, and Dirk Hovy. Freeseg-diff: Training-free open-vocabulary segmentation with diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 567-576, 2024.
[26] Lukas Karazija, Wafaa Bousselham, Andrei Bursuc, Thiemo Alldieck, and Patrick Pérez. Diffusion models for zero-shot open-vocabblar segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1010-1020, 2023.
[27] Feng Wang, Jieru Mei, and Alan Yuille. SCLIP: Rethinking self-attention for dense vision- language inference. In European Conference on Computer Vision, pages 315-332, Springer, 2025.
[28] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference Computer Vision, pages 188-304. Springer, 2022.
[29] Wafaa Bousselham, Andrei Bursuc, Thiemo Alldieck, and Patrick Pérez. Grounding every- thing: Emerging localization properties in liiounlanguage transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5678-5687, 2024.
[30] Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation. Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, 2025.
[31] Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Gustavo A Vargas Hakim, David Osowiechi, Ismail Ben Ayed, and Christian Desrosiers. Tfs-vit: Token-level feature stylization for domain generalization. Pattern Recognition, 149:110213, 2024.
12

---

[32] Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Gustavo A Vargas Hakim, David Osowiechi, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. Fds: Feedback-guide domain synthesis with multi-source conditional diffusion models for domain generalization. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), pages 8403, February 2025.
[33] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In proceedings of the IEEE/CVF international conference on computer vision, pages 9197-9206, 2019.
[34] David Osowiechi, Gustavo A. Vargas Hakim, Mehrdad Noori, Milad Cheraghalikhani, Ismail Ayed, and Christian Desrosiers. Tuflow: Unsupervised test-time training with normalizing flow. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2125 2126, Los Alamitos, CA, USA, jan 2023. IEEE Computer Society.
[35] Gustavo A Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Ismail Ben Ayed, and Christian Desrosiers. Clust3: Information invariant test-time training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1136-6145, 2023.
[36] David Osowiechi, Gustavo A. Vargas Hakim, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. Nc-ttt: A noise constrastive approach for test-time training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6078-6086, June 2024.
[37] Gustavo Adolfo Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. Clipartt: Light adaptation of clip to new domains at test time. arXiv preprint arXiv:2405.00754 2024.
[38] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAl 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceddigg, part III 18, pages 234-241. Springer, 2015.
[39] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. 2019.
[40] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Int. J. Comput. Vis., 88(2)-303-338, 2010.
[41] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[42] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[43] Holger Caesar, Jasper R. R. Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. CoRR, abs/1612.03716, 2016.
[44] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405 http/1005.3311 2014.
[45] Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Acde: The adverse conditions dataset with correspondences for semantic driving scene understanding. In Proceedings of the IEFF/CVF international conference computer vision, pages 10765-10775, 2021.
13

---

[46] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladien Koltun. Playing for data: Ground truth from computer games. European conference on computer vision, pages 102-118. Springer, 2016.
[47] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip Multilingual usioo-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14788 2025.
2:
[48] Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European conference on computer vision, pages 310-325. Springer, 2024.
[49] Ivona Najdenkoska, Mohammad Mahdi Derakhshani, Yuki M Asano, Nanne Van Noord, Marcel Worring, and Cees GM Snoek. Tulip: Token-length upgraded clip. arXiv preprint arXiv:2410.10034, 2024.
[50] Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin. On pitfalls of test-time adaptation. arXiv preprint arXiv:2306.03536, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, claim that we introduce a new method adapt VLMs for segmentation. In the paper, we present this method and prove its performance compared to state-of-art.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We mentioned limitations the last part of the conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[Yes]
Justification:
All our assumptions are included in the Introduction and the Methodology.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
present all our different settings which based previous methods for TTA. We also provide our code.
Justification: In the Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Everything detailed in our github link provided in the Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iiss.cc/ppll wwtpcccpppppppppeeeeeeeeeeeeeeee httpa////giieee/eeeSSbiiiiissss /Cc////oddiiisssseeeeeee sww.iissisiiiiiiiico for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Everything detailed in the section Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
All experiments have been conducted at least three times to ensure statistical robustness.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Everything detailed the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://neurips http///rrrp..ccpp.b..../ccccaaaa
Answer: [Yes]
Justification: We have considered all potential harms caused by the research process, socictal impacts, and potential harmful consequences, described in the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper addresses research field that is not currently utilized for direct practical applications, thereby limiting its immediate social impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In our paper, cite all sources that inspired work, and provide links in our git repository to the codes from which we drew inspiration.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
datasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We document all aspects of our research and provide link to our implementa- tion, along with step-by-step explanations for reproducing the results.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer:[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: This work does not use LLMs in the core methodology, scientific rigorousness, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hutt:://heaarigeccc//ooffrrrrere for what should or should not be described.
21

---

Test-Time Adaptation of Vision-Language Models for Open-Vocabulayy Semantic Segmentation Appendix
A Proof of Proposition
Unbiasedness and Variance Bound Proposition (Restated); Assume each per-template gradient g:(0) = Vo VolLUAML (Ti) + LLx(Li) has variance bounded by o². Then, the ensemble gradient, defined by V VoLaual + Et= (e)) is unbiased and satisfies the following variance bound:
T
d² T
E V (0)), Var(Ve
Var(g (0)) 1
T
Step 1: Unbiasedness. By linearity of expectation, we have:
E Linal
E
T Σ (c)
Σ [[g (l)) T
1
Hence, averaging the gradients from all T templates gives an unbiased estimate of the true gradient of the final loss.
Step 2: Variance Bound. Assuming independence with Var(g:) S o², we get
Var V Linal)
Var
1 (0(0))
Σ
T
T
(ar( - To² o²
T
7
7 T
Thus, the variance bound holds:
a²
Var V @Liinal)
5 T
This completes the proof of Proposition
B Implementation Details
Unless otherwise noted, all experiments utilize NACLIP 30 with ViT-L/14 as the OVSS backbone. We adapt only the LayerNorm parameters within the vision encoder, amounting to approximately 0.02% of the model's total parameters. Our adaptation setup follows prior work in classification [5] 37,, using the Adam optimizer with a fixed learning rate of 0.001 and 10 adaptation steps (iterations) across all experiments. Additionally we use batch of 2 images during adaptation. After each batch, model weights are explicitly reset to their original pre-adaptation values to ensure that each batch is adapted independently, without leveraging information from previously processed data. Following standard settings from [1] we use the default softmax temperature value of 100 in all experiments. All images resized 224 224 pixels. Due to the high resolution of images in the Cityscapes dataset, we split them into overlapping patches of size 224 x 224 pixels with an overlap of 112 pixels between patches. The segmentation predictions from these patches are aggregated reconstruct the final, full-resolution segmentation maps. No image augmentation techniques are applied during either the adaptation or evaluation phases.
All experiments are conducted on NVIDIA V100 GPUs equipped with 32GB memory. We implement approach using the PyTorch deep learning framework. To ensure statistical robustness and faimess in our comparisons, we repeat each experiment three times, reporting average performance along with standard deviation. We have provided detailed instructions and step-by-stey scripts in our repository clearly demonstrating how to generate datasets, perform the described test-time adaptations, and reproduce our reported results.
We adapt other baseline methods (TENT g. TPT 6,, WATT (. CLIPATTT 37 to work with spatial tokens. General adaptation hyperparameters (optimizer, learning rate, adaptation steps, and
22

---

Table 7: List of the seven prompt templates used in our MLMP method. These general- pprpose templates, originally proposed by the CLIP authors, serve as diverse textual views of each class and are not tailored to specific datasets or domains.
ID
Prompt template
T itap of {class}
T a bad photo of the {class),
73
origami {class}.
T photo of the large {class).
T a {class} in a video game.
T art of the {class}.
T7 photo of the small {class).
batch size) remain consistent with our setup. Method-specific hyperparameters or components are retained reported in their original implementations. Specifically, for WATT, used the sequential version (WATT-S) with default values I = and m = 5, for CLIPArTT we used the default I: = 3, and for TPT we used the 4 learnable tokens. Adapting these baseline methods to the segmentation task allowed us to systematically evaluate how various adaptation strategies- as prompt tuning (TPT), pseudo-labeling (WATT, CLIPArTT), prompt refinement (CLIPArTT), and weight averaging (WATT) perffomerm in the context of open-vocabulary segmentation.
C Template Details
Table Iists the seven prompt templates used in MLMP method. These templates were selected by the original CLIP author and are general-purposs not tailored to any specific image content. The CLIP repository also provides a full set of 80 prompt templates, which we use for larger-scale ablations.
More specifically, for the ablation in Figure of the main paper, we vary the number of templates T as follows:
T 1: the default CLIP prompt "a photo of {class}"
T 3: the first three templates {
7,,77) 7 T 7: all seven templates {Tr) ..... ] in Table 7
Table 7
. T 20: the first 20 templates from the 80-template pool
T 80: the complete set of 80 CLIP templates
D Dataset Details
This section details the datasets used in our experiments, along with the synthetic, real, and rendered domain shifts applied to evaluate robustness under distributional changes.
More specifically, conduct experiments
diverse set of segmentation benchmarks:
Pascal VOC (V20/V21) [40]: Contains 20 foreground object classes (v20) with a back- ground class (v21), widely used for benchmarking semantic segmentation tasks.
Pascal Context (P59/P60) []:: An extension of the Pascal VOC 2010 dataset, providing pixel-level annotations for more than 400 classes. Due to sparsity, a frequently used subset includes 59 object classes plus a background class, totaling 60 categories.
A large-scale dataset for semantic segmentation of urban street scenes. It comprises around 5,000 finely annotated images from 50 cities, recorded under various daylight conditions, featuring dynamic objects, varying layouts, and changing backgrounds, capturing significant natural domain shifts.
Cityscapes [42]:
http://gggh..oooooooo//oo//otaaa Cerith__III.... CLIP.ipynb
23

---

COCO-Stuff [3]:
Extends the original COCO dataset by adding annotations for back-
ground categories ("stuff"), resulting in 171 classes 80 objects, and 91 stuff categories.
subset of the original COCO dataset, consisting exclusively of 80 object categories without background annotations.
COCO-Object
ACDC 45 : Includes pixel-level annotations for 19 semantic classes, covering diverse driving scenes under fog, night, rain and snow conditions.
GTA-V [46]: Contains 24,966 synthetic images with pixel-accurate semantic labels; one version aligns to 34/19 classes compatible with real-world segmentation benchmarks.
In addition to the original versions of each dataset some of which already reflect natural domain shifts (e.g., Citsscaps)-we further assess the robustness of all methods by applying synthetic corruptions. Inspired by ImageNet-C [39], we apply 15 synthetic corruptions to evaluate the robustness of segmentation models under various perturbations. These corruptions include Gaussian noise, shot noise, impulse noise, defocus blur, glass blur, motion blur, zoom blur, snow, frost, fog, brightness variations, contrast variations, elastic transformations, pixelation, and JPEG compression. Each corruption is applied at severity level 5, representing the most challenging scenario.
We resize images from all datasets to 224 x 224 pixels. Due the high resolution of Cityscapes and ACDC images, we process them as overlapping patches of size 224 x 224 pixels (with overlaps of 112 pixels). Predictions for these patches are subsequently aggregated to produce the final segmentation maps.
This extended benchmark comprises total of 87 distinct test scenarios, encompassing synthetic, real, and rendered domain shifts. It provides a rigorous and comprehensive evaluation protocol that captures variations in resolution, scene diversity, object size, and semantic granularity.
E
Computational Complexity
In this section, we provide an exhaustive analysis of each test-time adaptation method's resource footprint by measuring: (i) latency, (ii) floating-point operations (FLOPs), (iii) peak GPU memory usage, and (iv) number of learnable parameters. For a fair comparison, all measurements were performed on a single test sample using the same NVIDIA V100 (32 GB) GPU. The results are summarized in Ta
We compare TENT, TPT, CLIPAITT, WATT, and our MLMP across all four complexity metrics. In TENT, WATT, CLIPArTT, and MLMP, only the LayerNorm parameters of the vision encoder are updated during adaptation, whereas TPT introduces additional learnable tokens the input of the text encoder. Additionally TENT, WATT, and MLMP use fixed sets of text features without a need for recalculating them during adaptation/evvaluaion in the "One-time Text Encoder" column) and then reused throughout both adaptation and evaluation phases. In contrast, TPT and CLIPAITT modify prompt embeddings or refine text templates during adaptation, requiring multiple forward passes through the text encoder.
all prompt templates can be encoded once (V
In terms of latency, MLMP completes both adaptation and evaluation in just 0.582 ms, which is only marginally slower than TENT (0.480 ms) but substantially faster than WATT (5.215 ms) and CLIPArTT (3.525 ms). Despite leveraging multi-level fusion and multiple prompt templates, MLMP maintains lightweight computational profile with only 82.4 GFLOPs and 82.9 GFLOPs for adaptation and evaluation, respectively-compeee comparable TENT and significantly lower than all other baselines. Furthermore, MLMP's ' peak memory usage is among the lowest at 2.093.9 9 MB, and like TENT, updates only 0.02% of the model parameters. These results highlight MLMP's efficiency: delivers rich representational capacity through multi-level and multi-prompt integration by boosting the results significantly (as shown in Table of the main paper), yet remains almost as lightweight as the simplest baseline.
F Performance with a Single Test Sample
As shown in Table 9 while TENT yields improvements on the original dataset, it leads to a perfor- mance drop V20-C. In contrast, our method, MLMP, consistently improves performance over the
24

---

Table 8: Computationallcompleee comparison across methods. For GFLOPs, only the forward- pass cost in adaptation and evaluation measured; by common practice, the cost pack-propggation in adaptation phase be approximated as twice the forward cost. A / in the second column indicates that all text information are encoded once and cached. A dagger (f) indicates that TPT adds additional parameters beyond the original network.
One-time Text Encoder
Time (sec.) 4
GFLOPs
Max Memory (MB) 4
Learn. Params (Ratio) +
Method
Adapt
Eval
Adapt
Eval
TENT TPT CLIPAITT
0.462
0.018
79.1
79.1
2,068.4
102,400 (0.02%)
x
0.445
0.031
275.6
275.6
2,583.1
3,072f (<0.01%)
X
3.494
0.031
1,755.5
275.6
8,928.5
102,400 (0.02%)
WATT
V
5.197
0.018
553.9
79.1
7,232,4
716,800 (0.17%)
MLMP (ours)
V
0.541
0.041
82.4
82.9
2,093.9
102,400 (0.02%)
no adaptation baseline, with gains of 8.77% on the otiginal data and 9.40% on the average across corruptions.
Table 9: mloU performance comparison when using single test sample for V20 dataset.
OVSS Backbone: NACLIP
Adaptation Method
Dataset: V20
No Adapt.
TENT
MLMP
Original
75.91
76.20
84.68
Gaussian noise
62.89
62.59
71.92
Shot noise
66.26
65.45
75.78
Impulse noise
63.16
63.34
72.35
Defocus blur
72.59
71.37
80.91
Glass blur
71.44
69.95
79.29
Motion blur
73.10
72.55
81.64
Zoom blur
59.03
60.64
69.99
Snow
71.49
70.03
80.64
Frost
65.38
65.96
74.33
Fog
70.69
69.39
80.77
Brightness
74.95
74.73
84.58
Contrast
71.51
69.74
79.78
Elastic transform
62.86
65.09
75.32
Pixelate
77.28
75.84
85.63
JPEG compression
72.59
70.85
83.20
V20-C Average
69.01
68.50
78.41
G
Generalization Across Model Variants
To evaluate the robustness and generality of our method, we conduct series of experiments using different model configurations within the segmentation pipeline. Specifically, we assess how MLMP performs when (i) changing the vision transformer backbone, (ii) switching between different open- vocabulary semantic segmentation (OVSS) formulations, and (iii) adopting an entirely different vasion-l model. These experiments demonstrate that our method maintains consistent improvements across a wide range of architectural and algorithmic configurations, confirming its flexibility and transferability.
G.1 Comparison Across VIT Backbones
To evaluate whether our method generalizes across different vision transformer backbones, we replicate the main experiments using ViT-B/16 and ViT-B/32 in place of the default ViT-L/14 model. These backbones represent lighter configurations, with fewer parameters and larger patch (32 and 16). As shown in Tables and []] MLMP continues to provide substantial improvements over all
25

---

baselines across both configurations, on both clean data (V20 Original) and under severe synthetic corruptions. These results confirm that the benefits of our multi-level and multi prommptppt adaptation strategy are not tied to model scale or specific architectural configurations, and remain effective even lower-resolution settings.
Table 10: Performance comparison of test-time adaptation methods using the ViT-B/16 backbone with NaCLIP as the OVSS model. Results are reported as mloU scores on the V20 dataset (original) and 15 corruption types. MLMP achieves the highest performance across all settings, demonstrating strong robustness even with a smaller backbone.
OVSS Backbone: NACLIP
Adaptation Method
Dataset: V20
No Adapt.
TENT
TPT
WATT
CLIPACTT
MLMP
Original
77.62
79.15 +0.06
77.63 40001
43.18 +0.07
72.70 #0.17
84.18 =0,7
Gaussian noise
48.00
52.04 #0.12
48.11 +:.00
18.83 #0.14
33.80 #0.52
61.67 =0.00
Shot noise
52.49
55.56 10.16
52.41 4:000
19.8640..1
37.62 #0.33
64.97 00110
Impulse noise
49.51
52.87 10.11
49.41 1:0.01
19.12 +0.23
35.92 +0.05
61.15 0.09
Defocus blur
68.03
69.85 1.0.4
67.88 1.0.01
37.54 1.017
56.77 10.19
76.71 0.00
Glass blur
62.17
65.14 10017
62.45 1.000
31.12 1:008
47.86 100115
72.62 0.044
Motion hlur
69.56
71.93 +0.02
69.54
36.89 1:0.5
58.64 +0.29
77.08 0.03
Zoom blur
47.34
52.30 10.19
47.38 1.001
22.83 #:012
33.52 =0.20
59.05 0.20
Snow
60.88
64.38 10.05
61.24
27.68 +:0.11
49.91
+:0388
71.41 0.155
Frost
55.45
58.38 10.14
55.44 1:0.00
29.66 1.0.23
46.94 10.04
67.42 0.009
Fog
67.07
70.01 10002
67.07 1:0.0
35.84 4:0.23
59.98 10.09
76.32 0.022
Brightness
73.33
75.14 10.17
73.23 4:000
40.87 1:009
67.30 10.33
82,14 1000
Contrast
60.30
63.30 10.04
60.20
29.68 10.12
48,42
70,022 10.0
Elastic transform
50.14
54,83 10001
50,00 1:0.01
23.32 1:0.44
43.67 1:000
63.65 0.07
Pixelate
75.48
77.44 1:003
75.31 1.0.01
42,26 4:0.88
67.33 10.08
83.29 1.0.1
JPEG compression
69.17
70.97 10007
69.15 1.001
34.94 1.000
60.30 10.54
79.35 00.12
V20-C Average
60.59
63.61
60.59
30.03
49.87
71.12
Table 11: Performance comparison of test-time adaptation methods using the VIT-B/32 backbone with NaCLIP as the OVSS model. Results are reported as mloU scores on the V20 dataset (original) and 15 corruption types. MLMP achieves the highest performance across all settings, demonstrating strong robustness even with a smaller backbone and smaller patch size.
OVSS Backbone: NACLIP
Adaptation Method
Dataset: V20
No Adapt.
TENT
TPT
WATT
CLIPAITT
MLMP
Original
72.43
72.83
1001
72.52
2000
50.71
20.10
67.67
20.5
79.95
2001
Gaussian noise
47.59
49.30 40.18
47.43
40.00
29.26
+0.09
37.36
40.17
59.27
1.06
Shot noise
51.80
54.43
40.1
51.83
40.00
31.82
20.15
41.07
40.21
63.73
20.1
Impulse noise
51.59 40.11
10.00
:0.01
37.65
1:017
58.81
0000
Defocus blur
60.23
61.86
1007
2000
36.54
48.97
2:0.21
Glass blur
54.59
56.73
54.80
28.70
37.96
65.34
Motion blur
59.53
60.49
59.65
35.80
47.61
66.71
7oom hlur
38.66
41.07
38.62 10000
21.72
24.14
24.39
1:0.8
49.53
1000
Snow
49.
51.01
48.88
27.55
20.17
40.30
62.30
11009
Frost
47.60
50.18 10.09
47.53 10.00
28.48
+0.15
39.23
1.012
60.22
0.00
Fog
56,25
59.47 40.88
56,22 10000
35.76
20000
47.63
#:0.9
68.18
1:000
Brightness
68.03
68.94
+0.04
67.95
+0.00
46.11
20002
61.07
#0.25
76.90
10099
Contrast
48.79
50.67
40.07
+0002 48.88
29.54
#00066
36.64
#:005
58,81
0:1.1
Elastic transform
52.73
55,59
40.20
52.75 40.00
29.86
#0.21
44.84
#0.188
65.39
0.09
Pixelate
68.61
69.15
40.01
68
2000
44,57
20.17
60.30
+0.41
77.28
1000
JPEG compression
63.86
65.12
4005
63.87 40.00
42.85
20.11
55.44
1.008
74.40
10009
V20-C Average
54.42
56.37
54.40
33.26
44.03
64.97
26

---

Table 12: Performance comparison of test-time adaptation methods using the original CLIP and SCLIP 177 as the OVSS model with a ViT-B/16 backbone. MLMP consistently improves performance across all settings, highlighting its strong generalization capabilities.
OVSS: CLIP
Adaptation Method
OVSS: SCLIP
Adaptation Method
Dataset: V20
No Adapt.
TENT
MLMP
V20
Adapt.
TENT
MLMP
Original
33.11
51,36
2000
61.47
20001
Original
78.20
79.12
84.91
Gaussian noise
22.74
37.78 2011
49.35
20002
Gaussian noise
45.65
49.72 101
61.20 20m
noise
23.67
38.86
Shot noise
50.21
54.09 #015
64.06
2um
Impulse noise
22.36
36.63
1015
47.81
1
Impulse noise
47.05
50.62 =a.6
59.73
=0.4
Defocus blur
31.85
48.33 :0.13
58.25
20M
Defocus blur
66.44
=006
75.03
sum
Glass blur
28.60
46,59
=0.29
56,02
Glass blur
62.01
64.79
72.22 =0.27
Motion blur
32.62
50.61
:0.00
59.55
10.31
Motion blur
68.95
70.81 =:.2
76.23
20.10
Zoom blur
23.42
39.43
47.65
Zoom hlur
45.12
48.50
57.84
Snow
28.38
48.05
20M
55.89
Snow
60.61
64.60
=0:30
71.70
201
Frost
26.20
1004
10.17
Frost
56.14
58.43
68.57
Fog
28.66
46,98
2015
56.62
2M02
Fog
68.34
70.03
$0.17
76.57
20.06
Brightness
33.71
51.77
=0.24
60.96
20.8
Brightnevs
82.91
1004
Contrast
26.06
42.86
53.55
Contrast
58.98
61.69
1:.0
69.57
20.10
Elastic transform
27.12
45.92
:0.11
51.02
:1.11
transform
56.45
10:0
1014
Pixelate
33.32
51.11
61.22
Pixelate
75.56
76,66
20:
82,99
2000
JPEG compression
31.64
49.76 =0:4
89.79 2:.6
JPEG compression
68.38
69.85 #011
78.69
=0.17
V21-C Average
28.02
45.33
$4.80
V21-C Average
59.98
62.49
70.82
G.2
Comparison Across OVSS Methods
Our method is designed to be flexible and agnostic to the underlying open-vocabulary semantic segmentation (OVSS) formulation. While all main experiments the paper use NaCLIP as the OVSS basiline-pparee with No Adapt., TENT, TPT, CLIPArTT, WATT, and our MLMP we further evaluate the generality of our approach by applying MLMP to two alternative OVSS methods.
Specifically, we consider the original CLIP D. adapted for pixel-wise segmentation via patch-level similarity, and SCLIP 277, which incorporates spatial priors into the vision-language matching process. As shown in Table 12 applying MLMP top of these OVSS baselines yields consistent improvements across both clean and corrupted settings.
G.3 Generalization to Emerging VLMs
To further assess the generality of our approach, we evaluate MLMP on Sig v2 47). one of the most recently introduced vision-language models. As shown in Table 13 MLMP continues to deliver consistent improvements across both natural and corrupted datasets, indicating that the core components of our method multi-level and multi-prompt aggregation generalizz well beyond CLIP-based architectures.
H
Effect of Longer Prompts
Recent studies such as Long-CLin CLIP 48 and TULIP [49 have explored extending the text length capability of Aision-language models, suggesting that richer linguistic descriptions may improve alignment. Motivated by this, we investigated whether incorporating longer and more descriptive prompts could further enhance MLMP's performance.
To this end, we generated extended templates derived from our original seven class-agnostic prompts using ChatGPT, together with extended class names where each category expressed in full natural language. For example, "a photo of the large (class)" becomes "a photograph of a very large (class), where the size of the object dominates the frame or is shown in contrast to smaller elements." while "aeroplane" becomes "a powered flying vehicle with fixed wings and engines, designed to transport people cargo through the air over long distances."
As summarized in Table 14, all experiments were conducted using the Long- 48 backbone. We evaluate three text variants: (1) the default short templates used throughout our main experiments, (2) Extended Templates, where each template replaced with a longer and more descriptive sentence, and (3) Extended Classes, where category names expanded into full natural-Languaa language descriptions.
27

---

Table 13: Performance comparison of test-time adaptation using SigLIP-2 as the OVSS model. Results are reported as mloU scores on the V20 dataset. MLMP shows improvement over the baseline in most corruption types.
OVSS Backbone: SigLIP-2
Adaptation Method
Dataset: V20
No Adapt.
ML.MP
Original
66.62
67.88 4:032
Gaussian noise
39.74
41.05
1:034
Shot noise
40.24
43,56
1:007
Impulse noise
40.76
42.43
4:0.22
Delocus blur
47.15
50.41
1:002
Glass blur
38.64
40.60
10.55
Motion blur
42.30
42.98
1:0.39
Zoom blur
27.29
29.20
1:03
Snow
3.85
4.84
Frost
21.32
23.12
Fog
70.73
70.86
1:004
Brighteess
18.45
19.14
1:0.27
Contrast
70.88
70.35
1:006
Elastic transform
38.41
38.71
1:005
Pixclate
57.85
59.14
1:007
JPEG compression
56.35
58.55
10.04
V20-C Average
40.93
42.33
Table 14: Effect of longer prompts and extended class descriptions on segmentation performance (mloU). All experiments use Long-CLIP CLIP as the OVSS model. Each pair of columns shows results without and with MLMP adaptation.
OVSS: LLng-CLIP
CLIP
Adaptation Setting
Dataset
Default Templates
Extended Templates
Extended Classes
No Adapt.
MLMP
Adupt.
MLMP
Adapt.
MLMP
V20
39. 64
54.45 10.09
35.55
53.43 10,06
25.66
43.92 1000
V20-C Average
29.36
48.71
26.92
48.90
21.09
39.01
V21
16.83
19.54
15.87
19.15
11.27
15.25
V21-C Average
13.64
18.57
12.98
18.27
10.23
14.66
For each variant, we report results with and without our MLMP adaptation (+ MIMMP. All scores correspond to mloU.
Overall, these results indicate that MLMP's adaptation mechanism effectively handles both concise and extended textual inputs without overfitting to prompt verbosity. We include this analysis to provide a clearer picture of MLMP's behavior under longer or more descriptive prompts.
I Effeet of Adaptation Iterations
The number of adaptation iterations in TTA varies across the literature. While some studies evaluate performance under single adaptation step, others perform multiple iterations to improve convergence. In our main experiments, we followed the common 10-iteration setup adopted in prior TTA works [8] 3.377 as it provides a consistent protocol for comparing vision-language adaptation methods such CLIPArTT and WATT. For fairness, used identical hyperparameters across all methods.
To further examine the influence of the iteration count, we also evaluated MLMP under a stricter one-ileration protocol, where only single adaptation step is performed. As summarized in Table [5]
ML.PP still improves over the baseline and outperforms TENT even with just one iteration, which demonstrates the effectiveness of our method even under a stricter setting.
28

---

Table 15: Evaluation of MLMP under single-iteration TTA protocol compared to standard baselines (mloU). All experiments use NACLIP as the OVSS model.
OVSS: NACLIP
Adaptation Method
Datasct
No Adapt.
TENT
MLMP
V20
75.91
76.80
79.88
V20-C Average
69.01
70.15
73,91
V2I
45.12
45.36 =0,00
50.24 =am
V21-C Average
40.75
41.02
46.07
Table 16: Comparison of episodic (reset-based) and online adaptation settings for MLMP and TENT (mloU). All experiments use NACLIP the OVSS model.
OVSS: NACLIP
Adaptation Setting
Dataset
No Adapt.
Episodic: TENT Episodic: MLMP
Online: TENT
Online: MLMP
V20
75.91
77.00
83.76 +0.00
76.44
79.19 +0.45
V20-C Average
69.01
69.33
77.58
69.32
71.77
J Episodic vs. Online Adaptation
We further investigate the difference between episodic (reset-based) and online adaptation settings in test-time adaptation (TTA). In our main experiments, all methods-inchin lines weere evaluated under the episodic setup. where the model is reset to its initial weights after each batch. This protocol avoids cumulative error propagation and is widely used in prior TTA works
MLMP and base-
To analyze the impact of continuous updates, we also tested an online version of MIMP, where model parameters are updated sequentially without resets and carried over the next batch. We used identical hyperparameters as in the episodic setting, and further experimented with lower learning rates and fewer adaptation steps for stability.
As shown in Table 16 we observe that while the online variant improves over baselines, still underperforms compared to the reset-based (episodic) setting. We believe this is primarily due to еттог accumulation and model drift, as documented in prior work showing that some recent classification TTA methods exhibit performance degradation in online setting-even when the distribution is static 50). This issue is particularly pronounced semantic segmentation, where dense, spatial predictions make the model more sensitive incorrect updates, and entropy minimization amplify early misclassifications. Over time, this leads the model to drift away from its well-initialized source parameters, reducing its ability generalize across the target domain.
Episodic (reset) adaptation avoids the runaway effects of carrying over errors, which is why it tends to be safer [2]. In contrast, fully online adaptation though attractive for its potential to continuously refine the model must confront these challenges.
We believe that online adaptation in segmentation can be improved through several future directions: (1) introducing occasional resets or using exponential moving average (EMA) of model weights to limit drift and reduce accumulated errors; (2) filtering high-entropy samples or gradients, which is particularly important segmentation due to its spatial granularity and sensitivity to local noise; and (3) incorporating provide a more reliable adaptation signal-though this may come with increased computational cost. Notably, the fact that our method works effectively with very few test samples and does not rely on accumulating state makes it efficient for real-world applications.
auxiliary self-supervised objective, such rotation prediction [1], to
K Visualization of Layer- Wise Confidence Weights
In the main paper (Figure 3, we presented the layer-wise confidence weights for the V20, Cityscapes, and COuffStuff datasets. Here, we extend the analysis by visualizing the weights for four additional datasets: V21, P59, P60, and COCO-Object in Figure 3 As with the earlier results, we plot the mean and standard deviation of the leamned weights across layers under various corruption types.
29

---

Pascal Context
Pascal Context
Drghil
0.30
0.30
Gaussiion
Defecus blar
Defocus blut
Snow
Show
compression
Weight
0.20
Weegh
9.20
0.15
0.15
0.05
0.05
0.00
0.00
616
622
414
Layer
Layer
Pascal VDC 21
CocoObject
0.35
Originl
0.14
Origind
Gausstan
0.25
IIEG compression
Wegge
Weight
0.10
0.02
0.00
0.00
0.
1m
197
19
122
Layer
Layer
Figure 5: Mean and standard deviation of weights of intermediate layers for several datasets.
The observed trends closely mirror those reported in the main paper. In particular, deeper layers consistently receive higher confidence scores, while lower and mid-level layers still contribute under corrupted conditions. This reweighting behavior reflects the adaptive nature of our fusion strategy. which dynamically emphasizes the most reliable representations depending on the dataset and соrruрtion type. These results further support the robustness and generality of our layer-wise uncertainty-awarc fusion mechanism across diverse segmentation benchmarks.
Visualization of Segmentation Maps
Figure 6 presents qualitative segmentation results the v20 dataset, comparing predictions from the non-adaptive baseline, TENT, and our MLMP method. MLMP demonstrates stronger spatial consistency and fewer semantic errors, particularly in challenging regions where both the baseline and TENT struggle. The integration of intermediate-layer supervision appears especially beneficial for refining small object boundaries and correcting fine-grained details.
we provide additional qualitative examples across range of corruption types, using NaCLIP as the our OVSS backbone. These include both successful and failure cases under Gaussian noise, defocus blur, snow, and JPEG compression. Overall, the results illustrate the robustness of MLMP in mitigating iniseeidduced artifacts and improving prediction confidence, while also highlighting failure modes that warrant further investigation.
Figures 15 15|
30

---

Figure 6: Qualitative results for No Adapt.. TENT and MLMP on V20 Original.
Figure 7: Failed Cases of MLMP on V20 Original.
Figure 8: Good Cases of MLMP on V20 for V20 gaussian noise corruption.
31

---

AGNITATIO
Figure 9: Failed Cases of MLMP on V20 gaussian noise corruption.
Figure 10: Good Cases of MLMP on V20 defocus blur corruption.
Inowye
Genurd Tuth
No Adiersti
TEXT
Figure 11: Failed Cases of MLMP on V20 defocus blur corruption.
32

---

Geound Thin
Adiypntiinn
TPNT
Figure 12: Good Cases of MLMP on V20 snow corruption.
Figure 13: Failed Cases of MLMP on V20 snow corruption.
Image
Genund Tuth
No Mo-kkiesttnnn
TEXT
Figure 14: Good Cases of MLMP on V20 JPEG compression corruption.
33

---

Genurd
Figure 15: Failed Cases of MLMP on V20 JPEG compression corruption.
M Detailed Results of the Main Paper
This section provides complete versions of the experimental results that were summarized or partially reported in the main paper. It includes full tables for ablation studies and detailed comparisons with baseline adaptation methods across various datasets and evaluation settings. These results offer more comprehensive view of the effectiveness and robustness of our proposed approach.
M.1 Ablation Studies
We provide here the detailed versions of the tables referenced in the ablation section of the main paper. Table 17 shows the detailed results of using different layer ranges in the proposed multi-level adaptation. Table 18 reports results for different strategies to integrate different prompt templates. Table presents a detailed analysis of the impact of the number of templates. Finally, Table 20 presents results for the combination of all proposed components, showing individual and combined contributions.
M.2 Comparison with Alternative Adaptation Methods
This section presents comprehensive tables with the full experimental results corresponding to those summarized in the main paper. Table 20 reports detailed results for the V21 dataset, Table [2 for P59, Table 23 for P60, and Table 24]for for the Cityscapes dataset. addition to the datasets reported above, we also include detailed results on the ACDC (45) dataset, which is specifically designed to capture real-world adverse conditions such as fog, night, rain, and snow. It is worth mentioning that in ACDC, approximately half of the adverse-condition images have corresponding reference (clean) views captured at nearly the same locations. While these reference images are captured at approximately the same location as the shifted ones, the scene content may differ (e.g., different vehicles or objects present) due to real-world variability. Following our main protocol, we perform test-time adaptation independently on both reference and adverse views, and report the mloU for each condition in Table 25 The results show that MLMP consistently improves over TENT and the non-adapted baseline across all conditions, confirming its robustness to real, non-synthetic distribution shifts.
34

---

Table 17: mloU performance over different layer aggregation strategies. Maximum value in each row is highlighted.
L24
L23-24 (last two)
L22-24 (last three)
L19-24 (last 25%)
L13-24 (last 50%)
L7-24
L1-24 (all layers)
ViT-L/14 Layer Range
(last)
(last 75%)
Original (V20)
77.000 =0.04
77.65
+0.02
=0.09
80.61
+0.05
80.50
=0.03
81.67
10004
78.79 #0.02
Gaussian noise
63.02 1006
64.41
10.5
65.39
10.13
66.88
10.18
66.88
1.002
67.82
1.001
63.06 1.0.09
Shot noise
65.88 1006
67.74
68.43
1004
70.11
70.40
10002
70.52
1..12
64.88
1.0022
Impulse noise
=0.4
65.53
=6.19=012
67.24
+0.17
67.15
=0.09
68.10
10001
62.09 10.05
Defocus blur
72.06 =0.12
72.93
+0.19
72.84
=0.02
76.10
20.16
76.37
=0.05
78.78
+0002
77.56 +0009
Glass blur
70.74 =0.07
+0.06
=0.5
=0.15
74.20
+0.12
75.53
#0.05
77.66
+0005
75.85 #0,02
Motion blur
73.50
73.74.1000
70.62
76.83
+007 77.50
20.07
79.39
10.16
77.22 10.15
Zoom blur
61.36
62.10
62.56-0.00
63.99
40.20
64.59
10.14
66.50
64.79
10.01
Snow
70.04
72.09
1004
72.56=0.0
74.47
40.12
74.41
10011
76.39 10002
73.72 10.07
Frost
67.01 =0.02
66,92
+0002
67.23
00.04
68,94
10.01
70.03
10.01
71.17 10.00
68.23
100022
Fog
70.54:0.0
71.44
10.07
71.50
:0.05
74.50
10.16
74.62
10.08
76.55 1.006
74.37
1:0.0
Brightness
75.61
76.27
76.80
79.16
10.07
79.64
81.34
10.4
79.11 1:0055
Contrast
70.51 =0.04
40.12
72.17
74.58
+0.08
75.42
+0.09
76.87
10006
74.09 +0.00
Elastic transform
65.78 =0.05
66.08
+008
66.06
=0.06
68.62
20.15
70.38
+0.09
70.59
00.14
67.91
#0.12
Pixelate
76.95 =0.12
78.38
+0002
78,46=0.0
80.70
81.11 =0.03
83.02
0.00
81.53 #0.02
JPEG compression
71.84 =0.15
73.88
74.40=0.0
+0.02
77.67
+0.03
78.73
0.00
75.87 10.19
V20-C Average
69.33
70.33
70.78
72.89
73.45
74.90
72.02
Table 18: mIoU performance comparison for different strategies integrate different prompt tem- plates.
Dataset: V20
Text
Params
Loss
Original
78.91 +0.77
74.46
=0.21
79.70 -0006
Gaussian noise
66.27 40.00
62.83
10.04
66.75
100001
Shot noise
69.78 10.10 67.10
#0.12
70.03
10004
Impulse noise
66.73 1:003
10.52 64.57
67.88
1:000
Defocus blur
74,05 10.10
70.28
10.16
74.31 1:0.09
Glass blur
73.31 +0.08 70.28
-0.34
74.38 1:036
Motion blur
75.20 +0.09
71.56
+0.09
75.72 00.04
Zoom blur
63.31 +0.11
61.29
=0.09
64.61 +0001
Snow
73.78 1002
70.10
10.30
74.66 1.001
Frost
68.90 1000 66.42
10.04
69.50 10.01
Fog
72,60 1003 67.63
10.07
73.33 10.03
Brightness
78.16 +002 74.05
+0.03
78.69 +0002
Contrast
73.75 +0.06 69.47
#0.24
74.09 +0,01
Elastic transform
68.41 1002 64.98
10.09
69.14 1:005
Pixelate
79.59
75.54
10.05
80.09 10.03
JPEG compression
74.98 4005
70.55
+0.11
75.56 +-0.02
V20-C Average
71.92
72.58
35

---

Table IoU performance of our method for different numbers of templates.
Dataset: V20
Template
Templates
Templates
20 Templates
80 Templates
Original
77.00 1004
79.48 =0.08
79.70 0.066
79.17 10.01
79.25 40002
Gaussian noise
63.02 #0.66
66.51
=0.10
66.75
0.001
63.94 10.09
66.02 40.02
Shot noise
65.88 10006
70.20
=0.03
70.03
10.04
68.59 10.01
69.00 10001 +0.00
Impulse noise
64.17 :0.04
67.57
=0.01
67.88 0.007
66.53 #0.06
66.91 +0002
Defocus blur
72.06 =:0.1
74.66
0.002
74.31 +0.09
73.89 12.12
74.09 +0.06
Glass blur
70.74 1:0007
74.38
0.12
74.38 0.36
73.78 1:009
73,88 1:0.7
Motion blur
73.50 1:00.0
75.80
75.72 1.004
75.19 10.04
75.19 10.04
Zoom blur
61.36 1:0007
63.47
0.00
64.61 0.01
63,50 10003
63.84 1:004
Snow
71.04:0000
74.09 =0.08
74.66 0.001
73.78 #0.03
73.91 #0.01
Frost
67.01 =0002
69.09
=0.01
69.50 0.00
69 69.08 #0.00
69, 10 +0.02
Fog
70.54 =0.07
73.69
=0.01
73.33 +0.03
72.87 #0.03
72.86 +0.05
Brightness
75.61 1:002
78.49
78.69 10.02
77.86 10.02
78.06
Contrast
70.51 1:0004
74.01
10.22
74.09 1..01
73.48 10.04
73.56 10009
Elastic transform
65.78 1:005
68.09
10.01
69.14 0.05
67.94 10.02
68.31 1:009
Pixelate
76.95 10.12
79.91
20.04
80.09
10003
79.15 10.06
79.41 10.05
JPEG compression
71.84 10.15
75.00
=0.05
75.56 0.02
74.45 +0000
74.75 +0.01
V20-C Average
69.33
72.33
72.58
71.74
71.93
Table 20: Detailed mloU comparison of MLMP components, showing individual and combined contributions.
Mali Level Fusius Matt-Proree loe. Imags- and Urmeery
Drightl
17.00
11.36
$166T-0m4
79.70
76.74
53.00
61.15
75.13
$1.76
Ganscin reiee
63.00 2m1
26.2..10
61.83 61
65.75 2m
61 sPa
61:96
2m:14
09 11:25m 2m
66 2m
69.00
2m
50.63 2m
61.15 2m14
71.13 =ap
Shotmois
55.55 1M8
57.49 1u0
70.51 .012
7003 Loss
68.07 Le03
00.03 L033
72.31 11:0
00.50 11/
73.12 1u0
72.89 Lu2
71,02 JU6
75.02 .Ms
Impiia
L8:1
1113
1110
68.77 1815
1009
67.17 .009
58.36
30.14
15.34
24.36
71.54
mmm
15.71
18.09 TK:0-40m
18.17
75.57
75.34
Motun har
77.16-111
75.73 anc
76.09 am.0
6m:00
79.90 23:4
T8 230
11.49 2010
18.25 2m
79:00-000
$1.41 -M
Zeom
51.16-501
24.22.1122
65.50-0155
00.41 2.13
52:09 2018
61.61
69:43 -0.1
1511
11/8
2018
2005
07.57
68:31
12.51
65.69
73.20
Fig
70.54 2007
12.75 2m0
76:55 .0t6
13.33 L0B
73.41 10:00
75.00
10:35
73.30
13:00
73.46
131
17.62 2308
15.96 200
75.81 2012
75.11 PM66
Bigghnss
75.1 400
77.23 am0
$1.34 -014
74,99 408
27:34 46:31
72.89 60:22
82,00 +1:00
77:30 77:30 61/8
12.18 4000
50.3 4m05
78.27 -015
$1.51 -011
Denzal
13.36.008 au3
66:33
77.42 61.3
4111
18.09 40/8
18.97 4M0
79:06 -035
Phodeu
36.55
70.55
53.50
61.34
51337
19G comprsscion
21
14.35 211
76.71-004
75.56 16.6-2m 20
74.77 2011
76.77 80:00
8051 23.6
76 21
19 19793m gm
17.08 g0
77:84 21:94-000
2000
$1.06 :041
Avarge
72.58
71.09
72.65
15.97
72.41
15.08
73.89
Table 21: mloU comparison of MLMP and baseline methods the V21 dataset, evaluated on both the original images and 15 corruption types.
OVSS Backbone: NACLIP
Adaptation Method
Datuset: V21
No Adapt.
TENT
TPT
WATT
CLIPArTT
MLMP
Original
45.12
45.65 +0.02
45.17 1:000
28.58 10.05
39.50 :0.04
50.78 #000
Gaussian noise
37.40
37.95 +0000
37.34 +:000
20.93 +0.07
30.05 10.18
43.59 :0.01
Shot noise
39.33
39.17 10.03
39.23 40.00
22.06 10.06
32.07 10.17
45.55 20002
Impulse noise
37.81
37.73 +0.04
37.78 40000
19.95 10.05
30.52.10.0
43.89 00001
Defocus blur
41.46
41.46 20.03
41,46 +0000
24.79 10.04
34,27 20.05
46.00 0.00
Glass blur
41.76
41.55 +0.01
41.82
24.61 +0.03
34.55 20.16
46.83 0.04
Motion blur
42.65
42.81
42.74-100.00
25.66
36.07
47.72 0.04
7oom blur
34.46
34.25 +0.00
34.44
20.74 10.05
28.44 10.07
39.07 0.00
Snow
40
40.47 +0.00
40.23 40.01
25.02 10.06
33.98
46.30 0.005
Frost
40.70
20.08
40.80 +0.00
23.43 +0.00
34.09 =0.01
46.78 0.00
Fog
42.50
40.07
42.47 +:000
24.95 +0.05
36.37 20.04
47.61 0.06
Brightness
44.21
44.64
44.27
27.07
38,44.10.1
49,89 000.8
Contrast
40.44
40.14
40.44
23.89 40.04
33.68
45.22 0.00
Elastic transform
40.63
41.78 10001
40.67 4:000
24.40
35.38
46.87
Pixelate
44.70
44.95 20003
44.79 +0.00
27.74 10.06
38,14 18.104
50. 0001
JPEG compression
43.05
42.87
43.05 +:000
26.04 +0.05
36.29
48.27 0002
V21-C Average
40.75
40.95
40.77
24.12
34.16
46.25
36

---

Table 22: mloU comparison of MLMP and baseline methods on the P59 dataset, evaluated on both the original images and 15 corruption types.
OVSS Backbone: NACLIP
Adaptation Method
Dataset: P59
No Adapt.
TENT
TPT
WATT
CLIPAITT
MLMP
Original
28.23
28.73 +0.02
28.26 +0.01
16.55 #0.05
24.60 =0.00
31.95 =0002
Gaussian noise
21.53
21.49
21.59
11.27
16.42
24.84 0.00
Shot noise
22.35
22.31
22.26 4:000
11.80
17.47
25.62 0.01
Impulsc noise
21.74
21.59
21.72
11.24
17.17
24.75
Defocus blur
25.42
25.14
25.41
14.31
20.71
27.95
Glass blur
25.03
24.70
25.01 1:0.00
20.63
27.66
Motion hlur
26.11
26.00
15.13
21.72 10.00
29.12 1003
Zoom blur
19.20
19.49
19.20 4:0.00
10.86
15.39
21.98 0.02
Snow
22.45
22.29
22.45 1:0.00
13.60
19.17
25,47
Frost
21.95
21.94
21.97 4:000
12.83
18.77
24.52
Fog
24.85
24.91
24.86
13.85
20.66
27.84
Brightness
27.39
27.80 10.01
4:.00
15.55 +0.04
23.56
30.63
Contrast
23.55
23.58 +0.01
23.54
19.00
26.95
Elastic transform
23.30
23.86 +0.02
23.30:10.01
+0.04
20.28 10.00
27. 00.01
Pixelate
27.61
27.55
27.62
15.84
23.47
31.23
JPEG compression
25.75
25.51
25.75
14.09
21.20
29.66
P59-C Average
23.88
23.88
23.88
13.37
19.72
27.03
Table 23: mloU comparison of MLMP and baseline methods on the P60 dataset, evaluated on both the original images and 15 corruption types.
OVSS Backbone: NACLIP
Adaptation Method
Dataset: P60
No Adapt.
TENT
TPT
WATT
CLIPAITT
MLMP
Original
24.95
25.29
24.98 10.01
14.77 10.44
21,88 10:01
27.99 20003
Gaussian noise
19.31
10.01
19.34
10.29 +0.03
14,91 10.01
22.22 =002
Shot noise
19.98
19.83 +0.02
19,91 4:001
10.82
15.82
22.81 =0001
Impulse noise
19.56
19.27
19.54
10.34
15.58
22.26 0.01
Defocus blur
22.74
22.38 #0.01
22.72
12.79
18.67
24.92 0.0.1
Glass blur
22.49
22.05
22.48 4:0.01
12.64
18.61
24.71
Motion blur
23.28
23.01
23.30
13.53
19.51 10.01
25.71
Zoom blur
17.42
17.58
17.42
14.07
19.70
Snow
20.06
19,80
20.06
12.29
17.20
22.64 0.00
Frost
19.71
19.61
19.73
11.59
17.02
22.02 0.0.1
Fog
22.20
22.09
22.20 4:001
12.50
18.53
24.89
Brightness
24.35
24.56
24.37 10.01
13.87 10.04
21.05 10.01
27.02
Contrast
21.09
21.00
21.08
12,04
17.24 10.01
24.17
Elastic transform
21.17
21.46 10.11
21.19
11,70
18.53 10.01
24.27
Pixelate
24.52
24.33
24.54
14.17
21.01
27.48
JPEG compression
22.99
22.65
2..04
12.65
19.05
26.24 10011
P60-C Average
21.39
21.25
21.49
12.08
17.79
24.07
37

---

Table 24: mIoU comparison of MLMP and baseline methods on the CityScapes dataset, evaluated on both the original images and 15 corruption types.
OVSS Backbone: NACLIP
Adaptation Method
Dataset: CityScapes
No Adapt.
TENT
TPT
WATT
MLMP
Original
29.49
30.54 +0.04
29.57 =0.01
20.77 +0.02
33.35 10.03
Gaussian noise
16.14
15.00 +0002
=0.01
=0.01
10.19 +0.01
15.32 +0000
Shot noise
20.18
19.38 40.01
20.15 =0.01
12.41 40.02
20.57
Impulse noise
16.37
14.59 +0.02
16.35 =0.01
8.56 +0.02
15.76 10005
Defocus blur
23.34
23.50 10002
23.46
15.63 +0.02
24.86 1.008
Glass blur
24.41
24.04 +0004
24.29
15,44 +0.01
25.70 10002
Motion blur
24.73
24.98 +0.00
24.91
14.21 +0.03
26.02 10001
Zoom blur
14.08
14.57 10006
14.08 =001
5.85 +0.02
14.04 20.04
Snow
19.88
20.14
19.90=0011
9.27 +0.03
22.23 +.003
Frost
16.78
16.47 10.02
16.78=001
10.69 40.01
17.12
Fog
22.47
22.87 +0.05
22.25
14.24
23.98 1.003
Brightness
28.45
29.30
28.47
19.99 +0.02
31.88 1.004
Contrast
16.10
16.35 +0.03
16.07 =0.01
11.05 +0.01
17.04 1..00
Elastic transform
29.14
30.16
29.02
19.99 +0.03
32.86 1:000
Pixelate
28.67
29.59
28.69=0.00
18.30
32.72 0.04
JPEG compression
23.69
23.62 +0.01
23.67 =0.01
15.98 +0.02
25.26 +.0.3
CityScapes- Average
21.63
21.64
21.60
13.45
23.02
Table 25: Detailed ACDC mloU (reference vs. adverse views). All experiments use NACLIP as the OVSS model. MLMP consistently improves over TENT and the non-adapted baseline across all weather conditions.
OVSS: NACLIP
Adaptation Method
Condition
No Adapt
TENT
MLMP
Fog (ref)
24.80
26.52 +0:03
31.28 +0003
Fog
23.88
26.89 10.04
33.33 1.0.04
Night (ref)
24.95
26.54 10.04
28.81 1.0.10
Night
22.12
24.17 1:0.00
24.76 1:0.03
Rain (ref)
24.79
26.62 +0000
31.10 #:0.03
Rain
23.86
26.84 1:004
32.44 1:0.04
Snow (ref)
22.10
24.27 1:0.4
28.23 1.0.02
Snow
23.54
27.25 +0005
30.59 +:0.03
Average (all ref)
24.16
25.99
29.85
Average (all domains)
23.35
26.29
30.28
38


Reviews:
[{"review_id": "6kKP8K7cYD", "reviewer": "Reviewer_t4bA", "summary": "The work proposes a novel TTA method—Multi-Level and Multi-Prompt (MLMP) entropy minimization—that adapts VLMs for segmentation using intermediate vision-encoder features and diverse text prompts at both global and pixel levels. The proposed method is plug-and-play and requires no extra data or labels. It further  introduces a comprehensive OVSS TTA benchmark with 7 datasets, 15 corruptions, and 82 test cases. Experiments show that the proposed method consistently outperforms classification-based TTA baselines in this setting.", "questions": "My main concern is \n\n1. how does the work select 7 prompts out of 80? Different selection might impact the performance.\n2. Comparison should be shown on realistic ACDC dataset.\n3. Technical novelty should be clarified.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength -\n\n1. Exhaustive ablations and comparison: The work showcases exhaustive ablation and comparison showing that the proposed method works well.  \n\n2. It proposes new benchmark - test time adaptation for Open vocabulary segmentation model.  \n\n3. The work is nicely written and well motivated.  \n\n\nWeaknesses - \n\n[A] Novelty: The technical contribution comprises of MLMP - that performs uncertainity aware entropy minimization of prediction probability across vision features, optimizing normalisation layers statistics. This has been the core idea of TENT work. Further, using multiple prompts[eg. OVSeg[1] have been used quite well for open-vocabulary segmentation. The technical novelty seems a bit limited.\n\n[1] Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP, CVPR 23\n\n[B] Comparison dataset - Although the comparison is quite exhaustive, but mostly focuses on synthetic corruptions (gaussian noise, zoom, motion blur etc) and doesn’t focus much on real world datasets. Prior test time adaptation works [2,3,4] primarily show results on adaptation from Cityscapes -> ACDC [2,3], realistic but synthetic SHIFT[2]. Would be good to add results atleast on realistic ACDC dataset.\n\n[2] Continual Test-Time Domain Adaptation, CVPR 22\n[3] Distribution-Aware Continual Test-Time Adaptation for Semantic Segmentation, ICRA 24\n\n[C] Selection of prompt: The work uses 7 prompt templates out of 80 CLIP standard prompts. However it doesnt seem straightforward to select the best performing set. How was this done? Further in these 7 prompts, there are prompts like - ‘a origami of <class_name>’, ‘a <class_name> in video-game’. How are such prompts helpful? They seem to out of domain for common corruption based images.\n\n[D] Minor: Any reason why last layer does not have higher confidence for COCOStuff dataset, as such is seen for PASCAL and Cityscapes dataset (Fig 3). Generally one would assume, last layer features to have more weightage in segmentation task as shown for PASCAL and Cityscapes datasets.", "was_revised": true, "final_justification": "Thanks for the nice work! I am recommending the final score of Broderline accept. Although the overall technical novelty appears limited agreed by other reviewers too, the task of test-time OVSS itself seems new, that might be relevant to the community. It further has detailed benchmark and analysis supporting the claims making it useful.", "rebuttal": "We thank the reviewer for their thoughtful evaluation and valuable feedback. We are pleased that they found the work to be well motivated and appreciated both the introduction of our OVSS TTA benchmark and the thorough experimental analysis. In what follows, we address the reviewer’s comments point by point.\n\n**A1. Clarifications on Novelty**\n\nWe appreciate the reviewer’s comment and welcome the opportunity to clarify our technical contributions. We agree that TENT is a well-established approach for test-time adaptation in classification, and we do not consider entropy minimization itself to be a novel contribution, nor is it the core of our method. \n\nOur main contribution lies in **test-time adaptation of open-vocabulary vision-language models**, a fundamentally different and largely underexplored setting compared to standard classification. Due to the dense, spatial nature of segmentation and the prompt-sensitive behavior of VLMs, directly applying methods like TENT, WATT, or CLIPArTT is non-trivial and often ineffective. We address these challenges by proposing a set of novel and carefully designed components: (1) A **multi-layer adaptation strategy** that integrates features from intermediate vision-encoder layers to capture complementary, shift-resilient cues (2) An **adaptive uncertainty-aware weighting strategy** that dynamically upweights more confident layers and downweights noisier ones, allowing the model to emphasize the most reliable feature representations during test time (3) An **intuitive multi-prompt aggregation mechanism at the loss level** to address increased prompt sensitivity in OVSS. Each of these components is validated through detailed ablations (see Table 1-3), and together they yield consistent, substantial gains across a wide range of scenarios. Our method is also **highly efficient**, adding only marginal overhead compared to TENT (please see Appendix E). \n\nIn addition to our design choices, we evaluate MLMP and adapted baseline methods exhaustively on a **comprehensive OVSS TTA benchmark suite** that now includes not only clean and synthetically corrupted data, but also realistic domain shifts (please see the t4bA-A3). This suite ensures fair and consistent evaluation, and we hope it will serve as a foundation for future TTA research in this domain.\n\nIn summary, we believe that the novel design of MLMP, its **plug-and-play** and **efficient formulation**, and its **strong empirical gains** across diverse and realistic settings represent a meaningful step forward in the emerging and largely underexplored field of test-time adaptation for open-vocabulary segmentation. Moreover, MLMP is **modular** and **can be seamlessly applied to a wide range of contrastive VLM-based OVSS methods**, enabling lightweight and effective adaptation without requiring additional supervision. It proves effective not only under synthetic corruptions and real-world domain shifts, but also enhances performance on clean, natural images.\n\n**A2. Differences with Multiprompt used in OVSeg**\n\nWe thank the reviewer for this helpful observation and take the opportunity to clarify key differences between our multi-prompt design and prior approaches such as OVSeg [1]. While OVSeg employs prompt ensembling, it does so during training by simply averaging text embeddings at the text encoder output, collapsing multiple linguistic features into a single representation. This results in a fixed perspective and lacks the diversity necessary for robust adaptation in a TTA setting.\n\nIn contrast, our method operates entirely at test time, and for the first time, introduces a **loss-level integration of multiple prompts during test time adaptation**. We demonstrate that prompt variability induces significantly greater uncertainty in segmentation (Fig. 1a), especially at the patch and intermediate-layer levels. To exploit this signal effectively, we directly incorporate prompt diversity into our optimization objective, treating each template as an independent “critic” and averaging their losses. Using multiple prompts in this way acts as a form of cross-modal regularization, encouraging more stable and generalized learning signals. While different from traditional image augmentation, it can be viewed as a lightweight and safe augmentation technique in the text modality. This design is further supported by Proposition 1, which shows that the resulting gradient is both unbiased and lower-variance, thereby promoting more stable and generalizable adaptation.\n\nWe also compare our approach in Table 3 to alternatives, including simple prompt averaging (as in OVSeg [1]) and parameter ensembling (as in WATT [2]), and show that our strategy outperforms both in adaptation effectiveness. While WATT incurs notable computational overhead, our method remains lightweight—even when using multiple templates—making it well-suited for dense prediction tasks such as OVSS.\n\n[1] Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP, CVPR 23\n\n[2] WATT: Weight Average Test-Time Adaptation of CLIP, NeurIPS 24\n\n**A3. Including Real Domain Shift**\n\nWe appreciate the reviewer’s suggestion to include evaluation on realistic, non-synthetic domain shifts. In response, we have conducted additional experiments on the ACDC [3] dataset, which contains real-world variations such as fog, night, rain, and snow. As in our main experiments, we perform test-time adaptation separately on both reference (clean) and shifted images within each domain. It is worth mentioning that in ACDC, half of the adverse-condition images have corresponding clean reference views. While these reference images are captured at approximately the same location as the shifted ones, the scene content may differ (e.g., different vehicles or objects present) due to real-world variability.\n\nAs shown in the table below, our method yields consistent improvements over both baselines across all domains. Compared to the non-adapted model, MLMP achieves substantial gains, around 6% mIoU improvement on reference images and over 7% mIoU in the presence of environmental domain shift. These results highlight **the robustness of our method in handling real-world domain shifts, demonstrating effectiveness beyond synthetic corruption scenarios**. We also note that the Cityscapes dataset, on which we already reported results in the main paper, exhibits natural distributional shifts due to variations in lighting, camera viewpoints, and urban layouts across different cities.\n| Condition| No Adapt.| TENT| **MLMP**|\n|-|-|-|-|\n| Fog (ref)| 24.80 ± 0.00| 26.52 ± 0.03 | **31.28 ± 0.03** |\n| Fog| 23.88 ± 0.00| 26.89 ± 0.04 | **33.33 ± 0.04** |\n| Night (ref) | 24.95 ± 0.00| 26.54 ± 0.04 | **28.81 ± 0.10** |\n| Night| 22.12 ± 0.00| 24.17 ± 0.00 | **24.76 ± 0.03** |\n| Rain (ref)| 24.79 ± 0.00| 26.62 ± 0.00 | **31.10 ± 0.03** |\n| Rain | 23.86 ± 0.00| 26.84 ± 0.04 | **32.44 ± 0.04** |\n| Snow (ref)| 22.10 ± 0.00| 24.27 ± 0.04 | **28.22 ± 0.02** |\n| Snow | 23.54 ± 0.00| 27.25 ± 0.05 | **30.59 ± 0.03** |\n| **Avg (all ref)** | 24.16 | 25.99| **29.85**|\n| **Avg (all domains)**| 23.35 | 26.29| **30.28**|\n\nTo further address the reviewer’s concern, we also evaluated MLMP on a rendered domain shift, using the GTAV dataset [4]. GTAV consists of photorealistic urban scenes rendered from a game engine and exhibits a distribution shift distinct from real-world imagery seen during CLIP pretraining. As shown in the table below, MLMP achieves consistent improvements over both TENT and the non-adapted baseline. This complements our ACDC results and demonstrates **MLMP’s robustness across both real-world and rendered test-time domains**.\n|Method|NACLiP|TENT|MLMP|\n|-|-|-|-|\n|mIoU (%)|25.09|26.62 ± 0.01| **28.84 ± 0.02**|\n\n[3] ACDC: The Adverse Conditions Dataset, ICCV 21\n\n[4] Playing for Data: Ground Truth from Computer Games, ECCV 16\n\n\n**A4. Selection of Prompts**\n\nWe thank the reviewer for this important question. As discussed in Appendix C, the seven prompt templates used in our method were selected directly from the original CLIP repository, without any tuning or dataset-specific filtering. These prompts are **general-purpose** and were originally proposed to offer **diverse textual views** across a wide range of classes—not tailored to specific datasets or domains. We intentionally adopted this fixed, predefined set to avoid any risk of target-data information leakage\n\nWhile we agree that prompt selection can influence performance, selecting or filtering prompts based on the target domain shift would violate the fundamental assumptions of test-time adaptation, where no access to shift-specific information is allowed. In fact, using domain-aligned prompts (e.g., referencing weather conditions) could introduce bias and unfairly benefit adaptation. Our goal was to evaluate MLMP under a fair and domain-agnostic setting.\n\n**A5. Insights into Layer-Wise Confidence Trends in COCO-Stuff**\n\nWe thank the reviewer for this insightful observation. Indeed, as shown in Fig. 3, the final vision layer receives lower confidence in the COCOStuff dataset compared to Pascal and Cityscapes. We believe this is due to the inherent properties of COCOStuff, which includes 171 classes, many of which are amorphous “stuff” categories (e.g., sky, grass, wall) with weak spatial boundaries and higher semantic ambiguity. In such cases, earlier and intermediate layers—preserving finer texture and edge information—often yield more reliable predictions than the deeper, class-aligned representations of the final layer. A similar trend is observed in the COCOObject dataset as well (please see Fig. 1 in the Appendix H).\n\nThis observation further supports the necessity of our proposed **uncertainty-aware multi-level fusion strategy**, which dynamically reweights intermediate features based on their entropy. Unlike fixed fusion schemes, our approach adapts to the specific input and dataset, allowing the model to emphasize more reliable layers under varying conditions."}, {"review_id": "ZJxIVHVnEQ", "reviewer": "Reviewer_rgyA", "summary": "The paper introduces MLMP (Multi-Level Multi-Prompt), a test-time adaptation (TTA) framework for open-vocabulary semantic segmentation (OVSS). MLMP focuses on dense prediction tasks by:\n- Integrating features from intermediate vision-encoder layers (e.g., ViT layers) to capture complementary, shift-resilient cues, with uncertainty-aware weighting based on entropy.\n- Minimizing entropy across multiple text-prompt templates at both global and local levels to leverage prompt sensitivity as a robust adaptation signal.\nThe authors also establish an OVSS TTA benchmark to demonstrate its effectiveness in adapting to domain shifts during inference.", "questions": "see weakness", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- The writing and architecture figure is clear and easy to follow.\n- The observation revealed in Fig 1.(a) is interesting. And the ablation accordingly (Fig 3) is vivid.\n\nWeakness:\n- Weak Motivation:\nI question the necessity of the OVSS TTA task. Most domain shifts in the authors' benchmark (e.g., blur, noise) are artificially created and lack practical significance. Such image shifts can be easily resolved through training-time data augmentation or post-processing. Modern segmentation frameworks no longer emphasize these augmentations because these shifts are already considered solved problems. Thus, re-investigating adaptation for them is of limited value. The authors should identify more compelling TTA tasks or scenarios.\n\n- Missing Comparison:\nBeyond zero-shot (ovss are considered as zss), the authors should compare with classical few-shot segmentation (fss) methods. These solutions also address domain shifts by adapting with a few samples without requiring training. The lack of comparison and discussion with such methods undermines the justification for OVSS TTA's necessity and validity.\n\n\n- Overclaim of generalization ability:\nThe claim that MLMP applies to \"any OVSS framework\" is incorrect. It clearly only works for pixel-to-text similarity matching frameworks, not query-based ones (e.g., SAM, SEEM) that use decoders for segmentation. Additionally, validation is limited to NACLIP, no other CLIP-based or non-CLIP frameworks are tested or mentioned. The \"any OVSS framework\" claim is obviously an overstatement.", "was_revised": true, "final_justification": "The author's response solves my concern, and the supplemented table makes the paper more readable. Thus I raise my score to 4: Borderline accept.", "rebuttal": "We thank the reviewer for their time and feedback. We are glad that they found the writing clear and appreciated the observations motivating our MLMP design. In the following, we address each of the reviewer’s comments in detail.\n\n**A1. General Motivation behind TTA for OVSS**\n\nWe appreciate the reviewer’s concern and would like to clarify the motivation behind the OVSS-TTA task. The main goal is to evaluate whether test-time adaptation can offer consistent improvements across a broad spectrum of potential shifts. As also noted by Reviewer t4bA, we believe the task is well motivated, and we agree with Reviewer tiCb on its high practical relevance, particularly given the unique challenges of zero-shot OVSS, where models are especially sensitive to distribution shifts due to the lack of supervised adaptation. This sensitivity is evident in Table 4 of the main paper, where the non-adapted model's performance drops from 75.91 mIoU on V20 (original) to 69.01 mIoU on V20-C (common corruptions), underscoring that even robust foundation models remain vulnerable to distribution shifts in the absence of test-time adaptation.\n\nWhile some of our benchmark shifts are synthetic (e.g., blur, noise), this follows standard practice for robustness evaluation, such as in ImageNet-C, and provides a controlled, reproducible setting to test the effectiveness of adaptation strategies. This is especially valuable in zero-shot scenarios, where the test distribution is inherently unknown. \n\n\nTo demonstrate that our approach goes beyond synthetic corruptions, we include results on more realistic domain shifts. In our response to t4bA-A3, we present evaluations on ACDC, which features real-world weather-related conditions, and GTA5, a distinct synthetic domain with different scene statistics. We also highlight intra-dataset variations within Cityscapes (e.g., lighting, camera viewpoints, and location diversity). Furthermore, our method improves performance not only under distribution shift but also on the original (unshifted) datasets (Table 4), which underscores the general utility of test-time adaptation in OVSS.\n\nIn real-world settings, it is unlikely that a model can fully account for all possible unseen target distributions. This is due to several factors: (1) the space of potential corruptions and environmental shifts is vast and essentially unbounded; (2) training on such a wide range of conditions would require extensive labeled and diverse data, which is often costly or impractical to obtain; and (3) the distribution encountered at deployment is typically unknown and may evolve over time, especially in open-world scenarios like OVSS.\n\nThese constraints make TTA not only relevant but necessary, as it allows models to adjust to novel domains without requiring retraining or labeled data.\n\n**A2. Comparison with Few-Shot**\n\nThank you for raising this point. While FSS methods do allow for adaptation using a small number of labeled examples, we believe they are fundamentally different from the TTA setting we explore in OVSS.\n\nFirst, most FSS methods assume access to a clean support set with annotated masks at inference time. In contrast, OVSS operates in a fully zero-shot setting, without any class-specific labels or examples at test time. The set of target categories is also unknown and potentially large, making it impossible to construct or assume a meaningful support-query structure.\n\nMoreover, while FSS methods do not require training-time fine-tuning, they still rely on labeled masks at inference, which can be difficult to obtain for open-vocabulary segmentation tasks, especially when dealing with arbitrary or long-tail classes.\n\nFor these reasons, we believe that FSS and TTA operate under different assumptions and are not directly comparable. However, we appreciate the suggestion and will include a discussion of this distinction in the revised version of the paper.\n\n**A3. Clarification on the Generality of MLMP Across OVSS Frameworks**\n\nWe would like to clarify that we do evaluate MLMP across multiple similarity-based OVSS frameworks to assess its generalization ability, as mentioned in lines 282–283 of the paper. Specifically, we test MLMP with (1) different OVSS methods (Table 6 in Appendix), and (2) different vision-language models, namely SigLIP-2 (Table 7 in Appendix) or LongCLIP (please see R-Fuid-A9) and (3) different backbones (Tables 4 and 5 in Appendix). These experiments demonstrate that MLMP is robust across architectures, feature extractors, and vision-language model variants within this class of methods.\n\nAs stated in line 117, our method is specifically tailored for the OVSS framework based on contrastive VLMs, which covers most existing approaches in open-vocabulary semantic segmentation. We will ensure this point is stated more clearly in the revised manuscript."}, {"review_id": "AAPKO2oNCR", "reviewer": "Reviewer_tiCb", "summary": "This paper proposes a novel test-time adaptation (TTA) method, MLMP, for adapting vision-language models (VLMs) to segmentation tasks during inference. MLMP performs multi-level and multi-prompt (MLMP) entropy minimization by integrating features from intermediate layers of the vision encoder and optimizing both global CLS tokens and local pixel-level representations using different textual prompt templates. MLMP is a plug-and-play module that can be applied to any segmentation network without requiring additional training data or labels, and it remains effective even when only a single test sample is available. A comprehensive benchmark suite for open-vocabulary semantic segmentation (OVSS) TTA is constructed, covering seven segmentation datasets, 15 common corruptions, and 82 diverse testing scenarios, providing a standardized and thorough platform for future OVSS TTA research.", "questions": "Please answer the questions in Weakness and refute the weaknesses I pointed out about Feature Fusion and Prompt Templates.", "limitations": "The authors discussed their limitations of their work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:  \n1、The focus on test-time adaptation for VLMs is of high practical relevance.  \n2、Theoretical analysis demonstrates that using multiple prompt templates helps reduce the variance of the loss gradient, laying a foundation for future research.  \n3、The experiments are extensive and thorough, supporting the proposed method from multiple perspectives.  \n4、The proposed method achieves significant performance improvements across multiple datasets.  \n\n**Weaknesses**:\n\n1、Feature fusion across layers and the use of image-level entropy as a loss are relatively common techniques, making this part of the method less innovative.  \n2、In the Experimental Setup section, it is stated that the adaptation process is conducted over 10 iterations with resets. However, as far as I know, TTA is typically evaluated under a one-epoch protocol to test effectiveness. How does the proposed method perform under that setting?  \n3、How exactly does the method support open-vocabulary settings? Line 120 mentions that a set of concepts C_k in C must be provided during inference. Does this imply that the model can only segment categories within the predefined set C? It is suggested to provide more detailed clarification to highlight the method's capability in handling open-vocabulary segmentation.  \n4、The method appears sensitive to the choice of prompt templates. Although multi-prompt optimization helps mitigate this issue to some extent, selecting more effective prompt templates to further improve performance remains an open challenge that warrants deeper investigation.", "was_revised": true, "final_justification": "Thank you for the author's response. I maintain my original rating.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We are pleased that they acknowledged the practical significance of adapting vision-language models during inference, appreciated our theoretical insights on prompt-based variance reduction, and recognized the strength of our experimental results. In the following, we address the reviewer’s comments point by point.\n\n\n**A1. Novelty of Feature Fusion and Image-Level Loss**\n\nThank you for the comment. While feature fusion across layers is indeed common in standard segmentation models (e.g., U-Net, HRNet, FPN), its use in TTA, especially in the context of OVSS, has not been explored to our knowledge. In our method, we go beyond basic fusion by introducing adaptive layer weighting, which allows the model to selectively emphasize features based on their relevance during adaptation. This is a key contribution that distinguishes our approach.\n\nAs for the image-level entropy loss, we note that this is not a common technique in segmentation, and especially rare in OVSS. Unlike pixel-level entropy minimization used in methods like TENT, we apply entropy regularization at the image-level representation (CLS token), encouraging more confident global predictions in a zero-shot setting. To our knowledge, this is the first use of such a loss in OVSS TTA.\n\nWe believe the overall design brings a novel perspective to this recent problem, a view that is also echoed in the feedback from R-Fuid.\n\n**A2. Evaluation Under One-Iterate TTA Protocol**\n\nThe number of adaptation iterations in TTA varies across the literature. While some works evaluate under a one-iteration (step) protocol, many others use multiple iterations. For example, the original TTT paper (Sun et al., 2020) as well as more recent vision-language TTA methods like CLIPArTT and WATT commonly use 10 adaptation iterations. Our choice to follow this convention is aligned with these prior works. For fairness, we use the same hyperparameters across all experiments and baseline methods.\n\nNonetheless, we agree it is valuable to assess performance under a single-iteration protocol. As shown in the table below, MLMP still improves over the baseline and outperforms TENT even with just one iteration, which demonstrates the effectiveness of our method even under a stricter setting.\n|Dataset|No Adapt.|TENT|**MLMP**|\n|-|-|-|-|\n|V20|75.91|76.80±0.01|**79.49±0.01**|\n|V21|45.12|45.36±0.00|**49.97±0.00**|\n|V20-C|69.01|70.15|**73.38**|\n|V21-C|40.75|41.02|**45.54**|\n\nWe will clarify this setup in the main paper and include the single-iteration results to make the evaluation more comprehensive.\n\n**A3. Open-Vocabulary Capability**\n\nWe thank the reviewer for this insightful question. The set of concepts C_k ⊂ C mentioned in Line 120 is provided at inference time and can be arbitrary—there is no restriction to a fixed or predefined vocabulary. This flexibility is what enables open-vocabulary segmentation: the model is not trained on specific categories and can adapt to any set of text labels at test time.\n\nIn our experiments, we follow the evaluation protocol used in prior OVSS works (e.g., NACLIP [1] ), which adopt a predefined set C purely for benchmarking purposes. However, this set can be changed at any time, and the model is capable of segmenting new, unseen categories that were not present during training.\n\nWe will clarify this aspect in the paper to better emphasize that our method fully supports open-vocabulary segmentation by design.\n\n[1] Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation, WACV 2025\n\n**A4. Selection of Prompts**\n\nThank you for the observation. We agree that prompt selection is an important aspect in vision-language models. As noted in Appendix Section C, we did not tune prompt templates for performance; instead, we used the general-purpose templates proposed by the original CLIP authors. These prompts are not tailored to specific image content, which helps ensure fairness and broad applicability.\n\nWhile our multi-prompt strategy helps mitigate sensitivity to any single template, we agree that exploring more effective or adaptive prompt selection strategies could further improve performance. However, such approaches often come with additional computational and memory costs, and we consider them complementary to our method. We view this as a promising direction for future work."}, {"review_id": "PG7EQVT37d", "reviewer": "Reviewer_Fuid", "summary": "The paper introduces a novel method for test-time adaptation of contrastive VLMs for open-vocabulary semantic segmentation. The method can be summarized in two steps:\n\n1. Update/adaptation: the authors tune the layer norm modules by minimizing the entropy of the distribution resulting from a (softmaxed) dot-product between prompt embeddings and image features. For this, they use (a) features from the last 75% of ViT blocks, averaged together, and (b) 7 prompt templates. They then calculate loss terms for each template and average them.\n2. Inference: the authors predict the segmentation map using the same formulation as CLIP zero-shot for image classification but on a spatial token basis. They use the same seven prompts and features as in the adaptation step, but they re-weight the visual features based on their batch-wise entropy. \n\nThe authors demonstrate that their method significantly improves performance compared to the non-adapted baseline on seven datasets, both in its \"clean\" version and when corrupted using the same image corruptions as those in ImageNet-C.\n\nTheir method also performs significantly better than other test-time adaptation strategies originally conceived for image classification but adapted to open-vocabulary semantic segmentation.", "questions": "1. The adaptation/inference loop is not fully clear to me. My understanding is that: you pick two test samples, perform 10 steps of adaptation, predict the masks of both samples, and then restore the model. Is this correct? If yes, why was a batch size of two chosen? Intuitively, I would expect that adapting for a single sample would work better.\n\t1. Would a larger (or smaller) batch size lead to better results?\n\t2. It could be interesting to explore an online version of the method, i.e., rather than discarding the updates, use them as a starting point for the next batch or sample. Would the performance degrade or improve significantly, as in [1]?\n2. WATT and CLIPArTT perform significantly worse than a non-adapted baseline. Do you have any insights on why this happens?\n3. Since the multi-prompt loss alone improves performance by a good margin, it could be interesting to see if further improvements can be achieved with longer prompts using either Long-CLIP [2] or TULIP [3].\n\n---\n\n- [1] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts, Yu Sun et al., ICML 2020.\n- [2] Long-CLIP: Unlocking the Long-Text Capability of CLIP, Beichen Zhang et al., ECCV 2024\n- [3] TULIP: Token-length Upgraded Clip, Ivona Najdenkoska, Mohammad Mahdi Derakhshani et al., ICLR 2025", "limitations": "The authors appropriately discussed the method's limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The paper is reasonably well-written and easy to follow. The method is novel and tackles a previously understudied area.\n2. The method significantly outperforms non-adapted backbones and existing baselines adapted for open-vocabulary semantic segmentation. These improvements are validated across various datasets, corruptions, model sizes, and model families, the latter two explored in the supplementary materials.\n3. The methodological choices are explained through extensive ablations.\n\n---\n\n### Weaknesses\n\n1. Some parts of the paper could be further polished and are a bit unclear.\n\t1. In Figure 1, I would recommend adding an x-axis label for the figure on the left and specifying in the caption what backbone is being used and what V20 means (it's not immediately clear that this refers to Pascal VOC). Furthermore, it's not clear how the variance in the left plot was computed. Is it computed over a single example? Or multiple?\n2. The paper claims that using multiple prompts prevents degenerate collapse and ensures that the model yields segmentations that are consistent across diverse linguistic formulations but never shows a case of degenerate collapse with a single prompt. I am also skeptical of the second claim, as to my understanding, the authors always use the same set of prompts for both adaptation and inference across the paper's experiments.\n3. In Figure 4, it could also be interesting to see the performance with no adaptation to see how much a single prompt improves over the baseline.\n4. On line 212, you mention the GTAV dataset but do not cite it.\n5. On line 155, the paper states that the hypothesis that layers contain complementary information is validated in Figure 1b. However, this figure only compares the overall performance of MLMP to a non-adapted baseline and does not provide insights into this specific claim.", "was_revised": true, "final_justification": "The authors addressed all my concerns and provided new and insightful experiments. I am thus increasing my score to 5.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We are encouraged that they found our method novel, well-motivated, and clearly presented, and that they appreciated the strong empirical results across diverse datasets, corruption types, and model families. We also thank the reviewer for highlighting the value of our extensive ablations and recognizing that our work addresses an understudied area. Below, we respond to each of the reviewer’s comments in detail.\n\n**A1. Improving Figure 1**\n\nThe variance is computed over all samples in the V20-C benchmark, which includes all corruption types. We appreciate the reviewer’s feedback and will incorporate all these clarifications to improve the clarity and completeness of the figure.\n\n**A2. Clarifications on Multi-Prompt Robustness Claims**\n\nTo illustrate how multiple prompts help prevent degenerate collapse, we conducted an experiment where we ran adaptation using either all seven templates or only a single one and averaged them, using the same 10 images for efficiency. The results are summarized in the table below.\nTemplate|No Adapt.|50 Steps|100 Steps|200 Steps|\n|-|-|-|-|-|\n|All Templates|38.20|65.42|64.75|65.74|\n|Average of single template|37.63|59.79|58.60|56.88|\n\nOn V20-C, single-template runs often show unstable behavior or steep drops as we can see thanks to the average, in contrast, the multi-template setup maintains high performance, with only a slight decline. These results support our claim that using multiple prompts acts as a regularizer and mitigates collapse during adaptation.\n\n**A3. Improving Figure 4**\n\nWe will update this figure in the final version by adding the performance with no adaptation (the quantitative results are already in the main table).\n\n**A4. Missing Citation for GTAV**\n\nWe will add the appropriate citation for the GTAV dataset in the final version.\n\n**A5. Typo in Line 155**\n\nThank you for pointing this out. The reference to Fig. 1b in that sentence is a typo. The intended evidence for the complementary nature of intermediate layers is in Table 1, which reports the ablation on fusion strategies, and in Figure 3, which visualizes the learned layer-wise confidence weights. We will correct the sentence and update the citation accordingly in the final version. \n\n**A6. Clarification on Adaptation Loop and the Effect of Batch Size**\n\nThe reviewer’s understanding of the adaptation/inference loop is correct: we adapt the model using a batch of two test samples for 10 steps, predict the masks for both samples, and then restore the original model parameters.\n\nWe chose a batch size of 2 mainly for computational efficiency; it provides roughly 2× faster adaptation compared to using a batch size of 1, which was important given the need to evaluate multiple adaptation methods across several datasets.\n\nWhile larger batch sizes can be beneficial in classification due to entropy bias correction, this is less relevant in segmentation, where supervision occurs at the pixel/token level. In fact, increasing the batch size may require additional tuning of hyperparameters and can hurt performance due to over-smoothing or weaker adaptation signals.\n\nTo address your intuition that a batch size of 1 may work better, we confirm this in Appendix Table 3. As shown below, our method performs well with a batch size of 1, supporting the robustness of the approach across batch configurations:\n|Dataset|No Adapt.|TENT (bs=1)|TENT (bs=2)|MLMP (bs=1)|MLMP (bs=2)|\n|-|-|-|-|-|-|\n|V20|75.91|76.20 ± 0.00|77.00 ± 0.04|84.68 ± 0.00|83.76 ± 0.00|\n|V20-C|69.01|68.50|69.33|78.41|77.58|\n\nWe will update the paper to make this clearer and explicitly reference the ablation results to improve transparency.\n\n**A7. Episodic vs Online Setting**\n\nWe thank the reviewer for the insightful suggestion regarding an online adaptation setting. We experimented with this setting using the exact same hyperparameters as our main method (i.e., learning rate and adaptation steps), but found that both our method and TENT eventually collapsed: while performance initially improved, it degraded after a few batches and dropped significantly thereafter.\n\nMotivated by the reviewer’s reference to [1], we further explored the online setting by reducing the number of adaptation steps and lowering the learning rate. The results are as follows:\n|Dataset|No Adapt.|Episodic: TENT|Episodic: MLMP|Online: TENT|Online: MLMP|\n|-|-|-|-|-|-|\n|V20|75.91 ± 0.00|77.00 ± 0.04|83.76 ± 0.00|76.44 ± 0.21|79.19 ± 0.45|\n|V20-C|69.01|69.33|77.58|69.32|71.77|\n\nWe observe that while the online variant improves over baselines, it still underperforms compared to the reset-based (episodic) setting. We believe this is primarily due to error accumulation and model drift, as documented in prior work showing that some recent classification TTA methods exhibit performance degradation in an online setting—even when the distribution is static [2]. This issue is particularly pronounced in semantic segmentation, where dense, spatial predictions make the model more sensitive to incorrect updates, and entropy minimization can amplify early misclassifications. Over time, this leads the model to drift away from its well-initialized source parameters, reducing its ability to generalize across the target domain.\n\nEpisodic (reset) adaptation avoids the runaway effects of carrying over errors, which is why it tends to be safer [2]. In contrast, fully online adaptation—though attractive for its potential to continuously refine the model—must confront these challenges.\n\nWe believe that online adaptation in segmentation can be improved through several future directions: (1) introducing occasional resets or using exponential moving average (EMA) of model weights to limit drift and reduce accumulated errors; (2) filtering high-entropy samples or gradients, which is particularly important in segmentation due to its spatial granularity and sensitivity to local noise; and (3) incorporating an auxiliary self-supervised objective, such as rotation prediction [1], to provide a more reliable adaptation signal—though this may come with increased computational cost. Notably, the fact that our method works effectively with very few test samples and does not rely on accumulating state makes it efficient for real-world applications.\n\nWe will include a discussion of these findings and potential improvements in the final version. \n\n[1] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts, ICML 2020. \n\n[2] On Pitfalls of Test-time Adaptation, ICLR 2023\n\n**A8. Insights on the Underperformance of WATT and CLIPArTT**\n\nThank you for the observation. Indeed, as shown in the table below, our own variant of MLMP using a pseudo-labeling loss performs worse than the non-adapted baseline, consistent with the behavior observed for WATT and CLIPArTT. We also show in Table 3 of the paper that parameter averaging, as done in WATT, does not improve performance in our setup either.\n\nWe believe this highlights a key difference between segmentation and classification tasks. In segmentation, pseudo-labeling relies on confidently predicting a hard label for every pixel, which can amplify noise and lead to unstable adaptation, especially under distribution shift. In contrast, entropy minimization provides a smoother learning signal by encouraging confident predictions without committing to hard labels, making it more effective in this context.\n|Dataset|No Adapt.|MLMP|MLMP PseudoLabel|\n|-|-|-|-|\n|V20|75.91|83.76 ± 0.00|70.06 ± 0.18|\n|V21|45.12|50.78 ± 0.02|43.78 ± 0.11|\n|V20-C|69.01|77.58|56.31|\n|V21-C|40.75|46.25|35.74|\n\nWe will clarify these insights in the paper to better explain why some adaptation strategies from classification may not transfer well to segmentation.\n\n**A9. Experience with Longer Prompts**\n\nThank you for the suggestion. We agree that exploring longer prompts, such as those enabled by models like Long-CLIP or TULIP, can be an interesting direction. To investigate this, we generated extended templates based on our 7 class-agnostic prompts using ChatGPT, as well as extended class names, where each class was described in natural language. For example, \"a photo of the large {class}.\" becomes \"A photograph of a very large {}, where the size of the object dominates the frame or is shown in contrast to smaller elements to emphasize its scale.\" and “aeroplane” becomes “aeroplane: a powered flying vehicle with fixed wings and engines, designed to transport people or cargo through the air over long distances”\n\nAs shown in the table below, MLMP consistently improves performance over the Long-CLIP baseline across all variants. However, using extended templates or extended class descriptions did not lead to further improvements over the base prompts. This suggests that while MLMP is robust and effective even with longer inputs, simply extending prompt length does not guarantee performance gains, possibly due to the complexity or ambiguity introduced by verbose descriptions.\n|Dataset|LongCLIP|LongCLIP + MLMP|Extended Templates|Extended Templates + MLMP|Extended Classes|Extended Classes + MLMP|\n|-|-|-|-|-|-|-|\n|V20|39.64|54.45 ± 0.09|35.55|53.43 ± 0.06|25.66|43.92 ± 0.04|\n|V21|16.83|19.54 ± 0.02|15.87|19.15 ± 0.04|11.27|15.25 ± 0.03|\n|V20-C|29.36|48.71|26.92|48.90|21.09|39.01|\n|V21-C|13.64|18.57|12.98|18.27|10.23|14.66|\n\nWe will add this analysis and discussion to the paper to provide a clearer picture of MLMP’s behavior under extended prompts."}]

Author rebuttal:

Meta-review: Paper was reviewed by four expert reviewers and received al positive scores of 3 x Borderline Accept and 1 x Accept. Initial reviews had a number of concerns that focused on (1) novelty, (2) lacking motivation, (3) lack of details (e.g., on how prompts are selected), (4) lack of comparisons to traditional methods, and (5) lack of clarity in certain places. However, rebuttal has largely addressed these concerns and reviewers appear to be satisfied with the provided responses. 

AC has carefully considered the reviews, rebuttal, discussion and the paper itself. AC agrees with [t4bA] that overall technical novelty of the approach is limited. At the same time, the simplicity of the approach and the comprehensive benchmarking and evaluation are commendable. On balance AC is in agreement with reviewers and is recommending Acceptance.

Authors are strongly encouraged to incorporate content from the rebuttal, especially clarification and additional experiments, into the main paper (and supplementals where appropriate). This will strengthen the paper.
Decision: Accept (poster)