Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptation
Authors: Zhixiang Chi
Li Gu
Huan Liu
Ziqiang Wang
Yanan Wu
Yang Wang
Konstantinos N Plataniotis
Venue: ICLR 2025 Poster (2025)
Keywords: Distribution shifts
Visual prompt
Foundation model
Few-Shot Test-Time Domain Adaptation
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose to learn data-specific knowledge from the input space to complement frozen CLIP, and adapt CLIP towards domain shift.

Abstract: Few-shot Test-Time Domain Adaptation focuses on adapting a model at test time to a specific domain using only a few unlabeled examples, addressing domain shift. Prior methods leverage CLIP's strong out-of-distribution (OOD) abilities by generating domain-specific prompts to guide its generalized, frozen features. However, since downstream datasets are not explicitly seen by CLIP, solely depending on the feature space knowledge is constrained by CLIP's prior knowledge. Notably, when using a less robust backbone like ViT-B/16, performance significantly drops on challenging real-world benchmarks. Departing from the state-of-the-art of inheriting the intrinsic OOD capability of CLIP, this work introduces learning directly on the input space to complement the dataset-specific knowledge for frozen CLIP. Specifically, an independent side branch is attached in parallel with CLIP and enforced to learn exclusive knowledge via revert attention.  To better capture the dataset-specific label semantics for downstream adaptation, we propose to enhance the inter-dispersion among text features via greedy text ensemble and refinement. The text and visual features are then progressively fused in a domain-aware manner by a generated domain prompt to adapt toward a specific domain. Extensive experiments show our method's superiority on 5 large-scale benchmarks (WILDS and DomainNet), notably improving over smaller networks like ViT-B/16 with gains of \textbf{+5.1} in F1 for iWildCam and \textbf{+3.1\%} in WC Acc for FMoW. \href{https://github.com/chi-chi-zx/L2C}{Our Code: L2C}

Published as a conference paper at ICLR 2025
LEARNING TO ADAPT FROZEN CLIP FOR FEW-SHOT TEST-TIME DOMAIN ADAPTATION
Zhixiang Chi', Li Gu", Huan Liu', Ziqiang Wang', Yanan Wu?t,
Yang Wanga,, Konstantinos N Plataniotis
University of Toronto, 2 Concordia University, 3 McMaster University zhixiang. chi 1a11mmal... www.ll.uoooono.c..
ABSTRACT
Few-shot Test-Time Domain Adaptation focuses on adapting a model at test time to a specific domain using only a few unlabeled examples, addressing domain shift. Prior methods leverage CLIP's strong out-of-distribution (OOD) abilities by gener- ating domain-specific prompts to guide its generalized, frozen features. However, since downstream datasets are not explicitly seen by CLIP, solely depending on the feature space knowledge is constrained by CLIP's prior knowledge. Notably, when using a less robust backbone like ViT-B/16, performance significantly drops on challenging real-world benchmarks. Departing from the state-of of-the-an art of inherit- ing the intrinsic OOD capability of CLIP, this work introduces learning directly the input space complement the dataset-specifie knowledge for frozen CLIP. Specifically, an independent side branch is attached in parallel with CLIP and enforced learn exclusive knowledge via revert attention. To better capture the dataset- -sausstt-peeiiccceeee label semantics for downstream adaptation, we propose to enhance the inter-dispersion among text features via greedy text ensemble and refinement. The text and visual features are then progressively fused in domain- aware manner by generated domain prompt to adapt toward specific domain. Extensive experi- ments show our method's 's superiority on 5 large-scale benchmarks (WILDS and DomainNet), notably improving over smaller networks like ViT-B/16 with gains +5.1 in Fl for iWildCam and +3.1 % in WC Ace for FMoW. Our Code: L2C
INTRODUCTION
Deep models excel when test and training data distributions align, but real-world scenarios often involve domain shifts (Gulrajani & Lopez-Paz, 2020; Taori et al., 2020; Su et al., 2024b; Yang et al., 2024a; 2023), leading to performance drop. Few-shot Test-Time Domain Adaptation (FSTT-DA) (Chi et al., 2024; Zhong et al., 2022) addresses this by introducing a test-time learning phase to adapt generic models unseen target domains using few unlabeled samples. It faces several challenges: i) limited domain-speciftt information due to a few unlabeled data (unseen target domains), ii) one-time adaptation for each target domain, iii) strict source- free environment during test-time on unseen target domains, and iv) handling diverse target domains with varying complexities and domain shifts.
Therefore, developing an adaptive learning system using source domain data crucial (Ahmed et al., 2021), 1), as it embodies dataset-specifii chowledge-includee MetaDMoFF (Zhong et al., 2022) adapts to unseen target domains by querying relevant knowledge from source expert models and then updating an adaptive student model through knowledge distillation (Ye et al., 2024; Yang & Ye, 2024; Hamidi et al., 2024). MABN (Wu et al., 2024b) learns source distributions during offline training and pinpoints domain speciffc parameters for updates during test-time. However, both MetaDMoEE and MABN involve model fine-tuning, which can compromise the inherent OOD generalization of vision foundation models like CLIP (Wortsman et al., 2022b).
labels, semantics, and domains.
VDPG (Chi et al., 2024) leverages CLIP's inherent OOD capabilities and robustness (Zhang et al., 2023) by operating solely its visual features. It compacts source domain knowledge into learnable knowledge bank. A generator then creates domain-specific prompts from this bank, conditioned on
Corresponding authors.

---

Published as a conference paper at ICLR 2025
WildCamm
FMoW
VOPG (ICLR24)
L2C (ours)
Bisne soofff
46.7
+5.33
13:1
choddy
Cemudiin Cuif
23-2
433
Feage muade:
VIT-5225
VITU214
VIR-1/66
(b) Comparison on ViT-B/16 and ViT-L/14. Figure I: VDPG preserves CLIP's OOD capabilities by operating in the feature space and relying on pretrained CLIP. However, with weaker backbone like ViT-B/16, performance drops on benchmarks like iWildCam and FMoW. Learning directly from the image space complements CLIP, improving ViT-B/16 shown in (b).
(a) High-level comparison.
the features of unlabeled data, steer the frozen CLIP features toward the target domain. While effective in generating diverse prompts across domains, VDPG has notable drawbacks: its ability to produce domain-specifie prompts and utilize source knowledge is limited by CLIP's general, non-dataset-specifee knowledge. As shown in Fig. 1b, with a much less robust backbone, ViT-B/16, VDPG suffers significant performance drop. Moreover, VDPG overlooks class semantic cues (Yoon et al., 2024), which are vital for downstream datasets but not addressed by its vision-only encoder.
In this work, we adopt VDPG's black-box approach retain CLIP's OOD capabilities and aim to improve it by adapting both frozen image and text features. On the image side, we attach a parallel module named CPNet learn directly from the input space and ComPlement the frozen visual feature of CLIP at its output. It contradicts the methods that involve feature interactions among intermediate layers (Yin et al., 2023; Xu al., 2023; Chi et al., 2018), and allows CPNet to remain lightweight and applicable for black-box settings. CPNet is encouraged via revert attention to focus on learning only the necessary dataset-specific information, both semantic and domain, that may be absent from CLIP's generalized knowledge. Fig. la demonstrates high-level comparison with VDPG.
Noting the significant diversity in image features across domains, even for the same group of classes, deduce that text features must also adapt accordingly. To benefit downstream adaptation, aim to enhance the discrimination among text features (termed inter-dispersion) (Cho et al., 2023a), reducing domain bias prior to adaptation. We propose greedy text ensemble strategy select prompt templates that improve this discrimination, combined with a lightweight refinement module that uses an inter-dispersion loss to further enhance class differentiation. Importantly, because the greedy ensemble is executed as a pre-processing step, the CLIP text encoder can be discarded when training starts, minimizing its impact on overall training costs (less than 0.01 % of total cost).
To adapt the complemented visual and enhanced text features towards the unseen target domain, we take advantage of CPNet which extracts the unique domain knowledge that CLIP may exclude. Specifically, we reshape the batched unlabeled data so that the inter-attention is computed among the batch instances. Within the same domain, the domaín information typically consistent across data instances (Zhong et al., 2022; Chi et al., 2024). It allows us to treat the propagated batch information as domain-specific knowledge. We then integrate it with a learnable domain cache to form a domain-specific prompt. This prompt guides the fusion of text and image features, enhancing the coherence domain-specific outputs, and thus adapting to a particular target domain.
We name our framework as Learning to Complement (I.2C) and our contributions are: 1) We propose a parallel CPNet to learn sptasct-specific knowledge to complement the generalized frozen CLIP visual feature; 2) We propose efffotless greedy ensemble and lightweight refinement to enhance the class-wise inter-dispersion for text features to benefit adaptation; 3) We improve the domain knowledge extraction process to adapt both text and visual features in domain-aware manner, 4) We evaluate L2C on benchmarks, especially on challenging real-world WILDS dataset with smaller backbones (i.e., +5.1 in Fl for iWildCam and +3.1 % in WC Acc for FMoW with ViT-B/16).
RELATED WORK
Distribution shifts often degrade the learning-based methods (Zhang et al., 202 To address this, Domain Generalization (DG) (Zhou et al., 2020; Lv et al., 2022) and Unsupervised Domain Adaptation (UDA) (Zhang, 2021; Peng et al., 2019; Pei al., 2018) have been explored. DG extracts
2

---

Published as a conference paper at ICLR 2025
comminninvariian features for multiple domains (Li et al., 2018; Long et al., 2018), but single model often falls short. UDA adapts source knowledge to unlabeled target data through extensive target-specific training, but its scale and resource demands limit practicality. PODA (Fahes et al., 2023) and ULDA (Yang et al., 2024b) achieve zero-shot adaptation by leveraging natural language descriptions of target domains without accessing data. In contrast, FSTT-DA uses domain cues from larget domain images, making it suitable for scenarios where descriptions or labels are unavailable.
Test-time adaptation (TTA) (Su al., 2022; 2024c;a) is an emerging learning paradigm that incorpo- rates an additional learning phase at test time before inference, to mitigate distribution shifts. This phase often utilizes unsupervised objectives like entropy minimization (Wang al., 2021; Niu et al., 2022: Zhang et al., 2022a; Gong et al., 2022; Zhao et al., 2023; Wang et al., 2024), teacher-student self-training (Yuan et al., 2023; Marsden et al., 2022; 2023; Liu et al., 2022a; Zhang et al., 2024), auxiliary tasks (Sun et al., 2020; Liu et al., 2023; Chi et al., 2021; Liu et al., 2022b), and contrastive learning (Chen al., 2022a; Wu et al., 2023; Liang et al., 2022) for supervision. Although effective, these approaches often require model fine-tuning challenges their scalability and intrinsic OOD capabilitiess in larger foundation models (Wortsman al., 2022a). Recent developments include the use of vision prompts (Han et al., 2023), which adapt by modifying only a minimal number of parameters to leverage the existing knowledge within large models (Zhang al., 2021b; bb; Gan et al., 2023). HybridPrompt (Wu et al., 2024a) and ProD (Wu et al., 2024a) introduce prompt- -haeed algorithms to extract domain-specificc knowledge to address domain shifts in cross- cross-domain few-shot learning (CD-FSL) where labelled support set avaibale (Guo et al.. 2020; Wang & Deng, 2021; Fu et al., 2023). However, these prompts are inserted into various layers and require access to the weights of the base model. Therefore, they incur additional computational costs and pose challenges in scenarios where privacy concerns or proprietary models limit flexibil- ity (An et al., 2022). Our work introduces a practical, gradient-fre adaptation method, enabling model deployment in black-box environments.
complex design of learnable parameters. It
Few-shot test-time domain adaptation (FSTT-DA). FSTT-DA utilizes few unlabeled data samples for domain adaptation, providing practical edge over instance- methods. MetaDMoBE (Zhong et al., 2022) separately trains a pool of domain domain-specific experts, a process that creates boundaries in knowledge transfer among source domains. MABN (Wu et al., 2024b) focuses on identifying and updating domain-specinf parameters via a self-supervised auxiliary branch. It makes its effectiveness dependent on the auxiliary task. VDPG (Chi et al., 2024) harnesses the inherent OOD generalization capabilities of VFMs (Zhang et al., 2023) to create a domain prompt generator that aligns VFM features specific domains, yet it is limited by a lack of dataset-specifie knowledge. In contrast, our method directly learns from the input space, effectively integrating dataset-specific knowledge with the robust OOD capabilities of foundation models.
Efficient tuning with side network. Recent trends favor employing a smaller, parallel side network over inserting learnable parameters into the main backbone. This approach has proven effective in dense prediction (Chen et al., 2022b; Xu et al., 2023) and recognition tasks (Fu et al., 2024; Wang et al., 2023; Sung et al., 2022). However, these methods typically require accessing or modifying the main backbone's 's intermediate features for efficient adaptation. Our proposed framework diverges by integrating a revert attention mechanism that learns dataset-specific knowledge, aiming to enhance the output of pre-trained foundation models without intervening in their internal processes.
PRELIMINARIES
Problem setting. In this study, we address Few-Shot Test-time Domain Adaptation (FSTT- DA) (Zhong et al., 2022; Wu et al., 2024b: Chi al., 2024). In this setting, a model is trained and then is tested on M target domains m==1- We assume distribution shifts occur between
on N labeled source domains D. = (DD = (I. (x), ] Jn=1. Im==-
with only input images: D. = (D = (2.))"
any source and target domain pairs and they share the same label space D, = V/. At test-time, for each target domain D a few-shot of unlabeled data x is used to adapt the model which is used for inference D This adaptation stage is source-free, source data are not used post-training. Appendix B depicts the setting of FSTT-DA.
Motivations. Foundation models like CLIP, trained on web-scale datasets (Oquab et al., 2023; Radford et al., 2021), have markedly improved downstream tasks (Cho et al., 2023b; Goyal et al., 2023). However, it remains significant challenge adapt these models unseen domains using
3

---

Published as a conference paper at ICLR 2025
Pre-processing training
Gradient
Text templaates Af Askoelugteee!
Inter-dispersion Lun
CLIP text encoder
Greedy text ensemble
Text feature refinement (Me, Ma)
namess Oal Sereeao
Text Image
Domaii Caclie
Path for dumain pronein
Legend
Unlabeld support
CLS
Path complement CLII leature fox each class
Lightweight
Cross anention
Peature
Reshape
CPNet
Frezet
Learnable
8,1. - I, (8),
Mutiplication
Addition
spectff knowledge
Adopting to dosaiin via Domain-sware fusion
DP
(DAF)
CLS
CLIP image encoder
000
CA
CA
Query set (2)
dissbuty
Ligit
Complenent
CA
Generalized knowledge
Figure 2: Training process of L.2C on source domains. (Top) For dataset, our greedy strategy selects text prompts with larger inter-dispersion which will be refined subsequently (T (Bottom) CPNet is proposed in parallel with CLIP image encoder to learn dataset-specift knowledge complement the generalized knowledge CLIP. To adapt to a domain, a few unlabeled data samples (support set S) used to first generate domain prompt DP via CPNet and domain cache. DP is then used to adapt all the data (query set Q with image feature: [(x and text feature: T in that domain via domain-awar fusion (DAF).
minimal unlabeled data as in FSTT-DA. A key approach (VDPG) has been proposed to harness their inherent OOD generalization capabilities (Chi et al., 2024). This involves using domain- specific prompts based few data features to adapt CLIP's s broad features to particular domains. Nevertheless, CLIP has not specifically seen the downstream datasets. The method deriving domain- specific knowledge strictly from generalized feature space has inherent limitations. Consequently, VDPG's reliance on CLIP's pre-trained knowledge restricts its performance. As shown in Fig. 1 and Table 1, using weaker model like ViT-B/16 yields poor results on the challenging WILDS benchmarks. These shortcomings have motivated us to develop an efficient framework that learns directly from the input space. Our approach not only taps into dataset-specific knowledge including semantics and distribution/domain cues to complement generalized CLIP features, but also leverages text features enrich label semantics, thus significantly enhancing adaptation capability.
METHOD
Overview. We aim to adapt both image and text features unseen domains, as illustrated in Fig. In 1ec.4.1, we introduce CPNet to acquire dataset-specific knowledge, complementing CLIP's visual features. Sec.4.2 covers our greedy ensemble approach and lightweight refinement for enhancing text features. In Sec.4.3, we first demonstrate the generation of commann-specifi prompts and then adapt the features using domain-aware fusion. Sec.4.4 outlines the training and inference process.
4.1 LEARNING DATASET-SPECIFIC VISUAL KNOWLEDGE TO COMPLEMENT CLIP
Parallel CPNet. Freezing CLIP is effective in retaining its OOD capability (Wortsman al., 2022b). We propose an independent CPNet in parallel with the CLIP image encoder to learn dataset-specific knowledge to complement CLIP. We
boldface to refer to a batch of images and æ to indicate one image. Given an image E IR XxXxXxXX xC it is first split into patches and encoded into embeddings with dimension d. A class token [CLS] is pre-pended to form the input (in) tokens in E [gIx(+1)xxd Let represent the CLIP image encoder, we denote its output as I(x"") E [lx We impose minimal architectural constraints on CPNet but only match its output dimension with that of the CLIP encoder, thus express CPNet (z:)) (I+((x)
4

---

Published as a conference paper at ICLR 2025
Given that CLIP has mastered extensive generalized knowledge, it is strategically beneficial for CPNet to only acquire necessary dataset-speciffe semantic and domain information not encompassed by CLIP. Consequently, we introduce a parameter-free Revert Attention (RT) mechanism (Chen et al., 2018) 8) to specifically target the learning of CPNet. We employ the Scaled Dot-Product Attention method (Vaswani et al., 2017) calculate the attention between their outputs and then compute its complement with respect to 1. The resulting reverted attention map A is reapplied to (P(x using dot product:
CP (x²))= = A CP(x where A softmax
CanfimaxCPP(2"" (CP(x"") I(x²))),
(1)
where represents the dot-product. This approach ensures that CPNet focused solely on learning information distinctive from CLIP, rendering efficiency and compactness (e.g., requiring only transformer blocks to complement 12-layer ViT-B/16 on the DomainNet dataset). The datasct-specific
information added back to complement the CLIP visual feature: ((xt) = I(x + Cp (x
4.2 ENILANCING THE LEARNING ON DATASET-SPECINIC LABEL SEMANTICS
For classification with CLIP, text features act class prototypes, generated by pairing class names with template (e.g.. A photo of a [CLASS]) ) We freeze CLIP leverage its OOD generalization, so the same set of text features is shared across do- mains. In Fig. in image embeddings for two selected class pairs across domains in DomainNet. The significant se-
Apple vs Buseball Bet
Ploor Lamp VS Light Bulb
caleulate the average difference
mantic variation, even within the same class pairs,
highlights the need for domain-specifie text features. Since CLIP relies on the unified space of text and image, we focus on adapting text features alongside image features.
Figure 3: Image embedding differences are calculated as minus cosine similarity for the displayed classes and domains, highlighting that semantic variations differ notably across domains, which requires customized text fea- tures for each domain.
To facilitate the adaptation process, we propose re- ducing domain biases in the text features by increas- ing their discrimination (inter-dispersion). This en- sures that text features remain neutral across all do- mains before adaptation. To achieve this, we in- troduce greedy text ensemble strategy processing step, selecting text templates that enhance disperiisersi.n. Let T(Pc) represent the text features from a prompt template P and class labels C using text encoder T. The anter-dispersion of text features is quantified by their uniformity in hypersphere (Cho al., 2023a; Wang & Isola, 2020), described as:
pre-
= Σ &
exp(-t|(T.(P) - (P))
(2)
where i.j represent ith jth class, t 2 by default. Assuming prompt templates, we sort them by increasing Luni values: { We begin our ensemble with the most uniform embedding, T(P)), and incrementally add others. The ndhh prompt is retained in the ensemble list E if it reduces the overall uniformity metric:
C.mi (Ane(Ave(( T(P*)))) < <Av))(AveEE)).
where Ave represents averaging ensemble. After selecting prompts greedily, we ensemble E into Tore RC via averaging, well-informed initialization that encapsulates CLIP''s text knowledge. Pseudocode for the greedy ensemble is provided in Appendix E. In Appendix F.2, we show another alternative quantify inter-dispersion. At this stage, the text encoder T can be discarded making the ensemble process effortless, occupying less than 0.01 % of the total cost as in Appendix D.2. To further improve the dispr-dis,,,,oon, we introduce lightweight module to refine T when training on the source data:
gre M.Trre Ma + Tgre
where M. R and Ma E Rdxd adjust along label and feature dimensions, respectively. Additionally, we apply the uniformity loss the output Lune ore
5

---

Published as a conference paper at ICLR 2025
4.3 ADAPTING TO DOMAIN VIA DOMAIN-AWARD FUSION
Domain prompt computation. We aim to compute a domain-specifie prompt to adapt both image (I(="))) and text (TT")) features towards unseen target domains. The source domain knowledge is critical in helping the computation of domain prompt (Chi et al., 2024). We follow the Cache-based learning methods (Zhang et al., 2022b; Zhu et al., 2023) to build learnable key (K E (R4xx)- value (V (VER domain cache to store and query such learned source knowledge, where is the cache size. Given batch of b unlabeled images from domain in FSTT-DA, K is used to compute the similarity between that domain and the source domains, which will be used to query the source knowledge from V.
To process x, we first transform it into an embedding x E R VDPG directly feeds x into transformer. Since the attention mechanism operates along the dimension, excluding the batch dimension b, this results in separate attention for each image in x This approach is non-intuitive, as domain knowledge should be mnstance-agnostie Instead, we propose computing interrelations within the batch (Blattmann et al., 2023) using the dataset-specific CPNet. To achieve this, we reshape x by combining the first two dimensions into X E xxxx)xxxxd This allows the attention mechanism to operate along the (b x 2) dimension, interleaving all the images in
Analogous to classification, where a CLS token aggregates global information for an image, we prepend a learnable domain token (D) that all information in X is propagated D through attention (Dosovitskiy et al., 2020). The prepended token [D, X is fed to CP. We then retrieve D from CP(DD, J query the source domain information from K-V cache by computing their similarity as sof faar (mmar(Kk") ). V. The domain prompt (DP) is the concatenation of the queried source knowledge and D which represents the domain-specifie knowledge of that domain:
DP E g((+1)xd |softmax (K5") V, D).
(5)
Domain-aware fusion. Once the domain prompt DP is obtained by Eq. 5 with unlabeled data, we aim adapt all the data -.'h in that domain. To this end, we propose cross- attenntttns among DP, [(xi") and T T to progressively fuse them with domain-aware fusion module: DAF(DP, ((x"").
T")). Specifically, we first separately project ixx* and T into that domain by conditioning on
DP using cross cooss-attention (CA) (Jaegle et al., 2021):
DP CA(K =T gre V = 9Te .Q= DP),
(6)
DP = CA(K = i(.""), V ), Q DP),
Note, that we omit the QKV weight matrices and the FFN layer for simplicity. Now, DP" and DP contain their modality information in the same domain, we then cross fuse them into other modality:
dm CA(K DP" V =DP" Q = i(x")))
dm CA(K = DP V = DP =
where dm represents domain. We then obtain their class token dm from dm and its corresponding text feature 7' from Tdm as [I TTdm) follow the original CLIP loss (Goyal et al., 2023; Radford et al., 2021) on the adapted text and image features as:
( )qmm)....,
" ], where B is the batch size. We finally
exp ((Tdm) (Tylme)
B
exp ((T.)m) (Tylm)
Ldlp
log
log Σ exp ((TØm) (Tadm))
(8)
ΣØ exp ((I6m)) (Tydm))
The final loss is defined as: Liotal = Letip + XLani where À balances two losses.
4.4 DOMAIN-CENTRIC LEARNING TO ADAPT
Training on source domains. Our ultimate goal is to adapt to unseen target domains using only few unlabeled data samples. It is essential to align the training objective directly with the evaluation protocol, embodying the system learning to adapt. Therefore, instead of uniformly sampling the data across domains, we follow VDPG to learn at the domain level and mimick the adaptation at test-time as in meta-learning (Finn et al., 2017; Chi et al., 2022; Gu et al., 2022).

---

Published as a conference paper at ICLR 2025
Algorithm Domain-centric learning to adapt
Algo. and Fig. show train- ing scheme. For each dataset, the text process is only executed once to obtain the ensembled text feature (I.2-3). The entire text encoder can be discarded before official training, For each itera- tion, we consider it as an adap- tation task on a randomly sam- pled source domain D. Two dis- joint support set (Xs) and query set (xo=yo) are sampled. (xs) is used to generate the domain prompt (L8-9). Then the com- plemented visual feature is com- puted for xe and adapted by the domain prompt (L10-12). The
Require: I/T: CLIP imageltet encoders; (Pa) ) P text prompe templates: C: C classes with names, D,: source domains; a. learning rate; CP: CPNet, K/V: K-V
domain cache; DAF: domain-aware fusion module:; Me/ MM: text refinement;
1: Greedy text feature ensemble
2: {T((E)) P
p==
D Compute and sort text features for all text prongt templates
3: Obtain pare via greedy ensemble using Eq. 3, then discard the text encoder.
4: Leaarning to complement CLIP and adapt to a particular domain
5: for itril to Max iteration do
6:
(xs), (x))Na)~ D. > Sample a souree domain and support and query sots x2 <-
7:
X3 <
> Form input embedlding > Agaregate domain information from support set Form domain prempt for domain 2 [(x²)) 1(x()) CP (x Compute complemented visual feature
8:
δ &(sl)
9:
DP [softma((K6") ) V.
10:
11:
Tare M,T Ma Tore
Retine ensembled text feature Tee I + (DF, T(x T(xtr) ) 1ar*) b Adapt query lowards domain D
12:
13: (CP, K/V, DAV, Mr /Ma) + (CP, K/V, DAF, M1/Ma) a Ceotal
end for
whole system is evaluated by the loss on the query set (L13).
Adapting to unseen target domain at inference. After iterations of the adaptation task trained on source domains, L2C is ready to adapt unseen domains. Algo.2 in Appendix C outlines the inference process, which consists of two phases: 1) For each target domain, a few unlabeled data samples are first drawn, and the domain prompt is obtained. Afterward, the K-V cache can be discarded. 2) The domain prompt is then used to adapt every data sample in that domain. Fig. 8a & 8b in Appendix demonstrate the two phases.
EXPERIMENTS
Datasets and evaluation. We follow VDPG to evaluate on DomainNet (Peng et al., 2019), which comprises 569K images across 345 classes in 6 domains. We follow the official seaee-one-domain-ou protocol to train 6 models and report accuracy. We also evaluate on 4 WILDS (Koh et al., 2021) benchmarks, known for their real-world challenges and notably low CLIP zero-shot accuracy (Chi et al., 2024). This includes classification benchmarks such as iWildCam (Beery et al., 2021), Camelyon17 (Bandi et al., 2018), and FMoW (Christie et al., 2018). Although CLIP is primarily designed for classification, we also adapt our framework for regression (PovertyMap (Ych et al., 2020)), detailed in Appendix F.3. Evaluation metrics include accuracy, Macro F1, worst-case accuracy, Pearson correlation (r), and its worst-case.
Architecture and training details. We use official CLIP pre-trained ViT-B/16 and ViT-L/14 as the foundation models. Their feature dimensions (d) are 768 and 1024 respectively. Therefore, our CPNet is stacked by regular transformer modules as in ViT with the same feature dimensions. The model is trained for 20 epochs with SGD using cosine decay with initial learning rates of 2.5e-3 and 1e 33 for WILDS and DomainNet. À is set to 0.1 to balance the losses. We use 16 images for adaptation at inference. Appendix G&H lists additional hyperparameters and the text prompts.
5.1 MAIN RESULTS
Evaluation on WILDS. The WILDS benchmarks reveal complex real-world domain shifts, like wild- camera setups, remote sensing, and medical imaging. It characterized by significant data imbalances at domain and class levels. CLIP demonstrates notably low zero-shot accuracies in these scenarios. However, as Table 1 indicates, our method substantially exceeds previous approaches. surpasses VDPG with improvements of 2.1 and 5.1 in Macro-F1 for iWildCam, and enhances WC Acc by 0.3% and 3.1% for FMoW with ViT-L/14 and VIT-B/16. respectively. ViT-B/16 shows notably weaker learning capabilities compared to ViT-L/144 Our method, which learns directly from the input space, effectively harnesses domain-speciff and data-specific knowledge, thus outperforming VDPG, particularly in models with lower capacities (i.e., ViT-B/16) across diverse WILDS datasets.
Evaluation on NomainNet. Table 2 presents the accuracy across various domains and their overall averages. Our approach significantly surpasses VDPG, in 4/6 and 5/6 domains with average accuracy
7

---

Published as a conference paper at ICLR 2025
Table Evaluation on challenging WILDS image testbeds under OOD conditions. It reveals that our method excels in both classification and regression tasks, significantly outperforming SOTA methods. Notably, with smaller network (ViT-B/16), our method surpasses VDPG due independently learned data-specific knowledge. (": results obtained using official code; i; main evaluation metrics WILDS; 9 3/8 channels utilized in PovertyMap as VDPG.)
IWildCam
Camelyon17
FMoW
PovertyMap 2 (Regression)
Method
Backbone
Acc
Macro F1
Acc
WC Ace?
Avg Acc
WC Pearson
Pearson
ERM
71.6 (2.5)
(1.3)
70.3 (6.4)
32.3
(3.0
0.45 (0.06)
0.78 (0.04)
CORAL
73.3 (4.3)
32.8 (0.1)
59.5 (7.7)
31.7 (1.24)
50.5 (0.36)
0.44 (0.06)
0.78 (0.05)
IRM
59.8 (3.7)
15.1 (4.9)
64.2 (8.1)
30.0 (1.37)
50.8 (0.13)
0.43 (0.07)
0.77 (0.05)
ARM-CMI.
CNNs
70.5 (0.6)
18.6
(1.24
27.2 (0.38)
45.7 (0.28)
0.37 (0.08)
0.75 (0.04)
ARM-BN
70.3 (2.4)
22.7)
87.2 (0.9)
10.640
42.0(0.211)
0.49 (0.21)
0.84 (0.05)
Meta-DMoE
(0.3)
(0.6)
(1.4((155
35.4
52.5
0.51 (0.04)
(0.03)
MABN
78,4(0.6)
38.3(1.2)
92.4(1.9)
36.610.41)
53.2(0.52)
0.56 (0.05)
0.84 (0.04)
Zero-shot (ZS)
V/T-
14.9
9.7
50.1
14.3
16.3
0.27
0.38
VDPU
CLIP
(0.2)
(0.3)
93.2 (0.3)
37.8 (0.5)
(0.3)
(0.02)
(0.02)
L2C (Ours)
73.4 (0.4)
35.2 (0.3)
94.2 (0.2)
40,9 (0.4)
54.8 (0.1)
0,50 (0.02)
0.80 (0.03)
Zero shot (ZS)
28.7
1.0
64.2
13.3
21.1
0.35
0.62
FLYP
VIT-L/14
72.2 (0.4)
(0.3)
46.0 (0.3)
(0.4)
VDPG
CLIP
78.8 (0.2)
46.5 (0.3)
96.0 (0.4)
46.4 (0.5)
61.9 (0.4)
0.51 (0.03)
0.83 (0.04)
L2C (ours)
77.3 (0.1)
48.6 (0.4)
96.1 (0.3)
46.7 (0.3)
(0.3
0.56 (0.02)
0.84 (0.03)
Table 2: Evaluation on DemainNet. Our method surpasses SOTA, achieving top accuracy in 4/6 and 5/6 domains, with average gains of +1.4 % and +2.2% % using ViT-B/16 and ViT-L/14, respectively.
Method
Backhone
Clip
Info
Paint
Quick
Real
Sketch
Avg.
ERM
58.1 (0.3)
18.8 (0.3)
46.7 (0.3)
12.2 (0.4)
59.1
49.8 (0.4)
40.9
Mixup CORAL
55.7 (0.3)
(8.5)
44.3 (0.5)
(0.4)
55.8 (0.3)
48.2 (0.5)
39,2
59.2 (0.1)
(0.2)
46.6 (0.3)
(3.4
(0.2)
50.1 (0.6)
41.5
MTL.
CNNss
57.9 (0.5)
18.5 (0.4)
(0.1)
(2.5
59.5
49.2
40.6
SegNet
57.7 (0.3)
(9.0)
45.3
(2.7
(0.5)
(0.2
40.3
ARM
49.7 (0.3)
16.3 (0.5)
(1.1
(..1)
(3.4)
43.5 (0.4)
35.5
MetaDMMBB
63.5 (0.2)
(0.3)
(0.2)
MABN
64.2
23.6
$1.5
15.2
64.6
$4.1
45.5
DePrompt
VIT-B/16 IMN
(0.6)
(0.1)
34,9 (0.1)
(0.2)
(0.3)
(0.3
48.3
Zero-shor (ZS)
69.9
48.2
65.4
14.5
82.3
62.5
$7.1
ERM
WTT-R166
(0.1)
(0.6)
46.5 (4.2)
18.5 (0:9)
58.7 (2.7)
$2.5
44.4
MIRO
CLIP
74.9 (0.2)
37.1 (0.4)
59.8 (0.6)
18.7 (1.2)
72.2 (0.2)
61.2 (0.9)
54.0
VDPG
76.3 (0.2)
(0.1)
67.8 (0.1)
(0.2)
81.5 (0.3)
66.6 (0.2)
59.8
L2C (ours)
75.6 (0.1)
52.1 (0.1)
69.4 (0.1)
17.3 (0.2)
85.5 (0.1)
67.1 (0.2)
61.2
Zero-shot (ZS)
WTL
78.1
$4.0
71.6
21.8
86.0
71.2
63.8
VDPG
CLIP
82.4
$4.9
73.1
22.7
85.0
73.2
65.2
L2C (ours)
82.3
58.7
75.2
24.0
88.6
75.4
67.4
improvements of +1.4 and +2.2 using VIT-B/16 and ViT-L/14, respectively. These improvements underscore the benefits of directly learning dataset-specific knowledge. In Appendix F.I, we further compare with prompt-basee based methods: CoOp (Zhou et al., 2022b), CoCoOp (Zhou al., 2022a) and side branch-based DTL (Fu et al., 2024)
5.2 ABLATION STUDIES
We conduct ablation studies on DomainNet-Inffe iWildcam and FMoW using CLIP ViT-B/16 on various components, including CPNet, Revert Attention (RT), text refinement (Text ref.), greedy ensemble (Greedy), uniformity loss (Lun),, DAF module and training schemes in Table 3. Note, if DAF is not incorporated, the domain branch is also omitted, which will discussed in Table
CPNet and revert attention. As highlighted in Index / vs. 2 of Table 3, incorporating CPNet alongside a frozen CLIP markedly enhances performance across all datasets. On particularly chal- lenging iWildCam and FMoW, there is exceptionally low zero-shot performance. However, effective learning of sppeifc knowledge from the input space results in substantial performance gains, specifically, +13.1 on F1 for iWildCam and 13.8% WC Ace for FMoW. Additionally, integrating reverted attention directs CPNet to assimilate knowledge overlooked by CLIP, sharpening its focus essential dataset-specific insights. This strategy leads to further enhancements (Index 4 vs. 5).
8

---

Published as a conference paper at ICLR 2025
Table Ablation on various components of our work on DomainNettInfc,, iWildCam and FMoW.
Info
iWildCam
FMow
Index
CPNet
Text ref.
Greedy
Luni
DAF
Training
Acc
Acc
F1
WC Ace
Acc
(ZS)
48.2
14.9
9.7
14.5
16.3
2
ERM
49.6
65.8
22.8
28.3
49.0
3
ERM
50.3
68.7
24.0
31.4
50.9
4
ERM
51.0
69.1
26.9
33.3
$1.6
RT
ERM
51.5
71.5
32.9
36.1
53.1
6
RT
ERM
51.7
72.2
33.2
36.0
53.8
7
RT
ERM
51.5
71.9
32.8
36.0
52.8
8
RT
Domain-centric
52.1
73.4
35.2
40.9
54.8
Infograph- 00D Acc
Willdcmm - -OD Acc
WWlldcam -OOD F1
FMoW - WR ACC
FMoW DOD Acc
31.5
4880-shot=48.22
Zero-shat-14:7
Zero-shot-9.7
Zero-shot- 14.5
Zero-shot-16.3
lsyers
lyyers
layers
layers
Figure 4: Analysis different number of transformer layers of CPNet.
Text refinement and greedy ensemble. Text features from frozen CLIP serve as initialization and refining them per Eq. 4 proves beneficial for subsequent adaptation (Index VS. 3). Since text features act as class prototypes, enhancing class discrimination is crucial. The goal is to make these features more discriminative by increasing the inter-dispersion among them, reducing domain biases. This results in more neutral features for unseen target domains. Therefore, using our greedy ensemble (Index 3 VS. 4) and enforcing the uniformity loss (Index 5 VS. 6) leads positive gains. Appendix F.4 provides visualization using t-SNE (Van der Maaten & Hinton, 2008), and Appendix F.6 reports sensitivity on A.
Domain-aware adaptation and fusion (DAF), The text and image features (xx:") and Thee are not inherently tailored to a specific domain. By integrating domain-aware fusion model, both features are adapted to that domain, facilitating the fusion of text and image modalities within that domain. Hence, Index 8 demonstrates substantial improvement over nomain-agnostic predictions (Index 6).
Training schemes. ERM samples batches uniformly without considering domain labels, misaligning it with the protocol of adapting specific domain using limited unlabeled data. In contrast, domain-centric learning to adapt optimizes at the domain level, treating each iteration as a task of FSTT-DA. Thus, domain-centric approach yields further improvements (Index 7 VS. 8).
Effect on number of transformer layers of CPNet. Fig. 4 illustrates the effect of the number of transformer layers in CPNet. Different downstream datasets require varying learning capacities depending on their complexity. For instance, while DomainNet is more stable, the more challenging remote sensing scenario FMoW requires additional learnable blocks to achieve reasonable perfor- mance(Wan Wang et al., 2022). Nevertheless, even with just I layer in CPNet, substantial gains have been with observed over zero-shot performance, thanks the complement of dataset-specifee knowledge.
Analysis on domain prompt DP. Our domain prompt DP aims to adapt the domain domain-agnostic features [(x=")) and gre towards particular domain. It consists of two components: knowledge queried from K-V cache, representing source domain knowledge and current domain knowledge D aggregated from its unlabeled data. We omit some parts of DP, reported in Table 4, equipping with both domain knowledge is essential to better adapt the features towards a domain.
Analysis domain information aggregation. VDPG independently processes all the unlabeled data and then aggregates their domain information via averaging. However, we perform simple reshaping and allow the attention to be performed every pair of the tokens in that data batch which

---

Published as a conference paper at ICLR 2025
Table 4: Ablation on domain prompt.
Table Ablation on domain information aggregation.
iWildCam
FMoW
iWildCam
FMoW
Domain prompt
Aggregation
Acc
F1
WC Ace
Acc
Acc
F1
WC Ace
Acc
K-V cache only
73.0
34.1
37.8
50.6
Mcan
73.2
34.4
37.8
52.8
D only
71.3
32.1
35.4
50.4
Max
71.9
34.5
36.7
52.6
DP
73.4
35.2
40.9
54.8
Reshape
5
73.4
35.2
40.9
54.8
Camelyon17
WWldCCm
FMoW
Poverly
DemainNet-Real
-0
2
Figure Correlation between every pair of V-vectors in K-V domain cache.
has shown superiority as reported in Table 5. For Mean and Max, do not reshape the tensor but take the mean or max over the batch dimension.
Analysis on K-V domain cache. L is the size of the domain cache as the number of learnable vectors. Ideally, each can condense the distinct domain specificity from the source domains. Such property is exhibited by computing the correlations between every pair of the V-vectors, illustrated in Fig. 5. Please note, that we did not apply external constraints on the cache (e.g.. correlation loss (Chi et al., 2024)). Appendix reports the sensitivity on the size of the cache.
Additional analysis on greedy ensemble and text feature uniformity. Table 6 reports the compari- son between the average ensemble (CLIP) and our greedy approach using ViT-B/16 while holding off other components the same. Greater gains more challenging WILDS benchmarks obscrved compared to more structured common objects as DomainNet. Fig. illustrate that a lower uniformity among text features is potentially beneficial for the final performance.
61.2
Gruady atsumbb
Fnsemble method
DomainNet (Ace.)
iWildCam
FMoW (WC Acc)
Ensemble (CLIP)
60.9
33.6
40.1
Final Accuracy
60.8
Groady ensemble
61.2
35.2
40,9
60.6
a
Table 6: Comparison between average ensemble (CLIP) and our greedy approach using ViT-B/16. The greedy ensemble improves across benchmarks, However, DomainNet contains more structured, common objects. Greater gains on more challenging WILDS benchmarks are observed.
60.2
(CAS5))
59.
028
0.50
0.34
0.35
0.30
Text feature uniformity
Figure 6: Final accuracy VS. text unifor- mity DomainNet with ViT-B/16.
6 CONLUSION
In this work, we introduce I.2C address FSTT-DA. L,C adapts model trained on source domains to unseen target domains using just few unlabeled data points. Our method builds on the inherent OOD capability CLIP, complementing with a parallel network that learns data-specific knowledge at the input space through revert attention. Additionally, to effectively integrate data-specific label semantics. To facilitate domain adaptation, we generate a domain prompt that guides the integration of enhanced text and visual features through domain-awe aware fusion. Our extensive experiments validate L2C's effectiveness, showcasing its superior performance across five large-scale benchmarks in DomainNet and WILDS.
propose greedy text feature ensemble
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
For a fair comparison, we use the VDPG codebase, with data processing following the official WILDS code. The pre-trained CLIP models are directly soureed from the OpenAl CLIP repository. Sample code for the greedy ensemble is provided in Appendix E. Other components, such as CPNet, DAF, text refinement, and K-V cache, utilize standard PyTorch functions. Pre-trained models and the full code will be released upon publication of the paper.
REFERENCES
Sk Miraj Ahmed, Dripta S Raychaudhuri. Sujoy Paul, Samet Oymak, and Amit K Roy- Koy-Chowdhury. Unsupervised multi-source domain adaptation without access to source data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021.
Qiyuan An, Ruijiang Li, Lin Gu, Hao Zhang, Qingyu Chen, Zhiyong Lu, Fei Wang, and Yingying Zhu. A sewaccypreserving unsupervised domain adaptation framework for clinical text analysis. arXiv preprint arXiv:2201.07311 2022.
Peter Bandi, Osear Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 38(2):550-560, 2018.
Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iwildcam 2021 competition dataset. arXiv preprint arXiv: 03494, 2021.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In JEEE Conference on Computer Vision and Pattern Recognition, 2022a.
Shuhan Chen, Xiuli Tan, Ben Wang, and Xuelong Hu. Reverse attention for salient object detection. In Proceedings of the European conference on computer vision (ECCV), 2018.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint urXiv:2205.0853: 2022b.
Zhixiang Chi, Xiaolin Wu, Xiao Shu, and Jinjin Gu. Single image reflection removal using deep encoder-decoder network. arXiv preprint 1rXiv:180.000090 2018
Zhixiang Chi, Yang Wang. Yuanhao Yu, and Jin Tang. Test-time fast adaptation for dynamic scene deblurring via meta-auxiliary learning. In Conference on computer vision and pattern recognition, 2021.
Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metafscil: A meta-learning approach for few-shot class incremental learning. In Conference computer vision and pattern recognition, 2022.
Zhixiang Chi, Li Gu, Tao Zhong, Huan Liu, Yuanhao Yu, Konstantinos N Plataniotis, and Yang Wang. Adapting to distribution shift by visual domain prompt generation. In Proceedings of the Twelfth International Conference Learning Representations, 2024.
Eulrang Cho, Jooyeon Kim, and Hyunwoo J Kim. Distribution-aware prompt tuning for vision- language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023a.
Junhyeong Cho, Gilhyun Nam, Sungyeon Kim, Hunmin Yang, and Suha Kwak. Promptstyler: Prompt-driven style generation for source-free domain generalization. In Proceedings of the IEFF/CVF International Conference on Computer Vision 2023b.

---

Published as a conference paper at ICLR 2025
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, and Raoul De Charette. Poda:
Prompt-driven zero-shot domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, 2017.
Minghao Fu, Ke Zhu, and Jianxin Wu. Dtl: Disentangled transfer learning for visual recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Yuqian Fu, Yu Xie, Yanwei Fu, and Yu-Gang Jiang. Styleadv: Meta style adversarial training for cross-domain few-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2023.
Yulu Gan, Yan Bai, Yihang Lou, Xianzheng Ma, Renrui Zhang, Nian Shi, and Lin Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In AAAI Conference on Artificial Intelligence, 2023.
Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lec. Note:
Robust continual test-time adaptation against temporal correlation. NeurIPS, 2022.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
Li Gu, Zhixiang Chi, Huan Liu, Yuanhao Yu, and Yang Wang. Improving protonet for few-shot video object recognition: Winner of orbit challenge 2022. arXiv preprint 4rXiv:2210.00/74. 2022.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01433 2020.
Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16, 2020.
Shayan Mohajer Hamidi, Xizhen Deng, Renhao Tan, Linfeng Ye. and Ahmed Hussein Salamah. How to train the teacher model for effective knowledge distillation. In European Conference on Computer Vision, 2024.
Cheng Han, Qifan Wang. Yiming Cui, Zhiwen Cao, Wenguan Wang. Siyuan Qi, and Dongfang Liu. E' 2vpt: An effective and efficient approach for visual prompt tuning. arXiv preprint vrXiv:2307.13770, 2023.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference Machine Learning, 2021.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao. et al. Wilds: A benchmark of wild distribution shifts. In International Conference on Machine Learning, 2021.
12

---

Published as a conference paper at ICLR 2025
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In European Confererence on Computer Vison, 2018.
Hanwen Liang, Niamul Quader, Zhixiang Chi, Lizhe Chen, Peng Dai, Juwei Lu, and Yang Wang. Self- supervised spatiotemporal representation learning by exploiting video continuity. In Proceedings of the AAAI conference on artificial intelligence, 2022.
Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, and Jin Tang. Few-shot class-incremental learning via eegrops-- data-free replay. In European Conference on Computer Vision, 2022a.
Huan Liu, Zijun Wu, Liangyan Li, Sadaf Salehkalaibar, Jun Chen, and Keyan Wang. Towards multi-domain single image dehazing via test-time training. In Conference on computer vision and pattern recognition, 2022b.
Huan Liu, Zhixiang Chi, Yuanhao Yu, Yang Wang, Jun Chen, and Jin Tang. Meta-auxiliary learning for future depth prediction in videos. In Winter Conference on Applications of Computer Vision, 2023.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. Advances in Neural Information Processing Systems, 2018.
Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, and Di Liu. Causality inspired representation learning for domain generalization. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Robert Marsden, Mario Döbler, and Bin Yang. Introducing intermediate domains for effective self-training during test-time. arXiv preprint arXiv:2208.077 07733, 2022.
Robert Marsden, Mario Döbler, and Bin Yang. Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction. arXiv preprint arXiv:2306.00633 2023.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In ICML, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/93, 2023.
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adapta- tion. In AAAI Conference on Artificial Intelligence, 2018
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In IEEE International Conference on Computer Vision, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In Advances in Neural Information Processing Systems, 2022.
Yongyi Su, Xun Xu, and Kui Jia. Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, 2022.
Yongyi Su, Yushu Li, Nanqing Liu, Kui Jia, Xulei Yang, Chuan-Shema Foo, and Xun Xu. On the adversarial risk of test time adaptation: An investigation into realistic test-time data poisoning, 2024a.
13

---

Published as a conference paper at ICLR 2025
Yongyi Su, Xun Xu, and Kui Jia. Towards real-world test-time adaptation: Tri-net self training with balanced normalization. Proceedings of the AAAI Conference on Artificial Intelligence, 2024b.
Yongyi Su, Xun Xu, Tianrui Li, and Kui Jia. Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering regularized self-training. IEEE Transactions Pattem Analysis and Machine Intelligence, 2024c.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with superviin for generalization under distribution shifts. In International Conference on Machine Learning, 2020.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Neural Information P Processing Systems, 2022.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 911)), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
Dequan Wang. Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test- time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
Di Wang, Jing Zhang, Bo Du, Gui-Song Xia, and Dacheng Tao. An empirical study of remote sensing pretraining. JEEE Transactions on Geoscience and Remote Sensing, 2022.
Haoqing Wang and Zhi-Hong Deng. Cross-domain few-shot classification via adversarial task augmentation. arXiv preprint arXiv:2104.14385, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. International conference on machine learning, 2020.
Yaoming Wang, Jin Li, XIAOPENG ZHANG, Bowen Shi, Chenglin Li, Wenrui Dai, Hongkai Xiong, and Qi Tian. Barleria: An efficient tuning framework for referring image segmentation. In The Twelfth International Conference on Learning Representations, 2023.
Ziqiang Wang, Zhixiang Chi, Yanan Wu, Li Gu, Zhi Liu, Konstantinos Plataniotis, and Yang Wang. Distribution alignment for fully test-time adaptation with dynamic online data streams. In European Conference on Computer Vision, 2024.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In IEEE Conference on Computer Vision and Pattern Recognition, 2022b.
Jiamin Wu, Tianzhu Zhang, and Yongdong Zhang. Hybridprompt: Domain-aware prompting for cross-domain fhw-shot learning. International Journal of Computer Vision, 2024a.
Yanan Wu, Zhixiang Chi, Yang Wang, and Songhe Feng. Metaged: Learning to continually learn in
generalized category discovery. In International Conference on Computer Vision, 2023.
Yanan Wu, Zhixiang Chi, Yang Wang, Konstantinos N Plataniotis, and Songhe Feng. Test-time domain adaptation by learning domain-ane -awarn--wwre batch normalization. In AAAI Conference on Artificial Intelligence, 2024b.
14

---

Published as a conference paper at ICLR 2025
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
En-hui Yang and Linfeng Ye. Markov knowledge distillation: make nasty teachers trained by self- undermining knowledge distillation fully distillable. In European Conference Computer Vision,
2024.
En-Hui Yang, Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information constrained deep learning for classification, arXiv preprint (rXiv:2309.09123, 2023.
En-Hui Yang, Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information constrained deep learning: Framework and preliminary results. In 2024 IEEE International Symposium on Information Theory (ISIT), 2024a.
Senqiao Yang, Zhuotao Tian, Li Jiang, and Jiaya Jia. Unified language-driven zero-shot domain adap- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Linfeng Ye, Shayan Mohajer Hamidi, Renhao Tan, and EN-HUI YANG. Bayes conditional distribu- tion estimation for knowledge distillation based on conditional mutual information. The Twelfth International Conference Learning Representations, 2024.
Christopher Ych, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature communications, 11(1):2583, 2020.
Dongshuo Yin, Xueting Han, Bin Li, Hao Feng, and Jing Bai. Parameter-efficient is not sufficient: Exploring parameter, memory, and time efficient adapter tuning for dense predictions. arXiv preprint arXiv:2306.09729 2023.
Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark Hasegawa-Johnson, Yingzhen Li, and Chang D Yoo. C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. arXiv preprint urXiv:2403. 2024.
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In CVPR, 2023.
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift, Advances in Neural Information Processing Systems, 2021a.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. In NeurlPS, pp.
2022a.
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip Tip-erapter: Training-fiee free adaption of clip for few-shot classification. In European conference on computer vision, 2022b.
Xin Zhang, Shixiang Shane Gu. Yutaka Matsuo, and Yusuke Iwasawa. Domain prompt learning for efficiently adapting clip to unseen domains. arXiv preprint arXiv:2111.12853, 2021b.
Xin Zhang, Shixiang Shane Gu, Yutaka Matsuo, and Yusuke Iwasawa. Domain prompt learning for efficiently adapting clip to unseen domains. Transactions of the Japanese Society for Artificial Intelligence, 2023.
Yang Zhang, Songhe Feng, and Jiazheng Yuan. Continual compositional zero-shot learning. In Proceedings of the Thirly-Third International Joint Conference on Artificial Intelligence, 2024.
Youshan Zhang. A survey of unsupervised domain adaptation for visual recognition. arXiv preprint 12Xiv:2/12.0674 2021.
15

---

Published as a conference paper at ICLR 2025
Bowen Zhao, Chen Chen, and Shu-Tao Xia. Delta: Degradation free fully test-time adaptation. In ICLR, 2023.
Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang, Yuanhao Yu, and Jin Tang. Meta-dmoe: Adapting to domain shift by meta-distillation from mixture-of-experts Advances in Neural Information Processing Systems, 2022.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In European Confererence on Computer Vison, 2020.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE Conference Computer Vision and Patern Recognition, 2022a.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 2022b.
Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
A LIMITATION
Our method enhances the generalized knowledge of robust CLIP, which serve as the primary con- tributor. However, when the downstream dataset significantly diverges from the pre-training, the load on our CPNet increases, necessitating larger network. Despite this, our approach focuses on acquiring the excluded knowledge directly from the input space. Consequently, the trade-off between computational demand and performance is more favouralle compared to previous methods (VDPG).
B ILLUSTRATION FOR FSTT-DA SETTING
Training (source domains)
Inference (unseen target domains)
Sparso
Tor
Taga
Trand
podd
Tage (oea))
Tage
Doeas
Thoining en domonia
Adipt saing a feu vnisicied
Figure 7: Illustration of FSTT-DA setting. After training on the source domains, the model adapts to each of the target domains using a few unlabeled data samples. Each target domain has a tailored model which will be used infer all of the data in that domain.
C INFERENCE PROCESS (ADAPTING TO A TARGET DOMAIN)
Algorithm Inference: adapting to an unseen target domain
Require: CLIP image encoder; D1": unseen target domain; CP: CPNet; K/V: K-V domain cache; DAF: domain-sware fusion module; iave trained feature:
1: // Compute the domain prompt
2: (xs) D
b Sample a unlaheled samples
target domain
3: K5 x8
Form input embeddings
4:
CP((D,
Aggregate domain information from unlabeled
5: [solimax KD V.
Compute the prompt for domain 27
6: Discard K-V domain cache
7: #/ Adapting every data in the target domain
8: every image xa D do
x Xg
Form input embeddings
10: T(x)") t 1(x()) + CPRT(s))) (x
b. Compute complemented visual feature
11: TH" 1dm + DAF(DP, f(x")) fare)
D. Adapt the image towards domuin DD"" Compute predictions
12: Logits DogiissCCoiiesssmllaittyyy mdm)
13: end for
Algo. 2 shows the process of inference for adapting to particular unseen target domain. It contains two phases:
Domain prompt computation. For unseen target domain, we first collect few unlabeled data samples and compute the domain prompt using the CPNet and the domain cache. Such a step is illustrated in Fig. 8a and L1-5 of Algo. 2. Please note, that after this step, the domain cache can be ignored.
Adapting the data using the domain prompt. Once the domain prompt is computed, it is utilized to adapt all data samples in that domain. This stage follows the process as in L7-12 of Algo. 2. Also illustrated in Fig. 8b, this step only involved CPNet, CLIP image encoder and proposed DAF.
17

---

Published as a conference paper at ICLR 2025
unlabeled data samples
doeak wkwi
Reshupe
Lightweight
Domaln
Cache
1. (00)
CPNet
(a) Computing the domain prompt (DP) for the target domain using few unlabeled data samples.
Lighteeigt OPhect
Teane lin foree
Data streen
Desaliranat Bikke
CLIP
Complenen
CA
Genrrleed hauvisdie
(h) Adapting every data in the target domain.
Figure 8: Inference process for adapting to unseen domain. Adapting to unseen target domain involved two phases. The first one is the domain prompt computation using a few unlabeled data sample as in (a). The second phase is to utilize the domain prompt for inferencing all the data in that domain as (b).
D COMPUTATIONAL RESOURCES
D.1 COMPUTATIONAL COMPARISON OVER THE ENTIRE FRAMEWORKS
Table Comparison on speed and memory (batch size of 64).
CPNet(1 layer)
CPNe1(3 layer)
CPNei(6 layer)
VIPG
Datasets
Camelyonl M.vertyyaap
DomaintNeLWWile
FMoW
All datasets
purams
49.2
Train meBo
3872
$554
8106
3672
Train speed(s/batel
0.97
Inferenee memory(MB)
2270
2330
2430
2798
Inference s/bacc(s
0.64
0.67
0.73
0.69
Table 7 reports the memory usage and speed during both training and inference. Despite introducing a parallel CPNet, the resource consumption remains comparable VDPG. While VDPG relies heavy guidance module, our framework primarily allocates parameters within CPNet. The increase in memory during training largely due tensor reshaping, allowing attention to applied across the entire batch. However, the domain prompt computation is performed only once per target domain and gradient-free. As shown Fig. 8b, the main computation during inference is streamlined- such the text encoder, text feature refinement module, and K- V cache all eliminated, making our framework highly efficient.
D.2 EFFICIENCY ON GREEDY TEXT ENSEMBLE.
Greedy Ensemble is executed pre-processing step before large-scale training (L3 of Algo. I). Once the text features for all templates are obtained, the entire text encoder can be discarded, making ensembling highly efficient. For instance, in DomainNet-real with batch size of 64, the image encoder requires 120K forward passes over 20 epochs, while the text encoder only needs 80 forward passes to compute the text features, resulting in minimal resource usage (<0.01%)
18

---

Published as a conference paper at ICLR 2025
E PYTORCH-LIKE SAMPLE CODE FOR GREEDY ENSEMBLE
Number of text prompt temolates
Number of classee feature dimension Tensor, shape-(P, C d]
TE
P sets of text embeddings
Score: Tensor, shape-[2] correeponding uniformity loss for TE
def uni uniformity_ eebed,, t-2):
text_embed; shape-iC, d]
return torch.pdis pdist (text_ _ebet_ p-2) pew (2. 0).0) mul (-t) .exp mean
def sort_ poct_unifermity (IE, Score)
sort_index torch.argsor. (Score) cpp numpy return [[(sort_idx]
def ensemble [TF_]ist): strck list, dim-0) (dim-0)
def seeeddyemserbletee (TE, Score):
final_TE []
Te_sorted fort_uniforriae (TE, Score)
take the prompt embedding with Teast und Formety lass as base Fimal_TE.append (TR_sorted
for range (1, 2):
temp_TE final_CB + TE TE_sorted[1]
if uni formity_loss (ensemble (temp_TR) ) < un Formi ty_ loss (ensemble (Ff Final_TR temp_TR
return ensemble (final_TE)
F ADDITIONAL EXPERIMENTS
F.1 COMPARISON WITH PROMPT- BASED METHODS AND SIDE BRANCH- BASED METHODS
We further provide comparison with prompt-based methods: CoOp (Zhou al., 2022b), Co- CoOp (Zhou et al., 2022a) and side branch-hased DTL (Fu et al., 2024). Since these methods are non-adaptive, we applied test-time optimization as TPT minimize the entropy (Shu et al., 2022).
Table 8 outlines the architectural differences. TPT relies on gradient updates, making gradient flow crucial and thus limited white-box settings. In contrast, VDPG and our proposed L2C generate domain-specifti prompts for adaptation, enabling them to operate in the more challenging black-box setting. Despite these constraints, L2C still surpasses all other methods, as shown in Table
Table 9: Performance comparison.
Table 8: Architectural comparison.
Method
Ave Acc. (DomainNet, VIT-B16)
Method
Black/wwhie box
Gradient test-time
CoOp CoOp + TPT
60.6
CoOp+ + TPT CoCoOp TPT
white box
roquired required required
CaCoOp
white box
CoCoOp TPT DTL DTL TPT TPT VDPG L2C (Ours)
60.4
DTL. + TPT
white box
57.6
VDPG
black box
gradient-free
59.2
L2C (Ours)
black box
gradicnt-frce
59.8
F.2 ALTERNATIVE CRITERIA FOR TEXT EMBEDDING UNIFORMITY
The criteria we used for text feature inter-dispersion among classes is determined by the uniformity in hypersphere (i.e., Eq. 2). It can also be determined by measuring the Average Text Feature
19

---

Published as a conference paper at ICLR 2025
Dispersion (ATFD) which calculates the distance of all class embedding their centroid (Yoon et al., 2024). For a text features T(Pc) with a prompt template P and class labels C, ATFD is computed as:
c
1 ATFD
Σ EE|C
|T centroid
(P
where T centroid
T(Pc),
(9)
C
C
where I1-112 measures the L.2 distance. Please note, a smaller ATFD indicates the class text features are closer to the centroid, therefore, the class features are more closely clustered. In contrast, larger ATFD indicates a more dispersed distribution of the text features. Therefore, to integrate ATFD into our greedy ensemble pipeline, we need to sort the features of text in reverse order based on ATFD and select the prompts that can maximize ATFD after ensemble. The loss function then becomes Letip X ATFD.
Liotal
Table reports selected text prompt templates for DomainNet with both Luni and ATFD. The selected templates match with each other and the average is also very close to each other. Table 11 reports a performance comparison of using Luni and ATFD on additional benchmarks, showing close results.
Table 10: Comparison between metries of using two different criteria for text feature inter-dispersion.
DomainNet (ViT-B/16)
Inter-dispersion criteria
funii
ATFD
blurry photo l).
blurry photo
embroidered ().
embroidered {}-
itap of the []-
itap [].
Selected prompt templates
my [].
[]-
itap of [].
itap of [}-
black and white of (1. black and white photo ().
Average uccuracy on domains
61.2
61.1
Table 11: Additional performance comparison of using two different criteria for text feature inter- dispersion.
iWildCam
FMoW
DomainNet
Method
Acc
Macro
WC
Acc
Curii ATED
73.4
35.2
40.9
54,8
61.2
40.8
F.3 ADOPT CLIP FOR REGRESSION IN POVERTYMAP DATASET
Regression task CLIP: Regression task requires a single number output, therefore, it equivalent to setting the number of output class as (Chi et al., 2024). Since the PovertyMap aims to estimate the wealth index for region, therefore, we simply use sentence prompt as the text input of the CLIP text encoder: A satellite image showing the wealth index, yielding a text embedding with output dimension as Td E glxd With the adapted image feature pm e R Bxd the logit is obtained by matrix multiplication between them, with shape E Bx1 Therefore, each image has single output number in our framework. For the regression task, we train the model using MSE loss, and omit the text uniformity loss as there is only one "class".
Surprisingly, without any training, the original CLIP model is able to predict zero-shot regression (e.g., the regional wealth index) just like zero-shot classification shown Table 1, the zero-shot regression shows positive correlation the wealth index). Our method can significantly boost performance by learning the complement knowledge and adapting to those unseen target domains.
F.4 T-SNE VISUALIZATION OF COMPARISON ON TEXT FEATURES
20

---

Published as a conference paper at ICLR 2025
(CLASS)
Groedy
Refined
10
-20
-10
10
Figure 9: t-SNE (Van der Maaten & Hinton, 2008) visualization of comparison on text features with prompt [CLASS], ), greedy ensemble and refined output (ViT-B/16 on DomainNet).
Fig. 9 shows the visualization of the features for different prompting methods and also our simple yet effective refinement output. It clearly shows that the features with prompt [CLASS] have more clustered features which are less discriminative. Our greedy ensemble greatly increases the distance among the class features. With simple refinement using Mc. Ma and the uniformity loss, the features are further separated.
Table 12: Sensitivity on domain cache size.
Table 13: Sensitivity on loss balancing weight A.
iWildCam
FMoW
iWildCam
FMoW
L size
Acc
F1
WC Acc
Acc
A
Acc
F1
WC Acc
Acc
69.2
32.8
33.2
50.6
0.01
73.2
35.1
40.8
54.7
73.2
35.0
40.9
54.8
73.4
35.2
40.9
54.8
10
73.4
35.2
40.7
54.5
1.0
73.5
34.9
40.4
54.8
F.5
SENSITIVITY ON DOMAIN CACHE SIZE:
Table 12 reports the performance with different size of domain cache. When size=1, the learning capability is too small. Increasing to 5 or 10 makes it more stable.
F.6 SENSITIVITY ON LOSS BALANCING WEIGHT A:
Table 13 reports the sensitivity on A. Our framework is less sensitive to À as it is only applied to the text features at the beginning, with only Mc and Ma optimize. The effect is the convergence speed, but the ultimate performance is quite stable.
G ADDITIONAL EYPER-PARAMETER
We utilize the same configurations of transformer blocks of VIT-B/16 ViT-L/14, therefore, the feature size can be matched between the main backbone and the CPNet. The only hyper-paramente we tune is the number of transformer blocks. We show the number of layers for different datasets and the total number of learnable parameters and compare them with FLYP and VDPG when ViT-B/16 is utilized in Table 14. The size of the domain cache is reported in Table 15. All the experiments can be conducted with single NVIDIA V100 GPU, We set the batch size as 64 (12 images for support and 52 images for query set). Each iteration runs 0.4 seconds.
21

---

Published as a conference paper at ICLR 2025
Table 14: Configuration of CPNet and the total number of Iearnable parameters.
L2C (ours)
FLYP VDPG
NomainNet iWildCam
Camelyon17
FMoW
MavertyMap
of transformer blocks
3
6
Learnable parameters
27.9M
27.9M
13.8M
49.2M
13.8M
149M
32.1M
Table 15: Size of domain cache L.
DomainNet iWildCam Camelyon17 FMoW PovertyMap
T.
10
10
5
10
H DETAILS ON TEXT PROMPTS TEMPLATES:
In this section, describe the candidate prompts used perform the greedy ensemble, the full list of attached in supplementary material (MS Excel spreadsheet SextProomtTeemplate Table 16 shows the selected text prompts among the candidate text prompt follows:
DomainNet & iWildCam: We use 80 text prompt templates that used for ImageNet (Deng et al., 2009) from CLIP (Radford al., 2021) and FLYP (Goyal et al., 2023).
FMoW: We use 14 text prompt templates from FLYP (Goyal et al., 2023). The prompts describe the photos in remote-sensing scenarios.
Camelyon17: We filter some text prompts from the 80 ImageNet text prompts that can be used to describe the medical issue and generate some using ChatGPT. In total, there are 56 text prompts.
Please note that PovertyMap is regression task, the prompts for this task and the experimental setting for regression are reported in Sec. F.3.
Table 16: Selected text prompts by our proposed greedy ensemble. i is replaced by class name.
iWildCam
DomainNet
FMoW
Camelyon17
blurry photo of the [].
blurry photo
acrial photo of () occania.
dark photo the ().
a dark photo of the ().
embroidered ().
(]. a black and white photo the ().
cropped photo
().
itap of the (}.
good photo (}.
black and white photo of the (}- black
itup of my [1-
photo O-
itap {}.
close-up photo of the (]-
black and white photo of ().
a photo many ().
stap of my {)-
a bright photo of the []-
itap of
good photo of a
22


Reviews:
[{"review_id": "XlxX4OniDT", "reviewer": "Reviewer_aphZ", "summary": "This paper introduces a novel approach for few-shot test-time domain adaptation (FSTT-DA) by learning to adapt frozen CLIP models to specific domains using only a few unlabeled examples. The method leverages strong CLIP`s OOD abilities and further learns on the input space to complement CLIP's generalized features with dataset-specific knowledge. Additionally, it enhances text features through a greedy ensemble strategy and proposes a domain-aware fusion approach for adapting both text and visual features toward a target domain.", "questions": "1. what are the additional number of parameters and computation costs?\n2. In Lines 510-511, the paper mentions that compared to previous work external constraints are not used. I am curious why these losses are not used. Will the results become worse when using them?\n3.\tHow does the similarity between the target domain and the source domain affect the method?\n2.\tIn Fig. 4, for some datasets, the increasing number of layers (parameters) of CPNet can not bring further improvement in accuracy but makes the accuracy lower.  What is the reason behind this? \n5.   Open question: why the FSTT-DA task with few-shot unlabeled data is selected as the testbed, what about the few-shot labeled data but also with the cross-domain (CD-FSL) [ref1,2,3] setting?  To me, the unlabeled and small labeled data don't come together quite often.  \n\n[ref1]  A Broader Study of Cross-Domain Few-Shot Learning\n[ref2] Cross-Domain Few-Shot Classification via Adversarial Task Augmentation\n[ref3] StyleAdv: Meta Style Adversarial Training for Cross-Domain Few-Shot Learning", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The overall writing is good and clear.\n2. Promising results are obtained in the studied benchmarks. \n3. Comprehensive evaluations are conducted, and the effectiveness of the proposed modules is also shown.", "weaknesses": "Novelty is somewhat limited.  I appreciate the proposed method improved the base CDPG obviously and the proposed method (as in Fig.2) also includes many submodules. However, except for the greedy text ensemble strategy, the techniques used in the submodels e.g., domain prompt, and cross-attention-based fusion all look not new.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "exZHJG66wh", "reviewer": "Reviewer_AGPh", "summary": "To address the Few-shot Test-Time Domain Adaptation task, this paper proposes a new method named L2C. \n\nCompared to previous work, L2C adds an input-space learning branch to the frozen CLIP model, overcoming limitations associated with relying solely on CLIP’s feature space. It introduces a greedy text ensemble and domain-aware fusion, enhancing the model's ability to capture dataset-specific semantics.\n\nExtensive experiments on five large-scale benchmarks demonstrate performance improvements, particularly with less robust backbones like ViT-B/16.", "questions": "I want to disccuss with authors:\n## Performance\n1. As the author mentioned, the proposed L2C shows better performance with the CLIP ViT-B model, but the performance improvement on the CLIP ViT-L model is limited. Therefore, I would like to know if the proposed L2C method is mainly effective for smaller or \"not well-trained\" models, and whether L2C might become less effective as models and data scale up.\n\n## Setting\nTo be honest, Few-shot Test-Time Adaptation (FSTTA) is a new field, so I may have some questions about the settings that I'd like to discuss with the authors.\n\n1. In the traditional TTA setting, once the model is trained on the source domain, most existing works, like TENT [1], make minimal or no changes to the pretrained model, focusing instead on designing methods for the inference stage. However, the L2C method modifies the training process on the source domain (Figure 2). I'm not certain if this approach is permitted.\n\n2. The figure7 shows that the FSTT-DA setting has several target domain, does the FSTTA will occur the catastrophic forgetting like CTTA setting? (CoTTA [2])\n\n3. This method uses prompt guidance and includes multiple domains, which reminds me of PODA [3] and ULDA [4]. I hope the authors can discuss or compare these methods in the paper to enhance its completeness.\n\n## Method\n1. As shown in Figure 1, can I understand that, compared to VDPG, L2C introduces a new trainable network to learn more domain-specific knowledge? If so, is the performance comparison fair, given that L2C has more trainable parameters and a larger overall network?\n\n[1] Tent: Fully Test-time Adaptation by Entropy Minimization\n\n[2] Continual Test-Time Domain Adaptation \n\n[3] PØDA: Prompt-driven Zero-shot Domain Adaptation\n\n[4] Unified Language-driven Zero-shot Domain Adaptation", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The research task Few-shot Test-Time Domain Adaptation seems practical and meaningful.\n\n2. With less robust backbones like ViT-B/16, the proposed method L2C shows significant performance improvement.", "weaknesses": "I have some questions about this paper that need further discussion. Please see them below.\n\nIf the authors can address my concerns, I am willing to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3lGJmwj3qt", "reviewer": "Reviewer_pBcp", "summary": "This paper presents a new test-time CLIP adaptation method that learns domain-specific knowledge complementary to CLIP’s general knowledge. A parallel CPNet is introduced to achieve this by revert attention from CLIP image encoder. A domain prompt is proposed for domain-aware fusion between text and visual features. The experimental results demonstrate the effectiveness of the proposed method on several benchmarks.", "questions": "1. The introduced modules to refine text prompt, i.e., M_c and M_d are not ablatively evaluated. Can you provide ablation study about this?\n2. In Table 3, the performance declines across most of indicators when domain-aware fusion is applied. Why would learning in a domain-aware manner lead to weaker performance?\n3. What is the impact of unlabeled samples when learning the target domain prompt? It is advisable to conduct sensitivity analysis of number of samples.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The experimental results show the improvement over state-of-the-art methods.\n2. This paper is well written and follows a good structure.\n3. The supplementary material is extensive, offering a lot of supplements and support to the main text.", "weaknesses": "1. The motivation behind the proposed method is not clearly explained.  As shown in Figure 1, The method is organized in two branches, fusing domain prompt and complementing dataset-specific knowledge. However, these two branches seem to have very similar functions,  i.e.,  incorporating domain-specific knowledge, which makes the whole pipeline seem redundant.\n2. As shown in experiments, it seems that CPNet plays an important role. However, it is not clear that whether the improvement is due to the incorporation of domain knowledge or the addition of extra network parameters.\n3. Some relevant prompt-based works exploring domain knowledge for mitigating domain shift is not discussed, e.g., HybridPrompt[1], Pro[2].\n    \n    [1] Wu et al., HybridPrompt: Domain-Aware Prompting for Cross-Domain Few-Shot Learning\n\n    [2] Ma et al., ProD: Prompting-to-disentangle Domain Knowledge for Cross-domain Few-shot Image Classification", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LdEWpdl6Wk", "reviewer": "Reviewer_Q3Mk", "summary": "This paper introduces \"Learning to Complement\" (L2C) for Few-shot Test-Time Domain Adaptation, addressing domain shift with minimal unlabeled data. The method adds a parallel module, CPNet, to learn dataset-specific knowledge and complement CLIP’s frozen features. A greedy text ensemble strategy enhances text feature discrimination, improving adaptation by reducing domain bias. The fused text and visual features are guided by domain-specific prompts, leading to superior performance on five benchmarks, especially with smaller backbones like ViT-B/16.", "questions": "*\tThe idea of this work is interesting and effective in FSTT-DA scenario, can the authors provide the source code to facilitate reproduction?\n\n*\tIn line 202, the authors mention that CPNet is introduced to acquire data-specific visual knowledge. However, throughout the paper, CPNet appears to learn dataset-specific knowledge. Could the authors clarify the difference between data-specific visual knowledge and dataset-specific knowledge?\n\n---\n\nI am open to discussing these points with the authors during the response period. If the concerns and questions are adequately addressed, I will consider raising my score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*\tThe organization of this paper is well-structured, making it easy to read and comprehend.\n\n*\tThe insight of learning dataset-specific knowledge is convincing and effective, making a significant contribution to the FSTT-DA community.\n\n*\tExperimental results are impressive, the proposed methods got SOTA in all proposed benchmarks.\n\n*\tOverall, the quality of the paper is commendable. The authors have conducted a thorough ablation study to examine the impact of various components.", "weaknesses": "*\tIn Section 4.2, although the authors highlight the improvement of the text dispersion strategy compared to [a], and demonstrate the effectiveness of the greedy text ensemble strategy for FSTT-DA, the advantages of these improvements over previous methods are not sufficiently discussed.\n\n*\tThe authors did not provide comparisons on the PACS dataset. Extra experiments on smaller datasets could further demonstrate the flexibility of the proposed method.\n\n*\tWhile the paper makes significant progress compared to VDPG and achieves SOTA performance, the authors do not analyze the challenges of FSTT-DA tasks, nor do they provide a discussion on how these challenges are addressed.\n\n[a] E. Cho et al. Distribution-Aware Prompt Tuning for Vision-Language Models. In ICCV, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)