RA-TTA: Retrieval-Augmented Test-Time Adaptation for Vision-Language Models
Authors: Youngjun Lee
Doyoung Kim
Junhyeok Kang
Jihwan Bang
Hwanjun Song
Jae-Gil Lee
Venue: ICLR 2025 Poster (2025)
Keywords: vision-language models
test-time adaptation
retrieval-augmented strategy
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Propose a novel retrieval-augmented test-time adaptation method for vision-language models

Abstract: Vision-language models (VLMs) are known to be susceptible to distribution shifts between pre-training data and test data, and test-time adaptation (TTA) methods for VLMs have been proposed to mitigate the detrimental impact of the distribution shifts.  However, the existing methods solely rely on the internal knowledge encoded within the model parameters, which are constrained to pre-training data.  To complement the limitation of the internal knowledge, we propose **Retrieval-Augmented-TTA (RA-TTA)** for adapting VLMs to test distribution using **external** knowledge obtained from a web-scale image database.  By fully exploiting the bi-modality of VLMs, RA-TTA **adaptively** retrieves proper external images for each test image to refine VLMs' predictions using the retrieved external images, where fine-grained **text descriptions** are leveraged to extend the granularity of external knowledge. Extensive experiments on 17 datasets demonstrate that the proposed RA-TTA outperforms the state-of-the-art methods by 3.01-9.63\% on average.

Published as a conference paper at ICLR 2025
RA-TTA: RETRIEVAL-AUGMENTE
TEST-TIME ADAPTATION FOR VISION-LANGUAGE MODELS
Youngjun Lee', Doyoung Kim', Junhyeok Kang", Jihwan Bang', .Hwnjun Song' Jae-Gii Lee
1 KAIST, 2 LG AI Research
{youngjun, lee, doyol9, jihwan. bang, songhwanjun, jaegiljekaist.ae junhyeck kangflgresearch..a
ABSTRACT
Vision-language models (VLMs) are known be susceptible distribution shifts between pre- training data and test data, and test-time adaptation (TTA) methods for VLMs have been proposed to mitigate the detrimental impact of the distribu- tion shifts. However, the existing methods solely rely on the internal knowledge encoded within the model parameters, which are constrained pre-training data. To complement the limitation of the internal knowledge, we propose Retrieval- Augmented-TTe -TA (RA-TTA) for adapting VLMs test distribution using external knowledge obtained from web-scale image database. By fully exploiting the bi- modality of VLMs, RA-TTA adaptively retrieves proper external images for each test image to refine VLMs' predictions using the retrieved external images, where une-grained text descriptions are leveraged to extend the granularity of external knowledge. Extensive experiments on 17 datasets demonstrate that the proposed RA-TTA outperforms the state-of-the-ae methods by 3.01-9.63% on average.
INTRODUCTION
In recent years, vision-language models (VLMs) pre- are-trained large corpora of image-text pairs have garnered significant attention (Radford et al. 2021 Jia et al. 2021 Li et al. 2022. Sun et al.)
2023 Zhang al. 2024a). When transferring the pre-trained knowledge of VLMs at test time, distribution shifts between pre-training data and test data deteriorate the zero-shot transferability of
VLMs (Bommasani et al. 202I et al. [022 Fang et al. [022 Santurkar al. (023). Thus,
a number of test time adaptation (TTA) methods for VLMs (Shu et al. 2022 Feng et al. 2023 Ma
al. 20233 Zhao al. 2024b; Karmanov al.) 2024 Zhang | 2024b) have been proposed to
mitigate the detrimental impact of the distribution shifts. For adapting to an unlabeled test image, the
existing methods typically rely on the outputs of VLMs on the test image, which are determined by
the pre-trained knowledge encoded within the model parameters. However, this internal knowledge learned from the pre-training data may be insufficient to address unseen test data potentially deviated
by distribution shifts (Agarwal et al. 2021 Menon et al. 2024 Parashar et al. 2024).
Due to the difficulty of updating pre-trained models frequently, retrieval-augmenter generation (RAG) (Gao et al., 2023 7hao et al. 2024a Fan et al. 2024) is proposed in language domains, and it incorporates external knowledge from document database into a query to complement new or focused knowledge absent in the pre-trained knowledge. Thus, in accordance with the philosophy of using external knowledge, we propose a aatrieval-augmeent approach for TTA with VLMs and refer to this approach
(RA-TTA). Figure aa) shows the overall proce- dure RA-TTA: (i) a VLM is requested to provide an answer in response to a test image; (ii) the test image is queried against a web-scale image database retrieve relevant and useful images for the purpose of TTA; (iii) the final answer is adjusted on the fly using both the answer from the VLM and the retrieved external images. The image database usually contains only the images without
eetrieval-aggmented--iee
labels and captions (Caron et al. 2019 Tian et al. 2021 Goyal et al. 2021).
It evident that the external images offer insights into a test image only when properly retrieved. For example, let's consider task that recognizes a vehicle type (c.g., CX-9) from an image. If test image depicts the front and side of vehicle while obscuring the rear, the external images with
"Coresponningg author.

---

Published as a conference paper at ICLR 2025
Test image
Pred
Pred
Target classes 4. Santafe Fi
geneertedd descriptions
2.033
10.5
C2 CX-9
LIM (
Fu Santafe FE has parametric patterns.
Faus CX-9 has boomerang-shuped headlight.
G.2
G.00.
Gx Acura RL
Eag! CX-9 has dual mufflers.
(C)
VLM
Preduetriesul
Offine
(iii) Refine
G 0.3
Online
0.6
Query Retrieval
G 001
Ca Seleet! image
Image
Retrieved images
Distorce
Sta-box
CX-9 9 hheallgght
vector DB
(iil)
embbedding of an external image
image-to-lmage rettieval
embedding of text description
description -used retrieval of R4-77A (Ours)
(a) Overview of the proposed RA-TTA.
(b) Overview of the description- retrieval.
Figure 1: Key idea of RA-TTA: (a) shows the overall procedure of RA-TTA; (b) elaborates on the description- based retrieval approach, where a VLM embeds a test image into the embedding space and identifies its pivotal features (text descriptions), e.g.. boomerang- headlight of CX-9, 9, and the neighboring images to the identified descriptions in the embedding space are retrieved.
pivotal features such as the headlight must be helpful, whereas those with non- visibe features such the taillight irrelevant features such as ski-box would not aid in the recognition. This objective aligns with RAG for retrieving documents that contain critical information relevant to given query. However, naive image-to-ings with the pivotal features. The primary reason is that an image, similar to lengthy document, often encompasses multiple diverse semantics, c.g., a vehicle image with headlight and ski-box in Figure (b). That is, images may be quite intricate to be encapsulated by single embeddings. Recall that, RAG, a document divided into multiple chunks, with each chunk intended to contain a specific piece of information (Lewis et al. 2020 Wang et al. 2024a). This chunking enables the capture of more relevant and coherent information for a query. On the other hand, because precise image segmentation is very costly and challenging, applying image segmentation to achieve an effect analogous to document chunking for images is infeasible.
similarity search (Iscen al. 2023) often fails retrieve the images
In this paper, to retrieve proper external images and ultimately improve the zero-shot transferability of a VLM at test time, we propose a description- basodd retrieval approach that fully leverages the bi-modality of VLMs. Figure [Tb) elaborates
its overall procedure: (i) Multiple (e.g., 20) fine- grained visual features for each target class are extracted the form of text descriptions, such as "Mazda CX-9 has bbomerang-shaper headlight," by LLMs. These text descriptions are extracted offline prior to test time. (ii) For each test image given online, text descriptions relevant to a test image are selected by image-to-text similarity search on the embedding space of VLM, which we regard as the pivotal features of the test image. (iii) External images aligned with these selected text descriptions are retrieved by text-to-image similarity search on the same embedding space. We contend that image retrieval through text descriptions is comparable to document chunking, as it divides images into multiple semantic chunks for effective retrieval. In Figure [Tb),, due to the text descriptions on the headlight of CX-9, its headlights are desirably dominant in the retrieved images. contrast, arbitrary SUV images with a ski-box could retrieved a naive image-to- image similarity search were employed. Moreover, we also propose description based adaptation approach that exploits the selected text descriptions for the adaptation with the retrieved images.
Overall, the proposed RA-TTA equipped with the description- retrieval and adaptation signifi- cantly outperforms the state-of-the-an TTA methods on 17 datasets, including the standard transfer learning and natural distribution shift benchmarks. According our extensive evaluation, the im- pact of external images was higher when the strength of distribution shifts became higher, and the advantages of the description-basel retrieval became more pronounced when test data exhibited
more complex semantics (see $ 4.2 and $ 4.6).
RELATED WORK AND PRELIMINARY
2.1 VISION-LANGUAGE MODELS
Vision models (VLMs), such as CLIP (Raffrrd et al., 2021), ALIGN (Ja et al. 2021) and LongCLIP (Zhang et al. 2024a). are pre-trained to align images with corresponding text descriptions, thereby understanding the relationships between arbitrary image-text pairs. These
2

---

Published as a conference paper at ICLR 2025
models have demonstrated excellent zero-shot transferability for image classification tasks, where classes are represented by text prompts, and images are classified based on the similarity to these prompts (Radford et al. (02T). Previous approaches employ coarse text prompts, such as "a photo of Chevrolet Impala," " which do not fully utilize the contextual richness of language. Recent ad- vancements shift towards incorporating fine-grained visual details in prompts, such as "Chevro- let Impala has sharp and muscular styling." to enhance representational capacity. For example, VisDesc (Menon & Vondrick, (022)) and CuPL (Pratt al. 2023) employ class-specifie, LLM- generated text descriptions as the prompts for VLMs. Waffee (Roth et al. (023) proposes a novel prompting technique for VLMs to improve the utility of the text descriptions by analyzing the effect of LLM-generated text descriptions. Following these studies, our research further explores the utility of fine-grained text descriptions to improve VLM efficacy.
2.2 TEST-TIME ADAPTATION FOR VISION-LANGUAGE MODELS
When transferring the zero-shot capabilities of VLMs, the distribution shift between pre-training data and test data is the main obstacle. Test-time adaptation (TTA) methods for VLMs have been proposed adapt VLMs the distribution shift. These methods adapt an input test image on the fly without any training requirements. TPT (Shu et al. 2022) uses data augmentation to enrich the test image, filters out unreliable augmented images based on prediction entropy, and then updates learnable prompts by minimizing the entropy of the reliable predictions. DiffTPT (Feng et al.) 2023) enhances TPT by augmenting an input image with generated images from a pre-trained diffusion model. C-TPT (Yoon et al., 2024) also enhances TPT by calibrating the prediction uncertainty. RLCF (Zhao et al.) (024b) introduces a CLIP based loss to avoid the pitfall of the entropy minimization. Because only single unlabeled test image is available, existing methods leverage the outputs of VLMs on the test image for the adaptation. However, they solely rely on the internal knowledge encoded in the VLM parameters, which are constrained to the pre-training data (Agarwal et al. 2021| 1] Menon et al.) 2024; Parashar et al., 2024). In contrast, we leverage external knowledge retrieved from an external image database, which could have new focused knowledge that is not present in the internal knowledge, thereby providing informative knowledge for unseen test images.
2.3 RETRIEVAL-AUGMENTED STRATEGY FOR VISION-LANGUAGE MODELS
Since not all knowledge can be encoded within VLM parameters, retrieval- augmented strategies have been adopted as a solution, which can leverage external knowledge. These strategies can be broadly categorized into training-based and training-free methods. Because training- methods exceed the scope of this work due to the necessity for extensive training and/or additional param- eters before test time, we focus on training-free ffree methods here. Training-free methods, including SuS-X (Udandarao et al.) 2002),, Neural Priming (Wallingford et al. (023), and Ret-Adapter (Ming) & Li 2024)), retrieve training datasets and use the retrieved datasets to adapt VLMs without back- propagation based on few-shot adaptation methods. However, since they conduct the retrieval before test time, their retrieved images are static; thus, they cannot adaptively cope with unseen test im- ages deviated by distribution shifts. On the other hand, we retrieve external images adaptive to test images on the fly and can flexibly handle distribution shifts. See Appendices A.T and A.2 for the details of training- based and training-free methods, respectively.
2.4 BACKGROUND ON THE CLIP MODEL
The contrastive language-image pre-trained (CLIP) model (Radford et al. 2021 consists of an image encoder f()) and text encoder g()), each of which maps an image x E 3xxxxx and a text prompt t into d-dimensional shared embedding space, respectively. By calculating cosine similarity between the mapped embeddings e' e Rd for the image and e" E Fdd for the text prompt, CLIP produces the image-text alignment score galign cos(e', ee"), which measures semantic similarity between the image v and the text prompt f. For text prompts, text descriptions that represent the fine-grained visual features of objects can be utilized. For each class c, multiple L. text descriptions (fo.)) are generated and mapped to the embedding space. The embeddings of these descriptions for each class are averaged, and the averaged embedding हे¹ is used as prototype for the class c. Using the prototypes, the prediction probability for an image x being class can be calculated as exp(cos(e' )/T)
p(c|=) = D exp(cos(e) o )/T)*
(1)
where C' the number of classes and is a temperature parameter.
3

---

Published as a conference paper at ICLR 2025
Test image
Step I: Deseription- Based Retrieval (53.2)
Step II: Description Based Adaptation ($3.3)
Imagee to Description Selection ($3)2.1)
Text descriptions
Description to Image Retrieval ($3.2.2)
Retrieved images
Description Based Relevance (53.3.1)
Relevance Knowlcdge Fusion
Augmented prediction
(53.3.2)
D3
Initial prediction
CB VLM
Figure 2: Overall procedure of RA-TTA.
RA-TTA: RETRIEVAL- TEST-TIME ADAPTATION FOR VLMS
3.1 PROBLEM STATEMENT AND OVERVIEW
Problem Statement. RA-TTA aims to adapt VLM (e.g., CLIP) to classify test image 2test as its label y by leveraging a set of N external image embeddings R { [k: E L} retrieved from an image vector database D = {ef Y/=1 where L is the index set of the N retrieved embeddings, D is the database size, and e IRd the image embedding computed by image encoder f(+) for an external image z In line with TPT (Shu et al. 2022), the training data unavailable, the training pipeline cannot be modified, and the adaptation conducted online single image.
Overview. Figure 2 illustrates the retrievalland-adaptaiie procedure of RA-TTA. In Step I, for a test image, description-based retrieval selects relevant text descriptions and retrieves external im- ages aligned with the selected text descriptions ($ 3.2). In Step II, description-bust calculates relevance scores for the retrieved images with respect the test image, and the initial prediction refined by the relevance scores to produce the augmented prediction ($ 3.3). Its pseu- docode presented in Appendix B
adaptation
3.2 STEP I: DESCRIPTION- BASED RETRIEVAL
Figure details the procedure of the description-basee retrieval step. The retrieved images should include the visible features of the target object in a test image 2e which are pivotal for recognizing the corresponding label y. To achieve this goal, multiple fine-grained visual features of each target class are extracted offline by LLMs in the form of text descriptions, which can be represented as T (t_((te,)f_ Based on the descriptions in T, we present description- based retrieval approach to retrieve images that include the pivotal features of 2e from D. The bi-modality of VLMs is fully exploited first through image-to-text search (Left of Figure 3 3.2.1 and then through rext-to-image
search (Right of Figure 3.2.2).
3.2.1 IMAGE-TO-DESCRIPPUI SELECTION (LEFT OF FIGURE
Robust Image-Text Alignment. The image-text alignment scores between test image ztest and the text descriptions in T are calculated for selecting relevant text descriptions. Misleading descriptions may be inaccurately aligned with the test image, mainly due to non-target objects, likely leading to the retrieval of irrelevant images. To select only relevant descriptions without misleading ones, enable VLM analyze test image from multiple perspectives, thereby understanding comprchensively. Specifically, the standard augmentation A(-), including random resized cropping and random flipping, is applied to test image test obtainiin the augmented images (= IM where M is the number of augmented images by A(.) and the original test image test is denoted Zp. Subsequently, the alignment scores of each description tæ, are calculated with the images in align A. That is, = ( I'm AJ,, where 5m,t,, = cossee e".;), e/ - =(xx), and =", Although misleading descriptions may result in inaccurately elevated alignment scores for a few certain augmentations, the majority of augmentations overlook these misleading descriptions; the scores of the misleading descriptions are generally low, with only a few exceptions. Thus, to avoid selecting the misleading descriptions (i.c., features), the p-percentile of the scores in St,.: is used as the representative image-text alignment score for the description fc,,, which is represented by aligo
align
= percentile, (St..) (S..,, percentile, (($milgn,) 1 mm)))
(2)
where percentile, (+) returns the 100p-th (p E [0, 1)) percentile of set. We choose = 0.75 or 0.90, with which the function returns the top 75% (i.c., third quartile (Q3)) or 90% value of S
For simplicity, we refer to e' and eT as image and text description, respectively, omitting the term "emhedding" hereafter.
4

---

Published as a conference paper at ICLR 2025
Deseription Grouping
Test image
f fetteey Leafett Group Se
Group S:
Selected text descriptions 3xp (*
Prototype-Baceh Retrieval
Robust Image-Text Aligument
A T/ext Text/ lmsgee snbedding Text pronotype Build text prototypes from text emnherddnngs
Target object
Nanddarget
Misleading description
Ofline
Description Selection
(irrelevant)
Hyundai Palisade
Three-Row SUV.
2. Retrieve nenrest images from text prototypes
Relevant description
percrntile e
boomerang
-shaped headlight.
Robust
Irrelevant description
Aligmment SCOTDS of Fee
aligument scores 87
Rg.
Rg (xg x
CX-9 has dual mufflers
RRetrieeed Images R (Rg. Rg, "
Figure 3: Description-base based retrieval. Left: Image-to-description selection. A test image aug- mented to produce multiple views, and the image-text alignment scores between the augmented images and the text descriptions are calculated. The top- descriptions are selected based on the third quartile (Q3) value of the alignment scores of each description to avoid misleading and irrel- evant descriptions. Right: Description-tor -o-imagg retrieval. The selected descriptions are grouped by their corresponding class, and those in each group are embedded and averaged to build text prototype. Finally, the external images closest to the prototypes are retrieved.
Consequently, the robust alignment scores of the test with the text descriptions in T become align
ST = ( ET-
Description Selection. Since the high image-text alignment score of an image-text pair represents the high correspondence between them, the top KD descriptions in Tk, are selected as the pivotal features of g.ees using the alignment scores in Sy, which can represented by
Tkp {he salign E Top-K(ST;; Kn)},
where Top-K( :;K) is function that selects the K highest values from set.
3.2.2 DESCRIPTION-TO-IMAGE RETRIEVAL (RIGIT OF FIGURE 3
Description Grouping, Before conducting the retrieval, we arrange the selected descriptions be- cause those of different classes could be mixed, where the corresponding classes of the descriptions in TKp are denoted as C= {cl te,t Tkg The text descriptions in Txe are grouped by their corre- sponding class (= where Ge includes those of class and represents the semantics of the class e with the features indicated by the belonging descriptions.
Prototype-Basel Retrieval. Based on the descriptions in Ge. we retrieve the external images that can be interpreted as representing the semantics of G. and attach as their pseudo label. First, the embeddings of the descriptions in Ge are simply averaged to build a text prototype ei, that serves as the representative embedding of the semantics in Ge- Then, the alignment scores between eff and the embeddings of external images in D= y are calculated, and the top Ks external images are retrieved using the alignment scores, which can be represented as
Rg. [x &glignn E Top-K(SD: Ks)};
where Sp { alignn D 1--1 and 8 aligo cos(e)". cτ. The same procedure is applied separately to each group G.. Since the same number (Ks) of images are retrieved for each group, the total number of retrieved images is N = Ks x |C]. Finally, R = (RR. leec is fed to the next step.
3.3 STEP II: DESCRIPTION -AAIID BASED ADAPTATION
By leveraging the set of retrieved external images, R= (R leec. the VLM''s initial prediction for test image fest is adjusted by an additional prediction based R, where the initial prediction p is conducted as in Eq. (T). Since each set external images, Rg., is associated with its pseudo label c, first caleulate the relevance score from test to the set of external images, Rg. which is then used to determine the probability of test belonging to the corresponding class c. Interestingly, calculate this relevance score, the selected text descriptions, Tk, in Eq. are exploited again
K -enaerest neighbor search can he performed very efficiently with conventional vector databases.
5

---

Published as a conference paper at ICLR 2025
( 3.3.1), where our description-ben adaption kicks in. Subsequently, both initial and additional predictions are fused to produce the augmented prediction ($ (.3.2). In short, this adaptation step can be viewed as ensemble procedure in which each retrieved external image casts a vote for the label of y sest based on its pseudo label.
3.3.1 DUSCRIPTION-BASE RELEVANCE
Semantic Gap. Since we use the pivotal features (i.e., text descriptions) of a test image to retrieve external images, it is logical to employ these descriptions to assess the relevance between test image and an external image. We intend to measure the disparity between two images concern- ing specific semantic aspect. We specifically introduce the semantic gap between two images, which pertains to the specific semantic aspect, in Definition 3.1 As illustrated in Figure 4 it is the difference between the distances from each image embedding to the embedding representing the semantic aspect of interest. For instance, when examining the front grille of vehicles, the semantic gap would signify the disparity between two vehicle images in terms of the clarity with which they depict the front grille.
Semantic gap
Target semantics
d2 d
di
d2
Figure 4: Semantic gap.
Definition 3.1 (SEMANTIC GAP). Given the descriptions in Ge, the semantic gap between two images z; and žj is the absolute difference between their cosine distances with the text prototype eg. of the descriptions in G. in the embedding space of a VLM,
(6) where e/ f c = fzz)). and eg. is the text prototype of the descriptions in Ge computed by averaging their embeddings.
gap(a.; Tj. Go) = [(1 - cos(e o).)) - (1 - -os(ee e)).)
Semantic Relevance Computation. Through the concept of the semantic gap, now derive the relevance score of a test image to the set of external images for a specific class. To enhance its reliability and robustness, we devise two simple yet effective techniques. First, we reuse the set A of augmented images for the test image z test instead of considering it only. As result, the relevance score is aggregated from the pair-wise semantic gaps between (M + 1) augmented images and Kş external images. Second, the significance of each image within each set, i.e., A or Rg., is factored into the relevance score. If an image located closer to the text prototype of the corresponding set in the embedding space of a VLM, the image should be considered more importantly in aggregating the pairwise semantic gaps. Mathematically, matrix Cg, ((M+1)x K) has the pair-wise semantic gaps between 2; E A and zn Rg.,, which can be represented
Cg. gaappp; 2f, Ge) IEIEAGERO.. (M
For the significance of each image, we additionally define two weight vectors U RM++1) and V E Rks; U contains normalized weights from the cosine similarity between the embedding of Iz E A and the prototype eº of A, where E is calculated by averaging the embeddings of the descriptions in TK, (see Eq. (4);: V contains normalized weights from the cosine similarity between the embedding of xy E Rg. and of (see Eq. (6). Considering these weight vectors, the pair-wise semantic gaps in Cg. are aggregated by the optimal transport (OT) framework (Villani et al. 2009), which can measure the distance between the two sets of weighted data points. Then, the relevance score to a class is formulated by
rel CG
OT U,)++
where O'aist (s:.-)) returns the aggregated distance based on the OT framework. The same proce- dure repeats for each group RRg (i.e., for each class), resulting in RS= {sg) a CEC). We further provide the details of OT in Appendix C
3.3.2 KNOWLEDGE FUSION FOR ADAPTATION
We derive the retrieval- based prediction P for test using the relevance scores in RS, and the pre- diction probability for the image x test of being a class can be represented as
exp(st))/1
c
p test)
Ecec exp(>)!)/T)
(9)
otherwise

---

Published as a conference paper at ICLR 2025
Finally, we fuse the retrieval- -ased prediction P and the initial prediction p using the entropy of each prediction, leading to the augmented prediction pang for the image xresst
p [c] x t a x p(c] | (eest) + (1 - a) x ((cl Here, a= exp(1/(=HH
(10)
where H (or H) is the entropy of P (or P).
4 EVALUATION
4.1 EXPERIMENT SETTING
Datasets. To evaluate the zero-shot transferabilityy we use standard transfer learning and natural distribution shift benchmarks, including 17 datasets that span wide range of image classifica-
tion tasks: ImageNet (Deng et a 2009). Flowers (Nilsback & Zisserman (2008). DTD (Cimpoi
ct al.
2014). Oxford pets (Parkhi et al. 2012). Stanford cars (Krause et al. (013). UCF101 (Soomro
et al.
2012). Ca (Fei-Fei et al. 2004). J. Food101 (Bossard et 2014)) SUN397 (Xiao al.
2010
FGVC aircraft Majj et al. 2013). 3, RESISC45 (Cheng et al. 2017). Caltech256 (Griffin et al,
2007
and CUB200 (Wah al. 201 transfer learning benchmarks, and natural distribution shift
benchmarks, including ImageNet adversarial (Hendrycks et al. 2021b), ImageNet V2 (Recht et al.
2019), ImageNet rendition (Hendryeks et al.
2021a), and ImageNet sketch (Wang al. 2019). The
details of each dataset can be found in Appendix
Compared Methods. We compare RA-TTA against four kinds of VLM adaptation methods that do not require training procedures before test time: (I) zero-shot CLIP baselines, (2) tuning-based methods, (3) text description-basee methods, and (4) retrieval-based methods. For zero-shot CLIP baselines, we include two zero-shot baselines of CLIP using default prompt, "a "ahto of a (class
name)." " and the ensemble of 80 hand-crafted prompts (Radford et al. (021). For tuning-batt based methods, we use TPT (Shu et al., 2022), C-TPT (Yoon et al.] [022)), and R.CF et al. 2024b), which update parameters through back-propagation from VLM outputs for a test image. We use VisDesc (Menon & Vondrick 2022), WaffleCLIP (Roth et al. 2023), ) and CuPL (Pratt et al. 2023)
as text description-base methods that leverage the contextual richness of language. For retrieval-
based methods, we include SuS-X-LC (Udandarao al., 2023) and Neural Priming (Wallinggor
et al. 2023). which retrieve external images from an image database offline.
Web-Scale External Image Database Construction. We construct the database for retrieval-based based methods, including SuS-X-LC, Neural Priming, and our proposed RA-TTA, with the following objectives: ensuring relevance downstream tasks and preserving the diversity and noisy nature of a web-scale database. Thus, we explore I. LAION2B (Schuhmann et al., (022) dataset that consists of 2B web-scale image-caption pairs for the construction. Following keyword-basee fast string matching on LAION2B (Wallingford et al. 2023 Parashar et al. 2024)) we downloaded images whose text captions contain the target classes and used the downloaded images as external images. This approach offers several advantages: it ensures the relevance and preserves the nature of web- scale database (Parashar et al., 2024), not to mention reproducibility. While using all images in LAION2B may seem appealing. it requires 100 TB of storage. We conjecture that our database achieves a practical balance between leveraging web-scale images and the complexity of evaluation. The details for the image database construction can be found in Appendix D.2 Implementation Details. We implement RA-TTA using the CLIP-B/IC (Radford et al. 2021 model as VLM. For generating text descriptions, we use GPT-3.5 Turbo (Ouyang al. 2022
and CuPL (Pratt et al. 2-tyle hand-written templates. We set the augmentation size M to 100.
We configure K for Top-K operations at KD 20 and Kş = 20. We use temperature param- eter 7 of 0.01, which is the default scale value of CLIP. We use the aforementioned configurations across all datasets because dataset-specific configurations are not preferred in TTA. We set p = 0.75 for transfer learning datasets except ImageNet and = 0.90 for ImageNet-based datasets. Further implementation details on the compared methods can be found in Appendix D.3] All implementa- tions are conducted using PyTorch 2.3.0 on an NVIDIA RTX 4090. The source code of RA-TTA is available at httttt:://gtthiiie com www.kow.iii.e wwt.:taiatttt..ttt -masstddmllllll dmw.b/ TTA
4.2
MAIN RESULTS
Table shows a comparative analysis of RA-TTA against the compared methods for 13 transfer learning benchmarks, evaluating how much the given methods can improve the zero-shot capability
7

---

Published as a conference paper at ICLR 2025
Table 1: Overall results for test-time adaptation (or transfer learning). We report the top-1 accuracy (%) for 13 standard transfer learning datasets. The "Avg." column indicates the average accuracy of 13 datasets. The best results are in bold, and the second best results are underlined.
PGV alreraff
Stanford cars
Floward02
Oxford pets
Callech101
RESISC45
Callech256
UCFF1
F0
SUN397
CUR200
Arg. (13)
IN-JK
DTD
CLIP
66.76
67.19
44.50
88.14
65.27
64.92
92.78
85.40
62,55
24.60
55.70
82. $2.80
58.08
66.05
Ensemble
68.37
65.85
45.21
88,20
66.34
67.41
93.77
85.41
65.79
24.39
58.35
85.8
58.61
67.19
TPT
69.08
69.18
47.04
87.44
66.55
68.04
93.79
86.34
65.32
23.31
56.84
85.37
60.11
67.57
C-TPT
68.32
45.27
88.25
65.48
65.50
93.39
84,95
64.55
24.39
25
66.90
RLCF
68.61
67.72
46.40
86,73
66.51
66.98
93.83
86.09
64,92
23,43
56,89
85. 18
57.91
67.02
VisDese
69.09
71.86
50.41
88.55
65 48
69.52
94.81
86.43
68.25
25.59
57.81
88.17
60.13
68.93
WaffeeLLPP
69.05
72.59
48.33
89,79
64.60
69.13
94.61
86.85
67.17
25.25
63.31
88. 10
59.83
69.12
CaPl.
69.78
75.92
58.22
91.47
66.92
67.80
94.24
86. 39
67.38
28.98
65.
88.07
60.18
70.81
Su8-X-LC
669.45
76.23
59.23
91.83
67.55
67.12
93.78
86. 13
67.78
29.41
65.22
88.75
59.12
70.89
Noural Priming
69.38
73.22
55.98
89,76
66. 13
68.02
94.71
87.01
67,86
27.32
63. 11
88. 50
57.14 14
69. 86
RA-TTA (Ours)
70.58
78.65
60.99
92.78
70.11
73.28
94.84
87.10
70.38
32.34
66.95
89,50
62.73
73.09
of VLMs. RA-TTA outperforms all existing methods, with an average improvement of 3.01-9.633 over the baselines, demonstrating its effectiveness in enhancing the knowledge of VLMs at test time. The tuning-based methods (Shu et al.] 2022 Yoon et al. 2024 Zhao et al., 2024b) cannot leverage knowledge from an external database and the contextual information from textual modality, thus fail- ing to adapt. On the other hand, RA-TTA achieves the best performance by leveraging the external knowledge and the textual information, even outperforming text description- based methods (Menon & Vondrick 2022 Pratt al. 2023 Roth et al. 2023) that adopt fine- tne-grained text descriptions but cannot use external knowledge unlike RA-TTA.
RA-TTA is particularly effective for specialized domain dataset, like RESISC45 for satellite im- ages, and fine-graine tne-grained datasets, such as Flowers 102, Stanford Cars, FGVC Aircraft, and CUB200 because RA-TTA retrieves a customized set of external images for each test image by identifying its pivotal features through fine-grained descriptions. In contrast, previous retrieval-based methods, including SuS-X-LC (Udandarao et al. 2023) and Neural Priming ( a fixed set of external images for a given class, which limits the granularity of extractable knowledge and ultimately results in inferior performance compared to RA-TTA.
(Wallingford et al., (022)) retrieve
Moreover, Table 2 illustrates the perfor- mance of RA-TTA for ImageNet vari- ants (Hendrycks et al. 2021b Recht et al. Hendrycks et al., 2021a Wang et al. 2019) which are typically used to evaluate the VLMs' robustness to natural distribution shifts, such artistic rendition and black and white sketches (Radford et al.) [021]] Shu al. (0022). RA-TTA outperforms the compared methods in terms of the average accuracy, demonstrating that RA-TTA can cope with natural distribution shifts effec- tively. Notably, the description- based meth- ods (e.g., CuPL (Pratt et al., 2023)) that em- ploy only fine- tne-grained descriptions without help from external knowledge show minor improvements compared to Ensemble (Radford et al. 2021). However, RA-TTA improves the accu- racy by 15, 15.49% over Ensemble, indicating that the effectiveness of RA-TTA arises not only from the use of fine-grained descriptions but also from the sophisticated integration of external knowledge.
Performance for natural distribution shifts. We report the top-1 accuracy (%) for four Im- ageNet variants. The "Avg." column indicates the average accuracy of 4 datasets.
Table 2:
IN-A
IN-V2
IN-R
IN-K
Avg. (4)
CLIIP
47.51
60.80
73.98
46.
57.12
Ensemble
50.04
61.89
77.58
48.29
$9.45
TPT
54.39
63.48
77.27
47.95
60.77
CTTPT
50.28
62.47
75.68
47.42
58.96
RLCF
56.52
63.37
77.04
48.09
61.26
VisDesc
50.17
62.76
75.25
48.25
59.11
WaMeeLIP
50.51
62.68
75.81
48.73
$9.43
CaPL.
50.23
63.00
78.16
49.60
60.25
SuS X-LC
49,91
63.22
77.82
49.18
60.03
Neural Priming
49.68
62.79
76.70
49.03
$9.55
RA-TTA (Ours)
59.21
64.16
79.68
50.83
63.47
The standard deviations for Tables I and 2 are presented in Appendix E. owing to the lack of space.
4.3 ABLATION STUDY
In Table we conduct ablation studies to understand the impact of the description based retrieval, the description-basel adaptation, and the image weighting scheme. (1) If the cesccrpttion-baeed
8

---

Published as a conference paper at ICLR 2025
retrieval is disabled, the retrieval procedure in Section 3.2 is simply replaced by image-to srrge-to-image similarity search. (2) If the description based adaptation is disabled, the semantic gap in Defini- tion B.I is simply replaced by image-to-image cosine similarity. (3) If the image weighting scheme is disabled, the weight vectors I and V in Eq. (8) are not used for aggregating the pair-wise dis- tances. We prepare three different variants, denoted by Var. 1, Var. 2, and Var. 3 in Table B by disabling three, two, and one component(s), re- retrieval
Table 3: Ablation studies. We report the top-1 accuracy (%) on the FGVC air- craft dataset, where the benefit of RA- TTA significant. Description-bant based re- trieval, description- adaptation, and im- age weighting are disabled separately.
spectively. While the description-bed is effective by itself (compare between Var. and Var. 2), its effect is clearly boosted when combined with the description-based adaptation (compare be- tween Var. and Var. 3). addition, the image weighting scheme adds a slight improvement (com- pare between Var. 3 and RA-TTA), Overall, all three components are shown to be effective, supporting the superior performance of RA-TTA. The results for other datasets be found in Appendix E..2 Furthermore, refer to Appendices E.3 an E.4 about
Retrieval Adaptation
Weighting Accuracy
Var.
29.39
Var. 2
30.91
Var. 3
X
31.96
RA-TTA
5 32.34
the results of changing the LLM for generating descriptions and varying the database size.
4.4 HYPERPARAMETER SENSITIVITY ANALYSIS
We conduct sensitivity analyses four crucial hyperparameters which influence the performance of RA-TTA. The results in Figure 5 report the averaged accuracy on 13 standard transfer learning benchmarks. Refer to Appendix B.5 for more results.
Effect of Augmentation Size. For mitigating the interfering effect of misleading descriptions, RA-TTA conducts augmentations analyze test image from multiple perspectives. We analyze the impact of the augmentation size M on the accuracy. As shown in Figure (5aa), increasing the augmentation size improves the accuracy, with a plateau observed around 25. This result suggests that while more augmentations are generally beneficial, the benefits diminish beyond a certain point (around 25), where the accuracy stabilizes.
Effect of KD for Description Selection. RA-TTA selects the top KD descriptions as the pivotal features of a test image. Figure (b) demonstrates the accuracy concerning the KD. RA-TTA performs best at KD 20, after which the accuracy declines sharply. This result can be un- derstood as increased KD may lead to selecting irrelevant or misleading descriptions, negatively impacting the following retrieval and adaptation.
Effect of Kg for Image Retrieval. RA-TTA retrieves the top Kg images using the selected text descriptions. Figure [i(c) illustrates the effect of Kg. As for KD, can infer that the accuracy drops because outlier images can be included as Ks increases. However, the accuracy remains stable across different values of Ks, with significant drop even Ks increases. This result suggests that when aggregating the semantic gap, representing the significance weights of retrieved images makes RA-TTA robust the retrieved outlier images.
Effect of Alignment Score Percentile. Figure 5 d) shows the accuracy based on the percentile value p for rejecting misleading descriptions. The best accuracy is achieved around 0.75, the third quartile value, which is optimal for selecting relevant descriptions while rejecting misleading descriptions. When the value is set too high (e.g., 1.0), the accuracy degrades, possibly due the inclusion of misleading descriptions.
4.5 EFFICIENCY ANALYSIS
In the context of TTA, the effi- ciency is indeed critical factor to consider. We provide an anal- ysis of computational complexity, including inference time and GPU peak memory. Based on Table the inference time of RA-TTA IS comparable to that of TPT, the representative TTA method. This is because FAISS [Johnson et al. 2021), an advanced and efficient search engine typically used for
Table 4: GPU inference time per sample (s/sample)
FGVC aircraft
Stanford cars
RESISC45
Avg. (3)
TPT RA-ITA (Ours)
0,103
0.155
0.95
0.118
0.113
0.117
0.121
0.117
which shows the GPU inference time per sample (i.e., s/sample),
RAG-based approaches, enables fast nearest-neighhor search. The analysis of GPU peak memory is provided in Appendix E.6 owing to the lack of space.
9

---

Published as a conference paper at ICLR 2025
73
73
Accuracy (%)
73
Accuracy (%)
73
Accuracy (%)
72
71
Accurscy
72
72-
71
S0S-X-LC
5.25.00
100
150
200
20 40
60
80
160
520 60 80
160
0.10.25
0.5 0.75 0.91.
Ks
(a) Augmentation size. (b) selected descriptions. (c) # retrieved images.
(d) Score percentile.
Figure 5: Effects of the four hyperparameters on the accuracy. The green dashed line denotes the accuracy of SuS-X-LC as a strong baseline.
4.6 QUALITATIVE ANALYSIS
In Figure we present illustrative examples of the Descrrption-bassee based retrieval of RA-TTA on the Stanford cars (Krause al. 2013) and RESISC45[Cheng et al. (2017) datasets. We observe in Fig- ure 6 that description-bass of test images because the retrieval procedure focuses specific piece of information, i.e., piv- otal feature, without being distracted by other irrelevant information. Additional visualizations are provided in Appendix E.7
retrieval allows us to retrieve external images with the pivotal features
Test image
Selected text descriptions
Retrieved images
specific features such as the distinetive grille with the Acura logo, the swept-back headlights...
12T
T21
GT: 2012 Acura RL Sedan
exterior styling, which includes sleek, low-slung profile with four doors...
12T
T2I
GT: 2012 Porsche Panamera Sedan
road bridge from top-down perspective, clearly revenling the intersections...
12T
T21
GT: Overload
...prominett structures such as cooling towers, smokestacks, and...
T21
GT: Thermal power plant Pivotal features
12T: Image-to-text search
T21: Text-to-image scarch
Figure 6: Examples of the images retrieved by our and RESISC45 datasets. The results of the description rescription-based based retrieval on the Stanford cars selection and the description-to- image retrieval are shown individually. A pivotal feature is indicated by red box in the images.
5 CONCLUSION
In this paper, we propose RA-TTA (Retrieval-Ae Augmeette Test-Time Adaptation), designed to lever- age the external knowledge from a web-scale image database on-the-fly. By fully exploiting the text descriptions, which represent fine-grained visual details, the description- retrieve external images with the pivotal features of a test image, and the description-basee adapta- tion allows a VLM to adapt to the test image using the inherent semantics of the retrieved images. Our extensive experimental results demonstrate superior zero-shot transferability compared to the state-of-the-ar methods across 17 datasets. Overall, we believe that our work sheds light on the effectiveness of external knowledge for the zero-shot transfer of VLMs..
retrieval ensures to
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This work adheres to the ethical guidelines and principles ICLR, ensuring that no harm was done to individuals, groups, environments during the study. A web-scale external image database may contain problematic images is constructed carelessly. Even though we use a previous version of LAION2B the data source, do not include the known problematic images, because normal, unproblematic images are selected by querying the class names of our downstream tasks. Since cleaned version of LAION2B has been released just a month ago, further strengthen our ethical standard, we will reconstruct our external image database using the cleaned version.
REPRODUCIBILITY STATEMENT
In accordance with ICLR's reproducibility guidelines, we provide a GitHub repository containing the code used for experiments: httppa://gtttub...mmm kmm ab/ RA-TTA,
ACKNOWLEDGEMENT
This work supported by Institute of Information & Communications Technology Planning & Evaluation (IIT)) grant funded by the Korea government (MSIT) (No. RS-2020-1100862, DB4DL: High-Usability and Performance In-Memory Distributed DBMS for Deep Learning, 50% and No. RS-2022-11220157 Robust, Fair, Extensible Data-Centric Continual Learning, 50%).
REFERENCES
Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications. arXiv preprint arXiv:2108.02815, 2021.
Rishi Bommasani et al. On the Opportunities and Risks Foundation Models. arXiv preprint vrXiv:2108.07258, 2021.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. 1o0d-ingg Discriminative Com- ponents with Random Forests. In ECCV, pp. 446-461, 2014.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised Pre-training of Image Features on Non-curated Data. In ICCV, pp. 2959-2968, 2019.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing Web- Scale Image-Text Pre-Training Recognize Long-Tail Tail Visual Concepts. In CVPR, 2021.
Xi Chen al. PaLI: Jointly-Saaler Multilingual Language-Iman Model. In ICLR, 2023.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote Sensing Image Scene Classification: Bench- mark and State of the Art. Proceedings of the IEEE, 1855119:1855-188, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing Textures in the Wild. In CVPR, pp. 3606-3613, 2014.
Marco Cuturi. Sinkhorn Distances: Lightspeed Computation of Optimal Transport. In NeurIPS, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-scale Hierarchical Image Database. In CVPR, pp. 248 203-255,222 2009.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang. Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models. In KDD, 2024.
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP). In ICML,, pp. 6216-6234, 2022.

---

Published as a conference paper at ICLR 2025
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning Generative Visual Models from Few Train- ing Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. In CVPR Workshop, pp.
2004.
Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning. In ICCV, pp. 2704-2714, 2023.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-Augmenteé Generation for Large Language Models: A Survey, arXiv preprint arXiv:2312. 2023.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised Pretraining of Visual Features in the Wild, arXiv preprint arXiv:2103.01988, 2021.
Gregory Griffin, Alex Holub, Pietro Perona, et al. Caltech-256 Object Category Dataset. Technical Report ENS-TR-2007-001 California Institute of Technology, 2007.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The Many Faces of Robustness: A Critical Analysis of Out-of-distribution Generalization. In ICCV, pp. 8340-8349, 2021a.
Dan Hendrycks. Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural Adversarial Examples. In CVPR, pp. 15262-15271, 2021b.
Ahmet Iscen, Mathilde Caron, Alireza Fathi, and Cordelia Schmid. Retrieval-Enhanced Contrastive Vixion-Text Models. ICLR, 2023.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Language Representation Learn- ing with Noisy Text Supervision. In ICML, pp. 4904-4916, 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-Scale Similarity Search with GPUs. IEEE Transactions on Big Data, 7:533-547, 0473-57,211 2021.
Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb Saddik, and Eric Xing. Efficient Test-Time Adaptation of Vision-Language Models. In CVPR, 2024.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D Object Representations for Fine- grained Categorization. In ICCV Workshop, pp. 554-561, 2013.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval-Augmented Gen- cration for wwwwlldgg--nttnningg NLP Tasks. In NeurIPS, 2020.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre- training Paradigm. In ICLR, 2022.
Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li. Learning Customized Visual Models with Retrieval-Augmented Knowledge. In CVPR, pp- 15148-15158, 2023.
Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. SwapPrompt: Test-time Prompt Adaptation for Vision-Language Models. In NeurlPS, 2023.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained 2013.
Visual Classification of Aircraft. arXiv preprint
Sachit Menon and Carl Vondrick. Visual Classification via Description from Large Language Mod- cls. In ICLR, 2022.
Sachit Menon, Ishaan Preetam Chandratreya, and Carl Vondrick. Task Bias in Contrastive Vision- Language Models. IJCV, 2024.
12

---

Published as a conference paper at ICLR 2025
Yifei Ming and Yixuan Li. Understanding Retrieval-Augmented Task Adaptation for Vision- Language Models. In ICML, 2024.
M. Neumann, A. S. Pinto, X. Zhai, and N. Houlsby. In-domain Representation Learning for Remote Sensing. In ICLR Workshop, 2020.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not Quantity: On the Interaction between Dataset Design and Robustness of CLIP. In NeurIPS, pp- 21455-21469, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated Flower Classification over a Large Num- ber of Classes. In ICVGIP, pp. 722-729, 2008
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training Language Models Follow Instructions with Human Feedback. In NeurlPS, 2022.
Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caver- lee, and Shu Kong. The Neglected Tails in LasounLL Models. In CVPR, pp. 12988-12997, 2024.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and Dogs. In CVPR, pp. 3498-35055 2012.
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What Does Platypus Look Like? Generating Customized Prompts for Zero-shot Image Classification. In ICCV, pp. 15691-15701, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In ICML, pp. 8748 8763, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Tmxae Generation. In ICML, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do Imagenet Classifiers Generalize ImageNet? In ICML, pp. 5389-5400, 2019.
Karsten Roth, Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waf- fling Around for Performance: Visual Classification with Random Words and Broad Concepts. In ICCV, pp. 15746-15757, 2023.
Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is Caption Worth a Thousand Images? A Study on Representation Learning. In ICLR, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An Open Large-scale Dataset for Training Next Generation Image-Tert Text Models. NeurIPS, 2022.
Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, et K-LITE: Learning Transferable Visual Models with External Knowledge. In NeurlPS, pp. 15558-15574 2022.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time Prompt Tuning for Zero-shot Generalization in Vision-Language Mod- cls. In NeurlPS, pp. 14274-14289, 2022.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A Dataset of 101 Human Action Classes from Videos in the Wild. Technical Report 12-01, Center for Research in Computer Vision, University of Central Florida, 2012.
Quan Sun, Yuxin Fang Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: Improved Training Techniques for CLIP Scale. arXiv preprint arXiv:2303. 15389, 2023.
13

---

Published as a conference paper at ICLR 2025
Bart Thomee, David A. Shamma, Gerald Friedland. Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The New Data in Multimedia Research. Communica- tions of the ACM. 59(2):64-733 2016.
Yonglong Tian, Olivier J Henaff, and Aäron Van den Oord. Divide and Contrast: Self-supervised Learning from Uncurated Data. In ICCV, pp. 10063-10074, 2021.
Vishaal Udandarao, Ankush Gupta, and Samuel Albanic. SuS-X: Training- free Name-only Transfer of Vision-Language Models. In ICCV, pp. 2725-2736, 2023.
Cédric Villani et al. Optimal Transport: Old and New, volume 338. Springer, 2009.
C. Wah, S. Branson, P. Welinder, Perona, and Belongie. The Caltech-UCSD Birds-200-2011
Dataset. Technical Report CNS-TR-2001-00 California Institute of Technology, 2011.
Matthew Wallingford, Vivek Ramanujan, Alex Fang, Aditya Kusupati, Roozbeh Mottaghi, Anirud- dha Kembhavi, Ludwig Schmidt, and Ali Farhadi. Neural Priming for Sample-Efficiees Adapta- tion. In NeurlPS, 2023.
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning Robust Global Represen- tations by Penalizing Local Predictive Power. NeurIPS, 2019.
Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, et al. Searching for Best Practices in Retrieval- Augmented Generation. arXiv preprint arXiv:2407.012// 2024a.
Zhengbo Wang, Jian Liang. Lijun Sheng, Ran He, Zilei Wang, and Tieniu Tan. A Hard-to-Beal Baseline for Training-Free CLIP-based Adaptation. In ICLR, 2024b.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun Database: Large-scale Scene Recognition from Abbey to Zoo. In CVPR, pp. 3485-3492, 2010.
Chen-Wei Xie, Siyang Sun, Xiong Xiong, Yun Zheng, Deli Zhao, and Jingren Zhou. RA-CLIP: Retrieval Augmented Contrastive Language-Iman Image Pre-training, In CVPR, pp. 19265-19274, 2023.
Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark Hasegawa-Johnson, Yingzhen Li, and Chang D Yoo. C-TPT: Calibrated Temt-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion. In ICLR, 2024.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-CLIP: Unlocking the Long-Text Capability of CLIP. In ECCV, 2024a.
Jingyi Zhang, Jiaxing Huang, Xiaoqin Zhang, Ling Shao. and Shijian Lu. Historical Test-time Prompt Tuning for Vision Foundation Models. In NewrIPS, 2024b.
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-Free Adaption of CLIP for Few-Shot Classification. In ECCV, pp. 493-510, 2022.
Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Hongsheng Li, Yu Qiao, and Peng Gao. Prompt, Generate, then Cache: Cascade of Foundation Models Makes Strong Few-shot Learners. In CVPR, 2023.
Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. Retrieval-Auus Augmentee Generation for Al-generated Content: A Survey. arXiv preprint arXiv:2402. 19473, 2024a.
Shuai Zhao, Xiaohan Wang. Linchao Zhu, and Yi Yang. Test-time Adaptation with CLIP Reward for Zero-shot Generalization Vision-Language Models. ICLR, 2024b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning Prompt for Vision- Language Models. International Journal of Computer Vision, 130:2337-2348, 2022.
Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement, 2023.
14

---

Published as a conference paper at ICLR 2025
A EXTENDED SURVEY
A.I TRAINING-BASED RETRIEVAL-AUGMENTE METHODS FOR VLMS
Within retrieval -uggeeetee strategies for VLMs, training- based methods have proposed (additional) pre-training schemes which make use of external knowledge as training datasets (Shen et al. 2022] Liu et al. 2023) and which introduce specifically designed modules for external knowledge (Xie) al. 2023 Iscen et al. 2023). K-LITE (Shen et al.) 2022) augments text supervisions by incor- porating external textual knowledge such as Wiktionary and uses these enhanced supervisions to train VLMs. REACT (Liu et al. 2023) fine-tunes VLMs by constructing datasets about downstream tasks through retrieval. RA-CLIP(Xie et al. (023) and RECO (Iscen et al. 2023) introduce addi- tional modules that allow VLMs utilize retrieved images, where the parameters are trained using large-scale image-text datasets (e.g., WebLI (Chen et al. 2023), YFCC (Thomee et al. 2016). and Conceptual Captions 12M (Changpinyo al.) 2021)) However, these methods require extensive training before test time, which demands high computational resources, making them challenging to use for resource- essource-omsstraiee applications. On the other hand, RA-TTA requires neither training nor additional parameters, hence facilitating its straightforward implementation.
A.2 TRAINING-FREE ADAPTATION METHODS FOR VLMS
Training-foe free adaptation methods for VLMs (Zhang et al. 2022 2023 Zhu et al. 2023 Wang et al.
2024b) have aimed adapt VLMMs to downstream tasks without updating learnable parameters. They can conduct the adaptation efficiently because they do not require backpropagation. Tip- Adapter (Zhang et al., (022) introduces non-parametric cache model that stores the embeddings of few-shot training images and the corresponding labels. Next, it computes the similarities be- tween the embeddings in the cache model and a test image, uses them as weight aggregate the labels in the cache model, and integrates the aggregated pseudo-labell with the logits of CLIP. In order to improve Tip-Adapter, CaFo (Zhang et al.) (002) adds vision foundation model called DINO, synthetic images from DALL-E (Ramesh et al. (02T)) I to expand the few-shot training data, and text descriptions generated by GPT-3 to enrich the language context. APE (Zhu al. 2023) improves Tip- Tip-Adapter by extracting useful features from the cache model, and ELIP-GDA (Wang) cct al. 2024b) applies the Gaussian Discriminant Analysis (GDA) to create a feature-based classifier. However, these methods only focus adapting VLMs without backpropagation and do not account for using external knowledge. In contrast, the goal of RA-TTA is leveraging external knowledge to complement the absence in the pre-trained knowledge, which is orthogonal to the objective of the training-free methods.
B PSEUDOCODE OF RA-TTA
The overall procedure of RA-TTA is described in Algorithm which is self-explanatory.
C
OPTIMAL TRANSPORT FOR CALCULATING SEMANTIC RELEVANCE SCORE
2009) is a framework for calculating the distance between two distributions, typically represented by sampled data points, Mathematically, each distribution is formulated as
Optimal Transport (OT) Villani et al.
U
U and V =
(11)
where x; is a sample from U. u; is a probabiiity mass of x;, i.e., EM function placed at .;. The same definition applies for V. Here, M and N denote the number of data points sampled from U and V, respectively. When calculating the distance, OT requires disparity matrix C representing pairwise disparities between samples from the two distributions. Then, to find the optimal transportation plan I", the optimization problem of OT is formulated as
i==1 u4=1, and 6z, is a Dirac delta
min C) AH()), I[ = {T e R M rix u, r =v). rell(u,v)
(12)
where Ce MxN is disparity matrix where each element C.j represents the disparity between x,; and Øj. u R is probability mass vector whose i-th element is u, v RV is probability mass
15

---

Published as a conference paper at ICLR 2025
Algorithm 1 RA-TTA: Overall Procedure
Require:
test image x/'0d, text descriptions T= -U- (lc,))=2. external image vector database D, an image encoder [()), a text encoder g(-), augmentation size M. for description selection p. K for image retrieval Ks. percentile value p
Ensure: final prediction *
1: / STEP I: DESCRIPTION BASED RETRIEVAL IN 3
IMAGE-TO- DODGERIIDDSIIIIIOONNN SELECTION 3.2.1
3: ST +- RobusttAligammen((2ee. T,M, M,, f.9)
4: TK, <- DDescripiionSelctinn(1 DD T)
5: /* DDESCRIPTIONNTO-IMM.E RETRIEVAL IN
6: G DescriplionCCrouping(SS.
7: <- ProbottppeBassdUeetrivvvll D, Ks.g)
8: /* STEP 11: DESCRIPTION-BASS ADAPTATION IN 3.3
9: / DESCRIPTION- CIMLITTION-BEEED BASED RELEVANCE SCORE COMPUTATION IN 3.3.1
10: C SemaattiCCagCCCmputtttnnn
,,,,,,,,, R, f.9)
RS RelevaaceSoreeCmmprttton
TKn:
12: /" KNOWLEDGE FUSION IN
3.3.2 3.3.22 2 //
13: p < CetrrevalBasedPPeeiiclaao
14: < InitialPrediction( Sest TT.
15: pa Fusion($:1) p)
return g
vector whose j-th element is Fj, and H(T) = -i, Pig log sj is an entropic regularizer for an efficient solution using the Sinkhorn- Knopp algorithm (Cuturi 2013). 1y denotes the all-one vector of dimension N. Using the optimal solution I", the distance between U' and V is (T" C).
A key advantage the OT framework its flexibility to incorporate a user-defined disparity function measure the distance between two distributions. In RA-TTA, a novel disparity (i.e., cost) function called semantic gap is introduced to calculate the semantic disparity between a test image and a retrieved image. Moreover, we adopt two techniques to enhance the reliability and robustness of the adaptation procedure:
We reuse the augmented images of the test image, previously employed in description-based retrieval. As a result, we have the test image set A (comprising the test image and its augmented images) and the retrieved image set Rg, that related the selected features of class c. We represent the importance of each image within these sets using alignment scores because the importance of individual images may vary. This importance is normalized, allowing the two sets to be interpreted as probability distributions.
Thus, calculating the distance between the test image and the retrieved images is transformed into calculating the distance between two distributions using our novel disparity function, semantic gap. Consequently, the optimal transport framework is adopted in this context.
D FURTHER DETAILS FOR EVALUATION
D. DETAILS OF EVALUATION DATASETS
In T provide detailed statistics on the evaluation datasets. We use the dataset configuration of CoOp (Zhou et al. 2022) except for Caltech256 and RESISC4S, which CoOp does not handle. We use test split from SuS-X (Udandarao al. 2023) for Caltech256, and adopt test split
from (Neumann et al. 2020 for RESISC45.
D.2 DETAILS OF EXTERNAL IMAGE DATABASE CONSTRUCTION
To construct web-scale external image database, we explore the text captions of LAION2B (Schuhmann et al. (022) and download the images whose captions contain the target class name, such as "Chevrolet Impala." Specifically, the SQLite full-text search (FTS) table of the text captions in LAION2B is first built, which enables efficient text search for the captions. Then, the class names of downstream tasks are extracted from the definition of downstream tasks. The class names queried to the FTS table, and the table returns the indexes of the captions that contain the
16

---

Published as a conference paper at ICLR 2025
Table 5: Detailed statistics of datasets used in evaluations.
Datasel
# Classes
Test samples
Remark
ImageNet
1,000
50,000
WordNet categories classification
Plowers102
2,463
Fine "fine-graieed flowers
1,692
Textural classification
Oxford Pets
3,669
Fine grained pels classlication
Stanford Cars
196
8,041
Fine grained car classification
UCF101
101
3,783
Human action classificntion
Callech101
100
2,465
Object clarsification
Food101
101
30,300
Fine-grained food classification
Sun397
classification
PGVCAireraft
100
3,333
Fine-grained aircraft classificatio
RESISC45
6,300
Remote sensing image scene classification
Caltech256
256
2465
Object clavsificaiion
CUB200
200
5794
Fine gruined birds classilication
ImageNet-V2
1,000
10,000
ImageNet variant of temporal shift
ImageNet-Skete)
1,000
50,889
ImageNet variant of sketches
InugeNet-AAversari Adversarial
200
7,500
ImageNet variant of adversarial samples
ImagsNes-Rendttin
200
30,000
ImageNett variant of artistic renditions
queried class names. Finally, the corresponding images to the returned indexes are downloaded. The database of each downstream task is constructed separately, and the database size for each down- stream task is reported in Table [6] Because the database is constructed using the class name, the relevance of the database for each dataset to the corresponding task could be ensured (Wallingford
al. 2023
Parashar al. 2024).
Table Detailed statistics of external image databases for the evaluation datasets.
Dataset
D
ImageNet
10,695,986
Flowers102
2,128,144
DTD
452,735
Oxford Pots
1,328.173
Stanford Cars
374,413
1,795,826
Caltech101
2,741,240
Food101
2,237,734
Sun397
6,267,408
PGVCAireraf
336,551
RESISC45
1,962,533
Caltech256
7,015,883
CUB200
644,720
The LAION2B consists 2B image-caption pairs from Web; thus, the downloaded images contain the diversity of the Web. At the same time, the text captions of LAION2B very noisy, and, as shown in Figure 7 the downloaded image could be irrelevant to the queried class name even if the caption contains the queried name, reflecting the Web's noisy nature. Therefore, the constructed databases preserve the nature of a web-scale image database.
D.3 DETAILS OF THE COMPARED METHODS
We evaluate ten methods compare with our proposed RA-TTA. The implementation details for the compared methods as follows.
CLIP (Radford et al. 2021): : A basic prompt "a photo of a/an <Class>" is used for the zero-shot @
Ensemble (Radford et al. 2021 The ensemble of 80 hand-crafted prompts is used as a text
prompt.
https:://gttuub.com////////
17

---

Published as a conference paper at ICLR 2025
Query: Pelican
Query: Praying mantis
Query: Prison
Query: Bird house
Query: Banana
Query: Ant
Query: Hen
BIRD HOUSE CAFE *
Figure 7: Examples of the downloaded noisy images with the corresponding queried class name from ImageNet.
TPT (huu et al. hyperparameters adopted.
2022): The official code released by the authors is used, and their default
2024): The official code released by the authors is used, and their default hyperparameters adopted.
C-TPT (Yoon et al.,
RLCF Zhao et al. 2024b): The official code released by the authors is used, and their default hyperparameters are adopted. The reward model same with the main model.
VisDesc (Menon & Vondrick 2022): ) The official code? released by the authors is used. The
default templates for generating text descriptions are adopted.
WaffleCLIP (Roth et al. 2023): The official codo relcased by the authors is used. The default
templates for generating text descriptions are adopted.
CuPL (Pratt et al. 2023): The official code released by the authors is used. As remarked in Appendix D.4] we slightly modified the default templates for generating text descriptions.
released by the authors is used. Because the validation dataset is unavailable TTA, hyperparameter tuning for each dataset prohibited. Thus, the same hyperparameterss the original paper. For the fair comparison between the retrieval-based methods, including SuS-X-LC, Neural Priming, and RA-TTA, the image database introduced in Appendix D.2 is used for the retrieval.
SuS-X-LC (Udandarao et al. 2023); The official code releerreleseed
used across all datasets, which reported to function well
Neural Priming (Wallingford et al., 2023): The official code released by the authors is used, and their default hyperparameters are adopted. As with SuS-X-T& X-IC, the image database introduced in Appendix D.2 used for the retrieval.
D.4 LLM TEMPLATES FOR GENERATING TEXT DESCRIPTIONS
In Table 7 we present the templates for GPT-3.5 Turbo (Ouyag et al. 2022 used to generate text descriptions for each dataset. The templates are based on those from CuPl. (Pratt et al. (023), but slightly modified them for GPT-3.5 Turbo. Since the CuPL templates were imitially designed for text-davinci-002, which is based on GPT-3 and has now been deprecated, some of these templates do not function expected with GPT-3.5 turbo. For instance, when prompted with the original CuPL template, "Describe an image from the internet of a goldfish," GPT-3.5 turbo responds, "I
httess///aLaub.cc//eee-----11111 TPT
18

---

Published as a conference paper at ICLR 2025
can't access the internet see the image of goldfish." Therefore, we carefully revised only those templates that do not function properly with GPT-3.5 Turbo to ensure they work as intended.
Table 7: CuPL style hand-written templates for extracting text descriptions from LLMs.
Dataset
CuPL. style templates
Describe photo () ()-
Describe what O O looks like.
ImageNet
How you identify }} (12
What does O) ( look like?
Describe photo of (} {1 with distinctive visual features.
Describe the shapes and
Describe the color and patterns of
Describe the overall appereance of
caption an image O (
What does O ( flower look like.
Flowers102
Describe appearance () ().
Visually describe ( {), a.pe of flower.
caption an image () (
What does
material look like? Please explain the chameteristics and provide examples of where it is typically found. surface look like? Please explain characteristis and provide examples where
What does
typically found. texture look like! Please explain the characteristics and provide examples of where it is typically found. object look like? Pleuse explain the characterittics and provide examples of where is typically found. explain the characteristics and provide examples where typically found. [] () puttern look like? Please explain the characteristics and provide examples of where is typically found.
What does
What does
does
() thing like?
What does
Oxford Pets
Describe what () () pet
like.
Visually describe (} aype of pet.
How can you identify ( (]-
Description of [} {}. aype cur.
caption photo of () OO:
What are the primary chamateristics of () (}?
Stanford Cars
Description of the exterior of ()
What are sdentifying characteristies of (). a type of car?
Describe an image of () ().
What does () lock like?
Describe what () () a type car, looks like.
What does a person doing look like.
UCF101
Describe the process of O-
How does person []-
Caltoch101
Describe what ( () looks like.
What does look like.
Describe shupes
structures
Describe photo of () () with distinctive visual features.
Visually describe () [}-
Describe what (} () looks like.
Food101
Describe photo () (} with distinctive visual features.
How can you tell that the food in photo is () {}?
Describe what (} ( looks like.
SUN397
How can you idlentify () ()?
Describe photo of () () with distinctive visual features.
RGVC Aireraft
Describe () [} aircraft.
Describe satellite photo of Q
RESISC45
Describe the satellite photo of (1 ()-
Describe aerial photo of
Describe what ( () looks like.
Caltech256
Describe photo () () with distinctive visual features.
Deseribe what () ( specics of bird, looks like.
What does () look like.
Visually deseribe ( a of bird.
CU/R200
caption of an image of () a of bird.
Describe appearance of () [].
What are the prominent features to idcotify O () bird.
Describe an art draswing of () ()-
Describe artwork showing (1 ().
Describe cartoon (}
Describe an origami ( ().
Describe deviant art photo depicting (} [}-
ImageNet rendition
Describe an embreidery
Describe graffiti art showing () ()-
()-
Describe painting of ().
Describe sculpture ( (
Deseribe black and white sketch of O ().
Describe toy ( {}.
Describe video game () (}-
Describe how black and white sketch () {} looks like. Describe black and white sketch of { ()-
ImageNet sketch
19

---

Published as a conference paper at ICLR 2025
E ADDITIONAL EVALUATION RESULTS
E.1 MAIN RESULTS WITH STATISTICAL SIGNIFICANCE
we report the standard deviation omitted in Tables ! and where the value after "*" represents the standard deviation over five different runs. Overall, RA-TTA demonstrates robust performance, exhibiting low variability across most datasets.
In Tables 8 and 9
Table 8: Statistical significance for the results in Table
Stanford Care
FGVC Afreaft
Flovorsin2
Oxford Pets
Caltech101
RESISCAS
Caltech256
UCF101
Food101
SUN397
CUR200
Ayg: (13)
IN-1K
DTT
69.08
69.18
47.01
$7.44
66.55
68.04
93.79
86.34
65.32
23.31
56.84
85.37
60.11
67.57
TPT
(4:005)
(=0.12)
(+0.23) 23)
(+0.09)
(40.00)
(40033)
(#0.08)
(+0.15)
(40.08)
(+0.21)
(40.19)
(=0.17)
(40.6)
(50.15)
68.32
69.43
45.27
88.25
63.48
65.50
93.39
84.95
64.55
24.39
56,02
85.25
58.84
66.90
C:IPT
(2307)
(=0.16)
(20.10)
(40.13)
(20.15)
(#0.20)
(20.22)
(29.21)
(40.22)
(40.17)
(20.00
(20.11)
(20.30)
(20.17)
68.61
67.72
46.40
86.73
66.51
66.98
93.83
86.09
64.22
23.43
56.89
85.18
57.91
67.02
RLCF
(2322)
(=0.31)
(40
(20.20)
(20.90)
(40
(20.28)
(2:0.22)
(43 25)
(40.27)
(26.18)
(20.11)
(20.31)
(20.25)
RA-ITA (Ours)
70.58
78.65
60.98
92.78
70.11
73.28
94.84
87.10
70.38
32.34
66.95
89.50
62.73
73.09
(#0.34)
(=0.10
(40.17) (+0.0)
(40 10
(40.22)
(40.08)
(+00004
(40.0%)
(+0.22)
(+0.00)
(=0.00
(++0.7)
(+0.11)
Table 9: Statistical significance for the results in Table
IN-A
IN-V2
IN-R
IN-K
Avg.
55.78
63.48
78.98
47.95
60.77
(00.17)
(=0.11)
(#0.10)
(#:0112)
(=00113)
50.28
62.47
75.68
47.42
58.96
CTTPT
(10.22)
(10.28)
(L0007)
(E0.13)
(10.18)
$6.52
63.37
77.04
48.09
61.26
RLCF
(200118)
(1:.22)
(1:0055)
(1:0114)
(:00.15)
RA-TTA (Ours)
59.21
64.16
79.68
50.83
63.47
(6022)
(=0.15)
(#0.01)
(#0.01)
(=00112)
E.2 RESULTS ON ABLATION STUDY
We present the implementation details for the variants of RA-TTA. For the image-to-image similar- ity search Var. 1, we use simple cosine similarity between image embeddings and retrieve Kg external images for test image. To replace the semantic gap computation for Var. 2, we adopt simple cosine similarity between image embeddings. To realize Var. 3, we just use uniform weights vectors for L/ and V.
In Table 10 we present results on ablation study across the standard transfer learning benchmarks, and the similar trends in Section 4.3 are observed, where description-ban based approaches, including Var. 3 and RA-TTA, outperform other variants.
Table 10: Results on ablation study for test-time adaptation (or transfer learning).
Stanford cars
FGV aireraft
Flowors102
Osford pets
Caltech1u1
RESISCAS
Caltech256
UCF101
Foed101
SUN397
CUR200
Avg. (13)
IN-IK
DTD
Var. 1
66.62
77.92
56.67
91.22
67.81
67.44
93.58
86.41
68.62
29.39
62.92
$8.43
60.78
70.75
Var.
68.81
78.81
57.17
91.93
68.71
69.02
94.05
86.93
69.41
30.91
65.01
88.61
61.27
71.59
70.45
69.95
94.35
72.67
RA-TTA (Ours)
70.58
78.65
60.98
92.78
70.11
73.28
94.84
87.10
70.38
32.34
66.95
$9.50
62.73
73,09
E.3 EFFECTS OF DIFFERENT LLMS GENERATING TEXT DESCRIPTIONS
The description-based retrieval relies on the LLMs to generate detailed text descriptions. Thus, analyze how different LLMs affect description quality and final adaptation performance, text descriptions generated by Claude-Sonnet are newly considered for this analysis. We compare the
20

---

Published as a conference paper at ICLR 2025
performance of CuPL Prrttt et al. achieve better performance with improved descriptions.
2023) investigate the quality of text descriptions, CuPL can
As shown in the first two rows of TT] CuPL with the de- scriptions of GPT works bet- ter than CuPL with those of Claude, implying that the text descriptions generated by GPT are better than those gener- ated by Claude. Thus, differ- ent I.LMs can affect descrip- tion quality even with the same generation prompts. Also, the last two rows of Table T1 show that RA-TTA with the descriptions
Table 11: Performance when using text descriptions generated by ChatGPT and Claude3. We report the top-1 accuracy (%) on the FGVC aircraft, Stanford cars, and RESISC45 datasets. The ployed LLMs are indicated with the method name.
FGVC aireraft
Stanford cars
RESISC45
Avg. (3)
CuPl.-Clande3 CuPL-ChaGPP RA-TTA+Cluudee RA-TTA+ChaGPP
28.05
66.38
$7.33
50.59
28.89
66.92
65.17
53.66
30.93
68.82
61.92
53.89
32.34
56.47
of GPT outperforms RA-TTA with those of Claude, implying that the better descriptions facilitate the adaptation of RA-TTA more effectively by improving the relevance and quality retrieved images. Notably, RA-TTA with the text descriptions of Claude outperforms the performance of the corresponding CuPL (i.e., CuPL+Claude3 vs. RA- RA-TTA+Claude3), demonstrating that RA-TTA effectively leverages external knowledge when the quality of the description is not very good.
E.4 IMPACT OF DATABASE SIZE ON ADAPTATION PERFORMANCE
Regarding the database of external images, we analyze the impact of its size on adaptation iefformance-ddpptatiin accuracy, storage costs, and inference time. For this analysis, we use random sampling to build subset of the original database. As shown in Table 12 accuracy im- proves alongside storage costs as the database size increases. This result expected because the likelihood of including informative samples for adaptation increases with larger database, albeit at the expense increased storage costs. Regarding inference time, thanks to efficient search engines, such FAISS (Johnson al. 2021), the increase in latency remains almost negligible, shown in the last row of Table [2
Table 12: Effects of the database size accuracy, storage costs, and inference time of RA-TTA.
We report the top-1 accuracy (%), storage costs (MB), and average GPU inference time per sample (s/sample) measured the proportion of the sampled database to the original database. The values the parentheses indicate the number of images in database.
single RTX4090 for the Stanford cars dataset. The percentage represents
5%
10%
20%
40%
60%
Original (374,413)
(13,720)
(37,441)
(74,882)
(149.765)
(224.647)
(299,530)
Accuracy
68.12
68.81
69.01
69.81
69.96
70.08
70.11
Storage costs (MB)
18.28
36.56
73.13
146.25
219.38
292.51
365.64
Inference time (s/smmple)
0.106
0.106
0.107
0.111
0.113
0.113
0.117
E.5 RESULTS ON SENSITIVITY ANALYSIS
We present hyperparameter sensitivity analyses on the standard transfer learning datasets, and the similar trends in Section 4.4
observed as shown in Figures
20
71
71
70
Aecuracy (%)
Accuracy (56)
69
70
Accuracy
69
70
68
67
RA-TTA SaS-X-LCC
65
69
25
100 M
5.20 40
60
160
520
60 S0
0.10.25
Ko
Ks
(a) Augmentation size. (b) # of selected descriptions.
(c) # of retrieved images.
(d) Score percentile.
Figure 8: Effects of the four hyperparameters on the accuracy of ImageNet-Ik:
21

---

Published as a conference paper at ICLR 2025
79
79
Accurscy (%)
Accuracy (%)
(%6
78
Acquracy
76
78
77
77
73
72
76
S65-X-LC
76
525.50
100
150
200
20 40 60
80
160
5.20 60
80
160
0.10.25
0.5 0.75 0.91.0
KD
Ks
p
(a) Augmentation size. (b) selected descriptions. (c) # of retrieved images.
(d) Score percentile.
Figure 9: Effects of the four hyperparameters on the accuracy of Flowers102.
62
Accuracy (5)
Accurncy (%)
Accuracy (%)
61
61
61
60
Accuracy
60
60
60
RA-TIA
59
SoS-X-LC
58
59
59
55
100
150
200
5020 40
160
520
60
160
0.1022
0.75
0.91.0
M
KD
Ks
(a) Augmentation size.
(b) # of selected descriptions. (c) # of retrieved images.
(d) Score percentile.
Figure 10: Effects of the four hyperparameters the accuracy of DTD.
Accuracy (%)
Accuracy (%)
Accaracy (%%
93
91
90
92
92
92
89
RA-TTA SG
5.
100
5.20 40
160
520 60
M
KD
Ks
(a) Augmentation size. (b) # of selected descriptions. (c) # of retrieved images.
(d) Score percentile.
Figure 11: Effects of the four hyperparameters on the accuracy of Oxford pets.
71
70
Accurey (%)
Accuracy (%)
Accuracy (%)
Accuracy (%))
70
69
69
69
68
68
68
S0S-X-LC
63
5.25.50
100
150
200
5:20 40
8D
160
5.2040 60
80
160
0.10.25
0.5
0.75 0.91.0
Ks
(a) Augmentation size. (b) selected descriptions.
(c) # retrieved images.
(d) Score percentile.
Figure 12: Effects of the four hyperparameters the accuracy of Stanford cars.
74
73
72
Accurscy (%)
72
72
70
71
Accuracy
Acouracy
68
70
70
70
69
69
69
68
RA-TTA
68
67
SuS-X-LC
525
100 M
150
5.0 40
60
160
520 40 60
80
0.10.25
0.91.0
KD
Ks
(a) Augmentation size. (b) # of selected descriptions. (c) # of retrieved images.
(d) Score percentile.
Figure 13: Effects the four hyperparameters on the accuracy of UCF101.
22

---

Published as a conference paper at ICLR 2025
95
Accurscy (%)
95
Accuracy (%)
95
Accuracy (%)
95
94
93
94
RA-TTA
91
S65-X-LC
5.5.50
100
150
200
20 40 60
80
160
520 40 60
80
160
0.10.25
0.5
0.75 0.91.0
Ks
(a) Augmentation size. (b) selected descriptions.
(c) # retrieved images.
(d) Score percentile.
Figure 14: Effects of the four hyperparameters the accuracy of Caltech101.
87
Accuracy (%)
Accuracy (%)
Accuracy (%)
87
87-
87
85
86
RA-TIA
86
86
83
SoS-X-LC
5
100
150
200
20 40 60
80
160
520 40 60
80
160
0.10.25
0.5 0.75 0.91.0
KD
Ky
p
(a) Augmentation size. (b) selected descriptions.
(c) # retrieved images.
(d) Score percentile.
Figure 15: Effects of the four hyperparameters on the accuracy of Food101.
Accuracy (%)
70
Accuracy (%)
70-
Accuracy (%)
70
69
69
69
69
68
68
67
SoS-X-LC
52
100
150
200
20 60
40
160
520 60
80
160
0.10.25
0.5 0.75 0.91.0
M
KD
Ks
p
(a) Augmentation size.
(b) # selected descriptions. (c) # of retrieved images.
(d) Score percentile.
Figure 16: Effects of the four hyperparameters on the accuracy of SUN397.
32
Accuncy (%))
32
(%)
Accuraey
81
Acuuaay
31
31
30
30
RA-TTA
30-
30
29
29
SUS-X-LC
29
28
5.5
100
150
20 40
60
160
5.20
60
80
160
0.10.25
0.5 0.75 0.91.0
M
Kp
K5
P
(a) Augmentation size. (b) # selected descriptions.
(c) # of retrieved images.
(d) Score percentile.
Figure 17: Effects the four hyperparameters on the accuracy of FGVC aircraft.
Accuracy (%)
67
Accurncy (9%)
Accuracy (%)
67
65
Accuracy
64
66
RA-ITA
65
Su5-X-LC
65
65
5
100
150
3.20-0 40
60
160
520
60 SO
0.10.25
M
KD
Ks
(a) Augmentation size. (b) # of selected descriptions. (c) # of retrieved images.
(d) Score percentile.
Figure 18: Effects of the four hyperparameters on the accuracy RESISC45.
23

---

Published as a conference paper at ICLR 2025
90
90
90
Accuracy (%)
Accuricy (%)
Accuracy (%)
88
87
86
89
RA-TIA
So5 CC
83
25
100 M
150
20 40
160
520
60
0.10.25
0.75
0.91.0
KD
Ks
(a) Augmentation size. (b) # of selected descriptions.
(c) # of retrieved images.
(d) Score percentile.
Figure 19: Effects of the four hyperparameters the accuracy of Caltech256.
Accuracy (%)
Accuracy (%)
Accuracy (%)
60
61
61
61
Accuracy
60
60
54
59
RA-TTA
59
89
SDS-X-LC
48
5.5 50
100
150
200
20 40 60
160
520 60
0.10.25
0.75 0.91.0
M
Ks
(a) Augmentation size.
(b) # of selected descriptions. (c) # of retrieved images.
(d) Score percentile.
Figure 20: Effects of the four hyperparameters on the accuracy of CUB200.
E.6 GPU MEMORY OVERHEAD FOR INFERENCE
RA-TTA does not require backpropagation (i.e., training-free), which contributes to the memory efficiency RA-TTA. As shown in Table [3| results lower GPU peak memory compared to TPT which uses backpropagation to update Iearnable prompts.
Table 13: GPU peak memory for TPT and RA-TTA. We report the GPU peak memory (MB) on the FGVC aircraft, Stanford cars, and RESISC45 datasets.
FGVC aircraft
Stanford cars
RESISC45
Avg. (3)
TPT
2213.53
3718.75 984.75
1358.72
2430.33
RA-TTA (Ours)
905.63
911.43
933.94
E.7 ADDITIONAL VISCALIZATIONS
In Figure 21. present illustrative examples of the rejected/selected descriptions in description- based retrieval on the Stanford cars (Krause et al. (013) and RESISC45$Cheng et al. (2017) datasets. Figure 21 illustrates that the selected descriptions pay attention specific piece of information, namely pivotal feature, whereas the rejected descriptions emphasize irrelevant features. Figure 22 briefly illustrates the workflow of RA-TTA on the Stanford cars and RESISC45 datasets.
24

---

and containers Published as a conference paper at ICLR 2025
Rejected text descriptions with irrelevant features
Selected text descriptions with relevant features
Test image
You can identify 2012 Porsche Panamera Sodan by looking for its distinct body style, four-door design, aggressive front fascia and signature Porsche logo the front and rear of the vehicle.
The 2012 Porsche Panamera Sedan features sporty exterior styling, which includes a sleek, low-slung profile with four doors, und wide wheel arches. It has distinctive sloping roofline and four-door layout, combining the elegance of coupe.
GT: 2012 Porsche
Panamera Sedan
thermal power plant in an acrial image would show large buildings with tall chimmeys and a network of power lines connecting to the grid. It also have fuel storage arcas, water reservoirs, and roads for transport.
thermal power plant in a satellite photo uppears as large industrial complex with prominent structures like cooling towers, boiler units, and smokestacks. The photo show heat emissions in the form of thermal plumes from the cooling towers and steam vents, well as energy production.
GT: Thermal
power plant
satellite photo of harbor typically shows ships and boats docked along piers and quays, with visible cargo handling. layout of harbor, such as breakwaters can seen along with surrounding infrastructure.
satellite photo of harbor typically shows body of water surrounded by land with docks, piers, and boats. The harbor may uppear as dark area within the surrounding land, with watercraft visible water.
GT: Harbor
Figure 21: Examples of the selected and rejected text descriptions in our description based retrieval on the Stanford cars and RESISC45 datasets. The results of the rejected text descriptions and the selected text descriptions are shown individually. An irrelevant feature colored yellow in descriptions, while relevant feature is colored red.
Selected text descriptions with relevant features
Test image
Retrieved images
Refined prediction
2012 Porsche Panamern Sodan features sporty exterior styling, which includes a sleek, low-slung profile with four doors, and wide wheel arches. It has distinctive sloping roofline and four-door layout, combining the elegance of coupe.
Before refinement
2012 BMW
M3 Coupe
After refinement
GT: 2012 Porsehe Panamera Sedan
2012 Porsche
Panamera Sedan
thermal power plant in a satellite photo appcars as large industrial complex with prominent structures like cooling towers, boiler units, and smokestacks. The photo may show heat emissions in the form of thermal plumes from the cooling towers and steam vents, well energy production.
Before refinement Industrial area
After refinement
Thermal power plant
GT: Themnal
power plant
satellite photo of a harbor typically shows body water surrounded by land with docks, piers, and boats. The harbor may appear as dark arca within the surrounding land, with watercraft visible in the water.
Before refinement Parking lot
After refinement Harbor
GT: Harbor
Figure 22: Examples of the brief workflow of RA-TTA on the Stanford cars and RESISC45 datasets. The test image is queried to select text descriptions that include relevant features, and then proper images are retrieved based on those text descriptions. The initial prediction is refined with the help of the retrieved images.
25


Reviews:
[{"review_id": "QOfumBWGZr", "reviewer": "Reviewer_UYSy", "summary": "This paper introduces the success of RAG into VLM for test-time adaptation, achieving performance improvements. Extensive experiments demonstrate the effectiveness of the method.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-written, and the framework is clear and easy to follow. Additionally, the provided code enhances reproducibility.\n2. Introducing RAG into VLM for test-time adaptation is a reasonable and meaningful idea.\n3. The experiments in this paper are thorough, with detailed descriptions. The method surpasses others on multiple datasets.", "weaknesses": "1. More training-free methods should be discussed and compared. In VLM-based transfer learning, training-free adapters [1] have significantly improved with a small number of image samples, as in CaF [2] with fine-grained descriptions based on classes. The proposed method is more complex, so I am curious about its performance advantage compared to these easy but efficient methods.\n\n2. This method involves generating descriptors for each test sample with a GPT-based generator, then self-filtering with the image itself, retrieving similar samples, and finally drawing conclusions. This is a relatively lengthy pipeline. How does its efficiency compare to other methods? Additionally, GPT usage is not free, though open-source models could be a workaround. Could this pipeline be simplified, perhaps by using improved prompts to avoid generating imaginary content or by creating visual descriptors to replace the image-to-description selection?\n\n3. Could more visualization results be provided, showing the generated descriptors -> filtered irrelevant descriptors -> refined inference results, similar to the presentation in Figures 3 and 4 of DVDet [3]?\n\nReferences:\n\n[1] Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling\n[2] Visual Classification via Description from Large Language Models\n[3] LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-Grained Descriptors", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A2DEycrj0m", "reviewer": "Reviewer_gDjz", "summary": "The paper proposes a novel retrieval-augmented test-time adaptation (RA-TTA) for improving VLM generalization on downstream datasets at test time. RA-TTA exploits additional visual information through retrieval of web-scale images to regularize VLM predictions at test time. Extensive experiments over various datasets show the effectiveness of the proposed method.", "questions": "as shown in Weaknesses", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The introduction retrieval augmentation into TTA problem is an innovative approach. Also, the proposed description-based retrieval technique aligns well with the multi-modal capabilities of VLMs.\n\nThe paper presents extensive experimental evaluations over 17 diverse datasets. The authors also conduct detailed ablation studies and sensitivity analyses.\n\nThe paper is well-organized and easy to follow.", "weaknesses": "The paper lacks some necessary discussions and analyses.\n\n(1) In the proposed method, different databases are sampled from large-scale datasets for each downstream dataset respectively. This may introduce additional computational requirements when adapting to different datasets. Will directly retrieving from the large-scale datasets like LAION2B help mitigate this issue? In addition, the computation overhead or inference time compared with TPT should be provided for examining the efficiency of the proposed method.\n\n(2) The retrieval process highly depends on the LLMs for generating detailed text descriptions. The authors are suggested to discuss how different LLMs might affect description quality and, consequently, the final adaptation performance.\n\n(3) How do the authors ensure that the retrieved images contain the expected features to augment the test image? For example, in the examples provided in the introduction, if the test image depicts the front and side of a vehicle, the retrieved images should ideally contain related information, such as headlights.\n\nSome recent and related studies are missing from the section Related work. The author should consider discussing the difference with these works [a,b,c,d].\n[a] Efficient Test-Time Adaptation of Vision-Language Models\n[b] Dual memory networks: A versatile adaptation approach for vision-language models\n[c] SwapPrompt: Test-Time Prompt Adaptation for Vision-Language Models\n[d] Historical Test-time Prompt Tuning for Vision Foundation Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VEyZS3BL44", "reviewer": "Reviewer_aU8U", "summary": "In vision-language models, existing test-time adaptation methods solely rely on the internal knowledge encoded within the parameters. To complement the limitation of the internal knowledge, this paper proposes retrieval-augment TTA for adapting VLMs to test distribution using external knowledge obtained from a web-scale image database. It uses fine-grained text descriptions for retrieving proper external images and refining VLM’s predictions with the retrieved external images. Extensive evaluations on 17 datasets validate that RA-TTA outperforms the state-of-the-art methods.", "questions": "1. The basic idea and pipeline are very similar to [1]. The differences and advantages should be claimed. \n\n2.  In Equation 8, why is optimal transport used, and what is its purpose?\n\n3. Figure 3 is not very clear in presenting the proposed method, specifically in prototype-based retrieval part.\n\n[1]  Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models, ICML 2024.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow. \n2. This work investigates test-time adaptation in visual language models. Due to the difficulty of updating pre-trained models frequently, TTA on visual language models is a highly researched topic.\n3. The proposed method can achieve state-of-the-art results on 17 benchmarks, including standard transfer learning and distribution shift. And it conducts ablation study to validate the effectiveness of proposed modules.", "weaknesses": "1.\tThe biggest issue with the article is its lack of innovation. In fact, the proposed method is very similar to [1], which is not even cited in this paper. [1] firstly retrieves relevant samples from a web-scale database using seed prompts (T2I) or seed images (I2I), then build a K-shot cache by selecting the Top-K similar images per class based on CLIP embeddings, and the final predictions is an ensemble of original predictions and few-shot cache predictions. The basic idea and pipeline are very similar to [1].\n2.\tIn Table2, the proposed method achieves inferior performance than TPT [2] and RLCF [3] in IN-V2 and IN-A. The authors should analyze the reasons behind this phenomenon. Does it indicate that the effectiveness of the method is related to the downstream datasets? \n3. The paper lacks clarity in describing certain operations. For instance, in the Equation 8, why is optimal transport used, and what is its purpose?\n\n\n[1] Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models, ICML 2024.\n\n[2] Test-time Prompt Tuning for Zero-shot Generalization in Vision-Language Models, NeurIPS 2022.\n\n[3] Test-time Adaptation with CLIP Reward for Zero-shot Generalization in Vision-Language Models. In ICLR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fz5kqzQzn8", "reviewer": "Reviewer_DF2M", "summary": "This work focuses on test-time adaptation (TTA) of vision-language models (VLMs) and proposes retrieval-augmented TTA (RA-TTA), aiming to improve the generalization ability of VLMs on downstream datasets. Specifically, RA-TTA incorporates external visual information by retrieving web-scale images with fine-grained text descriptions, which are then used to augment VLM predictions at test time. Unlike prior test-time adaptation methods that rely solely on the pre-trained model's internal knowledge, RA-TTA enhances test-time adaptation with retrieved images, consistently improving prediction accuracy across 17 datasets.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Introducing retrieval augmentation into test-time adaptation of VLMs is novel. Prior methods generally augment test images using simple augmentation strategies (e.g., crop, resize), while this work exploits the external knowledge form additional image data bases to augment each test image for better model predictions. Additionally, the proposed description-based retrieval could effectively leverages the bi-modality of VLMs, leading to more relevant retrieval results.\n2. The experiments conducted in this work are extensive. The authors benchmark the proposed RA-TTA with the SOTA methods on 17 diverse datasets, and provide comprehensive analysis experiments, including ablation studies and parameter sensitivity analyses.", "weaknesses": "1. RA-TTA highly relies on Large Language Models (LLMs) to generate fine-grained text descriptions for image retrieval. Therefore, the authors should discuss how different LLMs might vary in their effectiveness at generating useful descriptions, and how this impacts retrieval relevance and quality.\n2. The authors construct different data base for each downstream dataset by sampling form a large dataset such as LAION2B. The sampled database size may impact both computational cost and retrieval quality, for example, a larger sampled database size may bring better retrieval quality but may also increase latency and storage costs. Therefore, an analysis of how the sampled database size affects adaptation performance should be provided.\n3. Ensuring semantic diversity in text descriptions for each class name is crucial in RA-TTA. How is this diversity ensured during the generation process?\n4. Compared  with prior studies such as TPT, the proposed RA-TTA involves additional database construction process and retrieval process which may bring more computation cost and inference time. Therefore, the computational and storage overhead compared to TPT should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)