Multi-Label Test-Time Adaptation with Bound Entropy Minimization
Authors: Xiangyu Wu
Feng Yu
Yang Yang
Qing-Guo Chen
Jianfeng Lu
Venue: ICLR 2025 Poster (2025)
Keywords: Vision-Language Models
Zero-Shot Multi-Label Generalization
Test-Time Adaptation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A Multi-Label Test-Time Adaptation method with Bound Entropy Minimization objective.

Abstract: Mainstream test-time adaptation (TTA) techniques endeavor to mitigate distribution shifts via entropy minimization for multi-class classification, inherently increasing the probability of the most confident class. However, when encountering multi-label instances, the primary challenge stems from the varying number of labels per image, and prioritizing only the highest probability class inevitably undermines the adaptation of other positive labels. To address this issue, we investigate TTA within multi-label scenario (ML--TTA), developing Bound Entropy Minimization (BEM) objective to simultaneously increase the confidence of multiple top predicted labels. Specifically, to determine the number of labels for each augmented view, we retrieve a paired caption with yielded textual labels for that view. These labels are allocated to both the view and caption, called weak label set and strong label set with the same size k. Following this, the proposed BEM considers the highest top-k predicted labels from view and caption as a single entity, respectively, learning both view and caption prompts concurrently. By binding top-k predicted labels, BEM overcomes the limitation of vanilla entropy minimization, which exclusively optimizes the most confident class. Across the MSCOCO, VOC, and NUSWIDE multi-label datasets, our ML--TTA framework equipped with BEM exhibits superior performance compared to the latest SOTA methods, across various model architectures, prompt initialization, and varying label scenarios. The code is available at https://github.com/Jinx630/ML-TTA.

Published as a conference paper at ICLR 2025
MULTI-LABEL TEST-TIME ADAPTATION WITH BOUND ENTROPY MINIMIZATION
Xiangyu Wu! Feng Yu', Qing-Guo Chen', Yang Yang': Jianfeng Lu" Naanjing University of Science and Technology
2 Aliiaba International Digital Commerce Group
ABSTRACT
Mainstream test-time adaptation (TTA) techniques endeavor to mitigate distri- bution shifts via entropy minimization for multi-class classification, inherently increasing the probability of the most confident class. However, when encoun- tering multi-label instances, the primary challenge stems from the varying num- ber of labels per image, and prioritizing only the highest probability class in- evitably undermines the adaptation of other positive labels. To address this issue, we investigate TTA within multi-label scenario (MI-TTA), developing Bound Entropy Minimization (BEM) objective to simultancously increase the confi- dence of multiple top predicted labels. Specifically, to determine the number labels for each augmented view, we retrieve paired caption with yielded tex- tual labels for that view. These labels are allocated to both the view and cap- tion, called weak label and strong label set with the same size k. Follow- ing this, the proposed BEM considers the highest top-k predicted labels from view and caption a single entity, respectively, learning both view and cap- tion prompts concurrently. By binding top-k predicted labels, BEM overcomes the limitation of vanilla entropy minimization, which exclusively optimizes the most confident class. Across the MSCOCO, VOC, and NUSWIDE multi-label datasets, our MI.-TTA framework equipped with BEM exhibits superior perfor- mance compared to the latest SOTA methods, across various model architec- tures, prompt initialization, and varying label scenarios. The code is available at https: //tt:::////h/t// com/ wwwxx300/ML-TT111
INTRODUCTION
The advent of vision-language models (VLMs) (Radford et al., 2021; Li et al., 2023; Zeng al., 2024: Yang et al., 2024a) has facilitated remarkable generalization capabilities by pretraining on massive datasets. Nonetheless, VLMs such as CLIP (Radford et al., 2021), require sophisticated prompt learning when confronted with considerable discrepancies between training and testing do- mains, to prevent performance degradation due to distribution shifts occurring during testing time.
Fortunately, recent advancements (Shu et al., 2022; Feng et al., 2023; Ma et al., 2023; Liu et al., 2024b; Zhang et al., 2024b; Zhao et al., 2024a; Karmanov al., 2024; Yoon et al., 2024; Gao et al., 2024) allow for immediate adaptation to any distribution of test instance during testing time, which is known as Test-Time Adaptation (TTA). As pioneering works, TPT (Shu al., 2022) and its enhancement, DifITPT (Feng et al., 2023), select set of confident augmented views. learning instance-level prompt for each test instance. DART (Liu et al., 2024b) and DMN (Zhang et al., 2024b), to fully utilize the encountered knowledge from past samples, design dual-modal knowl- edge retention prompts and dynamic dual-memory networks, respectively, to adaptively incorporate historical knowledge. The central premise of these methods is entropy minimization, which aims to minimize inconsistency and uncertainty over the model predictions, and further increase the predic- tion probability of the highest confidence class, theory that is readily demonstrable.
Although entropy loss is advantagcous for TTA as an uncertainty metric, a natural question arises: Can it be reliably applied to instances with multiple positive labels? As illustrated in Figure (a), for the positive label set (keyboard, phone, remote, mouse, book), compared to CLIP, all methods consistently boost the probability of the most confident class, keyboard. Nonetheless, TPT (Shu
"Corresponding author

---

Published as a conference paper at ICLR 2025
(a). Changes of output logits compared to CLIP
(b). Results on varying number of labels
0.95-
80
logie:
6.50
0,85
MT-TTA
70
MB-TTA
0.80-
60
6.75
50
koebborrd
phone
remole
book
1.0-
40
0,95-
BLOE
0.90-
30
6,85
0.80
6.75
book
people refnigerator clock
3
5
10
Number of labels per image
Positive labels of image
Figure I: (a). Compared to CLIP (Radford et al., 2021), ML-TTA increases all positive label logits simulta- neously, while others focus only top-/ class. (b). Comparison of various methods images with varying numberr. Compared to CLIP, as the number of labels per image rises, the adaptability of TPT (Shu et al., 2022) and RLCF (Zhao et al., 2024a) in handling multi-label images shows a marked decrease.
et al., 2022) and RLCF (Zhao et al., 2024a) adversely impair the remaining positive labels. This indicates that existing TTA methods primarily focus on increasing the confidence of lop-/ label, leading to insufficient adaptation for other positive labels. Given this, we expect to treat the highest top-k positive labels as a single label, aiming to simultaneously increase the predicted confidence of multiple top-k labels. However, positive label sets are not known in advance in real applications.
Based on the preceding discussion, we investigate the TTA within multi-label scenario (MI-TTA)) and propose novel theoretical optimization objective named Bound Entropy Minimization (BEM), which posits that when the highest top-k predicted labels (k being the size positive label set) share identical probabilities, the entropy loss will uniformly increase the probabilities of all top-k classes. Consider multi-label test image with set of augmented views, to determine the number of positive labels for each view, we retrieve paired caption with derived textual labels for each view, which then serves as weak label set of size k for the corresponding view. Furthermore, owing to the aligned visual- with known positive labels, a premise corroborated by recent academic research (Guo et al., 2023; Zhao et al., 2024b; Li et al., 2024a; Wu al., 2024). Drawing inspiration from these findings, we conceptualize each paired caption as a pseudo-view possessing a known label set, termed strong label set, of the same size since the textual labels are directly derived from captions.
space of CLIP (Radford al., 2021), texts can be treated as pseudo-images
Upon determining the weak label set for each view and the strong label set for each paired caption, the proposed BEM objective binds the highest top-k predicted labels as a single label for both view and caption. By optimizing the view prompt and caption prompt, the model is encouraged to concur- rently increase the confidence of the top-k classes. Additionally, since some augmented views and paired captions may fail to capture the target label area, leading to misleading predictions, we adopt confidence selection utilized in TPT (Shu et al., 2022) to filter out "noisy" views and captions with high entropy (i.e., low confidence). Consequently, in this paper, starting from TPT, the developed ML-TTA framework equipped with BEM endows the CLIP's adaptability of multi-label instances during testing. Our contributions summarized as follows:
We examine the Minimization (BEM), which simultancously increase the probabilities of all highest top labels.
Test-Time Adaptation (ML-TTA) and propose Bound Entropy
BEM binds weak label set of view and strong label set of the caption as single label, respectively, learning instance-level view and caption prompts for adapting multi-label test instances.
On the MSCOCO, VOC, and NUSWIDE datasets, ML-TTA outperforms the original CLIP model as well as other state-of-the-art TTA methods designed for multi-class classification, across vari- ous model architectures, prompt initialization, and varying label scenarios.
RELATED WORK
2.1 TEST-TIME ADAPTION
Test-time adaptation (TTA) (Zhang et al., 2022; Shu et al., 2022; Ma et al., 2023; Karmanov et al., 2024; Zhao et al., 2024a; Lee et al., 2024; Chi et al., 2024; Ma et al., 2024) enables models to adapt
2

---

Published as a conference paper at ICLR 2025
changing distributions during testing time without accessing to the source domain data extensive target domain data. Within the spectrum of TTA settings, e.g., "fully" TTA (Wang et al., 2021; Zhao et al., 2023), "online" TTA (Lee & Chang, 2024; Lee et al., 2024), "continuous" TTA (Liu et al., 2024a; Song et al., 2023), and "prior" TTA (Wei et al., 2023; 2024), Conline"TTX (Shu et al., adapting to individual samples and is
2022; Karmanov al., 2024; Zhao al., 2024a) focuses particularly valuable in many application domains, such as autonomous driving, where weather con- ditions are constantly changing, and road monitoring, where traffic patterns are continually evolving. MEMO (Zhang et al., 2022) is the pioneering work that proposes consistent predictions across di- verse augmented views. Following this, TPT (Shu et al., 2022) notably enhances the generalization capabilities of the CLIP (Radford et al., 2021) model to unseen test data by entropy minimization. SwapPrompt (Ma et al., 2023) utilizes online and target prompts, enhancing the CLIP's adaptability by preserving historical information and alternating prediction. In contrast, TDA (Karmanov et al., 2024) adapts to streaming input data by constructing a dynamic key-value cache from historical data. RLCF (Zhao et al., 2024a) incorporates reinforcement learning to distill knowledge into more compact models. Among these works, MEMO (Zhang et al., 2022). TPT (Shu et al., 2022), and RLCF (Zhao al., 2024a) are particularly challenging, the model is reset after adapting test instance, obviating the need to retain historical knowledge, and thereby accommodating continu- ously shifting test distributions. Nonetheless, these methods are primarily designed for multi-class classification and may not be as effective in the more common multi-label scenario.
2.2 PROMPT LEARNING IN VLMS
Visual-language models (VLMs) (Li al., 2021; Wu et al., 2022; Yang al., 2024b; Li al., 2023; Wan et al., 2024; Zeng et al., 2024; Huang et al., 2024), trained on massive image-text pairs (Sharma et al., 2018; Schuhmann et al., 2022), have demonstrated remarkable proficiency in cross-task leam- ing. To further enhance the transfer abilities of CLIP (Radford et al., 2021), researchers have devel- oped various prompt learning techniques (Zhou et al., 2022b;a; Fu et al., 2024; Li al., 2024b; Wu et al., 2024). For instance, the groundbreaking work CoOp (Zhou et al., 2022b), and its advance- ment CoCoOp (Zhou et al., 2022a), are the first to propose optimizing context vectors to improve the generalization capabilities of CLIP. Maple (Khattak et al., 2023) introduces multimodal prompt learning method, designed to recalibrate both visual and language modalities. Dept (Zhang et al., 2024a) and PromptKD (Li et al., 2024b) take the challenge from the perspectives of knowledge retention and distillation, respectively, to promote robust generalization on novel tasks. Exploit- ing the aligned visual-language space of CLIP (Radford et al., 2021), TAI-DPT (Guo et al., 2023), PVP (Wu et al., 2024) and RC-TPL (Zhao et al., 2024b) propose to regard texts as images for prompt tuning zero-shot multi-label image classification. Investigations like DualCoOp (Sun al., 2022), DualCoOp++ (Hu et al., 2023), and VLPL (Xing et al., 2024) consider more intricate tasks, enhanc- ing multi-label classification capabilities in the partial-label scenario. In contrast, our study focuses on a training-free free paradigm, termed multi-label test-time adaptation, which obviates the need for the source training data and is exclusively the testing instance level.
METHOD
In Sec. 3.1, we review the entropy minimization widely used in TTA. In Sec. 3.2, we highlight the issue that vanilla entropy minimization predominantly increases the probability of top-/ predicted label and propose a new proposition Bound Entropy Minimization (BEM). In Sec. 3.3, we present a Multi-Label Test-Time Adaptation (ML-TTA) framework, incorporating BEM, which binds the highest top predicted labels of both augmented views and paired captions as an individual single label. ML-TTA consists of viewicaption constructing (Sec. 3. and label binding (Sec. 3.3.2).
3.1 PRELIMINARIES
The purpose of Test-Time Adaptation is to utilize each test instance once for immediate adaptation before inference, without any prior assumptions about the test data distribution. For the TTA of VLMs, let Mg denote the CLIP model trained on the training dataset Dimin {(xtrin ((((aii..(rrin)) |x! Xurai , y!rain E yrrinn i=1 ar The TTA approach, TPT (Shu et al., 2022), incorporates the Marginal Entropy Minimization (MEM) objective to adapt Me using solitary instance test from the testing y
dataset Diest_ {(x]ss y
xteat E 2 yest yieest Mn
Ji=1
3

---

Published as a conference paper at ICLR 2025
Given test instance x and set A of N random augmentation functions, x is first augmented N times to generate a set of different views, represented as X {x x = (x } j==1-
TTA aims to minimize the marginal entropy of these augmented views, encouraging the model to perform consistent and confident predictions. The entropy of an augmented view is defined as:
H(p(-1x)")) - - ply=-1xx 11x5*) log(p(y = =[x 1|x)*))
1=1
where I E ylest and is the number of labels in ) The core principle of TPT is to minimize the marginal entropy of the prediction probability distributions of selected confident augmented views by ratio т, thereby encouraging the model to make consistent predictions. After obtaining the average entropy of these confident views, denoted as H, TPT updates the prompt using a single gradient descent step based on H and performs immediate inference on this test instance. Once inference is done, the model's prompt and optimizer are reset promptly for adaptation to the next test instance. Owing to its simplicity and effectiveness, Marginal Entropy Minimization has emerged as de facto standard in modern TTA.
3.2 BOUND ENTROPY MINIMIZATION
can be observed that the TPT method selects a subset of confident augmented views with lower entropy (i.e., high confidence) from X continually minimizing the average entropy of these con- fident views to maintain consistent model predictions across these views. With respect to vanilla entropy minimization within TTA, the following proposition holds.
Proposition Consider the output logits of a confident view x, denoted as a (81, 82, $L), where, without loss of generality, we assume $1 > 82 > ... > SL. It can be deduced that the entropy loss H- = H (p(-1z)) decreases as 81 increases, and H increases as the sum of the remaining logits, Sreat = 2 86, decreases. Formally, this relationship can be expressed as:
OH
0H 0.
H V A detailed proof is provided in the Appendix. Following a single gradient descent update step, we can derive ((-11) (()) a Hl and s((+1) s(t)) a H, where a denotes the learning rate. Therefore, Proposition 1 indicates that the nature of entropy loss is to increase the probability of the most confident class while diminishing the cumulative probability of the rest classes. Hence, when adapting to single-label test instances, the goal of vanilla entropy minimization is solely maximize the probability of the top-/ predicted label, disregarding changes in the probabilities of the remaining labels.
V
Os, <0 and V H
(2)
dS
In contrast, in the context of multi-label test-time adaptation, where the test instance may include a set of positive labels Lp 4,2,,422,.... L. In this case, regardless of whether the top-/ pre- dicted label is the element of the positive label set Lp. the entropy loss will inevitably decrease the prediction probabilities of the other positive labels within Lp while increasing the probability of the most confident class. This may lead to the model overemphasizing the top-1 predicted label and inadequately adapting to the other positive labels. Therefore, for test-time adaptation in multi-label data, we propose the following proposition, termed Bound Entropy Minimization.
Proposition Consider the outpul logits of a confident view x, denoted as (s1, 82, ...,...,,, St.), where, without loss of generality, we assume 81 > 82 > >>>>> 8L- We define the modified logits as s' = (s's, $2...... where s/² a,++5. fs S with a, = $1 11--s s; and s = s; for > Here, a is a constant value that does not participate in differentiation. resulting in s = $1 for all i < k. Let S Ei 8t. For the modified logits s', we define the modified probability p' Softmax(s"), and the modified entropy as H' = -I- p.. p', It follows that:
H' 'H'
0H'
as; <0, 0, V and V H' as 2S
> 0.
(3)
A detailed proof is provided in Appendix. Likewise, after one step of gradient descent optimization, the prediction probabilities of all top-k predicted labels will further increase due to H.' << 0 for all i <= le and V H' > O. Therefore, from Proposition 2, to be robust against distribution shifts with multiple labels, it is crucial to determine the number of positive labels for adapting multi-labe test instances. In the following subsection, we will introduce a novel Multi-Label Time Adaptation framework by employing proposition 2 and incorporating text captions into the adaptation system.
4

---

Published as a conference paper at ICLR 2025
min! (aa
Day
Pillor
Rogeet
View Prongt
Imags
%bJ
Maiic trading
Ri
Ra
RA
Laids Couat
Counts: 2 Counts:
Nouns fiter
Image
Description Base
Stoue Banda
%/u
%
Schi
Sih
big dog sits next
A cat-stts gray tabli.
Vase and cup front bed.
Text
Captirs Proepp
Dog
Shodde
bi
Pillow
Text Beceder
Suject
A: Tuned Porametoes
# Frozen Parmeeeets
ply(T)
Figure Overview of proposed multi- test-time adaption.
3.3 MULTI-LABEL TEST-TIME ADAPTATION
3.3.1 VIEW-CAPTION CONSTRUCTING
Benefiting from the aligned space of CLIP, any image can be assigned most similar caption from an offline text description base based on similarity retrieval. As depicted in Figure 2, given a test image test and collection of random augmentation functions A A2,, AN x is first augmented N times to generate a set of different augmented views. For each augmented view, retrieve the most similar caption from offline text description database to serve as its paired caption. The views generating and caption allocating can be expressed as:
Xiest
[x(et)
test
X
Ax(
T
t
R, (x
(4)
where A: and R4 represents augmentation and retrieval by computing similarity. To streamline the retrieval process, we directly utilize the method proposed PVP (Wu et al., 2024), which employs LLama-2-7B (Touvron et al., 2023) construct the text description base, each text is description of a natural scene containing several categories. Then, CLIP is used to extract text embeddings and construct offline database of size B x d, where B denotes the number of test descriptions and d denotes the embedding dimension. More details of the text description base construction are provided in the appendix.
The goal of TTA is to calibrate the model for a single unlabeled test instance. Clearly, a single instance is insufficient for tuning the entire CLIP model to learn domain-specific knowledge. Con- sequently, as shown in Figure 2, akin to prompt tuning paradigm, we design two identical prompts, referred as view prompt and caption prompt, denoted V and C, respectively. Treating prompt tuning at test-time as way to furnish customized context for individual test instances. Benefiting from the aligned space of CLIP, the representations of images and texts share similar semantic formation, therefore, the paired caption can be considered as a "pseudo image" with accurate textual labels, encouraging the model learn visual-related knowledge and complementary information from views and captions jointly. For I. categories, we initialize the view and caption prompts with template "a photo of a [CLS],"" in which [CLS], represents the j-th label name, e,g., dog or cat, results in v and Cj. Once the paired views and captions are obtained, each view x.est on view prompts and for each caption the
compute the logits for on L caption prompts as below:
(Enc' ((Ene(xx(t) Enc (V;)). EnnT(v;))) Ioe
(Enc" (ttet)) Enc (c;));
(5)
where Encl and EncT represent the frozen image encoder and text encoder of CLIP, (.9) signi- fies the dot product. As stated in proposition 2, the crux of adapting multi-label instance lies in identifying the size of positive label set for each view xtat and caption t.est
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: Label Binding Algorithm
Input: Logits s; before label binding and the size of weak label set A**
Output: Modified logits &; after label binding.
my max, 8tz-
for j = 1 to L do
d.j detach (m, $t))
P Detach from gradient.
if (ay. S kx
then
8,y dij + 8ij
b Bind Siy if j-th label is in highest top-k* predicted labels.
end if else
8,J 8új end if
10 end for
(5:0. S11. 3,L)
11
3.3.2 LABEL BINDING
Obviously, the positive label set for xt_* not feasible to obtain directly. Fortunately, the textual labels for tiest, which we refer to strong label set, can be readily derived through noun filtering, e.g., A truck drives past black car with a suitcase on top. with extracted strong label ser being truck, car, suitcase. Moreover, this set can also serve as a pseudo-positive label set, termed the weak label set, for x Consequently, we treat the size of strong label set as the top-k bound highest
logits of captions, akin to views. The binding operation for stg
mut and 2
can be expressed as:
= ((mm tne
-8ij )+ I(Rank,
S
Sk*7
+ the
I(Rannk ylu sppton >> k=t
(6)
8 = -(mi
-si, 115 glat) )+s I(Rank,
glst +
I[Raank (ota is ss k
where (mix et al. (2017), si (s xtu sxlee binding, m² and m. 1ox indicator function, and Rank(x,s) indicates the descending rank of within s, 1xx² and ttth" denotes the size of weak label set of i-th augmented view and strong label set of i-th paired caption. The algorithm process of label binding is presented in algorithm 1. We provide a detailed label binding process using 3-class classification task in the Appendix.
gle +
(2) employs stop-gradient operation follow VQ-VAE den Oord x
$/2 ..... SiL denotes the logits before $422 ..... Si and sf = (s tex t
denotes the maximum logit of s; and s!" rrespectively, T(() denotes the
To reduce the noise brought by random augmentation and the noise in the caption caused by noisy views, we employ confidence selection to filter out noisy views and captions with higher entropy (i.e., lower confidence). Such noisy views may, due to random cropping augmentation, exclude the target label area, leaving only irrelevant background information. Similarly. the retrieved paired captions for these noisy views will lack any pertinent textual labels. We selected views and captions with lower predicted entropy by ratio т, yielding (x(S)) for views and (É M for captions. see Taking views X! an example, the probability of xtt" L labels denoted as p Softmax ) the average predicted entropy of the filtered low-entropy views can be expressed as:
Σ
Σ -o(==1x
Ilog log(p(y l[x(!*))
avg
Subsequently, the bound entropy optimization objective of view prompt V is to minimize the pre- dicted entropy through f For the objective of caption prompt C, we replace x in Eq.(7) with confident captions ieet obtain Ht
3.3.3 OVERALI OBJECTIVE OF MI.TTA
ML-TTA calculates the predicted bound entropy of confident augmented views and paired captions, optimizing both view prompt and caption prompt with a single step of gradient descent, and simul- taneously increasing the probability of highest top predicted labels. Then, the overall bound entropy

---

Published as a conference paper at ICLR 2025
loss is given by:
HBEM = A + Frase
(8)
After optimizing the prompts, ML-TTA immediately infers the test instance x and resets the parameters of the prompts (V and C) and the state of optimizer to adapt to the next test instance. During the inference phase, we separately compute the similarity between the view prompt V and the test instance x as well as the similarity between the caption prompt C and the test instance xles, and directly add these two similarities to obtain the final prediction result.
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
Benchmarks. We utilize the widely employed CLIP (Radford et al., 2021) model as source model and select the multi-label datasets VOC (Everingham et al., 2010), MSCOCO (Lin et al., 2014), and NUSWIDE (Chua et al., 2009) as target domains. The VOC dataset includes 20 categories, covering both VOC2007 and VOC2012 versions, which contain 4.952 and 5,823 test images, respectively. The MSCOCO dataset extends the category range to 80, and for testing purposes, we employ the validation sets of COCO2014 with 40,504 images and COC02017 with 5,000 images, as the test set labels are not accessible. The NUSWIDE dataset includes 81 categorics with a total of 83,898 test images of lower resolution, which presents a broader category spectrum than MSCOCO.
Implementation details. All experiments are based on the CLIP model, encompassing RN50, RN101, VIT-B/32, and ViT-B/16 architectures, each consisting of image encoder and a corre- sponding text encoder. For the initialization of the view and caption prompts, we employ the token embedding of the "a photo of a" hard prompt as initialization weights and another using learned prompts from CoOp (Zhou et al., 2022b) and MaPLE (Khattak et al., 2023). The learning rate for the view prompt is le-2, while for the caption prompt is 1e-3. For all settings, multi-label test- time adaptation is performed on single instance, i.e., the batch size is 1. The ratio for filtering confident views and captions O.I. The optimizer is AdamW (Loshchilov & Hutter, 2019) with a single update step, followed by immediate inference on the test instance. Following PVP (Wu al., 2024), we collect 100k text descriptions for each dataset, resulting in a total size of 300k text description base. All experiments are evaluated by the mean Average Precision (mAP) metric, defined as mAP Ele AP,, where L is the number of categories, and AP; is the under the Precision Recall curve for the i-th category.
4.2 COMPARISONS WITH STATE-OF -HEEE--F-TEEARRR
To our knowledge, our work is the first to investigate the feasibility of traditional entropy minimiza- tion in the multi-label setting. Therefore, in this section, select the original CLIP model and other SOTA methods for multi-class scenarios as baselines, including episdoic methods that do not require retaining historical knowledge (TPT Shu al. (2022), DiffTPT Feng al. (2023), RCLF Zhao et al. (2024a)) and online methods that do (DMN Zhang et al. (2024b), TDA Karmanov et al. (2024)).
Results different architectures. Table compares ML-TTA with both online and episdoic TTA methods on different CLIP (Radford et al., 2021) architectures, demonstrating the superior performance across various multi-label datasets. Specifically, for the RN50 and RN101 architectures on COC0201 2012/11177777 (Lin et al., 2014) datasets, MI-TTA achieves 4-55% improvement in mAP over the original CLIP (Radford al., 2021) model, whereas TPT (Shu al., 2022) and DifTTPT (Feng et al., 2023) yield only 1% enhancement. Despite introducing dual -uel-y network knowledge from historical samples, DMN (Zhang et al., 2024b) and TDA Karmanov et al. (2024) present a slight performance decline, due to intensifying the optimization bias towards top-/ label. Notably, RLCF (Zhao et al., 2024a) employs a reinforcement learning- based knowledge distillation and more adaptation steps, resulting in a catastrophic degradation in the multi-label adaptation performance for smaller models due to excessive optimizations of top-/ label. On the VOC2012/2017 (Everingham et al., 2010) datasets, TPT and Di also show 1~ 2% decrease in performance compared to CLIP, whereas ML-TTA still maintains 2~3% performance improvement, indicating the robustness of MI-TTT in multi-label adaptation across various model architectures and datasets.
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with CLIP and SOTAs on adapting multi-label instances with different architectures.
Methods
Epsdoic COcO0010 COCO2017 VOC2007 VOC2012 NUSWIDE Average
CLIP [CCL 2022]
47.53
47.32
75.91
74.25
41.53
57.31
DMN [CVPR 2024]
X
44.54
44.18
74.87
74.13
41.32
55.81
RN-50 50
TDA CVPR 2024]
X
48.91
49.11
76.64
75.12
42.34
58.42
TPT [NeurPPS 2022]
V
48.52
48.51
75.54
73.92
41,97
57.69
DIfTTP ICCV 2023]
V
48.56
48.67
75.89
74.13
41.33
57.72
RLCF [CCR 2024]
V
36.87
36.73
65.75
64.73
29.83
46.78
ML.TTA (Ours)
51.58
51.39
78.62
76.63
42.53
60.15
CLIP [ICML2022
V
48.83
48.15
76.72
74.21
41,93
57.97
DMN [CVPR 2024]
X
46.28
45.44
76.82
75.32
42.71
57.31
RN-101
TDA [CVPR 2024
X
50.19
49.78
78.12
77.13
43.13
59.67
TPT [NeurIPS 2022]
49.71
48.89
74.82
73.39
43.10
57.98
DIfTTPT JCCV 2023]
V
49.45
49,19
74.98
74.31
42.93
58.17
RLCF [CLR 2024]
40.53
39.79
71.21
69.63
31,77
50,59
ML-TTA (Ours)
52.92
52.24
78.72
78.13
43.62
61.13
CLIP [CML.202
V
50.31
50.15
77.18
76.85
42.90
59.48
DMN [CVPR 2024]
X
49.32
48.13
77.42
76.60
43.41
58.98
VIT-B/32
TDA [CVPR 2024]
X
51.23
51.49
77.62
77.12
44.13
60.32
TPT [NeurIPS 2022]
V
48.12
48.63
74.21
71.93
43.63
57.30
DIATPT JCCV 2023]
V
48.73
49.19
74.50
72.98
43.42
57.76
RLCF [CLR 2024]
50.28
49.59
77.12
76.83
43.29
59.42
ML.TTA (Ours)
52.83
52.99
78.70
77.97
44.12
61.32
CLIP [CML 2022]
V
54.42
54.13
79.58
79.25
45.65
62.61
DMN [CVPR 2024]
X
52.52
52.37
79.83
79.67
46.27
62.13
VIT-B/16
DART [AAAT 2024]
X
54.73
54.68
79.91
78.56
45.91
62.76
TDA [CVPR 2024]
X
55.21
55.46
80.12
79.92
46.72
63.49
TPT [NeurIPS 2022]
V
53.32
54.20
77.54
77.39
46.15
61.72
DIfFTPT ICCV
53.91
54.15
77.93
77.24
46.13
61.87
RLCF [CCR 2024]
V
54.21
54.43
79.29
79.26
43.18
62.07
ML-TTA (Ours)
57.52
57.49
81.28
81.13
46.55
64.80
For the vision transformer series architectures, compared CLIP (Radford et al., 2021), ML-TTA consistently achieves ~~ 4% mAP improvement on the COCO2014/2011 (Lin al., 2014) and VOC2007/2012 (Everingham et al., 2010) datasets. However, most TTA methods, except TDA (Kar- manov et al., 2024) and DART (Liu et al., 2024b), exhibit a slight performance decrement, par- ticularly among episodic methods. Additionally, we observed an intriguing observation: all TTA methods, excluding RLCF (Zhao et al., 2024a), fail to substantially enhance the mAP performance of CLIP (Radford al., 2021) on the NUSWIDE (Chua et al., 2009) dataset, with an improvement of merely about 1%. This may be attributed to the low image resolution of NUSWIDE dataset, where random data augmentation struggles to preserve sufficient visual information. Consequently, adapting to multi-labels for small targets may become research topic in the future.
Results on different prompt initialization. For this comparison, we adopt the learned prompt from CoOp (Zhou et al., 2022b) and Maple (Khattak et al., 2023) to initialize the prompt weights, replac- ing the template "a photo the application of both CoOp and Maple prompt weights in ML-TTA framework results in sig- datasets. For instance, the mAP
[CLS]" employed the original CLIP model. As shown in Table 2,
nificant enhancement of over 4% in mAP on the COCO2014/200 increases from 47.53% to 51.58% and from 47.32% to 51.39% prompt initialization, whereas other episdoic methods, TPT (Shu et al., 2022) and DiffTPT (Feng al., 2023), yield improvements of more than 1.5%. Moreover, MI-TTA also surpasses
COC02014/2017 with CoOp
8

---

Published as a conference paper at ICLR 2025
Table Comparison with SOTAs on adapting multi-label instances with different prompt initialization.
Methods
Epsdoic (CCo20004 COC02017 VOC2007 VOC2012 Average
CoOp [UCV2022]
56.12
56.35
79.14
77.85
46.74
63.24
CoOp
IDA [CVPR 2024]
X
56.93
57.15
80,20
78.58
47.82
64.13
TPT [NeurIPS 2022]
V
55.35
55.23
79.72
77.85
47.27
63.08
DINTPT [CCV 2023]
V
55.30
55.47
79.86
77.61
47.13
63.07
RLCF [[CLR 2024
V
56.72
56.18
80.15
78.24
47.62
63.78
ML-TTA (Ours)
59.68
59.33
83.17
81.36
48.12
66.33
Maple [CVPR2023]
V
62.18
62.35
85.34
84.79
48,42
68.62
Maple
TDA [CVPR 2024]
x
63.25
63.64
85.76
84.15
49.55
69.27
TPT [NeurIPS 2023]
V
63.36
63.75
85.04
83.92
48,90
69.01
DIfTTPT ICCV
62.93
63.14
85.15
83.78
48.81
68.76
RLCF [CLR 2024]
V
62.84
62.90
85.35
85,28
49.37
69.15
MI.-TTA (Ours)
64.75
64.86
86.40
85.69
50.21
70.38
TDA (Karmanov et al., 2024), which is designed by dynamically employing the historical sample knowledge, on both CoOp and Maple prompt initialization across all datasets.
Results on different label counts. Apart from the analysis of architecture and prompt initialization weights, we explore a more challenging scenario Table 3, where the COCO2014 dataset is divided into subsets with incrementally increasing numbers of im- age labels per part, e.g., (1,2) represents the number of labels L per image is either or 2. When L 21, TPT achieves only a neg- ligible improvement compared to CLIP and shows large considerable performance degradation in other situations as well as DifFTPT. RLCF improves significantly when {1, but its perfor- mance sharply declines as L increases. In contrast, our ML-TTA framework outperforms CLIP across all situations, demonstrating that ML-TTA not only can address the distribution shifts during testing but also effectively handle varying numbers of labels in testing instances.
Table Results different label counts.
Methods
{1,2]
{3.4]
{5,6,7)
{>=88
CLIP [CMML 2022]
62.76
55.41
49.89
41.07
TPT NNurlPS 2023]
62.88
53.05
45.57
37.43
Di [CCC 2023
61.97
52.67
44.32
36,89
RLCF [CLR 2024]
66.01
51.65
43.32
35.08
MI.TTA (Ours)
67.14
57.59
51.68
41.32
Results on adaptation complexity. Furthermore, analyze adapting time per test instance with methods that also do not require retaining historical knowledge. Table 4 shows that ML-TTA presents significant advantage compared to DiffTPT, which involves generating multiple pseudo-images via diffusion model, and RLCF, which requires dis- tillation from a teacher model along with more gradient update steps. Compared to the benchmark TPT, ML-TTA increases adapting time due to simultaneous optimizing view and caption prompts.
Table 4: Results on adaptation complexity.
Methods
TPT
(IETPT|RRCYS ML-TTA
Adapting Time 0.21s mAP
0.41s
0.45s
0.24s
48.52
48.56
36.87
51.58
Table 5: Ablation studies of different components.
4.3 ABLTION STUDIES.
RN50
VIT-B/16
Methods
Different components. In Table 5, we dis- cuss the effectiveness of different compo- nents within our proposed ML-TTA frame- work on the COC02014 and VOC2007 datasets, including view prompt (VP. i.e. TPT (Shu et al., 2022)), caption prompt (CP), and Bound Entropy Mini- mization (BEM). Across both RN50 and ViT-B/16 architectures, BEM consistently enhances the mAP performance of VP. CP, and VP+CP, which indicates the reasonable effective- ness of our proposed Bound Entropy Minimization objective. Furthermore, we observe that CP and
COC02014
VOC2007
COC02014
VOC2007
VP (i.e., TPT)
48.51
75.52
53.32
77.57
VP+BEM
48.96
76.31
53.58
77.89
CP
49.12
76.16
55.14
78.93
CP4BEM
49.54
76.75
55.64
79.58
VP+CP
51.22
77.98
$7.14
80.85
VP+CP+BEM
51.58
78.62
57.52
81.28

---

Published as a conference paper at ICLR 2025
CP+BEM always achieve superior performance compared to VP and VP+BEM in all settings. Such phenomenon shows treating text as a pseudo-image with a known label set to adapt multi-label test instance is more reliable than augmented views, the positive label set of views is pseudo.
4.4 FURTHER ANALYSIS
58
57
Table 6: Comparison with binary cross-entropy loss.
56
ML-TA
55
RN50
VIT-B/16
Methods
54
COC02014
VOC2007
COCO2014
VOC2007
53
CLIP
47.53
75.91
54.42
79.58
52
VP+CP+BCE
48.39
75.75
54.51
78.59
16 32
128
Number of augmented views
VP+CP+BEM
51.58
78.62
57.52
81.28
Figure 3: Results different number of views.
Loss functions. Here, we conduct comparison between Bound Entropy Minimization (BEM) and the conventional binary cross-entropy (BCE) loss function in multi-label classification tasks. Specifically. regarded the weak label set of confident views as hard labels for those views and the strong label set of confident captions as hard labels for those captions, then using BCE loss optimize the view and caption prompts. The results are shown in Table 6. Compared to CLIP, the mAP improvement using BCE loss on the COCO2014 is less than 1%. In contrast, our BEM objective surpasses BCE loss by 3~4% in mAP across all benchmarks, which demonstrates BEM not only more effective than vanilla entropy minimization but also more robust compared to binary cross-entropy loss. BCE loss not suitable for optimizing a single test instance.
Number of augmented views. Following TPT (Shu et al., 2022), we conduct parameter experiments of different numbers of augmented views on the COCO2014 dataset using ViT-B/16 architecture. As depicted in Figure 3, as the number of views increases from to 128, the mAP performance of RLCF and MI.TTA both show an upward trend and begin to stabilize at views. Surprisingly, the performance curve of TPT does not have any regularity, which implies that vanilla entropy minimization, by focusing only on the label with the highest probability, leads to unstable adaptation for lulti-label instances.
Table Results on different numbers of retrieved captions.
Number of retrieved cap- tions. We also investigate the impact of allocating dif- ferent numbers of retrieved captions for each augmented view on the performance of ML-TTA. As shown in Ta- ble 7, when only one caption is allocated to each view, ML-TTA outperforms CLIP or TPT by 3-4%. As the number of captions increases, the performance of ML-TTA gradually improves until stabilizes. For the VOC2007 dataset, too many captions can lead to a slight decrease in perfor- mance, as captions that are not highly similar to the view may introduce noisy positive labels that do not exist in the corresponding view.
Datasets
CLIIP
TPT
2
8
16
32
64
128
RNS0
coco2014
47.53
48.52
51.35
51.37
51.41
51.49
51.58
51.59
51.55
51.48
VOC2007
75.91
75.54
78.29
78.33
78.48
78.54
78.61
78.59
78.53
78.42
VTT-B/16
coco2014
54.42
53.32
57.23
57.33
57.41
57.48
57.49
57.52
57.55
57.58
VOC2007
79.58
77.54
81.06
81.12
81.21
81.24
81.28
81.19
81.15
80.98
CONCLUSION
In this paper, we investigate test-time adaptation framework (ML-TTA) designed for multi-label data without making any presumptions about the distribution of the test instances. The proposed Bound Entropy Minimization (BEM) objective overcomes the limitation of the vanilla entropy loss, which only optimizes the most confident class. By conceptualizing paired captions as pseudo-viens views with a known label set, ML-TTA employs BEM to adapt to multi-label test instances by allocating weak label set to each augmented view and strong label set to each paired caption, binding the top- k predicted labels with the highest probabilities. Extensive experiments on the MSCOCO, VOC, and NUSWIDE datasets demonstrate that ML-TTA framework outperforms the souree model CLIP and other tfate-of-the-ar test-time adaptation methods, across various model architectures, prompt initialization, and varying label scenarios.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported in part by Key Laboratory of Target Cognition and Application Technology (2033-CXTT-LC-005),
REFERENCES
Zhixiang Chi, Li Gu, Tao Zhong, Huan Liu, Yuanhao Yu, Konstantinos N. Plataniotis, and Yang Wang. Adapting distribution shift by visual domain prompt generation. In ICLR. OpenRe- view.net, 2024.
Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus-wide: a real-world web image database from national university of singapore. In CIVR, 2009.
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisser- man. The pascal visual object classes voc challenge. IJC,, 88(2)300-333, 2010.
Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In ICCV, pp. 2704-2714, 14, 2023.
Zhongtian Fu, Kefei Song, Luping Zhou, and Yang Yang. Noise-aware image captioning with progressively exploring mismatched words. In AAAI, pp. 12091-12099, 2024.
Junyu Gao, Xuan Yao, and Changsheng Xu. Fast-slow test-time adaptation for online vision-and- language navigation. In ICML. OpenReview.net, 2024.
Zixian Guo, Bowen Dong, Zhilong Ji, Jinfeng Bai, Yiwen Guo, and Wangmeng Zuo. Texts as images in prompt tuning for multi-label image recognition. In CVPR, pp. 2808-2817, 2023.
Ping Hu, Ximeng Sun, Stan Sclaroff, and Kate Saenko. Dualcoop++: Fast and effective adaptation to lulti-label recognition with limited annotations. TPAMI, 2023.
Longfei Huang, Xiangyu Wu, Jingyuan Wang, Weili Guo, and Yang Yang. Refining Visual Percep- tion for Decoration Display: A self-enhanced deep captioning model. ACML,, volume 260, pp. 527-542, 05 08 Dec 2024.
Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In CVPR, pp. 14162-14171, 2024.
Muhammad Uzair Khattak, Hanoona Abdul Rasheed, Muhammad Maaz, Salman H. Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, pp. 19113-19122, 2023.
Jae-Hong Lee and Joon- Joon-Hyuk Chang. Stationary latent weight inference for unreliable observations from online test-time adaptation. In JCML, 2024.
Jonghyun Lee, Dahuin Jung, Saehyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang, and Sun- groh Yoon. Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. In ICLR, 2024.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu-Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS, pp. 9694-9705, 2021.
Junnan Li, Dongxu Li, Silvio Savarese. and Steven C. H. Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, volume 202, pp. 19730-19742, 2023.
Yiming Li, Xiangdong Wang, and Hong Liu. Audio-free prompt tuning for language-audio models. In ICASSP, pp. 491-495, 2024a.
Zheng Li, Xiang Li, Xinyi Fu. Xin Zhang, Weigiang Wang. Shuo Chen, and Jian Yang. Promptkd: Unsupervised prompt distillation for vision-language models. In CVPR, pp. 2024b.

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, volume 8693, pp. 740-755, 2014.
Jiaming Liu, Senqiao Yang, Peidong Jia, Renrui Zhang, Ming Lu, Yandong Guo, Wei Xue, and Shanghang Zhang. Vida: Homeostatic visual domain adapter for continual test time adaptation. In ICLR, 2024a.
Zichen Liu, Hongbo Sun, Yuxin Peng, and Jiahuan Zhou. Dart:dual-mode adaptive online prompt- ing and knowledge retention for test-time adaptation. In Artificial Intelligence, pp. 14106-141 14,, 2024b.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Jiawei Ma, Po-Yao Huang, Saining Xie, Shang- Wen Li, Luke Zettlemoyer, Shih-Fu Chang, Wen- 2024.
Tau Yih, and Hu Xu. Mode: Clip data experts via clustering. In CVPR, pp.
Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. In NeurlPS, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139, pp. 8748-8763, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurlPS, volume 35, pp. -52294, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, pp. 2556-2565, 2018.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NewrlPS, 2022.
Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficient continual test-time adaptation via self-distileed regularization. In CVPR, pp. 11920-11929 2023.
Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. In NeurlPS, volume 35, pp. 30569-30582, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet. Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv, 2023.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn ing. In Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6306- 2017.
12

---

Published as a conference paper at ICLR 2025
Fengqiang Wan, Xiangyu Wu, Zhihao Guan, and Yang Yang. Covir: Coordinating cross-modal consistency and intra-modal relations for langualaa retrieval. In ICME, pp. 1-6, 2024.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
Jiaheng Wei, Harikrishna Narasimhan, Ehsan Amid, Wen-Sheng Chu, Yang Liu, and Abhishek Kumar. Distributionally robust post-hoc classifiers under prior shifts. In ICLR, 2023.
Tong Wei, Zhen Mao, Zi-Hao Zhou, Yuanyu Wan, and Min-Ling Zhang. Learning label shift cor- rection for test-agnostic long-tailed recognition. In JCML, 2024.
Xiangyu Wu, Jianfeng Lu, Zhuanfeng Li, and Fengchao Xiong. Ques-to-visual guided visual ques- tion answering. In ICIP, pp. 4193-4197, 2022.
Xiangyu Wu, Qing- Yuan Jiang, Yang Yang, Yi-Feng Wu, Qing-Guo Chen, and Jianfeng Lu. Tai++text as image for multi-label image classification by co-learning transferable prompt. In IJCAI, 2024.
Xin Xing, Zhexiao Xiong, Abby Stylianou, Srikumar Sastry, Liyu Gong, and Nathan Jacobs. Vision- language pseudo-labele for single-positive multi-labe learning. In CVPR, pp- 7799-7808, 2024.
Yang Yang, Fengqiang Wan, Qing- Yuan Jiang, and Yi Xu. Facilitating multimodal classification via dynamically learning modality gap. In NeurIPS, 2024a.
Yang Yang. Wenjuan Xi, Luping Zhou, and Jinhui Tang. Rebalanced vision vision-language retrieval considering structure-aware distillation. T7P, 33:6881 16892, 2024b.
Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark A. Hasegawa-Johnson, Yingzhen Li, and Chang D. Yoo. C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. In JCLR. OpenReview.net, 2024.
Yan Zeng, Xinsong Zhang, Hang Li, Jiawei Wang, Jipeng Zhang, and Wangchunshu Zhou. X2-vlm: All-in-one pre-trained model for vision-language tasks. TPAMI,
2024.
Zhang, Shihan Wu, Lianli Gao, Heng Tao Shen, and Jingkuan Song. Dept: Decoupled prompt tuning. In CVPR, pp. 12924-191 12933, 2024a.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. NeuriPS, 2022.
Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision-language models. In CVPR, pp. 28718- 28728, 2024b.
Bowen Zhao. Chen Chen, and Shu- Shu-Tao Xia. Delta: Degradation-fiee free fully test-time adaptation. In ICLR, 2023.
Shuai Zhao, Xiaohan Wang, Linchao Zhu, and Yi Yang. Test-time adaptation with clip reward for zero-shot generalization in languaee models. In ICLR, 2024a.
Xiongjun Zhao, Zheng- Yu Liu, Fen Liu, Guanting Li, Yutao Dou, and Shaoliang Peng. Report- concept textual-prompt learning for enhancing x-ray diagnosis. In ACM MM, 2024b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. CVPR, pp. 16795-16804, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. IJCV, 13009))2337-24444 2022b.
13

---

Published as a conference paper at ICLR 2025
Appendix for Multi-Label Test-Time Adaptation with
Bound Entropy Minimization
A PROOF
A.1 PROOF OF PROPOSITION
Proposition 1. Consider a model's output logits of selected view denoted as (81, 8,,..., 81.), where without loss of generality, we assume 81 > 82 > ... 81. follows that the entropy loss H H (p(([z)) decreases 31 increases, and H increases the sum of the remaining logits, Speat Σ 8/, decreases. Formally, this can be written as:
HH
0H
and
O.
0s1
a
Proof. We denote the predicted probability p(y = (2))
Σe exp pr for simplicity, where 8;
the logit of the i-th category. We first calculate the partial derivative of 8; with respect to pr:
Opr
expor 81
0s:
0s:
Σj=1 expsj
51,4 exps: St
exp St
(10)
プラ= exp Sj
exp 51
EE/=1 exp 83)
2
S1 PiP:
where 81,5 only if j. else is 8,,5 O. We can now directly calculate the partial derivative of H for St-
0H
L
log Pt
08;
8s;
1=1
L
Op
8pt
log pi + Pt
Ost
pt Os,
1=1
1
(ói log PiP log pt + St, - PIP.)
log + Pt
Prps log pipe)
(11)
1=1
(p. loglog + pc)
Σ pt
Pip; log Pt PiP.)
(pipe log p: - Pips log pt + PiP: PiPs)
PiPs log P
pt
where the fourth equivalent uses the property of 8.g and fifth equivalent uses ΣL_ Pt = 1. Since
we assume 51 > $2 > S,, then the probabilities have the same order pl P2 > PL therefor:
0H
Σ
L
PIP log Pi
Σ 1=2
L
piP log P1
<
0
(12)
0s1
pt
14

---

Published as a conference paper at ICLR 2025
0 for all / > second inequality, first calculate the sum of the partial derivative of H for all logits.
log Pi
1, therefor proof the first inequality in proposition 1. To prove the
L OH
Pi Pt
pipe log
0s;
1=1
i=1
1=1
(DiPs logp, PiPs log
(13)
==1 1-1
(pip: logp; P.pi logp:)
where we change the position index and / for the second term in the double summation to get the third equivalent. Now the second inequality is easy to get:
0H
0H
0Ses
0s:
3s:
2
0H
0s;
1=2
(14)
OH
0H
0s;
081
UH 0
081
A.2 PROOF OF PROPOSITION 2
proposition 2. Consider model's ousput logits of selected view x, denoted as ($1, S2,, $2.... ..... SL), where without loxs of generality, we assume 81 >> 82 > ... 81.- We define the modified logits as s' (6 ./,))..8'')) where s = 8; for < k with a; 81 8; and = 8, for k. Here, a, is a constant that does not participate in differentiation, resulting in s for all i S k. Let Sreat = 84. For the modified logits s', define the modified probability p' Softmax(s"),
and the modified entropy as H' = -t p/ logp', It follows that:
0H
OH 0S
Visk k and
0.
(15)
0s;
Proof. With the assumption 81 > >2..... 81 and the definition of s', we have 8 = 8/2
8/ 8 Use the conclusion in proposition for S k. we have:
0H OH' dsl
OH'
the pay log
Pizk log pf
x
Σ 1=1
Σ
< 0
as: Os ds, 0s
Pl
p
(16)
1=k+1
Vi < k
where bf pf. for i < & and as 5 8/ 5/ >
s/ Similar to the proof of 0, which has been proved in Equation. 13 to
proposition 1, we use the conclusion of EL_
Anl
Dz.
15

---

Published as a conference paper at ICLR 2025
prove the second inequality.
0H' SS
ÖH
88,
08
k+
0H
0s;
(17)
OH
ÖH
dst
0st
i=1
i=1
0H
Os:
B DETAILED LABEL BINDING PROCESS.
In this section, present certain example to showcase the calculation of Label Binding 3.3.2.
Label binding refers making the top-k predicted logits equal, as expressed below:
((m,
Jay >x[(Rank_(s) xlsa
S + xI((Rak_[<r)
(18)
Since label binding (making equal) is non-differentabble, we employ the stop-gradient operation in VQ- VQ-VAE van den Oord et al. (2017) for backpropagation, i.e. ((m," S1 siy to perform label binding.
Taking a 3-class classification task as an example with class labels of (1, assuming kx"" is 2, jest
and the label binding process s = [0.9, 0.7, 0.3] + = [0.9, 0.3). 82) represents the logit of the j-th class the i-th augmented view after label binding, e.g., 8/2 test
changes from 0.7 0.9. m, denotes the maximum value of s, which is 0.9. II()) is the indicator function. Rank_(a, b) The process for computing
indicates the descending rank of within b, e.g., (aank the bound logit for each class is as follows:
(00.9 0.9) + 0.9) 1(Rank (0.9,s) 0.9 I(Rank (0.9, > 2)
0.9 x [(1 2) 0.9 I(1>2)
0.9,
(0..9
0.7) + 0.7) (aRakk(00...p s 2) + 0.7 x I(Rank (0.7,s) > 2)
0.9 x 1(2 2 + 1 2)
(19)
0.9,
8/
(00.9 0.3) + 0.3) [[aakk 2) + 0.3 x [(Rank.(003,s)) > 2)
0.9 x 1(3 < 2) + 0.3 133
0.3,
label binding process changes the logits from 00.9 0..,7.7,.33 00.9 00.9,0.9,0.3)
C TEXT DESCRIPTION BASE CONSTRUCTION
Here, we present the construction of the text description base using Large language models (LLMs). Initially, for set of labels, denoted as L = {41, L2 (2.,,,,,, where represents the total number of labels across all multi-label datasets. Following PVP (Wu et al., 2024), we define prompt template to instruct LLama-2-7B (Touvron et al., 2023), generating descriptions that describe a nature scene, which is follows:
PROMPT: Make a sentence to describe a photo. Requirements: Each sentence should be less than 15 words and include keywords: (li, ..,............
where 4 .4, is a subset of with i < 5. We randomly sample categories from L and input these categories along with the prompt template into LLMs automatically generate text
16

---

Published as a conference paper at ICLR 2025
descriptions. After obtaining generated descriptions, employ the nouns filtering strategy used in PVP to extract textual labels for each description. Some examples are illustrated below:
1. A hot dog toaster is positioned next to stop sign.
2. A group of girls enjoying game of frishee while sitting on chairs.
3. The little boy dreams of becoming a pilot as he falls asleep with his aeroplane.
4. Remotes control the TV, allowing people to enjoy their favorite shows.
5. A motorbike speeds past man wearing a tie, as he holds a wine glass in one hand.
where the underlined words indicate the textual labels extracted from corresponding description. However, due to the uncontrollable quality and relevance of the paired captions generated by LLMs, these captions may not always accurately represent the image contents. In real-world scenarios, be- sides adopting confident-basee filtering strategy to filter out views and captions with high entropy (e.g., low confidence), we can also explore more robust strategies to retrieve paired captions, such as, constructing high-quality and content-rich text description databases, ensembling label sets from multiple captions, or improving the similarity retrieval strategy, thereby reducing the impact of noise on the model's adaptation.
17


Reviews:
[{"review_id": "ZRy8b5I61j", "reviewer": "Reviewer_jmBy", "summary": "This paper focuses on test time adaptation under a multi-label setting, this is an early work in this field.  This paper first analyzes why widely used entropy loss is not helpful in multi-label settings, and proposes a new method to adapt with multi-label. Then, the author proposes the view prompt and caption prompt to adapt the model for each instance. The experiments on three datasets show the effectiveness of the proposed method.", "questions": "1. The equ(6) is quite difficult to understand, more explanation is needed to show the meaning. The author should explain more about how weak labels and strong label is recognized in the proposed method, and the meaning of $\\hat{s}_{ij}^{x^{test}$.\n2. It is unclear which parameter is learnable in this method. The authors need to clearly point out all the learnable parameters.\n3. The authors could explain more about the motivation of the view prompt and caption prompt, and why they are useful for this setting.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper focuses on an important question.\n2. This paper has a good theoretical analysis.\n3. The proposed method achieves better result than baselines.", "weaknesses": "1. The equ(6) is quite difficult to understand, more explanation is needed to show the meaning. The author should explain more about how weak labels and strong label is recognized in the proposed method, and the meaning of $\\hat{s}_{ij}^{x^{test}$.\n2. It is unclear which parameter is learnable in this method. The authors need to clearly point out all the learnable parameters.\n3. The authors could explain more about the motivation of the view prompt and caption prompt, and why they are useful for this setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "me5PerzBY9", "reviewer": "Reviewer_97yB", "summary": "This paper proposes a novel method for Multi-Label Test-Time Adaptation (ML–TTA) using a technique called Bound Entropy Minimization (BEM). Unlike traditional test-time adaptation (TTA) that optimizes for the most confident single-label prediction, BEM increases the confidence of the top-k predicted labels simultaneously. This approach addresses the challenges associated with multi-label data where prioritizing one label can reduce the adaptation effectiveness for others. The framework also incorporates paired captions as pseudo-positive labels to guide adaptation. Experiments conducted on MSCOCO, VOC, and NUSWIDE datasets demonstrate that ML–TTA outperforms existing methods and the original CLIP model, showcasing superior adaptability across diverse architectures and prompt setups.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper demonstrates robust experimentation across diverse datasets (MSCOCO, VOC, NUSWIDE) and architectures (e.g., RN50, ViT-B/16), showcasing the generalizability and efficacy of the proposed method.\n2. The introduction of the Bound Entropy Minimization (BEM) for Multi-Label Test-Time Adaptation (ML–TTA) is a significant theoretical and practical advancement. It effectively addresses the challenges inherent in multi-label test-time adaptation, a space where traditional single-label approaches like entropy minimization fall short.", "weaknesses": "1. The method section, particularly the mathematical formulations and algorithmic details, could be more clearly presented. The explanations surrounding the implementation of label binding and how the paired captions are retrieved need additional clarity for readers less familiar with the intricate mechanisms of vision-language model adaptations.\n2. While the paper effectively shows ML–TTA's superiority over traditional methods, it would benefit from a more detailed discussion about the choice of baseline methods and potential reasons for their relative underperformance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dGLraDpbXp", "reviewer": "Reviewer_bCKi", "summary": "This paper introduces a Bound Entropy Minimization method for improving test-time adaptation in multi-label scenarios. BEM addresses the challenge of adapting multiple labels simultaneously. By integrating textual captions to determine the number of positive labels, the method enhances the confidence of several top predicted labels. The proposed Multi-Label Test-Time Adaptation (ML–TTA) framework leverages both visual and textual data, leading to superior performance across various datasets compared to state-of-the-art techniques.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed Bound Entropy Minimization (BEM) method presents an innovative solution to improve test-time adaptation in multi-label scenarios.\n2. The use of paired captions as pseudo-labels is a clever strategy to determine the number of positive labels for each test instance.\n3.  It considers both visual and textual modalities, optimizing for a more robust adaptation to distribution shifts.\n4. The figures are well presented.", "weaknesses": "1. More detailed motivation behind the model design is preferred. It is important to explain why the authors propose the method in this work.\n2. The proposed method involves multiple steps, including view augmentation, caption retrieval, and label binding, which might introduce complexity in practical implementation. Simplifying the process could enhance usability.\n3. The effectiveness of the method heavily relies on the quality and relevance of the paired captions. In real-world scenarios, captions might not always accurately represent the image content, which could affect performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qVrmkkBcIn", "reviewer": "Reviewer_LTFP", "summary": "The paper presents a novel approach to Test-Time Adaptation (TTA) for multi-label scenarios using a method termed Bound Entropy Minimization (BEM). The paper is well-structured, the problem statement is clear, and the proposed solution is innovative. The integration of view and caption prompts and the application of BEM to meet the test time adaptation are innovative to some extent. However, there are  some details should be clarified.", "questions": "See the weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper is well-structured, the problem statement is clear, and the proposed solution is innovative. \n2) The integration of view and caption prompts and the application of BEM to meet the test time adaptation are innovative to some extent.\n3) Compared with the latest and most advanced methods, the method in this paper achieves the best performance.", "weaknesses": "1) In your paper, the choice of top-k seems to be very important, so how do you determine the setting of k? You said \"we retrieve a paired caption with derived textual labels for each view, which then serves as weak label set of size k for the corresponding view.\" How do you make sure the selected weak label set is reliable?\n2) I can not see any explanation about the \"augmented view\" in this paper, what is the definition of it and what effort does it have in the framework?\n3) The comparison methods you selected in the paper may be not designed for multi-label datasets, so is this comparison fair? Could you add more ML-TTA specific framework to the results?\n4) Some details: Table 1 lacks a description of evaluation metric; Marking the second-best result in the experimental results is more beneficial to the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)