ICLR 2025

Published as a conference paper at ICLR 2025
MULTI-LABEL TEST-TIME ADAPTATION WITH BOUND
ENTROPY MINIMIZATION
Xiangyu Wu1,2, Feng Yu1, Qing-Guo Chen2, Yang Yang1∗, Jianfeng Lu1∗
1Nanjing University of Science and Technology
2Alibaba International Digital Commerce Group
ABSTRACT
Mainstream test-time adaptation (TTA) techniques endeavor to mitigate distri-
bution shifts via entropy minimization for multi-class classification, inherently
increasing the probability of the most confident class. However, when encoun-
tering multi-label instances, the primary challenge stems from the varying num-
ber of labels per image, and prioritizing only the highest probability class in-
evitably undermines the adaptation of other positive labels. To address this issue,
we investigate TTA within multi-label scenario (ML–TTA), developing Bound
Entropy Minimization (BEM) objective to simultaneously increase the confi-
dence of multiple top predicted labels. Specifically, to determine the number of
labels for each augmented view, we retrieve a paired caption with yielded tex-
tual labels for that view. These labels are allocated to both the view and cap-
tion, called weak label set and strong label set with the same size k. Follow-
ing this, the proposed BEM considers the highest top-k predicted labels from
view and caption as a single entity, respectively, learning both view and cap-
tion prompts concurrently. By binding top-k predicted labels, BEM overcomes
the limitation of vanilla entropy minimization, which exclusively optimizes the
most confident class. Across the MSCOCO, VOC, and NUSWIDE multi-label
datasets, our ML–TTA framework equipped with BEM exhibits superior perfor-
mance compared to the latest SOTA methods, across various model architec-
tures, prompt initialization, and varying label scenarios. The code is available
at https://github.com/Jinx630/ML-TTA.
1
INTRODUCTION
The advent of vision-language models (VLMs) (Radford et al., 2021; Li et al., 2023; Zeng et al.,
2024; Ya...