Efficient Open-world Test Time Adaptation of Vision Language Models

Authors: Manogna Sreenivas, Soma Biswas
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Test Time Adaptation, Vision Language Models, Robust learning, Domain Adaptation, Open World learning

Abstract:
In dynamic real-world settings, models must adapt to changing data distributions, a challenge known as Test Time Adaptation (TTA). This becomes even more challenging in scenarios where test samples arrive sequentially, and the model must handle open-set conditions by distinguishing between known and unknown classes. Towards this goal, we propose ROSITA, a novel framework for Open set Single Image Test Time Adaptation using Vision-Language Models (VLMs). To enable the separation of known and unknown classes, ROSITA employs a specific contrastive loss, termed ReDUCe loss, which leverages feature banks storing reliable test samples. This approach facilitates efficient adaptation of known class samples to domain shifts while equipping the model to accurately reject unfamiliar samples. Our method sets a new benchmark for this problem, validated through extensive experiments across diverse real-world test environments. Our code is anonymously released at https://github.com/anon-tta/ROSITA.git

Decision: reject
Reviewer scores: 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1730207967401

thread_type:
official_review

<thread_object>
creator: reviewer_fpup

<content>
summary:
This paper proposes ROSITA, a new approach for Open World Single Image Test Time Adaptation that utilizes Vision-Language Models (VLMs). This method enhances the differentiation between known and unknown classes through the use of feature banks and a novel contrastive loss function.

strengths:
**[New setup]** This work claims that it is the first to study open world single image test time adaptation using VLMs.

**[Extensive experiments]** This work has conducted extensive experiments across a diverse array of domain adaptation benchmarks.

weaknesses:
**[Incremental technical contribution]** The main contribution is the reduce loss, a specific contrastive loss, while the class identifier seems a reformulation of previous methods [1, 2] to this task. Given this fact, the technical contribution seems incremental.

[1] Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, 7 (2):179–188, 1936.

[2] Yushu Li, Xun Xu, Yongyi Su, and Kui Jia. On the robustness of open-world test-time training:
Self-training with dynamic prototype expansion. In ICCV, 2023.

**[Unconvincing updating parameter choices]** In section 2.3, the authors test using three different parameter groups for training. However, only these three groups are not comprehensive. Other parameters might be more suitable for updating at test time, e.g., the first layer or the last layer of the encoders. Moreover, additional parameters to keep the intact of the original model are also worth trying, e.g., adapter layers and LoRA.

**[Unclear illustration]** In Figure 1, there might be some questions to be raised: 1) Which dataset is used for experiments? Is it representative enough? If not, better use the results on multiple datasets. 2) Is the learning rate only hyperparameter to tune? What about optimizer type, training epochs and learning rate schedular? 3) Is it possible to update both the prompts and the LN parameters?

**[Missed baseline]** The more recent paper “Diverse Data Augmentation with Diffusions for Effective  Test-time Prompt Tuning, ICCV 2023” is not compared.

**[Experiments]** (i) In Table 1 and 3, the proposed method often outperforms other methods significantly while in Table 2 the performance is close to that of (K+1)PC. Could the authors explain this? (ii) In Table 4, the experimental results with $L_{Re}$ and $L_D$ should be given. (iii) In Table 6, it would be nice to include other methods.

questions:
What’s the main difference between CNN based open world TTA and VLMs based one?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730514018669

thread_type:
official_review

<thread_object>
creator: reviewer_m4nu

<content>
summary:
The article proposes ROSITA, a framework for test-time adaptation of vision language models in settings where, during inference, the model receives samples of both desired/target classes and undesired ones. The latter should be flagged by the model, as in standard open-set scenarios. ROSITA is based on two components: LDA-based separation of desired and undesired classes (following Li et al. (2023)), and a contrastive objective, updating LayerNorm parameters of the visual encoder. Experiments show that the approach outperforms common TTA approaches in this new, less-explored setting.

strengths:
1. Although it is hard to define closed vs. open set in the context of VLMs, the case of test-time adaptation when the test samples comprise both desired and undesired classes is an understudied problem that this paper aims to tackle.

2. The proposed method, ROSITA, is sound, exploiting a simple technique based on LDA to identify desired vs undesired samples (Li et al. (2023)) coupled with a contrastive objective to refine the feature representation. The various design choices are also motivated via ablation studies (e.g., Fig. 1, Tab. 4).

3. The method consistently performs well across various settings and data streams.

weaknesses:
1. As first statement for the focus of the work, lines 64-65 states this work adapts VLMs to work with single-image TTA. This sentence is misleading as multiple works already considered TTA with a single image: for instance, TPT (Shu et al. (2022)) and TDA (Karmanov et al. (2024)) are two examples. This sentence should be revised to avoid potential over claims and better contextualize the work, e.g., by focusing the claim on the differences in the specific setting considered and those of existing works on VLMs.

2. While the definition of the setting as open-world follows previous work (i.e., Li et al. (2023)), it is inaccurate as it does not follow its original meaning. Specifically, open-world recognition implies that classes are added overtime to the model, adding semantic to samples of unknown (or undesired, as in this context) classes [a]. This is not the case in this work, as the set of target classes is static and it is not updated. A more precise definition would be following (Lee et al. (2023)) and use "open-set TTA", as that denotes the presence of a set of target classes and OOD samples to be recognized. It would be helpful to clarify the particular meaning of open world (comparing it with previous use, e.g., [a]) and/or consider adopting the term "open-set TTA" throughout the paper for consistency with existing literature.

2. Currently, the set of baselines do not include any open-set TTA approach, beyond the adaptation of Li et al. (2023). In particular, there exist many potential techniques for detecting unknowns (e.g., MSP [b], max logic [c]) and previous works considered what happens when considering them as alternative OOD detection strategy (e.g., Lee et al. (2023), Tab. 5). As all the experiments and methods are held out with the same, LDA-based selection strategy (Appendix B.4, lines 809) it would be more thorough to include (i) alternatives to LDA's-based OOD detection (ii) simple combination of existing OOD identification algorithms and those for TTA. This would provide a more comprehensive evaluation of ROSITA's performance relative to a broader range of approaches.

4. Technically, the article heavily relies on the TDA scores (proposed already in Li et al. (2023)) to recognize undesired classes. At the same time, there has been already efforts in constructing contrastive-based objectives for TTA [d,e]. Tuning only LayerNorm layers is an already known concept in the literature [f] as well, reducing the impact of Sec. 2.3). In this context, the article merges these two approaches for improved TTA in the "open-world" setting but the technical contribution and the differences with what has been already presented should be clarified and potentially analysed in the manuscript. I would be helpful to include a paragraph to explicitly discuss how the approach differs from and improves upon existing contrastive TTA methods and LayerNorm tuning techniques, potentially highlighting the aspects (and challenges) that are especially relevant for open-set/world TTA. 

6. TDA, one of the closest baseline, is compared with the proposed approach only on Table 3 and the same goes for the other main baseline, i.e., Li et al. (2023). Ii is not clear what is the criterion behind excluding part of the baselines between tables, especially two main ones which could further strengthen the contribution of the manuscript itself. TDA and Li et al. (2023) should be consistently reported across all relevant experiments, and/or it should be explained why these baselines have been excluded from certain comparisons. This would help readers better understand the relative performance of ROSITA across different scenarios.

**Minors**:
- From the abstract (i.e., lines 15-17) and the introduction (lines 73-75, 81-84) it is not clear how ROSITA work and what are its technical contribution. Clarifying them would allow the reader to better follow one of the key messages of the manuscript.

- Lines 203-205, the meaning of "text hypothesis transfer" is not clear.

- Line 481, typo "identifierr" 

**Rerefences**:

[a] Bendale, Abhijit, and Terrance Boult. "Towards open world recognition." In CVPR, 2015.

[b] Hendrycks , Dan and Gimpel, Kevin. "A baseline for detecting misclassified and out-of-distribution examples in neural networks". In ICLR, 2017.

[c] Sungha Choi, et al. "Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes." ECCV, 2022

[d] Chen, Dian, et al. "Contrastive test-time adaptation." In CVPR, 2022.

[e] Yamashita, Kota, and Kazuhiro Hotta. "MixStyle-Based Contrastive Test-Time Adaptation: Pathway to Domain Generalization." In CVPR, 2024.

[f] Zhao, Bingchen, et al. "Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning." In ICLR, 2024.

questions:
1. Is the article the first to perform TTA on single samples and with OOD/undesired classes? If not, clarifying the differences would help to better understand the technical contribution.

2. How does the performance of the model change w.r.t. various design choices? (e.g., different OOD criterion, hyperparameters).

3. Is there a motivation behind the choice of competitors per method? And would it be possible to add other baselines as competitors?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536891155

thread_type:
official_review

<thread_object>
creator: reviewer_pk3p

<content>
summary:
This paper propose a novel method called ROSITA  for Open World Single Image Test Time Adaptation using VLMs.  

ROSITA employs feature banks and an innovative contrastive loss to enhance the distinction between known and unknown classes, allowing efficient adaptation to domain shifts while enabling the model to reject unfamiliar classes.

The extensive experiments demonstrate the effectiveness of the proposed method.

strengths:
1. The paper is well-written and easy to follow, with a clear motivation.
2. The experimental results are impressive, and the ablation study is thorough.
3. The method is novel, interesting, and effective in addressing the challenging problem of Open World Test Time Adaptation.

weaknesses:
1. The title of this paper uses '...... vision-language models' , It seems that author only used CLIP for experiments. It would be better to also study the performance with CLIP-large and other VL backbones.
2. The paper lacks a thorough discussion of the limitations of the current solution and does not provide suggestions for future work.
3. The author doesn't report the error bar.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732082087875

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Authors Response on common concern (1/2)

comment:
**Regarding Incremental Novelty:** 
The driving motivation for this work was to establish a strong benchmark for Open world TTA. Here, we attempt to make a convincing case about why we believe this work is a step forward in advancing research in TTA. This work progressed by us questioning and answering the following sequence of questions, in this order.
 
1. **Why is Open-set/world TTA a relevant problem?** TTA is a very relevant and important problem as it equips the model to adapt to the ever-changing domains in the real world. Most TTA works study the closed-set scenario where the test samples come from a predefined set of classes. Well, real world is seldom closed. On encountering an unfamilar object (open classes), a system equipped to perform closed-set classification would misclassify this unfamiliar object as one of the predefined closed set of classes. A system that identifies a bicycle (say unfamiliar object) as car (in closed-set) is not very smart. A system with the ability to say "I don't know" is better than the one that misclassifies a bicycle as car. Hence, we believe it is important to explore beyond the conventional closed set TTA to bridge the gap between research in TTA and real world TTA. Open-set/world TTA has been recently explored using TTA based on CNNs [1,2].    
2. **Why use VLMs?** Conventional TTA works primarily based on CNNs [7, 10], involve a source model (trained specifically for each dataset), say trained on CIFAR-10 clean data and is test time adapted to CIFAR-10C, its corrupted test set. Contrary to this, VLMs like CLIP has been trained on large scale (image, text) pairs. This enables CLIP to be used in an off-the-shelf manner to classify any dataset given the $C$ classes of interest without any specific retraining. _The text-based classifiers can be obtained for free using the text prompts like "A photo of a {classname}.", enabling effective zero shot $C$-way classification._ This makes VLM a natural candidate for TTA [3,4,5] and hence, we choose to explore its effectiveness for Open-world TTA.
3. **How do we setup the baselines?**
Recent works like TPT [3], PromptAlign [4], TDA [5] show that VLMs can be test-time adapted to improve results beyond zero-shot classification on a single-image basis. These however only address closed-set TTA. To adapt a method from closed-set to open-set, we need an additional module that distinguishes desired class samples from the undesired class samples. For this purpose, we adapt the LDA based OOD detector proposed in [1] for single image open-world TTA here. In our work, _we combine the single image TTA works on VLMs [3,4,5] with LDA based class identifier [1] to establish a strong benchmark for Open world Single Image TTA._
4. **Why use LDA based class identifier?**  This can be done by thresholding scores like MaxLogit, MSP[6]. However, it is non-trivial to choose a fixed threshold, especially in this single image TTA scenario where we do not have access to any/abundant data. In [1], they present a very simple and elegant LDA based OOD detector to estimate the threshold dynamically during TTA, which we adapt for our work. _We emphasize that we do not claim this to be our contribution, and we use it across all the baselines presented._ In ROSITA, we leverage the score statistics which we get for free from the LDA based class identifier to select reliable samples for updating the model using ReDUCe loss.    
5. **What is the best choice of parameters to update the model?** There is a clear gap between CNN [7,10] and VLM [3,4,5] based TTA works with respect to the parameter choice, which we attempt to bridge by presenting this analysis (Section 2.3).  As updating normalization layers has been successfully used in many CNN based TTA works[7], it is only natural to consider adapting normalization layers even in the case of VLMs. We believe Entropy Minimization [7] with _LayerNorm update_ [8] is the most natural and simplest baseline to initiate the study of TTA of VLMs. Surprisingly, none of the current TTA works [3,4,5,9] on VLMs establish this simple yet strong baseline (which is obvious in hindsight). The motivation to present this analysis is to establish this baseline, emphasizing that learnings from TTA works based on CNNs can indeed be translated to VLMs. Yes, we agree this is not novel. Nevertheless, we believe it is an important insight, and hope future VLM based TTA works can benefit from this.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732083591366

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the rebuttal

comment:
Dear reviewers, 

We sincerely thank you all for providing constructive feedback on our work. This has definitely helped strengthen our work from several perspectives.

Here, we summarize the major concerns raised by the reviewers' which we address during this rebuttal:

1. **Experiments with more Vision Language Backbones.**
2. **Justifying the choice of LDA based class identifier.**
3. **Regarding technical contributions and novelty of this work.**
4. **Extensive analysis on parameter choices for model update.**
5. **Additional baseline for TTA using VLMs.**

We are currently revising the manuscript based on the suggestions provided and including additional experimental results. In the meanwhile, **we look forward to actively discuss and address any further concerns through the discussion period.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732558526214

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Manuscript Revision

comment:
**Dear Reviewers,**

Thank you for the insightful feedback, which has helped us significantly to strengthen our work. We have revised the manuscript to incorporate your suggestions and additional experimental analyses done as part of this review process.  

Below, we summarize the major updates done in the manuscript:  
1. **Comprehensive analysis of learnable parameters** (Section 2.3 and Appendix C.7).  
2. Inclusion of **additional baselines**: **DPE** (NeurIPS'24) and **UniEnt** (CVPR'24) (Tables 1, 2, 3, and Appendix B).  
3. **Evaluation of different $C_d$ vs $C_u$ class identifiers** (Appendix C.4).  
4. **Key distinctions between ROSITA and prior works** (lines 391–410).  
5. **Detailed study of the effectiveness of the ReDUCe loss** (lines 109–117).  
6. **Performance evaluation of ROSITA with larger VL backbones** (Appendix C.9).  
7. **Discussion of limitations and scope for future work** (Appendix B.5).  

As the discussion phase is drawing to a close, we look forward to addressing any remaining questions or concerns you might have. Thank you once again for your time and valuable input!
</content>
</thread_object>

</thread>


Metareview: