Title: CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Test-Time Adaptation

Vision-Language Models

Unsupervised Image Classification

Online Adaptation
Abstract: Vision-language models (VLMs) like CLIP exhibit strong zero-shot capabilities but often fail to generalize under distribution shifts. Test-time adaptation (TTA) allows models to update at inference time without labeled data, typically via entropy minimization. However, this objective is fundamentally misaligned with the contrastive image-text training of VLMs, limiting adaptation performance and introducing failure modes such as pseudo-label drift and class collapse. We propose CLIPTTA, a new gradient-based TTA method for vision-language models that leverages a soft contrastive loss aligned with CLIP’s pre-training objective. We provide a theoretical analysis of CLIPTTA’s gradients, showing how its batch-aware design mitigates the risk of collapse. We further extend CLIPTTA to the open-set setting, where both in-distribution (ID) and out-of-distribution (OOD) samples are encountered, using an Outlier Contrastive Exposure (OCE) loss to improve OOD detection. Evaluated on 75 datasets spanning diverse distribution shifts, CLIPTTA consistently outperforms entropy-based objectives and is highly competitive with state-of-the-art TTA methods, outperforming them on a large number of datasets and exhibiting more stable performance across diverse shifts.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles test-time adaptation (TTA) for vision-language models (e.g. CLIP). The authors hypothesize that the classic entropy minimization objective is misaligned with the contrastive training paradigm of CLIP. To address this, they propose a batch-aware contrastive loss for TTA in vision-language settings. Besides, they introduce an outlier contrastive exposure loss to enhance outlier detection in open-set scenarios. Extensive evaluation on 75 datasets verifies robustness and effectiveness of the proposed method for TTA.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The idea for extending classic entropy minimization to better align with CLIP’s contrastive training is well motivated. The proposed batch-aware contrastive loss is conceptually clear and easy to follow.\n- The empirical evaluation is thorough and extensive, spanning a large number of datasets. The improvement shows solid improvements, especially under data corruption, where baseline solutions often fail to adapt.\n\n**Weaknesses**\n1. The performance of the proposed batch-aware contrastive loss seems highly depends on batch size (i.e., the number of samples in a batch). However, the effect of batch size on adaptation performance is not studied and discussed.\n2. While the authors hypothesize that the misalignment between pre-training and adapting objective is crucial for TTA, this claim remains untested. For example, the performance of applying a similar contrastive loss to classic cross-entropy pre-trained models (e.g., ResNet pre-trained on ImageNet) is unclear. Based on the authors’ hypothesis, will classic entropy minimization techniques outperform the proposed contrastive loss for classic models?\n3. Fig. 4b suggests that CLIPTTA avoids class collapse by preserving prediction diversity. However, the regularization term ($\\mathcal{L}_{\\text{reg}}$), which explicitly maximizes entropy in a batch, likely plays a major role in this effect. It is unclear how much of the improvement in preventing class collapse is due to the proposed contrastive loss itself vs the regularization.\n\nQUESTIONS:\n1. How does the number of samples in a batch affect the performance of the proposed contrastive loss for TTA (compare with entropy minimization loss such as TENT)?\n2. Though classic models trained with cross-entropy cannot compute the text-to-image entropy (the second term in eq. 3), the image-to-text entropy (the first term in eq. 3) could still be used. What’s the performance when applying the proposed contrastive loss to such classic models? How does it compare to traditional entropy minimization in this context?\n3. In Fig. 4 (b), futher ablations such as (1) TENT + $\\mathcal{L}\\_{\\text{reg}}$, and (2) CLIPTTA w/o $\\mathcal{L}_{\\text{reg}}$ would be helpful to verify the individual contribution of the contrastive loss in preventing class collapse.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
In this paper, the authors introduce CLIPTTA, a novel gradient-based test-time adaptation (TTA) method for vision-language models that employs a soft contrastive loss aligned with CLIP’s pre-training objective. They provide a theoretical analysis of CLIPTTA’s gradients, demonstrating how its batch-aware design enhances robustness against pseudo-label drift and class collapse. Extensive experiments validate the effectiveness of the proposed approach.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The idea to leverage soft contrastive image-text loss for adapting CLIP is well-motivated.

2. The method is clearly explained and the authors include sufficient analysis to show the benefits of the proposed method.


Weaknesses:

1. Several concepts are introduced without sufficient explanation—for example, gradient dynamics, pseudo-label drift, and natural continuity.

2. It is unclear how the proposed loss function helps minimize the domain gap during test-time adaptation.

3. The proposed method relies on a batch of test samples and cannot be applied to a single test sample.

QUESTIONS:
1. The authors only consider CLIP in this paper, can the proposed method be applied to other similar vision-language models?

2. How does the proposed method compare with RPL [1] and BATCLIP [2], which are both specifically designed for the same task?

3. While the proposed method may help prevent convergence to dominant classes, it remains unclear how it facilitates recovery from poor pseudo-labeling. Further explanation is needed.


[1] If your data distribution shifts, use self-learning

[2] BATCLIP: Bimodal Online Test-Time Adaptation for CLIP

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CLIPTTA, a test-time adaptation method for vision-language models like CLIP. It uses CLIP-style contrastive loss and a soft OOD filtering mechanism based on maximum softmax scores. A learnable threshold and optional memory bank further improve adaptation. Experiments show considerable performance in both closed- and open-set settings.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths:__\n\nThis paper presents a well-structured study on test-time adaptation for CLIP, evaluated across a wide range of datasets. The method demonstrates consistently strong performance, highlighting its robustness and general applicability.\n\n__Weaknesses:__\n\n1. The definition of ID/OOD is somewhat unclear for vision-language models like CLIP, given its undisclosed training data. A sample flagged as OOD by confidence-based filtering may simply reflect underrepresentation, not a true open-set case. A more formal definition would strengthen the framing.\n\n2. The adaptation is performed over 10 steps per sample, which raises questions about test-time efficiency in real-time settings.\n\n3. The method is evaluated solely on CLIP, without examining other vision-language backbones (e.g., SigLIP) or standard CLIP backbones (e.g., ResNet, ViT-L), which limits the generality.\n\n4. The ablation is restricted to typical TTA benchmarks and does not include cross-dataset evaluations commonly used in CLIP adaptation work. Comparing with non-CLIP-specific methods like Tent may also be suboptimal.\n\n5. The paper allows the CLIP logit scale parameter to be updated during test-time adaptation but does not analyse or report its behaviour. Given that CLIP’s pre-trained logit scale is calibrated for contrastive learning, adapting it at test time could introduce instability or overfitting.\n\nQUESTIONS:\nPlease refer to weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents CLIPTTA, a novel test-time adaptation (TTA) framework tailored for vision-language models such as CLIP. CLIPTTA addresses the limitations of entropy-based adaptation objectives by introducing a soft contrastive image-text loss, which better aligns with CLIP’s original contrastive pretraining. This loss helps reduce pseudo-label drift and mitigates class collapse during adaptation. Additionally, the paper proposes an Outlier Contrastive Exposure (OCE) loss, aimed at improving in-distribution (ID) and out-of-distribution (OOD) separation for more robust adaptation under distribution shifts. CLIPTTA is evaluated across an extensive benchmark of 75 datasets and shows consistent improvements over standard entropy-based approaches (e.g., TENT) and is competitive with recent state-of-the-art methods such as CLIPArTT.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1) The introduction of the soft contrastive loss is a principled design choice, closely aligned with CLIP’s pretraining, and serves as a more appropriate alternative to entropy minimization for VLMs.\n\n2) The proposed OCE loss improves robustness by explicitly encouraging separation between ID and OOD samples, which is critical for real-world generalization.\n\n3) The paper is clearly written and well-structured, with strong empirical results across a diverse and challenging set of benchmarks.\n\nWeaknesses:\n\n1) Computational Efficiency and Scalability:\nThe method’s reliance on large batch sizes for effective contrastive loss raises concerns about efficiency and scalability. Although the paper mentions using two NVIDIA V100 GPUs, it does not report runtime, memory usage, or how performance scales with smaller hardware setups. A quantitative analysis of resource requirements would strengthen the work.\n\n2) Unexplored Failure Cases of Contrastive Loss:\nThe paper does not examine scenarios where soft contrastive loss might break down—e.g., when pseudo-labels are highly correlated within a batch, or when the batch contains a high proportion of OOD samples. Such conditions could undermine the contrastive objective, and identifying these limits would be helpful.\n\n3) Limited Analysis Across Dataset Types:\nAlthough evaluated on 75 datasets, the paper offers limited insight into how CLIPTTA performs under different types of distribution shift (e.g., adversarial, synthetic, or natural corruptions). Highlighting specific strengths or weaknesses across shift categories would improve understanding of the method’s robustness.\n\nQUESTIONS:\n1) Could the authors provide a runtime and memory usage comparison with other TTA methods? Given the reliance on large batch sizes for contrastive loss, how does CLIPTTA scale to lower-resource settings?\n\n2) Can the authors provide more detailed per-dataset or per-shift-type analysis? Are there specific types of distribution shifts (e.g., adversarial, synthetic corruptions) where CLIPTTA struggles or particularly excels?\n\n3) In Figure 4, how does CLIPTTA compare to baselines in terms of class collapse and deterioration ratio? Could this be discussed more explicitly?\n\n4) How does the Class-wise Confident Memory (CCM) handle OOD samples during updates? Are there safeguards to prevent incorporating noisy or low-confidence predictions?\n\n5) Does optimizing the OCE loss introduce a trade-off in classification performance on in-distribution samples? If so, how significant is this trade-off?\n\nLIMITATIONS:\nMissing limitations and not properly discussed in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}