Title: Statistics Caching Test-Time Adaptation for Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision-Language Models

CLIP

Test time

Transfer Learning
Abstract: Test-time adaptation (TTA) for Vision-Language Models (VLMs) aims to enhance performance on unseen test data. However, existing methods struggle to achieve robust and continuous knowledge accumulation during test time. To address this, we propose Statistics Caching test-time Adaptation (SCA), a novel cache-based approach. Unlike traditional feature-caching methods prone to forgetting, SCA continuously accumulates task-specific knowledge from all encountered test samples. By formulating the reuse of past features as a least squares problem, SCA avoids storing raw features and instead maintains compact, incrementally updated feature statistics. This design enables efficient online adaptation without the limitations of fixed-size caches, ensuring that the accumulated knowledge grows persistently over time. Furthermore, we introduce adaptive strategies that leverage the VLM's prediction uncertainty to reduce the impact of noisy pseudo-labels and dynamically balance multiple prediction sources, leading to more robust and reliable performance. Extensive experiments demonstrate that SCA achieves compelling performance while maintaining competitive computational efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SCA, a Test-Time Adaptation strategy for VLMs (e.g. CLIP), that uses a cache of statistics. The method deviates from previous cache-based methods in that the features are not directly stored, and pseudolabels are not fully trusted based on a low entropy criterion. The pipeline is fast, as it does not require gradient propagation through the model (as in prompt tuning alternatives), and only fuses the zero-shot logits with the cache logits, using adaptive fusion parameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper proposes an alternative to popular cache based approaches for TTA, that does not require storing features directly. The main advantage is that the criterion does not rely on low entropy, which might be deceiving; incorrect features could be stored accidentally. \n2. The statistics-based cache is easy to compute, and the pseudo-label selection approach is interesting to avoid taking zero-shot predictions directly. \n3. Having adaptive fusion hyperparameters is very useful, as most of previous works finetune their hyperparameters on each dataset.\n4. Results show a competitive performance in several of the widely used benchmarks.\n\nWeaknesses:\n\n1. This work disregards another branch of TTA research that is also based on gradient, but that has proven to be competitive, such as WATT, or even TENT.\n2. Other datasets that are popular in the field of TTA are not considered; corruptions (CIFAR-10/100-C, Imagenet-C), VisDA-C, etc. \n3. The symbol \\theta is used equally for the softmax temperature as for the cumulative probability threshold in Eq. 9.\n\nQUESTIONS:\n1. What is the impact of adapting on a single image and not with a batch? Specially when computing the Gram matrix G, having a few more samples could introduce more information from the batch in a transductive way.\n2. What would the performance be on corrupted datasets (see weaknesses). Some research ignore those datasets that were traditional in the field of TTA, and that significantly decrease the performance of CLIP. \n3. The ablation study on Table 4 is not totally significant, can other datasets be included?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on test-time adaptation for vision-language models. The authors claim that existing methods struggle to achieve robust and continuous knowledge accumulation during test time and propose Statistics Caching test-time Adaptation (SCA), a novel cache-based approach. Specifically, SCA formulates the reuse of past features as a least squares problem, avoiding the storage of raw features and instead maintaining compact representations. In addition, SCA introduces adaptive strategies to reduce the impact of noisy pseudo-labels and dynamically balance multiple prediction sources. Extensive experiments on two benchmarks demonstrate that SCA achieves strong performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1. The paper proposes Statistics Caching test-time Adaptation (SCA), a novel cache-based approach that enables robust and continuous knowledge accumulation during test time. Storing compact information instead of raw features is a promising alternative to enable efficient adaptation.\n\n2. Comprehensive experiments, including comparisons with state-of-the-art methods and ablation studies, are provided.\n\n**Weakness**\n\n1. The motivation requires more detailed explanation. Although the authors provide Figure 2(a) to illustrate that exsiting approach leads to performance degradation, it would be helpful to analyze what types of features are forgotten during adaptation and how knowledge forgetting affects performance. In addition, how do low-entropy and high-entropy samples affect the learning process and contribute to information forgetting? The authors could also provide performance curves of the proposed method as a comparison to better demonstrate its effectiveness.\n\n2. The authors formulate feature storage as a least squares problem. It would be better if they could provide some theoretical justification or insights to explain why this approach yields better compact features for test samples. Also, the authors could compare it with other feature compaction strategies, such as using class prototypes and updating them with certain rules.\n\n3. Prediction fusion is a common strategy in VLM adaptation for improving prediction quality. It is somewhat conflicting to use entropy to weight predictions from different sources, especially since the authors themselves note that entropy is not a reliable measure of prediction accuracy (e.g., lines 40–69). Have the authors tried alternative weighting strategies, and have they analyzed the robustness of the entropy-based weighting in assigning correct importance to different predictions?\n\n4. I noticed that the authors use a prompt ensembling strategy to initialize prompts. However, prior work such as TPT and DiffTPT uses only a single prompt (\"a photo of a\") for initialization. This may introduce unfairness in comparison, as the ensemble strategy can yield a stronger baseline.\n\n5. Some feature visualizations should be provided, as they would help illustrate how stored features contribute to adaptation.\n\nQUESTIONS:\nPlease see the weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates test-time adaptation (TTA) for vision-language models (VLMs) such as CLIP. In this field, two approaches have been proposed: (i) test-time prompt tuning, which optimizes trainable prompt parameters, and (ii) feature caching, which preserves past features and uses them for inference on new test samples. Test-time prompt tuning is computationally expensive because it optimizes prompts using backpropagation with unsupervised objectives such as entropy minimization. On the other hand, feature caching is computationally efficient because it does not perform optimization and instead uses past effective features classified with pseudo labels for inference. However, the paper argues that existing feature caching suffers from \"forgetting\" and \"blocking\" due to misclassified sample features because the cache size is limited. To address this, the paper proposes statistics caching test-time adaptation (SCA) to extract useful knowledge from all past test sample features. SCA formulates the classification problem using past features, such as ridge regression, and sequentially updates the Gram matrix required to obtain the closed-form solution. Additionally, to handle noisy pseudo-labels, it introduces dynamic soft pseudo-label assignment, which uses only the class with the highest prediction probability. SCA constructs the final prediction as the entropy-weighted sum of the zero-shot logits from CLIP and the logits computed from the cache. Experiments demonstrate that SCA achieves superior performance and high computational efficiency compared to recent TTA methods for VLMs, including test-time prompt tuning and feature caching.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n+ **S1.** The proposed method in this paper significantly reduces storage costs compared to existing methods by representing information from all past test samples as a Gram matrix. Leveraging all of the test samples for TTA might be a new perspective.\n+ **S2.** The dynamic soft pseudo-label assignment proposed in this paper has been successful in significantly improving performance.\n### Weaknesses\n- **W1.** The forgetting and blocking issues arising from existing feature caching, which are the primary motivations for this paper, may not be practically significant challenges. First, as shown in Figure 2, these issues have only been reported in a limited number of datasets, indicating limited generalizability. Second, as shown in Table 4, the performance improvements of the proposed method are primarily attributed to dynamic soft label assignment. Without dynamic soft label assignment, the average DPE and SCA performance would be comparable (69.40 in Table 1 vs. 69.32 in Table 4). Since this technique is not directly and necessarily related to feature caching, it can be combined with DPE. Addressing these concerns would further strengthen the clarity of the paper's contributions.\n- **W2.** The negative aspects that may be introduced by the proposed method have not been sufficiently discussed. The proposed method sequentially accumulates knowledge from all test samples into a Gram matrix. As the paper claims, this certainly has positive effects, but it may also have negative effects. For example, in situations where multiple domain shifts coexist, if samples belonging to a specific domain shift become dominant, the Gram matrix will contain statistics biased toward that domain shift. In such a situation, if samples with minor domain shifts appear, can the proposed method robustly handle them? The paper raises a primary research question, “Is this acclaimed knowledge accumulation process truly robust and effective over time?” on L42, but the evidence that the proposed method addresses this question is insufficient.\n- **W3.** The method for determining hyperparameters is unclear. The paper introduces three hyperparameters, $\\gamma, \\beta, \\tau$, into the proposed method and states that these are fixed to specific values in the experiments (L211-212). How were these fixed values discovered? As a result, the sensitivity of $\\beta$ and $\\gamma$ is confirmed in Figure 5, but nothing is mentioned about $\\gamma$. If the tuning was based on the average value of the test set, this would damage the reliability of the paper. In any case, the protocol for determining the hyperparameters should be clearly stated.\n\nQUESTIONS:\nPlease see the weakness section and address the concerns.\n\nLIMITATIONS:\nThe main paper does not include a limitation section. The limitation should be discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Statistics Caching Test-Time Adaptation (SCA), a training-free test-time adaptation method for CLIP that replaces fixed-size caches with running class-wise statistics. Instead of storing features, it updates class means using soft pseudo-labels weighted by confidence. Predictions are made by combining this stats-based classifier with CLIP’s original output. The method avoids cache issues like forgetting or blocking and shows strong accuracy gains across benchmarks with the adaptive instance-level fusion strategy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The motivation is clear, and has a well-structured presentation with Figure 1.\n2. Replaces fixed-size caches with statistical summaries, directly addressing forgetting and blocking issues in cache-based adaptation.\n3. SCA outperforms prior methods like TDA and DPE across multiple benchmarks, showing robustness with the proposed hyperparameter strategy picking.\n4. Computationally efficient: the method avoids optimization and maintains low overhead per sample, with inference time comparable to other lightweight approaches.\n\n**Weaknesses:**\n1. The results on ResNet backbone with OOD benchmark is weaker than previous SOTA DPE.\n2. No ablations on bigger scope dataset like ImageNet, small-scale datasets like DTD is used for ablation studies.\n3. Efficiency comparisons lack clarity, unclear whether data augmentation cost is included.\n\nQUESTIONS:\n1. Lines 176–177 state that allowing |Ω| to vary with τ and the sample’s confidence results in softer pseudo-labels for uncertain samples and sharper ones when confident. Could you provide more evidence for this claim, either empirically or through visualization?\n\n2. The results on ImageNet with the ResNet backbone appear weaker than DPE. Could you clarify why your method underperforms in this setting and whether it's due to architecture sensitivity or some other factor?\n\n3. In your efficiency comparison (Table 3), do the reported numbers include the cost of data augmentation? If not, could you provide a breakdown to help interpret the practical runtime tradeoffs?\n\n4. How did you determine the default values for your hyperparameters? Were they tuned on a subset of data, or selected based on prior work?\n\n5. Why are there no ablation studies on ImageNet? Given its scale and diversity, this would provide a more rigorous validation than DTD, which is relatively small.\n\n6. How would you explain the performance drop between Table 1 and Table 2? Is it due to the number of classes, the nature of the domain shift, or semantic similarity in CLIP’s textual space? Some analysis here would help interpret the results.\n\n7. Lines 242–244 present an important insight. Could you consider highlighting this point earlier or making it more prominent in the Table 3 caption, for instance? It feels like a core comparison factor for those who are not familiar with DPE and might be overlooked.\n\nIf my concerns are addressed, I will raise my score.\n\nLIMITATIONS:\nThe method introduces some storage overhead due to maintaining feature statistics. It partially mitigates error accumulation, but in settings with low domain shift, the performance gains seem to become smaller.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}