Title: Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment
Venue: NeurIPS 2025
Accepted: yes
Keywords: Test-time Adaptation

Vision-Language Models

Distribution Shifts
Abstract: Test-time adaptation (TTA) enhances the zero-shot robustness under distribution shifts by leveraging unlabeled test data during inference. Despite notable advances, several challenges still limit its broader applicability. First, most methods rely on backpropagation or iterative optimization, which limits scalability and hinders real-time deployment. Second, they lack explicit modeling of class-conditional feature distributions. This modeling is crucial for producing reliable decision boundaries and calibrated predictions, but it remains underexplored due to the lack of both source data and supervision at test time. In this paper, we propose ADAPT, an Advanced Distribution-Aware and backPropagation-free Test-time adaptation method. We reframe TTA as a probabilistic inference task by modeling class-conditional likelihoods using gradually updated class means and a shared covariance matrix. This enables closed-form, training-free inference. To correct potential likelihood bias, we introduce lightweight regularization guided by CLIP priors and a historical knowledge bank. ADAPT requires no source data, no gradient updates, and no full access to target data, supporting both online and transductive settings. Extensive experiments across diverse benchmarks demonstrate that our method achieves state-of-the-art performance under a wide range of distribution shifts with superior scalability and robustness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ADAPT, a novel method for test-time adaptation (TTA) that is both backpropagation-free and distribution-aware. The core idea is to reframe TTA as a probabilistic inference problem, where class-conditional feature distributions are modeled as Gaussians. This allows for a closed-form, training-free solution that efficiently adapts vision-language models to new test distributions using dynamically updated \"knowledge banks\" of high-confidence samples.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\nI think the main strength of this work lies in its efficiency and practicality. By eliminating the need for backpropagation and iterative optimization, ADAPT presents a compelling solution for real-time or resource-constrained scenarios, a significant advantage over many existing TTA methods. I also found the framework's flexibility to be a major plus; its ability to operate in both online and transductive settings without requiring source data makes it broadly applicable.\n\n# Weaknesses\n\nDespite the strong results, I have a few concerns:\n\n1. The method's core assumption is that class features follow a Gaussian distribution with a shared covariance matrix (lines 128-129) and a uniform class prior (line 134),  it may be too simplistic for real-world data distributions that could be multi-modal or highly skewed. As noted in prior work [1, 2, 3], the assumption of uniform prior is often violated in the real world, which could degrade the performance of methods requiring such an assumption. An analysis of potential failure cases would strengthen the paper.\n\n2. The literature review contains several critical gaps that weaken the paper's novelty claims. The backpropagation-free TTA approach using pre-trained classifier weights as prototype anchors closely parallels [4]'s vision-only methodology—creating linear layers from CLIP's text embeddings is essentially a direct extension of their approach. Other backpropagation-free TTA methods [6, 7] for vision or vision-language model are missing. Additionally, the \"Knowledge banks\" mechanism resembles those in [3, 5] ([5] is cited but not discussed), and ADAPT's update strategy (e.g., Eq. 9) appears very similar to existing work. The paper requires proper citations and clearer differentiation from these prior works to establish its contributions.\n\n[1] NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation (NeurIPS 2022)\n\n[2] Robust Test-Time Adaptation in Dynamic Scenarios (CVPR 2023)\n\n[3] DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation (ICML 2025)\n\n[4] Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization (NeurIPS 2021)\n\n[5] DynaPrompt: Dynamic Test-Time Prompt Tuning (ICLR 2025)\n\n[6] Test-Time Model Adaptation with Only Forward Passes (ICML 2024)\n\n[7] Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025)\n\nQUESTIONS:\n1. What is Oracle ADAPT and how does it differ from ADAPT in the transductive setting?\n2. Consider adding memory and computation analysis to strengthen ADAPT's contribution. This should favor ADAPT since it avoids backpropagation.\n3. Have you explored class-specific covariance matrices? What's the accuracy vs. computational complexity trade-off?\n4. Why exclude the current test sample from class mean updates in online but include all samples (via soft labels) in transductive settings?\n5. How does the knowledge bank perform on high inter-class similarity datasets? Could low confidence across all classes stall updates and hurt adaptation?\n6. How did you tune the regularization parameter β in Eq. 5, and how sensitive is performance to its value?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Gaussian Discriminant Analysis (GDA)-based test-time adaptation (TTA) method for CLIP. The GDA framework is initialized with textual embeddings of prompted class names. At test time, it models visual features from the image encoder and yields predictions in a backpropagation-free manner. The evaluation includes commonly used benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The work is well-motivated, and the paper is well-written.\n2. The method design appears principled and is supported by theoretical analysis.\n3. The evaluation includes sufficient benchmarks, and the ablation studies are well-structured.\n\n**Weaknesses**\n\nThe major concern lies in the exact performance advantage of the proposed method compared to existing backpropagation-free TTA baselines, especially memory-based TTA methods like TDA [18]. My understanding is that the main difference between ADAPT and memory-based methods is that memory-based approaches use a KNN-like strategy to model test-time visual features and yield predictions, while ADAPT uses a GDA-based approach to do the same. Since the text embeddings are fixed, both can incorporate prior knowledge from textual prompts. Although Tables 2 and 4 show performance gains of ADAPT over TDA, Table 7 indicates that ADAPT benefits significantly from GPT-generated prompts. To my knowledge, TDA employs hand-crafted prompts similar to the “CLIP Template” in Table 7. If we compare ADAPT and TDA using similar prompts, their performances seem comparable. This suggests that the observed performance gains in Tables 2 and 4 may largely result from the use of stronger GPT-generated prompts rather than from inherent advantages of the GDA-based method itself. Therefore, the advantage of the proposed GDA-based approach over memory-based methods remains uncertain based on the current manuscript.\n\nQUESTIONS:\n1. The authors may need to compare ADAPT with previous memory-based methods under comparable prior knowledge from textual prompts to more convincingly demonstrate the superiority of the proposed algorithm.\n2. Given that ADAPT ultimately adopts prompts from AWT [57], AWT may need also be compared and discussed.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ADAPT, a novel backpropagation-free and distribution-aware test-time adaptation method designed for VLMs. By modeling class-conditional feature distributions via gaussian discriminant analysis and updating them with high-confidence test samples in a constructed knowledge bank, the method provides closed-form, training-free and efficient adaptation. ADAPT operates under both online and transductive settings and outperforms both backpropagation-based and training-free baselines across natural shifts, corruption robustness and fine-grained categorization tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. ADAPT avoids iterative updates or backpropagation at test time by formulating TTA as a probabilistic inference task with gaussian assumptions, enabling low-latency adaptation.\n2. ADAPT explicitly estimates class-conditional feature distributions with a shared covariance matrix and uses knowledge banks for regularization.\n3. ADAPT achieves SOTA performance on ImageNet variants and 10 fine-grained datasets under both online and transductive settings, while also being more efficient in time and memory usage.\n\nWeaknesses:\n\n1. While the Gaussian Discriminant Analysis formulation enables efficient closed-form inference, the assumption of unimodal Gaussian distributions with a shared covariance across classes may not capture the complexity of real-world data, especially in multimodal or highly structured feature spaces. This simplification can result in biased or suboptimal predictions in challenging domains (e.g., fine-grained classification or long-tail distributions), potentially limiting ADAPT's effectiveness in practical deployment scenarios.\n2. The method depends on confidence thresholds, bank sizes and prior means from CLIP templates, which may require careful tuning.\n\nQUESTIONS:\n1. What is the effect of significant noise in the early high-confidence predictions on long-tail adaptation?\n2. What kind of historical samples or priors does ADAPT need?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a gradient free method for adapting vision–language models at test time. The prediction is treated as Gaussian mixture class assignment, where the parameters of each class are updated based on a set of highly confident representatives of that class. \nThe paper treats two settings: (i) online: as each test sample arrives, its embedding is scored, and if confident, added to the per-class bank. Then, the class Gaussian parameters are recomputed in closed form before making that sample’s prediction. (ii) once the entire unlabeled test set is available, the method builds the banks from the most confident samples.\nThis allows the predictions, modeled as GMM assignment, to adapt to test time distribution changes.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\nThe paper is well written and it proposes a simple and clean solution to test time adaptations. The empirical evaluation is extensive and convincing.\n\n**Weaknesses:**\n- The memory banks, build from high confidence samples, are central to the method’s efficiency, but they also introduce a potential failure, especially if significant shift occurs. If the initial CLIP zero-shot predictions are substantially misaligned with the new domain, their confidence scores will become unreliable, causing the memory banks to be built from misclassified examples. This, in turn, will corrupt the estimated class means and shared covariance.\n- Similarly, the question of how having classes with very few high-confidence samples affects the estimation of the parameters, and therefore the results.\n-  The paper offers no formal guarantees on how quickly the estimated parameters converge, or bound on the divergence from the true distribution. \n- Evaluation relies exclusively on CLIP’s embedding space.  It’s unclear how well its assumptions and mechanisms transfer to other models.\n- The method may be  highly dependent on the randomized order of samples in the online case. However, no standard error or other form of confidence is reported for repetitions of each experiment. \n\nDespite these limitations, the paper offers a simple, elegant solution to a significant problem. I would be happy to revisit my evaluation—and potentially raise my score, if the authors address the identified weaknesses, particularly by reporting standard deviations or confidence intervals for their results.\n\nQUESTIONS:\n- Have the authors considered a bank-pruning step (periodically removing samples whose likelihood falls below a threshold)? This can help addressing some of the weaknesses.\n- The shared covariance assumption seems very restrictive. Have the authors considered experimenting with separate covariance matrices, even on a small scale example?\n\nLIMITATIONS:\nSee weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}