DOTA: Distributional Test-Time Adaptation of Vision-Language Models

## Abstract

Abstract Vision-language foundation models (e.g., CLIP) have shown remarkable performance across a wide range of tasks. However, deploying these models may be challenging and unreliable when significant distribution gaps exist between the training and test data. The training-free test-time dynamic adapter (TDA) is a promising approach to address this issue by storing representative test samples to guide the classification of subsequent ones. However, TDA only red naively maintains a limited number of reference samples in the cache, leading to severe test-time catastrophic forgetting when the cache is updated by dropping samples. In this paper, we propose a simple yet effective method for Test-time Continual Adaptation ( TCA ). Instead of naively memorizing representative test samples, TCA continually estimates the distributions of test samples, allowing the model to adapt to the deployment environment. The test-time posterior probabilities are then computed using the estimated distributions based on Bayes’ theorem for adaptation purposes. To further enhance the adaptability on the uncertain samples, we introduce a new human-machine collaboration paradigm which identifies uncertain samples, collects human-feedback, and incorporates it into the TCA framework. Extensive experiments validate that TCA enables CLIP to continually learn, resulting in a significant improvement compared to current state-of-the-art methods.

## 1 Introduction

Recent advances in vision-language foundation models have shown remarkable vision understanding capabilities across a broad range of tasks by training on web-scale image-text pairs (radford2021learning; DBLP:conf/icml/LavoieKIAWCB24; zhai2023sigmoid). Taking CLIP as an example, it can conduct zero-shot classification without the need for additional training data using predefined prompts (radford2021learning). However, CLIP may still face challenges when handling various specific applications during test time, especially when there is a significant distribution gap between the training and test data (shu2022test; karmanov2024efficient; feng2023diverse).

Test-time adaptation methods are typically employed to address the distribution gap between the training and test datasets by fine-tuning the original model during test time (boudiaf2022parameter; chen2022contrastive; DBLP:conf/iclr/WangSLOD21). Test-time adaptation aligns well with real-world applications where models need to adapt to new environments quickly. There are two primary lines to achieve test-time adaptation on the vision-language foundation models. Early works advocate learning prompts during test time with the test data (shu2022test; feng2023diverse). However, these methods require significant computational resources to optimize the learnable prompts via backpropagation. This significant resource overhead makes them unsuitable in applications when fast inference speed is widely required. Therefore, a more efficient method, Training-Free Dynamic Adapter (TDA), has been proposed (karmanov2024efficient) recently. To avoid the training process with backpropagation, TDA maintains a lightweight cache during testing to store representative test samples and guide the classification of subsequent test samples.

Although TDA has achieved significant efficiency compared to previous prompt tuning methods, it still faces challenges due to the limited cache capacity. Specifically, TDA naively preserves a limited number of typical samples in the cache during test time and dynamically updates the cache with higher classification-confidence samples. This strategy leads to test-time forgetting, because when new confident samples are added, the previous cached samples must be discarded. As a result, relying solely on a few high-confidence samples stored in the cache leads to a biased classifier.

To address the above issue, we introduce a novel method called Distributional Test-Time Adaptation (TCA). TCA continually estimates the distribution of test samples to adapt the test environment. Specifically, under the mild assumption that the embedding distribution of each class follows a Gaussian distribution (hastie1996discriminant), we propose an efficient method to continually estimate the distribution of different classes. Once the distributions of different classes are estimated, we can easily calculate the posterior probabilities of subsequent test samples based on Bayes’ theorem and obtain a test-time classifier for test-time adaptation. Similar to TDA, this process does not require gradient backpropagation, avoiding the complex computational overhead during testing, leading to approximately 20 times faster inference speed. Moreover, unlike TDA memorizing representative test samples, TCA can continually adapt to the test environment. Last but not least, to further improve the performance of the model in dealing with uncertain or risky samples during test-time adaptation, we introduce a new human-in-the-loop paradigm. This approach enables the model to detect uncertain samples and then adapt during test time with the aid of human-feedback. In summary, the contributions of this paper are:

- •
We propose a novel distributional test-time continual learning framework which promotes the performance of existing visual-language foundation models in downstream tasks.
- •
Within this framework, we propose a simple yet effective method to enhance the foundation model by efficiently estimating the distribution of different categories during test time.
- •
We first define the test-time adaptation problem with human-feedback, which allows the model to detect high-uncertainty samples and perform test-time adaptation under human-feedback.
- •
Extensive experiments on diverse datasets validate the effectiveness of the proposed method, demonstrating a significant improvement.

## 2 Related work

Test-time adaptation (TTA) focuses on addressing the distribution shift between training and test data by learning from the test data. Early efforts to improve TTA performance primarily involve adjusting batch normalization layers and designing unsupervised objective functions (nado2020evaluating; wang2020tent; khurana2021sita; lim2023ttn). For example, TENT (wang2020tent) optimizes the affine parameters in batch normalization layers by minimizing the entropy of the prediction probability.
MEMO (zhang2022memo) applies variant augmentation methods to a single test sample and optimizes model parameters by minimizing the entropy of the prediction probability. To enhance the performance of vision-language models during testing, TPT (shu2022test) introduces adaptive text prompts and optimizes the prompts through entropy minimization. Building on this, DiffTPT (feng2023diverse) leverages pre-trained stable diffusion models to generate diverse augmented data for use in test-time prompt tuning. However, TPT and DiffTPT rely heavily on gradient backpropagation to optimize the prompts, making them computationally expensive and resource-intensive during testing. TDA (karmanov2024efficient) proposes a lightweight test-time adaption method by storing representative test samples. Compared to TDA, which naively stores typical test samples, we achieve continuous adaptation by estimating the distribution of test samples, leading to a more efficient and adaptive solution.

Uncertainty estimation aims to estimate the reliability of decision. Traditional methods for uncertainty estimation often require additional training processes. For example, ensemble learning (lakshminarayanan2017simple; liu2019accurate) and Bayesian neural networks (mackay1992practical; gal2016dropout) estimate uncertainty by obtaining the distribution of prediction. However, these methods typically introduce additional computational costs during inference. To address this, regularization-based methods have been proposed to constrain the confidence of the model during training, preventing overfitting and thereby improving uncertainty estimation (malinin2018predictive; sensoy2018evidential; han2022trusted; han2024selective). However, these methods focus on modifying the training process, such as altering the model architecture or loss function, to estimate uncertainty. They are not applicable to foundation models that have already been fully trained. Therefore, in this paper, we focus on estimating uncertainty during the inference stage using test samples.

Vision-language models have demonstrated strong vision understanding capabilities benefiting from training on large-scale datasets (radford2021learning; zhai2023sigmoid; DBLP:conf/icml/LavoieKIAWCB24). Among them, CLIP (radford2021learning) is the most representative method by maximizing the similarity between image and their corresponding text embeddings. To further enhance performance of CLIP on downstream tasks, prompt learning-based methods have been proposed by optimizing the prompts of the text encoder (zhou2022cocoop; zhou2022coop; bai2024id; khattak2023maple). Moreover, to reduce the computational cost associated with gradient calculations in prompt learning, efficient CLIP adaptation methods have been introduced (gao2024clip; zhang2022tip; DBLP:conf/iclr/WangLS0WT24; li2024graphadapter; yu2023task). These methods enable downstream task adaptation using only a small number of training samples in the embedding space. Orthogonal to above methods, this paper focuses on continuously adapting to environments during testing by leveraging test samples.

## 3 Method

### 3.1 Zero-Shot Classification with prompt

#### Zero-shot classification.

During the pre-training stage, CLIP(^1^11While this paper primarily focuses on CLIP, our approach is also applicable to other similar models.) trains its image and text encoders using large-scale image-text pairs. This is achieved by maximizing the cosine similarity between the image and text embeddings through contrastive loss. Unlike traditional classifiers trained on closed-set labels, CLIP leverages open-set semantic information in the image-text pairs to learn a broader range of visual concepts. Consequently, during the test stage, CLIP can perform zero-shot classification without additional training. Specifically, given a test sample $\bm{x}$ for $K$-class classification, where $\bm{x}$ represents the image embedding obtained from the image encoder, the corresponding zero-shot prediction probability $P^{\texttt{zs}}_{k}$ for class $k$ is calculated as:

$$ $P^{\texttt{zs}}_{k}(y=k|\bm{x})=\frac{\exp(\cos(\bm{x},\bm{w}_{k})/\tau)}{\sum _{k=1}^{K}\exp(\cos(\bm{x},\bm{w}_{k})/\tau)},$ (1) $$

where zs refers to zero-shot. $\bm{w}_{k}$ is the classification weight for class $k$, obtained by encoding the corresponding prompt, e.g., “a photo of {class}”, with the class token replaced by the specific category name. $\tau$ is the learned temperature parameter in CLIP, and $\cos(\cdot,\cdot)$ denotes the cosine similarity. The above classification process can be understood as comparing the obtained image embedding with the text prompt and selecting the most similar category as the final decision.

### 3.2 Distributional Test-time Adaptation

#### Motivation.

When CLIP is deployed in various environments, the performance tends to degrade due to the changes of data distribution, especially when the test data has a significant distribution gap from the CLIP training data. Test-time adaptation can effectively adapt the foundational model to new environments quickly during the test stage. Current state-of-the-art method TDA maintains a cache during test-time to preserve representative samples of different classes, which then guide the classification of the following test samples. However, TDA may lead to a severe test-time forgetting problem when the cache is updated due to only maintaining the embeddings of very limited test samples without learning the underlying relationships between the sample and label. To this end, we propose distributional test-time adaptation (TCA), which aims to continuously learn from test-time data by estimating the test sample distribution. Specifically, as shown in Fig. 1, we propose to online estimate the data distribution of samples in the current test environment during testing. Once obtaining the distribution, we can leverage Bayes’ theorem to naturally infer the test-time posterior distribution of different classes for new test samples to adapt the test-time environment.

Figure: Figure 1: Framework of the proposed method. During test time, a stream of test samples is evaluated with original zero-shot classifier, and we estimate the distributions for the test samples during testing, enabling the model to continually learn from the test samples and the zero-shot classification probabilities. As the number of test samples increases, the estimated test sample data distribution will become more accurate. Finally the test-time classifier can then be obtained using the estimated distributions according to Bayes’ theorem for test-time adaptation.
Refer to caption: extracted/5886588/fig/framework2.png

#### Classification with Gaussian discriminant analysis.

Formally, inspired by classical Gaussian discriminant analysis (hastie1996discriminant), we assume that the embedding distribution of each class $k$ follows a Gaussian distribution, i.e., $P(\bm{x}\!\!\mid\!\!y\!\!=\!\!k)=\mathcal{N}(\bm{\mu}_{k},\bm{\Sigma}_{k})$, where $\bm{\mu}_{k}$ and $\bm{\Sigma}_{k}$ are the mean vector and covariance matrix of class $k$, respectively.
Using Bayes’ theorem, the posterior probability $P(y\!=\!k|\bm{x})$ of class $k$ can be given by

$$ $P(y\!=\!k|\bm{x})=\frac{P(\bm{x}|y\!=\!k)P(y\!=\!k)}{P(\bm{x})},$ (2) $$

where $P(\bm{x})=\sum_{k=1}^{K}P(\bm{x}|y\!=\!k)P(y\!=\!k)$ and $P(y\!=\!k)$ is the prior probability.
In practice, we set the prior probability to $1/K$ for simplicity.
Then $P(y\!=\!k|\bm{x})$ can be obtained with

$$ $P(y\!=\!k\!\mid\!\bm{x})=\frac{\exp(f_{k}(\bm{x}))}{\sum_{k=1}^{K}\exp(f_{k}( \bm{x}))},$ (3) $$

where $f_{k}(\bm{x})=-\frac{1}{2}(\bm{x}-\bm{\mu}_{k})^{T}\bm{\Sigma}_{k}^{-1}(\bm{x}
-\bm{\mu}_{k})-\frac{1}{2}\log|\bm{\Sigma}_{k}|$.

#### Parameter estimation with zero-shot predictive probability.

We can conduct classifier updating with the Gaussian discriminant analysis. Unfortunately, during testing, we cannot access to the ground-truth labels for the $N$ test samples, whose input embeddings are denoted as ${\{\bm{x}_{n}\}}_{n=1}^{N}$. Therefore, we try to use the zero-shot predictive probability to estimate the distribution of test samples. Specifically, we first estimate the zero-shot posterior probability $\{P^{\texttt{zs}}_{k}\}_{k=1}^{K}$. Then, we maximize the test-time posterior probability by estimating the means $\{\hat{\bm{\mu}}_{k}\}_{k=1}^{K}$ and covariances $\{\hat{\bm{\Sigma}}_{k}\}_{k=1}^{K}$. This process can be viewed as a single iteration of the EM algorithm (moon1996expectation), where obtaining the zero-shot classification probability corresponds to the expectation step, and estimating $\{\hat{\bm{\mu}}_{k},\hat{\bm{\Sigma}}_{k}\}_{k=1}^{K}$ based on the zero-shot predicted probability corresponds to the maximization step. Formally, $\{\hat{\bm{\mu}}_{k},\hat{\bm{\Sigma}}_{k}\}_{k=1}^{K}$ can be estimated with the following equations (hastie1996discriminant):

$$ $\hat{\bm{\mu}}_{k}=\frac{\sum_{n=1}^{N}P^{\texttt{zs}}_{k}(y\!=\!k\!\mid\!\bm{ x}_{n})\bm{x}_{n}}{\sum_{n=1}^{N}P^{\texttt{zs}}_{k}(y\!=\!k\!\mid\!\bm{x}_{n} )},\quad\hat{\bm{\Sigma}}_{k}\!=\!\frac{\sum_{n=1}^{N}P^{\texttt{zs}}_{k}(y\!= \!k\!\mid\!\bm{x}_{n})(\bm{x}_{n}\!-\!\hat{\bm{\mu}}_{k})(\bm{x}_{n}\!-\!\hat{ \bm{\mu}}_{k})^{T}}{\sum_{n=1}^{N}P^{\texttt{zs}}_{k}(y\!=\!k\!\mid\!\bm{x}_{n })}.$ (4) $$

#### Test-time distribution parameter estimation.

When estimating data distribution at test time, one another challenge is that we evaluate the test samples sequentially in a streaming manner instead of accessing all samples simultaneously. This necessitates a strategy to appropriately adjust the estimation method in Eq. 4 through effective initialization, and then allowing the parameters to be updated quickly as new test samples arrive. To achieve this goal, we propose a simple method which initializes the distribution parameters and then updates them in test time. Initialization of $\{\hat{\bm{\mu}}_{k},\hat{\bm{\Sigma}}_{k}\}_{k=1}^{K}$. We can initialize the estimated mean of different classes in a way that aligns it with the original zero-shot classifier $\{\bm{w}_{k}\}_{k=1}^{K}$:

$$ $\vspace{-0.1em}\hat{\bm{\mu}}_{k}^{0}=\bm{w}_{k}\quad\text{and}\quad\hat{\bm{ \Sigma}}_{k}^{0}=\sigma^{2}\bm{I},$ (5) $$

where $\sigma^{2}$ is a hyperparameter that determines the initial variance and $\bm{I}$ is the identity matrix. Update of $\{\hat{\bm{\mu}}_{k},\hat{\bm{\Sigma}}_{k}\}_{k=1}^{K}$. We employ the update form described in (dasgupta2007line), which is capable of estimating Gaussian distribution parameters in an online setting. Specifically, given a batch of test samples at step $t$, the updated $\hat{\bm{\mu}}_{k}^{t},\hat{\bm{\Sigma}}_{k}^{t}$ can be computed based on the $\hat{\bm{\mu}}_{k}^{t-1},\hat{\bm{\Sigma}}_{k}^{t-1}$ as follows:

$$ $\hat{\bm{\mu}}_{k}^{t}=\tfrac{c^{t-1}_{k}\hat{\bm{\mu}}_{k}^{t-1}+\sum_{n}P^{ \texttt{zs}}_{k}(y=k\mid\bm{x}_{n})\bm{x}_{n}}{c^{t-1}_{k}+\sum_{n}P^{\texttt{ zs}}_{k}(y=k\mid\bm{x}_{n})}\text{ and }\hat{\bm{\Sigma}}_{k}^{t}=\tfrac{c^{t- 1}_{k}\hat{\bm{\Sigma}}_{k}^{t-1}+\sum_{n}P^{\texttt{zs}}_{k}(y=k\mid\bm{x}_{n })(\bm{x}_{n}-\hat{\bm{\mu}}_{k}^{t-1})(\bm{x}_{n}-\hat{\bm{\mu}}_{k}^{t-1})^{ T}}{c^{t-1}_{k}+\sum_{n}P^{\texttt{zs}}_{k}(y=k\mid\bm{x}_{n})},$ (6) $$

where $c_{k}^{t-1}$ is the sum of the confidences of the cumulative number of observed samples of class $k$ at step $t-1$, and $c_{k}^{0}=1$, with $c_{k}^{t}$ updated as $c_{k}^{t}=c_{k}^{t-1}+\sum_{n}P^{\texttt{zs}}_{k}(y\!\!=\!\!k|\bm{x}_{n})$. Then, we can use Eq. 3 to calculate the test-time adapted posterior probability.
In practice, to reduce computational complexity when inverting the covariance matrix $\hat{\bm{\Sigma}}_{k}$, similar to the approach in (anderson1958introduction; friedman1989regularized), we approximate the covariance by averaging across all classes, reducing the number of matrix inversions from $K$ to $1$, thereby improving efficiency. Additionally, we apply shrinkage regularization to the precision matrix to enhance the stability of the inversion process as follows: $\hat{\bm{\Lambda}}=[(1-\epsilon)\hat{\bm{\Sigma}}+\epsilon\bm{I}]^{-1}$,
where $\epsilon=10^{-4}$ is the shrinkage parameter. The term $\epsilon\bm{I}$ ensures that the eigenvalues of the covariance matrix are well-conditioned, maintaining the desired properties such as positive definiteness and rank stability.

### 3.3 Test-time adaption with human-feedback

Figure: Figure 2: Pipeline of test-time adaptation with human-feedback. Test-time uncertainty estimation is employed to identify unconfident samples, prompting the input of human-feedback. The feedback, combined with the prediction of model, is then utilized for test-time adaptation.
Refer to caption: x1.png

#### Test-time adaption with human-feedback.

The continuous test-time adaptation method enhances model performance by estimating the data distribution of incoming test samples. However, relying solely on zero-shot predicted probability distributions for this estimation may lead to inaccuracies, particularly for originally uncertain samples. The predicted probabilities of these uncertain samples often fail to provide reliable information for accurate distribution estimation. To address this, we propose a new task that incorporates human-feedback during test-time adaptation, establishing a simple yet effective human-in-the-loop paradigm. Specifically, after the model is deployed, we aim to obtain label information on uncertain samples with human in real-time and use it for test-time adaptation. This approach enables quick and effective performance improvements on uncertain samples during testing.

Figure: Algorithm 1 The distributional test-time adaptation pseudocode of TCA.

#### Test-time uncertainty estimation.

To achieve the test-time adaption with human-feedback, we first define the test-time uncertainty estimation task, which aims to determine whether the current test sample is uncertain based on the information from the previous test samples stream. Formally, given a test sample $\bm{x}_{i}$ and the previously tested samples $\{\bm{x}_{n}\}_{n=1}^{i-1}$, our objective is to evaluate whether the current sample $\bm{x}_{i}$ is uncertain, leveraging information from both the previous inference samples $\{\bm{x}_{n}\}_{n=1}^{i-1}$ and $\bm{x}_{i}$ itself. To achieve this goal, we propose a simple yet effective method based on the confidence scores of past samples(^2^22Here we only propose a simple yet effective solution, and leave the task of improving the performance of test-time uncertainty estimation to future work.). Specifically, we store the confidence scores of all past test samples and use this information to determine whether the current test sample falls within the lowest percentile of confidence scores. Formally, given the confidence score $s_{i}$ of the current sample $\bm{x}_{i}$, where $s_{i}=\max(\{P^{\texttt{zs}}_{k}(y\!\!=\!\!k|\bm{x}_{i})\}_{k=1}^{K})$, we classify $\bm{x}_{i}$ as uncertain if:

$$ $s_{i}\leq s_{\gamma}$ (7) $$

where $s_{\gamma}=\text{percentile}(\{s_{n}\}_{n=1}^{i},\gamma)$ represents the value at the $\gamma$-th percentile of the confidence scores $\{s_{n}\}_{n=1}^{i}$, with $\gamma$ indicating the proportion of scores when sorted in ascending order. In other words, $s_{\gamma}$ corresponds to the score below which $\gamma$ proportion of the sorted confidence scores fall.
Moreover, $\gamma$ can be viewed as a hyperparameter that controls the proportion of samples classified as uncertain, which can be used to control the degree of human involvement during the testing process. Compared with the traditional method of judging whether the decision is uncertain only based on the current sample, we can obtain relative uncertainty estimation to improve the adaptability of model to the test data distribution and more robust threshold setting. Then as shown in Fig. 2 when sample is uncertain, we can collect human-feedback, manually determine its true label, and use the method in Sec. 3.2 to continuously update the model.

### 3.4 Adaptive fusion of zero-shot and test-time classifier

As the number of test samples increases, the reliability of the estimated test sample distribution improves (dasgupta2007line). However, when the number of test samples is insufficient, the estimated distribution may be unreliable. To address this, we introduce a dynamic zero-shot classification and test-time result fusion approach, allowing the model to rely more on zero-shot classification when the test-time distribution estimation is insufficient. Formally, the final fusion probability is defined as follows:

$$ $P_{k}(y=k|x)=\tfrac{\exp(\cos(\bm{x},\bm{w}_{k})/\tau+\lambda f_{k}(\bm{x}))}{ \sum_{k=1}^{K}[\exp(\cos(\bm{x},\bm{w}_{k})/\tau+\lambda f_{k}(\bm{x}))]},$ (8) $$

where $\lambda=\min(\rho c,\eta)$. Here, $c$ represents the number of test samples, and $\rho$ and $\eta$ are hyperparameters that control the weight of the test-time classifier logits. The value of $\lambda$ increases with the number of test samples when this number is insufficient, gradually approaching the maximum value $\eta$. This approach encourages the model to rely on the zero-shot classifier results when the test samples are insufficient to estimate the distribution, mitigating the potential negative impact of the test-time classifier. The whole pseudo code is shown in Alg. 1.

## 4 Experiments

We conduct extensive experiments to validate the performance of TCA. Specifically, we first compare TCA with current state-of-the-art methods and then conduct ablation studies.

#### Benchmarks.

Consistent with prior works (shu2022test; feng2023diverse; karmanov2024efficient), we conduct our main experiments on natural distribution shifts and cross-domain generalization scenarios. For the natural distribution shifts scenario, we utilize multiple datasets including ImageNet (deng2009imagenet), ImageNet-A (hendrycks2021natural), ImageNet-R (hendrycks2021many), and ImageNet-S (wang2019learning), which serve as measures of the robustness of our approach. In the cross-domain generalization scenario, we evaluate the performance of the model across 10 diverse image classification datasets, each representing a distinct domain with different classes: Aircraft (maji2013fine), Caltech101 (fei2004learning), Cars (krause20133d), DTD (cimpoi2014describing), EuroSAT (helber2019eurosat), Flower102 (nilsback2008automated), Food101 (bossard2014food), Pets (parkhi2012cats), SUN397 (xiao2010sun), and UCF101 (soomro2012dataset). This benchmark provides a comprehensive evaluation of the adaptability of the model during test time across various class spaces.

#### Comparison method.

We compare the proposed method with the following method: (1) TPT (shu2022test) is a test time prompt tuning method. (2) DiffTPT (feng2023diverse) introduces more diverse test sample augmentation with diffusion model. TPT and DiffTPT require gradient backpropagation to update prompt, so they require greater computational cost. (3) TDA (karmanov2024efficient) introduce an efficient test-time adaption method do not need backpropagation, which works with a cache containing representative samples to conduct test time adaption with these samples. To be consistent with the previous works (shu2022test; karmanov2024efficient), we also include the baseline zero-shot performance of CLIP, using the ensemble of 80 hand-crafted prompts (radford2021learning). The results of the above methods are both obtained from the original paper.

**Table 1: Top-1 accuracy (%) under the natural distribution shifts scenario. For clarity, the best and second-best results that do not require human-feedback are shown in bold and underlined, respectively. TCA $5\%$ and $15\%$ feedback indicate test-time adaptation with human-feedback on uncertain samples, with approximately $5\%$ and $15\%$ of the samples being uncertain $(\gamma=0.05\text{ or }0.15)$. BP-free and continual adaption indicate whether the method does not require gradient backpropagation and has the ability of continuous adaptation.**
| Method | BP-free | Continual adaption | ImageNet | ImageNet-A | ImageNet-R | ImageNet-S | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP-ViT-B/16 | ✓ | ✗ | 68.34 | 49.89 | 77.65 | 48.24 | 61.03 |
| TPT | ✗ | ✓ | 68.98 | 54.77 | 77.06 | 47.94 | 62.19 |
| DiffTPT | ✗ | ✓ | 70.30 | 55.68 | 75.00 | 46.80 | 61.95 |
| TDA | ✓ | ✗ | 69.51 | 60.11 | 80.24 | 50.54 | 65.10 |
| TCA | ✓ | ✓ | 70.68 | 61.19 | 81.17 | 51.33 | 66.09 |
| TCA $5\%$ feedback | ✓ | ✓ | 71.01 | 61.44 | 81.41 | 52.13 | 66.50 |
| TCA $15\%$ feedback | ✓ | ✓ | 71.83 | 61.83 | 81.78 | 53.34 | 67.20 |
| CLIP-ResNet-50 | ✓ | ✗ | 59.81 | 23.24 | 60.72 | 35.48 | 44.81 |
| TPT | ✓ | ✓ | 60.74 | 26.67 | 59.11 | 35.09 | 45.40 |
| DiffTPT | ✗ | ✓ | 60.80 | 31.06 | 58.80 | 37.10 | 46.94 |
| TDA | ✓ | ✗ | 61.35 | 30.29 | 62.58 | 38.12 | 48.09 |
| TCA | ✓ | ✓ | 61.82 | 30.81 | 62.81 | 37.52 | 48.24 |
| TCA $5\%$ feedback | ✓ | ✓ | 62.12 | 31.01 | 63.04 | 37.86 | 48.51 |
| TCA $15\%$ feedback | ✓ | ✓ | 62.77 | 31.13 | 63.34 | 38.48 | 48.93 |

**Table 2: Top-1 accuracy (%) under the cross-domain generalization scenario. For clarity, the best and second-best results that do not require human feedback are shown in bold and underlined, respectively. TCA $5\%$ and $15\%$ feedback indicate test-time adaptation with human feedback on uncertain samples, with approximately $5\%$ and $15\%$ of the samples being uncertain.**
| Method | Aircraft | Caltech101 | Cars | DTD | EuroSAT | Flower102 | Food101 | Pets | SUN397 | UCF101 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP-ViT-B/16 | 23.22 | 93.55 | 66.11 | 45.04 | 50.42 | 66.99 | 82.86 | 86.92 | 65.63 | 65.16 | 64.59 |
| TPT | 24.78 | 94.16 | 66.87 | 47.75 | 42.44 | 68.98 | 84.67 | 87.79 | 65.50 | 68.04 | 65.10 |
| DiffTPT | 25.60 | 92.49 | 67.01 | 47.00 | 43.13 | 70.10 | 87.23 | 88.22 | 65.74 | 62.67 | 65.47 |
| TDA | 23.91 | 94.24 | 67.28 | 47.40 | 58.00 | 71.42 | 86.14 | 88.63 | 67.62 | 70.66 | 67.53 |
| TCA | 25.59 | 94.32 | 69.48 | 47.87 | 57.65 | 74.67 | 87.02 | 91.69 | 69.70 | 72.06 | 69.01 |
| TCA $5\%$ feedback | 26.73 | 94.56 | 70.95 | 49.82 | 65.00 | 76.86 | 87.17 | 92.78 | 70.49 | 75.26 | 70.96 |
| TCA $15\%$ feedback | 28.65 | 95.01 | 73.01 | 53.78 | 76.60 | 79.70 | 87.41 | 93.54 | 71.82 | 79.33 | 73.89 |
| CLIP-ResNet-50 | 16.11 | 87.26 | 55.89 | 40.37 | 25.79 | 62.77 | 74.82 | 82.97 | 60.85 | 59.48 | 56.63 |
| TPT | 17.58 | 87.02 | 58.46 | 40.84 | 28.33 | 62.69 | 74.88 | 84.49 | 61.46 | 60.82 | 57.66 |
| DiffTPT | 17.60 | 86.89 | 60.71 | 40.72 | 41.04 | 63.53 | 79.21 | 83.40 | 62.72 | 62.67 | 59.85 |
| TDA | 17.61 | 89.70 | 57.78 | 43.74 | 42.11 | 68.74 | 77.75 | 86.18 | 62.53 | 64.18 | 61.03 |
| TCA | 18.06 | 88.84 | 58.72 | 45.80 | 47.15 | 68.53 | 78.61 | 87.33 | 63.89 | 65.08 | 62.20 |
| TCA $5\%$ feedback | 18.81 | 89.25 | 59.22 | 47.10 | 59.36 | 69.63 | 78.75 | 88.28 | 64.65 | 68.04 | 64.31 |
| TCA $15\%$ feedback | 19.62 | 89.98 | 60.34 | 51.83 | 68.19 | 72.59 | 79.06 | 88.96 | 65.96 | 72.46 | 66.90 |

### 4.1 Comparison with State-of-the-arts Methods

#### Results under the natural distribution shifts scenario.

We first compare TCA with state-of-the-art methods in the context of natural distribution shifts. Tab. 1 presents the experimental results, revealing several key observations. (1) Leveraging distribution modeling of the representation of test data, TCA achieves superior performance without requiring gradient backpropagation. For instance, using the CLIP-ViT-B/16 backbone network, TCA outperforms the second-best method by an average of 0.99%, achieving state-of-the-art results across all datasets. (2) Performance of TCA can be further improved by incorporating human feedback. For example, with the ViT-B/16 backbone, introducing human feedback for approximately 5% of uncertain inference samples during test-time adaptation leads to an additional average performance improvement of 0.41%.

#### Results under the cross-domain generalization scenario.

Then we compare TCA with state-of-the-art methods under the cross-domain generalization scenario across 10 diverse image classification datasets, each from a distinct domain with different classes. Tab. 2 presents the experimental results. From the experimental results, we can get similar conclusions as in the natural distribution shifts scenario. First, the proposed method achieved the best performance on most datasets and the top two performance on all datasets. For example, when using the ViT-B/16 backbone network, the average performance was improved by 1.47%. Secondly, introducing human feedback during the reasoning adaptation process can further improve the performance. Especially on the EuroSAT dataset, after selecting 5% of uncertain reasoning samples for test adaptation, the performance on the ViT-B/16 and ResNet-50 backbone networks was improved by 7.35% and 12.21% respectively.

**Table 3: Comparisons of our TCA with other methods in terms of efficiency (Testing Time) and effectiveness (Accuracy). The final column shows the accuracy gain compared with the baseline.**
| Method | Testing Time | Accuracy | Gain |
| --- | --- | --- | --- |
| CLIP-ViT-B/16 | 11.82min | 68.34 | 0 |
| TPT | 447min | 68.98 | +0.64 |
| DiffTPT | 1346min | 70.30 | +1.96 |
| TDA | 22min | 69.51 | +1.17 |
| TCA (Ours) | 22min | 70.68 | +2.34 |

#### Inference time comparison.

To illustrate the efficiency of the proposed method, we conduct evaluation about the inference time using the ViT-B/16 backbone on the ImageNet (deng2009imagenet) dataset. The experimental results are shown in Tab. 3. From the table, we can see that the proposed method is faster than the methods that require gradient backpropagation. For example, TCA is 24 times faster than TPT, and 61 times faster than DiffTPT.
Therefore, test-time adaptation methods that require gradient backpropagation may not be applicable during deployment due to the performance limitations of the inference device. At the same time, compared with TDA, the speed of the proposed method is comparable, but the performance is higher.

### 4.2 Ablation Studies and Further analysis

Figure: Figure 3: Line chart depicting the improvement in model performance as the number of encountered test samples increases, compared to the performance of standard zero-shot classification.
Refer to caption: x2.png

#### Analysis of continuous learning ability.

When testing on the ImageNet dataset, we record the performance of the most recent 5,000 test samples and compare them with the original zero-shot classifier performance, recording the relationship between the improvement in model performance and the number of test samples seen. The results are shown in Fig. 3. From the experimental results, we can see that the proposed method gradually improves the model performance as the number of test samples increases. In contrast, the improvement of TDA first increases and then decreases, and it is unable to continuously learn from the test data stream. We show the performance of the last 50% of test samples and all samples on more datasets in Tab. 4. The experimental results clearly show that the performance of the last 50% of test samples is significantly higher than the overall performance. The above improvement is due to the fact that the estimated distribution becomes more reliable as the number of observed test samples increases.

**Table 4: Performance of TCA with ViT-B/16 across multiple datasets, comparing overall accuracy and the last 50% of test samples to show continuous adaptability.**
| Method | Aircraft | Caltech101 | Cars | DTD | EuroSAT | Flower102 | Food101 | Pets | SUN397 | UCF101 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TCA (All test samples) | 25.59 | 94.32 | 69.48 | 47.87 | 57.65 | 74.67 | 87.02 | 91.69 | 69.70 | 72.06 | 69.01 |
| The last 50% of test samples | 27.11 | 94.65 | 69.88 | 50.95 | 57.48 | 75.89 | 87.10 | 93.02 | 70.67 | 73.20 | 70.00 |

#### The necessity of distribution estimation.

We compared the performance of the TCA with a simplified version that only uses the mean, excluding the estimation of the Gaussian distribution by removing the covariance matrixs. This experiment aimed to understand the necessity of continual distribution estimation in enhancing model accuracy. The experimental results are shown in Tab. 5. The third row in the table presents the accuracy reductions across different datasets when the covariance matrix is removed. The results indicate a consistent decrease in accuracy across all datasets, with a particularly notable drop of 3.41% on the UCF101 dataset. These findings highlight the importance of continual distribution estimation.

**Table 5: Ablation study comparing the performance of TCA with a variant that uses only the mean, excluding the estimation of the Gaussian distribution (by removing the covariance matrix). The significant drop (third row) in model performance without distribution estimation highlights the importance of distributional test-time adaptation.**
| Method | Aircraft | Caltech101 | Cars | DTD | EuroSAT | Flower102 | Food101 | Pets | SUN397 | UCF101 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TCA | 25.59 | 94.32 | 69.48 | 47.87 | 57.65 | 74.67 | 87.02 | 91.69 | 69.70 | 72.06 | 69.01 |
| w/o covariance | 24.99 | 92.09 | 67.29 | 45.62 | 54.99 | 70.89 | 86.40 | 90.11 | 67.62 | 68.65 | 66.87 |
| -0.60 | -2.23 | -2.19 | -2.25 | -2.66 | -3.78 | -0.62 | -1.58 | -2.08 | -3.41 | -2.14 |  |

#### Effects of different uncertainty sample selection strategies.

To evaluate the effectiveness of the proposed confidence-based test-time uncertainty estimation for selecting samples to collect human feedback , we designed two alternative strategies for comparison. First, we randomly selected inference samples for human feedback. Second, we replaced the confidence in the proposed method (as described in Sec. 3.3) with the maximum cosine similarity. The experimental results, shown in Tab. 6, demonstrate that the confidence-based uncertainty sample selection method significantly improves test-time adaptation performance compared to random selection and the cosine similarity-based approach. However, designing more effective methods for identifying uncertain samples to collect human feedback remains an open problem, which we leave for future exploration.

**Table 6: Top-1 accuracy (%) of experimental results using the ViT-B/16 backbone with different methods for selecting uncertainty samples for human feedback. Random, Similarity, and Confidence refer to Randomly selecting inference samples, selecting based on zero-shot cosine similarity, and selecting based on the confidence of the zero-shot classifier, respectively.**
| Feedback Percentile | Method | Aircraft | Caltech101 | Cars | DTD | EuroSAT | Flower102 | Food101 | Pets | SUN397 | UCF101 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 5% | Random | 26.58 | 94.36 | 70.22 | 48.94 | 65.25 | 75.48 | 87.08 | 92.07 | 70.18 | 73.43 | 70.36 |
| Similarity | 27.06 | 94.36 | 70.30 | 50.24 | 63.38 | 76.17 | 87.11 | 92.42 | 70.28 | 74.41 | 70.57 |  |
| Confidence | 26.73 | 94.56 | 70.95 | 49.82 | 65.00 | 76.86 | 87.17 | 92.78 | 70.49 | 75.26 | 70.96 |  |
| 15% | Random | 28.68 | 94.69 | 71.57 | 50.83 | 74.63 | 76.37 | 87.15 | 92.34 | 70.93 | 75.71 | 72.29 |
| Similarity | 29.46 | 94.56 | 72.27 | 53.84 | 71.09 | 78.97 | 87.24 | 93.08 | 71.42 | 76.55 | 72.85 |  |
| Confidence | 28.65 | 95.01 | 73.01 | 53.78 | 76.60 | 79.70 | 87.41 | 93.54 | 71.82 | 79.33 | 73.89 |  |

#### Accuracy analysis of the selected uncertain samples.

We evaluate the zero-shot classification accuracy of the selected uncertain samples. The experimental results are shown in Tab. 7. From the table, we can see that the uncertain samples found using the proposed confidence-based method usually have lower zero-shot classification accuracy. The zero-shot classifier averages 64.59% accuracy, but for the 5% uncertain samples found by our method, it drops to 25.87%. This demonstrates that the proposed method accurately detects samples with low classification confidence, enabling efficient label collection through a human-in-the-loop approach.

**Table 7: Top-1 accuracy (%) of uncertainty samples selected by different methods. ‘Baseline’ indicates the original zero-shot classification results. Lower accuracy suggests better identification of uncertain samples by the method.**
| Feedback Percentile | Method | Aircraft | Caltech101 | Cars | DTD | EuroSAT | Flower102 | Food101 | Pets | SUN397 | UCF101 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| - | Baseline | 23.22 | 93.55 | 66.11 | 45.04 | 50.42 | 66.99 | 82.86 | 86.92 | 65.63 | 65.16 | 64.59 |
| 5% | Similarity | 19.32 | 91.95 | 51.76 | 30.36 | 5.00 | 42.86 | 54.56 | 50.00 | 55.61 | 32.22 | 43.36 |
| Confidence | 11.80 | 68.35 | 25.00 | 15.87 | 20.51 | 9.63 | 31.74 | 37.93 | 19.79 | 18.09 | 25.87 |  |
| 15% | Similarity | 21.37 | 95.74 | 58.94 | 32.70 | 13.73 | 37.89 | 63.40 | 65.74 | 55.91 | 45.68 | 49.11 |
| Confidence | 11.36 | 71.81 | 29.81 | 18.12 | 19.63 | 20.16 | 44.91 | 52.04 | 30.66 | 21.42 | 31.99 |  |

## 5 Conclusion and future work

We propose a method for continuous test-time adaptation, which enhances the original zero-shot classifier by continually adapting through online estimation of the test sample distribution and obtaining test-time posterior probabilities. To achieve this, we introduce an online distribution parameter estimation method that can estimate the distribution of test samples during testing by using the prediction probabilities from the zero-shot classification of the data stream samples. Additionally, to further adapt to uncertain samples that the base model may encounter during deployment, this work is the first to define the task of test-time adaptation, which detects uncertain samples and collects human feedback labels. By leveraging the human feedback on uncertain samples, the proposed continuous adaptation method is further improved. TCA demonstrates superior performance and comparable speed across various scenarios. In the future, we believe that exploring better test-time uncertainty estimation methods to collect human feedback and conduct test-time adaptation represents a promising direction in Human-AI collaboration.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. **Test Distribution Estimation:** In line 212 and from equations (4) and (6), it is described such that a batch of test samples arrive at each time step. However, prior baselines TPT, TDA perform single image TTA. Further, in Implementation details, in line 706, it is mentioned batch size is 1. Please clarify if single image TTA is done here as well, for fair comparison. If so, what does $n$ refer to in equations (5) and (6). How are $\mu_k$ and $\sigma_k$ estimated in single image TTA. Are you storing features, as done in TTA as well, along with the statistics?

2. **Selection of uncertain samples:**
Confidence is softmax applied over the similarity scores only right? Why is there such a large discrepancy using these two similar metrics (Table 7)? Also, is this similarity and confidence estimated from zero shot classifier or the classifier proposed? And why'd you choose what you choose?

3. **Sensitivity to hyperparameters:** How sensitive is the method on the choice of the parameters $\sigma, \eta, \rho$, as it's not practical to assume access to validation data before actually doing TTA? 

4. **Human in the loop TTA:** Please refer to the weaknesses and address the relevant concerns raised.
soundness: 2
strengths: - They propose to estimate the class distributions in an online manner. 
- The adaptive fusion of zero shot text classifier based predictions and the distribution based feature similarities is simple and intuitive. 
- This being a backpropogation free approach, is very light-weight computationally, which is a great advantage for TTA.
- The paper is well written and easy to follow, however several clarifications are required.
summary: The authors address the problem of Test Time Adaptation of Vision Langugage models. They propose to continuously estimate the distribution of test samples, which they leverage through Bayes theorem, to make the final test predictions. They also collect human feedback to receive supervision for uncertain samples.
weaknesses: 1. **Distributional Test Time Adaptation:** In a single image TTA setting, the whole section 3.2 is quite unclear. There is no **batch** of samples in this setting. So, how are the class distributions actually updated at each time step.

2. **Samples for distribution estimation:** Are all test samples used for updating the class distributions? Wouldn't the use of low confident/uncertain samples lead to bad parameter estimates in eqn(4)?

3. **TTA with human feedback:** While this is one of the major contribution of the paper, it appears to only result in modest improvements. 5% and 15% is a lot of data to ask labels for, from a human. However, the results improve only of the order of 1-2%. This makes the efficiency of this whole process questionable.

4. **Performance evaluation of TTA with human feedback:** As the test samples arrive in an online manner and based on uncertainty, how is the final accuracy evaluated here? Are these samples inclusive when evaluating accuracy? If so, you should be using the ground truth as predictions for actively labeled samples. Then the accuracy should be up by about 5% or 15%. As this is not the case in the results reported, are the labeled samples excluded from evaluation? This needs to be clarified. For fair comparison, all results should be reported on the complete test set, even when using human feedback.

5. **Need stronger baselines for TTA with human feedback:** To study the role of TTA with human feedback, stronger baselines need to be established, using different selection strategies, like random, confidence, entropy etc. and report the accuracy of complete test set. As all strategies have same amount of labeled samples included, the performance improvement due these strategies as well as the gains wrt no human feedback can be assessed.   

6. **Amount of human feedback:** 5% and 15% is a lot of supervision and this may not be feasible during test time. It's more practical to ask labels for about 1-2% of test data. Experiments with stronger baselines, with lesser supervision, with correct evaluation method, is required to actually understand and evaluate the role of human feedback in TTA.

7. **Choice of hyperparameters:**
In Implementation details, it is mentioned that validation sets are used to choose the hyperparameters. However, in TTA, one does not have access to any data from the test distribution apriori. Hence, validation data is not accessible in practice. Well, if one had access to validation data for test data, it provides a lot more information and could be used for more than just hyperparameter tuning.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. This paper is novel and interesting; would you consider making the code open source?
2. In Table 1, for ResNet-50, DOTA’s performance is only slightly better than TDA, with an average improvement of just 0.15%. I would like to understand the reason for this marginal gain.
soundness: 3
strengths: 1. This paper is well written and easy to follow.
2. The idea is interesting,  the motivation of this paper is clear as well as the novelty of the method.
3. The proposed method is extensively tested against prior work and outperforms on a variety of tasks/baselines.
summary: The author presents DOTA, a method that adapts to deployment conditions by continually estimating test sample distributions rather than memorizing them. Using Bayes’ theorem, Dota computes posterior probabilities for real-time adaptation. A human-in-the-loop feature also gathers feedback on uncertain samples, enhancing adaptability. Experiments show Dota outperforms current methods with continuous test-time learning.
weaknesses: 1. This paper does not include a detailed case study focused on a particular domain or a challenging dataset.
2. This paper does not include experiments assessing the model's sensitivity to hyperparameters. A detailed analysis of hyperparameter tuning could offer valuable insights into the robustness and generalizability of the proposed approach.
3. The paper could benefit from additional visualizations illustrating Test-time adaption with human feedback

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How does the performance change w.r.t. the data stream?
2. What is the impact of the various hyper parameter?
3. How does the test-time forgetting phenomenon happen?
4. How does the model compare with other baselines, e.g., DMN?
5. How does the work relate to existing ones on prototypical networks and the various TTA settings?
soundness: 2
strengths: 1. DOTA revisits principles in the literature on continual learning via nearest class mean classifiers (e.g., [a,b]) for improving the performance of VLMs at test time. Overall the approach is easy to implement and can be considered a valid baseline for future works, performing continuous TTA without the need for storing a cache, as in TDA.

2. The article is well-structured and easy to follow, guiding the reader through all the design choices. 

3. DOTA is effective (as shown in the comparisons with TDA, e.g., Tab. 1 and Tab. 2) and computationally cheap (as shown in Tab. 3). 

4. Fig. 3 provides an analysis of how the performance of the two models (TDA and DOTA) vary w.r.t. the number of samples, showing the advantages of the latter.

**References**:

[a] Mensink, Thomas, et al. "Distance-based image classification: Generalizing to new classes at near-zero cost." IEEE transactions on pattern analysis and machine intelligence 35.11 (2013): 2624-2637.

[b] Bendale, Abhijit, and Terrance Boult. "Towards open world recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
summary: The article addresses the problem of test-time adaptation of vision-language models. Differently from previous works (e.g., TDA, Karmanov et al. 2024) that use a cache to store samples, the proposed method, DOTA, stores an online estimate of the statistics (mean and variance) of each class of interest.  These statistics are then used during inference to refine standard CLIP predictions.  An active learning strategy exploiting this statistic is also proposed to improve the performance of the model further, asking a user to annotate the least confident examples. Experiments on a wide range of tasks show the efficacy of this approach.
weaknesses: 1. DOTA continually updates its estimates of the statistics. Those might be affected by various factors linked to the experimental protocol, (e.g., order of the classes in the stream, batch size) as well as hyperparameters choice (i.e., the initial variance value, the shrinkage $\epsilon$, $\lambda$'s hyperparameters). Currently, the article does not provide too many insights on these factors, with the analysis mostly limited to the active learning percentile (i.e., Fig. 3). To assess the robustness of the model and provide a thorough study of its performance, it would be interesting to show results across multiple data ordering (i.e., currently it is not clear how many orders have been tested) and whether the performance changes w.r.t. the particular stream considered, even on the edge-cases where the data is non-i.i.d. [c]. Moreover, the hyperparameters may impact the speed of adaptation (e.g., variance initialization, $epsilon$) as well as how much the pretrained model is considered (e.g., $\lambda$s): studying their impact is essential to fully evaluate the complexity of the approach and potential difficulties in applying it on real-world scenarios. 

2. While TPT and DiffTPT are strong models for test-time adaptation, they work on the episodic setting, i.e., where adaptation is held out on a single sample, and then the model is reset to its previous state. The possibility of storing/using test-time data (assuming coherence in the sequence) is a non-negligible advantage that DOTA has (and that it shares with TDA). This makes both the "continual adaptation" mark on TPT (Tables 1 and 2) potentially misleading, as well as TDA the only true baseline acting under the same priors of DOTA. To make the results stronger, it would be beneficial to add more baselines, such as DMN [d]. 

3. Following on the previous points, adapting to an evolving stream is a much more nuanced problem, where correlation between consecutive data may play an important role. Thus, various TTA settings with different types of stream and data dependencies (e.g., practical TTA [e], universal TTA [f]) could have been considered to further show the effectiveness of the approach. 

4. A key motivation behind DOTA relies on the test-time forgetting of TDA (lines 52-56). However, there are no experiments demonstrating this point (beyond the quantitative advantages of DOTA). An analysis clearly showing this phenomenon (and how DOTA is more robust to it) would strengthen the motivation behind the approach. 

5. The active learning strategy proposed to refine the performance for uncertain samples (Section 3.3) is a nice addition to make the approach more coherent but it lacks competitors. For instance, also TDA could employ a similar strategy (as the update of the cache is based on the confidence of the predictions). Moreover: (i) the accuracy with random feedback is also very close to those achieved with the proposed strategy (e.g., 0.6% gap on average in Tab. 6, 5% percentile); (ii) for the confidence-based scoring to work, the model is assumed to be calibrated, something not always true and that needs a proper discussion [g]; (iii) in Tab. 7 the accuracy of the random baseline is not reported: this is an important reference to put results into perspective.  

6. Related work (Section 2) provides a limited discussion on the various types of TTA settings (e.g., [e,f]) as well as on previous methods employing online updates of statistics for continual learning/open world recognition [b] or prototype-based few-shot learning [I,j]. Expanding the discussion would help to better contextualize the work in the current literature.

**Minors**: 

- Footnote 1 hints that the model could be applied beyond CLIP. However, there are no experiments confirming this claim. It would have been more thorough to show other models (e.g., SigLIP [h]) to support it.

- Table 4 shows the results only for DOTA. It would be interesting to see the same analysis for the other baselines (e.g., TDA) to contextualize/provide a reference for the results.


**References**:

[a] Mensink, Thomas, et al. "Distance-based image classification: Generalizing to new classes at near-zero cost." IEEE transactions on pattern analysis and machine intelligence 35.11 (2013): 2624-2637.

[b] Bendale, Abhijit, and Terrance Boult. "Towards open world recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

[c] Gong, Taesik, et al. "Note: Robust continual test-time adaptation against temporal correlation." Advances in Neural Information Processing Systems 35 (2022): 27253-27266.

[d] Zhang, Yabin, et al. "Dual memory networks: A versatile adaptation approach for vision-language models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.

[e] Yuan, Longhui, Binhui Xie, and Shuang Li. "Robust test-time adaptation in dynamic scenarios." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[f] Marsden, Robert A., Mario Döbler, and Bin Yang. "Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

[g] Tu, Weijie, et al. "An Empirical Study Into What Matters for Calibrating Vision-Language Models." International Conference on Machine Learning. 2024.

[h] Zhai, Xiaohua, et al. "Sigmoid loss for language image pre-training." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[i] Snell, Jake, Kevin Swersky, and Richard Zemel. "Prototypical networks for few-shot learning." Advances in neural information processing systems 30 (2017).

[j] De Lange, Matthias, and Tinne Tuytelaars. "Continual prototype evolution: Learning online from non-stationary data streams." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: see the Weaknesses
soundness: 3
strengths: 1. The writing and figures are good and easy to understand.

2. The DistributiOnal Test-time Adaptation (DOTA) method for Vision-language foundation models without BP is simple yet effective during testing in new target domain, achieving a significant improvement compared to current state-of-the-art methods in most of the datasets.

3. This paper first define the test-time adaptation problem with human feedback, allows the test-time adaptation for uncertain samples with human feedback.

4. An adaptive final fusion probability is introduced to mitigate the potential negative impact when the number of test samples is insufficient.
summary: The authors propose a Distributional Test-time Adaptation (DOTA) method, which adapt the pretrained Vision-language foundation models (e.g., CLIP) to the test target domain by estimating the distributions of different categories for test samples continually. The authors further introduce a human feedback collaboration method which identifies uncertain samples to further enhance the adaptability. Extensive experiments on diverse datasets validate the effectiveness of the proposed method.
weaknesses: 1. The proposed method estimate the data distribution of samples in the current test environment during testing, but there lack some evidence or visualization. Why updating the feature distribution for different categories works better?

2. The DOTA method seems somewhat similar to the T3A method [R1], which continually maintains a memory bank for prototypes during the testing stage. Could the authors clarify and analysis the difference and the advantages of the proposed method?

[R1] Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization

3. The proposed method seems works for not only VLMs but also other models in all classification tasks. Is it suitable for traditional TTA or Domain Generalization tasks?

4. There could give more details and explanation about the $f_k(x)$ in Eq.(3).

5. The uncertainty estimation method is simple. Is there any other uncertainty estimation method (like entropy) better?

6. There missing an ablation study for the adaptive fusion probability.

Meta-review: This paper tackles the test-time adaptation problem for CLIP using an approach termed distributional test-time adaptation (DOTA). The approach has two main ideas: the first idea is to estimate and keep track of the distribution parameters (namely mean and variance) of test samples, and the second idea is to integrate the approach with a human-in-the-loop strategy, which asks human labelers to correct wrong zero-shot predictions when the model’s confidence is lower than a threshold. The reviewers generally appreciated the first idea of distributional adaptation but remained deeply concerned about the human-in-the-loop design, which does not bring significant improvement and lacks comparisons with related work. In the panel discussion, the reviewers specifically pointed out that the marginal improvement brought by the extra human labor cost is insufficient and thus do not give support for full acceptance. After reading the paper and the reviews, the AC agrees with the view that the human-in-the-loop design does not bring enough improvement and therefore needs further justifications. Since this is the first work that combines test-time adaptation with human-in-the-loop, it is important to set up a fair, convincing benchmark for future research. The AC recommends that the paper be rejected and encourages the authors to further refine the human-in-the-loop part.

Decision: Reject