Binary-Feedback Active Test-Time Adaptation

Authors: Taeckyung Lee, Sorn Chottananurak, Junsu Kim, Jinwoo Shin, Taesik Gong, Sung-Ju Lee
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: test-time adaptation, domain adaptation, deep learning, machine learning

Abstract:
Deep learning models perform poorly when domain shifts exist between training and test data. Test-time adaptation (TTA) is a paradigm to mitigate this issue by adapting pre-trained models using only unlabeled test samples. However, existing TTA methods can fail under severe domain shifts, while recent active TTA approaches requiring full-class labels are impractical due to high labeling costs. To
address this issue, we introduce a Binary-feedback Active Test-Time Adaptation (BATTA) setting, which uses a few binary feedbacks from annotators to indicate whether model predictions are correct, thereby significantly reducing the labeling burden of annotators. Under the setting, we propose BATTA-RL, a novel dual-path optimization framework that leverages reinforcement learning to balance binary feedback-guided adaptation on uncertain samples with agreement-based self-adaptation on confident predictions. Experiments show BATTA-RL achieves substantial accuracy improvements over state-of-the-art baselines, demonstrating its effectiveness in handling severe distribution shifts with minimal labeling effort.

Decision: reject
Reviewer scores: 5, 5, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730258668680

thread_type:
official_review

<thread_object>
creator: reviewer_zvdw

<content>
summary:
Deep learning models often face performance issues when there are domain shifts between training and test data. since Test-Time Adaptation (TTA) methods have limitations, and active TTA approaches with full-class labels are impractical due to high costs, this paper proposes a Binary-feedback Active Test-Time Adaptation (BATTA) setting and a method named BATTA-RL to address the performance degradation of deep learning models due to domain shifts when testing. Specifically, in BATTA, an oracle provides binary feedback (correct/incorrect) on model predictions. This feedback is integrated in real-time to guide continuous model adaptation. A dual-path optimization framework is proposed to leverage binary feedback and unlabeled samples, which is balanced by reinforcement learning. Experimental results demonstrates the effectiveness of the proposed methods.

strengths:
1. This paper proposes a novel and practical active learning test-time adaptation paradigm. It presents an innovative approach that combines binary feedback with unlabeled samples, effectively addressing the issue of domain shifts in deep learning models when testing. In this setting, an annotator provides binary feedback (indicating whether a model prediction is correct or incorrect) instead of full-class labels. This reduces the labeling burden significantly as binary feedback requires less information compared to full - class labels. For example, the human effort and error rate in providing binary feedback are much lower than in full-class labeling as demonstrated by previous studies.

2. The proposed method, BATTA-RL, shows promising results in improving model performance with minimal labeling effort, which is a significant contribution to the field of test-time adaptation.

3. The experimental setup is comprehensive, and the comparisons with existing methods are thorough, providing strong evidence of the superiority of the proposed paradigm.

weaknesses:
1. It would be beneficial for the article to further explain why the reward functions for the two paths are set differently. For example, in Binary Feedback - Guided Adaptation (BFA), the reward function value is -1 in the incorrect case, while in Agreemend - Based Self - Adaptation (ABA), the reward function value is 0 in the case of disagreement. What are the considerations behind such designs? This clarification would enhance the understanding of the proposed method and its underlying mechanisms.

2. It would be advisable for the article to further explain in the appendix, using formulas if possible, how the baselines of previous TTA and ATTA are adapted to fit into the proposed setting to ensure fairness. This would provide more transparency and a deeper understanding of the experimental setup and comparisons made in the study.

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730461850811

thread_type:
official_review

<thread_object>
creator: reviewer_afeq

<content>
summary:
The paper introduces Binary-feedback Active Test-Time Adaptation (BATTA), a novel TTA setting for adapting deep learning models to domain shifts at test time using binary feedback from human annotators. The authors address limitations in prior active TTA methods, which suffer from high annotation costs, especially in multi-class settings. To mitigate this, they propose BATTA-RL, a reinforcement learning-based approach with a dual-path optimization strategy. BATTA-RL combines Binary Feedback-guided Adaptation (BFA) for uncertain samples and Agreement-Based Self-Adaptation (ABA) for confident samples, enhancing model performance on challenging test distributions. Experiments across multiple datasets demonstrate that BATTA-RL outperforms existing TTA methods.

strengths:
1. **Reduced Labeling Costs**: BATTA minimizes labeling demands by using binary feedback from human annotators instead of requiring full-class labels. This significantly reduces the annotation burden, making it more feasible for real-world scenarios than current ATTA methods.

2. **Dual-Path Optimization with Reinforcement Learning**: BATTA-RL combines Binary Feedback-guided Adaptation (BFA) for uncertain samples and Agreement-Based Self-Adaptation (ABA) for confident samples. Introducing reinforcement learning by binary feedback optimization is interesting and novel in TTA. 

3. **Strong Experimental Results**: BATTA-RL consistently outperforms competing TTA methods and even surpasses the ATTA method with full-class labels.

weaknesses:
1. **Experimental Setting**: Appendix D.1 mentions multiple epochs of adaptation for all datasets. However, in TTA settings—where real-time, stream-based tasks are required—multiple epochs of adaptation are impractical. This approach hinders real-time inference capabilities.

2. **Scalability of BATTA**: Without extensive experimentation on large-scale datasets, the scalability of BATTA remains uncertain. Testing on a dataset like ImageNet-C with 1000 classes would be insightful. In scenarios with high model error rates, where most feedback might be "incorrect," there is a risk of BATTA collapsing. Additionally, as shown in Table 1(c), BATTA only surpasses SAR by less than 0.3%, which is a marginal improvement.

3. **Frequency of Human Intervention**: The requirement of annotating 3 samples per batch of 64 suggests a substantial annotation budget, even with binary feedback. The need for human intervention in every test batch implies that annotators must be continuously available while the system operates. Reducing the annotation budget and frequency could make the system more practical.

4. **Computational Complexity**: Monte Carlo dropout likely introduces additional computational demands. The manuscript lacks a thorough comparison of computational complexity against competing methods. The statement that “experiments were mainly conducted on NVIDIA RTX 3090 and TITAN GPUs, with BATTA-RL consuming 5 minutes on PACS” is vague. Specific hardware details and wall-clock time comparisons with other methods, per test sample, are necessary for clarity.

questions:
Please refer to **Weakness**.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730634268095

thread_type:
official_review

<thread_object>
creator: reviewer_pxu6

<content>
summary:
Motivated by the high annotation budget of the previous active TTA, where an oracle provides accurate ground-truth labels for selected samples, this paper defines a more realistic active test-time adaptation setting (binary feedback active TTA) with relatively weak assumptions. To achieve this, the authors propose an RL framework, BATTA-RL, consisting of Binary Feedback-guided Adaptation (BFA) and Agreement-based Self-Adaptation (ABA). BFA is proposed to learn from the valuable feedback information and ABA is proposed to improve self-training with MC dropout predictions. The evaluation experiments are conducted on CIFAR10-C, CIFAR100-C and Tiny-ImageNet-C. The results show the effectiveness and excellent improvement of the proposed method.

strengths:
1. This paper introduces reinforcement learning to test-time adaptation and provides the unsupervised TTA method with low-cost feedback to improve the robustness of the TTA method.
2. The paper is well written and easy to follow, and the experiments on several benchmark datasets validate its effectiveness.

weaknesses:
1. Using the ensemble predictions of multiple data or feature augmentation (e.g., dropout) to estimate the certainty of the samples or to obtain the robust predictions is already well known in the TTA tasks, such as MEMO[A] and CoTTA[B]. One suggestion for improvement would be for the authors to compare different uncertainty estimation strategies, e.g. those used in MEMO and CoTTA.
2. Using the ensemble predictions to obtain the uncertainty of the samples could be more time consuming. It's better for the authors to calculate the wall clock time for the proposed method and compare it with others.
3. To demonstrate the effectiveness of the proposed sample selection method used in BFA, a comparison with random selection is necessary. What is the performance of the BFA module using feedback from randomly selected test samples rather than those from top-k uncertainty samples?

[A] MEMO: Test Time Robustness via Adaptation and Augmentation
[B] Continual Test-Time Domain Adaptation

questions:
See the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715027690

thread_type:
official_review

<thread_object>
creator: reviewer_8qrn

<content>
summary:
The paper proposes using binary feedback for active test-time adaptation (TTA) through reinforcement learning. It begins by employing MC-dropout to perform multiple forward passes. Based on the softmax output from MC-dropout, uncertain samples (those with low confidence) are selected and sent to an annotator for binary feedback. This feedback is then utilized in the learning process through the REINFORCE loss. For more certain samples, an Agreement-Based self-Adaptation (ABA) module is implemented, encouraging the model to maintain consistent predictions via a reward signal.

strengths:
1. The concept of using binary feedback to guide the model is intriguing.
2. Utilizing MC-dropout to estimate sample confidence is also a promising approach.
3. Experimental results demonstrate significant improvements in model performance.

weaknesses:
1. The signal acquisition process may be impractical; relying on an annotator could be costly or unrealistic in certain settings. However, advancements in foundation models may help address this issue soon.
2. The computational cost of MC-dropout is notable. Since multiple inference steps are required to gather this information, its application to TTA may raise efficiency concerns. Reviewers expect the authors to include experiments that evaluate the running time of the proposed method.
3. The fairness of the experiments is questionable. Given that this approach utilizes multiple MC-dropout steps, it effectively functions as a form of ensemble prediction, which naturally enhances performance. In contrast, other methods typically rely on a single forward pass.

questions:
Please refer to the Weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1731264175653

thread_type:
official_review

<thread_object>
creator: reviewer_uywh

<content>
summary:
BATTA is a novel approach for active test-time adaptation that uses binary feedback from a human to adapt pre-trained models to new domains, allowing reasonable labeling costs compared to methods requiring full-class labels. The BATTA-RL method integrates binary feedback on uncertain samples with self-adaptation on confident samples within a reinforcement learning framework. Extensive experiments demonstrate that BATTA-RL surpasses state-of-the-art test-time adaptation methods.

strengths:
1. The approach of using reinforcement learning, instead of the commonly used (direct) entropy minimization objective or cross-entropy loss for TTA is intriguing and adds a new perspective.
2. The approach to not solely rely on human feedback but to incorporate self-adaptation on confident samples is realistic and efficient, making it a practical strategy for TTA.

weaknesses:
1. The experiments conducted in this paper seem quite limited. It is common in current TTA to conduct evaluations on benchmarks such as ImageNet-C (additionally, ImageNet-R, and VisDA-2021). Also, to demonstrate that the use of reinforcement learning algorithms for active test-time adaptation is generally powerful, it would be important to include experiments in more complex scenarios, such as those proposed in SAR [1]: i) dynamic shifts in the ground-truth test label distribution leading to imbalanced distributions at each corruption, ii) single test sample scenarios, and iii) combinations of multiple distribution shifts in more challenging and biased situations.
2. Since the current experiments are conducted on relatively easier datasets like CIFAR-10 and Tiny ImageNet, it is likely that there are a relatively higher number of confident samples. However, in scenarios with fewer confident samples, such as those in ImageNet-C or ImageNet-R (severity level 5), wouldn't the burden of human feedback increase? In such cases, it appears that the performance may work similarly to SimATTA (full labeling). If so, what would be the key distinguishing factor of the proposed method?
3. Furthermore, as noted in DeYO [2], there can be cases where a model makes confident predictions due to spurious correlations. Would BATTA-RL still demonstrate outstanding performance when evaluated on benchmarks with significant spurious correlations? Including related observations or at least discussions on this point would be valuable.



[1] Niu, S., Wu, J., Zhang, Y., Wen, Z., Chen, Y., Zhao, P., & Tan, M. Towards Stable Test-time Adaptation in Dynamic Wild World. In The Eleventh International Conference on Learning Representations.


[2] Lee, J., Jung, D., Lee, S., Park, J., Shin, J., Hwang, U., & Yoon, S. Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors. In The Twelfth International Conference on Learning Representations.

questions:
Please address the concerns mentioned above in weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732145817213

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
We sincerely thank all reviewers for their thorough evaluation and constructive feedback. We are encouraged that the novelty and significance of our work have been recognized, and we appreciate the opportunity to strengthen our contribution through your valuable suggestions.

In our global response, we first highlight the paper's major contributions, then address the common concerns raised across reviews, and finally detail our revision changes for clarity. We have organized our response to help reviewers easily track how their feedback has been incorporated into the improved manuscript.

###  Major contributions

- Introducing a novel and practical active learning test-time adaptation paradigm (Reviewer uYWh, 8qrn, ZvDw, afeQ, ZvDw).
- Proposing a novel dual-path optimization algorithm with reinforcement learning (Reviewer uYWh, Pxu6, afeQ).
- Significant improvements over TTA/ActiveTTA methods with various datasets/scenarios (Reviewer 8qrn, Pxu6, afeQ, ZvDw).


### Common concern

- Additional experiments (Reviewer uYWH and afeQ): We conducted additional experiments on large-scale datasets and adaptation scenarios and summarized the results in the table below (also appears in  Table 3 in Section 4). In all datasets and scenarios under the BATTA setting, our BATTA-RL consistently outperformed the TTA and active TTA baselines. BATTA-RL's superior performance on large-scale datasets such as ImageNet-C demonstrates its effectiveness in large-scale test-time adaptation. The key insight is that BATTA-RL formulates both binary feedback and unlabeled sample adaptation as a single reinforcement learning objective, where the reward signals seamlessly guide the model's adaptation. The use of MC-dropout provides a robust uncertainty estimate while optimizing MC-dropout, which prevents the TTA model from overfitting and leads to a stable adaptation in large-scale datasets. Also, agreement-based adaptation (ABA) provides a robust adaptation with confident samples without requiring a fixed threshold.


| Dataset | SrcValid | BN-Stats | TENT* | EATA* | SAR* | CoTTA* | RoTTA* | SoTTA* | SimATTA* | BATTA-RL |
|---------|-----------|-----------|---------|---------|--------|----------|----------|----------|------------|------------|
| ImageNet-C | 14.43 | 26.88 | 0.93 | 30.87 | 35.15 | 22.55 | 26.80 | 36.02 | 19.50 | **36.59** |
| ImageNet-R | 33.05 | 35.08 | 29.10 | 37.14 | 36.64 | 35.02 | 34.35 | 31.00 | 35.63 | **38.59** |
| ColoredMNIST | 50.49 | 45.59 | 44.92 | 45.59 | 45.74 | 45.60 | 48.90 | 59.45 | 93.66 | **96.75** |
| VisDA-2021 | 27.36 | 26.46 | 20.38 | 27.82 | 27.41 | 26.46 | 27.23 | 27.71 | 22.80 | **29.30** |
| DomainNet | 54.82 | 54.41 | 18.80 | 59.49 | 57.78 | 54.40 | 56.41 | 54.82 | 58.41 | **60.85** |

Table R1. Accuracy (%) comparisons on additional datasets. Note: CoTTA results will be updated by this Sunday. --> Updated.


| Setting | SrcValid | BN-Stats | TENT* | EATA* | SAR* | CoTTA* | RoTTA* | SoTTA* | SimATTA* | BATTA-RL |
|---------|-----------|-----------|---------|---------|--------|----------|----------|----------|------------|------------|
| Imbalanced (non-iid) | 57.70 | 26.58 | 23.79 | 17.54 | 26.63 | 26.58 | 50.66 | 76.34 | 75.62 | **86.91** |
| Batch size 1 | 57.70 | 27.82 | 10.04 | 27.82 | 28.12 | 27.82 | 10.14 | 45.92 | - | **70.17** |

Table R2. Accuracy (%) comparisons on additional scenarios. 



- Computational efficiency (Reviewer 8qrn, Pxu6, and afeQ): We conducted a comprehensive runtime analysis by measuring the average wall-clock time per batch across different methods on the Tiny-ImageNet-C dataset. 
Our results in the table below (also appears in Table 4 in Section 4) show that BATTA-RL requires 4.19 ±0.06 seconds per batch, positioning it between simpler TTA methods (0.33-1.68s) and more complex approaches like CoTTA (26.63s) and SimATTA (45.45s).
The runtime profile demonstrates that BATTA-RL achieves a favorable balance between computational cost and performance, particularly considering its significant accuracy improvements over faster baselines while maintaining substantially lower processing time than methods like SimATTA.

| Method | Src | BN-Stats | TENT* | EATA* | SAR* | CoTTA* | RoTTA* | SoTTA* | SimATTA* | BATTA-RL |
|---------|-----|-----------|--------|--------|-------|---------|---------|---------|------------|------------|
| Avg. Time (s) | 0.18 | 0.33 | 1.03 | 0.98 | 1.02 | 26.63 | 1.68 | 1.25 | 45.45 | 4.19 |

Table R3. Average wall-clock time per batch (s) comparisons.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses the challenge of adapting deep learning models to test-time domain shifts with minimal labeling costs. It introduces Binary-Feedback Active Test-Time Adaptation (BATTA), where an oracle (human) provides binary feedback (correct/incorrect) on selected predictions instead of full-class labels.

While this paper presents an interesting and novel angle with merits in its use of RLHF to update the model, the AC agrees with Reviewer 8qrn that ATTA could benefit significantly from leveraging a stronger foundation model. This includes not only using an advanced foundation model as an oracle to generate RL rewards but also at least experimenting with a larger teacher model to periodically guide the model at test time and measure the resulting performance gains. Although the authors state that exploring foundation models as oracles is a direction for future work, this investigation would greatly enhance the current paper by providing a deeper understanding of the benefits of human-in-the-loop approaches.