Test-Time Ensemble via Linear Mode Connectivity: A Path to Better Adaptation

Authors: Byungjai Kim, Chanho Ahn, Wissam J. Baddar, Kikyung Kim, HUIJIN LEE, Saehyun Ahn, Seungju Han, Sungjoo Suh, Eunho Yang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: test-time adaptation, domain adaptation, linear mode connectivity

Abstract:
Test-time adaptation updates pretrained models on the fly to handle distribution shifts in test data. While existing research has focused on stable optimization during adaptation, less attention has been given to enhancing model representations for adaptation capability. To address this gap, we propose Test-Time Ensemble (TTE) grounded in the intriguing property of linear mode connectivity. TTE leverages ensemble strategies during adaptation: 1) adaptively averaging the parameter weights of assorted test-time adapted models and 2) incorporating dropout to further promote representation diversity. These strategies encapsulate model diversity into a single model, avoiding computational burden associated with managing multiple models. Besides, we propose a robust knowledge distillation scheme to prevent model collapse, ensuring stable optimization and preserving the ensemble benefits during adaptation. Notably, TTE integrates seamlessly with existing TTA approaches, advancing their adaptation capabilities. In extensive experiments, integration with TTE consistently outperformed baseline models across various challenging scenarios, demonstrating its effectiveness and general applicability.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729937228204

thread_type:
official_review

<thread_object>
creator: reviewer_sf3m

<content>
summary:
This paper considers a new problem, test-time ensemble (TTE), which aims at using multiple models generated during TTA. This paper first formulates the test-time ensemble problem. The paper also proposes the weight average and dropout as the baseline methods to evaluate the performances. 

The contributions can be summarized as: 

(1) The author revealed that TTA models exhibit linear mode connectivity, an in- triguing insight that simplifies and enhances the adaptation process.

(2) The author introduced Test-Time Ensemble (TTE), a novel and computationally efficient approach that not only enriches model representations but also stabilize TTA optimization through de-biased and noise- robust knowledge distillation.

(3) TTE integrated effortlessly with existing TTA methods, enhancing adaptation in diverse scenarios and showing potential for applicability to future TTA methods.

strengths:
(1) This paper proposes the new problem, test-time problem, which is different from previous test-time adaptations. I believe this problem has some practical applications.

(2) The paper proposes some simple baseline methods that can effectively address the problem.

weaknesses:
(1) The analysis of test-time adaptation does not inspire the new methods. The moving average ensemble methods are popularly adapted in self-supervised learning and ensemble methods. I consider the Linear Mode Connectivity theory should tell the reason and the situation that the models generated during test-time adaptation. 


(2) Limited technical novelty: this paper proposes the two-branch structure and leverage the weight average to improve the performance. Similar techniques are implemented in https://github.com/huggingface/pytorch-image-models. I do not see anything new compared to what have been proved in image classification.

(3) Unclear description. In section 3, the de-biased distillation subsection does not describe clearly where the bias comes from. I suggest the author should explain the bias again. Also, I can not understand what the connection between the spike phenomena of the accuracy curve and the bias.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730147314705

thread_type:
official_review

<thread_object>
creator: reviewer_nkqc

<content>
summary:
The paper introduces the Test-Time Ensemble (TTE), a method designed for TTA using the theory of weights space ensemble, which can be used on top of different TTA methods. The authors show different results for TTA over corruptions with different baselines, and the method seems to work pretty well. Furthermore, the authors also provided results for continual TTA, which is interesting.

strengths:
The paper is well-written, easy to follow, and detailed. I liked how the authors presented the work and motivated toward the problem. Furthermore, the results are motivating, and the idea seems easy to implement on top of different methods (as demonstrated by the authors), which can be beneficial for the community if the authors also provide the full code for reproducibility.

weaknesses:
Personally, I did not see many problems with the paper, but I would suggest the authors proofread again to avoid problems such as the following typo "with lager and more complex" -> "with larger and more complex" in the introduction or "Adaptvie momentum" ->  "Adaptive momentum."  

If the authors work on the following points, it will improve a lot the quality of the work: 
- I am not so convinced by section 3.2, DE-BIASED AND NOISE-ROBUST KNOWLEDGE DISTILLATION. Could you clarify this a bit more in this section? And maybe make it more clear in the paper.

-  In Equation 5, there is no hyperparameter to balance the terms. I think it should be included, right? 

- For Table 3, with continual TTA, the authors compared all other baselines with TTE with DeYO, but not TTE with other variants as well, and for continual TTA, I don't understand how the performance can still perform well in the direction of the adaptation without degrading too much as we can see in the other approaches (for example DeYO goes from 28.1 to 3.7 and then 7.2), for me it only make sense if you ensemble with zero-shot model as well (or reset the model weights) but if this is the case it should be done for all other baselines as well.
 
- For Table 4, column V2 seems strange, as almost all results in a) are 68.9, even DeYO and DeYO with TTE. Could you also add more results with other batch sizes? (the batch size can play an important role in different algorithms, which can benefit DeYO and not the others. For instance, I recommend taking a look at the paper "Bag of Tricks for Fully Test-Time Adaptation, IEEE/CVF Winter Conference on Applications of Computer Vision. 2024",  which shows the role of batch size in some of the TTA algorithms.

- I would suggest revisiting some of the baselines for Tab 1. I would also consider a baseline with other methods of the local ensemble as well, such as SWA with TTA, and for Tab 4. I would also add other methods with TTE (maybe in the supp. material).


Rebuttal period: All my points were answered well, and I am quite happy with the effort of the authors during the rebuttal phase.

questions:
Here, I am adding the questions that I find relevant for improving the work quality; some of them were already discussed in the Weaknesses section:

- Could you clarify this a bit more in section 3.2? How is it important for the method?

- Do you think the batch size can impact the results, especially the ones provided for the continual TTA?

Please consider answering the points on the weaknesses as well. Furthermore, I am open to discussion, and I think that the work has a good potential for the community.

Rebuttal period: After carefully reading all the answers provided by the authors, I feel that my questions were answered, and I don't have any additional questions. I am confident to change my decision from "marginally above the acceptance threshold" to "accept, good paper".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730969119719

thread_type:
official_review

<thread_object>
creator: reviewer_xgvc

<content>
summary:
The paper proposes a method called Test-Time Ensemble (TTE), which uses an ensemble strategy to dynamically enrich model representations during online test-time adaptation (TTA). TTE constructs an ensemble network by averaging the parameter weights of different TTA models, which are continuously updated using test data. This weight averaging technology captures model diversity and improves representation quality without increasing the computational burden of managing multiple models. TTE further combines dropout to promote diverse collaboration of representations within TTA models, and also proposes a debiased and noise-resistant knowledge distillation scheme to stabilize the learning of TTA models in the ensemble. TTE can be seamlessly integrated with existing TTA methods, enhancing their adaptive capabilities in various challenging scenarios.

strengths:
1. TTE utilizes an ensemble strategy to dynamically enrich model representations during online test-time adaptation (TTA), which is an interesting approach.
2. TTE constructs an ensemble network by averaging the parameter weights of different TTA models, and this weight averaging captures model diversity, improving representation quality without increasing the computational burden of managing multiple models.
3. TTE further promotes the diversity of representations within TTA models by combining with dropout, and proposes a debiased and noise-resistant knowledge distillation scheme to stabilize the learning of TTA models in the ensemble.
4. The experiments are extensive and the results are superior to other compared methods.

weaknesses:
1. The TTE method involves multiple hyperparameters, such as the momentum coefficient , dropout ratio and temperature, which may affect the stability and generalization of the method. Further research on how to reduce the dependence on hyperparameters is crucial.
2. TTE integrates many well-established techniques such as ensemble, dropout, knowledge distillation, which have been utilized in TTA or few shot learning. The combination has weaken the novelty of the paper, and the unique contributions of the paper should be classified.
3. How to conduct the weight-space ensemble without adding computational complexity? Will the technique increase storage consumption?
4. The knowledge distillation-based debiasing and anti-noise strategies proposed in the paper may not be able to completely solve the problem of noisy test data. How to solve the scenario that the pseudo labels are incorrect?

questions:
1. Why the results of CoTTA in Continual TTA with non-i.i.d. conditions are only 2.2% and 3.4%?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731070137474

thread_type:
official_review

<thread_object>
creator: reviewer_drm4

<content>
summary:
This paper introduces a novel test-time ensemble approach that can be seamlessly integrated with existing TTA models to enhance adaptation. Specifically, the proposed framework reduces domain gaps through two ensemble strategies: weight averaging of TTA models and dropout. Additionally, a knowledge distillation strategy is employed to mitigate both noise and bias for improving model robustness under different distribution shifts. 
Extensive experiments are conducted in different TTA scenarios to demonstrate the superiority of the proposed method over existing baselines.

strengths:
1. Writing quality is good. The paper is well-structured, and clearly written. 
2. Good insights. This paper explores TTA as a domain generalization problem, uncovering linear connectivity within TTA models. This perspective suggests that domain generalization techniques could enhance model representations for TTA tasks. 
3. SOTA performance. The proposed method achieves the state-of-the-art performance via the integration with different TTA models in various scenarios.
4. Ablations. Ablation experiments are provided to verify the effectiveness of the proposed modules.

weaknesses:
1. Although the results presented in Tables 3 show the performance improvement achieved by the proposed framework in the continual TTA scenario, it is unclear how the method enhances baseline performance in later adaptation stages. Additionally, I would like to know if the proposed method addresses the issue of catastrophic forgetting in this context. 

Additional question:
Is it possible to extend the proposed benchmark construction method to dense prediction tasks, such as semantic segmentation? It would be very meaningful if it can be applied to various tasks.

questions:
Please refer to the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732354268928

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the reviewers for their thoughtful feedback and valuable insights. Your comments have helped us identify areas to clarify and strengthen our work. Below, we summarize the key strengths of our contribution, recognized by reviewers.

### **Summarized contributions recognized by the reviewers.**

- The problem formulation, **exploring TTA as a domain generalization problem**, is well-motivated [drM4, NKQc] and is initially proposed in this paper [sF3M].

- **Uncovering linear mode connectivity within TTA models** provides valuable insights [drM4] and enables the development of an **adaptive ensemble strategy** for improving model representations during online TTA [drM4, xGvc].

- TTE also introduces a **debiased and noise-robust knowledge distillation scheme** that stabilizes TTA optimization [xGvc].

- TTE achieves **state-of-the-art (SOTA) performance**, surpassing conventional TTA models [drM4, xGvc], supported by well-motivated experimental results [NKQc].

- **Extensive ablation studies** validate the effectiveness of the proposed modules and highlight their contributions to the overall performance [drM4, xGvc].

- TTE is **practical, easy to implement with baseline models** [NKQc, sF3M], and beneficial to the research community [NKQc].

### **Major revisions in the updated manuscript**

Following the reviewers' suggestions, we have thoroughly revised the paper and uploaded the updated version, which we believe has significantly enhanced the quality of this paper. Here, we would like to highlight the major revisions, as follows.

#### **Introduction**
- The introduction has been revised to clarify the concept of TTE as an adaptive weight ensemble scheme.
- The discussion of computational costs has been softened to provide a more balanced perspective.

#### **Preliminaries**
- Additional analysis has been conducted to validate the property of linear mode connectivity under various potential TTA scenarios, further supporting the motivation for TTE.

#### **Methods**
- The distinct contributions of adaptive ensemble strategies and their significance in online TTA have been clarified and are now more prominently presented in the revised paper.
- Section 3.2 has been revised to improve the clarity and understanding of the proposed knowledge distillation method.

#### **Experiments**
- Additional experiments were conducted with other baseline models (Tent[1]+TTE and SAR[2]+TTE). We revisited the experiments in Tables 3 and 4 to further validate the effectiveness of TTE.
- Additional experiments under continual TTA with non-i.i.d. conditions demonstrated the stability of TTE and its ability to prevent catastrophic forgetting.
- Additional comparison studies with stochastic weight averaging [3], a prior method for offline domain generalization, were performed to highlight the advantages of TTE.

Furthermore, we will incorporate all the responses provided during the discussion phase into the final manuscript. For details, please refer to our responses to each reviewer’s comments.

> References
> 1. Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. International Conference on Learning Representations (ICLR) 2020.
> 2. Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. International Conference on Learning Representations (ICLR) 2023.
> 3. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. Conference on Uncertainty in Artificial Intelligence (UAI) 2018.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a method called Test-Time Ensemble (TTE), which enriches model representations during online test-time adaptation (TTA) inspired by domain generalization. The proposed method is well-motivated and technically solid. The experimental results are extensive and convincing. The authors successfully addressed the major concerns raised by the reviewers about the technical details, novelty, possible extension, experiments, and writing during the rebuttal. All reviewers agree to accept this paper. Overall, it is a technically solid paper that meets the expectations of the ICLR community.