Title: Buffer layers for Test-Time Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Test-time adaptation

source-free adaptation

buffer layer

layer insertion
Abstract: In recent advancements in Test Time Adaptation (TTA), most existing methodologies focus on updating normalization layers to adapt to the test domain. However, the reliance on normalization-based adaptation presents key challenges. First, normalization layers such as Batch Normalization (BN) are highly sensitive to small batch sizes, leading to unstable and inaccurate statistics. Moreover, normalization-based adaptation is inherently constrained by the structure of the pre-trained model, as it relies on training-time statistics that may not generalize well to unseen domains. These issues limit the effectiveness of normalization-based TTA approaches, especially under significant domain shift. In this paper, we introduce a novel paradigm based on the concept of a \textit{Buffer} layer, which addresses the fundamental limitations of normalization layer updates. Unlike existing methods that modify the core parameters of the model, our approach preserves the integrity of the pre-trained backbone, inherently mitigating the risk of catastrophic forgetting during online adaptation. Through comprehensive experimentation, we demonstrate that our approach not only outperforms traditional methods in mitigating domain shift and enhancing model robustness, but also exhibits strong resilience to forgetting. Furthermore, our \textit{Buffer} layer is modular and can be seamlessly integrated into nearly all existing TTA frameworks, resulting in consistent performance improvements across various architectures. These findings validate the effectiveness and versatility of the proposed solution in real-world domain adaptation scenarios. The code is available at https://github.com/hyeongyu-kim/Buffer_TTA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel concept of \"Buffer Layer\" for Test Time Adaptation (TTA) scenarios. This concept breaks through the traditional method of relying on normalization layer updates and addresses the problems of statistical instability of normalization layers in small batch sizes. The design of the buffer layer preserves the integrity of the pre-trained model and mitigates the risk of the catastrophic forgetting problem during online adaptation. Experimental results show good performance improvement in CIFAR and ImagNet corruption datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed Buffer Layer is a plug-and-play module and can be seamlessly integrated into any pre-trained model architecture. This modularity makes it highly applicable to a wide range of models without requiring significant modifications to the original architecture.\n2. The approach effectively mitigates the risk of catastrophic forgetting, which is a common issue in online adaptation scenarios. By preserving the integrity of the pre-trained backbone and isolating the adaptation process to the Buffer Layer, the model retains critical knowledge learned during pre-training. This ensures that the model does not lose its original capabilities while adapting to new domains, making it more robust and reliable in dynamic environments.\n3. The Buffer Layer shows strong compatibility with existing TTA methods, consistently providing additional gains when combined with traditional normalization-based adaptation techniques. This integration enhances performance across different test-time batch sizes, demonstrating its effectiveness in both small and large batch scenarios.\n\nWeaknesses:\n1. Limited Backbone Diversity: While the paper claims that the Buffer Layer can be integrated across various architectures, the experiments are limited to ResNet-based backbones. The effectiveness of the Buffer Layer on other popular architectures, such as those based on Transformers, remains unexplored. Given the increasing prevalence of Transformer-based models in various domains, it is essential to validate the Buffer Layer's performance on these architectures to fully assess its applicability.\n2. There is no detailed analysis of the computational complexity or testing time cost after integrating the proposed Buffer Layer. Understanding the additional computational overhead introduced by the Buffer Layer is crucial for practical deployment, especially in resource-constrained environments. A thorough analysis of the trade-offs between performance gains and computational costs would provide a more comprehensive evaluation of the Buffer Layer's practicality.\n3. The proposed Buffer Layer shares some similarities with existing techniques such as Adapters or Side Tuning, which are primarily used in parameter-efficient fine-tuning.\n4. The paper does not include experiments with a batch size of 1, which is a common scenario in real-world applications, especially in online or streaming settings.\n\nQUESTIONS:\nSee the weaknesses. By addressing these weaknesses, the paper would provide a more comprehensive and robust evaluation of the Buffer Layer, further solidifying its contributions and potential impact in the field of test-time adaptation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConventionally, test-time adaptation methods often update the parts of the model parameters, such as the affine parameters in the normalization layers, or update the entire parameters, but these methods suffer from the problem of ineffectiveness when the batch size is small and the catastrophic forgetting problem. This paper addressed these problems by introducing a light-weight network module, called Buffer Layer, and updating it while freezing the original network parameters.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengthes**\n1. The proposed method is simple and effective, and　it has a significant contribution to this research area: This paper proposed a very simple method that adds lightweight modules to the base network, yet it effectively addresses the problems of the conventional approaches. In addition, in the experiments, the proposed method demonstrated its effectiveness when introduced to several existing methods, which indicates the broad compatibility of the method. These results demonstrate the potential to completely replace the conventional approach of training only normalization layers, and I believe that this will be a significant contribution to this field.\n2. Comprehensiveness of the Experiment: The experiments and analysis were comprehensive and adequately supported the claims of this study.\n3. Paper organization: This paper is well-written and easy to follow.\n\n\n**Weaknesses**\n1. Comparison to parameter-efficient fine-tuning (PEFT) methods: The proposed method is similar to the existing PEFT methods (e.g., LoRA), which update only a small number of additional parameters. It is necessary to discuss the difference between these methods and the proposed method.\n2. Applicability to the transformer-based networks: The proposed method seems to be designed for the convolution-based networks like ResNet, but there remains a concern about the applicability to the non-convolution-based networks like Vision Transformer. The modern neural network is mainly built upon the transformer-based architecture, and thus, it is necessary to investigate the applicability to such transformer-based models. In addition, this is related to the weakness 1., LoRA is applicable to the transformer-based models, so it is interesting to compare the effectiveness of the proposed method and LoRA.\n3. Analysis about the forgetting: In the experiments, it is shown whether the proposed method is effective against forgetting, but there seems to be a lack of discussion or analysis of why it is effective. The authors explained this is because the original model parameters are frozen, but I think the activations inside the network are changed by the buffer layer training, and this change may cause catastrophic forgetting. I believe that analyzing the mechanism by which the proposed method prevents forgetting will lead to a further understanding of the proposed method.\n4. Minor points:\n    + Typo in l135: Sec Sec. -> Sec.\n    + Figure 3 is not referred to in the main text.\n    + In Figure 3, I'm not sure what \"DF\", \"KW\", and \"KWC\" mean.\n\nQUESTIONS:\nThe paper was clear and easy to understand, so I don't have any additional questions. I have summarized my concerns in the \"weakness\".\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the problem of test-time adaptation (TTA) from the angle of which parameters to update, instead of how to update them. The proposed method builds on the idea that most approaches update the batch norm (BN) parameters of the model, however these parameters are highly sensitive to small batchsizes. Similar to EcoTTA, the method keeps the model untouched and instead adds a Buffer module on top of initial layers of the network, which is adapted instead of the BN parameters using the usual entropy or augmentation consistency losses. In a comprehensive set of experiments, authors show that adding their Buffer Module on top of existing TTA boosts their accuracy under different types of data corruption settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* The method is simple but effective. The Buffer module can be added on top of most TTA approach, without any modification, to improve their robustness to corruptions at test time.\n\n* As the method is rather simple to explain, the paper instead builds on evaluating it under a broad range of settings. Overllal, results of these experiments are convincing.\n\n* The paper is quite well written. Thanks !\n\nWeaknesses:\n\n* Although mentioned very quickly in Section 2.1, the paper does not properly present the efforts of previous works to address the instability of BN update for small batchsizes. Those include SAR which replaces the BN by Layer Norm or Group Norm, and RoTTA which updates the statistics using an exponential moving average. \n\n* Some of the reported results do not match with those published in previous works. For example, in Table 3, authors report a classification error of 94.34% for SAR in the BN=2 and GN setting, while the error reported in the SAR paper for the same setting is 65.5%.\n\n* Moreover, some comparisons in the experiments are a bit unfair. For instance, the Layer Norm variant of SAR  (on a ViT backbone) achieves much stronger results, but authors only evaluate their method on CNN backbone. Furthermore, EcoTTA is not included in the comparison since it requires initializing the external module with source data. However, a better picture would be obtained by adding this method, with this initialization and also using a random initialization like in the paper.\n\n* As the module starts with randomly initialized weights, its prediction for the first test samples might be very noise. This problem is alleviated by employing a small alpha to reduce the module's initial influence. I believe this problem should be discussed with greater depth in the paper.\n\nQUESTIONS:\nPlease see the weaknesses above. Technically, my current evaluation is closer to a Weak Accept. I would be willing to upgrade my score to Accept if my concerns are addressed by authors.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper proposes a light-weight buffer layer to address the catastrophic forgetting problem of TTA. With extensive experiments, the paper shows that this simple plug-and-play modification of the model can achieve surprisingly better performance when inserted into any other TTA methods.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is easy to read and follow. The motivation is clear.
2. The proposed module is simple and can be easily integrated into any other TTA methods, which makes it have wide applications.
3. All the experiments, including conventional CIFAR-10 & CIFAR-100C datasets, and several datasets with domain shifts, validate the effectiveness of the proposed method.

Weakness:
1. The proposed method is similar to the weight expansion of continual learning, which should be discussed in the related work.
2. How does the number of paralleled buffer layers (e.g., multiple 1x1 conv) influence the final performance?
3. In Table 1., some results of employing buffer layer drops for ROID,  which should be explained why it happens and when the proposed method works or not.
4. How does the proposed method compare to LoRA, which can be employed at BN or Conv layers?

QUESTIONS:
See the above Weakness.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03