Buffer layers for Test-Time Adaptation
Authors: Hyeongyu Kim
GeonHui Han
Dosik Hwang
Venue: NeurIPS 2025 poster ()
Keywords: Test-time adaptation
source-free adaptation
buffer layer
layer insertion
Primary area: deep_learning
TL;DR: We propose a modular Buffer Layer for Test-Time Adaptation that overcomes the limitations of normalization-based methods, offering superior robustness and generalization under domain shift.

Abstract: In recent advancements in Test Time Adaptation (TTA), most existing methodologies focus on updating normalization layers to adapt to the test domain. However, the reliance on normalization-based adaptation presents key challenges. First, normalization layers such as Batch Normalization (BN) are highly sensitive to small batch sizes, leading to unstable and inaccurate statistics. Moreover, normalization-based adaptation is inherently constrained by the structure of the pre-trained model, as it relies on training-time statistics that may not generalize well to unseen domains. These issues limit the effectiveness of normalization-based TTA approaches, especially under significant domain shift. In this paper, we introduce a novel paradigm based on the concept of a \textit{Buffer} layer, which addresses the fundamental limitations of normalization layer updates. Unlike existing methods that modify the core parameters of the model, our approach preserves the integrity of the pre-trained backbone, inherently mitigating the risk of catastrophic forgetting during online adaptation. Through comprehensive experimentation, we demonstrate that our approach not only outperforms traditional methods in mitigating domain shift and enhancing model robustness, but also exhibits strong resilience to forgetting. Furthermore, our \textit{Buffer} layer is modular and can be seamlessly integrated into nearly all existing TTA frameworks, resulting in consistent performance improvements across various architectures. These findings validate the effectiveness and versatility of the proposed solution in real-world domain adaptation scenarios. The code is available at https://github.com/hyeongyu-kim/Buffer_TTA.

Buffer layers for Test- Time Adaptation
Hyeongyu Kim! Geonhui Han' Dosik Hwang! 12.3
School of Electrical and Electronic Engineering, Yonsei University
Department of Radiology and Center for Clinical Imaging Data Science, College of Medicine, Yonsei University Artificial Intelligence and Robotics Institute, Korea Institute of Science and Technology {lion4309, hgh6945, dosik basik-hwang@yonsei. ac.kr)
Abstract
In recent advancements in Test Time Adaptation (TTA), most existing methodolo- gies focus on updating normalization layers to adapt to the test domain. However, adaptation presents key challenges. First, nor-
the reliance normalization-bas malization layers such as Batch Normalization (BN) are highly sensitive to small batch sizes, leading to unstable and inaccurate statistics. Moreover, normalization- based adaptation is inherently constrained by the structure of the pre-trained model, it relies on training- time statistics that may generalize well to unseen do- mains. These issues limit the effectiveness of normalization based TTA approaches, especially under significant domain shift. In this paper, we introduce novel paradigm based on the concept of Buffer layer, which addresses the fundamental limitations of normalization layer updates. Unlike existing methods that modify the core parameters of the model, our approach preserves the integrity of the pre- trained backbone, inherently mitigating the risk of catastrophic forgetting during online adaptation. Through comprehensive experimentation, we demonstrate that our approach not only outperforms traditional methods mitigating domain shift and enhancing model robustness, but also exhibits strong resilience to forgetting. Furthermore, our Buffer layer is modular and can be seamlessly integrated into nearly all existing TTA frameworks, resulting in consistent performance improve- ments across various architectures. These findings validate the effectiveness and versatility of the proposed solution in real-world domain adaptation scenarios. The code is available at ftthtps:: :/gp:::///tttiie thtbb...t..tt/..///g com/ hamm/hyenngyuuBee kim/ Hann/Bufer.... _TAA
1 Introduction
Recent progress in deep learning has led to remarkable advancements, largely driven by deep neural networks (DNNs) and large-scale datasets 10 enabling unprecedented performance across various tasks. However, models still face challenges when tested on data that differs from the training data, known as domain shift, which occurs when the data distribution in the target domain diverges from the source domain [8] To tackle this, several strategies have emerged, including domain adaptation (DA) and domain generalization (DG) 3 26 30). DA adapts models target domains using labeled source data and unlabeled target data, but it requires access to both source and target domain data, which is often not feasible in real-worlds settings. Furthermore, DA methods are typically not online and cannot continuously adapt after deployment. DG, on the other hand, aims to generalize across domains without requiring target domain data during training, making it useful for unseen domains, but it struggles with capturing domain-specific patterns and often needs a diverse set of source domains, which may always be available.
"Equal contribution
i Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Dest-Time/Att Adaptation (TTA) has emerged as a practical solution to address domain shift, allowing models adapt the target domain during the inference phase typically without requiring access to source domain data 25 The core idea of TTA is to adjust the model based on the unlabeled target domain data while leveraging the pre-trained model, focusing on minimizing the discrepancy between the target and source domains, often by using criteria such as entropy minimization. TTA methods typically update either small subset of parameters, such as normalization layers (particularly Batch Normalization, BN) 04 23 17 29
16 4 12 22 I1 18 66, or the entire model [9 24 [. However, these strategies face key challenges. Normalization based methods suffer from unreliable statistics with small batch sizes, while full model updates can incur high memory and computational costs due to backpropagation through the entire network. These limitations hinder the practical deployment of TTA in essource-constraiigg scenarios.
In this work, we revisit test-time adaptation by introducing lightweight and modular Buffer layer that can be seamlessly inserted into any pre-trained model, enabling efficient adaptation during inference without modifying the backbone parameters. Unlike prior methods that rely on updating normalization layers fine-tuning the entire model, our approach delegates the adaptation task to an auxiliary network specifically designed to mitigate domain shift at test time. This design circumvents the instability of normalization-bin bassd methods under small batch sizes and avoids the computational burden of full backpropagation through the main network. Moreover, Buffer layer can optionally be co-trained with normalization parameters, offering flexible and extensible mechanism for robust adaptation in diverse deployment scenarios.
We evaluated our method across several widely used TTA benchmarks, including CIFAR10-G CIFAR- 10-W, CIFAR 100-C, and ImageNet-C, comparing it against state-of-the-ae techniques spanning both conventional and temporally aware adaptation methods. Our approach is highly modular and method-agnostic, allowing seamless integration into diverse frameworks for a wide range of TTA tasks. Notably, when combined with normalization-bal TTA methods, our Buffer layer consistently provided additional gains, highlighting its compatibility and effectiveness as a general enhancement module. In addition, Buffer layer serves as an isolated adaptation unit, effectively mitigating catastrophic forgetting by preserving the original model parameters, which is common limitation in conventional TTA approaches.
adaptation, which is used in most existing
Contributions
(1) A modular and scalable adaptation framework: We introduce a sluu-andd-lay Buffer layer that can be seamlessly integrated into any pre- pre-trained model architecture. Its modularity makes it applicable across a wide range of test-time adaptation scenarios without altering the original network.
(2) Consistent performance gains and broad compatibility: The proposed Buffer layer delivers substantial accuracy improvements across diverse TTA benchmarks. It also enhances performance of existing normalization-bast based methods when combined, demonstrating strong compatibility.
(3) Forgetting- resilient resiilint adaptation design: By isolating adaptation into a dedicated component, our approach mitigates catastrophic forgetting of pre-trained knowledge, a common and critical limitation in many online TTA techniques.
2 Background
2.1 Test-Time Adaptation
TTA has emerged as a practical solution to domain shift, enabling models to adjust to target distribu- tions during inference without requiring access to source domain data. By leveraging a pre-trained model and adapting it on unlabeled target samples, TTA climinates the need for retraining or supervi- sion, making it appealing for real-world deployment under privacy and computational constraints. Existing TTA methods can be broadly categorized based on what components of the model are adapted and how the adaptation performed. One common approach is BN adaptation, where methods such as [14] 23 18 17 update BN affine parameters via entropy minimization. Another category pseudo-label -bsed adaptation, which uses confident model predictions as supervisory signals 227 24 15 Methods like [28 [6] 21 leverage test-time augmentation and consistency regularization to improve robustness. Despite their differences, these approaches exhibit trade-offs between flexibility, stability, computational cost, and applicability in source-free settings.

---

Conventional TTA Methods
Buffer TTA
pre teee-traindd models
Cando
Reili
BatchWrrmd
Can20
Reuli
Monil
BatchNormD
Update Norm
Fine tunittunigg
Figure I: Overview of our test-time adaptation framework. Unlike prior methods that rely on updating normalization layers or fine-tuning the entire model, which require backpropagation and can suffer from instability under small batch sizes, additive modules that require warm-up phases, proposed Buffer layer enables direct test-time adaptation without any additional training. It operates on any type of objectives mitigate domain shift, acting as a lightweight and modular adaptation unit that preserves the original model parameters and prevents catastrophic forgetting.
2.2 Limitations and Reformulation of the Problem
Despite these advancements, most TTA research has predominantly focused on how to update a model, for example through entropy minimization, consistency regularization, or confident sample selection, while paying relatively little attention to what to update. Typical update targets include either the full model BN layers, both of which involve trade-offs between stability and computational efficiency,
Although more recent works such as EcoTTA 21 and L-TTA (20) attempt to address these limitations by freezing the backbone and adapting only lightweight auxiliary components, they still require warm-up phase using source data for initialization. This reliance on the source domain undermines the premise of source-free adaptation and limits their applicability in practical deployment scenarios.
In this work, we reformulate the problem by proposing to update dedicated auxiliary layer, referred to the Buffer layer, which is integrated into the network in a modular fashion. Unlike existing approaches, our method delegates the adaptation process to this compact and trainable unit, leaving the backbone untouched. This design mitigates catastrophic forgetting and supports stable adaptation under small batch sizes while maintaining computational efficiency. Moreover, our approach operates in fully online and truly source-free manner, requiring additional initialization access to source data at any stage.
3 Methods
3.1 Motivation
TTA methods have predominantly focused updating normalization layers, particularly batch normalization (BN), under the assumption that domain shift can be mitigated by adapting normaliza- tion statistics and affine parameters, i.e., (Js: Ts Ts, Ba) and their target counterparts (μ4, dz, nh (.). While intuitive, this assumption reduces domain shift to marginal distribution alignment and neglects more complex, class-conditional, or higher-order shifts.
Moreover, normalization layers are not explicitly designed for adaptation. Their reliance on batch statistics makes them unstable under small batch sizes, and modifying them risks disrupting the pretrained backbone. This motivates us to move beyond normalization layers and explore a structurally decoupled alternative.
In contrast to conventional BN-based methods, we propose a lightweight and modular Buffer layer that serves as an external adaptation unit, inserted into the network without altering the original architecture. By isolating adaptation using this auxiliary layer, we achieve scalable and stable mechanism for source-free free TTA, without relying BN statistics or internal parameters of the model.

---

3.2 Buffer layer: Modular Adaptation for Source-Free TTA
We introduce the Buffer layer, lightweight convolutional module for test-time adaptation, inserted in parallel to the pretrained backbone without altering its original parameters. As illustrated in Fig [ each Buffer layer consists of simple 1x1 and 3x3 convolutions whose outputs are scaled by learnable coefficient, and residually added to the original activations. Buffer layers are only inserted in the early stages of the network and can be optimized using any standard TTA objective, such as entropy minimization or consistency regularization.
In contrast prior approaches that directly update BN layers, our method externalizes adaptation to modular unit that only activates during inference. This ensures stable updates even under small batch sizes and avoids disrupting pretrained statistics. Furthermore, the Buffer layer can be optionally co- co-trained with normalization parameters, offering flexibility to adapt jointly when batch statistics are reliable. See Sec. 4. for architectural details.
4 Experiments
4.1 Baseline methods
We revisit a range of representative TTA methods and reinterpret them from the perspective of what and how to update. Many otate-of-the-art approaches, such as TENT (23] CMF [II], EATA [17], SAR [8], DO [12, and ROID [16], are built upon updating BN layers in various ways. For example, TENT minimizes prediction entropy to update BN affine parameters, while EATA adds sample filtering and regularization strategies. Despite their differences in how they update parameters, these methods share a common design choice in what they update, namely the BN layers.
In our experiments, we retain each method's original update mechanism (how to update) but replace the update target (what to update) from BN (@ BN) to our proposed Buffer layer ( (@ Buffer). This setup enables a controlled comparison to isolate the effect of changing the adaptation unit itself. Specifically, we implement TENT @ Buffer, EATA @ Buffer, SAR @ Buffer, and so forth, where only Buffer layers are updated during test time while the rest of the backbone including BN layers remain frozen. This design highlights the adaptability and generalizability of our Busfer layer across diverse TTA strategies. All experiments were conducted with three different random seeds, and the reported results represent either the mean or the mean with standard deviation.
4.2 Results
4.2.1
CIFAR 10-C& & CIFAR 100-C
Table Classification error rate (%) on CIFARIO-C and CIFAR Batch sizes (BS) of 2, 4, and 16 are evaluated across WRN28, ResNeXT (CIFAR-1OC) and WRN40, ResNeXT (CIFAR-100C).
Metladi
(CPAARIOO))
RexNeXT (CTFARID-C)
WIINAD (0SC)
RodNeX (CIFARCIO-C))
H5-2
B5-4
BS-
BS-2
BS-4
BS-16
BS-2
HS-4
BS-16
B5-2
BSad
BS-16
BN
32 L000
31, 200
301900
1.582 1A46
13.26-400
12.76
42.54.0M L4.
41.79.
40 00.39- 1920
42 12311400
28,40
37.
37.30
BBN
$3.56
23.12
66-50-1122
17.65+am0
94.59
50.74
Buffer
37.05 1811
25.11 608
30 02 4.3
20.52 1MM
19.35 19.35-400 -313
11.44-54
80.00
35 72 1..9
39.30 30 30000.0
91 80 LBA
48.17 40
8BN
45.49.. 664
34.18 1036
20.97 4.5
$7.12 A0TS
23.34 100
14.13 100
50.58 JUS
58.25 AU37
40 18 LOU3
20 BAA
67 18
36.58 4.3
10/000
(95.21)
3'13.99
(90.15)
(W2.045
(90.24)
(W19.205
191.99
PBN
Kinuffer
mm
41
14
17g
10)
(00.50
De 12
SiBufler
35.6-84
25.55 07
30 30:30-m
28.61
19.15 00.17-004
$0.55.m
58
36
45 363361
33.80 33.44-1
(W47.22
(W17544)
(W43.84)
CMF
CBN
41.2 4/0
21.51 mn
10.05 412
18.07
$3.45 13.45
21.31
18.85 -003
10.88
11.71
95
$6.09
35 10.10 -03
96.07 6.0
74.20-24
34 4:31
165.6
(90.60)
(W0.07
1W23.62
(42.45)
(44.55
(W0.92)
(90.20)
177
(W26.15)
(W0.4))
ROID 15
16 Mi 11
30 11
413
13.01 23%
1.47
30
4000
17
RuTTAA 23
BIBN
Vingfler
31 15
21 57
21 57 57
13:71
71
44 44.5
44 14
42 14
43. 14
42. 14
(90. 50g
(W0.20
(90.30
(W1.27)
(M0. 53)
(W2.011
(W2.3 3kg
(W1) 565
(90.64)
(4)
CuTTA
70.54
48.5 59
22
30.53
13.76
00.84-00
33.00am
43.1
91
62.97
38.33
MaCosttust]]
28.05 an
22.65
oul
18.35 4/6
35 usc
21.20 415
11.39 am
63.56 oux
$1.96 auo
39.72 ouu
67. 04.44 4:30
$3,01 cou
36.75 415
Table T reports the classification error rates on CIFARIO-C and CIFAR100-C under various small cest-time batch sizes (2, 4. and 16). Across all evaluated architectures, including WRN28, WRN40.

---

and ResNeXT, we observe that integrating the proposed Buffer layer (@ Buffer) consistently and substantially improves performance over the original BN-based counterparts.
The performance gains are particularly notable in low batch-size regimes such as BS=2 and BS=4, where traditional BN-hased adaptation methods often suffer due to unreliable batch statistics. For example, TENT @ Buffer reduces error by up to 15.51% on WRN28 under BS=2, and similar trends observed with Yo These results demonstrate that our Buffer layer not only generalizes well across diverse TTA algorithms but also offers robust alternative to BN in small batch size scenarios.
4.2.2 CIEAR-1O-W
Table 2 shows the results on CIFARIO W,
Table 2: CIFARIO-W. Bests are in bold. Green background indicates performance improvement.
challenging benchmark. Despite the difficulty, our Buffer layer consistently improves perfor- mance across all baselines. Notably, under small batch sizes (e.g., BS=2), our method achieves substantial gains: for instance, TENT from 89.30% to 64.14%, and Yo from 84.37% to 46.96% when only the Buffer layer is up- dated. This demonstrates the effectiveness of approach in overcoming the instability of normalization-bat methods under limited sta- tistical contexts.
CIFARIOW
Method
BS
16
Source
OBN
89. 40.30+
B4 72
59 66
TENT
Budeer
41:00:214
30
Buffor
83.74:2224
GBN
68 1-
$9 59
39 71 1:1
EATA 1
OF Billfrr
$8.89
30.02:281
29 91100111
Buffer
22645
OFNN
47.78 te0
35 2:
29. 28 -::0
CMF
11
OF Buller
26.00-240 10
30003 m-3-
@BN--ufferr
62.001147
However, as the batch size increases, the perfor- mance gap between @ Buffer and the baseline narrows, and some cases, even underperforms compared to the original method. Interestingly, in most cases, when both the Buffer layer and normalization parameters are updated jointly during adaptation (@ BN+Buffer) the performance consistently improved. We attribute this to the varying optimization difficulty inherent in each method; methods like CMF and SAR may benefit more from additional degrees of freedom during adaptation. This suggests that the Buffer layer can still contribute positively when combined with normalization updates, even in cases where frozen BN alone is insufficient. Moreover, as batch size increases, the influence of BN becomes more prominent and reliable, making joint updates with the Buffer layer more beneficial. These observations highlight the effectiveness of the co-updating strategy in leveraging the strengths of both components.
DeYo
14
WUN
Ruffor
84,57+
4B 26.95111
63 63:22 +557
34.81 129
28.30 20
39 00.00--..
RuNfeefffer
67.22.1
GINN
42. 26
36 d2
50 73 in
SAR KK
2R Rudlir @ @DNeeuffer
00:22
50 80-22.0
42.35 Seaus
20.35 S8800
30 00065-2000
41.81 Shee
36 24-24-20-
20-93 a2
RDID 16
C Buffer
41 11.00
36 30:2010
30.06 00:00.mmmm
41.77 SE0ES
36.16 36:16:2331
4.2.3 Would Buffer work with GroupNorm? ImageNet-C Results
Table 3: ImageNet-CC C. Bests are in bold. Green background indicates performance improvement.
Dataset
Nel-C
Models
Res50XBN)
Resv2_50IGN)
Method BS
16
2
16
Source
82.03
72.80
@BN
96.29-+0.11
78.90-+0.16
63.61+021
94.51+0007
85.63-+0.07
72.81+0021
TENT
26
@ Buffer
93.14.10.04
81.11
71.04.1001
94.2110006
85.43.0.10
72.24
BNN+Bfffer
96.15-+-0.13
78.66+00.13
63.48+00115
94.55+0.07
85,49+00.12
72.30+0.39
BB
93,300+0004
80.32++0.02
02.88
94.41+0008
85.64410.08
72.32+0.15
BATA
Buffer
93.22:1:0044
81.12.40.1
71.09:10:04
94.25:00008
91.43:10334
71.95
BBN+Bfffer
93.28++0.04
80.28:20.05
62. 16.:::00111
94.44:10.33
85,43-40.34
70.83+0011
BB
99.32-00.01
97.56++0.29
64.80:00004
96. 06.7244:225
83.23+0.111
69.30
CMF |I
@ Buffer
93.24
80.95-10.05
71.0310.144
95.28.10.1
89.89.10.27
69.52-1041
@ BN+Buffer
99.35540000
97.8940031
64.46
97.45 97.45:10.07
82.01 +0.40
65.51=0003
@ BN
93.2940.04
81.51+0.29
94.44
85.55=20.10
70,98=0.15
DeYo 12
@ Buffer
93.23
81.11 10.08
70.81 L022
94.24.10.07
85.57.1...1
73.84.10.8
@ BBN+uufee
93.29
90.81+00.88
68.88
94.25-+0.08
85.70++0.11
69. 69.16
BN
93.31
81.07 10
06
94.34110.05
85.67:20.12
25.91:0.25
SAR 18
Buffer
93.25 1..04
$1.03.1..00
71.09 71.09.10.05 1.0.5
94.25 0.006
85.45.1...
72.60 0.09
BN+Buffer
93.32-40004
81.05++0.31
65.53+40.31
94.46+0004
85.45--0.16
72.69+0025
BN
97,22++1.44
87.88+0063
61.51+0.18
94.68+0.38
85.0740.088
70.70++0007
ROID
Buffer
93.27:10.03
81.18:10.09
0.15
94.32:4:005
06.48-10.09
69.59-10.23
BNffeffer
97.12:+0.12
87.69:t0.12
61.35+0.17
94.51+0006
84.00:20.16
67.98++0.26
5

---

Similar to the observations on CIFAR- ImageNet-Crre results reveal that jointly updating the Buffer layer and norm layers becomes effective as the batch size grows (Table B). This trend reinforces the notion that BN statistics become more stable and reliable with larger batches, allowing co-adaptation with the Buffer layer to yield improved performance. As previously noted in [18], however, BN causes instability during test-time adaptation, especially under small batch regimes.
To address this limitation and further demonstrate the model-agnostic nature of our Buffer layer. we extended our evaluation to a ResNetV2 [ architecture that employs Group Normalization (GN) instead of BN. Experimental results on ImageNet-C with the GN-based ResNetV2 show trends consistent with those observed in the BN-based ResNet50, indicating that our Buffer layer remains effective regardless of the type of normalization used. These findings highlight the flexibility and general applicability of our method across diverse normalization schemes and network architectures.
4.2.4 On Large Batchsizes
Based on previous observations, we further evaluate the performance of our method under large-batel batch settings by jointly updating both the Buffer layer and BN parameters (Fig2). This setting aligns with the practical regime where BN benefits from stable batch-level statistics. Across datasets, including ImageNet-C, this co-adaptation strategy continues to yield improved performance, surpassing the baselines. These results confirm that the Buffer layer remains effective even when normalization-bant based adaptation is reliable, highlighting its complementary role in diverse test-time scenarios.
CIFARIO-C (BS=512)
CIFAR100-C (BS-256)
CIFARIO-W (BS=128)
ImageNet-C (BS=128)
Error (56
Error (96)
34
Error (%))
Error (%6
39
11
TENT
EATA SAR Do
CMP
ROID
TENT
EATA
SAR
Do
CMP
ROID
TENT
EATA SAR Ye CMF
ROID
TENT
EATA
SAR
DeYo
CMF
ROID
Baseline
@
Buffer
Figure 2: Classification error rates (%) across datasets on large batch sizes.
4.2.5
Continuously Changing Domains
Table 4: Classification error rate (%) on CIFARIO-C and CIFAR100-C under continuously changing environments, comparing models with and without Buffer. Accuracy is averaged over three different random seeds. Bold numbers indicate the highest accuracy.
Method
Ganes.
Shot
Implu.
Defac:
Glass
Motin
Zaom
Snow
Froet
Fog
Brigh.
Centr
Flest
Pisel
Ipsg
Avg
TENT BN
65.22
52.05
85.00
86.72
B3333
$4.32
B5.11
$8.39
90.71
PT.11
90.92
91.33
91.33
90.55
90.31
36.30
18 Dumeer
35.30
31.42
39.06
25/99
$4.43
25.05
24:16
26.86
26.26
23.84
19.94
22.45
33:47
27.57
32.51
29.17
EATA [7] IFN
2036
44.85
38.17
32333
64:39
59.73
49:30
40.82
32.93
34.47
47.37
49.35
61.20
39.97
02.54
31.92
6 Rugier
34.50
30.20
37.54
24.97
42066
25.29
24:11
27.04
26.05
24.33
19:52
21.38
33.49
27.38
32.41
28:72
CIPARIO4:
DBN
45.47
28.44
28.79
25.75
19:44
22.45
31.44
31.75
31.30
Butter
39.00
37.06
05.57
24:31
06.82
24.90
24.64
28.79
29:06
25.86
19:56
22.89
35.24
31.99
38.25
31.61
Yo or BN
66:30
15.38
71.28
71300
12.55
71.85
74.82
70.30
76.71
09.30
CKF
26.05
15.22
19.29
23.59
27.70
Butler
38:22
01:45
25.61
26.16
18:01
19.32
26.81
31.44
27.74
RODD [Td
27.26
15.19
207440
20.50
10.95
18 Rutter
36.12
34.35
42.28
23:45
45.23
24:25
23.81
26.95
28:15
24:51
18.35
21.39
34.41
29.55 28.88
36.03
29.96
TENT te BB
95.72
90.02
95.54
59.17
95.77
90.00
95.02
58.45
98.27
58.70
59.25
58.47
95.39
98.41
98.78
98.35
48 Rutior
90.59
97.45
97.94
08.21
95.44
56.39
98.25
98:27
98.25
08.
90
09.44
99..50
97:94
97.96
08.02
97.00
EATT [2] PPN
65.15
51.22
85.41
85.65
65.39
$3.09
83:35
$9.24
39.96
90:12
90.04
92.89
90.85
90:45
91.09
$7.35
86 Bulle
60.73
53.85
56.33
45:00
601.22
47.54
46.99
52.27
$2.53
55.84
45.66
49.73
57.24
$1.57
59.23
53.05
CIPARIONE
SAN HI GBN
63.13
07.45
06.71
01.72
09357
09:30
03000
64.89
01.05
09.33
35.34
01.15
67.33
67.09
00.41
65.77
di Buffer
58.67
57.75
59.92
44/46
59:47
45.96
45:55
52.25
51:76
58:34
44.04
46.72
53.43
48:46
55.28
52.14
DeY3 DN
91.32
97.71
98:03
97.93
97.33
98.05
95.03
98.06
98.37
98.52
98.58
98.42
98.35
98.71
98.82
97.82
Buller
98.29
99699
99:09
CMI GEN
96.33
97.84
97.99
95.76
99.03
V6.07
45.95
55.5
49.55
[6]
16.35
Butle
54.13
49.63
30.05
42.22
46.15
53.68
46.51
55.09
36.01
We evaluate our method under continuously evolving domains reflect real-world deployment. This dynamic scenario challenges models remain both responsive and stable as input distributions

---

gradually shift over time. Across most adaptation baselines, replacing BN updates with our Buffer layer yields consistently superior performance throughout the domain shift trajectory. As shown in T our method exhibits strong temporal stability and sustained accuracy, even as the input distribution drifts further from the source domain. This demonstrates that the Buffer layer remains effective in changing and evolving environments where traditional normalization- based approaches often struggle to keep pace.
We further note that this robustness in streaming scenarios is closely related to the Buffer layer's resistance to catastrophic forgetting. By preserving the original model's parameters and decoupling adaptation into an external module. the network retains critical source-domain knowledge, which in turn strengthens its adaptability over time. Our approach is thus well-suited for long- deployment, as it preserves source knowledge and maintains adaptability over time.
4.2.6 Catastrophic Forgetting: Buffer Never Forget!
We further investigate the potential issue of catastrophic trrgetting-weeeee adaptation to the target domain degrades performance on the original source distribution- particlllrrr in the context of CIFAR- 10-W,, which simulates a real- real-world-liks continuous distribution shift (Fig.3). This benchmark provides a long sequence of input data with gradually evolving characteristics, making well-suited for evaluating the stability of test-time adaptation methods. Such settings are especially relevant in practical scenarios, where models are deployed in dynamic environments and must adapt without sacrificing previously acquired knowledge. To assess this, evaluate TENT and its Buffer--augentte variant on both the target data and the original source data.
DF
KW
KWC
0.8
0.8
BS=4
0.6
0.6
0.4
0.5
0.4
0.2
0.2
10
10a
10
10'
10a
10aa
10
10a
10
# data stream
data stream
of data stream
BS=16
0.6
0.8
0.4
0.4
0.5
0.2
0.2
10'
10
10
10
10
10
10'
10
10
# data stream
data stream
of data stream
lossoe errer (TENT (BN)
lagasi crme rue (TENT BN)
annnc enor náe (TENT Bafferr
larget coor rai: (TENT (
Figure 3: Catastrophic forgetting experiments on WRN28, CIFAR-IO-W. Blue: TENT @ @Buffer, Red: TENT @BN
Interestingly, we find that TENT @ @Buffer not only improves performance on the target domain but also preserves accuracy on the source domain significantly better than the original TENT. This is attributed to the fact that our method does not update the BN layers, which are essential for main- taining source- domain statistics. By isolating adaptation into the external Buffer layer while keeping the core network intact, our approach enables effective test-time adaptation without overwriting learned representations. Notably, even under prolonged and continuous distribution shift, our method maintains stable performance, whereas TENT suffer from performance degradation as adaptation progresses. These findings collectively demonstrate that our design is not only theoretically sound but also empirically robust in dynamic test-time scenarios.
The importance of avoiding catastrophic forgetting goes beyond preserving source-domain accuracy. In practice, we observe that once a model begins to forget the source domain, its adaptation capacity to the target domain also deteriorates over time. This suggests that forgetting disrupts the foundational representations learned during pretraining, which are crucial for effective generalization. Conse- quently, methods that fail to preserve source knowledge may experience compounding errors in the target domain, turning forgetting into critical bottleneck for reliable deployment. This underscores the necessity of stable adaptation mechanisms like our Buffer layer that safeguard existing knowledge while enabling localized adjustments.
7

---

4.3 Ablation Studies
4.3.1 Design Exploration of the Buffer layer
Our previous experiments demonstrate that attaching Buffer layer to a pre-trained model, without modifying the backbone itself, enables effective test-time adaptation. While this decoupled adaptation strategy has proven successful, it remains unclear which architectural configurations of the Buffer layer best support this behavior. As illustrated in Fig. the design space of the Buffer layer includes multiple options, ranging from Ix and 3x3 convolutions to the inclusion of BN, as well as variations in their placement within the network. In this work, we systematically explore several architectural variants (e.g., combinations of convolutions with learnable scaling factors a and B, optional BatchNorm) uncover effective configurations. All ablation experiments in this section conducted using TENT @ Buffer, where only the Buffer layer is updated during test bist-time adaptation while all networks including BN layers remain frozen. As shown in Table and Table we further examine how different structural choices and insertion locations of the Buffer layer impact adaptation performance. Our goal is not to propose single optimal design but rather to investigate what aspects of localized Buffer module contribute most to adaptation performance. This exploration is motivated by the intuition that different layers of pre-trained model may require distinct forms of correction, and thus, one-size- fils-alllllllll Buffer structure may not be sufficient.
Empirically, we find that a dual-path configuration combining 1x1 and 3 convolutions (Module @), applied after the activation layer (iii,, achieves a strong balance, yielding robust performance across diverse settings. In all (iii)-type placements, Module ( consistently outperforms other designs. However, for (ii) ti)-ttyp placements, the optimal configuration appears to be batch-size dependent: when the batch size is small, single 1 1 convolution performs best, whereas for larger batch sizes, the combined xx and 3x3 structure yields superior results. This suggests that the best module design depends both its complexity and the batch size used during adaptation.
Building on the architectural choice identified as most effective in Table 5 (Module (., we further investigate where in the network this Buffer layer should be placed. As shown in Table we analyze the effect of inserting the Buffer layer at different stages of the backbone-earlly middle, or late (a,b,c) under varying batch size regimes. Interestingly, Buffer layer is strongly influenced by the batch size. When the batch size is small, attaching the Buffer layer only the early stages yields the best results, suggesting that correcting low-level features is critical under unstable hatch statistics. In contrast, with larger batch sizes, combining Buffer layers at both early and middle stages consistently leads to better performance.
observe that the optimal placement of the
Table 5: Block-level Error Rates.
Block-und unit Resalts
CIFARIO
CIFARIOO
BS4
BS128
BS128
2
1
4
2
4
4
( (Conv2D)
29.73
29.57
31.63
19.53
19.44
47.92
51.68
85
35.20
33.23
34.81
33.18
(ii) (BN2D)
29.25
29.26
30.08
29.63
20.22
18.84
19.84
18.73
47.97
48.14
49.19
48.03
35.01
32.62
34.61
32.48
(ii) (RcLU)
29.36
29.63
29.95
29.35
20.28
19.12
19.49
19.03
48.20
48.12
49.47
47.99
34,45
32.40
34.21
32.31
Table 6: Module-level Error Rates.
Giil
Configuration
CIFARI0
CIFAR 100
ReLU
balire Medul
(a) (b)
(c)
BS4
BS128
BS4
BS128
BotchD
29.35
19.03
47.99
32.31
33.86
19.04
80.17
32.38
Conv2D
64.64
20.98
98.43
75.20
Butler Module
33.43
18.33
80,80
31.72
Baffer
63.41
19.97
98.45
71.16
67.47
20.27
98.46
69.80
BN
66.94
19.55
98.43
67.83
Figure 4: Module design of Buffer layer.
8

---

Across all configurations, we find that inserting the Buffer layer at the final stage of the model tends degrade performance. This supports the hypothesis that domain shift in most cases are primarily driven by low-level distributional changes rather than high-level semantic shifts. Additionally, the batch size appears to influence not just statistical reliability but also the effective capacity of the adaptation module. Specifically, larger batches enable more stable gradient estimates, which may allow deeper or more distributed Buffer structures to be optimized more effectively. This insight underscores the importance of jointly considering both placement and batch dynamics when designing robust lest-time adaptation modules.
4.3.2 Effect of a
We analyze the role of the scaling coefficient used in the Buffer layer, which controls the strength of test-time adaptation (Table 7). Although a set as a learnable parameter, we observe that its behavior highly sensitive initialization. In particular, smaller initial values of tend to perform better in low batch-size regimes (e.g.. BS=2), preventing overfitting to noisy or unstable gradients. Conversely, larger initializations
benefit large batch sizes by enabling more aggressive adaptation. This correlation suggests that the optimal setting of o is not universal but instead depends on the available test-time batch size. We observe this trend consistently across both settings, with and without frozen BN layers, indicating that a plays a critical role regardless of whether normalization statistics are updated. These findings highlight the importance of designing principled initialization strategy or dynamically adjusting a based on batch statistics for stable and effective adaptation.
Table 7: a sweep results on CIFAR100-C
TENT
@ Buffer BS16
TENT @ BN + Buffer
a
BS2
BS4
BS8
BS64
BS256
BS2
BS4
BS8
BS16
BS64
BS256
1c-5
97.93
65.99
39.60
34.83
31.70
32.88
lc-5
98.46
94.51
78.73
50.46
32.81
31.24
le-4
97.88
65.73
39.76
34,91
31.73
32.97
le-4
98.46
94.56
78.96
50.70
32.76
31.24
le-3
97.89
64.98
39.76
34.85
31.58
32.70
le-3
98.43
94.71
78.97
51.22
32.74
31.18
le-2
97.96
66.84
39,59
34.76
31.47
32.18
le-2
98.54
95.07
80.82
51.81
32.84
30,90
le-1
98.30
68.87
39.89
35.05
31.69
30.96
le-1
98.77
96.95
89.08
69.32
36.42
31.11
5
Conclusion and Limitations
In this work, we proposed a lightweight and modular Buffer layer for source-free, fully online test- time adaptation. By decoupling the adaptation process from the backbone and targeting only the external buffer modules, our approach enables stable and architecture-agnosti adaptation even under small batch sizes or severe domain shifts. Extensive experiments across diverse datasets and TTA baselines validate the effectiveness and generalizability of the proposed method.
Despite these promising results, several limitations remain. First, the optimal architecture of the Buffer Layer has yet to be identified. Interestingly, placing the buffer after activation functions (c.g.. ReLU) consistently yields better performance, but the underlying reason for this remains unclear and deserves further exploration.
Second, while treat the scaling factor learnable parameter, its optimization is sensitive to initialization. A principled strategy for initializing or dynamically tuning a is still lacking, and future work could explore meta-learned or data-aware initialization schemes.
Third, the current adaptation objective is directly borrowed from existing TTA methods, without explicitly considering the characteristics of the Buffer layer. Designing optimization target tailored to the Buffer's 's role, potentially incorporating constraints or priors reflecting its residual nature, could further enhance adaptation stability and performance.
Despite certain limitations, the proposed Buffer layer demonstrates notable effectiveness. It achieves strong performance without the nource-dependent warm-up phase required by prior auxiliary ap- proaches, and remains robust under small batch sizes, thereby mitigating batch dependency issues. Moreover, it effectively suppresses catastrophic forgetting, highlighting that domain adaptation can be achieved without reliance on normalization. These findings suggest a promising new paradigm for source-free adaptation.

---

6 Acknowledgement
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) 200S-2222-160703822 DS-2025-02217919 RS-2025-022150077 University (R$2020-1120136) the KIST Institutional Program (2E33801, 2E33800), and Yonsei Signature Research Cluster Program of 2024 (2024-22-0161).
Artificial Intelligence Graduate School Program at Yonsei
References
[1] Dian Chen, Dequan Wang, Trevor Damell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 295-305, 2022.
[2] Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai image database. In 2009 IEFE conference on computer vision and pattern recognition, pages 248-255. Tece, 2009.
Fei-Fei. Imagenet: large-scale hierarchical
[3] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, training of neural networks. Journal of machine
Mario March, and Victor Lempitsky. learning research, 17(59), 2016.
[4] Taesik Gong, Yewon Kim, Taeckyung Lee, Sorn Chottananurak, and Sung-Ju Lee. Sotta: Robust lest-time adaptation on noisy data streams. Advances in Neural Information Processing Systems, 2023.
[5] Dan Hendrycks Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.1 122266 2019.
[6] Junyuan Hong, Lingjuan Lyu, Jiayu Zhou, and Michael Spranger. Mecta: Memory-cconomic continual test-time model adaptation. 2023 International Conference on Learning Reprexentations, 2023.
[7] Minguk Jang, Sac-Young Young Chung, and Hye Won Chung. Test-time adaptation via self-training with nearest neighhor information. arXiv preprint arXiv:200.11079,, 2022.
[8] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: A benchmark of in- the-wild distribution shifts, In International conference on machine learning, pages 5637-5664. PMLR, 2021.
[9] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 28, 2020, Proceedings, Part V 16, pages 491-507. Springer, 2020.
[10] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[11] Jae-Hong Lee and Joon- Joon-Hyuk Chang. Continual momentum filtering parameter space for online test-time adaptation. In The Twelfih International Conference Learning Representations: 2024.
[12] Jonghyun Lee, Dahuin Jung, Sachyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang, and Sungroh Yoon. Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. arXiv preprint arXiv:2403.0 07366, 2024.
[13] Scunghun Lec, Sunghyun Cho, and Sunghoon Im. Dranet: Disentangling representation and adaptation networks for unsupervised cross-domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15252-15261, 2021.
[14] Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Pattern Recognition, 80:109-111 117 2018.
[15] Jiaming Liu. Sengiao Yang, Peidong Jia, Renrui Zhang, Ming Lu, Yandong Guo, Wei Xue, and Shanghang Zhang. Vida: Homcostatic visual domain adapter for continual test time adaptation. arXiv preprint arXiv:2306..04: 2023.
[16] Robert A Marsden, Mario Döbler, and Bin Yang. Universal test- time adaptation through weight ensem- bling, diversity weighting, and prior correction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2555 2565-2555,,40 2024.
10

---

[17] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International conference machine learning. pages 16888-16905. PMLR, 2022.
[18] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation dynamic wild world. arXiv preprint EXXx:2332.12200 2023.
[19] Skyler Seto, Barry-John Theobald, Federico Danieli, Navdeep Jaitly, and Dan Busbridge. Realm: Robust entropy adaptive loss minimization for improved single-samps sample test-time adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2062-2071, 2024.
[20] Shin and Hyun Kim. L-tta: Lightweight test-time adaptation using a versatile stem layer. Advances in Neural Information Processing Systems, 37:39325-39344 2024.
[21] Junha Song. Jungsoo Lee, In Kweon, and Sungha Choi. Ecotta: Memory-efficieee continual test time adaptation via self delf-distilled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11100--11999 11929, 2023.
[22] Xiaoxiao Sun, Xingjian Leng, Zijian Wang, Yang Yang, Zi Huang, and Liang Zheng. Cifar-1--warehouse: Broad and more realistic testbeds in model generalization analysis. arXiv preprint wrXiv:2310.04414 2023.
[23] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
[24] Qin Wang, Olga Fink. Luc Van Gool, and Dengxin Dai. Continual dest-time domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7201-7211 2022.
[25] Zixin Wang, Yadan Luo, Liang Zheng, Zhuoxiao Chen, Sen Wang, and Zi Huang. In search of lost online test-time adaptation: survey. International Journal of Computer Vision, pages 1-34, 2024.
[26] Garrett Wilson and Diane J Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(5):1-46, 2020.
[27] Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15922- 15932, 2023.
[28] Marvin Zhang, Sergey Levine, Chelsea Finn. Memo: Test time robustness via adaptation and augmen- tation. Advances in neural information processing systems, 35:38629-38666 2022.
[29] Bowen Zhao, Chen Chen, and Shu- Tao Xia. Delta: degradation-free fully test-time adaptation. arXiv preprint 2rX.v:3301. 2023.
[30] Kaiyang Zhou, Ziwei Liu, Qiao, Tao Xiang, Chen Change Loy. Domain generalization: A survey. IEEE transactions on pattern analysis and machine intelligence, 45(4):336 4415, 2022.

---

A.1 Implementation Details
In this work, we compare our proposed approach against several state-of-the-art test-time adaptation (TTA) baselines. For fair comparison, we carefully reproduce each method based on official imple- mentations and papers, and unify the training environment as much as possible. Below, we detail the optimization and hyperparameter settings used for each method.
All experiments including TENT, DeYo, SAR, CMF, and ROID are optimized using the Adam optimizer with a learning rate of le-3, 3 0.9, and zero weight decay.
EATA shares the same optimizer configuration TENT (Adam, LR=1e-3, 3 0.9, WD=0.). Additionally, we set the Fisher regularization strength to 1.0 and the confidence margin to 0.4, following the original implementation. For the source distribution sampling, we 2000 samples to compute the Fisher information matrix.
These unified settings ensure that performance differences primarily arise from the intrinsic mecha- nisms of each method, rather than discrepancies in optimization or tuning.
All experiments on CIFARIO-C. CIFAR100-C, and ImageNet-CC [S] are conducted under Severity 5 settings, following standard protocol to evaluate robustness under the highest level of corruption.
CIFAR-1O-W [22] is a web-collected dataset constructed to evaluate model robustness under realistic distribution shifts. is composed of three distinct subsets-DF (Diffusion), KW (Keyword), and KWC (Keyword with Cartoon)-each reflecting different data generation strategies and semantic characteristics. To comprehensively evaluate the generalization ability of each adaptation method, conduct separate experiments on all three subsets.
All experiments performed using NVIDIA RTX A6000 GPUs.
For the experiment in Section 4.2.6 (Continuously Changing Domains), we use a batch size of 16 during adaptation, which balances stability and responsiveness to gradual domain shifts.
For the analysis in Section 4.3.2 (Effect of a), we intentionally deviate from the configuration presented in Section 4.3. Specifically, we insert the Buffer layer only after very first single activation layer, rather than applying throughout the network. This simplified setting isolates the effect of the scaling parameter o, allowing for more controlled analysis its influence on adaptation performance.
All Buffer layers are composed of randomly initialized convolutional modules, without any pretraining. They optimized solely during test time, reinforcing the simplicity and modularity of the proposed approach.
Following the common practice in test-time adaptation, we configure BatchNorm(BN) layers to use target-domain batch statisties for normalization, while keeping the affine parameters frozen. This allows the model to respond to distributional shifts in the input without altering any trainable components of the normalization layers.
A.2 Pseudo-code of Implementing Buffer Adaptation
Algorithm Test-Time Adaptation with Buffer layer
1: Input: test sample x, pretrained model Fo. buffer module Bg, TTA algorithm Age
2: Output: adapted prediction y
3: Initialize adaptation method Ag
Attach Buffer module B& in parallel to Fg
5: Freeze all parameters in Fa
Enable gradients for
7: < Fo(x) + (x)
C (5)
9: Update Φ using V L
10: return y
12

---

The adaptation procedure with the proposed Buffer layer is intentionally designed to be simple and modular, as illustrated in Algorithm[[ Given any existing TTA method, the only modification required is to attach an external Buffer layer in parallel to the pretrained model and enable gradient updates for the buffer's parameters. The backbone network remains entirely frozen, ensuring that the source- domain representations are preserved, while the Buffer layer acts as a residual path that provides localized feature-level corrections during adaptation. This design not only minimizes implementation overhead but also ensures broad compatibility across different architectures and optimization schemes. As result, our method can be easily incorporated into existing TTA pipelines with minimal code changes, without disrupting the original model structure or its pretrained functionality.
A.3
Mixed Domains
Table 8: CIFAR100-C under mixed domain shifts.
Considering realistic deployment scenarios, where the test distri- bution may shift continuously or vary across time, it is crucial to evaluate TTA methods under dynamic, mixed-doman conditions. As highlighted in 127] robust adaptation in such non-stationary
Dataset
ImageNet-C
Models
BS=2
environments requires models to generalize across a sequence of
@ @BN
98.92
TENT
@ Buffer
98.63
heterogeneous target domains without explicit domain boundaries.
@ BN
76.53
Inspired by this, we argue that mixed-domain evaluation provides more rigorous and practical testbed for TTA methods, as better reflects the challenges of real-world streaming inference. Accord- ingly, we include experiments under mixed corruption settings to assess the robustness and stability of our method across diverse and evolving domain shifts.
EATA
@ Buffer
76.58
@ BN
98.95
CMF
@ Buffer
83.41
@ BN
96.28
DeYo
Buffer
76.65
@BN 76.42
SAR 18
Buffer
76.42
As shown in Tab[8] our method demonstrates strong performance under mixed-domain test scenarios. Across diverse corruption types and varying domain orders, the proposed Buffer layer consistently maintains low darget-domain error while preserving source-domain accuracy, even in the absence of domain boundaries or resets. These results highlight the adaptability and stability of our approach in dynamic TTA settings, reinforcing its practical utility for real-world applications where domain shifts occur unpredictably and continuously.
@BN 95.76
ROID
@ Huffer
76.90
A.4 Further Experiment on Catastrophic Forgetting
As discussed in the main text, apply- ing existing TTA approaches such as Tent to BN layers can lead to source domain forgetting, a phenomenon that becomes significantly more pro- nounced under relatively small batch sizes, often resulting in increased tar- get domain eиror. In contrast, the pro- posed Buffer layer exhibits strong resistance to such forgetting, main- taining source performance while en- abling stable adaptation to the target domain. Tab. p presents results on the CIFAR100-C dataset, further confirm-
Table 9: Source and target errors on CIFAR100-C dataset, ResNeXT, batch size 16, Gaussian Noise.
Moving μt,
Eiror Rate
0.8
0.8
800
3200
5600
8000
9600
of data
Error Rate
Fixed ja.
0.8
0..4
0.2
800
3200
5500
8000
9600
of data stream
ing that this robustness generalizes be- yond the settings reported in the main experiments.
Source errox (TENT BN)
Target enirr (TENT (BBN)
Souree ernor (TENT Ruffer)
Turget erice (TENT Ruffer)
While catastrophic forgetting has been recognized as a critical challenge in TTA, prior studies have paid limited attention to establishing a standardized and fair evaluation protocol for measuring it. In particular, the question of how best evaluate a model's retention source-domain performance after adaptation remains largely unaddressed in the existing literature [17. This leaves open an important methodological consideration when assessing forgetting, should the adapted model be evaluated on doree- data using updated source statistics (moving μ, a)? Or should the model
13

---

instead be evaluated using the target-domain statistics fixed during adaptation, thereby preserving the post-adaptation state (fixed p. ot)
BS4
BS16
Moving μ. a Error Rate
0.8
0.6
0.5
0.4
0.2
100,00 0000
200 200,000 ,000
300, 300,000 0000
100,000
0000
200,000 ,00
300, 300,000 ,000
# of data stream
# of data stream
Fixed μ., d
Error Rate
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
100, 000
200, 200,000 .000
300,000
100. 000 200, 0,000
300, 300,000 $,00000
# of data stream
of data stream
Smanaz-cmer emce (TFNT (-NN)
Targe mner (TENT # RN)
Scunat sroe (TENT Bullie)
Tagee crmee (TRNT te Bulller
Figure 5: Catastrophic forgetting experiments on WRN28, CIFARIO-W W (KW). Blue: TENT @BN, Red: TENT @ Bufffer.
To clarify this methodological ambiguity, we conduct experiments under both evaluation protocols. By comparing both settings, we are able to more precisely assess the impact of the Buffer layer on preserving source-domain performance, particularly under small batch size conditions. While both protocols yield complementary insights, we consider the use of source-domain statistics (moving 14, a)) to be more practical and informative, as it reflects realistic deployment scenarios in which source- aourne- calibration may still be available or preferable. Importantly, these moving statistics collected during the standard forward pass over unlabeled source-domain data, and do not require any additional supervision or training overhead. From this perspective, using moving μ and o is not only cost-free but also fully consistent with test-time constraints. Accordingly, we adopt this protocol for the results presented in the main paper. As shown in Tab. 9 and Fig. 5 our method demonstrates consistently strong anti-forgetting performance across CIFARTO0-C and CIFAR1O-W W, supporting the efficacy of the Buffer layer in mitigating forgetting even under challenging conditions. This effect is also clearly reflected in the source-domain results shown in Fig. 6
This finding highlights a clear deviation from the commonly observed trade-off in TTA, wherein performance improvements on the target domain are typically accompanied by degradation on the source. The Buffer layer, however, achieves simultancous gains in both domains, indicating a more favorable balance between adaptation and retention.
central factor behind this effect the non-intrusive and parallel architecture of the Buffer layer. Unlike conventional methods that adapt BN layers by updating their affine parameters, our approach avoids modifying any pretrained, learnable parameters the backbone. In many existing methods, such updates overwrite source-domain representations, and once altered, the original alignment the source distribution becomes difficult to recover. In contrast, the Buffer layer operates as structurally independent residual branch that leverages only the input's batch statistics-mest and variance- for adaptation. Notably, this auxiliary layer can be interpreted as implicitly fulfilling the role of affine transformation in parallel and externalized manner, enabling domain-specifee modulation without interfering with the main pathway. As result, it preserves the integrity of source-domain features while allowing effective target-domain adaptation, thereby offering enhanced robustness against catastrophic forgetting.
By maintaining a strictly parallel configuration, the Buffer layer allows source-domain inputs to be processed exclusively through the unmodified backbone, entirely bypassing adaptation-specins paths. This architectural decoupling provides compelling explanation for the observed reductions in both source and target domain errors achieved without violating the trade-off constraints that typically characterize TTA.
14

---

A.5 Feature-level Analysis
To further validate the effectiveness of the proposed Buffer layer, we conducted an analysis at the feature level by examining how different adaptation strategies affect internal representations. Following the methodology of [23], we visualized the distributional statistics caaistics-ssecificlllly the channel-wise mean and variance- of intermediate features to observe the model's response to domain shift. For this purpose, we randomly selected a subset of feature maps from the output of stage2 (corresponding to the 18th layer in ResNexT), which captures mid-level semantics critical to downstream predictions.
@ Source Domain
Source (Eroo: 00 04%)
(b) 6BBN (Firoe: 99.09%) (c) (oButTer (Emor 11 17%)
@ Target Domain
(4) Source (Frvor: : 72. 33%)
(b) Tent GBBN (Frroe: 60.29%)
(c) Tent ogButter (Emor (40,17%)
(4) Oraole (Eror: 38.46%)
Figure 6: Feature distribution comparison across adaptation strategies CIFAR100-C (batch size 16). Despite being applied only at the early stage of the network, Buffer-based adaptation (orange) yields feature statisties (mean and variance) that more closely aligned with the Oracle model (green), compared to TENT applied at BN layers (blue). This suggests that the Buffer layer effectively propagates adaptation signals throughout the network, enabling target-aware representations even in deep feature spaces.
We compared four adaptation settings: (a) Source (no adaptation). (b) TENT applied to BN layers (TENT@ TEENTBBN, (c) TENT applied to the Buffer layer (TENT (TENT@Buffer). and (d) Oracle, which is trained with full access to target-domain labels using cross-entropy loss, following the setup in [23] All experiments were conducted on the CIFAR 100-C dataset with batch size of 16. The visualizations indicate that the feature distributions obtained from TENT@ Buffer are more closely aligned with those of the Oracle model than those from TENT@BX.
This result suggests that the Buffer layer enables a more effective form of test-time adaptation by producing internal representations that better reflect the target-domain characteristics. The alignment with Oracle-level feature statistics provides further support for the Buffer layer's s capacity to generalize under domain shift without requiring access to target supervision.
A.6
Why Buffer layer works well?
Unlike conventional TTA methods that rely on modifying internal components of the pretrained model, our approach introduces adaptation ssternally-throuee a structurally independent Buffer layer. This design choice is not merely architectural; it fundamentally alters how adaptation interacts with the existing representation space. By avoiding direct updates to the backbone, the Buffer layer prevents destructive interference with source-domain features, common cause of catastrophic forgetting in BN-based adaptation. Instead, the pretrained backbone remains intact, serving as a stable foundation throughout the adaptation process.
More importantly, this separation enables a distinct mode of representation learning. Instead of altering or overwriting existing features, the Buffer layer introduces complementary target-specific activations that coexist with the pretrained representations. This leads to an effective expansion of the class-conditional feature space, as the model learns to associate semantic concepts with a broader
15

---

range of domain-specitte variations. Such behavior is conceptually aligned with findings in multi- domain and domain generalization literature, where diverse exposure distributional
altering label semantics- hasttticcchhhhh--saaaa been shown to improve generalization [13] In this light, the Buffer layer can be interpreted as enabling a form of incorporate new domain signals without sacrificing previously acquired knowledge.
it allows the model to
In contrast recent additive-layer approaches [2] 20 that inject adaptation modules directly into intermediate layers of the backbone, often modifying the main information flow, the Buffer layer remains fully external and modular. While such additive methods may retain partial structural sep- aration, they still introduce parameter updates or architectural interference that can compromise source-domain representations. The Buffer layer, by comparison, performs domain-specific modula- tion purely through residual pathways and without altering any pretrained parameters, effectively emulating the role of affine adaptation
parallel, non-destructive manner.
Overall, the Buffer layer enables a form of adaptation that sidesteps the destructive interference commonly observed in BN-based or entangled additive-layer methods. By maintaining clean separation between the adaptation mechanism and the pretrained model, it preserves the model's original capabilities while selectively enhancing its responsiveness target-domain signals. This balance between stability and adaptability offers a scalable and reliable foundation for test-time deployment in dynamic or continuously shifting environments.
16

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency. research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: All main claims are including paper's contributions and scope in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
17

---

Justification:
See Sec.5, Concusions and Limitations
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specificatinn, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texl system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't itn't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: In 2. 2.Background and 3. Method sections, we provide assumptions, and in 4. Experiment we provide proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the methods are fully reproducible enough, regardless of whether the code and data are provided or not. The code will be made public too.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will provide code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wttsssubmsiiiooooioooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //ipps.cc/publi///uul
Codeeubmiiiie wwwpebbmmssiooPoo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
19

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e-g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Supplementary.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: seeds.
We include standard deviation, and ran three experiments on different random
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: Yes]]
Justification: Yes, in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
20

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp///eeurpps...cpppbbb.iiiit
Answer: [Yes]
Justification: Yes
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss it at the front and the end of the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper poses such risks.
21

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e-g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released. the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: N/A
22

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (ttpe:///eerrrpsccccooofeereee// for what should or should not be described.
23


Reviews:
[{"review_id": "JuRSEHebP4", "reviewer": "Reviewer_SMLc", "summary": "The paper proposes a light-weight buffer layer to address the catastrophic forgetting problem of TTA. With extensive experiments, the paper shows that this simple plug-and-play modification of the model can achieve surprisingly better performance when inserted into any other TTA methods.", "questions": "See the above Weakness.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is easy to read and follow. The motivation is clear.\n2. The proposed module is simple and can be easily integrated into any other TTA methods, which makes it have wide applications.\n3. All the experiments, including conventional CIFAR-10 & CIFAR-100C datasets, and several datasets with domain shifts, validate the effectiveness of the proposed method.\n\nWeakness:\n1. The proposed method is similar to the weight expansion of continual learning, which should be discussed in the related work.\n2. How does the number of paralleled buffer layers (e.g., multiple 1x1 conv) influence the final performance?\n3. In Table 1., some results of employing buffer layer drops for ROID,  which should be explained why it happens and when the proposed method works or not.\n4. How does the proposed method compare to LoRA, which can be employed at BN or Conv layers?", "was_revised": true, "final_justification": "I keep my rate since all my concerns have been addressed", "rebuttal": "Thank you for your insightful comments. We appreciate the opportunity to improve our manuscript and have addressed your points below.\n\n---\n\nW1 :  **Weight expansion of continual learning**\n\nR1: Although our Buffer Layer is applied only at test time—whereas classic weight‑expansion methods add modules during training—both paradigms freeze the backbone parameters and introduce compact auxiliary layers to mitigate catastrophic forgetting. Recognizing this conceptual similarity, we will expand the Related Work section with a comprehensive comparison to these training‑time expansions (e.g., Progressive Neural Networks, adapter modules) and recent low‑rank adaptation techniques such as LoRA.\n\n\n---\n\nW2 : **The number of paralleled buffer layers** (e.g., multiple 1x1 conv)\n\nR2 : Thank you for this excellent question. In the main manuscript (Fig. 4 & Table 5) we focused on the type of Buffer layer, but examining how the number of parallel layers affects performance is indeed very insightful. As shown below, we swept from 1 up to 5 parallel 1×1 buffers (and one mixed 1×1 + 3×3 + 5×5) under batch sizes 4 and 16 on CIFAR10-C with WRN28 architecture, as with our main manuscript :\n\n-----   \n\n| Configuration    | BS = 4 Error (%) | BS = 16 Error (%) |\n|------------------|------------------|-------------------|\n| single 1×1conv         | 29.16            | 21.22             |\n| double 1×1conv          | 29.02            | 20.87             |\n| 1×1conv + 3×3conv        | 29.15            | **20.37**             |\n| 3* 1×1conv           | 29.07            | 20.71             |\n| 1×1conv + 3×3conv + 5×5conv  | 29.83            | 20.41             |\n| 4* 1×1conv           | **28.96**            | 20.58             |\n| 5* 1×1conv          | **28.96**            | 20.56             |\n\nAs shown in the table above, an intriguing phenomenon emerges as we increase the number of parallel 1×1 conv buffers: under very small batch sizes (BS = 4), error steadily decreases—from 29.16% with a single 1×1 layer down to 28.96% with four or more—demonstrating the clear benefit of extra 1×1 paths in small‑batch regimes. However, these pure 1×1 expansions remain suboptimal at BS = 16, never outperforming our original 1×1 + 3×3 configuration. Conversely, the 1×1 + 3×3 + 5×5 mixed buffer matches BS = 16 performance but falters at BS = 4, highlighting a distinct trade‑off between kernel complexity and batch size. Overall, across both small and moderate batches, our 1×1 + 3×3 design proves the most robust and reliable choice. We appreciate this insightful question—further exploration of buffer count and kernel interactions is indeed a promising direction for future work.\n\n---\n\nW3 : **Performance drop for ROID**\n\nR3 : We thank the reviewer for this insightful point. We hypothesize that ROID’s weight ensembling—implemented via an exponential moving average over the buffer’s randomly initialized weights—biases the buffer back toward its noisy starting state, impeding effective adaptation. To test this, we removed the weight-ensembling component and applied our Buffer Layer. The CIFAR-10-C error rates are reported below:\n\n| ROID | BS 2 | BS 4 | BS 16 |\n|------------------|------------------|-------------------|-------------------|\n| @BN        | 93.22            | 51.95             | 34.01 |\n| @Buffer         | 72.92           | 50.67             | 35.15|\n| @Buffer, no EMA        | **71.16**            | **47.73**        | **33.86**|\n\nRemoving weight ensembling consistently reduces error across all batch sizes, confirming that the ensembling bias toward random initial weights was the primary bottleneck. We will include these ablation results and their analysis in our revised manuscript.\n\n---\n\nW4 : **Comparison to LoRA**\n\nR4 :For a fair comparison, we built a LoRA-based test-time adaptation variant on WideResNet-28 (WRN28) using the official ConvLoRA implementation. Additionally, we inserted LoRA adapters into the same early convolutional blocks where our Buffer Layer is placed, and updated only those adapters with TENT’s entropy-minimization objective at test time. The resulting CIFAR-10-C error rates are shown below:\n\n| Method      | Config     | BS2 |BS4 | BS16 |\n|-------------|------------|------------|-------------|--------------|\n| Tent| LoRA(early)    | 39.20       | 29.85       | 21.02        |\n| |  LoRA(full)       | 87.13      | 66.42       | 24.39        |\n|         | @BN         | 82.56      | 51.36       | 23.12        |\n|             | @Buffer     | **37.05**  | **29.11**       | **20.32**    |\n\nOur Buffer Layer consistently outperforms the low-rank weight modulation of LoRA under the same test-time adaptation regime. We believe this is because domain shifts often manifest as localized, spatially varying distortions that are better addressed by our convolutional residual branches, whereas global low-rank updates struggle to correct such input-specific feature misalignments.\n\nThank you for this insightful question. Addressing it has helped us deepen our analysis and strengthen our empirical results. We will add this comparison and discussion to the revised manuscript. Please let us know if you have any further comments!"}, {"review_id": "iPhGgHc6bs", "reviewer": "Reviewer_RLdc", "summary": "The paper investigates the problem of test-time adaptation (TTA) from the angle of which parameters to update, instead of how to update them. The proposed method builds on the idea that most approaches update the batch norm (BN) parameters of the model, however these parameters are highly sensitive to small batchsizes. Similar to EcoTTA, the method keeps the model untouched and instead adds a Buffer module on top of initial layers of the network, which is adapted instead of the BN parameters using the usual entropy or augmentation consistency losses. In a comprehensive set of experiments, authors show that adding their Buffer Module on top of existing TTA boosts their accuracy under different types of data corruption settings.", "questions": "Please see the weaknesses above. Technically, my current evaluation is closer to a Weak Accept. I would be willing to upgrade my score to Accept if my concerns are addressed by authors.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* The method is simple but effective. The Buffer module can be added on top of most TTA approach, without any modification, to improve their robustness to corruptions at test time.\n\n* As the method is rather simple to explain, the paper instead builds on evaluating it under a broad range of settings. Overllal, results of these experiments are convincing.\n\n* The paper is quite well written. Thanks !\n\nWeaknesses:\n\n* Although mentioned very quickly in Section 2.1, the paper does not properly present the efforts of previous works to address the instability of BN update for small batchsizes. Those include SAR which replaces the BN by Layer Norm or Group Norm, and RoTTA which updates the statistics using an exponential moving average. \n\n* Some of the reported results do not match with those published in previous works. For example, in Table 3, authors report a classification error of 94.34% for SAR in the BN=2 and GN setting, while the error reported in the SAR paper for the same setting is 65.5%.\n\n* Moreover, some comparisons in the experiments are a bit unfair. For instance, the Layer Norm variant of SAR  (on a ViT backbone) achieves much stronger results, but authors only evaluate their method on CNN backbone. Furthermore, EcoTTA is not included in the comparison since it requires initializing the external module with source data. However, a better picture would be obtained by adding this method, with this initialization and also using a random initialization like in the paper.\n\n* As the module starts with randomly initialized weights, its prediction for the first test samples might be very noise. This problem is alleviated by employing a small alpha to reduce the module's initial influence. I believe this problem should be discussed with greater depth in the paper.", "was_revised": true, "final_justification": "Although I initially considered upgrading my score to 5, I was not satisfied with the authors' answer regarding the comparison to other approaches. In particular, they did not compare their method with the Layer norm variant of SAR, as suggested in my comment, and instead reported results for the weaker Batch norm and Group norm variants.", "rebuttal": "We’re grateful for your thoughtful feedback, which has helped us refine our manuscript. Below, we provide our detailed responses to your comments.\n\n---\n\nW1 :  **Lack of presentation of previous works on instability of BN updates with small batch sizes**\n\nR1 : We thank the reviewer for highlighting the important work on stabilizing BN in low‑batch regimes, notably SAR, which replaces BatchNorm with LayerNorm or GroupNorm, and RoTTA, which smooths BN statistics via an exponential moving average. We fully recognize the value of these approaches. In our revision, we will expand Section 2.1 with a detailed survey of SAR’s normalization swap and RoTTA’s EMA‑based statistic update, and clearly contrast them with our Buffer Layer strategy—where we preserve BN and instead introduce lightweight parallel modules for adaptation. This addition will both credit these contributions appropriately and strengthen our related‑work discussion.\n\n---\n\nW2 : **Discrepancy in SAR results**\n\nR2 : We apologize for the confusion. The 65.5% error reported in the SAR paper refers to a **ResNet50 with GroupNorm**, whereas our Table 3 uses the **ResNetV2** backbone from Big Transfer [Kolesnikov et al., ECCV 2020], which—despite **also employing GN**—is a **different architecture**. Our purpose of including ResNetV2 results was to demonstrate the broad applicability of our method across diverse backbones.\nTo resolve this misunderstanding and clarify our comparisons, we have reproduced the SAR setting on ResNet50 (GN), with the following results:\n\n| Method | BS 1 | BS 2 | BS 4 | BS 16 |\n|------------------|------------------|-------------------|-------------------|-------------------|\n| Tent  @ GN               | 85.30 | 80.49 | 77.37 | 71.60 |\n| Tent @ Buffer | 71.58 | 78.48 | 71.58 | 69.66 |\n| SAR @ GN | 66.03 | 67.78 | 66.75 | 67.89 |\n\n\nAlthough the SAR paper only reports results at batch size 1, our reproduction (**66.03 vs. 65.5 on SAR paper Table 4**) is in close agreement, supporting the validity of the reproducibility and the reliability of our evaluation pipeline. We also applied our Buffer Layer on this ResNet50(GN) setup and observed a consistent error reduction on Tent, underscoring the method’s effectiveness across architectures. We appreciate this insightful question and will update to clearly distinguish between the different backbones and their respective results.\n\n---\n\n\nW3 : This question touches on two separate issues, so to ensure clarity we will address them in two parts:\n\nW3-1 :  **SAR on CNN(BN)/ViT(LN) : Fair Evaluation issue**\n\nR3-1 : We appreciate the reviewer’s observation that SAR was designed specifically for GN or LN (on Vision Transformers) and that evaluating exclusively on CNNs with BN may seem unfair. We have already included **ResNetV2(GN)** comparisons in Table 3 of the main manuscript. To further demonstrate our method’s generality beyond CNNs, we adapted the Buffer Layer to a **ViT‑B/16** backbone, by converting the Buffer Layer with its 2D convolutional path into a simple 1D convolution over the (B, N, C) token sequence. We emphasize that, while our original 1×1 + 3×3 design targets 2D feature maps and **may not be optimal** for Transformers, this proof‑of‑concept shows extensibility. The results below confirm that TENT@Buffer (1D conv on ViT‑B/16) outperforms the SAR LN variant at Batchsize 4. (To further validate reproducibility, we began by verifying that our reproduction of the SAR setting at batch size 1 closely matches the 65.5% reported in the SAR paper).\n\n| Method | | Error rate(%) |\n|------------------|------------------|------------------|\n| Tent |@BN                | 54.40 |\n| |@ Buffer | **52.07**|\n| CMF| @BN| 67.77 |\n| |@ Buffer | 66.74 |\n| SAR|@BN    | 53.59 |\n| | @ Buffer | 54.09 |\n| Deyo|@BN                 | 65.05 |\n| |@ Buffer | 65.62 |\n| ROID      |   @BN       | 68.87 |\n| |@ Buffer | 66.02 |\n\n\nWe think that a transformer‑tailored buffer module might yield even greater gains, and this remains an exciting direction for future research. We will update Section 3.2 to include this comparison and clearly highlight our method’s broader applicability. Thank you for this excellent question, which has allowed us to rigorously validate our approach once again.\n\nW3-2 :  **EcoTTA with/without Initialization**\n\nR3-2 : Thank you for this valuable suggestion. As noted, EcoTTA requires source‑data warming, which is why we did not include it originally. However, we agree that presenting both the warm‑up and fully source‑free variants of EcoTTA will enrich our experimental comparison. Below we report EcoTTA error rates on CIFAR‑10‑C (WRN28 backbone) both with source‑data warm‑up and without (fully source‑free), alongside our TENT@Buffer results:\n\n|                      | **BS2** | **BS4** | **BS16** |\n|----------------------|--------:|--------:|---------:|\n| EcoTTA, random init  | 88.64 | 88.15 | 83.26 |\n| EcoTTA, warmup       | 82.72   | 53.78   | 21.95    |\n| TENT @buffer         | 37.05   | 29.11   | 20.32    |\n\nAs shown, TENT@Buffer outperforms EcoTTA across all batch sizes, even when EcoTTA leverages source‑data warm‑up. This mirrors the EcoTTA authors’ observation that small‑batch regimes degrade their stability—necessitating an additional ‘adaptBN’ module—whereas our Buffer Layer achieves robust, stable performance on small batches without any source data, demonstrating a clear advantage. \n\n---\n\nW4 : **Initial noise from random buffer weights**\n\nR4 : Thank you for this excellent insight and for giving us the opportunity to delve deeper into our buffer design. In addition to the small‑batch gradient noise you highlighted, we believe there are two primary sources of noise in our adaptation procedure: (1) the inherent variance from very small batch updates, and (2) the random initial weights of the buffer itself.\nTo isolate the latter, we compared our standard random initialization against a zero‑initialized buffer across a range of alpha values (1e‑5, 1e-3, 1e‑1). As summarized in the table below, the zero‑init buffer shows minimal sensitivity to α, maintaining stable error rates, whereas the randomly initialized buffer’s error varies by nearly 1 pp across the same α sweep. Surprisingly, at large batch regime, the zero‑initialized buffer even outperforms our original random‑init results—underscoring that initialization‑induced noise can be as impactful as batch‑size effects. These observations echo similar findings in LoRA and other PEFT works and point to initialization strategy as a rich direction for future exploration.\n\n| init_method | lr        | BS2     | BS16     | BS256    |\n|-------------|-----------|---------|----------|----------|\n| random| 1.00E-05  | 74.85   | 35.87    | 33.11    |\n|             | 1.00E-04  | **74.67** | 35.85    | 33.10    |\n|             | 1.00E-03  | 74.98   | 35.87    | 33.10    |\n|             | 1.00E-02  | 75.00   | 35.81    | 33.09    |\n|             | 1.00E-01  | 74.87   | **35.80** | **32.91** |\n| zero| 1.00E-05  | 75.07   | 34.34    | 31.77    |\n|             | 1.00E-04  | 75.12   | 34.32    | 31.79    |\n|             | 1.00E-03  | 75.24   | 34.27    | 31.82    |\n|             | 1.00E-02  | 75.19   | 34.39    | 31.80    |\n|             | 1.00E-01  | **74.76** | **34.21** | **31.45** |\n\nThank you again for this valuable feedback, which helped make the paper substantially richer. If you have any further comments or follow-up questions, we would be grateful to hear them."}, {"review_id": "KN1yOCa5zP", "reviewer": "Reviewer_kqmB", "summary": "Conventionally, test-time adaptation methods often update the parts of the model parameters, such as the affine parameters in the normalization layers, or update the entire parameters, but these methods suffer from the problem of ineffectiveness when the batch size is small and the catastrophic forgetting problem. This paper addressed these problems by introducing a light-weight network module, called Buffer Layer, and updating it while freezing the original network parameters.", "questions": "The paper was clear and easy to understand, so I don't have any additional questions. I have summarized my concerns in the \"weakness\".", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengthes**\n1. The proposed method is simple and effective, and　it has a significant contribution to this research area: This paper proposed a very simple method that adds lightweight modules to the base network, yet it effectively addresses the problems of the conventional approaches. In addition, in the experiments, the proposed method demonstrated its effectiveness when introduced to several existing methods, which indicates the broad compatibility of the method. These results demonstrate the potential to completely replace the conventional approach of training only normalization layers, and I believe that this will be a significant contribution to this field.\n2. Comprehensiveness of the Experiment: The experiments and analysis were comprehensive and adequately supported the claims of this study.\n3. Paper organization: This paper is well-written and easy to follow.\n\n\n**Weaknesses**\n1. Comparison to parameter-efficient fine-tuning (PEFT) methods: The proposed method is similar to the existing PEFT methods (e.g., LoRA), which update only a small number of additional parameters. It is necessary to discuss the difference between these methods and the proposed method.\n2. Applicability to the transformer-based networks: The proposed method seems to be designed for the convolution-based networks like ResNet, but there remains a concern about the applicability to the non-convolution-based networks like Vision Transformer. The modern neural network is mainly built upon the transformer-based architecture, and thus, it is necessary to investigate the applicability to such transformer-based models. In addition, this is related to the weakness 1., LoRA is applicable to the transformer-based models, so it is interesting to compare the effectiveness of the proposed method and LoRA.\n3. Analysis about the forgetting: In the experiments, it is shown whether the proposed method is effective against forgetting, but there seems to be a lack of discussion or analysis of why it is effective. The authors explained this is because the original model parameters are frozen, but I think the activations inside the network are changed by the buffer layer training, and this change may cause catastrophic forgetting. I believe that analyzing the mechanism by which the proposed method prevents forgetting will lead to a further understanding of the proposed method.\n4. Minor points:\n    + Typo in l135: Sec Sec. -> Sec.\n    + Figure 3 is not referred to in the main text.\n    + In Figure 3, I'm not sure what \"DF\", \"KW\", and \"KWC\" mean.", "was_revised": true, "final_justification": "The author's rebuttal has largely addressed my main concerns, such as applicability to ViT, comparison to other PEFT methods, etc., so I would like to maintain my initial “Accept” rating with higher degree of confidence.", "rebuttal": "We’re grateful for your detailed feedback. It helped us improve the manuscript, and our responses to your points are below\n\n---\n\nW1 :  **Comparison to parameter‑efficient fine‑tuning (PEFT) methods**\n\nR1 : While both our Buffer Layer and PEFT methods like LoRA freeze pretrained weights and learn only a few extra parameters, they contrast sharply in purpose and setting: PEFT leverages labeled downstream data for supervised fine‑tuning on a known task, whereas our Buffer Layer performs unsupervised test‑time adaptation, updating on unlabeled samples to correct domain shifts at inference. Architecturally, LoRA injects low‑rank updates into existing weight matrices, while we append lightweight parallel 1×1 and 3×3 convolutional branches alongside the backbone. We have provided **a detailed LoRA comparison in R2** and will expand the Introduction and Related Work to underscore these key differences.\n\n---\n\nW2 : **Applicability to the transformer-based networks**\n\nR2 : Thank you for highlighting the importance of evaluating on Vision Transformers and comparing to PEFT methods in that setting. First, to demonstrate that our framework can be effectively applied to ViT, we converted each 2D convolutional branch into a simple 1D convolution over the (B, N, C) token sequence of a ViT‑B/16 backbone. The results on ImageNet-C with Batchsize 4 are as follows:\n\n| Method | Error rate(%) |\n|------------------|------------------|\n| Tent                 | 54.40 |\n| Tent @ Buffer | **52.07**|\n| CMF| 67.77 |\n| CMF @ Buffer | 66.74 |\n| SAR                | 53.59 |\n| SAR @ Buffer | 54.09 |\n| Deyo                 | 65.05 |\n| Deyo @ Buffer | 65.62 |\n| ROID                | 68.87 |\n| ROID @ Buffer | 66.02 |\n\nAs the table demonstrates, the Buffer Layer shows its effectiveness on ViT-B/16, confirming that the approach generalizes beyond convolutional backbones.\n\nFor the LoRA‑based TTA variant, we initialized all LoRA adapters to zero and leveraged TENT’s entropy‑based update mechanism—mirroring our buffer update strategy but within the LoRA parameterization. The results are as below : \n\n| Method      |BS4 | BS16 | BS64 |\n|-------------|-----------|------------|------------|\n| Source (No Adaptation)     | 59.51     | 59.51      | 59.51      |\n| LoRA + Tent | 95.27     | 86.67      | 71.44      |\n| Tent @Buffer     | 54.48     | 56.88      | 58.88      |\n\n\nAs shown, TENT\\@Buffer outperforms the LoRA-based adaptation on the **same Transformer backbone**, confirming our buffer strategy’s competitiveness **beyond CNNs**. While structurally and conceptually similar to LoRA in that both freeze the backbone and add a small adaptation module, the test-time adaptation setting exposes a clear difference: our parallel convolutional residual branches with controlled corrections are more efficient and better suited to correcting domain shift than LoRA’s low-rank weight updates, making it a distinct (and in this case more effective) design. We agree that a Transformer-specific buffer module could yield further gains. Thank you again for this insightful suggestion.\n\nTo further discuss and verify whether the positional preference for Buffer Layer insertion that we discussed in the main manuscript carries over to Transformers, we conducted an additional ablation study on ViT-B/16, varying where the Buffer is attached (e.g., at different depths). \n\n| BS4 | Error rates|\n|------------------|------------------|\n| Layer1 | 52.23|\n| **Layer2** | **52.07**|\n| Layer3 | 54.73|\n| Layer4 | 55.23|\n| Layer5 | 55.23|\n| Layer6 | 54.28|\n| Layer7 | 55.68|\n| Layer8 | 55.50|\n\nAs in Table 6 of the main text, performance varies with insertion location: attaching the Buffer at early depths produces the largest gains, while later placements yield smaller improvements. This finding both increases consistency with the patterns reported in the main manuscript and is an interesting result in its own right—it suggests that the insertion dynamics we observed in CNNs have a similarity in Transformer architectures, reinforcing the generality of the design.\n\n---\n\nW3 : **Analysis about the forgetting**\n\nR3 : The reviewer’s point is well taken, and we’ve been exploring the same question: if the buffer injects corrections in parallel, why doesn’t that induce forgetting? We believe the answer is that the Buffer Layer behaves more like a **conditional residual adapter** than an overwrite. Its output is added to the backbone’s output in a data-dependent way, with a learned scale, so the original representation is preserved and only gently adjusted based on the input: on **target** inputs the residuals actively compensate for domain shift, whereas on **source** inputs the added corrections tend to be near-neutral and do not significantly perturb the representation. Because the backbone is frozen, upstream features remain fixed, and the buffer’s influence is localized and tempered, avoiding wholesale drift.\n\nSupporting this, Supplementary Figure 2 shows that after adaptation the model’s target feature distribution moves closer to the oracle (as expected), while the source data distribution largely retains its original shape. Even though the buffer adds a residual correction, it does not collapse or significantly distort the source representation, which is consistent with a conditional, controlled adjustment rather than a destructive change. This behavior, improved target alignment with minimal impact on source, suggests why catastrophic forgetting is mitigated in our approach.\n\n\n---\n\n\n\n**[Minor Points]**\n- Actually, it is \"**See** Sec\", Not \"Sec Sec\". We will change this sentence to such as 'Note Sec~' etc.\n- Thank you so much for your detailed comments. We will add referring  Fig.3. in the main text.\n- Sorry for the inconvenience. The CIFAR‑10‑W dataset is a realistic‑domain‑distribution benchmark for evaluating domain shift, comprising KW: keyword search domains, KWC: keyword with color domains, and DF: diffusion‑generated domains; we will add detailed descriptions in the main text.\n\nWe’re grateful for your careful and insightful review; it’s been extremely helpful in strengthening the paper. Thank you."}, {"review_id": "DzSNEJFL6l", "reviewer": "Reviewer_S6cX", "summary": "This paper proposes a novel concept of \"Buffer Layer\" for Test Time Adaptation (TTA) scenarios. This concept breaks through the traditional method of relying on normalization layer updates and addresses the problems of statistical instability of normalization layers in small batch sizes. The design of the buffer layer preserves the integrity of the pre-trained model and mitigates the risk of the catastrophic forgetting problem during online adaptation. Experimental results show good performance improvement in CIFAR and ImagNet corruption datasets.", "questions": "See the weaknesses. By addressing these weaknesses, the paper would provide a more comprehensive and robust evaluation of the Buffer Layer, further solidifying its contributions and potential impact in the field of test-time adaptation.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed Buffer Layer is a plug-and-play module and can be seamlessly integrated into any pre-trained model architecture. This modularity makes it highly applicable to a wide range of models without requiring significant modifications to the original architecture.\n2. The approach effectively mitigates the risk of catastrophic forgetting, which is a common issue in online adaptation scenarios. By preserving the integrity of the pre-trained backbone and isolating the adaptation process to the Buffer Layer, the model retains critical knowledge learned during pre-training. This ensures that the model does not lose its original capabilities while adapting to new domains, making it more robust and reliable in dynamic environments.\n3. The Buffer Layer shows strong compatibility with existing TTA methods, consistently providing additional gains when combined with traditional normalization-based adaptation techniques. This integration enhances performance across different test-time batch sizes, demonstrating its effectiveness in both small and large batch scenarios.\n\nWeaknesses:\n1. Limited Backbone Diversity: While the paper claims that the Buffer Layer can be integrated across various architectures, the experiments are limited to ResNet-based backbones. The effectiveness of the Buffer Layer on other popular architectures, such as those based on Transformers, remains unexplored. Given the increasing prevalence of Transformer-based models in various domains, it is essential to validate the Buffer Layer's performance on these architectures to fully assess its applicability.\n2. There is no detailed analysis of the computational complexity or testing time cost after integrating the proposed Buffer Layer. Understanding the additional computational overhead introduced by the Buffer Layer is crucial for practical deployment, especially in resource-constrained environments. A thorough analysis of the trade-offs between performance gains and computational costs would provide a more comprehensive evaluation of the Buffer Layer's practicality.\n3. The proposed Buffer Layer shares some similarities with existing techniques such as Adapters or Side Tuning, which are primarily used in parameter-efficient fine-tuning.\n4. The paper does not include experiments with a batch size of 1, which is a common scenario in real-world applications, especially in online or streaming settings.", "was_revised": true, "final_justification": "This work proposed a plug-and-play Buffer module, showing performance gain cross different baselines. The rebuttal adressed most of my concerns. It seems like a PEFT method for TTA in an unsupervised manner. I keep my score towards borderline accept.", "rebuttal": "Thank you very much for your detailed and insightful comments. We’ve tried to answer each concern as clearly as possible and have made the corresponding revisions.\n\n---\n\nW1 :  **Limited Backbone Diversity**\n\nR1 : We appreciate that multiple reviewers have raised the need to validate our Buffer Layer **beyond ResNet backbones**. To that end, we conducted experiments on a ViT-B/16 model: we preserved our parallel-kernel design by converting each 2D convolution into a 1D convolution over the (B, N, C) token sequence and attached the Buffer Layer immediately after each Transformer encoder block’s attention output. Despite being originally optimized for CNNs, this straightforward adaptation yields a meaningful gain on ImageNet, as shown below:\n\nThe results on ImageNet-C with Batchsize 4 are as follows:\n\n| Method | Error rate(%) |\n|------------------|------------------|\n| Tent                 | 54.40 |\n| Tent @ Buffer | **52.07**|\n| CMF| 67.77 |\n| CMF @ Buffer | 66.74 |\n| SAR                | 53.59 |\n| SAR @ Buffer | 54.09 |\n| Deyo                 | 65.05 |\n| Deyo @ Buffer | 65.62 |\n| ROID                | 68.87 |\n| ROID @ Buffer | 66.02 |\n\nAs the results show, the Buffer Layer also works on Transformer backbones. That said, here we have merely taken the CNN-optimized Buffer and converted its parallel convolutions into simple 1D counterparts—this is a straight-forward, proof-of-concept adaptation. A buffer module designed specifically for Transformers (e.g., one that better leverages the self-attention structure) or a more architecture-agnostic/generalized design would likely yield even stronger results.\n\n---\n\nW2 : **Analysis of the computational complexity or testing time cost**\n\nR2 : Thank you for this excellent question. To give a sense of the overhead, we include a summary comparison in the table below. \n\n\n| Method       | # of Trainable Parameters              |                      | Vram Usage (GiB) |             | Inference Time (s) (per batch) |             |\n|--------------|----------------------------------------|----------------------|------------|-------------|-------------------------------|-------------|\n|              | WRN28 (CIFAR10)                        | Res50 (ImageNet)     | WRN28      | Res50       | WRN28                         | Res50       |\n| Tent @BN         | 18K / 36.5M (0.05%)                    | 53K / 25.6M (0.21%)  | 2.2       | 3.4       | 9.35                          | 11.15       |\n| Tent @Buffer     | 1.79M / 38.3M (4.67%)                  | 77.9K / 25.6M (0.30%)| 2.9       | 3.9        | 20.45                         | 15.28       |\n| Tent @Buffer+BN  | 1.81M / 38.3M (4.74%)                  | 131K / 25.6M (0.51%) | 2.9       | 3.8        | 20.4                          | 15.30        |\n| SAR @BN        | 18K / 36.5M (0.05%)                    | 30.6K / 25.6M (0.12%)| 2.2       | 3.3        | 19.06                         | 19.91       |\n| SAR @Buffer | 1.79M / 38.3M (4.67%)                  | 77.9K / 25.6M (0.30%)| 2.7       | 3.7        | 31.04                         | 23.49       |\n| SAR @Buffer+BN  | 1.81M / 38.3M (4.74%)                  | 108K / 25.6M (0.42%) | 2.7       | 3.6        | 32.21                         | 24.27       |\n\n\n\nAs shown, integrating the Buffer Layer incurs a modest increase in number of parameters, vRAM, and latency. There is some added test-time cost—especially during adaptation—because the Buffer requires additional computation—but because only a small number of parameters is trained and updates are localized, the extra computational burden remains light. Considering the substantial accuracy gains under domain shift, we believe this trade-off is favorable in many practical settings.\n\nMaking the Buffer Layer even more efficient while preserving its effectiveness (e.g., via low-rank approximations of Buffer Layers, conditional activation, or lightweight architectural variants) is a promising direction for future work.\n\n---\n\nW3 : **Similarity to PEFT methods**\n\nR3 : **Comparison to parameter-efficient fine-tuning (PEFT) methods**\nWe agree that the Buffer Layer might be seen conceptually similar to PEFT techniques such as Adapters, Side Tuning, and LoRA—each freezes (or largely preserves) a core model and adds a lightweight auxiliary component. However, the **goal and setting** differ: PEFT methods are typically applied with **labeled data** to fine-tune for a known downstream task, whereas our Buffer Layer is designed for **unsupervised test-time adaptation**, correcting domain shift on-the-fly using only unlabeled target samples. Architecturally, while LoRA and Adapters introduce low-rank or bottleneck transformations inside or alongside weight matrices, and Side Tuning adds a parallel branch with supervision, our Buffer Layer uses parallel convolutional residual corrections (1×1 + 3×3) that are conditioned and scaled to gently adjust representations without overwriting them.\n\nTo make this distinction concrete, we will include two sets of controlled empirical comparisons: one with **LoRA applied to convolutional backbones** (original version and matching the depth and insertion points of our buffer and using the same TENT entropy-minimization objective for test-time adaptation) and another with **LoRA on transformer-based backbones** (e.g., ViT), again under the same unsupervised adaptation protocol. \n\n| Method      | Config     | BS2 |BS4 | BS16 |\n|-------------|------------|------------|-------------|--------------|\n| Tent| LoRA(early)    | 39.20       | 29.85       | 21.02        |\n| |  LoRA(full)       | 87.13      | 66.42       | 24.39        |\n|         | @BN         | 82.56      | 51.36       | 23.12        |\n|             | @Buffer     | **37.05**  | **29.11**       | **20.32**    |\n\nThe table above reports ConvLoRA results on WRN28 / CIFAR-10 under the same test-time entropy-minimization objective used for our Buffer Layer. As shown, Buffer outperforms ConvLoRA, despite the superficial structural similarity of both methods (freezing the backbone and adding a small adaptation module). We believe the gap arises because the Buffer Layer’s parallel convolutional residual branches can apply localized, input-conditioned corrections—which is crucial for handling spatially varying domain shifts—whereas ConvLoRA’s low-rank weight updates are more global and less capable of capturing those fine-grained distortions. In short, sharing the high-level idea isn’t enough: the specific design of the Buffer Layer appears better suited to the test-time domain adaptation setting, as evidenced by the performance difference.\n\n\n\n| Method | Error rate(%) |\n|------------------|------------------|\n| Tent                 | 54.40 |\n| Tent @ Buffer | **52.07**|\n| CMF| 67.77 |\n| CMF @ Buffer | 66.74 |\n| SAR                | 53.59 |\n| SAR @ Buffer | 54.09 |\n| Deyo                 | 65.05 |\n| Deyo @ Buffer | 65.62 |\n| ROID                | 68.87 |\n| ROID @ Buffer | 66.02 |\n\nThe table above shows results on ImageNet-C with ViT-B/16 (batch size 4); it again performs well and exhibits the same behavior described in the main paper. This illustrates that, although the Buffer Layer may superficially resemble PEFT, applying PEFT directly to TTA does not yield comparable results.\n\n---\n\nW4 : **Evaluation on Batchsize 1**\n\nR4 : Thank you for this important question. The reason we did not include vanilla BS=1 results for most of our reported baselines (TENT, EATA, DeYO, etc.) is that they are BatchNorm-based, and BatchNorm collapses in the extreme BS=1 regime—baseline accuracy on corrupted CIFAR-10-C and CIFAR-100-C falls to near chance (e.g., ~10% and ~1%, respectively)—so there is effectively no meaningful signal for the Buffer Layer to build on in those settings. In other words, when the underlying adaptation mechanism fails catastrophically due to BN’s instability at batch size 1, simply stacking a buffer on top does not salvage it, which is why those pure BS=1 results were not included.\n\nTo validate, we do have two complementary scenarios where BS=1 is feasible and the Buffer Layer still delivers gains:\n\n**ResNet50 with GroupNorm** : GroupNorm is relatively insensitive to batch size because it normalizes across channels/groups rather than across the batch. In this configuration, TENT (and by extension our adaptation pipeline) remains functional at BS=1, and we observe that attaching the Buffer Layer further improves performance, confirming that the method can work in true single-sample online settings when the normalization backbone is stable.\n\n|  Resnet50 (GN) (Error rate %) ↓ |  |\n|-----|-----|\n| | BS1|\n|Tent @GN | 85.30|\n|Tent @Buffer | **71.58**|\n\n\n**Semantic segmentation** : As also highlighted in the TENT paper, segmentation naturally provides dense, per-pixel supervision-like signals even with batch size 1, since each image yields many spatial locations. We evaluated a Deeplabv2 model under a day-to-night shift on the CARLA dataset, attaching the Buffer Layer to the encoder. Despite BS=1, we observed a meaningful increase in mIoU, demonstrating that the Buffer Layer can be effective in real-world streaming/online scenarios where each “batch” is a single image with rich internal structure.\n\n\n| | Deeplabv2 (mIOU ↑) |\n|-----|-----|\n| | BS1|\n|Source (No Adaptation) | 52.75|\n|Tent @BN | 55.98|\n|Tent @Buffer | **56.68**|\n\n\nTogether, these results indicate that although standard BN-based classifiers collapse at BS=1, the Buffer Layer can still be effective in single-sample settings when the normalization or task structure supports stable updates—making batch size 1 a valuable direction for further study. Thank you for this suggestion. The insightful questions provided have truly enriched the paper and helped us think more deeply about the practical aspects of TTA."}]

Author rebuttal:

Meta-review: This paper introduces a Buffer Layer for test-time adaptation (TTA), a lightweight parallel module that adapts at inference while keeping the backbone frozen. The design addresses the instability of normalization-based TTA and mitigates catastrophic forgetting. The method is simple, modular, and effective, showing consistent gains across diverse datasets and TTA baselines. The authors provided extensive additional experiments (e.g., ViTs, GN backbones, LoRA comparisons, initialization and complexity analyses) that strengthened claims of robustness and generality. Conceptually related to PEFT methods, with somewhat incremental novelty. Some evaluation details (e.g., SAR results, computational cost, Transformer applicability) remain limited, though the rebuttal addressed them substantially. Despite minor concerns, the paper makes a solid and practical contribution to TTA. I recommend acceptance (poster).
Decision: Accept (poster)