L-TTA: Lightweight Test-Time Adaptation Using a Versatile Stem Layer
Authors: Jin Shin
Hyun Kim
Venue: NeurIPS 2024 poster (2024)
Keywords: Test-Time Adaptation
Domain Shift
Discrete Wavelet Transform
Gaussian Modeling
Primary area: deep_learning_architectures

Abstract: Test-time adaptation (TTA) is the most realistic methodology for adapting deep learning models to the real world using only unlabeled data from the target domain. Numerous TTA studies in deep learning have aimed at minimizing entropy. However, this necessitates forward/backward processes across the entire model and is limited by the incapability to fully leverage data based solely on entropy. This study presents a groundbreaking TTA solution that involves a departure from the conventional focus on minimizing entropy. Our innovative approach uniquely remodels the stem layer (i.e., the first layer) to emphasize minimizing a new learning criterion, namely, uncertainty. This method requires minimal involvement of the model's backbone, with only the stem layer participating in the TTA process. This approach significantly reduces the memory required for training and enables rapid adaptation to the target domain with minimal parameter updates. Moreover, to maximize data leveraging, the stem layer applies a discrete wavelet transform to the input features. It extracts multi-frequency domains and focuses on minimizing their individual uncertainties. The proposed method integrated into ResNet-26 and ResNet-50 models demonstrates its robustness by achieving outstanding TTA performance while using the least amount of memory compared to existing studies on CIFAR-10-C, ImageNet-C, and Cityscapes-C benchmark datasets. The code is available at https://github.com/janus103/L_TTA.

L-TTA: Lightweight Test-Time Adaptation Using a Versatile Stem Layer
Jin Shin, Hyun Kim
Department of Electrical and Information Engineering and RCEIT Seoul National University Science and Technology
Seoul, Korea
{shinjin0103, hyunkin)sssoollteeee @seoul tech. ac.
Abstract
Test-time adaptation (TTA) is the most realistic methodology for adapting deep learning models the real world using only unlabeled data from the target do- main. Numerous TTA studies in deep learning have aimed at minimizing entropy. However, this necessitates forward/hackward processes across the entire model and is limited by the incapability to fully leverage data based solely on entropy. This study presents a groundbreaking TTA solution that involves a departure from the conventional focus minimizing entropy. Our innovative approach uniquely remodels the stem layer (i.e., the first layer) to emphasize minimizing new learn- ing criterion, namely, uncertainty. This method requires minimal involvement of the model's backbone, with only the stem layer participating in the TTA process. This approach significantly reduces the memory required for training and enables rapid adaptation to the target domain with minimal parameter updates. Moreover, maximize data leveraging, the stem layer applies discrete wavelet transform the input features. It extracts multi-frene multi-frequency domains and focuses on minimizing their individual uncertainties. The proposed method integrated into ResNet-26 and ResNet-50 models demonstrates its robustness by achieving outstanding TTA performance while using the least amount of memory compared existing studies on CIFAR-IO-C, ImageNet-C, and Cityscapes-C benchmark datasets. The code is available https://gthbb..com//aasS.1....,
Introduction
From the inception of deep learning research, the problem of overfitting, where model's 's performance becomes biased towards its training dataset, has been a persistent issue. Various solutions such as
normalization and regularization have been proposed address these issues 26 56 30 Additionally, the need for trained models to adapt to the diversity of data distributions in the real world has also emerged recently. This phenomenon, known domain shift, has resulted in the proposal of various scenarios and practical studies [63] Scenarios for addressing model domain shifts are generally introduced with the key themes of domain adaptation (DA) or domain generalization (DG). These differentiated by their capability to access the data in the target domain. To clarify, DA can access all domains but can only use unlabeled data in the target domain [6
58 2). whereas DG can only train in the source domain [66] 34 49 35 Recent source-free free DA has indicated that accessing source domain data is generally infeasible. It has proposed a scenario where the target domain adapts using a teacher model that generates pseudo labels [48 B1 [2] However, this approach involves the significant cost of maintaining the large memory required for the teacher model.
64
*Corresponding author: Hyun Kim, email: Syunkim@xcoultechh.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

0000
PC Sattmar
Eminges
TENT
ECoTTA
Ours
Perward
Backward
Ojdate
Figure 1: Diagram comparing the forward/backwan flow and update process with TENT [59] and ECOTTA [5 illustrated as representative algorithms for entropy minimization and memory-efiiiiet methods, respectively. The red lock icon indicates the absence of TTA execution.
Within our current understanding, the most realistic research topic for deploying deep learning models in the domain shift field at present is test time adaptation (TTA) [39]. This approach relies solely on pre-trained model parameters and data from the target domain without generating pseudo labels. Typically. TTA research focuses on two critical observations: (1) the variation in batch normalization (BN) statistics when domain shift occurs models are trained to minimize the entropy of predictions [59]. Notably, recent studies based on entropy minimization improved the performance by simultaneously passing various augmentations through the input at test-time and either minimizing the average entropy (65] or filtering out noisy data points with high entropy that hinder training [45]. Although these studies enhanced the adaptation performance, they were limited by the substantial training costs and memory usage (which reduced their practicality). For these reasons, TTA performance improvement must be achieved, but in practical terms, there is need to simultaneously eliminate unnecessary processes and conserve memory resources as much as possible due to gradient, which causes the largest power consumption in training.
and (2) the possibility of adaptation when
We summarize the fundamental problems of such studies into three main aspects: (1) Forwarding the entire model process to obtain the entropy involves cost. (2) To enable the backward process, a substantial cost would be incurred to reach the initial normalization layer. (3) The absence of method to ensure data independence may result in biased learning in the incorrect direction if noisy data are not identified, and even such data are identified, this process can increase the training time.
Our research begins with the hypothesis that fine-tuning the first convolutional (CONV) layer, known as the stem layer, can significantly impact the TTA results. This is based on the understanding that domain shifts in input images affect model outcomes. To overcome the drawbacks of the aforementioned entropy minimization, we introduce new paradigm for fast and high-performance TTA. This is achieved using reconstructed stem layer that incorporates a Gaussian channel attention layer (GCAL) constructed on the squeeze and excitation (SE) block (25) and domain embedding layer (DEL) based on two-dimensional discrete wavelet transform (2D DWT) (29 Our main contributions can be summarized as follows:
To implement lightweight TTA, we minimize the uncertainty of channel-wise attention in intermediate features extractable from GCAL rather than an entropy minimization strategy. As depicted in Fig. D this involves omitting both forward and backward processes for all the parameters except those in the reconstructed stem layer. Using this stem layer pivot point without storing additional gradients provides the fastest training and significantly reduces training costs.
We propose the integration of DEL component within the stem layer to maximize the training effect a single data point. 2D DWT in the stem layer helps collect domain-invariant edge information efficiently and improves model generalization by providing multi-views with redundant content information before CONV operations. Furthermore, forwarding features extracted from multiple frequency domains to GCAL enables the calculation of uncertainty on an individual basis, thereby maximizing data utilization.
Additionally, DEL includes inverse DWT (IDWT), thereby enabling non- non-invasive design. This yields a shape identical to the intermediate feature obtained from the existing stem layer's CONV operation. This, in turn, enables convenient integration into any model utilizing convolutional neural network (CNN)-based backbone for image classification and wide range of applications (i.e., high scalability).

---

Experimentally, our method surpasses the state-of- art (SOTA) with a 5.4% higher prediction accuracy on CIFAR-10-C, including 15 types of corruptions. On the large dataset, ImageNet-C [22] it displays an accuracy lower than that of the SOTA model by 2.7%; however, it achieves a significant performance with only 1.7% of memory usage compared to the SOTA model. Furthermore, compared with ECOTTA [55], which is the SOTA in terms of memory efficiency, our approach uses 5.9% of memory and displays a marginal decrease in accuracy, by only 0.3%. These results highlight our method's 's effectiveness in balancing performance and memory efficiency. In terms of scalability, compared with EcoTTA, our method achieved a 4.6% improvement in mean intersection over union (mloU) with 0.04% of memory usage in a semantic segmentation task under four weather conditions (e.g., brightness, foggy, rainy, and snow).
2 Backgrounds
2.1 Domain Shift Scenarios
Deep learning models learn from high-dimensional data such as images and natural language. These models are influenced significantly by the data distribution [18]. Data derived from the source domain, denoted as D2 = { { Ve
and target domain, denoted as D, {(a,, ve (O....N-I) generally exhibit significantly different distributions. Therefore, it could be challenging for a model trained and fitted on data from D. to infer data from D.. Scenarios designed to address this domain shift are classified into DG and DA depending on whether there is direct access to the D, distribution during training. Both strategies primarily aim to align the feature space with domain-inveen invariant content information and style information sensitive variations. In DG, where access D, is unavailable, auxiliary networks are frequently employed to learn the adjustment of means and standard deviations to fit the domain [15] or use generative models to create D. adversarial to the distribution of D, and [49] 35 Recently, multiple augmentations were employed in conjunction with knwwedee regarding the domain directly. This secured superior generalization performance [6]. However, this competitive trend increases the load regarding training costs and inference, thereby necessitating improvements in practicality and efficiency. In DA, where direct access to D, is feasible, model generalization is more convenient. However, a significant characteristic is the unavailability of labels. Early DA research involved training with labeled data from D, [16 58 64). However, in practical applications, storing and utilizing the D, dataset in memory is considered unrealistic. To overcome this issue, source-free DA employs pseudo-labeling techniques by dually leveraging pre-trained models [48] 31] Nonetheless, these approaches still have the burden of maintaining the dual models in real-world application deployment as well as the problem of confirmation bias.
train simultaneously
an equal number of meta-networks to comprehend
2.2
Test Time Adaptation
Like DG and DA research, TTA is scenario developed to solve domain shifts. The focus is on more efficient learning during test-time (39) TTA essentially assumes training using only data ag without the corresponding label v Within these constraints, TTA adapts to the distribution of Dr by partially fine-tuning the layers of a pre-traaee trained model.
BN STAT 42 indicates the widespread covariate shift across all the domains that need to be addressed and adjusts the fixed parameters of BN layers, specifically, the mean and variance, at test-time. TENT 59 is the baseline for many TTA approaches. It proposes a strategy to minimize the prediction entropy and significantly improve the performance by updating statisties and affine transformation parameters for each batch. MEMO 655 applies multiple augmentations to the input at test-time and minimizes the average (marginal entropy) of entropies obtained by passing the model through, thereby demonstrating better adaptation effects. EATA [45] extends TENT as the baseline. It expounds that data points with high entropy do not contribute adaptation and proposed criteria for establishing a threshold. SAR [46] advances further by addressing realistic mixed distribution variations, small batch sizes, and imbalanced label distribution variations. aims for stable TTA by allocating the same class to all the samples and proposes a strategy to minimize the sharpness of the loss surface. REALM (53] achieves the highest performance among entropy minimization- Unlike other methodologies, REALM introduces a framework based on self-supervised learning that enables the inclusion of noise samples in training without skipping. However, based on existing research, achieving TTA through entropy minimization incurs the basic training cost for performing
studies.
3

---

forward/backward operations globally in the model. Moreover, it has limitations in assessing whether the data suitable for TTA through entropy.
For these reasons, there is a growing interest in whether the proposed methods can be applied efficiently in the real world. In particular, EcOTTA (55) introduces pioneering method minimize memory consumption during the TTA process. This is achieved by partially placing auxiliary networks for TTA outside the main network. The approach ensures that only the BN layers included in the main and auxiliary networks pass through during backpropagation, significantly shortening the path. This simultaneously reduces the memory required for gradients and the time consumed. MECTA [2] follows a training flow similar to ЕcoTTA, where all layers of the model are forwarded and a specialized normalization layer is introduced to minimize memory usage, particularly within the cache. However, it relies on methodologies like TENT and EATA to perform TTA. SFT [33] based on MEMO divides the model into subsets and introduces a method of learning only certain layers based on the distribution of Dr. This provides insight into the fact that learning the initial layers is more effective in supporting adaptation learning than the subsequent ones. Notwithstanding these efforts, it is necessary to forward the entire model for TTA, owing the training based on entropy. Moreover, during backpropagation, the problem of reaching batch normalization at the beginning of the backbone remains while the gradients of many layers are still being calculated.
Therefore, approaches that can adapt at test time without relying on entropy minimization have also been introduced. DDA [17] has proposed a method that directly projects imputs obtained from D, to D, using generative model without performing expensive retraining. However, this can yield more expensive results in maintaining additional systems, similar to augmentation.
2.3 Discrete Wavelet Transform
The wavelet transform (WT) provides flexible time-frequenee resolution by analyzing signals at multi scales, making it a more effective tool than the Fourier transform in signal processing [4]. Moreover, owing to the short and localized characteristics of wavelet functions, WT is suitable for processing non-stationary signals [54]. Based on these advantages, 2D DWT (2 is commonly used to extract detail and edge information in the spatial domain. Additionally, it allows for multi-level decomposition until both the width and height of the input image are powers of two. This enables aggressive summarization and extraction of essential information. Through the inverse transform. the original image can be reconstructed highly accurately with minimal information loss. The 2D DWT process involves sequentially applying one-dimensional DWT (1D DWT) 44 to the rows and columns of the input data, For 1D DWT, we operate using the simplest wavelet family, iee., the Haar wavelet [36]. It should be noted that Haar be performed conveniently through CONV operations (47). In the first stage, the horizontal direction decomposition is performed to split a single original feature map into two components: the low-frequency component (LFC), which generally contains visually intuitive information and represents the basic image structure, and the gigh-frequency component (HFC), which includes finer details of the image and a small amount of edge information. In the second stage, a vertical decomposition is performed as the two filters are transposed, as illustrated in Fig. 2 The detailed 2D DWT process presented in Appendix [
3 Proposed Method
3.1 Overview
Motivation: To improve the performance of TTA, numerous studies have focused on efficiently minimizing the entropy of model predictions obtained from unlabeled data xÇ. Although this has enhanced the accuracy, efforts reduce the memory consumption, which significantly impacts the power consumption, and perform TTA faster have been relatively few. Nevertheless, these research areas are essential for applying TTA in real-world scenarios. In practice, using augmentation in image vre-processing during training. dynamically filtering data advantageous for training, or replicating pre- trained weights generates a significant load. Furthermore, we identify a key drawback most existing TTA studies that adopt entropy. Even without parameter updates, calculating entropy necessitates sarwaddbbacwwan passes through the entire model layers. This imposes a fundamental training cost. This limitation motivates us to resolve the existing issues in TTA.

---

Sarch Nerm
Layer
Excitation
Sigwwld
Unverninnty
Channd Afication
PAPAP
Input
DWT
CONY
GCAL
IDW
Outpat
Figure 2: Overview of our method including the reconstructed stem layer. @ and denote element- wise multiplication and CONV operation, respectively, while dh and dhg respectively denote the low and high filters for DWT, described in Appendix
Goal: eliminate existing drawbacks and minimize training costs, we aim to provide an alternative to entropy minimization as a TTA solution. Specifically, our design goals are summarized as follows: Practicality in training: Minimizing the resources required for training, such as memory and data, to attain acceptable a reasonable prediction accuracy in D,- 2) Scalability: Designing to be non-invasively and conveniently applicable CNN-based tasks without modifying other layers. 3) Data leveraging: Maximizing usability from independent data to achieve TTA within constraints, even with small batch sizes or a single batch.
The sequential influence of CNN layers results in significant variations in prediction quality across different input image domains. By fine-tuning the first representation of the input image, fast adaptation to D; is possible. Therefore, instead of expensive entropy, we extract and minimize the channel-wis wise uncertainty from the reconstructed stem layer adapt D;-
Approach:
Workflow: As shown in Fig. 2 the reconstructed stem layer comprises the original CONV layer and two key architectures in this study, GCAL and DEL. The reconstructed stem layer follows the sequence of DWT, CONV, GCAL, and IDWT. The processes DWT and IDWT are encompassed in DEL. As depicted in Fig. 2 GCAL is the only key architecture that enables adaptation and outputs channel-wise attention and its associated uncertainty in an end-to-end manner. Similarly, as with an SE block, the extracted attention is scalar and applied to each feature on per-channel basis. Meanwhile, the uncertainty is minimized through negative log-likelihood (NLL) loss. In DEL, DWT decomposes the input feature map into multiple frequency domains while maintaining spatiality, This step is performed before CONV. It allows for capturing more diverse features from independent data, thereby better grasping the differences between D, and D.. By performing IDWT the end of the stem layer, the shape of the features is restored to its original state before the stem layer is modified. This enables the non-invasive application of the stem layer in pre-trained models and synthesizes important redundant information from multiple views to enhance the generality. While injecting the stem layer into a pre-trained model for warm-up, we jointly train reduce task-specific prediction enrors using suitable loss terms (i.e., cross-entropy) in conjunction with the uncertainty extracted from GCAL minimized through NLL loss. should noted that since uncertainty minimized only during training, in the TTA setting, the procedures after IDWT are not required.
3.2 GCAL: Gaussian Channel Attention Layer
GCAL is a key architecture that enables TTA using only the stem layer. It remodels the SE block to predict uncertainty. Please note that the SE block (25 is not incorporated into other layers and is trained end-to-end. It dynamically extracts channel attention "scalte defined below from intermediate features and performs recalibration. This enhances the representational capacity of the CONV and, ultimately, the prediction accuracy.
Tecale Foc(X, o(Wg8(W1,
(I)
5

---

where X represents the intermediate feature obtained after performing global average pooling, which in ResNet [20] comprises 64 channels. The weight W1 reduces the channel size of X based on predetermined hyperparameter, whereas W22 conversely expands back to its original channel size. o and 8 denote the sigmoid and ReLU functions [43. respectively.
In Gaussian Yolov3 [8. [ rather than predicting variables such the bounding box coordinates, the model is designed to extract the mean and variance of Gaussian distribution to quantify the uncertainty. Similarly, in this study, the output of the SE block, Tseates is represented by Gaussian parameters, namely, mean Ti and variance T The probability density function for quantifying the uncertainty of Yecule defined as follows:
1 277h:
Co 23
p(u Tie (a)))))>=
275:
V
For implementation, without significantly altering the Fae structure, we obtain 7/4 and "Σ by increas- ing the channel size by 100%. This is shown in Fig. These two Gaussian parameters are defined as follows:
2ie oWW3 X)), 7s: a (W3'
To perform TTA, we minimize the Gaussian parameter "I to reduce the uncertainty of "scale: which is multiplied channel-wise. It should be noted that Yscale "he However, because Tp is determined dynamically based on the input, it is challenging to determine the ground-truth (GT) μ in the TTA setting. Therefore, we aim to modify the SE block to function primarily as extractor of uncertainty for TTA. In all the training settings, we set the GT of 7h to the maximum value of the sigmoid function (= 1) enable training. Reflecting this approach, redefine the NLL loss to minimize r (an uncertainty) in conjunction with Eq.
C-1 Σ 1=0
L Luncertainty
llog(p Tw. (s:)),
where C represents the total number of channels intermediate features. Eq. 4g.4means minimizing uncertainty for all channels, which applies equally to pretraining and 'TA processes.
3.3 DEL: Domain Embedding Layer
In recent TTA scenarios, the entropy minimization model is traditionally learned by filtering out data that is evaluated to have low entropy. However, contrary to this trend, [61] experimentalyy shows that HFC derived from input images helps generalize the model. This means that high entropy actually contributes significantly to improving prediction accuracy. This entropy ambiguity must be avoided improve the individual usability of single input data z However, to solve this problem, is clearly challenging to determine the entropy threshold and control strength without additional training costs test time.
To alleviate entropy ambiguity, we propose DEL. which encapsulates GCAL and CONV layers with DWT and IDWT layers, shown in Fig. [ This decomposes the independent xf into various frequency domains and allows for end-to-end learning of uncertainty for each channel. Accordingly, we redefine the loss term for uncertainty with Eq. 4 follows:
1 1 C-1N-1 CN
Luncertainty
ΣΣ
Σ
-log(pe, (Myti Yas 7s))),
(5)
j=0
where N represents the number of frequencies decomposed into DWT. As shown in Fig. 3(a), the DWT layer maintains spatiality without resynthesis even after frequency decomposition, unlike the well-known transformation method 4 51]. even after decomposition. This makes learning spatial structures, such as edges or patterns, possible without additional computation. The IDWT layer, located last, makes the input of the subsequent layer have the same shape as when there is no DEL and maintains the spatiality of the feature. As a result, DEL enables non- non-invasive design.
Furthermore, we propose omnidirectional decomposition (ODD), which additively decomposes LFC and HFC simultaneously. As shown in Fig. Bia, when DWT is performed once, it can be observed that edge information overlapping with LFC still remains in HFC. We decompose edge and noisy

---

LPC,.
HFC
HPC
HFC
DWT
LFC
-
TTA
By
DWY
DWT
DWV
Original
DWT Level-2
(b)
Figure 3: (a) visualizes the proposed ODD process in detail on given input. (b) shows the variance of uncertainty according to the domain shift of LFC (=LFCLL) and HFC (=H
information as much possible level that maintains spatiality, making it possible to calculate uncertainty single input x' more sensitively. Additionally, as a side effect, generalization performance is further enhanced by the model concretely identifying noisy data contained in HFC.
Fig. ()b visualizes the changes the uncertainty map for each of the 64 channels for LFC and HFC as we perform our proposed TTA. Denoted Ary and Af: are defined as follows:
A |log(F (Fee (X&: W) - Fse (X,,; W)), 4 = (Log(Fse (X,::)) Fee Fxe(Xx; (6)
In A7 LFC, there is a clear difference in the uncertainty for each channel obtained with X, and X, even before performing TTA. On the other hand, A. shows that FSE(.) does not have the capability properly obtain uncertainty about X,, Therefore, we prove through A that uncertainty can be correctly extracted by performing TTA.
3.4 Interval Training for Continual Setting
For performance assessment of TTA, achieving high prediction performance with low training cost is important, but it also important to design it to enable continuous training. To overcome catastrophic forgetting, existing studies [55] 62) make this possible by maintaining teacher model of the same shape as the target model or applying regularization techniques for each layer. Instead of these complicated strategies, reload the weights of the pre-trained stem layer and update them according to D2 at every short interval. This an efficient scenario when initialized weights consume less memory and simultaneously perform fast TTA, and the proposed reconstructed stem layer realizes this. The short interval setting allows for natural adaptation even when the domain shifts frequently and preserves the generalization performance of the pre-trained model.
4 Experimental Results
In this section, we quantitatively and qualitatively demonstrate three contributions related to TTA obtained by applying stem layers reconstructed by DEL and GCAL to CNN-based models. We evaluate the practicality in training (i.e., comparison accuracy and memory efficiency with SOTA TTA methods in Tab. T and Fig. 4) scalability (i.e., applicability
image classification and semantic segmentation in Tab. 2). and data leveraging (i.e., assessment of the TTA performance at small batch sizes in Fig. 7 In Appendices[[IIp and [F] we further rigorously evaluate the proposed method by extending the model, using challenging datasets, and exploring generalization scenarios (i.e., source-free DA), along with an ablation study for hyperparameters.
4.1 Implementation Details
In the experimental settings, the model backbones use ResNet-26 ResNet-50 (0) These are pre-trained with ImageNet [[4] To evaluate the robustness of the model in the existing TTA setting,
7

---

Table I: Comparisons of prediction error (%) with prior TTA methods CIFAR-1O-C with ResNet-22 and ImageNet-C with ResNet-50.
Method
Noise Gauss.
Blur
Weather Frost
Digital
Avg.
Shot Impul.
Defoc.
Glass
Motion Zoom
Snow
Fog
Brit.
Contr:
Elastic
Pixel JPEG
CIFAR 10
Source
86.9
82.6
81.8
11.4
50.2
18.9
9.1
16.4
26.4
18.4
7.1
24.5
22.8
64.0
28.3
36.6
GCAL
62.4
53.4
51.7
13.5
16.6
26.4
21.7
31.0
21.3
61.7
22.5
30.8
TENT 59
39.4
38.8
47,9
19.9
45.0
23.2
20.6
28.
32.1
24.5
16.1
26.7
32.4
30.6
35.5
30.7
MEMO65
43.5
39.9
43.3
26.4
44.4
25.1
25.0
20.9
28.3
22.8
11.9
28.3
21.1
42.8
21.7
29.7
SFT 33
31.9
26.7
44.2
18.4
20.2
23.4
20.7
25.4
24.5
21.9
25.1
24.2
EATA 45
33.9
32.8
41.4
19.4
42.4
20.5
20.1
22.4
27.1
22.7
13.8
24.0
24.0
29.3
26.5
26.7
SAR
46.4
40.9
50.1
20.2
21.7
20.8
29.5
23.9
25.5
24.3
39.5
27.4
30.3
REALM 53
27.8
25.4
35.5
13.5
37.7
17.4
16.9
20.4
22.3
19.0
12.9
18.0
23.1
22.0
24.2
22.5
Ours w/o DEL
33.1
29.1
31.3
36.9
11.4
12.8
13.1
11.6
6.9
18.4
15.7
24.5
17.8
Ours
31.0
26.8
30.5
9.2
34.9
11.4
13.1
13.8
12.2
7.5
18.4
12.6
22.6
17.1
ImaggNNt-CC
Source
97.8
97.1
82.1
90.2
85.2
77.5
83.1
76.7
75.6
41.1
94.6
83.1
79.4
68.4
82.0
Ours w/o GCAL
82.7
82.6
86.2
79.1
89.1
84.2
75.7
74.8
67.3
73.1
37.5
84.6
74.1
43.2
55.2
72.6
TENT
81.4
78.5
MEMO 65
81.5
79.5
81.6
82.9
87.4
78.2
73.1
59.6
53.0
63.6
30.5
63.5
80.8
67.9
46.7
68.8
DDA 17
57.6
56.7
57.7
83.4
80.4
78.1
74.0
59.9
86.3
38.8
74.8
62.5
53.4
45,9
64.9
EATA 45
75.2
71.7
74.3
81.9
82.7
71.5
70.7
55.5
55.7
58.4
29.1
55.4
73.0
53.2
63.5
SAR 46
76.6
73.4
76.1
81.6
84.6
71.4
69.6
55.1
55.3
74.3
27.7
55.5
85.2
53.0
43.9
65.5
REALM
71.1
50.3
Ours w/o DEL
75.8
75.0
76.7
85.1
88.3
76.0
68.1
62.0
58.7
53.0
30.3
82.7
63.4
62.5
58.0
67.7
Ours
79.4
78.8
75.8
72.4
63.5
58.5
53.5
73.1
57.0
41.4
54.5
64.9
use the benchmark dataset with the well-established 15 types corruptions (e.g., noise, blur, weather, and digital) [22). Each corruption is applied to the validation set of the original dataset and and ImageNet-C.
has identical content information. These are called CIFAR-1O-C. CIFAR- respectively. The severity of the corruption is divided into five levels. We conventionally apply the most severe level (= 5) for an unbiased comparison with the other methods. Additionally, to measure the memory reliably, we refer to the officially provided code of TinyTL [5]. It is identical to that measured by EcoTTA [55). This considers the memory for storing model parameters and gradients. The gradient size increases exponentially according to the batch size. A detailed evaluation setting of image classification and semantic segmentation models presented in Appendix A
140
(191/2412, and (6411))
Burts
Qut
Memory Usipe (ML))
104
98 3%
84
82
CIFAR.DBCC
Metries
Figure 4: Comparison of memory usage in sin- gle iteration on CIFAR-100-C and ImageNet-C datasets.
Figure 5: Training time comparison of various TTA methods on ResNet-50.
4.2 Main Results
Tab. reports prediction errors with regard to accuracy for the image classification task. mentioned in Sec. DEL included in the stem layer can display generalization performance without additional

---

Table 2: Comparison of mloU (%) for semantic segmentation in Cityscapes- using DeepLabV3Plus with prior TTA methods.
Method
Bright.
Foggy
Frost
Snow
Avg.
Mem. (MB)
Source
60.4
54.3
30.0
4.1
37.2
Ours w/o GCAL
69.9
62,9
47.2
41.6
55.4
BN 42
69.1
61.0
44.8
39.1
53.5
280
TENT 59
70.1
62.1
46.1
40.2
54.6
2721
Eco
S
70.2
62.4
46.3
41.9
55.2
918
Ours w/o DEL
72.0
65.7
49.6
45.3
58.1
0.4
Ours
70.6
63.5
53.1
52.0
59.8
0.4
assumptions regarding learning, and this exhibits average improvement of 5.8% and 9.4% in CIFAR-10-C and ImageNer-C, respectively, compared with the baseline (i.e. source Ours w/o GCAL... It should be noted that we apply the proposed interval training to meet the continual TTA condition. these experiments, the weights of the reconstructed stem layer are re-initialized to the pre-trained weights every 20 iterations for CIFAR-IO-C and every 10 iterations for ImageNet-C. As shown in Tab. M for CIFAR-10-CA it shows a 5.4% higher improvement on average than SOTA. Meanwhile, although the results of ImageNet-C in Tab. d not exceed the SOTA performance, achieves accuracy comparable to SOTA with significantly less memory usage as can be seen in Fig. 4 In addition, it is noteworthy that the proposed method achieves 3.9% better performance than MEMO 665. which performs entropy minimization using various augmentations in ImageNet-C.
Ultimately, the goal of injecting the stem layer is to demonstrate an efficient TTA scenario that considers the trade-off between efficient prediction and memory consumption, and our method displayed an overwhelming performance advantage in terms of memory-saving compared with prior works. In Fig. 4 we illustrate the memory usage for ResNet-50, with a focus on BN STAT, entropy minimization-ban
research, and memory-saving ЕсоTTA [55]. Herein, TENT [59]. EATA [59), and REALM [53] displayed equal memory consumption with the result of entropy minimization- based research. Compared to these three approaches, proposed approach consumes only 28.6%, 2.8%, and 8.8% memory for CIFAR- 100-C and 28.6%, 9, 1.7%, and 5.9% memory for ImageNet-C, respectively, for performing TTA.
In Fig. 5 we compare the proposed method with three representative approaches from entropy minimization-based TTA (i.e,, TENT), diffusion-driven TTA (i.e., DDA), and memory-efficient TTA (i.e., EcoTTA) methodologies in terms of latency on a general heterogeneous system (i.e., CPU and GPU). Our method demonstrates 4.25x, 49.56x, and 1.76x faster performance in total latency, respectively, indicating that the proposed L-TTA achieves significantly faster training times than existing TTA methods. It is also important to note that since TENT and ЕсоТТA perform the TTA process using a single independent model, their latency tends to depend on memory usage. In contrast, DDA shows considerable CPU usage due the image preprocessing tasks such noise synthesis, which are not handled by the GPU. These CPU-intensive processes can lead to substantial power consumption unless dedicated hardware is available. Therefore, despite lower memory usage, using such methodologies for field applications should be avoided as much as possible.
The proposed stem layer can be conveniently expanded regardless of the task because only the initial parts of the entire network are changed. To demonstrate this, Tab. 2 provides additional results the semantic segmentation task. Even without performing TTA (i.e., Ours w/o GCAL), mloU is improved by an average 18.2% and 0.8% compared with the source model and TENT, respectively. When TTA is applied (i.e., Ours), it improves mIoU by 4.6% compared to EcoTTA with only 0.04% of memory usage.
4.3 Ablation Studies
Interval adjustment the continual scenario. As discussed Subsection 3.4 we propose interval training to satisfy the continuous TTA scenario. Accordingly, we conduct an ablation study to observe the prediction error variations using the ResNet-26 model, as reported in Tab. T In this experiment, adjust the iteration interval for performing one round of the TTA process and observe the resulting

---

prediction error. In Fig. 6, O and Full iter. on the x-axis represent the cases of no TTA and completing I epoch, respectively, with the intermediate values showing results for iteration increments of steps. The results demonstrate that applying GCAL and DEL to the stem layer consistently yields better outcomes, with steady decline in error rate as iterations increase. Furthermore, both methods also outperform REALM, SOTA approach, after 10 iterations. Notably, the proposed L-TTA (=DEL+GCAL) achieves a 5.7% performance improvement after completing epoch.
Evaluation on small batch sizes. By implementing DWT in DEL, we provide foundation for generalization and maximize our capabilities by minimizing individual uncertainties across various frequency domains with spatial properties. We contend that this approach assists data in conveniently adapting independently the target domain. In Fig. 7 we present the results of performing TTA on ImageNet-C for small batch sizes (=1, 2, 4. 8), in conjunction with the representative methods of entropy minimization (i.e., TENT 59 , EATA 45 and SAR
46 The conventional methods exhibit near-complete loss of classification capability, with average prediction error of 99.89% when the batch size one. Meanwhile, our proposed method demonstrates higher performance, with predic- tion error of 75.16%. Additionally, it is visually evident that our method is less sensitive to variations in batch size, with a difference of 8.37% between batch sizes one and eight, whereas TENT, EATA and SAR exhibit instability with differences of 31 1145% 22.67% and 30.86%, respectively. This is interpreted as a result of the significant dependency of the conventional methods on BN layers.
GCAL ONLY
Average Ervor (%6
Average Ervor (%)
REALM
5.7
Iteration
Batch size
Figure 6: Comparison of prediction error (%) across increasing TTA iterations on CIFAR- 10-C with ResNet-26.
Figure 7: Assessment of TTA effectiveness for small batch sizes in comparison with entropy min- imization baselines on ImageNet-C.
5 Conclusion
We propose novel method for lightweight TTA that utilizes the stem layer reconstructed with DEL and GCAL to minimize the uncertainty across multiple frequency domains rather than the entropy. This architecture improves the model's generalization performance without assumptions regarding training and can be applied conveniently and non-invasively to models with a CNN as the backbone. Our approach demonstrates performance equivalent superior those in SOTA studies with well-known robust benchmark datasets, showcasing the fastest and most memory-efficient results suitable for real-world scenarios.
Acknowledgement
This research was partly supported by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program 11--2024-S--2222-015555 supervised by the ITTP(Inttitute for Information & Communications Technology Planning & Evalua- tion) and Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of
10

---

References
[1] Shun-ichi Amari Backpropagation and stochastic gradient descent method. Neurocomputing. 5(4-5):185-196, 1993.
[2] Yiyao An, Ke Zhang, Yi Chai, Qie Liu, and Xinghua Huang. Domain adaptation network base on contrastive learning for bearings fault diagnosis under variable working conditions. Expert Systems with Applications, 212:1 1802, 2023.
[3] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time udaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 0344-8353, 2022.
[4] Oran Brigham. The fast Fourier transform and its applications. Prentice-Han Inc., 1988.
[5] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinyt:: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems, 33:1 1285- 11297, 2020.
[6] Jin Chen, Zhi Gao, Xinxiao Wu, and Jiebo Luo. Meta-causal learning for single domain generalization. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7683-7692, 2023.
[7] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018.
[8] Jiwoong Choi, Dayoung Chun, Hyun Kim, and Hyuk-Jae Lee. Gaussian yolov3: An accurate and fast object detector using localization uncertainty for autonomous driving. Proceedings of the IEEE/CVF International conference computer vision, pages 502-511, 2019.
[9] Jiwoong Choi, Dayoung Chun, Hyuk-Jae Lee, and Hyun Kim. Uncertainty object detector for autonomous driving embedded platforms. 2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), pages 16-20. 2020.
[10] Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne T Kim, Seungryong Kim, and Jaegul Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 11580-11590, 2021.
[11] Sungha Choi, Seunghan Yang, Scokeon Choi, and Sungrack Yun. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. European Conference on Computer Vision, pages 140-458. Springer, 2022.
[12] Dayoung Chun, Seungil Lee, and Hyun Kim. Usd: Uncertainty- based one-phase learning to enhance pseudo-label reliability for semi-supervised object detection. IEEE Transactions on Multimedia, 2024.
[13] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[15] Xinjic Fan, Qifei Wang. Junjie Ke, Feng Yang. Boqing Gong, and Mingyuan Zhou. Adversarially adaptive normalization for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8208 8217, June 2021.
[16) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096-008 2016.
11

---

[17] Jin Gao, Jialing Zhang, Xihui Liu, Trevor Darrell, Evan Shelhamer, and Dequan Wang, Back to the source: Diffusion-driee adaptation test-time corruption. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11786-11796 2023.
[18] Theodoros Georgiou, Yu Liu, Wei Chen, and Michael Lew. A survey of traditional and deep learning-based feature descriptors for high dimensional data in computer vision. International Journal of Multimedia Information Retrieval, 93)):135-170, 2020.
[19] Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Robust continual test-time adaptation against temporal correlation. Advances in Neural Information Processing Systems, 35:27253-27266, 2022.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
[21] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of out-of-distribution generalization. In Proceedings of the IEEF/CVF international conference computer vision, pages 8340-8349, 2021.
[22] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint 1rXiv:1903.12230 2019.
[23] Dan Hendryeks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 15262-15271 2021.
[24] Junyuan Hong, Lingjuan Lyu. Jiayu Zhou, and Michael Spranger. Mecta: Memory economi continual rest-time model adaptation. In 2023 International Conference on Learning Represen- tations, 2023.
[25] Jie Hu, Li Shen, and Gang Sun. Squeeze- -ede--aand-----eieee networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[26] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. pmlr, 2015.
[27] Yusuke Iwasawa and Yutaka Matsuo, Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34:2427-2440, 2021.
[28] Hemalatha Kanagaraj and V Muneeswaran. Image compression using haar discrete wavelet transform. In 2020 5th International Conference on Devices, Circuits and Systems (ICDCS), pages 271-274, IEEE, 2020.
[29] Hyun Kim, Albert No, and Hyuk-Jae Lee. Spiht algorithm with adaptive selection of compres- sion ratio depending on dwt coefficients. IEEE Transactions on Multimedia, 20(12):320-3211, 2018.
[30] Jan Kukačka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxonomy. arXiv preprint arXiv:1710.10686. 2017.
[31] Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, al. Universal source-fres free domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4554-4553, 2020.
[32] Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896. Atlanta, 2013.
[33] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation distribution shifts. arXiv preprint arXiv:220.I1166. 2022.
12

---

[34] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5400-5409, 2018
[35] Lei Li, Ke Gao. Juan Cao, Ziyao Huang, Yepeng Weng. Xiaoyue Mi, Zhengze Yu, Xiaoya Li, and Boyang Xia. Progressive domain expansion network for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 224 233, 2021.
[36] Ming Li, Zheng Ma, Yu Guang Wang, and Xiaosheng Zhuang. Fast haar transforms for graph neural networks. Neural Networks, 128:188- 198, 2020.
[37] Qiufu Li and Linlin Shen. Wavesnet: Wavelet integrated deep networks for image segmentation. In Pattern Recognition and Computer Vision: 5th Chinese Conference, PRCV 2022, Shenzhen, China, November 4-7, 2022, 2022, Proceedings, Part IV, pages 325-337. Springer, 2022.
[38] Qiufu Li, Linlin Shen, Sheng Guo, and Zhihui Lai. Wavecnet: Wavelet integrated cnns to suppress aliasing effect for noise-robust image classification. IEEE Transactions on Image Processing, 30:7074-7089, 2021.
[39] Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. arXiv preprint arXiv:2303.15361 2023.
[40] Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Pellot-Gurlet Taylor Mordan, and Alexandre Alahi. Tit++: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems, 34:21 808-22120, 2021.
[41] Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, friendly vision transformer. arXiv preprint arXiv:2110.02178 2021.
and mobile-
[42] Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' A'mour, Balaji Lakshminarayanan, and Jasper Snock. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
[43] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-I0), pages 807-814, 2010.
[44] Xuan Truong Nguyen, Hyuk-Jae Lee, and Hyun Kim. A low-cost hardware design of 1-d spiht algorithm for video display systems. IEEE Transactions on Consumer Electronics, 64( (441)244-52 2018.
[45] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International confer- ence machine learning, pages 16888-16905. PMLR, 2022.
[46] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. arXiv preprint arXiv:2302. 12400, 2023.
[47] Mostafa Parchami, Saman Bashbaghi, and Eric Granger. Video-based face recognition using en- semble of haar-like deep convolutional neural networks. In 2017 International Joint Conference on Neural Networks (IJCNN), pages 4625-4632. IEEE, 2017.
[48] Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Augoo: augmentation consistency-guided self-training for source-free domain adaptive semantic segmentation. arXiv preprint a0Xiv: 10140, 2021.
[49] Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Proceedings of the IEEF/VVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
13

---

[50] Sriprabha Ramanarayanan, Balamurali Murugesan, Keerthi Ram, and Mohanasankar Sivaprakasam. De-wenn: A deep cascade of wavelet based convolutional neural networks for mr image reconstruction. In 2020 IEFE 17th International Symposium on Biomedical Imaging (ISBI), pages 1069-1073. IEEE, 2020.
[51] K Ramamohan Rao and Ping Yip. Discrete cosine transform: algorithms, advantages, applica- tions. Academic press, 2014.
[52] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems, 33:11539-11551, 2020.
[53] Skyler Seto, Barry-John Theobald, Federico Danieli, Navdeep Jaitly, and Dan Busbridge. Realm: Robust entropy adaptive loss minimization for improved kigle-sample test-time adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2062-2071, 2024.
[54] Jin Shin and Hyun Kim. RI-spiht: Reinforcement learning-based adaptive selection of compres- sion ratios for 1-d spiht algorithm. IEEE Access, 9:82485-82496, 2021.
[55] Junha Song. Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficienn continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11920-11929, 2023.
[56] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 2016
[57] Vladimir Vapnik. Principles of risk minimization for learning theory. Advances neural information processing systems, 4, 1991.
[58] Riccardo Volpi, Hongseok Namkoong, Ozan Sener. John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems, 31, 2018.
[59] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
[60] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[61] Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 8684 8694, 2020.
[62] Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 7201-7211, 2022.
[63] Garrett Wilson and Diane J Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(5):1-46, 2020.
[64] Xiang Xu, Xiong Zhou, Ragav Venkatesan, Gurumurthy Swaminathan, and Orchid Majumder. d-sne: Domain adaptation using stochastic neighborhood embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019.
[65] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35:38629-38642. 2022.
[66] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep image generation for domain generalisation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13025-13032 2020.
14

---

A Appendix: A Detailed Implementation Setting of the Proposed Models
The image classification process includes applying reconstructed stem layer followed by 10 epochs of warm-up training using cross-entropy loss [57]. In both TTA and warm-up settings, the batch size and learning rate set to 128 and 0.05, respectively. For CIFAR-10 and CIFAR-100, the model utilizes weights pre-trained on ImageNet and is trained for 150 epochs, using the same batch size and learning rate configuration as in the ImageNet training setup. The optimizer used is standard SGD W. consistent across all datasets. This indicates the non-necessity for additional search for typerparameters in real fields. The only different parameter is the reduction size of the SE block, which was determined experimentally through ablation studies in Appendix D CIFAR-IO-C and LIFAR-100-C have relatively small number of classes and are assigned a reduction of eight. ImageNet-C is assigned a reduction of 32.
To evaluate the performance of TTA in semantic segmentation, Cityscapes-C was generated by synthesizing four weather corruptions with a severity of out of 15 corruptions in the Cityscapes dataset [ Additionally, for an unbiased comparison with the settings of EсоTTA (55), we
the DeepLabV model [7] with the backbone of ResNet-50 and pre-train this model with the instance selective whitening strategy proposed in RobustNet [TO] To warm up the stem layer, 32,000 iterations are trained with Cityscapes with batch size of 8. For TTA, Cityscapes-C is learned with a batch size of 2, and the performance metrics mIoU and memory usage are compared with those of existing methods.
The source code is available for reproducibility at the GitHub link provided in the abstract. The seed for random number generator is commonly used as 42. However, please note that this does not significantly affect the accuracy of the model. We conducted experiments
15 main corruptions using 10 additional seeds (i.e. 3, 10, 21, 22, 43, 99, 318, 500, 565, 777), along with the previously used seed (42). Each experiment was run for 10 iterations in single batch per seed, resulting in an average accuracy of 64.85% with standard deviation of 0.02. All experiments were conducted on a system equipped with l Xeon Gold 5218R CPU and an NVIDIA Tesla A 1100 80G GPU. However, the minimum GPU memory required to conduct the single experiment was measured to be approximately 12GB.
B Appendix: Operation Process of 2-D Discrete Wavelet Transform
The 2D DWT involves sequentially applying a ID DWT to the rows and columns of the input data. For example, in the case of the Haar wavelet [36), the low and high filter banks are defined as follows:
1 5 V
-E V2
V2
In the first stage, it is formulated in the horizontal direction decomposition follows:
#1
W-1
H-IW-1
Lijj
Σ
Σ (Xem,n
Vt + +2m+11m * 1
Σ
(X2m.n * vo + X2m+1 17)
7= m==
M=0 m=0
(8)
This describes the splitting of single original feature map into two components: the LFC and the HFC. Here, Xm. denotes the pixels of the input image. W and denote the width and height of X, respectively. * denotes the CONV. From CONV perspective, the two filter banks denote the kernel weights, and m and n denote the stride the horizontal and vertical directions, respectively.
In the second stage, a vertical decomposition is performed as the two filters are transposed to operate in Eq. The exception is that each X is replaced by L and H. respectively, and because it is decomposed vertically for L and H, the indices m and n are interchanged in Eq. 8 Because of the decomposition, divided into L.1 and H and H decomposed into H L and H. The LFC (L.L) generally contains visually intuitive information such as the overall image brightness. It exists in the range [0, +2Pmaa]] Herein, Pmaar is the maximum value of image pixels (8 bits 255). This area's 's information represents the basic image structure. It is significantly correlated with the HFCs. Meanwhile, HFCs (i.e,, LH, H L. H) include finer details of the image and a small amount of edge information including information regarding diagonal variations and the image texture or pattern. The range of values is within [-Pman//2 2, + Pmnar/ 2]. These obtained frequency domains have been variously analyzed and applied for image processing purposes [38 50 37).
15

---

C Appendix: Rigorous Assessments of Model Robustness
To demonstrate that the proposed method can adapt not only to the introduced corruptions but also to challenging domain shifts, we conduct experiments three well-known rigorous datasets with challenging themes separated from ImageNet. This experiment is reported as prediction error (%),
same as all experiments. As shown in Tab. 3 it is almost impossible to have classification ability on the test datasets of Imagenet-Skedl have already been pre- pre-trained with the same domain. However, this can be alleviated by applying the proposed DEL structure to ResNet-50, and when TTA is performed in real time with the proposed
[2T] ImageNet-R [60] and ImageNet-A [23] even though they
method, the model accuracy can be improved 2.84%, 40.91% %, and 5.49% for each ImageNet series.
Table 3: Evaluation of the proposed method using ResNet-50 on challenging ImageNet series.
Dataset
Source
Ours w/o GCAL
Ours
SengeNett-Setthhi 21
75.84
73.93
73.00 (-2.84)
ImageNett R 60
99.43
62.77
58.52 (-40.91)
ImateNet-A
X3
99.58
94.35
94.09 (-5.49)
D Appendix: Ablation Studies on Architectural Exploration
In our experiments, we assign a DWT level of two for DEL mentioned in Sec. 3.3 To experimentally demonstrate the advantages ODD, Tab. | we describe the results according to the DWT level on the 10-CR- CIFAR-100-C, and ImageNet-C datasets. To summarize, providing a more varied multi-view through ODD rather than only decomposing one time with DWT yields higher generalization performance and prediction accuracy in TTA. The average improvement is 1.5%.
Table 4: Comparison of TTA results in terms of average prediction errors for source, DWT level 1, and level 2 decompositions using ResNet-50 on CIFAR-10-C, CIFAR- and ImageNet-C.
Table 5: Comparison TTA results in terms of aver- age prediction errors using ResNet-50 on CIFAR- 10-C, CIFAR-100-C and ImageNet-C, focusing on the influence of the SE block's reduction scale.
Method
CIFAR-I0-C CIFAR-100-C ImageNet-C
REDUCTION
CIFAR-IO-C
CIFAR-100-C IMAGENET-C
Source
36.6
53.6
82.0
18.0
37.5
67.4
DWT Level
33.3
52.4
72.9
17.9
39.2
67.2
TIA
20.5
39.3
65.2
DWT Level 2
30.8
50.8
72.6
19.2
38.2
66.5
TTA
18.0
37.5
64,9
32
18.0
39.4
64.9
Tab. 5 presents an ablation study on the reduction size, a hyperparameter of the SE block that constitutes GCAL.. In the three datasets used for the experiments (i.e., CIFAR-10-C, CIFAR-100-C. and ImageNet-C datasets), 8, 4, and display the best results, respectively. However, no significant difference exists except when the reduction size is 16 in CIFAR-10-C. It should be noted that regardless of the reduction size, we exceed the SOTA results. Meanwhile, for large datasets with many classes such ImageNet-C, minimizing the reduction appears to yield linearly better advantages. Therefore, we empirically assign a reduction size of 32 for ImageNet-C.
E Appendix: Further Case Studies
In Sec. we compare our results primarily with studies that show significant improvements in accuracy by performing complicated processes. In Tab. [ we demonstrate that our accuracy is reasonable by presenting the prediction errors on CIFAR- 10-C and ImageNet-C using ResNet-26 and ResNet-50 models, respectively. For a comprehensive performance comparison, including memory usage, Tab. compares our proposal on CIFAR-10- 10-C and CIFAR-100-C using ResNet-50 against existing TTA methodologies. Notably, it shows superiority even methods like CoTTA, which
16

---

Table Comparisons of prediction error (%) with prior TTA methods on ResNet-50.
Method
Gause.
Shot Impul.
Defoe. Glass Motion
Zoom Snow
Frost
Fog
Brit.
Conte. Klastic
Pixel
JPKG
Avg.
Mem. (MB)
CIFAR-10-C
Source
65.6
60.7
74.4
28.9
79.9
46.0
25.7
35.0
49.4
54.7
13.0
83.2
41.2
46.7
27.7
48.8
BN STAT
18.0
17.2
29.3
10.7
27.2
15.5
8.9
16.7
14.6
21.0
9.3
12.7
20.9
12.4
14.8
16.5
91
TENT59
16.6
15.7
25.7
10.0
21.8
13.8
8.1
11.9
13.8
17.5
8.7
10.0
19.1
11.5
13.8
15.0
925
TTT++ 00
18.2
16.9
28.7
10.5
26.5
14.5
8.9
16.5
14.5
20.9
9.0
9.0
20.4
12.3
14.7
16.1
1877
[W
17.3
16.1
26.1
10.6
25.6
14.1
8.7
15.6
13.6
18.6
8.8
10.0
19.3
12.0
14.2
15.4
1971
45
17.2
14.9
23.6
10.2
23.3
13.2
8.5
14.0
12.5
16.5
8.6
9.4
17.2
11.0
12.7
14.2
925
CoTTA 62
16.2
15.0
21.2
10.4
22.8
13.9
8.4
15.1
12.9
19.8
8.6
11.3
17.5
10.5
12.2
14,4
2066
BooTTA (Mem-)
55
16.5
14.5
24.3
9.7
23.7
13.3
8.8
14.7
12.9
17.0
9.1
9.4
17.6
11.4
13.1
14,4
296
Eco.TA (Acc
65
16.6
14.4
23.6
9.8
23.4
12.7
8.6
14.5
12.6
16.5
8.7
9.0
17.0
11.3
12.6
14.1
498
Ours
22.4
18.8
28.3
7.8
30.4
9.9
6.5
8.6
9.3
9.2
4.2
5.6
17.3
11.6
22.1
14.1
6.4
CIFAR-100-C
Source
84.7
83.5
93.3
59.6
92.5
71.9
51.8
66.6
77.6
81.8
44.3
91.2
72.2
76.6
$6.5
73,8
SPA
48.1
46.7
60.6
35.1
58.0
41.8
33.2
47.3
43.5
54,9
33.5
35.3
19.8
38.4
40.8
44.5
91
5ENT
44.1
42.7
53.9
32,6
52.0
37,5
30.5
43.4
40.2
30.4
31.4
45.1
35.0
37.6
40.1
926
TTT+
48.1
46.5
60.8
35.1
57.8
41.6
32.9
46.8
43.3
33.3
34.0
50.0
38.1
40.6
44.3
1876
SWRNSP
48.3
46.5
60.5
35.1
57.9
41.7
43.5
33.5
35.1
49.9
38.3
40.7
44.4
1970
45
44.8
41.9
52.6
33.0
51,1
37.8
30.3
40.1
30.1
45.2
35.2
37.4
40.0
926
CoTTA
43.6
42.8
50.1
34.2
51.6
39.2
31.1
39.6
32.2
43.1
35.8
36.7
40.2
2061
EcoITA (Mem+)
55
44.8
40.3
49.2
32.3
50.1
36.3
39,9
44.6
33.7
45.3
36.3
37.7
39.5
296
EcoITA (Acce)
65
44.9
40.4
48.9
32.7
49.7
36.9
29.3
40.8
39.0
44.4
33.6
44.0
35.7
37.8
39.3
498
Our
52.4
49.9
55.7
25.8
56.8
27.4
23.6
30.7
31.9
18.6
23.5
41.2
318
50.6
36.7
6.4
maintain dual models during training. Additionally, our method improves the average prediction by 2.6% while using 98.71% % less memory compared to EcoTTA, which is designed to use more memory to enhance accuracy on CIFAR-100-C.
In all experiments of this paper, results are presented using ResNet. To address concerns regarding the applicability to other architectures, reconstruct the stem layer of the MobileViT[4[], which combines vision transformer and a CONV structure, using our proposed method to perform TTA. We use MobileViT-XXX: VTT-XXS, the most suitable model for mobile environments, which sets the kernel size to 3 3,3, unlike the vanilla ResNet series. As shown in Tab. 7 we conduct an ablation study for three cases (i.e., Ours w/o GCAL, Ours w/o DEL, and Ours) and observe similar trends to those in Tab T By applying the proposed stem layer, the average accuracy is improved by 15.7%.
Additionally, we discuss two studies similar to our approach and compare performance with source- free DA (SFDA) studies similar to the TTA scenario. SFDA generates learning criteria such as pseudo-labels prototypes to update the target model, requiring dual maintenance of well-trainee model, resulting in higher memory consumption compared to TTA. To address this, T3A [27 and LAME 33 propose classifier adjustment- bejjuttment-based methods similar to approach. However, since they only update the last layer, the learning cost for criterion generation and forwarding is continuously experimental results with similar approaches, representative TTAs, and SFDA studies on both CIFAR-IO-C and CIFAR-100-C using ResNet-50 and ResNet- 18, demonstrating better performance with extremely low memory consumption. It should be noted that the SFDA scenario consumes approximately twice as much memory as TENT 59
required. Tab. X and 9 compare
Table 7: Ablation Study of Prediction Errors (%) the CIFAR-10-C using Mobile
Method
Gauss.
Shot Impul.
Defoc
Glass
Motion Zoom Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
Snurce
78.9
75.4
78.1
11.3
49.3
15.9
8.5
13.7
23.4
17.6
6.1
15.6
20.6
50.6
25.7
32.7
Ours w/o GCAL
78.0
73.6
79.9
10.6
48.9
16.2
8.1
13.4
17.6
6.4
16.0
20.2
24.9
24.9
30.8
Ours w/o DEI.
34.2
29.0
32.6
8.8
33.2
10.6
8.5
12.2
13.9
11.8
6.0
8.7
18.3
14.9
26.7
18.0
8.4
10.7
17

---

Table 8: Comparison of Prediction Errors (%) be- tween the proposed method and high-cost method- ologies on ResNet-50.
Table 9: Comparison of Prediction Errors (%) be- tween the proposed method and high-cost method- ologies on ResNet-
Method
CIEAR-IO-C CIFAR-100-C
Method
CIFAR-10-C CIFAR-100-C
Source
29.15
60.34
Source
42.3
66.6
TENT 59
14.27
40.72
Pseudo-
32
21.6
43.1
T3A 27
15.44
42.72
TENT 59
18.8
40.3
CoTTA 62
14.4
40.2
LAME
44.1
68.8
Ours
14.1
36.7
CoTTA
17.8
44.3
NOTE
17.6
41.0
Ours
15.8
39,5
F
Appendix: Limitations and Discussions
Our proposed lightweight TTA approach reconstructs the stem layer to extract the channel-wise uncertainty and updates the parameters before the BN layer by minimizing the uncertainty. It quickly adapts the imitial representation to be robust to D; by consuming less memory by not learning subsequent layers. However, it can be observed that the stem layer has less impact on improving the robustness of the model as the number of classes increases, as shown in the experimental result of this paper (e.g., Tab. T This is considered curse of dimensionalityy Furthermore, Fig. shows the mean prediction errors of four corruption themes (i.e., noise, blur, weather, and digital) as the number training iterations increases. Here we not apply the interval training strategy proposed in Sec. 3.4 As shown in ImageNet-C. the iteration that yields the best performance for each theme different, and continuously minimizing uncertainty will cause the model to diverge. To prevent this, perform interval training. These observations small number classes like CIFAR 100-C, which converges stably even without interval training. Moreover, described in Tab. demonstrate its practicality by showing sufficient performance improvement autonomous driving scenarios such as semantic segmentation.
not observed in datasets with relatively
To address these challenges, several unexplored potential solutions could be considered. The issue of handling a large number of classes, as identified this study, may be mitigated by applying the channel attention uncertainty minimization strategy not only in the stem layer but also selectively in subsequent layers. Additionally, the overfitting-related regularization problem could tackled by actively filtering out data enhance generalization performance, seen in methods like EATA and RE3LM or by adapting to input data rather than unconditionally minimizing uncertainty. However, such approach should be designed enable the network to autonomously adjust to independent data, aligning with our research philosophy.
ImageNet-C
Mosin Enron
Moan Emaon
Itration
Itarzion
Figure 8: Mean error rates (%) by corruption type on ImageNet-C and CIFAR-100-C with increasing TTA iterations in ResNet-50.
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
It is provided in Sec. 3.1 as well as in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
is provided Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
Since the evidence supporting our proposal is closer to empirical analysis
rather than theoretical results, it is explained along with Fig. (] in Sec. 3.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All major details required for reproduction are reported. (with code in the supplementary materials)
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have created a README file and provide pytorch code as supplementary materials. Additionally, we provide shared links to well-known repositories for downloading pre-trained weight files.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
Suuidess/odeSuumissiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and basclines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include all the important information needed for experiments. (see the Sec. and Appendix A))
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We compare the performance of commonly proposed methods and various studies using prediction error (%). Additionally, Appendix F explains fixed intervals based statistical evidence.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Our proposal is memory minimization technique in TTA, which is shown with the Tab. 6 and Fig- We also report experimental environment (i.e,, CPU and GPU) and minimum required amount memory in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect. with the NeurIPS Code of Ethics
Answer: [Yes]
Note that our code can used under normal circumstances without special consideration.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We note that our paper does not specifically consider broader impacts unrelated to security social issues.
22

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic a algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
Note that our paper does not specifically consider safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We utilize code published by as well as code from publicly accessible repositories like GitHub, and we explicitly acknowledge this usage.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Note that our paper does not need to consider new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Note that our paper does not need to consider Crowdsourcing and Research with Human Subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: In our paper, we note that there are no potential factors that could be significant enough require notification to the IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "udg1zfBm9O", "reviewer": "Reviewer_H2Kq", "summary": "The paper focuses on reducing the memory usage of test time adaptation (TTA) by remodelling the first layer. The authors apply discrete wavelet transform to input features and use squeeze and excitation blocks to get the per-channel uncertainty. The proposed loss function minimizes the -per-channel uncertainty. The memory usage of the proposed method is significantly less than methods that require full forward and backward passes of the backbone.", "questions": "Please check the weakness.", "limitations": "The authors should address the limitations more.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses an practical problem for TTA.\n- The proposed method significantly reduces the memory consumption.", "weaknesses": "- **Gaussian Channel Attention Layer**\n    - The authors should explicitly show the loss function for pre-training and TTA for clarity. During pre-training, is equation 4 or 5 (the uncertainty loss) added to the pre-training loss?\n    - The authors should dedicate more effort to explaining why minimizing per-channel uncertainty can help TTA, which is more important than just the method itself.\n    - I can’t understand why the mean is set to 1 and why setting the mean to one can make SE block an uncertainty extractor.\n    - It seems this approach has nothing to do with attention. Could the author elaborate on the naming?\n- Unclarity of presentation\n    - Where is DEL in Figure 2? Both the caption of Figure 2 and section 3.3 mentioned DEL in Figure 2.\n    - In the caption of Figure 2, it mentions *. Where is it?\n    - The authors should briefly explain the psi_k  and LL/LN/HL/HH  in caption 2.\n    - “We jointly train it to reduce task-specific prediction errors using suitable loss terms (*i.e.*, cross-entropy) in conjunction with the uncertainty extracted from GCAL minimized through NLL loss. “ Only cross-entropy is used or equation 4/5 is added as well?\n    - During TTA, only equation 5 is minimized?\n    - Equation 6, what’s the W tilde? The authors need to define variables before using them.\n    - What information does figure 3(b) try to convey? What does the high and low uncertainty difference mean for TTA?\n    - “the absence of a method to ensure data independence”. What does the data independence mean here？\n- Since this paper is about efficient TTA, except for memory usage, it should also report, training time and flops.\n- Grammar\n    - line37:  “the possibility of adaptation when trained to minimize the entropy of predictions”  missing subject.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review on our manuscript. Please see our detailed responses to your comments outlined below.\n\n**Weakness-1.**\n\n>- Yes, please see lines 230-231 of the manuscript.\n“*Eq. 4 means minimizing uncertainty for all channels, which applies equally to pretraining and TTA processes.*”\n>- Our proposed GCAL serves to extract uncertainty (see Eqs. 2 and 3) instead of channel attention, which can be obtained by exploiting the squeeze and excitation layer (SE layer).\nAs described in the manuscript, learning the extracted channel attention ($=\\mu$) to always converge to 1 is equivalent to not using channel attention.\nInstead, since we can maintain the same $\\mu$ in the source and target domains, we can identify gaps in the variance ($=uncertainty$) and minimize them through NLL loss.\nThis is demonstrated in *Figure 3 (b)* and *Eq.6.*\n*Note that Figure 3 (b) is an $8\\times8$ heatmap of the uncertainty of the intermediate feature, which is 64 ($8\\times8$) channels.*\n>- As mentioned above, the channel attention ($=importance$) extracted by the squeeze and excitation (SE) layer is disabled by fixing it to 1. The SE layer learns as it trains and obtains the ability to determine the channel attention of intermediate features, but we want to change its role to extract uncertainty. \n>- It refers to the channel attention ($=importance$) extracted by the Squeeze and Excitation Layers, not the concept of attention in the Self-attention mechanism. \n\n**Weakness-2.**\n\n>- The structure of DEL encapsulates the GCAL and CONV layers with DWT and IDWT. In Figure 2, you can see that the DWT and IDWT encapsulate the GCAL and CONV layers. \n>- In Figure 2, you can find the '$*$' located with $\\psi_k$ within the yellow box (bottom right) describing DWT and IDWT. \n>- Owing to the limited number of pages in the manuscript, the explanation of these notations is inevitably shortened, and we instead explain in the caption that they can be found in *Appendix B*.\n>- Cross Entropy and Eq.5 are trained jointly while pretraining, and only Eq.5 is used while performing TTA. \n>- Yes.\n>- $W$ is the weight for the corresponding function $F_{se}$. It is used the same as in *Equation 1*. \n>- The four $8\\times8$ heatmaps show the difference in attention of 64 channels of intermediate features for the original domain (Source) and corruption (Target) for the same image. In the observations before TTA (left), we can see that LFC has a fundamentally large difference and HFC has almost no difference. However, in the post-TTA observations (right), we observe that LFC has nearly no difference from before TTA and HFC has a very large difference. This significant change in observations equates to a significant change in HFC's uncertainty about the target domain by performing TTA. This proves the claim that HFC can help the generalization process and provides more confidence in the uncertainty minimization process.\n>- Data independence means that no matter what data from the target domain is input, TTA can be performed with reduced uncertainty. We present this as a contribution because existing TTA studies have used strategies to minimize entropy by filtering only data with low entropy. \n\n**Weakness-3.**\n\n>- To measure training time and FLOPs fairly, we record the average of five runs of each methodology using the profiler officially provided by the Pytorch framework used in our experiments. As shown in Table A, compared to TENT, training time and KFLOPs are improved by $4.26\\times (CPU: 4.18\\times \\/ GPU: 11\\times)$ and $11428\\times$, respectively.\n\n**Table A**\n>| Method | CPU (ms) | GPU (ms) | KFLOPs |\n|:------:|:--------:|----------|:------:|\n| TENT   |  1519.4  |  40.965  |  81922 |\n| Ours   |  362.83  |  3.7162  |  7.168 |\n\n**Weakness-4.**\n\n>- We reorganized the sentence with “The possibility of adaptation when the models are trained to minimize the entropy of predictions.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nCw2LV9hj9", "reviewer": "Reviewer_ah9h", "summary": "In this paper, the authors enhance the efficiency of test-time adaptation (TTA) without necessitating forward/backward passes of the main model. To this end, they introduce a domain embedding layer consisting of a two-level discrete wavelet transformation, which extracts meaningful components in different frequencies. Then, based on the extracted frequency components, they estimate the channel-wise uncertainty with a squeeze and excitation module, which is minimized during testing for TTA. Experiments on benchmark datasets of corrupted images demonstrate the effectiveness of the proposed method. However, some concerns require further addressing. My detailed comments are as follows.", "questions": "5.\tDiscussions with the related TTA methods can be enhanced, including [2-3] for test-time uncertainty reduction, and [4] for backpropagation-free TTA. It’s recommended that the authors include the discussions to ensure a more comprehensive study.\n6.\tIt appears counter-intuitive that BN_STAT [C] requires more memory than the proposed method in Figure 4. Since online TTA performs both adaptation and inference on the current test samples before processing the incoming ones, the cost of forward passes can be ignored when viewed as a fundamental procedure in standard inference.\n7.\tThe result of the proposed method under a small batch size is unpersuasive. From Figure 5, L-TTA with a batch size of 4 stills deteriorates compared with standard inference, which achieves an error rate of 82.0%.\n8.\tIt’s recommended to include comparisons on ViT-Base to verify the effectiveness of the proposed method on larger networks.\n\n[A] Tent: Fully test-time adaptation by entropy minimization, ICLR 2021.\n\n[B] Surgical fine-tuning improves adaptation to distribution shifts, arXiv 2022.\n\n[C] Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift, arXiv 2020.\n\n[1] FEATHER: Lifelong Test-Time Adaptation with Lightweight Adapters, arXiv 2024.\n\n[2] Test-time Adaptation for Machine Translation Evaluation by Uncertainty Minimization, ACL 2023.\n\n[3] Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting, arXiv 2024.\n\n[4] Test-Time Model Adaptation with Only Forward Passes, ICML 2024.", "limitations": "The authors discussed and there are no significant limitations.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThis paper introduces a lightweight TTA solution that only updates the first stem layer without necessitating the forward/backward passes of the main model. This broadens the applicability of TTA in practice, e.g., on resource-limited devices or in latency-sensitive scenarios.\n2.\tThe two-level discrete wavelet transformation provides information from multi-views without changing the shape of output features. This benefits its integration into existing models.\n3.\tExperiments on benchmark datasets of corrupted images demonstrate the effectiveness of the proposed method.", "weaknesses": "1.\tThe proposed method necessitates altering model training for warm-up initialization, which incorporates dependency on the source dataset. This reliance restricts its applicability to the fully test-time adaptation setting, where source data is unavailable for data privacy consideration [A]. Moreover, this may make the model drift toward the source samples as discussed in [1]. \n2.\tThe performance of the proposed methods may be hindered due to the neglect of the coupling between the shallow model and the deeper model. The authors solely update the stem layer with the local uncertainty in a gradient-isolated manner. However, such optimization does not guarantee the enhancement of the deeper model, and may easily lead to overfitting in a challenging dataset as evidenced in Figure 6. In essence, L-TTA may serve as an image augmentation technique, which enhances meaningful image features without considering the main model.\n3.\tI’m concerned that the stem layer does not provide sufficient learning capacity. From SFT [B], shallow layers are ineffective in handling feature-level and output-level distribution shifts. As shown in Table 3, the proposed method offers a marginal improvement on ImageNet-Sketch and ImageNet-A with TTA.\n4.\tThe experiment results require careful review. For example, in Table 1, the performance of TENT [A] and EATA[B] is significantly lower than the results reported in the original paper. In Table 3, the performance of the source model on ImageNet-R is also inconsistent with the value reported in [B]. It’s advisable to provide a detailed configuration of hyperparameters for the compared methods in the appendix to ensure comparison fairness.\n\n<=======================after rebuttal========================>\n\nI appreciate the authors’ response, and most of my concerns have been addressed. Here, I still have the following suggestions:\n\n1.\tConduct ablation experiments to demonstrate the impact of the warm-up strategy and how the number of source samples used for warm-up affects the method’s performance.\n\n2.\tClearly discuss the application scope of the proposed method and the risk that the local uncertainty optimization might lead to an overfitted trivial solution, along with potential solutions to mitigate this issue.\n\nOverall, I believe this paper introduces a promising new paradigm for addressing super lightweight TTA, which is in high demand in real-world applications, and the proposed local uncertainty minimization also offers new insights to the TTA community. By carefully reading other reviewer’s comments and the rebuttal, considering the paper’s current scores and it has addressed the concerns of Reviewer sFE2 and iLad, I increase my rating accordingly for a kind support. I highly encourage the authors to consider my suggestions above and include more discussions with recent works such as [1-4] if the paper gets accepted.\n\nLastly, please consider releasing the code if the paper is accepted, and I would like to try it out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and thoughtful comments on our work. We are grateful for your expertise and feedback with this much insight. :) We joyfully respond to each comment as shown below.\n\n**Weakness-1.**\n>- TENT also uses pre-trained weights; indeed, our methodology can be trained from scratch instead of warming up. The warm-up is just a time-saving trick. (Naturally, note that the warm-up also uses the same source dataset used in the pre-training phase).\n\n**Weakness-2. & Weakness-3.**\n>- As the concerns written in Weaknesses 2 and 3, L-TTA performs the optimization on a shallow layer, which is a limitation for challenging datasets with a large number of classes. Our research satisfies the needs of mobile environments, where memory usage due to training must be drastically minimized while still achieving results that can show acceptable accuracy. \n\n**Weakness 4.**\n>- In Table 1, TENT and EATA are performed under the same training conditions as REALM for a fair comparison with REALM, which is reported as the state-of-the-art performance in this manuscript.\n>- In Table 3, we show the results of fine-tuning the pre-trained model (ImageNet-1K) provided by the Pytorch framework with the same training conditions as TTA on ImageNet-R. However, we note that the error shown by EATA is quite low and needs further verification with the official code provided by EATA.\n>- To address the concerns in this comment, we can provide a detailed configuration in the appendix.\n\n**Question 1.**\n>- [A] focuses on language models, and the uncertainty described there is regarded as the same term as entropy in the entropy minimalization shown in the manuscript. In other words, uncertainty is estimated as a consequence of the classification result, and Monte Carlo Dropout is used for this purpose.\n>- The uncertainty expressed in [B] focuses on calibration to improve the performance of the Fisher regularization proposed in EATA by reducing the uncertainty about the data through the whole network and sub-networks. Here, uncertainty is estimated as the discrepancy in the predicted labels between networks.\n>- [C] is introduced as a methodology that assumes resource-constrained environments such as FPGAs and non-configurable parameters and is limited to vision transformer models. It also proposes an approach that focuses on the embedding layer similar to the proposed L-TTA, but adds an activation shifting module instead of learning to optimize the embedded token to the current model parameter.\n\n>- Reference\n    + [A] Test-time Adaptation for Machine Translation Evaluation by Uncertainty Minimization, *ACL 2023.*\n    + [B] Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting, *arXiv 2024.*\n    + [C] Test-Time Model Adaptation with Only Forward Passes, *ICML 2024.*\n\n**Question 2.**\n>- In our paper, even the forward pass is only performed on the stem layer, so for a more rigorous comparison, Fig. 4 shows the measured results of the overall memory usage, assuming the model is up in memory. \n\n**Question 3.**\n>- The y-axis shown in Figure 5 has a typo: it is now written as $[100 - 95 - 90 - 85 - 95 - 90 - 85]$, which is correct to be written as $[100 - 95 - 90 - 85 - 80 - 75 - 70]$.\nTherefore, the results for batch size 4 in Table 4 show a better result than 82.0%. We provide the table as shown below to provide accurate data.\n| Method | Batch Size (1) | Batch Size (2) | Batch Size (4) | Batch Size (8) |\n|:------:|:--------------:|:--------------:|:--------------:|:--------------:|\n| TENT| 99.89| 93.25| 79.82| 68.43|\n| EATA| 99.86| 99.52| 95.25| 77.19|\n| Ours|**75.16**|**72.14**|**68.75**|**66.79**|\n\n**Question 4.**\n>- We additionally provide experimental results for CIFAR10-C/CIFAR100-C on the Swin Transformer-Base model[D].\nAs can be seen in the table below, we can achieve $1.95\\times$ and $2.95\\times$ better results on each dataset by applying our proposed approach.\n>- **CIFAR10-C**\n>      - |     Method    | Gauss. |  Shot | Impul. | Defoc. | Glass | Motion | Zoom | Snow | Frost |  Fog | Brit. | Contr. | Elastic | Pixel |  JPEG |  Avg. |\n|:-------------:|:------:|:-----:|:------:|:------:|:-----:|:------:|:----:|:----:|:-----:|:----:|:-----:|:------:|:-------:|:-----:|:-----:|:-----:|\n|Baseline| 42.44  | 37.92 | 43.64  | 4.61   | 23.49 | 7.90   | 3.31 |**4.09**| 7.55  |**7.15**|**1.76**|**4.56**|9.46| 13.68 | 14.29 | 15.06 |\n|Ours|**33.16**|**29.30**|**40.64**|**4.38**|**20.46**|**7.00**|**2.97**| 4.28 |**5.99**|8.57|1.93|5.01|**9.20**|**9.91**|**13.79**| **13.11** |\n>- **CIFAR100-C**\n>      - | Method | Gauss. | Shot | Impul. | Defoc. | Glass | Motion | Zoom | Snow | Frost | Fog | Brit. | Contr. | Elastic | Pixel | JPEG | Avg. |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Baseline | 74.88 | 72.39 | 73.99 | 18.39 | 63.04 | 25.52 | 16.09 | 20.95 | 27.91 | **27.04** | **10.92** | **24.69** | 30.28 | 37.93 | 40.03 | 37.60 |\n| Ours | **65.97** | **62.67** | **72.75** | **17.79** | **53.00** | **23.98** | **15.31** | **20.89** | **25.10** | 30.40 | 12.18 | 27.51 | **28.11** | **25.94** | **38.15** | **34.65** |\n>- Reference\n>      - [D] Swin transformer: Hierarchical vision transformer using shifted windows, *2021 CVPR.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pmasmbea76", "reviewer": "Reviewer_sFE2", "summary": "This paper proposes a novel test-time adaptation (TTA) method, L-TTA, that minimizes uncertainty instead of entropy. The method involves remodeling the stem layer of the network to minimize uncertainty, which significantly reduces memory overhead and enables rapid adaptation to the target domain. The stem layer applies a discrete wavelet transform to the input features to extract multi-frequency domains and minimize their individual uncertainties.\nThe paper presents a thorough evaluation across various tasks, demonstrating competitive results and significant reductions in memory overhead.", "questions": "1. The authors seem to provide memory usage results only for the ResNet-50 backbone. This raises a concern about whether the memory usage performance of L-TTA is consistent across different backbones.\n1. The costs of network training and inference are divided into spatial and temporal dimensions. Usually, there is a trade-off between memory usage and inference time. The authors claim to have achieved the fastest training, but no relevant experimental results are provided. If the authors still claim that L-TTA has advantages in training speed, experimental results should be provided.\n1. As a suggestion for future work, the authors could consider modifying the initial layers of the network to achieve different trade-offs between memory usage and accuracy.\n1. Parts (b) and (c) are missing in Fig. 2.\n1. In Eq. 2, $p(\\gamma_\\mu, \\gamma_\\Sigma; \\mu)$ denotes $\\mu$ as a parameter, which is unconventional. It is suggested to change it to $p(\\mu; \\gamma_\\mu, \\gamma_\\Sigma)$.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The idea is novel and interesting, with only the stem layer participating in the TTA process.\n1. The paper provides a comprehensive evaluation of different tasks.\n1. Existing experiments show competitive results, with significant reductions in memory overhead.", "weaknesses": "1. The method section of the paper is confusing, particularly the core components GCAL and DEL.\n   1. Why set the GT of $\\gamma_\\mu$ in GCAL to the maximum value of the sigmoid function (= 1)? This seems to encourage the SE block to output the scale parameter as 1, thus producing a trivial solution that keeps the input unchanged.\n   1. Moreover, the description of DEL is unclear. The authors state that DEL encapsulates GCAL and CONV layers with DWT and IDWT layers, but specific operators are not provided.\n\n1. The organization of the experimental section is also somewhat confusing.\n   1. Table 1 lacks the results for CIFAR100 and the memory usage which is the core contribution of the paper.\n   1. Table 6 presents memory usage results on CIFAR10/100, but the network structure is altered to ResNet-50.\n   1. Figure 4 also presents memory usage results on CIFAR100/ImageNet, but only ResNet-50 results are shown in a bar chart.\n1. There remains a gap between L-TTA and the baseline on ImageNet-C in Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review on our manuscript. Please see our detailed responses to your comments outlined below. Please see our detailed responses to your comments outlined below.\n\n**Weakness 1.**\n>1. GCAL is designed based on the squeeze and excitation layer (SE Layer), where the output of a sigmoid function is defined as the channel attention ($=\\mu$) of an intermediate feature. In L-TTA, learning to fix the output at $1$ means not using this feature.\nInstead, it is designed to simultaneously extract the variance ($=uncertainty$) and minimize it with *Eq.4/Eq.5*. The idea is to pre-minimize the uncertainty for the source domain when performing the pre-training and then to obtain the difference in uncertainty under the same channel attention when the target domain comes in at test time.\n>2. For simplicity, we define DEL as the structure in which DWT and IDWT encapsulate the current stem layer ($=7\\times7$ CONV layer) and our designed GCAL. DWT and IDWT are not unique operations that we propose, and we illustrate them in the bottom right corner of *Figure 2*. The specific operations are described in detail in *Appendix B* with *Eqs. 7 and 8*. \n\n**Weakness 2.**\n>- **(1 & 2)**: While it would be nice to include comprehensive results to avoid confusion, as in Weakness 2-1 and 2-2, we focus on a fair accuracy comparison with the state-of-the-art performance (i.e., REALM) in Table 1 to show the trade-offs (ResNet26-CIFAR10 and ResNet50-ImageNet) effectively. In Figure 4, on the other hand, we have organized the manuscript to simply compare memory usage.\nTo take these concerns into account, we simultaneously compare accuracy and memory usage for CIFAR10-C/CIFAR100-C on ResNet50, the same model as ImageNet in Table 1, for comparison with EcoTTA in Table 6.\nWe may consider adding such a clarification to the manuscript.\n>- **(3)**: In the manuscript, we show the accuracy and memory usage by using ResNet26 and ResNet50 backbone.\nResNet26 and ResNet50 both use the same Stem layer ($=7\\times7$ CONV layer), so the memory usage depending on the model parameters is the same, and the only reason the values differ is based on the difference in batch size.\n\n**Weakness 3.**\n>- In this study, we propose that Ours' performance is significant when considering the trade-off between memory usage and accuracy at test time.\n>- For example, ‘Ours’ performs **2.7%** worse than REALM in terms of accuracy but **98.3%** better in terms of memory (see *Figure 4*).\n\n**Qeustion 1.**\n>- We reconstruct and train only the stem layer ($=7\\times7$ CONV layer) for TTA, so the same memory usage is measured on ResNet Series (Resnet-26, 101, 152, etc.) and advanced architectures such as ResNext[A], which have the same kernel size as the corresponding stem layer. \n\n>- **Reference**\n>      - [A] Aggregated residual transformations for deep neural networks, *2017 CVPR.*\n\n**Qeustion 2.**\n>- To measure training time and FLOPs fairly, we record the average of five runs of each methodology using the profiler officially provided by the Pytorch framework used in our experiments. As shown in Table A, compared to TENT, training time and KFLOPs are improved by $4.26\\times$ (**CPU**: $4.18\\times$ / **GPU**:$11\\times$) and $11428\\times$, respectively.\n> | Method | CPU (ms) | GPU (ms) | KFLOPs |\n|:------:|:--------:|----------|:------:|\n|TENT|1519.4|40.965|81922|\n|Ours|362.83|3.7162|7.168|\n\n**Qeustion 3.**\n>- In the proposed architecture, the only parameter that can be considered for trade-off is the **reduction size** of the SE Block. For this purpose, we perform an ablation study in *Appendix D*. (See the table below)\n> | Reduction | CIFAR-10-C |  CIFAR-100-C | ImageNet-C |\n|:---:|:---:|:---:|:---:|\n| 4 | 18.0 | **37.5** | 67.4 |\n| 8 | **17.9** | 39.2 | 67.2 |\n| 16 | 19.2 | 38.2 | 66.5 |\n| 32 | 18.0 | 39.4 | **64.9** |\n\n**Qeustion 4. & Qeustion 5.**\n>- It’s our mistake. We accept the reviewer's comments and will revise the manuscript to address Questions 4 and 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "otTFi78P9O", "reviewer": "Reviewer_iLad", "summary": "This paper introduces L-TTA, a novel lightweight approach to test-time adaptation (TTA) that focuses on the stem layer of deep neural networks. The method incorporates a Domain Embedding Layer (DEL) using discrete wavelet transforms and a Gaussian Channel Attention Layer (GCAL) to minimize stem-layer uncertainty rather than full-network entropy. By updating only the stem layer during adaptation, L-TTA significantly reduces memory and computational requirements compared to existing TTA methods.", "questions": "1. Please discuss more memory-efficient TTA methods such as MECTA [a].\n1. Could the authors elaborate more on the sentence below from Section 3.3? Specifically:\n> In recent TTA scenarios, the entropy minimization model is traditionally learned by filtering out data that is evaluated to have low entropy. However, contrary to this trend, [61] experimentally shows that HFC derived from input images helps generalize the model. This means that high entropy actually contributes significantly to improving prediction accuracy.\n   - How are high-frequency components (HFC) related to entropy in TTA?\n   - What evidence supports the claim that \"high entropy contributes significantly to improving prediction accuracy\"?\n   - How does this observation inform the design of L-TTA?\n1. Given that L-TTA focuses on modifying the stem layer, how might this approach apply to test-time domain generalization scenarios that often require consideration of high-level information?\n\n\n\n[a] Hong, Junyuan, et al. \"Mecta: Memory-economic continual test-time model adaptation.\" 2023 International Conference on Learning Representations. 2023.", "limitations": "1. The proposed L-TTA appears to be primarily designed for vision applications. We would appreciate a discussion of broader applications.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The paper proposes a fundamentally new way to perform TTA, focusing on the stem layer and uncertainty minimization rather than entropy minimization across the full model. This enables the real-world application of lightweight TTA.\n1. L-TTA achieves competitive performance on standard TTA benchmarks while using orders of magnitude less memory than prior methods.", "weaknesses": "1. **Limitations in continual adaptation from frequent resets**: The method relies on frequent resets (every 10 iterations) to maintain performance, especially for complex datasets like ImageNet-C. This frequent resetting could leverage the benefits of intensive warm-up rather than demonstrating true adaptation capabilities from GCAL and DEL. This raises questions about the method's stability and effectiveness for continual adaptation. A deeper analysis of different reset intervals and alternative stabilization techniques is needed to fully understand and potentially improve the method's capabilities for continual adaptation.\n1. **Lack of statistical significance**. The paper does not report any error bars for the experimental results. It is straightforward to understand that the experiments are only run on a single seed (42).\n1. **Incomplete and inconsistent experimental details**:\n   - CIFAR100-C results are excluded from the main experiment table.\n   - The comparison with EcoTTA, a relevant baseline for lightweight TTA, is limited to only the Cityscapes-C dataset.\n   - Only four weather corruptions were used in Cityscapes-C without any justifications.\n   - The small batch size experiments in Figure 5 lack comparison with SAR, a method specifically designed to address small batch size issues in TTA.\n\n*Important*: Although the methodology is solid and brings a significant drive to lightweight TTA, I rate it a weak reject due to the abovementioned weaknesses. I would increase the score if the rebuttal addresses the issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the depth of insight you have provided. We have carefully considered your review and have provided our detailed responses to each of your comments below. :)\n\n**Weakness 1.**\n>- Our proposal aims to adapt to an extremely constrained environment, *i.e.,* by backward only in the stem layer with minimal memory. In Appendix F: Limitations and Discussions, we acknowledge the limitations of continuously processing datasets with many classes, such as ImageNet-C, but we show that for small numbers of 100 or less, **such as Cityscapes-C, CIFAR10-C, and CIFAR100-C, the effectiveness of TTA can be maintained reliably without resetting. Moreover, it performs better than 10 iterations.**\n\n**Weakness 2.**\n>- We perform the experiments as shown in Table C for the 15 main corruptions by applying 10 additional seeds (3, 10, 21, 22, 43, 99, 318, 500, 565, 777) apart from the single seed (42) used in the experiment.\n>- We report a mean of 64.85% (64.9% in Table 1) with a standard deviation (std) of 0.02.\n>- **Table C**\n>| Corruption | S_3 | S_10 | S_21 | S_22) | S_43 | S_99 | S_318 | S_500 | S_565 | S_777 | std. | Avg. (Err.) |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Gaussian_noise|78.28|78.33|78.32|78.34|78.23|78.27|78.37|78.29|78.30|78.30|0.04|78.30|\n|Shot_noise|77.82|77.86|77.81|77.78|77.75|77.85|77.82|77.76|77.82|77.84|0.04|77.81|\n|Impulse_noise|81.31|81.32|81.32|81.39|81.35|81.38|81.37|81.41|81.29|81.28|0.04|81.34|\n|defocus_blur|75.26|75.31|75.19|75.28|75.21|75.20|75.19|75.23|75.17|75.14|0.05|75.22|\n|glass_blur|81.87|81.77|81.82|81.88|81.80|81.79|81.83|81.81|81.77|81.76|0.04|81.81|\n|motion_blur|73.08|73.01|72.98|73.03|72.94|73.04|72.90|73.02|73.00|72.97|0.05|73.00|\n|zoom_blur|64.25|64.17|64.29|64.32|64.26|64.29|64.20|64.31|64.22|64.26|0.05|64.26|\n|snow|68.98|68.96|68.94|68.88|68.91|68.90|68.86|68.97|68.97|68.99|0.04|68.94|\n|frost|58.68|58.64|58.70|58.64|58.66|58.51|58.70|58.67|58.69|58.66|0.06|58.65|\n|fog|54.57|54.57|54.51|54.56|54.54|54.43|54.50|54.50|54.56|54.51|0.04|54.52|\n|brightness|33.36|33.34|33.35|33.31|33.36|33.34|33.39|33.34|33.38|33.36|0.02|33.35|\n|contrast|73.36|73.31|73.29|73.49|73.28|73.35|73.38|73.34|73.40|73.28|0.07|73.35|\n|elastic_transform|58.20|58.39|58.28|58.24|58.30|58.25|58.18|58.32|58.18|58.26|0.07|58.26|\n|pixelate|40.16|40.15|40.11|40.20|40.21|40.09|40.12|40.13|40.14|40.12|0.04|40.14|\n|jpeg_compression|53.79|53.86|53.84|53.84|53.94|53.86|53.75|53.84|53.85|53.72|0.06|53.83|\n|Avg. (total corr.)|64.86|64.87|64.85|64.88|64.85|64.84|64.84|64.86|64.85|64.83|0.02|64.85|\n\n**Weakness 3.**\n>- **(1 & 2):** we show the experimental results, including CIFAR10-C and CIFAR100-C on ResNet50 with EcoTTA, as shown in Table 6 (see Appendix E). The experimental results are as follows. In this experimental environment, we achieve state-of-the-art performance in terms of both accuracy and memory usage.\n>- **3:** This TTA study covers the trade-off between model prediction accuracy and memory usage. For fair comparison with the state-of-the-art research (i.e., EcoTTA) on the autonomous driving setting, we only use four weather corruptions. \n>- **4:** In Table E below, we evaluate the four methodologies, including SAR, on the same ResNet50 with roughly the same parameters. SAR is experimented with using the official code on GitHub uploaded by the authors, using the provided dataset, and set to 'resnet50_bn_pytorch' mode by changing only the batch size. \n>- **Table E**\n> | Method | batch size (1) | batch size  (2) | batch size  (4) | batch size  (8) |\n|:---:|:---:|:---:|:---:|:---:|\n|TENT|99.89|93.25|79.82|68.43|\n|EATA|99.86|99.52|95.25|77.19|\n|SAR|99.86|93.28|81.20|69.00|\n|Ours|75.16|72.14|68.75|66.79|\n\n**Question 1.**\n>- **1:** MECTA has a similar training flow to EcoTTA, which involves forwarding all layers of the model and proposes a special Normalization layer to minimize memory usage (represented by cache). However, it relies on methodologies such as TENT and EATA to perform TTA. We solve the fundamental problem by performing forwarding only on the proposed and reconstructed stem layer as an independent TTA methodology and show compliant memory usage as shown in Table B. \n>- **Table B**\n> | Method | CIFAR10 | CIFAR100 | ImageNet-C |\n|:---:|:---:|:---:|:---:|\n|MECTA|130|130|397|\n|EcoTTA|296|296|438|\n|Ours|**6.4**|**6.4**|**26**|\n\n**Question 2.**\n>- **2-1:** In TTA, entropy refers to the confidence in the classification result of an input image. For example, when represented as a graph, high entropy indicates low confidence, which is the shape of a smooth graph. In [A], HFCs and LFCs are extracted from an image and then inferred separately, where HFCs are correct and take high entropy, but LFCs are incorrect and take low entropy. \n>- **2-2:** In [A], it is experimentally shown that noise data (HFC), which is at a sufficient quality that the object cannot be identified through the Fourier transform, has better accuracy than object data (LFC) extracted from the same image. \n>- **2-3:** In this work, we discuss the differences between the conventional TTA's arguments and those of [A] and extract the channel attention of the intermediate features obtained by decomposing the input into LFC and HFC with DWT into GCAL. As shown in Fig. 3(b), we show that the variation of the channel attention of HFC over LFC after performing the TTA is higher than that of L-TTA, indicating that the design basis of the latter argument is the same.\n\n>- **Reference**\n>- [A] High-frequency component helps explain the generalization of convolutional neural networks, *2020 CVPR.*\n\n**Question 3.**\n>- Domain Generalization uses multiple source domains for training and employs networks to generate adversarial domains. L-TTA's uncertainty minimization and frequency decomposition approaches are believed to enhance high-level information with these generative models. Table 1 shows that \"Ours w/o GCAL\" results indicate model generalization be feasible without TTA.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are thankful to the reviewers for their thoughtful reviews. Our efforts to respond to your questions and comments have led us to identify improvements to be made in the manuscript, which we have incorporated in the final manuscript. 

We have endeavored to provide detailed comment-by-comment responses to each reviewer's official review. Due to the lack of available space in individual responses, we refer to the Appendix for some of our answers.

Furthermore, to briefly summarize our paper once again, we focus on the utilization of the reconstructed stem layer for minimal effort to make the network robust in mobile environments. Therefore, in our experimental results, we focus the most on memory usage compared to prior works and aim to show acceptable prediction performance.

**Common issues**

- To simply summarize the question about the test time adaptation (TTA) process, the proposed TTA minimizes the uncertainty extracted from the gaussian channel attention layer (GCAL) only with Eq. 4 and Eq. 5.
    - Eq. 4: TTA without DEL / Eq. 5: TTA with DEL
    - The mean ($=\mu$) in the manuscript refers to the channel attention of the intermediate feature.
    - Fixing it to 1 is intended to measure uncertainty at the same attention without distinguishing between source and target domains.
- In response to requests for clarification on limitations and scalability-related work, we provide this through further experimentation.
- Two reviewers asked for a discussion of similar papers and we provide a response.
- We only recorded about memory usage, but there are comments about training time and computation, so we provide the measurements in a table.
    - We have only covered memory usage because it's the most critical metric that consistently impacts power consumption in edge devices.
- We have been receiving requests about the statistical significance (i.e. error bars) of our results, so we have performed additional experiments and report them in a table.

For additional comments, please use the 'Official Comment' button and we will respond as soon as possible.

Meta-review: This paper proposes a novel test-time adaptation (TTA) method, L-TTA, that minimizes uncertainty instead of entropy. The method involves remodeling the stem layer of the network to minimize uncertainty, which significantly reduces memory overhead and enables rapid adaptation to the target domain. The stem layer applies a discrete wavelet transform to the input features to extract multi-frequency domains and minimize individual uncertainties. The paper presents a thorough evaluation across various tasks, demonstrating competitive results and significant reductions in memory overhead. After the discussion stage, three of four reviewers vote for acceptance of the submission.
Decision: Accept (poster)