Architecture-Agnostic Test-Time Adaptation via Backprop-Free Embedding Alignment
Authors: MA Xiao
Young D. Kwon
Pan Zhou
Dong Ma
Venue: ICLR 2026 Poster (2026)
Keywords: Test-time adaptation; efficiency; feature space; embedding alignment
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: This paper propose a lightweight forward-only test-time adaptation approach using covariance alignment in embedding space.

Abstract: Test-Time Adaptation (TTA) adapts a deployed model during online inference to mitigate the impact of domain shift. While achieving strong accuracy, most existing methods rely on backpropagation, which is memory and computation intensive, making them unsuitable for resource-constrained devices. Recent attempts to reduce this overhead often suffer from high latency or are tied to specific architectures such as ViT-only or CNN-only.
In this work, we revisit domain shift from an embedding perspective. Our analysis reveals that domain shift induces three distinct structural changes in the embedding space: translation (mean shift), scaling (variance shift), and rotation (covariance shift). Based on this insight, we propose Progressive Embedding Alignment (PEA), a backpropagation-free and architecture-agnostic TTA approach. By applying a novel covariance alignment procedure at each intermediate layer, PEA efficiently corrects the embedding distortions with only two forward passes. Extensive experiments demonstrate that PEA achieves state-of-the-art performance in both accuracy and efficiency, while also proving versatile across different architectures including ViTs and CNNs.

Published as a conference paper at ICLR 2026
ARCHITECTURE-AGNOSTE TEST-TIME ADAPTATION VIA BACKPROP-FREE EMBEDDING ALIGNMENT
Xiao Ma!, Young D. Kwon', Pan Zhou', Dong Ma
"Singapore Management University, Singapore SSassung AI Center-Cambride Cambridge, United Kingdom *University of Cambridge, Cambridge, United Kingdom
x siaoma. 2022 20228ph smu edu. sg, yd. srronssamsunge com panzhou @ smu edu. sg, cm87 @ea.m.ac.........
ABSTRACT
Time Adaptation (TTA) adapts a deployed model during online inference mitigate the impact of domain shift. While achieving strong accuracy, most exist- ing methods rely on backpropagation, which is memory and computation inten- sive, making them unsuitable for resource-constrainee devices. Recent attempts to reduce this overhead often suffer from high latency or are tied to specific ar- chitectures such as ViT-only CNN-only. In this work, revisit domain shift from an emhedding perspective. Our analysis reveals that domain shift induces three distinct structural changes in the embedding space: translation (mean shift), scaling (variance shift), and rotation (covariance shift). Based on this insight, we propose Progressive Embedding Alignment (PEA), backpropagation-Fie frre and architecture-agnostic TTA approach. By applying a novel covariance alignment procedure at each intermediate layer, PEA efficiently corrects the embedding dis- tortions with only two forward passes. Extensive experiments demonstrate that PEA achieves state-of-the-ar performance in both accuracy and efficiency, while also proving versatile across different architectures including ViTs and CNNs. Code released at https: : / / thub. com/ Twte/eeeeaaaaaaaa PaaeeMaaaaa/PEAA11 a0/PEA_TTA TTA.
INTRODUCTION
Deep neural networks (DNNs) have achieved remarkable success across a wide range of computer vision tasks (Pouyanfar et al., 2018). However, their performance often degrades significantly under distribution shifts between the training data and unseen test data a challenge that frequently arises in real-world and real-time applications (Koh et al., 2021; Sun et al., 2022). To address this limitation, DNNs must be able to adapt effectively to such shifts. Test-time adaptation (TTA) (Liang et al., 2025) has recently emerged on-the-fly using unlabeled test batches as they arrive during inference. By continually adjusting to data distributions, TTA mitigates the performance degradation caused by domain shifts and enhances the robustness deployed models.
promising paradigm, enabling pretrained model to be fine-tuned
Most mainstream TTA approaches rely on either pseudo- labeling (Wang et al., 2022; Marsden et al., 2024; Lee & Chang, 2024) is self-supervised strategy that assigns provisional labels the current test batch and updates the model based on these label estimations. In contrast, entropy minimization (Wang et al., 2020; Niu et al., 2022; 2023) is an unsupervised method that encourages the model to produce more confident predictions directly from unlabeled data. Despite their effectiveness, both approaches suffer from fundamental drawback: they depend on backpropagation. Specifically, they require backward passes and gradient storage across multiple layers during adaptation, which introduces substantial computational and memory overhead. This reliance makes them unsuitable for deployment in resource-constrained settings, such as edge devices or real-time applications. Recent methods like SPA and CMF cannot deploy on edge devices due to exceeding 10GB memory requirements (Table l).
entropy minimization. Pseudo-
To mitigate the inefficiency of backpropagation, several recent studies have proposed lightweight alternatives via reducing the overhead gradient-bart based updates. For example, MECTA (Hong al., 2023) combines model pruning with entropy minimization to reduce gradient computation.
"Corresponding author.

---

Published as a conference paper at ICLR 2026
EcoTTA (Song et al., 2023) replaces heavy convolutional blocks with lightweight netaa-s to lower backpropagation costs. Similarly. L-TTA (Shin & Kim, 2024) observes that shallow layers contribute most to adaptation and thus restricts updates to the stem layers, simplifying the process. More recently, some methods attempt to remove backpropagation altogether. FOA (Niu et al., 2024), for instance, performs derivative- prompt search for Vision Transformers (ViTs) (Dosovitskiy et al., 2020), thereby eliminating backward passes and reducing memory usage. However, FOA still incurs high latency, achieving competitive accuracy requires a large number of forward passes (e.g..2))
second major limitation of existing efficient TTA methods lies in their lack of architectural gener- ality. While full buckpropagation-busees approaches are broadly applicable to both CNNs (He et al., 2016) and Transformers (Vaswani et al., 2017); most efficient variants are narrowly tailored. For instance, FOA is designed exclusively for ViTs via prompt tuning and cannot be applied to CNNs. Conversely, methods like ECoTTA and MECTA are tailored to ResNet-style CNNs that rely on batch normalization layers, rendering them ineffective for Transformer architectures.
In this paper, we introduce PEA, a backpropagation -free and architecture-agnosiie method for efficient TTA. Our approach is motivated by principled analysis of how domain shifts distort inter- mediate feature representations. Specifically, our analysis reveals that features from shifted domains consistently diverge from source-domain features through three structural transformations: (i) mean shift, which displaces global feature centroids, analogous to a translation of the distribution; (ii) variance shift that modifies the spread of features and inter-class spacing, corresponding to scal- ing, and (iii) channel-wise covariance shift, which modifies inter-feature correlations, effectively rotating the feature space and reorienting class relationships.
Grounded in these observations, PEA progressively aligns feature covariances at each model block during inference. thereby enhancing the quality of final-layer representations and improving pre- diction reliability. Specifically, PEA implements a two-forward-per tifies the layer-wise shifts, and then, based on these shifts, assigns weights for each block im- plement covariance alignment across all layers' embeddings. Unlike prior methods, PEA is both backpropagation-fiee and architecture-agnosiic making it applicable to both CNNs and Transform- ers. This provides a unified and efficient solution TTA. Our main contributions are as follows:
procedure: the first pass iden-
Our analysis of intermediate embeddings uncovers the essence of domain shifts, which can be characterized as translations, scalings, and rotations of the embedding space.
We propose PEA, an approach that adapts using only two forward passes per batch without back- propagation, allowing efficient adaptation with minimal memory and compute overhead.
PEA is the first unified TTA framework that seamlessly generalizes across both CNNs and Trans- formers using identical procedures. Experiments CIFAR-C and ImageNet-CC demonstrate that it achieves competitive or superior performance compared to state-of-the-arr methods, while main- taining high efficiency with successful deployment on resource-constrained edge devices.
2 RELATED WORK
Conventional Test-Time Adaptation. TTA has emerged as practical solution for mitigating do- main shifts that can severely degrade model reliability in deployment (Wang et al., 2024; Liang et al., 2025; Xiao & Snock, 2024). The core idea is to update pretrained model online using only the incoming unlabeled test batches, without requiring access source data or ground-truth labels.
Early TTA studies primarily focused on updating the model's normalization layers. For example, simply recalibrating batch normalization (BN) statistics was found recover some of the accuracy lost under distribution shifts (Benz. et al., 2021). Building on this idea, entropy-based optimiza- tion techniques such as TENT (Wang et al., 2020) and EATA (Niu et al., 2022) update gradients online under the guidance of prediction entropy, often combined with sample filtering or dynamic reweighting to improve stability. These methods established the foundation for unsupervised TTA, which adapts models based solely their confidence without relying external labels.
In parallel, another branch of work leveraged the model's own predictions as supervision signals. These self-supervised strategies fine-tune the model with pseudo-labels generated from the current test batch. Representative examples include mean-teacher adaptation (Wang et al., 2022). meta- learned initialization for rapid convergence (Bartler al., 2022), and improved label robustness via
2

---

Published as a conference paper at ICLR 2026
symmetric cross-entropy (Döbler et al., 2023). More recent efforts further stabilized this process through ensembling (Marsden et al., 2024) and Kalman filter refinement (Lee & Chang, 2024).
Despite their differences, both unsupervised and self-supervised TTA methods share key limita- tion: they rely on backpropagation during adaptation. The need to compute gradients and store inter- mediate activations largely increases memory and computation overhead, limiting their practicality on resource-conee conourrc--oosttriieed devices and motivating the development of more efficient alternatives.
Efficient Test-Time Adaptation. Recent TTA research has increasingly focused on improving efficiency from various angles. Memory-aware gradient-basel methods aim to reduce the footprint of backpropagation. For example, T3A (Iwasawa & Matsuo, 2021) only adapt the final classifiers to make the adaptation lightweight, but suffer from the sub-optimal improvement. MECTA (Hong al., 2023) prunes gradient paths and normalizes only selected layers lower activation storage, while ECоTTA (Song et al., 2023) leverages compact meta-networks to minimize backpropagation overhead. L-TTA (Shin & Kim, 2024) enhances efficiency by restricting adaptation shallow stem layers in CNNs, and TinyTTA (Jia et al., 2024) combines early-exit classifiers with ensembling for low-memory adaptation on microcontrollers.
Notably, forward-only approaches eliminate gradient computation entirely. For example, LAME (Boudiaf et al., 2022) adjusts classifier decision boundaries post hoc without any gradi- ent updates, though its limited adaptability can reduce accuracy. FOA (Niu et al., 2024) employs derivative-free prompt optimization for Vision Transformers, substantially lowering memory usage but incurring high latency due to the large number of forward passes required.
Moreover, FOA includes an activation shifting module that is related to our work. However, while FOA applies simple mean shift only to the final-layer CLS token and relies primarily on test-time prompt optimization with many forward passes, PEA is statistics-driven and architecture-agnosiic motivated by the empirical observation that domain shift induces systematic embedding changes across layers (mean, variance, and covariance) for both ViT and ResNet, PEA performs block-wise covariance alignment to progressively pull target embeddings back toward the source distribution. This produces a fine-grained laser-wise realignment of the representation.
Overall, existing efficient TTA methods either rely on backpropagation, leading to high memory and computational costs, or are limited to specific architectures (e.g., CNN-only or ViT-only designs). In contrast, our method presents a unified forward-only framework that delivers fast, memory-efficceen adaptation while maintaining strong accuracy across both CNNs and Transformers.
3 MOTIVATION: ANALYSIS OF DOMAIN SHIFT
Although contemporary TTA methods have achieved empirical success, they often treat domain shift black-box problem, focusing on high-level strategies like entropy minimization and prompt tun- ing without exploring the root cause of performance degradation. This motivates our central ques- tion: what is the essence of domain shift? We approach this question from the perspective of the embedding space, hypothesizing that misalignment in intermediate representations is a key driver of performance drop under domain shift. To test this, we conducted an empirical analysis using a ViT model trained on CIFAR10 (source) and evaluated on CIFAR1O-C with F Fog (target). We applied t-SNE to visualize the intermediate embeddings from ViT block 3, focusing on three repre- sentative classes for the sake of clarity in illustration. The resulting visualizations consistently reveal three distinct structural transformations in the embedding space, as shown in Figure 1. We conduct more similar experiments and observe the same phenomenon, which can be found in Appendix A.
Our analysis reveals that, despite its varied forms, domain shift primarily manifests through three characteristic geometric changes in the embedding space:
(i) Translation (Mean Shift). As shown Figure I(a), the most fundamental effect of domain shift is translation of the feature distribution. The global centroid of the target domain's 's embeddings is displaced relative to that of the source domain. As a result, the embedding magnitudes in the target shifted domain become misaligned with the parameters learned by the source model. While this is the most common form of shift addressed by conventional TTA methods, it is often only one part of more complex problem.
(ii) Scaling (Variance Shift). Beyond a simple translation, domain shift significantly alters the scaling of the entire feature distribution, corresponding to variance shift. As depicted in Figure
3

---

Published as a conference paper at ICLR 2026
Class (src)
Class (dom)
Class (src)
Class (dom)
Class 3 (src)
Class (dom)
Sre global centroid
class centraid
Dom global centroid
Dom class centroid
(a) Translation (Mean Shift)
(b) Scaling (Variance Shift)
(c) Rotation (Covariance Shift)
Figure 1: Impact of domain shift on intermediate layer embeddings. Feature distributions of three classes from block of the VIT model are visualized. Each subfigure illustrates different type of shift: translation, scaling, and rotation. More experiments can be found in Appendix A.
1(b), the global "cloud" of features changes its overall shape and density. Some layers may exhibit a more compact feature distribution, where the embeddings are compressed closer to their mean, while others become more dispersed, expanding outwards. This observation is consistent with the insight in GALA (Sahoo et al., 2025) and PALM (Maharana et al., 2025). This non-uniform scaling across layers cannot be corrected by simple global normalization; instead, requires a layer-specific approach to align the variance change in feature space.
Our most erucial observation the presence of covariance shift in the feature space. This indicates systematic change in the correlation among the embedding dimensions. The shift mainly appears as a coherent geometric transformation of the feature cloud, resembling rotation and shearing. As visualized in Figure I(c), this distortion goes be- yond simple translation and scaling, fundamentally altering the relative orientation and arrangement of class clusters.
(iii) Rotation (Channel-wise Covariance Shift).
4 PROGRESSIVE EMBEDDING ALIGNMENT
Based on the analysis above, a natural TTA solution to progressively realign the shifted em- beddings toward the source distribution across model layers. However, applying such alignment presents two key challenges: (1) Since intermediate features are automatically Iearned and propa- gated through the model layers, even small misalignments early layers can accumulate and cause significant degradation in deeper representations. (2) TTA typically operates with small batch sizes (e.g., 64 or fewer) on the devices, making it difficult reliably estimate feature statistics.
To address these issues, we propose Progressive Embedding Alignment (PEA), a simple yet effective method that incrementally refines intermediate representations through robust covariance alignment. To tackle the challenge of accumulating errors, our method employs distance-aware weighted covariance alignment strategy that progressively interpolates between the original and aligned em- beddings based on their degree of shift, ensuring robustness and preventing over-correction. To overcome the challenges of small batch sizes, we introduce two techniques: an exponential moving average (EMA) to accumulate historical estimates of statistics, and lightweight data augmenta- tion to diversify input samples and enrich the feature distribution observed at test time. Unlike many prior TTA methods that require updating model parameters to fit the shifted domain, PEA is entirely buckpropagatoon-fre and crchilecture-agnoste operating solely on intermediate features. The complete PEA pipeline is summarized in Algorithm 1 (Appendix B).
4.1 DISTANCE-AWARE WEIGHTED COVARIANCE ALIGNMENT
The key objective of our method is to progressively realign the test-time intermediate features with the source-domain distribution each block of the DNN. We achieve this using a Whitening- Coloring Transform (WCT) (Cho et al., 2019) that geometrically transforms the target-domain fea- tures to match the structure of the source domain. However, as we mentioned in the first challenge above, applying covariance alignment too aggressively risks over-correction and misalignment. To balance this, we introduce a distance-aware weighting mechanism that adaptively combines the orig- inal and aligned features based on their layer-specific statistical discrepancy. Our method operates
4

---

Published as a conference paper at ICLR 2026
in two stages: offline stage that extracts source statistics prior to deployment, and an online stage that performs dynamic alignment at test time through a two-forward pass procedure.
Offline Stage. Prior to test-time deployment, compute and store the source feature statistics for each block of the model using the training set. These include the source mean vector μa, and covariance matrix s, These pre-computed statistics serve as the source geometry toward which we realign the test-time features. This offline process requires only forward pass through the train- ing data and does not involve any gradient computation or backpropagation. Once computed, the statistics require only minimal storage (about 30MB for ViT-Base) and enable deployment without ongoing source data access, making our approach practical for real-world deployment scenarios.
Online Stage, At test time, each incoming batch undergoes two forward passes. The first pass estimates the degree of domain shift each layer to determine the appropriate alignment strength. The second pass then performs the actual feature alignment using WCT. Unlike prior forward-on]) methods (Niu al., 2024) that require multiple runs to optimize prompts, our approach achieves adaptation with just two two forward passes.
Pass Estimating Alignment Weights. The goal the first pass is to measure how much the current batch deviates from the source distribution at each block. To achieve this, we forward the test batch through the network to extract the intermediate feature activations F; E BR For each block compute the batch mean 1b. and variance of These statistics characterize the current batch's distribution. To quantify the shift, we calculate statistical distance between the batch and source distribution:
(1) This distance captures both center shift (translation) and scale mismatch at each layer. We then normalize these raw distances across all layers using min-max scaling to obtain the alignment weight wy E [0, 1]:
di Ma.1 Ma, + || -
d: 1--miin4 di
Wi max di min; di
The weight wt reflects how strongly the features block i should be aligned: layers with minimal shift receive near-zero weights (i.e., skip alignment), while those with high discrepancy are corrected more aggressively.
Performing Weighted Feature Aligument. In the second forward pass, we reprocess the batch through the model and apply WCT-based alignment at each block. Let the updated test-time batch statistics be Ir, and Σt,, which may be computed either from the current batch or from EMA tracking (see Section 4.2). We then apply the whitening-colee colorinn transformation:
Pass 2:
Y = (F: - p4, E 1/251/2 Pa.1
In Eq. 3, first whiten the test features by removing domain-specifet variations using the target- domain mean and the square root of its covariance matrix. We then re-color the features with the source-domain covariance and mean restore the geometry of the source distribution.
Instead of directly replacing the original feature with the aligned output, we blend them using the previously computed weight:
= - wy) + wy Yi
(4)
The combination of F and Yi ensures that features are only shifted when necessary, maintaining stability for well-aligned layers while correcting mismatched ones.
One of the main computational bottlenecks in alignment lies in the operations on covariance matrices, especially the computation of the matrix square root 1 and its inverse E-1/ To perform this efficiently and stably, we use eigendecomposition tailored for symmetric positive semi- definite (SPSD) matrices. Given covariance matrix Σ, we first compute the eigendecomposition Σ = V VVT, where V contains the eigenvectors and A contains the eigenvalues. The square root and inverse square root are then computed as:
x VA 1-1/2_ VA 1
This cigendecomposition simplifies the computation of the matrix square root and its inverse, effec- tively avoiding the high computational burden of general matrix operations. Overall, our method in- troduces minimal overhead: the eigendecomposition used for alignment is computationally efficient
5

---

Published as a conference paper at ICLR 2026
due to the moderate feature dimensionality at each layer (typically 128- 1024), and it only applied during the forward pass. Crucially, our approach is entirely gradient-free and model-agnostte - does not require backpropagation and task-specific tuning. All operations are performed on interme- diate feature activations, allowing for seamless integration with a wide range of architectures (e.g.. CNNS and ViTs) and low-latency deployment on resource-constrained devices.
4.2 ROBUST STATISTICS ESTIMATION VIA EMA
The effectiveness of the embedding alignment critically depends the accurate estimation of the target domain statistics (J44,1, Σr,1) However, test-time deployment, especially on resource- constrained devices equipped with limited memory, often necessitates small batch sizes (e.g.. 64 or fewer), resulting in unreliable statistical estimates when derived from a single batch. To mitigate this issue, we maintain an Exponential Moving Average (EMA) strategy of the target feature statistics to accumulate historical batches to yield more stable and robust estimation over time. For each new batch i, the EMA is updated with a momentum parameter m.
(f-1) + m pb,t 4 = (1 - m) Mil
E -1-m)) -
+ m En,1
(6)
While EMA ensures stability, can be slow to adapt to sudden and fast domain shifts, causing the model to be anchored outdated statistics. To solve this problem, incorporate spike domain shift detection mechanism based on prediction entropy.
Spike detection uses the model's prediction confidence sudden drop in confidence (i.e., a sharp rise in entropy) often indicates that the model is encountering new, unfamiliar domain (Ma al., 2025). We track an EMA of the batch average prediction entropy. E and compare it to the instantancous entropy of the current batch, He. A spike is flagged if the current entropy surpasses the historical average by a fixed threshold tent
signal for detecting domain shift. A
(7) If an entropy spike is detected, the EMA statistics (Ht,), Σ.,) are immediately reset to those of the current batch. The detection module allows the model to rapidly adapt to the new data distribution, ensuring both stability during gradual shifts and agility during abrupt ones. The EMA update is com- putationally lightweight, involving only simple averaging per layer with negligible cost. Memory usage also minimal, requiring storage of just two small tensors per block.
Spike if: H, > Eems + Oent
4.3 DATA ENRICHMENT VIA LIGHTWEIGHT AUGMENTATION
To further enhance the estimation of the target batch distribution, we introduce a lightweight data enrichment strategy based on simple and low-cost augmenlutions (Simonyan & Zisserman, 2014). These augmentations include common geometric transformations such as horizontal flips, random crops, and mild rotations. They are computationally inexpensive and preserve the semantic consis- tency of the domain. For each input image, we generate K augmented views. This data augmenta- tion is integrated into both forward passes during the online adaptation stage:
Pass 1: As described in Section 4.1, the first forward pass is used to estimate the layer-wise distribu- tion discrepancy by computing the feature statistics of the current batch. To enhance the estimation under small batch sizes, we apply augmentation to each image and process the resulting K-view batch in the first forward pass. We then compute the alignment distance in Eq. using this enriched batch, which results in more robust and stable weight estimation for each layer.
The second forward pass performs the actual alignment using the WCT transformation shown in Eq. 3. As in the first pass, we augment the batch into K views and apply the WCT align- ment across all views. After obtaining K sets of aligned predictions, we aggregate them through uniform averaging:
Pass 2:
K
pred
logits
KKK
1=1
The feature enrichment and ensembling not only improve the stability of embedding alignment but also enhance final predictions by incorporating multiple complementary views of the data. Despite introducing multiple views per input, the augmentations are lightweight and require no additional model parameters or backward passes. As result, the added cost is limited to repeated forward

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of accuracy (%) on ImageNet-C using ViT-Base and ResNet-50 with memory consumption on server. Aug and BP indicate whether the approaches utilize data augmentation and backpropagation. In FOA, F specifies how many forward passes per batch.
Medel
Aug
Nethool
aloo
irppl.
defoe.
plao
modionn
2000
froi
Sag
brigh.
cust..
cluc
pisel.
Arg.
Mem.
Latency
50.7
17.5
11.3
15.
35.1
11.3
00.5
17.5
32.6
15.3
55.
SAR
50.4
63.3
62.6
14.3
56.1
19.1
54.5
63.1
05.6
65.
16.3
68.8
16.3
00.3
66.7
62.7
6181
0.94
Yeat
17.1
58.1
19.2
64.7
43-3
16.5
936
62.6
00.5
653
18.0
59.7
40.8
68.3
68.8
58.8
6008
6:31
VI2
FTIA (77
POX ()
00.5
01.5
63.1
66.
65.9
51.6
16.9
15.1
46.1
61.0
50.4
004
57.3
06.8
68.6
70.9
71.
736
K0.0
68.9
66.0
66.6
61.
15.0
72.9
73.
71.8
66.1
871
1.75
CMP
00:0
0:-3
ML.9
16.6
15.4
62-4
093
59.1
67.9
72.7
78.4
65.2
69-4
T3.
73.0
66.9
10HB1
0:53
PEA
58.9
60.4
64.5000
PEA NM
61.2
61.5
[2.]
$5.6
$2.4
51.4
62.1
70.4
70.7
74.5
50.9
65.8
71.7
73.4
73,6
06 Sch.
1957
0.59
22:2
23-7
21.3
2000
10122
21:5
25.1
21.5
35.1
20.5
67.7
25.6
14.0
13.1
27.3
20.8
817
0.17
Teat
27.4
18.6
17.4
32.5
44.9
MECTA
19.3
32.9
18.6
16.1
15.0
31.5
45.0
14.9
45.1
63.0
71.1
33.5
47.7
53.2
39.4
55.0
4425
0.50
EESTTA
3.9
6.2
3.5
7.5
9.6
24.3
41.
48.3
31.9
M6.7
N6.
16.5
41.1
46.2
46.2
50S
5871
0.6
RosNet
L-TA
34.8
22.5
12.1
1833
23.3
31.6
52.5
66.1
16.6
43.5
41.8
56.5
LAW
19.5
25.2
28.8
22.5
27.5
16.1
45.9
42.1
40.2
53.7
66.5
52.8
52.5
52.8
59.9
11734
0.00
PEA
22.6
25.0
22.0
15.0
25.1
17.3
45.2
89.3
68.7
648
13.5
S1.
53.0
43.5
52.2
02
22-72111
983
0.3630.2
PEA AM
26.2
25.8
25.5
24.8
36.8
41.6
51.3
68.6
52.4
55.0
47.9
448668.
passes with minor geometric transforms, making this approach highly efficient and practical even on memory-coostrained edge devices.
Fundamental Methodological Difference of PEA: Existing TTA methods typically update the affine parameters of normalization layers through backpropagation. i.e., they adapt the model to fit the shifted domains, using techniques such as entropy minimization and data augmentation. How- ever, as discussed in (Press al., 2024), the absence of ground labels at test time often cause embedding drifts over successive iterations, resulting in suboptimal performance or even leading to catastrophic forgetting.
In contrast, our approach adopts fundamentally different strategy: rather than modifying the model, we align the shifted embeddings with the source distribution. This eliminates the need for backprop- agation, ensuring that the original model parameters remain intact and robust, thereby completely mitigating catastrophic forgetting.
5
EXPERIMENTS
5.1 DATASETS AND BASELINES
Following recent works in TTA (Shin & Kim, 2024; Niu et al., 2024), we conduct comprehensive evaluation across multiple datasets. Specifically, we use CIFAR10-C, CIFAR 100-C, and ImageNet-C, each of which introduces 15 common corruption types applied to the original test sets. We adopt the most severe corruption level (severity = 5) and batch size of 64 throughout all experiments. To simulate a realistic online domain shift scenario, we follow the lifelong continual test-time adaptation setting in CoTTA (Wang et al., 2022; Niu et al., 2022), where corrupted samples are streamed sequentially at test time. Compared to always adapting each domain from the source domain, our continual setting is more realistic and challenging.
Datasets and Models.
For backbone models, we adopt both ResNet-50 (He et al., 2016) and ViT-Base (Dosovitskiy et al., 2020) on the ImageNet-C and CIFAR 100-C datasets. For CIFAR we evaluate using ResNet-50 and ViT-Tiny to account for the dataset's 's smaller scale. This diverse selection demonstrates that our method generalizes effectively across both CNN and Transformer-basee architectures.
Baselines. We compare our proposed PEA with several efficient TTA approaches as well as state- of-the-art performance-driven methods. For efficient CNN-based TTA, we include EсоTTA (Song et al., 2023), MECTA (Hong et al., 2023), and L-TTA (Shin & Kim, 2024). For ViT-specific adap- tation, we evaluate FOA (Niu et al., 2024). which performs forward-only prompt optimization. We also evaluate entropy minimization-ban -based methods including Tent (Wang et al., 2020), EATA (Niu et al., 2022), and SAR (Niu et al., 2023). Finally, we include recent state-of-the-an approaches based on pscudo-labeling and data augmentation: CMF (Lee & Chang, 2024), LAW (Park et al., 2024) and SPA (Niu et al., 2025)'. Details of the implementation and additional clarifications are provided in Appendix C.
'The SPA results reported in the original paper are obtained under a single-domain adaptation setting, where the model is reset hefore each corruption. In our experiments, we more challenging lifelong TTA setting, where the model adapts continuously across all domains without reset.
7

---

Published as a conference paper at ICLR 2026
Table 2: Adaptation accuracy (%) on CIFARIO-C and CIFARI00-C using VIT and ResNet.
Method
Model
Dataset
Adapt
(F=27)
SPA
Aug
VIT
CIFARTOU-C
CIFARIO-C
61.6
76.5
61.3
75.8
76.2
76.5
81.9
83.3
68.0
73.0
83.3
71.7
76.2
83.784,7:0.1
75.7-0.2
84.7+1k.1
77.0+00.2
Dataset
No Adret
TENT
PATA
MECTA
BoOTTA
L-TTA
CMP
LAW
PEA
PEA with Aug
ResNet
62.5
81.2
81.3
80.2
81.2
78.6
81.8
81.80..1
83,4t0.1
CIFARTOU-C
33.5
45.6
48.8
51.0
53.960.2
54.6109.3
5.2 MAIN RESULTS ON IMAGENET-C
Table presents the classification accuracy and variation (averaged over 5 runs with different random seeds) for each domain, together with the memory consumption and per-batch inference latency measured on the server.
For ViT-Base, without adaptation, the baseline VIT model achieves average accuracy of 55.5%. Although existing methods such as Tent (Wang et al., 2020) and EATA (Niu et al., 2022) offer moderate improvements to 58.8% and 60.7%, they incur substantial memory overhead (more than 6 GB) due to backpropagation-baaee based updates. More recent backprop-free method FOA (Niu et al., 2024) and SOTA SPA (Niu et al., 2025) achieve stronger accuracy (up 66.1% and 64.6%) but with high latency (up to 3.33s) or memory consumption (over 10 GB). By contrast, our PEA achieves 64.5% accuracy with only 887MB of memory and 0.3 latency. When combined with augmentation (PEA Aug), performance further improves to 66.5%, surpassing FOA with better latency. This demonstrates that PEA not only provides competitive accuracy but also delivers exceptional memory and latency efficiency, making it highly suitable for real-time or on-device deployment.
For ResNet-50, TTA baselines such as Tent, EATA, and CMF improve performance to up to 43%, but again at the cost of large memory (more than 5.9 GB) and higher compute demand. PEA out- performs all low-cost adaptation methods with an average accuracy of 42.7%, using only 983MB of memory. With augmentation, it reaches 44.8%, outperforming all existing backprop-fire methods like ECOTTA and L-TTA by a large margin.
Efficiency and Accuracy Trade-off: Our approach achieves highly favorable balance between bustness and efficiency. Unlike backpropagation-bun saaeepe based TTA methods, PEA delivers strong adap- tation performance while consuming significantly less memory and maintaining low latency. This lightweight yet effective design makes PEA highly suitable for practical deployment, especially in resource-constrained devices real-time systems, we will further discuss it in Section 5.6.
5.3 RESULTS ON CIFARIO-C AND CIFAR 100-C
We also evaluate the performance of PEA on the CIFARIO-C and CIFAR100-C using both ViT and ResNet. As shown in Table 2 (for more details see Section D.I), PEA consistently outperforms existing TTA approaches. Notably, under the VIT backbone, PEA achieves 77.0% accuracy on CIFARIO-C and 84.7% on CIFAR100-C when lightweight augmentation is applied, substantially outperforming augmentation-ben based basclines like CMF and SPA. Even without augmentation, PEA attains competitive results (75.7% and 83.7%). demonstrating its intrinsic robustness. Similar trends are observed with the ResNet backbone, where PEA achieves 83.4% on CIFAR1O-C and 54.6% on CIFAR 100-C, again outperforming strong baselines including MECTA, EcOTTA, and L-TTA
In addition, we observe that mugmentation-baser methods such as CMF and SPA show relatively lim- ited gains on these small-scale datasets compared to their performance on larger dataset ImageNet-C. This suggests that excessive reliance on augmentation alone may not generalize well across dataset scales. In contrast, PEA demonstrates strong generalization across both model architectures and dataset types. Importantly, it achieves this without updating any model parameters and is entirely backprop-free, making it naturally compatible with both CNN and Transformer architectures.
5.4 RESULTS ON SMALL BATCH SIZE
Table 3 presents the performance our method under varying batch sizes (BS = 4, 16, 64) on both CIFAR 100-C and ImageNet-C, using ResNet-50 and ViT-Base. We observe that while accuracy slightly drops as the batch size decreases, our method retains high performance even under very small batches. On CIFAR100-CC the ViT-Base model achieves 77.0% with BS=64 and maintains strong 70.0% even with BS=4, a modest 7.0% drop. In contrast, the ResNet-50 50 model sees a smaller absolute decline (from 54.6% 51.8%), but its overall accuracy remains much lower. A similar
8

---

Published as a conference paper at ICLR 2026
Table 4: Evaluation on Jetson Orin Nano using CIFAR 100-
C with batch size of 64. Methods marked as incompatible (X) fail due insufficient memory on the target device (3.5
Table 3: Results on small batch sizes on CIFAR 100-C and ImageNet-C.
GB). Memory requirements for them are shown in Table 1.
Datasets
Model
Accuracy (%)
VIT Bose
ResNet 50
BS 4
16
BS 64
Lanency (chitich) Monvry (MR)
Lateney (efeecc)
Menny (MB)
ResNet-50
51.8
Adaae
Adapt
100-C
ViT-Base
70.0
75.7
77.0
ImaguNet-CC
ResNet-50
VIT Base
63.3
41.7
65.8
44.0
66.5
44.8
FDA
8
Aug
9.8003
PEA Aug
2388
Table 5: Adaptation accuracy (%) on mixed domain on CIFARIO0 dataset.
No Adapt
Tent
EATA
SAR
FOA (K=9)
CMF
SPA
PEA
ViT-Base
61.6
61.2
61.2
62.0
62.6
69.0
71.4
72.0
No Adapt
Tent
EATA
MECTA
EcoTTA
CMF
L-TTA
PEA
ResNet-50
33.5
17.4
16.5
40.2
7.3
7.3
13.4
47.4
trend is observed on ImageNet-C, where ViT-Base drops by 3.2% and ResNet by 3.1%. As shown in Table 9 in Appendix D.3, our method performs better than other baselines. Further, we evaluate our method under extremely small batch sizes (BS=1 and BS=2) to simulate a streaming inference setup. The results are reported in Section D.4.
5.5 RESULTS ON MIXED DOMAINS SETTING
We further evaluate PEA under mixed-domain setting CIFAR100-C where all 15 corruption types (severity 5) are merged into a single pool and randomly shuffled. Consequently, each mini- batch contains samples from multiple corruptions (on average 64/15 =s 4.3 samples per corruption), emulating a realistic deployment scenario with rapid and unpredictable domain shifts within a batch. This setting is substantially more challenging than the single-domain protocol.
Table 5 summarizes the mixed-domain results. With ViT-Base PEA achieves 72.0% accuracy, im- proving over the non-adapted source model by 10.4% and outperforming all baselines. Notably, entropy-minimization methods (Tent and EATA) provide no gains (61.2%), indicating that simply updating on heterogeneous batches can be ineffective, while even strong baselines such as CMF (7 (71.4%) remain below PEA. On the more challenging ResNet-50 backbone, the gap is larger: Tent and EATA substantially degrade performance (17.4% and 16.5%) and several methods struggle (e.g., ECoTTA 7.3% and L-TTA 13.4%). whereas PEA reaches 47.4% and surpasses the strongest base- line, MЕCTA which performs 40.2%, demonstrating robust adaptation under rapid and irregular domain shifts. To further highlight the effectiveness of PEA under mixed-domain shifts, we provide additional visualizations in Section
5.6 EVALUATION ON EDGE DEVICE
To assess practical deployability, we evaluate the system performance on the Jetson Orin Nano, resource-constrainge edge device with 8 GB of shared memory. only 3.5 GB of which is accessible deep learning applications due OS and system overhead. We test all methods under a default setting with batch size 64 on CIFAR 100-C. Table 4 reports both the latency (in seconds per batch) and peak memory usage (in MB) for ViT-Base and ResNet-50 backbones.
Due to limited memory, many TTA methods fail to run on-device, especially those requiring back- propagation (e.g., Tent, EATA, MECTA. SAR). In contrast, our method (PEA) successfully runs on both backbones, maintaining reasonable latency (4. for ViT, 3.0s for ResNet) and modest mem- ory usage (101IMB and 976MB, respectively). With augmentation enabled, performance trade-offs increase modestly, but still remain within edge constraints. While both FOA and L-TTA are com- patible with edge devices, FOA incurs extremely high latency, rendering it impractical for real-time time applications. In contrast, L--TTA is fast but consistently underperforms in accuracy across all three

---

Published as a conference paper at ICLR 2026
datasets, discussed in Section 5.2 and 5.3. Notably, the forward-only design of PEA ensures patibility with edge settings, where low memory footprint and gradient-free inference are critical, showing its strong potential for real-world deployment without sacrificing adaptation effectiveness.
5.7 ABLATION STUDY
We conduct an ablation study to quantify the contribution of each major component in PEA using the VīT-Base model both CIFARI00-E and ImageNet-C. Table 6 summarizes the incremental performance improvements by the proposed components. Starting from the unadapted baseline, introducing only the covariance alignment module (Cov Align Only) brings significant gain on CIFAR 100-C (from 61.6% to 67.0%), demonstrating that aligning feature second-order statistics strong and lightweight signal for domain correction. However, this setting results sharp drop on ImageNet-C (down to 25.2%), due to over-alignment across all layers. Since ImageNet more challenging and features greater domain complexity, the per-batch estimation of the target distribution becomes less reliable, leading misaligned feature transformations.
Adding the layer-wise distance-based weighting mechanism (+ Weight- ing) mitigates the misalignment on ImageNet-C, boosting performance from 25.2% to 52.9%. This high- lights the importance of selectively applying alignment only to blocks that exhibit significant distribu- tional shifts. The improvement on CIFAR100-C more modest but still positive, suggesting that the weight- ing scheme contributes to robustness across datasets. Incorporating exponential moving average (EMA) for estimating test-time statistics (+ Weighting, EMA) provides a large boost both datasets (75.7% on CIFARI00-C and 64.5% on ImageNet-C). The EMA strategy accumulates stable statistics over time, which is especially beneficial when the test-time batch size small or noisy. This component ensures the alignment based reliable statistics rather than volatile per-batch estimates. Finally, adding data enrichment via lightweight augmentations (+ Weighting, EMA, Aug) yields the highest accuracy 77.0% on CIFAR 100-C and 66.5% on ImageNet-C. The multiple views not only help stabilize the estimation of target statistics but also improve final predictions via ensemble averaging.
Table 6: Ablation study of PEA using ViT-Base model on CIFARI00-C and ImageNet-C.
Ablation
Acc. (CIFARI00-C)
Acc. (ImageNet-C)
No Adapt
61.6
55.5
Cov Align Only + Weighting
67.0
25.2
68.3
52,9
Weighting, EMA
75.7
64.5
Weighting, EMA, Aug
77.0
66.5
Overall, each component contributes complementary benefits to the final performance, and their combination enables PEA to maintain high accuracy under diverse corruptions while being backprop-free and resource-efficient. See Appendix D.5 for more hyperparameter evaluations.
6 CONCLUSION
This work begins by revisiting the impact of domain shift on intermediate model embeddings, identi- fying three core transformations: mean shift (translation), variance shift (scaling), and channel-wise covariance shift (rotation), which systematically distort the feature space across layers. Motivated by this insight, we propose PEA, a lightweight, backpropagation-free, and architecture-agee aggnosi test-time adaptation approach that progressively aligns embeddings through layer-wise covariance correction using only two forward passes. Experiments across 3 datasets, including evaluations on resource-constrained edge devices, demonstrate that PEA achieves state-of-the-art accuracy and efficiency, offering practical and generalizable solution for robust real-world deployment.
Limitations. While PEA provides a lightweight, backpropagation-fiit free solution that generalizes across model architectures, it requires extracting source statistics from the training data prior to de- ployment. While this is acceptable in the standard TTA setting (Song et al., 2023; Niu et al., 2024), such source statisties may not always be available in some practical scenarios. That said, PEA does not require access to the full source dataset: using only 10% of the training data to compute these statistics is sufficient maintain strong performance (see Section D.5.2 2.2 for detailed results).
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
This work does not involve human subjects, sensitive personal data, or applications with direct societal ethical risks. The datasets used in experiments publicly available benchmarks that have been widely adopted in the research community. We believe the contributions of this paper align with the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
We have taken several steps ensure the reproducibility of results. All datasets employed in this study are publicly available and described in the main text. Detailed descriptions of model architec- tures, hyperparameters, and training protocols are provided in the paper and appendix. Furthermore, we provide pscudocode and implementation details in the appendix, and the complete source code will be made available if the paper is accepted.
REFERENCES
Alexander Bartler, Andre Bühler, Felix Wiewel, Mario Döbler, and Bin Yang. Mt3: Meta test- time training for self-supervised test-time adaption. In International Conference on Artificial Intelligence and Statisties, pp. 3080-3090. PMLR, 2022.
Philipp Benz, Chaoning Zhang. Adil Karjauv, and In So Kweon. Revisiting batch normalization for improving corruption robustness. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pp. 494-503, 2021.
Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-fiee free online test- time adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8344-8353 2022.
Wonwoong Cho, Sungha Choi, David Keetae Park, Inkyu Shin, and Jaegul Choo. Image-to-image translation via group-wise deep
transformation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10639-10647, 2019.
Mario Döbler, Robert A Marsden, and Bin Yang. Robust mean teacher for continual and gradual test- time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7704-7714, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Junyuan Hong, Lingjuan Lyu, Jiayu Zhou, and Michael Spranger. Mecta: Memory-economic con- tinual test-time model adaptation. 2023 International Conference on Learning Representations, 2023.
Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic do- main generalization. Advances Neural Information Processing Systems, 14:2427-24400 2021.
Hong Jia, Young Kwon, Alessio Orsino, Ting Dang, Domenico Talia, and Cecilia Mascolo. Tinytta: Efficient test-time adaptation via early-exit ensembles on edge devices. Advances in Neural In- formation Processing Systems, 37:43274-43299 2024.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, pp. 5637-5664. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2026
Jae-Hong Lee and Joon-Hyuk Chang. Continual momentum filtering on parameter space for online test-time adaptation. In The Twelfth International Conference on Learning Representations, 2024.
Jian Liang. Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distri- bution shifts. International Journal of Computer Vision, 133(1):31-44, 2025.
Xiao Ma, Young D Kwon, and Dong Ma. On-demand test-time adaptation for edge devices. arXiv preprint ar arXiv:2505..00 00986, 2025.
Sarthak Kumar Maharana, Baoming Zhang, and Yunhui Guo. Palm: Pushing adaptive learning rate mechanisms for continual test- test-time adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 19378 19388,1986,,, 2025.
Robert Marsden, Mario Döbler, and Bin Yang. Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2555-2566 2555,5565.2 2024.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning. pp. 16888-16905. PMLR, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. arXiv preprint arXiv:2302. 12400, 2023.
Shuaicheng Niu, Chunyan Miao, Guohao Chen, Pengeheng Wu, and Peilin Zhao. Test-time model adaptation with only forward passes. arXiv preprint arXiv:2404.0163 2024.
Shuaicheng Niu, Guohao Chen, Peilin Zhao, Tianyi Wang, Pengcheng Wu, and Zhiqi Shen. Self- bootstrapping for versatile test-time adaptation. In Forty-second International Conference on Machine Learning, 2025. URL https: /top/oppeevveeeeeee wwt.ieewwee//eeee...ee.. forum? wwt.::iidiiiieeeeeee
Junyoung Park, Jin Kim, Hyeongjun Kwon, Ilhoon Yoon, and Kwanghoon Sohn. Layer-wise auto- weighting for non stationaty test-time adaptation. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision, pp. 1414-1423, 2024.
Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei- Ling Shyu, Shu-Ching Chen, and Sundaraja S Iyengar. A survey on deep learning: Algorithms, techniques, and applications. ACM computing surveys (CSUR), 5155):1-36, 2018.
Ori Press, Ravid Shwartz-Ziv Yann LeCun, and Matthias Bethge. The entropy enigma: Success and failure of entropy minimization. arXiv preprint arXiv:2405.05012 2024.
Sabyasachi Sahoo, Mostafa ElAraby, Jonas Ngnawe, Yann Batiste Pequignot, Frédéric Precioso, and Christian Gagné. A layer selection approach to test time adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 20237-20245 2025.
Jin Shin and Hyun Kim. L-tta: Lightweight test-time adaptation using versatile stem layer. Ad- vances Neural Information Processing Systems, 37:39325-39349 2024.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 14Xivv1400.1555 1556, 2014.
Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11920-11929, 2023.
Tao Sun, Mattia Segu, Janis Postels, Yuxuan Wang, Lue Van Gool, Bernt Federico Tombari, and Fisher Yu. Shift: synthetic driving dataset for continuous multi-task domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 21371-21382. 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2026
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint vrXiv:2006.10722 2020.
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7201-7211, 2022.
Zixin Wang, Yadan Luo, Liang Zheng, Zhuoxiao Chen, Sen Wang, and Zi Huang. In search of lost online test-time adaptation: A survey. International Journal of Computer Vision, pp. 1-34, 2024.
Zchao Xiao and Cees GM Snock. Beyond model adaptation at test time: A survey, arXiv preprint 1xXiv:2411003666 2024.
13

---

Published as a conference paper at ICLR 2026
Class 1 (src)
Class (dam)
Class (src)
Class 2 (dom)
Class 3 (src)
Class (dom)
block2
block4
block5
block6
block7
blockB
block9
Figure 2: Impact of domain shift on intermediate layer embeddings using CIFAR10-C. We visualize the features of class plane, dog, and frog in Domain Fog across ViT block to block 9.
A DOMAIN SHIFT IN THE EMBEDDING SPACE
As discussed Section 3, domain shifts manifest as structural distortions in the intermediate fea- mean shift, variance shift, and covariance
ture space of deep models. These distortions-naned chift-occur consistently across all layers of the network. In this section, we provide additional empirical evidence to support this analysis by visualizing feature distributions under domain shift using ViT models on CIFARIO-C.
Visualization of Feature Shift Across Layers and Domains. We conduct detailed visualiza- tion of intermediate features extracted from ViT blocks through 9 on two corruption types from CIFAR 10-C: Fog and Gaussian Noiss We focus three representative classes-plane, dog, and frog-to illustrate how domain shift affects the geometry of class embeddings at different depths of the model.
Figures 2, 3 and 4 show the progressive deformation of class-wise embeddings under these two corruption domains. These results complement our earlier analysis and reveal consistent patterns across domains and layers.
From the visualizations, we observe that: All layers are affected by geometric distortions. Across all blocks, we observe consistent evidence of (i) mean shift, where class centers drift from their source positions; (ii) variance shift, indicated by altered spread and scale of feature clusters; and (iii) channel-wise covariance shift, where the orientation and shape of the clusters change due to altered inter-channel relationships; 2) The severity of distortion varies across layers. Different layers exhibit different sensitivities to each type of transformation; 3) Different domains impact features in distinct ways. Although both Fog and Gaussian Noise induce all three types of shifts, the degree and pattern of deformation vary. This reflects the domain-specific characteristics of the corruption types- e.g., Fog tends to cause smoother global drifts, while Gaussian Noi se leads to more irregular scatter.
Moreover, in domains such as Fog. which are relatively easy to adapt, the transformations at deeper layers more faithful than at shallow layers, suggesting the architecture progressively correct
14

---

Published as a conference paper at ICLR 2026
Class 1 (src)
Class (dom)
Class (src)
Class 2 (dom)
Class 3 (src)
Class (dom)
block2
block4
block5
block6
block7
blockB
block9
Figure 3: Impact of domain shift on intermediate layer embeddings using CIFAR10-C. We visualize the features of class plane, dog and frog in Domain Gaussian Noise.
the shift features propagate. In contrast, in domains such Gaussian Noise, deeper layer features deteriorate (pronounced scaling shrinkage and rotation), indicating that domain character- istics strongly shape the final representations and can hinder self-correction in depth.
These insights reinforce the central hypothesis our work: domain shift induces systematic, layer- wise geometric transformations in the embedding space. They also motivate proposed method, which explicitly corrects such distortions through progressive covariance alignment at each interme- diate block.
Visualization of Feature Shift on CIFAR100-C. To further validate that the observed structural shifts in embedding space are not specific EIFARIO-C, extend visualization CIFAR 100- C. We select three representative classes-pine pine tree, bicycle, and bee-from the CIFAR100-6 dataset and examine their intermediate representations under domain shift caused by Shot Noise, a common corruption in CIFARI00-C. The results are shown in Figure 5.
Similar to the patterns observed in COFARIO-C, we find that domain shift consistently induces sys- tematic geometric transformations in the embedding space: mean shift (translation), variance shift (scaling) and covariance shift (rotation). While the inter-class topology often preserved, these structural distortions displace features away from the decision boundaries, ultimately degrading classification performance. In particular, even though class relationships remain recognizable, the shifted features no longer be correctly classified due to their increased distance from the source- aligned classification regions.
These shifts manifest across multiple blocks of the ViT model, reinforcing claim that domain shift affects not only the output layer but also the intermediate representations systematic and structured manner. The consistency of these patterns across both CIFAR10-C and CIFAR100- highlights the generality our observation and motivates the need for intermediate- realign- ment strategies, such the one introduced in PEA.
15

---

Published as a conference paper at ICLR 2026
block2
block4
block5
block6
block7
blockB
block9
Figure 4: Impact of domain shift intermediate layer embeddings using CIFAR1O-C. We visualize the features of class plane, dog and frog in Domain Defocus Blur.
Cless 1 (src)
Class (dom)
Class (src)
Cless 2 (dom)
Class 3 (src)
Class (dom)
block1
block2
block3
block4
block5
block6
block7
block8
block9
Figure 5: Impact of domain shift intermediate layer embeddings using CIFAR100-C. We visu- alize the features class pine tree, bicycle and bee Domain Shot Noise.
16

---

Published as a conference paper at ICLR 2026
Algorithm PROGRESSIVE EMBEDDING ALIGNMENT (PEA)
Require: source stats (, E batch X, views K
1: Pass 1: Estimate alignment weights
2: Augment input: Xaug Augment
3: Extract block features Fj; compute alignment weights wy using Eq. 1 and Eq.
4: Pass 2: Align and predict
5: Extract Pi from Xug
6: for each block 7 do
7: Update Pe,1, Σt with EMA (Eq.
8: Apply covariance alignment: Yi (Fi 1(,1)
1/11122111 2'1 + pad
// Eq.3
9: Fuse: Fy = Fi) w.Yi Y
// Eq.4
10: end for
11: Compute logits for all views: logits, Classifier (Zt)
12: Final prediction: predinal - * Σ logits;
// Eq.
13: return pre
B PSEUDO CODE
To clarify the workflow of our PEA, we present its pseudo-code in Algorithm 1, which outlines the step-by-step process of progressively aligning domain-shifted features toward the source distribution during test-time adaptation.
C IMPLEMENTATION DETAILS
We use a default batch size of 64 for all evaluations, consistent with prior works (Niu et al., 2022; Lee & Chang. 2024). All methods are implemented and tested on server equipped with an NVIDIA A5000 Ada GPU. To evaluate the feasibility of deployment under real-world constraints, we also compare PEA with efficient TTA baselines on an edge device: the Jetson Orin Nano, which includes Cortex-A 778AE CPU and an 8GB shared RAM mobile GPU.
For our PEA, we set the EMA momentum m 0.02 ensure stable yet responsive estimate of feature statistics. To detect domain shifts, we use an entropy spike threshold 0em 1.0; when this exceeded, the EMA statistics are reset. We tested EMA momentum values of 0.01, 0.02, 0.05, 0.1 and found the performance to be robust, with accuracy fluctuations of at most 1 1% (see D.5). For augmentation, we use random horizontal flips and random resized crops (scale = 0.9). Each input generates K = 2 augmented views, which, combined with the original, produce views used for prediction ensembling. Since the augmentation an optional technique based on the available mem- ory, also report the results without augmentation. All other hyperparameters, such learning rates and optimization settings for baseline methods, are taken from their official implementations maintain fair comparison conditions. Note that we use prior works' original implementations without modification to ensure fair assessment of each method's 's realistic resource requirements.
Clarification of SPA. For SPA, although the paper (Niu et al., 2025) claims that it can generalize to both CNNs and ViTs, the official code provided by the authors includes only the ViT implemen- tation. Consequently, we report comparisons with SPA only on ViT. As shown in Table 1, while SPA achieves competitive performance, it incurs extremely high memory consumption (exceeding 10GB), making it impractical for efficiency driven applications.
D FURTHER DETAILS FOR EVALUATION
D. DETAILED RESULTS ON CIFARIO-C AND CIFAR100-C
While the main paper (Table 2) reports average adaptation performance across the 15 corruption types of CIFARIO- and CIFAR100-C 100-C, we provide the full per-domain performance this appendix to offer granular view of model robustness.
17

---

Published as a conference paper at ICLR 2026
Table 7: Detailed accuracies (%) on CIFARIO-C.
Medell
Methols
shox
Ingull
delox.
glass
mation
J00m
6ROW
Frust
log
brigh.
contr:
clust
pive.
JINIE
Avg.
No Aclapt
-5.8
29.8
09.3
K2.1
7.0
82.1
54.5
91.8
89.3
87.3
35.5
K3.4
X..2
51.8
X1.4
76.3
SAR
25.4
54.7
70.0
81.9
69.8
81.7
53.9
89.0
$7.5
83.6
95.3
83.2
81.7
51.5
79.5
75.8
Test
43.8
50.1
69.6
81.6
71.2
82.6
$4.6
91.7
89.7
88.0
95.2
90.5
83.5
49.0
FATA
49.8
82.1
71.0
82.3
91.8
95.5
83.4
FOA (F
71.6
88.2
76.4
87.7
91.1
91.6
90.9
89.1
95.2
76.5
81.3
POA (P- 9)
60.9
69.8
75.9
87.6
73.2
85.7
90.4
91.4
90.2
89.5
95.1
$4.9
84.5
68.2
80.5
81.9
CMF
66.2
71.6
71.1
82.8
75.6
87.5
90.5
89.6
90.5
S8.1
94.8
$6.3
83.33
83.3
833.2
833.3
SPA
72.3
79.2
61.8
80.3
69.8
78.9
85.7
80.4
85.2
70.1
91.5
72.7
70.1
69.9
74.9
76.2
PEA
66.0
77.8
88.5
77.4
88.3
95.5
86.9
81:7
Aug
88.6
78.8
88.4
95.8
83.1
No Adapt
31.2
39.5
26-2
66.3
15.6
62.9
70.3
85.8
50.2
84.7
93.0
$1.0
71.3
25.4
09.4
62.5
TENT
63.9
68.5
65.
89.1
67.4
87.4
90.5
87.7
86.4
90.2
93.9
91.4
80.6
83.2
72.2
81.2
EATA
63.9
68.4
65.
89.3
60.3
87.4
90.5
87.4
86.8
90.
95.8
91.4
80.8
82.9
72.0
81.3
MECTA
69.2
64.9
89.6
69.1
68.2
88.1
87.8
90.9
96.5
92.6
83.5
73.2
82.0
88.7
67.4
91.5
L-TTA
64.4
68.9
64.0
88.7
69.3
87.1
90.3
87.4
87.0
90.0
95.3
90.5
80.6
84.0
72.9
81.2
CMF
70.3
75.6
69.6
76.5
68.3
76.6
81.5
82.4
82.9
84.7
89.5
87.3
76.4
81.8
76.4
78.6
PEA
65.5
69.1
73.2
88.1
69.8
86.7
91.1
89.2
88.9
91.9
94.6
93.4
80.9
78.3
73.3
K2.3
PEA
68.3
72,0
75.6
89.1
71.2
87.4
91.2
89.5
89.3
92.3
95.0
93.8
82.0
79.7
74.6
83.4
Table 8: Detailed accuracies (%) on CIFARIO0-C.
Model
Methods
gaMss.
shot
impul
defoc.
glass
motion
200m
Snow
Frost
fog
brigh.
centr:
clast.
pisel.
Avg.
Adapt
42.4
54.1
73.1
47.5
12.8
80.3
57.1
N6.0
44.0
47.7
SAR
44.5
73.0
Tost
43.4
54.8
73.4
47.3
73.0
75.2
80.3
79.5
54.3
86.0
39.8
64.4
46.8
57.2
61.2
EATA
42.5
43.8
54.5
73.2
47.9
73.0
75.0
80.1
79.7
55.6
85.7
42.0
65.1
48.9
57.2
61.6
VIT
FOA (F 27)
FOA (F=
44.5
43.5
48.5
49.0
52.8
55.3
73.5
74..3
55.3
50.5
74.5
75.8
78.0
78.5
81.0
80.5
80.8
31.9
75.3
76.0
87.2
K7.2
76.2
75.4
70.1
70.0
55.6
55.1
60.7
61.3
68.0
67.2
66.0
61.1
76.4
79.5
SPA
20.8
74.2
52.2
77.8
79.2
77.5
86.0
61.7
55.4
71.7
PEA
62.1
64.8
71.5
81.2
65.9
80.0
82.6
82.7
83.6
82.1
87.4
36.4
70/7
20.8
64.4
75.7
PEA Aug
64.3
67.2
74.2
81.9
67.5
80.6
83.0
83.9
84.4
83.2
K7.9
86.8
72.3
72.7
65.6
77.0
No Adaatt
12.5
14,0
5.5
34..3
16.7
36.3
42.8
53.4
16.1
52.0
68.5
39.1
35.7
9.4
32.4
33.5
TENT
31.9
69.5
32.1
32.7
70.2
MECTA
30.2
33,9
29.3
58.4
32.3
55.7
63.1
55.7
51.9
60.8
70.0
61.4
47.3
49.2
10.5
49.7
ResNet
EcOTTA
17.7
22.3
18.8
60.5
23.8
57.1
63.5
55.7
51.9
60.7
70.6
65.9
41.3
43.9
30,0
45.6
L-TA
30.8
34.4
31.2
57.8
34.3
56.1
63.1
57.0
57.0
61.7
70.1
64.9
48.6
53.7
42.5
50.9
CMF
38.6
46.5
37.5
45.8
37.7
46.0
53.1
52.7
53.5
53.0
63.2
584
46.8
54,0
48.8
39.1
60.1
Aug
35.5
36.9
39.1
64.0
30.8
60.8
67.1
60.7
60.4
66.6
74.7
68.7
51.5
52.3
41.2
Table presents the detailed results on CIFARIO-C. Our method, PEA, consistently outperforms prior baselines across most corruption types for both ViT and ResNet backbones. Notably, PEA + Aug achieves the highest overall accuracy, benefiting from robust aligmment and enriched feature diversity. The improvement is especially pronounced under severe corruptions such impulse noise and pixelate, where domain shifts are more extreme.
Table 8 shows the corresponding breakdown for CIFAR 100-C. Similar trends are observed: PEA and its augmented variant deliver consistent gains across nearly all corruption types. On both VIT and ResNet, PEA Aug achieves the best performance on most corruptions, highlighting the strength of our progressive alignment and augmentation strategy.
These detailed results further demonstrate that method generalizes well across wide range of perturbation types, offering both strong average performance and consistent robustness under diverse corruption scenarios.
D.2 VISUALIZATION OF THE ALIGNED FEATURES
To better understand how our proposed PEA progressively corrects domain shifts across layers, we visualize the intermediate embeddings of three representative classes- -ine-ppineeee tree, bicycle, and bee from CIFARI00-C under the Contrast corruption. The upper row Figure shows the feature distributions of the source (circle markers) and shifted domain samples (triangle markers) at three representative layers (blockl, block6, and blockl I), along with their respective class centroids (stars for source and hexagons for domain). The visualizations clearly reveal that the domain features drift away from the source distributions in all intermediate layers, manifesting as embedding translation, scaling, and rotational shifts.
The lower row presents the domain features after applying our adaptation approach. We observe that the domain clusters become progressively more compact and align closely with the source clusters,
18

---

Published as a conference paper at ICLR 2026
59 (src)
59 (dom)
8 (src)
8 (dom)
6 (src)
6 (dom)
* Src class centroid
Dom class centroid
blockl (src vs shifted)
block6 (src vs shifted)
block11
(src vs shifted)
block1 (src and aligned)
block6 (src and aligned)
block11 (src and aligned)
Figure 6: Visualization of intermediate embeddings before and after PEA adaptation on CIFARI00 C. We display the features of the pine tree, bicycle, and bee classes under the CIFAR 100-C Contrast corruption.
Table 9: Detailed accuracies (%) under batch size of 4.
Model
Method
No Adapt
SAR 63.0
Tent 62.0
EATA 62.2
POA (F=
CMF 62.0
SPA 62.6
PEA 63.3
ViT-Base
55.5
62.0
No Adapt
TENT 1.7
EATA 18.5
MECTA 28.2
ECOTTA 1.2
L-TTA 34,5
CMF 17.6
PEA
ResNet-50
27.8
41.7
and the class centroids from the domains converge. Notably, by the final block (block1 I), the previously shrinking domain features are largely pulled back to their original source positions. These results demonstrate that PEA can systematically reduce the three types of distortions across and successfully realign the feature spaces to the source distribution.
D.3 SMALL BATCH SIZE COMPARISON
Table 9 provides detailed comparison PEA against existing TTA baselines under a severely con- strained setting (batch size = 4). While conventional entropy-minimizztion methods such as SAR, Tent, and EATA provide consistent improvements around 62%, and FOA (F 9) reaches similar accuracy, our method further improves performance to 63.3%. For the ResNet-50 backbone, the difference between methods even more striking. Without adaptation, the baseline model achieves 27.8% accuracy. Most minioziiiinnnizztioon methods either fail or perform poorly. While MECTA and I.-TTA achieve moderate improvements, they still lag behind our method, which attains 41.7%.
The primary reason for the failure on ResNet is that most existing TTA baselines rely updat- ing BatchNorm (BN) statistics. This process requires sufficiently large batch sizes to estimate sta- ble mean and variance values; otherwise, the updates become noisy and cause severe performance degradation. Under small-batch regimes, such as those common on edge devices, these baselines therefore collapse in accuracy. In contrast, PEA avoids this limitation by depending BN up- dates or backpropagation. Instead, it realigns embeddings using pre-computed source statistics and lightweight covariance alignment, which remain stable with very small batches. This design
19

---

Published as a conference paper at ICLR 2026
Table 10: Results on BS=1 and BS=2 on CIFARI00-C and ImageNet-C with ViT-Base. indi- cates that the method fails under BS=1.)
ImageNet-C
CIFAR100-C
Method
Batch Size
Batch Size =
Batch Size
Batch Size
No Adapt
55.5
55,5
61.6
61.6
SAR
58.6
62.0
62.0
64.2
CMF
F
61.5
F
61.5
FOA (K-9)
F
64.4
F
66.9
SPA
59.5
60.8
46.5
70.6
PEA
61.6
62.5
69.5
69.9
Table 11: Effect of momentum m and entropy threshold fen on average accuracy (%) on CIFAR100-
C. Each cell reports accuracy with the average number of entropy spikes (+).
Accuracy (%) (Spikes) 6ent
m
0.5
0.8
1.0
1.5
0.01
75.3 (16)
75.4(9)
75.0 (7)
74.6 (2)
0.02
75.7 (17)
75.8
75.5
75.5 (2)
0.05
75.7 (17)
75.7 (6)
75.7
75.7 (2)
0.10
75.4 (17)
75.3 (8)
72)
75.3 (1)
makes PEA inherently more robust under constrained batch sizes, ensuring consistent performance
across both CNN and ViT backbones.
D.4 STREAMING EVALUATION WITH EXTREMELY SMALL BATCHES (BS=1/2)
To reflect realistic streaming deployment where test inputs arrive (or few) at time,
evaluate all methods under extremely small batch sizes (BS=1 and BS=2). This regime is particularly challenging for test-time adaptation because reliable batch statistics difficult estimate and some methods become numerically unstable or memory-inefficient when the batch degenerates to single sample. Table 10 summarizes the results on ImageNet-C and CIFAR100-C with ViT-Base.
PEA remains stable and consistently improves over the non-adapted source model in both datasets, achieving strong performance even at BS=1 (e.g., 61.6 on ImageNet-C and 69.5 on CIFARI00 C). In contrast, several baselines fail at BS=1 (e.g., CMF and FOA), highlighting their reliance on larger batches iterative test-time optimization. These results validate PEA's suitability for stream- ing test-time inference: by performing lightweight, statistics-driver progressive alignment without backpropagation, PEA can operate reliably when only minimal batch information available.
D.5
MORE RESULTS OF ABLATION STUDY
D.5.1 HYPERPARAMETERS
We further analyze the sensitivity of our method to the two hyperparameters: the EMA momentum m, which controls the update rate of domain statistics, and the entropy threshold 0ems which is used for detecting distribution shifts. To this end, we sweep range of values for both parameters and report the average accuracy across the 15 CIFAR100-C domains, together with the number of entropy spikes triggered during evaluation, as shown in Table 11.
The results show that very small momenta (e.g., m 0.01) underperform with accuracies around 75.0-75.4%, while moderate values of m 0.02-0.05 consistently achieve the best performance (75.5-75.85 88)). Larger momentum (y = 0.10) again reduces accuracy to about 75.3-75.49 44.. For the entropy threshold, low values such as 0cnt 0.5 lead to frequent resets (16-17 spikes), whereas high values like 6mm = 1.5 almost disable resets (1-2 spikes). Accuracy remains stable across thresholds with differences within 1 1%, but excessively low thresholds slightly degrade performance due to too many resets, while overly high thresholds risk ignoring meaningful shifts. The balanced setting of Pen 0.8-1.0 1.0 achieves both high accuracy 75.5%%..% and moderate spike counts.
20

---

Published as a conference paper at ICLR 2026
Table 12: Impact of the fraction of source data used to adaptation accuracy(%).
Source Data Used
CIFARIO-C VIT
CIFARI00-C ViT
5%
83.240.1
76.420.2
10%
83.240.1
76.8+0.2
20%
83.420.1
77.0=0.1
50%
83.4+0.1
77.0+0.1
100%
83.420.1
77.0.0.1
Table 13: Impact of the number of test-time augmentation views K top-1 accuracy (%).
Dataset
K = 0
K
K 2
K
K = 4
ImageNet-C
64.5
66.2
66.5
66.6
66.6
CIFAR100-C 75.7
76.6
77.0
77.0
77.1
In conclusion, method is not highly sensitive these hyperparameters, with accuracy variation contained within about 1 1%. For all main experiments, we adopt m 0.02 and Pem = 1.0 as they provide the best trade-off between responsiveness and stability.
D.5.2 SENSITIVITY TO SOURCE DATA SIZE FOR OFFLINE STATISTICS
PEA requires access to a small amount of source data before deployment to estimate and store per-block source statistics. To understand how much source data is needed in practice, we vary the fraction of source samples used in this offline stage and evaluate the resulting adaptation accuracy. As shown in Table 12, PEA is largely insensitive to the amount of source data used to compute the offline statistics. On CIFAR performance saturates quickly even with 5% of source data. On CIFAR 100-C, using only 10% source samples already achieves near-saturated performance (76.8% 77.0% with 100%), and 20% matches the full-data result. Importantly, this preprocessing is performed once and entirely offline; at test time, PEA only relies on the stored statistics and never accesses source data again.
D.5.3 NUMBER OF VIEWS
In PEA, the data augmentation will add more views (K) of the data capture accurate statistics. Our default setting in the paper is K = 2. We add more ablation on the number of augmentation views from K augmentation consistently improves performance over = O (no augmentation), with the largest gain from K 0 to K K = 2, the improvements saturate and become marginal (;0.1%) 1%). Since increasing K also increases compute and memory, we adopt 2 as the default capture most of the benefit of augmentation while keeping the overhead modest.
4 in the following Table 13. We observe that multi-view
and a smaller but still clear gain from K
Beyond
E DOMAIN SHIFT IN RESNET EMBEDDING SPACE
In this section, we illustrate that the embedding-space shift for ResNet is consistent with that ob- served in VIT. We visualize the embedding space of ResNet-50 on CIFARIO-C and observe the same systematic shifts in mean, variance, and channel-wise covariance across layers. Specifically, Figure and Figure 8 show the Zoom Blur and Frost domains, respectively, where the embedding shifts closely match those in VIT, indicating that the effect driven by domain shift rather
21

---

Published as a conference paper at ICLR 2026
Class 1 (src)
Class (dom)
Class (src)
Class 2 (dom)
Class 3 (src)
Class (dom)
layer1_ biko
layer1 blkk
layer2 blk00
layer2_blke bikk
layer3 bik0
layer3 b_blk2
layer3_blke bikk
layer4 bik0
layer4 blk2
Figure 7: Impact of domain shift intermediate layer embeddings using CIFAR100-C. We visu- alize the features in Domain Zoom Blur for ResNet-50.
Class 1 (src)
Class (dom)
Class (src)
Class 2 (dom)
Class 3 (src)
Class (dom)
layer1, biko
layer1 bik2
layer2 bik0
layer2_blk bik3
layer3_blke
layer3 blk2
layer3_blks
layer4_brl biko
layer4
Figure 8: Impact of domain shift intermediate layer embeddings using CIFAR100-C. We visu- alize the features in Domain Frost for ResNet-50.
22

---

Published as a conference paper at ICLR 2026
Class 1 (src)
Class (dom)
Class (src)
Class 2 (dom)
Class 3 (src)
Class (dom)
block1
block2
block3
block4
block5
block6
block7
block8
block9
Figure 9: Impact of mixed domains on intermediate- layer embeddings on CIFAR100-C. We visu- alize ViT-Base features for classes.
than architectural design. This aligns with our experimental results, where PEA also improves CNN backbones (ResNet-50) on CIFAR10-C/100-C and ImageNet-C.
F MIXED-DOMAIN EMBEDDING SHIFT VISUALIZATION
To better understand why PEA remains effective under the mixed-domain scenario, we visualize the intermediate embeddings of ViT-Base on CIFARI00-2 in a mixed-domain setting, Specifically, we construct mixed CIFAR100-C stream where all corruptions (severity 5) are combined and shuffled, so that each batch contains samples from multiple domains. For subset of 3 classes, we extract features from 9 ViT blocks (from shallow to deep) and jointly project the clean and mixed- domain embeddings. Figure 9 shows the resulting embeddings for all 9 blocks.
As expected, compared the clean case, mixed-domain embeddings exhibit lower intra-class com- pactness and increased spread, since each class now aggregates samples from heterogeneous cor- ruptions. However, the key observation is that the distortion is still highly systematic across layers. This matches our hypothesis that domain shift, even when composed of multiple human- "domains" " (fog, snow, blur, etc.), induces a geometric shift in the embedding space.
G USE OF LLM IN THIS PAPER
We emphasize that large language models (LLMs) were used solely for polishing the writing and improving readability. No part of the technical content, experimental design, analysis, results relied on L.M--generted material. All research ideas, implementations, and evaluations are original to the authors.
23


Reviews:
[{"review_id": "FF7TRncUaF", "reviewer": "Reviewer_mDae", "summary": "This paper proposes **Progressive Embedding Alignment (PEA)**, a novel Test-Time Adaptation (TTA) method that is **backpropagation-free**, **architecture-agnostic**, and highly efficient. The authors aim to overcome two major limitations of existing TTA work: 1) the high computational and memory overhead of backpropagation-based methods (e.g., Tent, EATA), which makes them unsuitable for edge devices , and 2) the architecture-specificity of recent efficient methods, which are often tailored for either CNNs or ViTs, but not both.\n\nThe core idea is motivated by a principled analysis of domain shift. The authors empirically demonstrate that domain shifts induce a combination of three consistent geometric distortions in the intermediate embedding space:\n1.  **Translation** (mean shift) \n2.  **Scaling** (variance shift) \n3.  **Rotation** (channel-wise covariance shift) \n\nPEA works by explicitly correcting these distortions. In an offline step, it pre-computes and stores the source-domain statistics (mean $\\mu_{s,l}$ and covariance $\\Sigma_{s,l}$) for each block $l$. Then, at test time, it uses a two-pass, forward-only procedure:\n* **Pass 1:** Estimates the statistical distance between the current batch and the source to compute a layer-wise alignment weight $w_l$.\n* **Pass 2:** Applies a Whitening-Coloring Transform (WCT) to realign the batch features to the source statistics. The final, corrected feature $F_l'$ is a weighted blend of the original and aligned features, $F_{l}^{\\prime}=(1-w_{l})F_{l}+w_{l}Y_{l}$, which prevents over-correction .", "questions": "- Are recent methods used to reduce the overhead of backprop only tied to vit or cnn or have high latency? if so can you share related work to that?\n\n-  In equation 1 why you didn't use the mahalanobis distance between each input in the batch and the source distribution instead of relying on the batch statistics and in that case even for bs 1 the method shall work preoperly out of the box without having to think about the scale difference between mean and variance as the current distance can be overwhelmed by one measure over the other (if we have more mean shift or more variance). \n\n- Can you ablate the choice of distance and what happens if we simply use the shift in mean ?\n\n- how do you select the fixed threshold used to flag spikes in feature alignment\n\n- Can you ablate the choice of k views augmentaion to see its effect on the main results ?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Clear Motivation and Novelty: The analysis of domain shift as a geometric transformation (translation, scaling, rotation) in the embedding space is intuitive and provides a strong foundation for the proposed method. The idea of reversing these distortions via a weighted WCT, without any gradient updates, is a novel and elegant approach to TTA.\n\n- Exceptional Efficiency (Memory and Latency): The paper's primary contribution is its practicality. By being backprop-free and requiring only two forward passes , PEA is significantly more efficient than SOTA methods like CMF, SPA, or EATA. This is powerfully demonstrated in the edge device evaluation (Table 4), where PEA runs successfully on a Jetson Orin Nano, while numerous backprop-based baselines (Tent, EATA, CMF, etc.) fail due to OOM errors.\n\n- Strong Empirical Results: PEA (especially with augmentation) achieves state-of-the-art or highly competitive accuracy on all three benchmarks (ImageNet-C, CIFAR10-C, CIFAR100-C). \n\n- Generality (Architecture-Agnostic): This is a key advantage. The method works \"out-of-the-box\" for both CNNs (ResNet-50) and Transformers (ViT-Base) using the identical procedure", "weaknesses": "- Source Data Requirement: The method's primary limitation, which is acknowledged by the authors26, is that it is not \"source-free.\" It requires access to the original source training dataset in the offline stage to pre-compute the source statistics ($\\mu_{s,l}$, $\\Sigma_{s,l}$)27. While this is a valid setup for TTA, it is a stronger assumption than methods that only require the pre-trained model.\n\n-  Storage Overhead: While the inference compute/memory is low, PEA introduces a storage cost by requiring the mean vector and covariance matrix for each aligned block to be stored. The paper mentions this is minimal (~30MB for ViT-Base), but this cost should be explicitly reported for all models in the main tables to provide a complete picture of the resource trade-offs.\n\n- Statistics Estimation: The method's effectiveness hinges on the quality of the estimated test-time statistics ($\\mu_{t,l}$, $\\Sigma_{t,l}$)28. While the EMA helps, the authors note that performance could degrade with extremely small batches (e.g., BS=1) or high class imbalance29. A formal evaluation at BS=1, a true \"online\" setting, would be valuable to fully test this boundary condition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qe6gZvicHa", "reviewer": "Reviewer_bwXx", "summary": "This paper proposes Progressive Embedding Alignment (PEA), a backpropagation-free and architecture-agnostic Test-Time Adaptation (TTA) method. The key idea is to interpret domain shifts as three geometric transformations in the embedding space—translation (mean shift), scaling (variance shift), and rotation (covariance shift)—and correct them progressively via Whitening-Coloring Transform (WCT)–based covariance alignment. PEA performs only two forward passes per test batch, without modifying model weights, and can be applied to both CNNs and ViTs. Experiments on CIFAR10-C/100-C and ImageNet-C show that PEA achieves competitive accuracy with minimal memory and latency, demonstrating its suitability for real-time or edge deployment.", "questions": "How to decide the threshold for detecting domain changes (Eq. 7)  across different models and domains during online testing? An ablation study (with and without Eq.7) and sensitivity analysis on this threshold would be helpful.\n\nHow does the number of augmented views in PEA affect performance? More detailed sensitivity analyses on this factor are preferred.\n\nHow does PEA perform under mixed domain shifts? It would be helpful to see results for both CNN and ViT backbones in such scenarios.\n\nThe reported results of SPA appear to be lower than those in the original paper. Could the authors clarify this discrepancy? Since SPA is a strong backpropagation-based method, it is understandable that PEA (focus on forward-only) has lower performance than SPA, but ensuring consistent and accurate SPA results would make the comparison more convincing.\n\nSuggestion:\n\nFOA also includes an alignment component, namely “activation shifting,” which appears conceptually related to the proposed PEA. However, PEA performs alignment in a more fine-grained, layer-wise, and comprehensive manner. It would be much better to discuss this relationship in the paper.\n\nI would like to consider increasing my score if these are well addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The embedding-space interpretation of domain shift (translation, scaling, rotation) is elegant and intuitive. It provides a clear geometric motivation for the proposed covariance alignment strategy.\n\nPEA is architecture-agnostic for both CNNs and ViTs.\n\nThe fully forward-only, memory-light, and backprop-free natures make PEA ideal for edge devices. It achieves low latency (~0.3s) and small memory footprint (<1GB) while maintaining strong accuracy.\n\nDevice-level tests on Jetson Orin Nano enhance practical relevance.", "weaknesses": "As the proposed method relies on statistical estimation, its performance degrades under small batch sizes. And, I am wondering how does the method performs when the batch size is as small as 2? Moreover, with augmentation enabled, could PEA still operate effectively with a batch size of 1?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OmZVEQ4TOd", "reviewer": "Reviewer_9W85", "summary": "This work proposes PEA, a backpropagation-free and architecture-agnostic test-time adaptation (TTA) approach that aims to enhance adaptation while reducing computational bottlenecks. Unlike most TTA methods that rely on backpropagation—which is computationally expensive at test time—or are architecture-specific, PEA provides a general alternative. The authors observe that the representations of shifted domain samples across layers are geometric transformations of the source representations, primarily translation, scaling, and rotation. Building on this insight, PEA performs two forward passes: the first collects batch statistics per layer, and the second applies a correction based on source statistics. Experimental evaluations on ResNet-50 and ViT-B across corrupted datasets (CIFAR-10-C, CIFAR-100-C, and ImageNet-C) demonstrate accuracy gains when combined with test-time augmentation. Additional experiments on low-resource devices further highlight the method’s memory efficiency.", "questions": "* Class representations appear already well separable after only 3 blocks (out of 9). How do you explain this observation?\n\n* Line 167: “This systematic shift does not lead to a random feature distortion; rather, it manifests as a geometric rotation and shearing of the entire feature cloud.” Why would we expect a random feature distortion in the first place?\n\n* In pass1 paragraph of line 290: How are K views of multiple augmentations obtained within a single forward pass? Please clarify this point.\n\n* For clarity, in the main text, please specify what a “block” corresponds to in the ViT architecture, and explicitly indicate that the referenced block 3 is the third out of nine blocks.\n\n* Please cite prior work using test-time augmentation (e.g., Simonyan and Zisserman, 2014).\n\nReference\n\nSimonyan, Karen, and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is well written and easy to follow.\n\n* The paper proposes a method with two interesting properties: backpropagation-free and architecture-agnostic, which increases the applicability of TTA across different models and devices.\n\n* The evaluation on low-resource devices is a valuable contribution, highlighting the method’s efficiency in practical settings.", "weaknesses": "* The paper emphasizes the existence of discrepancies in representations between source and target distributions and their variation across layers. However, this adds limited new understanding. By definition, distribution shifts alter centroids and other statistical descriptors, and much of the domain adaptation literature already focuses on aligning source and target distributions.\n\n* The hypothesis that shifts reduce to simple geometric transformations in embedding space is overly strong, and the visualizations in Figure 1 and Appendix A do not provide convincing evidence. For example, as long as representations differ, a translation between centroids is always possible and therefore not surprising. Moreover, these are linear operations, and evidence from only two transformations is insufficient to claim generality. Non-linear or more complex transformations may well occur in intermediate representations, and the current evidence does not rule this out.\n\n* A critical assumption is that source statistics for all layers are accessible at test time. This may not hold in many realistic scenarios and sets the method apart from approaches that do not make such an assumption. This should be acknowledged explicitly as a limitation.\n\n* The conclusions in lines 160–163 and 139-141 overlook prior fine-grained analyses, such as Sahoo et al. and Maharana et al., which emphasize that not all layers are equally affected and advocate for layer-wise adaptation. \n\n\n* The detection mechanism based on the entropy of predictions is limited. Entropy is a weak OOD detection score compared to alternatives available in the literature (e.g., OpenOOD benchmark).\n\nReferences\n\nSahoo, Sabyasachi, et al. \"A layer selection approach to test time adaptation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 19. 2025.\n\nMaharana, Sarthak Kumar, Baoming Zhang, and Yunhui Guo. \"Palm: Pushing adaptive learning rate mechanisms for continual test-time adaptation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 18. 2025.\n\nOpenOOD benchmark: https://github.com/Jingkang50/OpenOOD", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PzJtCN65lJ", "reviewer": "Reviewer_Lsas", "summary": "The authors propose a new backpropagation-free test-time adaptation method to calibrate the pre-trained model to the target domains using two forward passes per sample. The authors observe that the distribution shift can cause three distinct structural changes in the features of intermediate layers. Based on this observation, they design a Progressive Embedding Alignment (PEA) to correct the feature distortions. One forward is used to estimate the level of distribution shift, and the other is used to recompute the intermediate features to tackle the distribution shift. Moreover, they also use the exponential moving average and data argumentation to enable stable and robust adaptation. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors empirically reveal that the distribution shift can cause the three distinct structural changes in the feature space, including translation (mean shift), scaling (variance shift), and rotation (covariance shift).\n2. The authors propose a Progressive Embedding Alignment (PEA) to calibrate the feature without updating the parameters of the model, which can originally mitigate catastrophic forgetting. \n3. The results of the ablation study demonstrate the necessity of each component in the proposed method.", "weaknesses": "1. The authors propose a feature re-computation method to calibrate the pre-trained model. However, a similar idea that updates the intermediate features has been proposed by the previous method[A]. What is the essential difference and advantage of the proposed method? \n2. The authors show the distinct structural changes of the ViT model in Figure 1. However, it is unclear whether this observation is related to this specific architecture. More discussion or experimental results on CNN models or models with different normalization layers should be provided.\n3. The proposed method requires computing the statistics of a batch of test samples, so the distribution of different domains in the same batch will greatly affect the accuracy of the statistical distribution. When the test samples within the same batch come from different domains (mixed data domains), the proposed method may suffer from unreliable distribution estimation and cannot guarantee a stable performance.\n4. The Spike detection is proposed to detect the domain changes and then reset the estimated statistical information for the new domain. However, it is unclear how to decide the threshold \\theta_{ent} for different domains, especially under challenging test scenarios such as imbalanced label shifts per SAR.\n5. The data enrichment is designed to enhance the estimation of the target batch distribution by generating K augmented views for each test sample. However, it is unclear how the hyperparameter K affects the adaptation performance and the computational cost.\n6. More BP-free test-time adaptation methods should be discussed and compared in the experiment, including but not limited to [B-D].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)