TRUST: Test-Time Refinement using Uncertainty-Guided SSM Traverses
Authors: Sahar Dastani
Ali Bahri
Gustavo Adolfo Vargas Hakim
Moslem Yazdanpanah
Mehrdad Noori
David OSOWIECHI
Samuel Barbeau
Ismail Ben Ayed
Herve Lombaert
Christian Desrosiers
Venue: NeurIPS 2025 poster ()
Keywords: Structured State-Space Models
VMamba
Test-Time Adaptation
Distribution Shift
Primary area: deep_learning

Abstract: State Space Models (SSMs) have emerged as efficient alternatives to Vision Transformers (ViTs), with VMamba standing out as a pioneering architecture designed for vision tasks. However, their generalization performance degrades significantly under distribution shifts. To address this limitation, we propose TRUST (Test-Time Refinement using Uncertainty-Guided SSM Traverses), a novel test-time adaptation (TTA) method that leverages diverse traversal permutations to generate multiple causal perspectives of the input image. Model predictions serve as pseudo-labels to guide updates of the Mamba-specific parameters, and the adapted weights are averaged to integrate the learned information across traversal scans. Altogether, TRUST is the first approach that explicitly leverages the unique architectural properties of SSMs for adaptation. Experiments on seven benchmarks show that TRUST consistently improves robustness and outperforms existing TTA methods.

TRUST: Test-Time Refinement using Uncertainty-Guided SSM Traverses
Sahar Dastani 1.2 1.2+
Ali Bahri
Gustavo Adolf Vargas Hakim
Moslem Yazdanpanahl
Mehrdad Noori'
David Osowiechi! Herve Lombaert
Samuel Barbeau'
Ismail Ben Ayed
Christian Desrosiers'
LIVIA, ILLS, ÉTS Montréal, Canada,
Mila Quebec Al Institute, PPolytechnique Montreal
Abstract
State Space Models (SSMs) have emerged as efficient alternatives to Vision Trans- formers (ViTs), with VMamba standing out as pioneering architecture designed for vision tasks. However, their generalization performance degrades significantly under distribution shifts. To address this limitation, we propose TRUST (Test- ffest-Time Refinement using Uncertainty-Guut Guided SSM Traverses), a novel test-time adaptation (TTA) method that leverages diverse traversal permutations to generate multiple causal perspectives of the input image. Model predictions serve as pseudo-labels to guide updates of the Mamba-specific parameters, and the adapted weights are aver- aged to integrate the learned information across traversal scans. Altogether, TRUST is the first approach that explicitly leverages the unique architectural properties SSMs for adaptation. Experiments on seven benchmarks show that TRUST consistently improves robustness and outperforms existing TTA methods. The code is available at: https:/ //gi gitt..u..c. hatm/Saaarastaantruunn
Introduction
The field of visual representation learning has advanced rapidly due to the ability of deep neural networks to extract rich and generalizable features. Convolutional Neural Networks (CNNs) [1, 2, 3, 4, 5] excel in modeling local patterns through strong inductive biases but struggle with global context. Vision Transformers (ViTs) [6. 7, 8, 9] address this challenge using a self-attention mechanism, although at higher computational cost. More recently, State Space Models (SSMs) [10, 11, 12] emerged as scalable alternative, providing global receptive fields with linear complexity. Despite their efficiency, the performance of SSMs degrades under distribution shift. This is mainly due to violations the Independently and Identically Distributed (i.i.d) assumption, which is often disrupted in real-world settings. While generalization strategies well developed for CNNs and ViT's, vision-specific strategies for SSMs are still lacking-
This work focuses on VMamba [13], vision-adapted variant of the Mamba architecture [14], designed for sequential visual processing. VMamba introduces 2D Selective Scan (SS2D), four-way traversal mechanism that scans image patches along predefined spatial directions. However, this directional processing introduces strong inductive bias by aligning internal representations with fixed traversal paths [15], which may hinder generalization under distribution shifts. Additionally, the hidden states of VMamba store historical context over the traversal sequence. When exposed to unseen domains, this context may accumulate domain-spectin artifacts, leading to amplified bias during propagation and ultimately degrading generalization [16].
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To address these challenges. we propose Test-Time Refinement using Uncertainty-Guu Guided SSM Traverses (TRUST), a novel Test Time Adaptation (TTA) strategy specifically designed for SSMs that leverages VMamba's 's internal traversal mechanism. As illustrated in the offline phase of Figure 1, different traversal permutations result in varying prediction entropy levels, revealing the sensitivity of the model to causal ordering. TRUST systematically generates multiple traversal permutations by reordering the four directional scans, exposing the model to diverse causal perspectives of the same input, At test time, we compute the entropy of predictions from each permutation and select the ones with the lowest entropy, which are associated with more stable and domain-robust hidden states.
Inspired by previous work on flat minima and weight averaging [17, 18], our method aggregates the outputs of different traversal permutations to implicitly explore a broader and flatter region of the loss landscape. Each permutation activates a distinct computational pathway through the model, effectively sampling different local minima. Selecting the top-k traversal permutations with the lowest predictive entropy and computing a weighted average of their model parameters mitigates the impact of noisy or uncertain predictions. This enhances generalization without requiring source data or model retraining.
We outline the main contributions of our work as follows:
We propose TRUST, the first TTA approach specifically designed for Namba-based vision mod- els. Unlike prior methods that depend on data augmentation or auxiliary models, our approach takes advantage of the internal traversal dynamics of VMamba to improve generalization under distribution shift.
TRUST introduces novel weighted averaging strategy to promote robustness, which permutes traversal directions and averages model weights from the most confident paths.
We validate our method on seven standard benchmarks, where it consistently outperforms existing TTA methods, establishing new state-of-the-art results.
2
Related Work
Test Time Adaptation (TTA) a paradigm of Domain Adaptation (DA) [19] that intends to enhance model generalization during deployment. Specifically, a pre-trained model is adapted to incoming data batches without requiring label supervision or having access source data. As a pioncering approach, Tent [20] minimizes prediction entropy an adaptation objective, based on the principle that distribution shifts in the target domain reduce the confidence of model in its predictions. Examples of Tent- based approaches include using confidence thresholds for sample selection before adaptation [21, 22], minimizing the sharpness of the entropy loss [23], using class-balanced memory queues for better adaptation sampling [24], and meta-learning the entropy loss [25], among other alternatives [26, 27, 28). The literature also includes several other adaptation strategies that do not minimize entropy, such as contrastive learning [29, 30], Laplacian optimization [31, 32], and prototype-based pseudo-labed labeling [33]. While flexible, these techniques do not capitalize on the unique properties of specific architectures such VMamba.
Architecture--seeific TTA. Recent advances in TTA have introduced a variety of model-specific strategies. Among those, FOA [34] loosely adheres to ViTs by incorporating gradient-fre learning of additional prompts in the form of token embeddings. For Vision-language Models (VLMs) [35, 36, 37, 38), recent strategies incorporate text information an auxiliary tool for adaptation. Examples include prompt learning via entropy minimization [35], positive and negative caching with text- based pseudo-labels [36], transductive adaptation using conformal pseudo-captions [38), and weight averaging across text templates [37]. SSM-oriented TTA has also been explored to smaller extent. In recent work, STAD [39] uses Markov processes to learn time-varying prototypes for classifying examples during inference and adapts the classification head of the model.
Weight Averaging for Robust Adaptation. Initially introduced in the context of Domain Gener- alization (DG) [40, 41, 42], weight averaging has gained popularity as an effective technique for enhancing model robustness to domain shifts [43, 18]. Notably, methods like SWAD [18] aim to identify flat minima in the loss landscape by minimizing loss fluctuations across diverse inputs. This is often accomplished by averaging the weights of models trained on different image augmentations. While similar ideas have been adopted in the TTA literature [23, 37, 44, 45, 46), they typically rely on explicit input diversification, which introduces additional computational overhead. In contrast,

---

Offine
Adaptation Phose
Evaluation
Cross Scan
Reondaring
56 Block
Cross
Pressl Dien
Wi
Nor Normolization Layer
Feed Forward Network
M
Nor woltaaattiionn
MI May
C
Weight
averaged
bant
W
W
Ioad
Faue USs
Fll Co.g: Uloo
VSS Block
fu- catd
Dx rboe
vas Block
2-5
cleasPar
weight Space
Weight Aversogin
VSS Block
clucs
clossafer
Selecbe
Input Patches
VSS Block
25
dleesierr
Hip Marror Ric
TRUST
Figure 1: An overview of the proposed method. Our network consists of three stages: offline, adaptation, and evaluation. In the Offline stage, multiple traversal permutations are generated and ranked by entropy. The top-K most confident permutations are selected. During adaptation, each permutation reorders the cross-scan traversals, and the Mamba-specific state-space parameters are updated accordingly. The resulting models are then merged via parameter-space averaging. The final averaged model used for inference in evaluation stage.
approach leverages the intrinsic diversity of SSM-baser traversals to induce variation directly in the weights our model, without requiring external data augmentations.
Method
Section 3.1 introduces key concepts in TTA and SSM, laying the groundwork for our approach. In Section 3.2, we propose traversal permutations generate complementary causal perspectives of the input, followed by a weight averaging strategy in Section 3.3 for robust and efficient inference. Figure shows an overview of our method.
3.1 Preliminaries
Test Time Adaptation (TTA) in Image Classification. Consider classification model composed of a feature processor fe. parameterized by 6, and a classifier h& parameterized by Φ. The model is pre-trained on source dataset D, = {(X,, 9z) N, ) consisting of images X, E RW H3 and corresponding labels yj {1...., C}. where C' is the number of classes. Formally, the model learns a map F, X. >> Vr, between the input distribution X, and output distribution ), from the source. Training D. is performed using a supervised loss Liain (typically cross-entropy). At test-time, the model is exposed to target dataset D. [X;] N, NNN
N,,N,1 +1. which lacks labels and follows different distribution, giving rise to the map F; Xr - D/. Although the source and target datasets share the same label space (),s Vd),, their joint distribution differ, i.e., P(X,, V.) + P(X,)))) Such difference is referred to as covariate domain shift [31].
State Space Model (SSM) Formulation. SSM-based models [10, 14] map 1D input sequence x(t) E IR to an output (t) E through a learnable hidden state h(t) E R. governed by linear dynamical system:
h'(() Ah(() + Bx((), y(1) = Ch(l),
(1)
where A E NNN, B Nx1 and R are learnable parameters. To integrate this formulation into deep learning, it discretized using the zero-order hold (ZOH) method with step
3

---

size A E R, resulting in the discrete recurrence:
h(t) Ah(t - 1) + Bx(t), y(t) Ch(t), where A = exx(AA), B=AB. (2)
Unrolling this recurrence over time yields ID convolution with kernel K E TR,, where I. E N denotes the kernel length. This form enables efficient parallel computation during training.
== x= K (CB, CAB CA CA'"BB) )
SS2D for 2D Visual Processing. To extend 1D state-space models to visual data, VMamba
introduces the SS2D module, which reshapes an input image X E " into a sequence of T non-overlapping patches (x, (x))[1- As shown in Fig. 1, these patches traversed in a predefined causal order determined by one of four canonical directions using the Cross-Scan module: left-to- (d). For each direction, the patch sequence is processed recurrently by a shared, discretized ID SSM, ensuring causality and directional consistency, The outputs from all four scans are then aggregated using the Cross-Merge module.
right (a), top-to-bottom (b), right-to-left (c), and bottom-to-top
3.2 Traversal Permutation
To address the limitations of VMamba under distribution shifts, we propose a TTA strategy that lever- ages VMamba's 's intrinsic directional traversal mechanism. We define set of traversal permutations
p = [2,,52..., ANJ.
(4)
where each n, represents a unique ordering of the four canonical sean directions, with m {a, b, c, d} corresponding to the original ordering used in VMamba (see Fig. for the definition of directions a, b, e and d). We assess the predictive confidence of the model under each traversal permutation T, P by computing the Shannon entropy of its output distribution. Permutations are then ranked in ascending order of entropy, and the top-K permutations with the lowest entropy are selected (the offline mode of Figure 1):
Prolected (T Teg
Tsk }.
(5)
During the adaptation phase, for each selected permutation Tip 1, 1,..., K, the input is processed according to the corresponding traversal order. Denoting p(X; ni, [0, 1]c as the output of the model for an image X when using traversal permutation Ti, we compute pseudo-label ýs by taking the class with maximum predicted probability:
Uk =
arg max (I......C)
[p(X; mi))
The Mamba-specific parameters are then updated to minimize the average cross-entropy (log loss) between the model prediction and the corresponding pseudo-label over each target sample in batch B:
θ arg min
1
Σ
log [p(X; Tie )1p.
0
B
XEB
This is achieved using propapropng as in standard methods. After optimization, the adapted parameter set 02 is cached, allowing a subsequent ensemble-based aggregation at inference.
Our method processes the same image through multiple directional permutations, enabling VMamba exploit complementary causal views of the input. These distinct trajectories expose the model to both global consistency (identical token set) and local variation (different hidden-stee statee evolutions). which helps find flatter minima offering a stronger out-of-distribute Crucially, re-ordering prevents Nomain-specifie artifacts from always entering the recurrence at the same time step. If artifact is embedded in a corrupted patch X,,, then under the default traversal this patch always appears at time step = fz in the processing sequence. The hidden state update that step becomes:
(OOD) generalization [18].
h (tte) f (t= 1), x., + +e)
Since the hidden state h() is recurrent and accumulates over time, injecting early in the sequence allows its influence to propagate through many subsequent updates. Therefore, the impact of the artifact depends heavily on the position of ta within the sequence. In our method, we apply traversal

---

permutation usg. which changes the order in which patches are processed. Under Tee the corrupted patch x,, appears at time step = Ti, (t&), leading to the updated hidden state:
(Th (te))
(3) (te) 1), x +6).
Across different permutations, the corrupted patch enters the sequence at varying time steps. This shifts the effect of artifact different hidden states, breaking its consistent influence pattern. During test-time adaptation, these variations allow the model to learn diverse responses, and the subsequent weight averaging of adapted models helps suppress artifact-induced bias.
3.3 Leveraging Traversal Permutations Through Weight Averaging
To improve both stability and generalization under distribution shifts, we aggregate the adapted models obtained from the top-K traversal permutations. During the evaluation phase, we utilize the averaged weights
0
A
(10)
K
and evaluate the model using the default traversal path T1. In this setting, each 6g represents an adapted model under distinct traversal permutation. This results in single, consolidated model that captures the benefits of nulti-directional adaptation.
This simple yet effective averaging strategy draws inspiration from the notion of flat minima in the loss landscape [43, 18), where parameter configurations that lie in flat regions tend to exhibit greater generalization and robustness perturbations. Weight averaging encourages convergence region in parameter space with low curvature, thereby reducing sensitivity to input variations and improving performance under distribution shifts.
To further illustrate this point, Figure represents the test loss surface over model parameters under Gaus- sian noise corruption from the ImageNet-C dataset. Each triangle marks model adapted via a different traversal permutation, while the central cross denotes the weight-averaged model. The axes depict linear interpolations between parameter sets, offering a 2D view of the high-dimensional landscape. This visu- alization demonstrates how different traversal per- mutations lead diverse optima, and how weight averaging converges toward smoother, lower-loss region, enhancing robustness and generalization.
3.6420
3,5099
3,5011
3,4544
3,4427
3.4356
3.4245
Figure 2: Loss surface of model parameters.
Unlike traditional TTA approaches that rely on data augmentation, external source data, or auxiliary objectives to generate diverse model variants for weight averaging, approach leverages the internal architecture of VMamba, specifically, its canonical traversal mechanism. By permuting traversal scans, induce structural variability that yields diverse perspectives of the same input, enabling effective adaptation without external augmentation.
4 Experiments
In this section, we present comprehensive evaluation of our proposed method across seven bench- mark datasets. We begin by describing the datasets used for evaluation, followed by implementation details and the baselines employed. We then report main results and conclude with set of ablation studies to analyze key components our approach.
4.1 Datasets
We evaluate the performance and generalization of our method across a wide range of TTA bench- marks, covering corruption robustness and domain generalization. For corruption-based robustness, we use CIFARIO-C [47], CIFAR 100-C [47], and ImageNet-C [47], which apply 15 corruption types (c.g., noise, blur, weather, digital artifacts) at five severity levels. These datasets scale from 10
5

---

(CIFARIO-C) to 100 (CIFARI00-C) to 1000 (ImageNet-C) classes, enabling systematic analysis of model degradation.
For domain generalization, we assess on PACS [48], ImageNet-S [49), ImageNet- V2 [50], and ImageNet-R [51]. PACS includes four visual styles with seven shared classes, using leave-one- domain-out evaluation. ImageNet-S features sketch-style abstractions: ImageNet-V2 provides cleaner, independently collected test set; and ImageNet-R introduces style shifts across 200 classes with diverse artistic renditions. Together, these benchmarks test adaptability to unseen distributions, styles, and abstractions.
4.2 Implementation Details
We perform adaptation exclusively on the Mamba-specific state space parameters within the SS2D module of each VMamba block, while keeping the rest of the model frozen. The base VMamba model trained with batch normalization layers. The adaptation is guided using a pseudo-labeling strategy, where confident model predictions are treated as supervision targets in cross-entropy loss. This enables self- sepe-sueee adaptation without requiring access to ground- -ruth labels. The adaptation proceeds in an online manner, with model updates applied sequentially (i.e., without resetting the weights to their pre-adaptation values). Optimization is performed using the Adam optimizer with learning rate of 10-1 and batch size of 128, ensuring consistent dynamics and fair comparison across benchmarks. All experiments were conducted using single NVIDIA A6000 GPU.
4.3 Baselines
We compare method against several state-of-the-art TTA approaches. Source Only refers to the performance of VMamba without any adaptation. ETA [21] minimizes an entropy loss modulated by admleead weighting mechanism, updating only the affine parameters of normalization layers. LAME [31] enforces smooth decision boundaries by introducing Laplacian regularization the model predictions. SAR [23] promotes robustness by explicitly optimizing for flat minima using sharpness-awand minimization. SHOT [19] aligns the classifier by minimizing entropy while encouraging confident and diverse predictions, using fixed feature extractor. Tent [20] adapts the model by updating only the affine parameters of normalization layers via entropy minimization.
We evaluate two variants of our approach: TRUST naive, which adapts only the Mamba-specific SS2D parameters, and TRUST, which further enhances robustness by averaging over multiple traversal permutations during adaptation.
4.4 Main Results
Table 1 presents the Top-1 accuracy under the highest corruption severity (level 5) for CIFARIO-C, CIFAR100 100-C, and ImageNet-C datasets. TRUST consistently outperforms all baselines across the three datasets, demonstrating strong generalization under severe distribution shifts. also surpasses its naive variant, underscoring the effectiveness of permutation-basee strategy in enhancing robustness.
CIFARIO-C. TRUST achieves a mean accuracy of 77.5%, outperforming both Tent and SHOT by margins of 11.0% and 10.7%, respectively. On challenging corruptions, TRUST yields the largest improvements over SHOT, such as elastic (7.5%), glass blur (10.8%), and motion blur (7.4%). Compared to its naive variant, TRUST provides a further 3.3% boost. These results highlight the benefits of traversal diversity in enhancing corruption robustness.
CIEARI0O-C. On a more fine-grained benchmark, TRUST attains a mean accuracy of 54.3%, ex- ceeding SHOT and SAR by 12.3% and 12.4%, respectively. The largest absolute improvements over SHOT are observed under challenging corruptions such as impulse noise (23.2%), contrast (18.9)) and shot noise (16.1 16.1%% 1)), demonstrating the robustness of our method under severe distribution shifts. also improves upon TRUST naive by 4.5%
ImageNet-C. TRUST achieves a strong mean accuracy of 56.1% on the large-scale ImageNet- benchmark, outperforming both Tent and SHOT by 14.4%, and TRUST naive by 2. 2.7%. On challenging corruptions, TRUST yields the largest improvements over SHOT, such as glass blur (14.8%), elastic (19.3%), and jpeg compression (14.4%), reflecting the advantage of our method in mitigating both spatial distortions and digital artifacts.

---

jpeg compressien
gatarian miise
impulse noive
defoems blur
Shot nolse
glaw blur
Zoom blus
brightmess
pixelate
classic
Frost
1og
Method
Mean
Source only
48.4
73.5
52.6
78.7
77.3
69.6
63.7
59.0
CIFARIO-C
ETA [TA[21]
46.7
48.3
44.8
73.5
52.6
73.0
78.6
75.7
71.4
71.2
85.7
69.6
63.7
67.9
59.0
65.8 (1001)
LAME [31]
46.7
48.3
44.8
73.5
52.6
73.0
78.6
71.8
75.8
77.2
85.7
69.6
63.7
67.9
59.0
65.9
SAR [23]
47.7
49.5
74.3
53.4
738
79.1
72.5
76.5
78.0
86.1
70.8
64.5
68.9
60.0
66.8 (10.9)
SHOT [19]
47.8
49.7
16.3
74.3
53.7
74.0
79.3
72.6
76.6
78.1
86.3
70.7
64.5
68.9
59.9
66.8 (10.9)
Tent 220
47.3
49.2
45.8
74.2
53.1
73.7
79.1
72.2
76.3
77.9
86.1
70.4
64.3
68.7
59.6
66.5 (10.6)
TRUST naive
58.9
61.8
62.0
79,8
60.9
79.1
82.6
80.5
81.8
83.6
88.8
81.8
70.3
75.1
66.0
74,2 (18.3)
TRUST
63.1
67.8
70.3
81.0
64.5
81.4
85.0
83.2
85.4
85.8
90.1
85.7
72.1
79.1
68.6
77.5
(1110)
Source only
21.0
22.1
18.3
50.6
27.7
51.0
56.2
45.3
50.6
52.4
65.3
43.2
39.0
41.7
33.4
41,2
CIFARI00-C
[TA]
21.2
22.3
18.7
50.8
27.8
$1.2
56.3
45.4
50.8
52.7
65.5
43.5
39.2
42.0
33.7
41.4 (10.2)
LAME [31]
22.1
562
33.4
41.22
SAR [23)
21.9
22.8
19.3
51.1
28.2
51.5
56.7
46.4
51.4
53.1
65.8
44.2
39.9
42.8
34.2
41.9 (10.7)
SHOT [19]
21.9
22.9
19.1
51.4
28.5
51.8
56.8
46.3
51.4
53.3
66.0
44.2
39.8
42.7
34.3
42.00
(20)
22.6
51.1
28.2
56.7
53.1
39.7
34.0
(10.6)
TRUST naive
32.1
32.8
34.1
56.9
35.4
51.2
61.6
54.6
57.8
60.1
69.6
55.6
46.6
50.8
41.0
49.8 (19.6)
TRUST
37.8
38.9
42.3
60.9
36.6
60.8
65.4
59.0
62.2
64.5
63.1
50.3
56.6
44.9
54.3
Source only [21]
24.3
26.1
25.1
22.2
23.2
35.4
43.2
49.3
48.4
56.9
70.0
26.8
45.1
43.7
41.4
38.7
IanggeNe-C
26.4
28.4
27.2
23.5
24.6
37.2
45.1
50,8
51.0
58.8
70.6
29.1
47.7
46.9
45.0
40.8 (12.4)
LAME [31]
26.1
23.2
56.9
26.8
41.4
38.8 (10.1)
SAR SAR1231
26.5
29.2
28.0
24.5
25.3
37.4
45.1
51.0
51.7
59.1
70.5
31.5
48.2
48.6
46.3
41.5 (12.8)
SHOT
28.0
30.1
28.8
25.0
26.0
45.7
51.0
51.5
59.1
70.6
30.2
48.4
47.8
45.8
41.7
Tent [20)
27.8
30.0
28.8
24,9
25.9
38.0
45.5
51.0
51.3
59.1
70.6
30.0
48 2
47.8
45.7
41.7 (13.0)
TRUST naive
43.4
45.6
44.9
38.3
36.6
53.0
54.9
57.1
60.2
66.0
12.2
50.2
59.0
61.1
58.5
53.4 (114.))
TRUST
49.4
57.9
66.8
(137.4)
Table 1: Top-1 classification accuracy (%) under various corruption types on CIFAR10-C, CIFAR100- C, and ImageNet-C datasets. Increases/decreases mean accuracy compared to performing no adaptation (Source only) is highlighted in green/red color.
In addition to corruption-specific robustness, our method demonstrates strong generalization to broader distribution shifts, as summarized in Table 2. On ImageNet-S, our method achieves 41.5% accuracy, outperforming SHOT by a substantial margin of 8.9%, and providing a modest 0.4% improvement over the naive variant. For ImageNet- V2. TRUST attains 64.0%. exceeding SHOT by 1.6% and providing a 0.6% gain over its naive variant. The most significant boost is observed on the ImageNet-R benchmark, which features real- world renditions of ImageNet categories. Here, our method achieves 44.3%, surpassing both Tent and SHOT by 12.4%, and outperforming TRUST naive by 4.6%. On the PACS dataset, our model reaches 69.9% accuracy, offering consistent improvements over SHOT and Tent by 2.5% and over the naive variant by 2.8%. These consistent gains across varied benchmarks highlight the adaptability of our permutation- based strategy, which not only improves corruption robustness but also scales effectively to domain generalization tasks.
Method
CIFARIO-C
CIFAR100-C
ImageNet-C ImageNet-S
ImageNet-V2 ImageNet-R
PACS
Source only
65.9
41.2
38.7
31.4
62.2
31.3
66.7
ETA [21]
65.8 (10.3)
(1.4(10.2)
40.8 (12.))
31.4
62.2
31.3
66.7
LAME [31]
65.9
41.2
38.8 (10.1)
31.4
62.2
31.3
66.7
SAR [23]
66.8 (10.9)
41.9 (10.7)
41.5 (128)
(1.6
(2.4.(002)
(2.0
67.3 (10.6)
SHOT [19]
66.8 (10.9)
42.0 (10.8)
41.7 (13.0)
32.6
62.4 (10.2)
31.9 (T0.6)
67.4 (10.7)
Tent (20)
66.5 (10.6)
41.8 (10.6)
41.7 (13.0)
(2.1)
62.3 (f0.1)
31.9 (T0.6)
67.4 (10.7)
TRUST naive
74.2 (18.3)
49.8 (18.6)
53.4 (1147)
41.1 (19.))
63.4 (11.2)
39.7 (18.4)
67.1 (10.4)
TRUST
77.5 (111.6)
54.3 (113.)
56.1 (117.4)
41.5 (116))
64.0 (11.8)
44.3 (31.0)
69.9 (13.2)
Table 2: Top-1 classification accuracy (%) across datasets. For CIFARIO-C, CIFAR100-C, and ImageNet-C, values are averaged over all corruptions; for ImageNet-S, V2, R, and PACS, they reflect test set accuracy. Increases/decrease, in mean accuracy compared to performing no adaptation (Source only) is highlighted in green/red color.
4.5 Ablation Study
In this section, we present a comprehensive ablation study on the CIFARIO-C dataset to assess the impact of key factors on the performance of our model. Specifically, we analyze the effects of batch size, augmentation types, number of traversal permutations, number of adaptation iterations, aggregation strategies, effect of traversing type in evaluation and computational overhead.
7

---

Batch Size. Figure 3 demonstrates that TRUST maintains a consistent advantage over Tent across different batch sizes. Even at smaller sizes such 16 and 32, it delivers an accuracy improvement of around 8%. This indicates that our approach remains reliable and effective, even when operating with limited data per batch.
Augmentation Impact. To evaluate whether standard data augmentations can serve as effective alternatives traversal permutation strategy, we compare TRUST against common augmentations including rotation, random cropping, and color jitter. As shown in Figure 4, these traditional augmentations yield only modest improvements over the baseline source-only model on CIFARIO 10- C, with accuracies ranging from 65.99 9% to 68.3%. In contrast, our method achieves a substantial performance boost, reaching 77.5%, which is significantly higher than all augmentation baselines. This indicates that simple augmentations fail to sufficiently address distribution shifts.
78
77.5
77.5
Accuracy (9)
74
72.5
72
TRUST Tent
Accuracy
70.0
68.3
67.5
65.9
66.9
68
65.0
62.5
128
Batch Size
Source
Rotation
Crop
Jitter
TRUST
Figure 3: Accuracy comparison between TRUST and Tent across varying batch sizes on CIFARIO- C dataset.
Figure 4: Performance comparison between stan- dard augmentations and TRUST CIFAR10-C dataset.
Number of Traversal Permutations. Figure 5 shows how accuracy varies with the number of traversal permutations across CIFARI0- CIFARI00-C, and ImageNet-C. Overall, increasing the number of permutations consistently improves performance on all datasets. On CIFAR10-C, accuracy rises from 75.6% (2 permutations) to 77.6% (8 permutations). For CIFAR1OM 100-C, the gain is from 51.7% to 54.7%, and for ImageNet-C. from 54.4% peak of 56. 1% six permutations before slightly dropping to 55.5% at eight. These results confirm that incorporating multiple traversal permutations enhances robustness under distribution shifts. Although accuracy improves with more permutations, gains diminish after six, suggesting that six permutations strike a good balance between performance and computational efficiency. For breakdown of memory overhead, see Figure
Number of Adaptation Iterations. Figure 6 illustrates that accuracy improves progressively with an increasing number of iterations. Despite the potential for further improvement with more iterations, we chose to use only single iteration in subsequent experiments to enable faster adaptation without compromising on competitive performance.
78.8
77.1
TRUST
78.6
Accuracy (%)
78.4
CIFARLO
Accuracy
78.2
CIFAR100-C
ImageNet-C
78.0
$6.1
54.7555
77.8
543
2.0
4,0
6.D
8.0
77.4
Traversal Permutation Number
Iteration
Figure 5: Effect of traversal permutation count accuracy across three datasets.
Figure 6: Model performance across adaptation iterations on CIFARIO-C dataset.
Aggregation Strategy. We evaluate two baseline strategies for aggregation across traversal permu- tations: (1) an ensemble approach that averages model predictions from independently processed traversal orders, and (2) repetition baseline, where the same traversal permutation is applied & times,

---

followed by weight averaging. As shown in Figure 7, both baselines yield lower performance on CIFARIO-C, with ensemble achieving 68.1% and repetition 69.6%, compared 75.6% with our method.
The ensemble variant offers traversal diversity, but because its individual models averaged, it lacks parameter alignment. The repetition baseline, in contrast, repeats the same traversal and updates the model sequentially; the weights, therefore, remain aligned, yet no traversal diversity is introduced. Both baselines perform worse than our method, demonstrating that neither unaligned traversal diversity (ensemble) nor aligned repetitions without diversity (repetition) is sufficient on its own. By combining these two ingredients, traversal diversity and parameter alignment, our approach achieves better generalization under distribution shift.
weight-
Effect of Traversal Permutation in Evaluation. Figure 8 illustrates how accuracy varies with different traversal permutations during evaluation. We observe that the choice of permutation impacts the performance. The default permutation used in the original VMamba architecture, "abed", achieves the highest accuracy 77.5%, while others, such "bade", yield lower performance (e-g.. 71.6%). Given this sensitivity, we adopt the "abed" traversal permutation, on which the model was trained, for all evaluations ensure consistency and leverage the strongest baseline.
80
75.6
72.5
Accuracy (%)
74
72
Accuracy
76
15.3
75.7
69.6
74
68
Ensemble
Repetition
TRUST
70
abdc
adcb
bacd
Aggregation Methods
Traversal Permutation Evaluation
Figure 7: Accuracy comparison of different ag- gregation strategies on CIFAR10-C dataset.
Figure Impact traversal permutation during
evaluation CIFARIO-CC dataset.
Computational Overhead. We evaluate GPU memory usage function of the number of traver- sal permutations used during parallel adaptation. Since only the SS2D blocks are updated, we instan- tiate one SS2D block per traversal while sharing the rest of the network. Traversals are batched and routed their corresponding SS2D blocks, and their outputs then concatenated. This design minimizes com- putational overhead. As shown Figure 9, memory usage increases moderately, 1.1x, 1.3x, 1.5x, and 1.7x for 2, 4, 6, and 8 traversals, respectively, relative to the 2.1 GB baseline (measured with batch size 1). In sequential mode, forward and backward passes are
GPU Memory (GB) w
X1.3
x1.1
2.1/GB:
2
Number of Traversals
Figure 9; GPU memory usage across traversals. performed separately for each traversal. We measure per-traversal time of approximately 135 ms (with the batch size of 1), leading to a total cost memory usage remains nearly constant, while adaptation time grows linearly. Conversely, in parallel mode, adaptation time remains nearly constant, with moderate memory overhead, offering a practical trade-off given the performance gains. For more details, refer the Supplementary Material.
x 135 ms for K traversals. As expected,
Conclusion
In this paper, we introduced TRUST, test-time adaptation strategy specifically designed for vision- oriented state-space models. Our method addresses two key challenges in VMamba under distribution shift: (1) the strong inductive bias introduced by fixed traversal seans, and (2) the accumulation of domain-specific artifacts in hidden states during sequential processing. To overcome these issues, TRUST exploits directional traversal mechanism Vammaa generate complementary causal views of the input, performing adaptation based each. The resulting models aggregated via

---

weighted parameter averaging, promoting convergence toward flatter, more robust regions in the loss landscape. Experiments on seven standard benchmarks show that TRUST consistently outperform popular TTA models including Tent, SHOT, and SAR, in image classification tasks. Our method does not yet generalize to the medical domain. Future work will adapt TRUST to address data scareity, modality differences, and privacy constraints in medical settings.
Acknowledgements
We appreciate the computational resources and support provided by Compute Canada and the Digital Research Alliance of Canada.
References
[1] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International conference on learning representations, 2014.
[2] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[3] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[4] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International conference on machine learning, pages 6105
0055-612 14. PMLR, 2019.
[5) Zhuang Liu, Hanzi Mao, Chao-Yual Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11976- 1986, 2022.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Heigold, S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2020.. 2020.
[7] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[8] Xiaosong Zhang, Yunjie Tian, Lingxi Xie. Wei Huang, Qi Dai, Qixiang Ye, and Qi Tian. Hivit: A simpler and more efficient design of hicrarchical vision transformer. In The Eleventh International Conference on Learning Representations, 2023.
[9] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[10] Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2024.
[11] Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry Hungry Hippos: Towards language modeling with state space models. In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
[12] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. In ICLR, 2023.
[13] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. VMamba: Visual state space model. Advances neural information processing systems, 37:103031-103066 2024.
10

---

[14] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, In First Conference on Language Modeling, 2024.
[15] Sahar Dastani, Ali Bahri, Moslem Yazdanpanah, Mehrdad Noori, David Osowiechi, Gustavo Adolfo Vargas Hakim, Farzad Beizaee, Milad Cheraghalikhami, Arnab Kumar Mondal, Herve Lombaert, et al. Spectral state space model for rotation invariant visual representation learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 23881- 23890, 2025.
[16] Shaocong Long, Qianyu Zhou, Xiangtai Li, Xuequan Lu, Chenhao Ying, Yuan Luo, Lizhuang Ma, and Shuicheng Yan. DGMamba: Domain generalization via generalized state space model. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 3607-3616,
2024.
[17] Nitish Shirish Keskar. Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. On large-batch training for deep learning; Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
[18] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: Domain generalization by seeking flat minima. Advances Neural Information Processing Systems, 34:22405-22418,, 2021.
[19] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning, pages 6028-6039. PMLR, 2020.
[20] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[21] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 16888-16905. PMLR, 17-23 Jul 2022.
[22] Jungsoo Lee, Debasmit Das, Jaegul Choo, and Sungha Choi. Towards open-set test-time adaptation utilizing the wisdom of crowds in entropy minimization. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 16380-16389, October 2023.
[23] Shuaicheng NIu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In Internetional Conference on Learning Representations, 2023.
[24] Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEFF/CV Conference on Computer Vision and Pattern Recognition, pages 15922-15932, 2023.
[25] Sachin Goyal, Mingjie Sun, Aditi Raghunanthan, and Zico Kolter. Test-time adaptation via conjugate pseudo-labels. Advances in Neural Information Processing Systems, 2022.
[26] Taesik Gong, Yewon Kim, Taeckyung Lee, Sorn Chottananurak, and Sung-Ju Lee. SoTTA: Robust test-time adaptation noisy data streams. Advances in Neural Information Processing Systems, 36, 2024.
[27] Yongcan Yu, Lijun Sheng, Ran He, and Jian Liang. STAMP: Outlier-aware test-time adaptation with stable memory replay. In European Conference on Computer Vision, pages 375-392, 2024.
[28] Zhengqing Gao, Xu-Yao Zhang, and Cheng-Lin Liu. Unified entropy optimization for open-set test-time adaptation. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 23975-23996 23984, June 2024.

---

[29] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 7201-7211, 2022.
[30] Dian Chen, Dequan Wang. Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 295-305, 2022.
[31] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-fiee online test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8344-8353, 2022.
[32] Haopeng Sun, Lumin Xu, Sheng Jin, Ping Luo, Chen Qian, and Wentao Liu, Program: Prototype graph model based pseudo-label learning for test-time adaptation. In The Twelfth International Conference on Learning Representations, 2024.
[33] Minguk Jang, Sae- Young Chung, and Hye Won Chung. Test-time adaptation via self-training with nearest neighbor information. In The Eleventh International Conference on Learning Representations, 2024.
[34] Shuaicheng Niu, Chunyan Miao, Guohao Chen, Pengcheng Wu, and Peilin Zhao. Test-time model adaptation with only forward passes. In International Conference on Machine Learning, pages -3315,, PMLR, 2024.
[35] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 15:14274-1428 14289, 2022.
[36] Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14162 14171, 2024.
[37] David Osowicchi, Mehrdad Noori, Gustavo Adolfo Vargas Hakim, Moslem Yazdanpanah, Ali Bahri, Milad Cheraghalikhani, Sahar Dastani, Farzad Beizaee, Ismail Ben Ayed, and Christian Desrosiers. WATT: Weight average test time adaptation of CLIP. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[38] Gustavo A Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. CLIPArTT: Adaptation of CLIP to new domains at test time. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), pages 7092-7101, February 2025.
[39] Mona Schirmer, Dan Zhang, and Eric Nalisnick. Temporal test-time adaptation with state-space models. arXiv preprint ar arXiv:2407.12492 2024.
[40] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
[41] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with MixStyle. In International Conference on Learning Representations, 2021.
[42] Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Gustavo A Vargas Hakim, David Osowiechi, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. FDS: seodbbac--guidee domain synthesis with multi-source conditional diffusion models for domain generalization. In Pruceedings of the Winter Conference on Applications of Computer Vision (WACV), pages February 2025.
[43] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018 UAI 2018, pages 885-885. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
12

---

[44] Ali Bahri, Moslem Yazdanpanah, Mehrdad Noori, Sahar Dastani, Milad Cheraghalikhani, David Osowiechi, Farzad Beizace, Gustavo Vargas Hakim, Ismail Ben Ayed, and Christian Desrosiers. Test-time adaptation in point clouds: Leveraging sampling variation with weight averaging. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 266-275. IEEE, 2025.
[45] Moslem Yazdanpanah. Ali Bahri, Mehrdad Noori, Sahar Dastani, Gustavo Adolfo Vargas Hakim, David Osowiechi, Ismail Ben Ayed, and Christian Desrosiers. Purge-Gate: Backpropagation- free test-time adaptation for point clouds classification via token purging. arXiv preprint arXiv:2509.09785 2025.
[46] Ali Bahri, Moslem Yazdanpanah, Sahar Dastani, Mehrdad Noori, Gustavo Adolfo Vargas Hakim, David Osowiechi, Farzad Beizaee, Ismail Ben Ayed, and Christian Desrosiers. SMART- PC: Skeletal model adaptation for robust test-time training in point clouds. arXiv preprint WwX1v 2025.
[47] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[48] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542-55500 2017.
[49] Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in neural information processing systems, 32, 2019.
[50] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400 PMLR, 2019.
[51] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli. Mike Guo, et al. The many faces of robustness: critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference computer vision, pages 8340-8349, 2021.
[52] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. International journal of computer vision, (8(2):303-338. 2010.
[53] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam- Gyn Cho, Seong-Whar Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 891-898, 2014.
[54] Mehrdad Noori, David Osowiechi, Gustavo Adolfo Vargas Hakim, Ali Bahri, Moslem Yazdan- panah, Sahar Dastani, Farzad Beizaee, Ismail Ben Ayed, and Christian Desrosiers. Test-time adaptation of vision-language models for open-vocabulary semantic segmentation. arXiv preprint arXivv2505.21844, 2025.
[55] Dong-Hwan Jang, Sangdoo Yun, and Dongyoon Han. Model stock: All we need is just a few fine-tuned models. In European Conference Computer Vision, pages 207-223. Springer, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract is the summary of the content of the paper and the introduction is clearly state the main contributions the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the last part of the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide details of implementation and experiments in section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
The data available online and we will release the code with the supplementary
material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https: //iips.cocc httppscccppuull///geeeee http://guieeee/oeeeeeeeee
ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Provided in section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Experiments were run with single seed. We plan to include multi-seed seed results in future revisions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides details on the hardware used, and reports a computational overhead analysis in the ablation section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that NeurlPS Code of Ethics https: : http///eurrpp..eeee http://uuuripp.ccpueeeeeeee httpp//////.i.cttiiiciiicceellll
didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: Yes, we do.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: No relevant.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No relevant.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We followed the protocol.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The paper does not introduce any new datasets or benchmarks. All experiments are conducted on existing public datasets using open-source baselines.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not relevant.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
Justification: This work does not use LLMs in the core methodology, scientific rigorousness, or originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https //tprr//nnu...eeeeeee wtw.urips.c//oooffeeeeeeeeeeeeee //t/////eeeneee/2225//////////// 20LM/LLM for what should or should be described.
20

---

TRUST: Tese-Time Refinement using Uncertainty-Guider SSM Traverses Appendix
Pseudo-code
In this section, give the pseudo-code for proposed test-time adaptation method, TRUST. This pseudo-code provides a concise summary of the key steps involved in our approach, offering a high-level abstraction of the implementation. Algorithm outlines the overall TRUST procedure for test-time adaptation. For each corruption in the evaluation set, the model is first reset to its original (pre-adaptation) weights. The input is then processed using the FORWARD AND_AND__A function (Algorithm 2), which outputs the adapted prediction and a list of model parameter sets 0, each corresponding to a different traversal permutation used during adaptation. These parameters averaged to obtain the final adapted parameters, Orinal which are loaded back into the model. The final prediction is then computed by re-evaluating the model on the same input z.
Algorithm TRUST
1: for each corruption do
2: MODEL.RESET
3: (out, 0) <- NODDL.FORWARD_AD_AADAA
4: Olnull <- MEAN(()
5: MODEL.LOAD_STATTEE BALLLLADDSTAT_DICTTCOC
6: out MODEL.EVALLATEGE)
7:
end for
Algorithm 2 defines the FORWARD __NND_ AND_ADAPT ADAPT function. For each traversal permutation Ti P, the model performs forward pass with input and permutation Tig The cross-entropy loss between the model's prediction and the pseudo-labels is computed, and gradient descent step is performed to update the model parameters. Each updated parameter state is stored in the list A. After all permutations have been processed, the method returns the final output and the list of different model parameters.
Algorithm 2 FORWARD BROWARD_AND_ADATE AND_ADAPT(E))
1: < empty list
2: for each Tu, P do
3: out + model(z, Wiy)
4: loss CE(out, ( (reudo_labels)
5: loss.backward()
6:
optimizer.step()
7:
appimizzerzero_graadde
8:
0.append (nnpeeednmodlll.rraeeeeeee))
9: end for
return (out,
B Finetuning Process
The backbone model, VMamba, originally trained on ImageNet-C, which contains 1,000 classes. To enable test-time use on corrupted datasets with different label sets, we fine-tune only the classifier layer on the clean version of each target dataset, keeping the rest of the model frozen. This procedure is applied to CIFAR10-C. CIFAR 100-C, and PACS, using learning rate of 5e-4 for 300 epochs. For PACS, which includes four domains (photo, art painting, cartoon, and sketch), we follow the standard protocol: one domain is held out for evaluation while training on the remaining three. Specifically, we use the photo domain as the held-out test set. For datasets such as ImageNet-S, ImageNet-V2, and ImageNet-R, which share the same label space ImageNet, no fine-tuning is required.
21

---

C Efficient Parallel Implementation
We mentioned in the main paper (Computational Overhead section) that our method supports an efficient parallel adaptation strategy. Figure 10 provides a detailed diagram of this process, In parallel mode, handle K traversal permutations simultaneously. A batch B is first split into K subsets, each corresponding to different permutation Tig- Each subset is then passed to an independent SS2D TRUST Version block, where the SS2D parameters are adapted in parallel while the rest of the model remains shared across all paths, this design significantly reduces memory usage.
After adaptation, the outputs are concatenated back into single batch, which allows for efficient GPU utilization, For evaluation, we perform a weight averaging step across all adapted SS2D modules, producing single unified SS2D TRUST Version block. This averaged block is then used for inference on the full batch. By leveraging parallelism in both data and traversal space, our method achieves scalable and low-overhead test-time adaptation while preserving performance across permutations.
Weight Average SS2D TRUST Version
B
ss2D TRUST Version
Concat
B
B
ss2D TRUST Version
B
ss2D TRUST Version
Adaptation
B
Batch of data
SS2D TRUST Version
B
Weight everaged
W Kig
Adapted weights
Evaluction
with pereutatioo Tyy
Figure 10: Detailed diagram of TRUST in Parallel mode.
D Standard Mode
Table reports Top-1 accuracy under various corruption types severity level 5 for CIFAR10- C, CIFARI00-C, 100-C, and ImageNet-C in the standard setting, where the model is reset for each test batch. On CIFAR1O-C, TRUST achieves the highest mean accuracy of 66.2%, surpassing Tent and SHOT by 0.2%. On CIFARI00-C, reaches 41.7%, outperforming all baselines by at least 0.4%, and improving over its naive variant by 0.5%, underscoring the benefits of permutation-based refinement, On the large-scale ImageNet-C. TRUST attains 39.9%, exceeding Tent by 1.1%, and outperforming its naive counterpart by 1. 1.2%. These results highlight the scalability and robustness of permutation- aware adaptation across diverse datasets. Table 3 evaluates performance under domain- level distribution shifts. TRUST achieves 31.6% on ImageNet-S, 62.3% on ImageNet- V2, and 31.5% on ImageNet-R. On the PACS benchmark, it obtains 71.3%, outperforming its naive variant by 4.6%. While gains are narrower in these less corrupted settings, TRUST consistently improves generalization, particularly under significant domain shifts.
E TRUST with Batch Norm Adaptation
In this experiment, we compare two adaptation strategies within our framework: updating only the BN layers versus updating the SS2D parameters. As shown in Table 4, our method outperforms Tent both settings, by 3.1 % when using BatchNorm adaptation and by 1 11.8% with SS2D adaptation. We opt to proceed with SS2D adaptation, traversal permutations directly influence Mamba-specific parameters encoded in the SS2D blocks. Updating these parameters is therefore essential to fully capture the effects of naversal-based modifications.
22

---

Method
CIFARIO-C CIFARIOO-C
ImageNet-C
ImugeNet-S
ImageNet-V2
ImugeNet-R
PACS
Source only
65,9
41.2
38.7
31.4
62.2
31.3
66.7
ETA
65.8 (30.3)
41.2
38.8 (70.1)
31.4
62.2
(1.4 (10.1)
66.7
LAME
65.9
41.2
38.8 (10.1)
31.4
62.2
31.4
66.7
SAR
(10.1)
41.3 (10.1)
38.8 (10.1)
31.4
62.2
31.4
67.1 (19.4)
SHOT
66.0 (10.1)
41.3 (10.1)
38.9 (70.2)
31.4
62.2
31.4 (70.1)
67.3 (10.6)
Tent
66.0 (10.1)
41.3 (10.1)
38.8 (10.1)
31.4
62.2
31.4 (10.1)
67.2 (10.5)
TRUST naive
65.9
41.2
38.9 (10.3)
31.5 (10.D)
62.3 (10.))
31.5
66.8 (10.1)
TRUST
66.2 (10.3)
41.7 (10.5)
39.9 (11.2)
(10.2)
62.3 (10.1)
31.5 (10.2)
71.3 (14.6)
Table 3: Top-1 classification accuracy (%) across datasets in standard setting. For CIFARIO-C, CIFAR100-C, and ImageNet-C, values averaged over all corruptions; for ImageNet-S, V2, R, and PACS, they reflect test set accuracy. Increases/decreesss in mean accuracy compared to performing no adaptation (Source only) is highlighted in green/red color.
brightans
gausiion
impulie
glass
10000O
200m
Frost
contras
elaxtic
pisclate
fng
Method
Source only
24433
26.1
25.1
22.2
23.2
35.4
43.2
49.3
48.4
56.9
70.0
26.8
45,1
43.7
41.4
38.7
Tent
27.8
30.0
28.8
21.9
25.9
38.0
45.5
51.0
51.3
59.1
70.6
30,0
48.2
47.8
45.7
41.7
HN
TRUST
32.8
35.1
34.0
26.8
28.5
40.8
47.7
52.9
53.8
61.1
71.3
34.1
50.7
51.8
50.2
44.8
Tent
29.6
31.7
34.7
25.1
22.0
45.7
44.8
46.0
55.8
62.2
69.8
32.5
52.8
56.8
54.3
44.3
SS2D
TRUST
46.8
49.4
48.5
42.8
40.8
57.1
57.9
57.3
61.7
06.8
71.9
54.9
61.4
63.6
60.2
56.1
(1108)
Table 4: Comparison of adaptation strategies using BN and SS2D parameters on ImageNet-C dataset. Increases/decreases in mean accuracy compared Tent is highlighted in green/red color.
F
Mean Entropy of Different Traversal Permutations
As illustrated in Figure 11, varying the order of spatial traversals in VMamba leads to substantial differences in mean entropy across datasets, revealing the sensitivity of the model's ' internal rep- resentations to traversal patterns. Higher entropy values indicate less confident predictions, often aligning with reduced robustness under distributional shifts. This highlights the critical role of traversal selection in ensuring reliable adaptation. To offer a comprehensive analysis, we report entropy values for all permutations across CIFARTO-C, CIFAR100-C, ImageNet-C, ImageNet-S, ImageNet-V2. V2, ImageNet-R, and PACS. Notably, the top-2, top-4, and top-6 traversal permutations with the lowest entropy, highlighted with green cross-hatching (i.e., diagonal lines overlaid on the heatmap cells), consistently reappear across datasets. This pattern suggests that certain traversal orders inherently yield more stable and confident model outputs, providing a principled foundation for selecting effective traversal subsets in approach.
CIRARIO-C
1.36
0.96
9:87
1,50
1.30
1.53
1.38
3.3G
1.56
1.36
1.44
1.53
1.32
1.30
0.93
OFAR100-C
2.35
2.01
2.53
2.59
2.43
2.89
2.58
2.56
2.50
2.69
2.66
2.84
2:47
2.67
2.49
2.81
ImageNet-C
6.48
6,56
5.46
5.55
5.48
6.61
6.61
4.68
ImageNets
445
555
6.29
5.54
4.55
6.67
6,61
6.64
6.57
5.25
5.32
5.25
6.2
6.12
5.24
5.11
5.31
5:29
539
5.24
5.43
InageNet-V2
2.5
ImageNet-
5.1
6.29
4.69
6.25
6.20
6.33
6.24
3.53
3.81
3.53
5.36
5.19
6.53
4.13
6.01
8:01
4.12
5:30
BACS
1.06
0.85
0.94
0.36
1.04
aded
actill
arab
4d60
lood
bade
Arard
dbbc
abile
dbick
Permutation Traverses
Figure 11: Mean entropy of different traversal permutation across seven benchmarks.
G Combination of TRUST with Augmentation
According to Figure 4 in the main paper. Jitter augmentation yields the highest performance among the augmentation strategies evaluated. Building on this, we examine the effectiveness of combining Jitter with our proposed method, TRUST, under two augmentation settings: (1) applying different Jitter augmentations for each traversal permutation, and (2) applying different Jitter augmentations per batch. The former setting achieves a modest improvement of 0.3% over the source-only baseline, while the latter results in slightly higher gain of 1.8%. However, both improvements are substantially
23

---

lower than the performance boost achieved by the original TRUST, which yields an 11.6% increase. These results highlights that although augmentation contributes performance, the core strength of TRUST lies in its ability to adapt representations based on traversal-specifee dynamics rather than augmentation diversity alone.
hrightness
gunssiun
defocus
m0000n
200m
Erost
S00W
tag
closic
pixelate
1898
Method
Mean
Source only
46.8
48.4
45.0
73.5
52.6
73.0
78.7
718
75.8
77.3
85.7
69.6
63.7
67.9
59.0
65.9
TRUST
per penmutation
(103)
TRUST Jitee per batch
57.0
53.4
57.2
70.0
18.9
65.6
79.6
77.5
77.9
77.3
83.2
79.3
57.9
65.4
60.6
67.7 (1L8)
TRUST
63.1
67.8
70.3
810
64.5
81.4
85.0
83.2
85.4
85.8
90.1
85.7
72.1 79.1 68.6
77.5
Table 5: Comparison of augmentation impact on CIFARIO 10-C dataset. Increases/decreases in mean
accuracy compared performing adaptation (Source only) is highlighted in green/red color.
H High Entropy Traversal Permutations
We have also tested the performance of TRUST using the top-k high-entropy (i.e., less confident) traversal permutations. As shown in Table 6, this setting leads to significant performance drop of 12.5% compared to the source-only baseline. This highlights the critical role of traversal selection in method. High-entropy permutations, which correspond to uncertain and unstable model predic- tions, introduce noise into the adaptation process and hinder effective generalization. These findings further support our strategy of entropy-based traversal filtering, where low-entropy permutations are prioritized to ensure reliable and robust adaptation.
brightmes
imgulse
delocus
elasicc
shot
glors
2000
SHOW
1p02
Method
Mean
Source only
46.8
48.4
45.0
73.5
52.5
73.0
78.7
71.8
75.8
77.3
85.7
69.6
63.7
67.9
59.0
65.9
TRUST (topkk high enrropy)
38.5
32.2
36.4
57.9
37.7
61.5
64.9
52.5
64.0
70.3
74.1
49.3
57.9
56.2
48.2
53.4 (112.))
TRUST (lop-k low entropy)
72.1
(1136)
Table 6: Comparison of TRUST performance with low and high entropy traversal permutations
CIFAR10-C dataset.
I TRUST Application beyond Classification Settings
We conducted additional experiments on segmentation tasks using various datasets, including Pascal VOC21 [52] and Pascal Context 59 [53], applying corruptions following the protocol of [54]. The results demonstrate that TRUST performs well in segmentation, outperforming methods like Tent. This further supports the generalizability and effectiveness of our approach beyond classification settings.
joo compression
EGusssian san noise
impalse noike
defocus hlur
metion blur
Shol nubse
glass blur
Z0om blur
brightness
Dataset
Frost
SnOW
CONEEAS
clasic
pixelate
tog
Method
Mcan
Source only
29.1
33.1
28.3
21.0
8.2
33.1
25.4
50,9
50.3
70.7
76.5
63.9
25.5
22.2
59.2
39.8
V21
Tent
33.0
35.7
32.0
22.3
14.7
38.2
25.3
46.5
49.0
60.2
63.9
66.2
38.5
28.8
43.9
39.9
TRUST
38.8
42.0
38.7
29.8
22.6
45.1
29,8
50.5
53.5
63.4
66.4
68.5
45.1
37.7
48.6
45.4
(15.6)
Source only
17.1
19.6
17.4
27.4
14.9
29.2
19.5
30.2
28.5
42.1
50.8
41.0
23.9
30.4
38.4
28.7
Tent
17.6
18.9
17.8
22.2
15.9
27.5
17.9
26.9
30.0
36.7
41.9
42.9
25.8
28.2
28.3
TRUST
24.4
27.4
25,4
24.6
21.2
30.1
19.8
29.8
32.8
39,2
42.4
43.2
31.5
36.1
31.6
30.6
(11.9)
Table 7: Segmentation performance under different corruptions of V21 and P59 datasets.
J Performance Comparison with ViT-based Method
To provide a fair evaluation, we adapted the ViT-based method from [34] by replacing its backbone with VMamba and modifying it to be compatible with VMamba backbone. Specifically, we replaced
24

---

the CLS token (absent in VMamba) with the mean of all tokens and adjusted the number of learnable tokens to match VMamba's dimensionality. This ensures that both methods operate under the same architectural constraints. Our experimental results demonstrate that TRUST, which is specifically designed to exploit VMamba's 's traversal mechanism, significantly outperforms the adapted baseline, highlighting its robustness under distribution shift.
brightness
gausstan
shot
impulse
defocus
glass
motion
200m
frost
rog
contrast
elastic
pixelate
ipg
Method
Mcan
Source only
24.3
26.1
25.1
22.2
23.2
35,4
43.2
49.3
48.4
56.9
70.0
26.8
45.1
43.7
41.4
38.7
Tent
27.8
30.0
28.8
24.9
25.9
38.0
45.5
51.0
51.3
59.1
70.6
30.0
48.2
47.8
45.7
41.7
FOA
17.8
19.4
18.5
15.1
18.2
22.7
28.6
38.7
33.9
44.3
57.4
22.7
38.2
40.9
41.7
30.5
(18.))
TRUST
48.5
71.9
63.6
60.2
(1174)
Table 8: Performance comparison across corruption types with the ViT-based Method.
K
Weight Diversity Across Traversals
To analyze the consistency of the SS2D block parameters under different traversal permutations, we compute statistics based on the L2 norms of each parameter tensor across six weverrsl-adaptee models. We extract each weight or bias tensor from each model, flatten it, and compute its L2 norm. We then calculate the mean of these norms to reflect the overall magnitude across traversals. To quantify variability, we also compute the standard deviation across the corresponding elements of these tensors and report the average these deviations.
Mean and Std for Biases
15.0
12.5
L2 Nom
10.0
7.5
5.0
0.0
31 Layer Number
Figure 12: Mean and standard deviation of the 1.2 norm for the bias parameters
Figure 12 shows the mean and standard deviation of the L2 norm for the bias parameters, while Figure 13 shows the same for the weight parameters. These visualizations indicate that although the magnitude of parameters varies across layers, the variation across traversals remains relatively low. This suggests that SS2D block parameters adapted with different traversal orders remain geometrically close in parameter space. Such consistency is conceptually aligned with the findings in the Model Stock method [55], which emphasizes the benefits of maintaining proximity in the weight space across fine-tuned or adapted models, such enabling effective weight averaging and improving generalization. This experiment is conducted using the CIFAR-IOC dataset.
Mean and Std for Weights
50
40
80
20
10
Layer Number
Figure 13: Mean and standard deviation of the L2 norm for the weight parameters
25


Reviews:
[{"review_id": "7RBlWaqATo", "reviewer": "Reviewer_GHW6", "summary": "The authors introduce a method called TRUST, to adapt vision SSMs at test-time in out-of-distribution conditions. The core idea in TRUST is to leverage variations of the patch traversal mechanism in the VMamba style architectures. Specifically, the authors permute the order of traversals and pick top-k orders that have minimal entropy on the test-time distribution. Then, the authors increase the sharpness of the predicted distribution by self-training on the pseudo-labels assigned by each of the top-k traversal orders to obtain a weight per traversal order. Eventually, the authors average the weights across all the top-k permutations to get the network weights. Through experiments, the authors demonstrate that TRUST outperforms algorithms like TENT and SHOT, when applied to the base VMamba model. The authors demonstrate these results on ImageNet and CIFAR corruption datasets, and other OOD shifts including PACS, ImageNet-S/v2/R. The authors further conduct several ablations to establish the impact of batch-size, input augmentations, number of traversal permutations, adaptation iterations, ensembling methods, specificity to permutations and computational costs with increase in traversals. Overall, within the purview of the conducted experiments, the authors propose a useful TTA method specifically for VMamba style architectures.", "questions": "I think addressing the points outlined under weaknesses would help improve my rating of the submission. Specifically, the sensitivity to a particular traversal order and assessing the computational overhead vs performance gain trade-off (where best possible performance is squeezed out of a combination of strong augmentations and prior TTA methods).", "limitations": "Yes. The authors do acknowledge the limitations of the proposed method.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\nThe following points highlight the strengths associated with the submission.\n\n1. The paper is generally well-written and easy to follow. The authors do a good job of motivating the architecture specific TTA method with TRUST. There is novelty in the sense that authors explicitly exploit the traversal procedure of VMamba to improve generalization. That said, this comes with the cost of being limited to a specific architecture.\n2. The authors conduct thorough experiments to outline the method’s effectiveness. The controlled set of experiments across differing visual distribution shifts clearly demonstrate that varying the traversal mechanisms in itself can provide substantial utility over applying more architecture-agnostic methods like TENT or SHOT. The margins over the base algorithms are quite substantial in some cases, highlighting the usefulness of the proposed method.\n3. Further, to justify and assess the impact of different design decisions, the authors conduct thorough ablations – ranging from the impact of batch-sizes, to specificity to permutation orders. For future work attempting to build on top of TRUST, these ablations serve as useful starting points to iterate on design decisions.\n\n**Weaknesses:**\n\nI have the following weaknesses to point out in the submission.\n\n1. The model performance being sensitive to the traversal order (abcd > badc) seems odd. Ideally, at test-time, one doesn’t have access to ground-truth to infer the best performing permutation. The submission might benefit from a little discussion about this observation. This being an outcome despite weight averaging seems especially odd – the model is likely carrying over some bias from the source training dataset that TRUST isn’t able to overcome. Concretely, two things would help improve the submission in this regard – (1) an investigation about why this is happening? and (2) if it’s possible to find a reliable heuristic to choose the best test-time permutation.\n2. The method generally seems very computationally intensive – both in terms of memory and latency. A natural counter-opinion is to ask if similar performance gains can be squeezed out of existing methods (combining augmentations, finding the best TENT, SHOT TTA variants) instead of adopting such a computationally expensive method. Understanding the limits or the experimental settings in which it’s feasible to apply TRUST will help understand downstream use-cases.", "was_revised": true, "final_justification": "I primarily had two concerns -- (1) sensitivity to the traversal order, (2) combating computational complexity. For the former, the rebuttal provided by the authors sufficiently addresses my concern (and emphasizes + ties together other design decisions). For the latter, the authors address my concern adequately by providing appropriate comparisons and highlighting efforts to deal with the computational trade-off.\n\nGiven that my concerns are sufficiently addressed, I am inclined to increase my rating of the submission.", "rebuttal": "We thank the reviewers for the valuable comments and are pleased that TRUST was recognized as a novel and well-motivated method for Mamba-style architectures. We appreciate the positive remarks on the clarity of our writing, the effectiveness of our approach, and the depth of our experiments and ablations. Below, we address the reviewer comments in detail.\n\n**1-1. Why is the model performance sensitive to traversal orders?**\n\nState Space Models (SSMs) are inherently direction-sensitive, meaning that the order in which input patches are traversed has a significant impact on model performance. According to the Mamba formulation (Eq. 1 of our paper), the hidden state at each step ($h(t-1)$) directly influences the next ($h(t)$). Therefore, the choice of traversal order propagates its effects through the entire sequence of hidden-state updates. This directional dependency also comes from the fact that the hidden states learned during training are aligned with the original traversal order. When a different patch order is used at test time, the model is exposed to a sequence it has not encountered before, resulting in a mismatch between learned and test-time dynamics. As noted in the Mamba paper, this directional bias can cause the model to encode information more effectively along the original traversal path.\n\n\n$h(t) = A h(t-1) + B x(t), \\quad y(t) = C h(t)$\n\n**1-2. How the top-k traversal orders are selected without labels?**\n\nWe emphasized that our selection of the top-k traversal permutations is entirely **unsupervised** and **does not require access to labels**. As detailed in Section 3.2 and illustrated in Figure 1 (offline phase) of the paper, we rank traversal orders solely based on the average **Shannon entropy** of predictions on a test dataset. We conducted experiments on ImageNet-C and selected the top-k traversal permutations based on the lowest prediction entropy. We observed that these top-k permutations remained largely consistent across different test data distributions. This finding is illustrated in Figure 2 of the supplementary material.\n\n**1-3. Find a reliable heuristic to choose the best test-time permutation**\n\nTo identify the most effective test-time permutation, we adopt Shannon entropy as a self-supervised confidence metric. While low entropy does not strictly guarantee better accuracy, it serves as a strong proxy for model certainty. In particular, entropy captures the sharpness of the predicted probability distribution, lower entropy indicates more confident predictions, which often correlates with more reliable traversal orders. Therefore, entropy offers a useful heuristic that helps rank permutations by their potential to produce accurate and stable outputs in the absence of labeled test data.\n\n**2-1. Computational Complexity**\n\nOur method supports an efficient parallel adaptation strategy, which we represented in Figure 1 of supplementary material and we discussed its computational overhead in the ablation study of the paper.  In this mode, we handle K traversal permutations simultaneously. One batch is first split into K subsets, each corresponding to a different permutation. Each subset is then passed to an independent SS2D TRUST Version block, where the SS2D parameters are adapted in parallel while the rest of the model remains shared across all paths. After adaptation, the outputs are concatenated back into a single batch, which allows for efficient GPU utilization. For evaluation, we perform a weight averaging step across all adapted SS2D modules, producing a single unified SS2D TRUST Version block. This averaged block is then used for inference on the full batch. \n\n**2-2. Could the combination of augmentation and TENT yield comparable performance?**\n\nWe explored the use of different augmentations as an alternative to multiple traversal permutations in Figure 4 of the paper. That experiment evaluates the impact of weight averaging across models adapted with various augmentations versus those adapted with different traversal permutations. To ensure a fair comparison, we match the number of data augmentations to the number of traversal permutations. As shown in the table below, different traversal permutations (**TRUST**) consistently outperforms different augmentations, highlighting the superior effectiveness of permutation diversity in enhancing the model performance.\n\n\n### CIFAR10-C:\n\n| Source only | Tent  | Rotation | Jitter | Crop  | TRUST |\n|:-----------:|:-----:|:--------:|:------:|:-----:|:-----:|\n|    65.9     | 66.5  |  66.8    | 68.3   | 66.9  | 77.5  |"}, {"review_id": "RM5AKtnH38", "reviewer": "Reviewer_sdHM", "summary": "The paper proposes TRUST, a TTA method that capitalizes on the architectural properties of VMamba, a vision-adapted State Space Model. It tackles the issue of performance degradation of SSMs under distribution shifts, which arises from strong inductive biases and the accumulation of domain-specific artifacts in hidden states due to fixed traversal paths. TRUST generates multiple traversal permutations of input images, which are then used to create diverse causal perspectives. By selecting the top-K permutations with the lowest predictive entropy and averaging the Mamba-specific parameters, the method mitigates the impact of noisy predictions and promotes robustness. The authors validate TRUST on various corruption-based and domain generalization benchmarks, showing consistent improvements over baseline TTA approaches.", "questions": "* Could you provide a more detailed analysis of the computational overhead, particularly the time complexity, of the traversal permutation generation and selection process in the offline phase? How does the scalability of this component affect the practical application of TRUST, especially with larger models or an increased number of permutations?\n * Given that TRUST's core mechanism is highly specific to the architectural properties of VMamba and its SS2D module, could you elaborate on the potential challenges or modifications required to adapt the underlying principles of uncertainty-guided traversal to other neural network architectures, such as CNNs or standard Transformers?\n * Regarding the \"confidence paths\" and the selection of permutations with the lowest entropy: Could you provide a deeper theoretical or empirical explanation as to why these specific paths consistently lead to better generalization and more stable, domain-robust hidden states, beyond the current explanation of exploring flatter minima? Additionally, could you discuss the sensitivity of the results to the choice of the hyperparameter K (number of top permutations selected) and the methodology used to determine its optimal value?", "limitations": "* Dedicated Limitations Section: Authors should include a \"Limitations\" section to acknowledge the inherent constraints and potential drawbacks of their method. This could include:\n   * Computational Overhead: Acknowledge and quantify the potential computational cost associated with generating and selecting traversal permutations, especially in real-time or resource-constrained environments.\n   * Specificity to VMamba: Explicitly state that the method's reliance on VMamba's unique architectural properties limits its direct applicability to other model architectures. Discuss whether the core ideas could be generalized and what challenges might arise.\n   * Hyperparameter Sensitivity: Discuss the sensitivity of the results to key hyperparameters, such as the number of top-K permutations, and the robustness of the method across a wider range of hyperparameters.\n   * Theoretical Guarantees: Address the lack of strong theoretical guarantees for why \"confident paths\" (low entropy permutations) consistently lead to better generalization.\n * Discussion on Potential Negative Societal Impact: It's crucial to discuss any potential negative societal impacts, even if seemingly minor or indirect. This could involve:\n   * Bias Amplification: Consider whether the adaptation process could inadvertently amplify existing biases present in the training data, especially when applied to real-world, diverse datasets.\n   * Adversarial Robustness: While improving robustness to natural distribution shifts, discuss if the method inadvertently introduces new vulnerabilities to adversarial attacks or if its effectiveness in adversarial settings is yet to be explored.\n   * Ethical Considerations in Deployment: Briefly touch upon the ethical considerations when deploying such adaptive models, particularly in high-stakes applications where errors due to unaddressed biases or unforeseen shifts could have significant consequences.\n   * Environmental Impact: While not explicitly explored, a brief mention of the energy consumption associated with extensive training and adaptation (if applicable) could be considered, aligning with broader discussions on sustainable AI.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n * Novelty in SSM Adaptation: TRUST is presented as the first TTA approach explicitly designed for Mamba-based vision models, leveraging their internal traversal dynamics rather than relying on external data augmentation or auxiliary models. This is a significant contribution to the field of TTA for emerging architectures like SSMs.\n * Effective Use of Architectural Properties: The method effectively utilizes the unique traversal mechanism of VMamba to generate diverse causal perspectives of the input, which is a clever way to induce variability and explore flatter minima in the loss landscape without additional computational overhead from data augmentations.\n * Strong Experimental Results: The paper provides comprehensive experimental validation on seven standard benchmarks, including CIFAR10-C, CIFAR100-C, ImageNet-C, PACS, ImageNet-Sketch, ImageNet-V2, and ImageNet-R. TRUST consistently outperforms existing state-of-the-art TTA methods, demonstrating significant improvements in robustness and generalization under various distribution shifts.\n * Clear Ablation Studies: The paper includes \"TRUST naive\" as a baseline which adapts only Mamba-specific SS2D parameters, while \"TRUST\" further enhances robustness by averaging over multiple traversal permutations during adaptation. This effectively highlights the contribution of the proposed permutation and weight averaging strategy.\n\nWeaknesses\n * Complexity of Traversal Permutations: While novel, the concept of systematically generating and selecting traversal permutations based on entropy might introduce a non-trivial computational overhead during the offline phase, especially if the number of permutations or the size of the model increases. The paper states that \"All experiments were conducted using a single NVIDIA A6000 GPU\", but doesn't elaborate on the time taken for the offline phase. Further analysis on the scalability of this permutation generation and selection process would strengthen the paper.\n * Limited Generalizability to Other Architectures: The method is highly specific to the architectural properties of VMamba and its SS2D module. While this is presented as a strength, it also implies that the core mechanism of TRUST might not be directly transferable to other backbone architectures (e.g., CNNs or standard Transformers) without significant modifications. This limits its broad applicability across the wider machine learning community.\n * Detailed Explanation of \"Confidence Paths\": The paper mentions that TRUST \"averages model weights from the most confident paths\", and selects permutations \"with the lowest entropy, which are associated with more stable and domain-robust hidden states\". While the link to flat minima is made, a deeper theoretical or empirical dive into why these \"confident paths\" (low entropy permutations) consistently lead to better generalization beyond the current explanations would be beneficial.\n * Discussion on Hyperparameter Sensitivity: The method involves selecting the top-K permutations. A more detailed discussion on the sensitivity of the results to the choice of K, and how this hyperparameter is determined, would improve the completeness of the experimental analysis.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and are pleased that TRUST was recognized as a novel and effective TTA method for Mamba-based vision models. We appreciate the positive remarks on our use of traversal dynamics, strong experimental results across diverse benchmarks, and the clarity of our ablation studies. Below, we address the reviewer comments in detail.\n\n**1-1. Time complexity of the traversal permutation generation and selection in the offline phase.** \n\nIn the offline phase, we evaluate our pre-trained model under different traversal permutations and select the top-k permutations with the lowest entropy values, corresponding to the most confident predictions. Since no model adaptation occurs at this stage, the procedure is equivalent to a forward-pass evaluation and is computationally lightweight. For instance, on the PACS dataset, evaluating for a single permutation takes only 4 seconds for all images in this dataset. Extending this across all 24 permutations results in a total evaluation time of 1.5 minutes, which is a one-time cost. Our experiments show that the top-K permutations remain consistent across all test datasets, as visualized in Figure 2 of the supplementary material. This consistency stems from the similarity of these permutations to the original traversal order used during pre-training, as reflected in their high confidence scores. Therefore, this procedure can be performed only once on a single dataset to determine the appropriate Ks, and then used across all test datasets.\n\n**1-2. How does the scalability of this component affect the practical application of TRUST, especially with larger models or increased number of permutations?**\n\nAs mentioned previously, each permutation in the offline phase is evaluated using only a forward pass, no backpropagation is involved. The total evaluation time scales linearly with the number of permutations. In terms of memory, the model size remains constant regardless of the number of permutations, since the same model is reused for each forward pass. Therefore, the memory required is simply that needed to run a single evaluation of the model.\n\n**2. Elaborate on the potential challenges or modifications required to adapt to other NNs such as CNNs or ViTs.**\n\nMamba is emerging as a powerful alternative to both CNNs and ViTs across a range of vision tasks. Unlike ViTs, which rely on self-attention and quadratic complexity with respect to sequence length, Mamba is built on SSMs that enable linear-time sequence modeling, making it highly efficient for long input sequences. This makes Mamba especially attractive in settings with limited compute resources. Despite being more lightweight, Mamba retains competitive or even superior performance in many tasks. Several recent papers [1, 2, 3] have shown that Mamba outperforms ViTs on image classification, segmentation, and detection, while using less memory. \n\nPrevious test-time adaptation methods, such as TENT and SHOT, are not well-suited for Mamba-based models, as demonstrated in Table 2 of our paper. Due to the unique architecture of Mamba and favorable properties, such as linear-time complexity, we focused on designing a strategy specifically tailored for Mamba to effectively handle domain shifts during test time in vision applications. Extending our approach to CNNs or ViTs would undermine our core objective, as it would forgo the key advantage of leveraging Mamba’s efficient and sequential computation. \n\n**3-1. Deeper theoretical explanation of why these confidence paths consistently lead to better generalization, beyond the current explanation of exploring flatter minima?**\n\nWe provide a deeper theoretical justification based on **SWAD** [5]. In our setting, each traversal permutation $\\pi_i$ defines a distinct causal ordering through which VMamba processes input patches, resulting in different trajectories of hidden states $\\{\\mathbf{h}(t)\\}$. Low-entropy (high-confidence) permutations consistently correspond to stable recurrent dynamics, where the influence of corruptions is minimized. This stabilizes the hidden-state evolution, yielding low-variance predictions and reducing sensitivity to domain-specific perturbations. From a loss landscape perspective, such paths correspond to regions of low curvature, as smoother predictions typically reflect flatter neighborhoods around $\\theta_i$ (parameters of adapted model with $\\pi_i$).\n\nTherefore, selecting low-entropy permutations acts as a proxy for identifying solutions that are both dynamically stable (in terms of hidden-state recurrence) and geometrically robust (in terms of curvature). Averaging over the adapted weights from these confidence paths further concentrates the model in a flat, low-loss region of parameter space. This synergy between entropy-based selection and weight averaging aligns directly with the generalization theory of SWAD [5], and provides a principled explanation for the observed robustness of TRUST across domains.\n\n\n\n**3-2. Sensitivity of the results to the choice of the hyperparameter K (number of top permutations selected) and the methodology used to determine its optimal value?**\n\nBy applying multiple traversal permutations, we expose the model to diverse causal perspectives of the same input. This variation encourages the model to learn different adaptation patterns at test time. Increasing the number of traversal permutations introduces more variation and, as shown in works such as Model-Soups [4], a greater number of diverse, correctly-aligned representations (our top-K permutations) can enhance the effectiveness of weight averaging. Figure 5 in the paper (table below) empirically supports this observation across three datasets. Our experiments shows that increasing permutation diversity improves robustness, with performance gains saturating around six permutations. This can offer a good trade-off between accuracy and efficiency.\n\n| Traversal Permutation Number | CIFAR10-C (%) | CIFAR100-C (%) | ImageNet-C (%) |\n|------------------------------|---------------|----------------|----------------|\n| 2                            | 75.6          | 51.7           | 54.4           |\n| 4                            | 77.1          | 53.7           | 55.6           |\n| 6                            | 77.5          | 54.3           | 56.1           |\n| 8                            | 77.6          | 54.7           | 55.5           |\n\n**Limitations**\n\nThank you for the thoughtful suggestions. We will incorporate a dedicated Limitations section in the final version, addressing potential societal impacts, including bias, adversarial robustness, ethical considerations, and environmental footprint.\n\n[1] Vmamba: Visual state space model. Advances in neural information processing systems, 37, 103031-103063.\n\n[2] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model.\n\n[3] LocalMamba: Visual State Space Model with Windowed Selective Scan. CoRR.\n\n[4] Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning (pp. 23965-23998). PMLR.\n\n[5] Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34, 22405-22418."}, {"review_id": "rvWjMZ4DLQ", "reviewer": "Reviewer_pWep", "summary": "TRUST addresses two key challenges faced by VMamba under distribution shifts: (1) the strong inductive bias introduced by fixed traversal scans, and (2) the accumulation of domain-specific artifacts in hidden states during sequential processing. To tackle these issues, TRUST introduces a test-time adaptation strategy specifically designed for vision-oriented state-space models.\n\nTo mitigate the first challenge, TRUST proposes Traversal Permutation, which involves training multiple versions of the model using different traversal orders. For the second challenge, TRUST merges the weight embeddings by averaging the parameters across all trained model variants, effectively reducing domain-specific biases.\n\nExperiments demonstrate that TRUST outperforms existing SOTA strategies and significantly improves VMamba's performance in cross-domain classification tasks.", "questions": "1. How is the value of K (i.e., the number of traversal permutations or model variants) determined across different images or datasets? Is K fixed, adaptive, or tuned per dataset or image? Clarifying this would help assess the generalizability and scalability of the method.\n\n2.  While the proposed strategy is evaluated on classification tasks, is it applicable to other vision tasks such as object detection or semantic segmentation?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strength: \n\n1. The paper proposes a novel strategy for adapting VMamba to domain-specific data.\n\n2. The strategy does not rely on external source data, making it easily transferable across datasets from different domains.\n\n3. The proposed method significantly improves VMamba’s performance in cross-domain classification tasks and outperforms other adaptation methods applied to VMamba.\n\n4. The writing is clear and easy to follow.\n\nweakness:\n\n1. The paper would benefit from a more thorough explanation of how the simple weight averaging across multiple models leads to performance gains. A deeper analysis or ablation study could strengthen the justification.", "was_revised": true, "final_justification": "Having reviewed your response and the comments from the other reviewers, I will be keeping my original score.", "rebuttal": "We thank the reviewer for their thoughtful review. We are glad our paper was recognized as novel and practical in addressing VMamba's limitations under distribution shifts. We appreciate the positive comments on the method's effectiveness, its source-free nature, and the clarity of the writing.\n\n**1. Explanation of how the simple weight averaging across multiple models leads to performance gains.**\n\nThe concept of weight averaging seeks to identify a solution in parameter space that lies within a flat region of the loss landscape. Such flat minima are associated with low loss values that remain stable under small perturbations to the model parameters. Models converging to these regions tend to generalize better under distribution shifts, as their performance is less sensitive to noise or minor changes in input data and parameters. In our approach, the integration of traversal permutations with weight averaging is inspired by **Robust Risk Minimization (RRM)** [1]. RRM aims to minimize the worst-case empirical loss within a neighborhood around the current parameters, formulated as:\n\n$E_{D}^{\\gamma}(\\theta) = \\max_{\\|\\Delta\\| \\leq \\gamma} \\hat{E}_{D}(\\theta + \\Delta)$\n\nwhere $\\gamma > 0$ defines the neighborhood around the model parameters $\\theta$, and $\\Delta$ represents small perturbations. While we do not directly optimize this objective, the principle aligns with our use of weight averaging across traversal permutations. By averaging the adapted weights, we converge toward a flatter region in weight space, improving generalization during test-time adaptation and enhancing robustness to distribution shifts.\n\n\n**2. How is the value of K (i.e., the number of traversal permutations or model variants) determined across different images or datasets? Is K fixed, adaptive, or tuned per dataset or image?**\n\nAs shown in Figure 5 of the paper (table below), we experimented with different values of K across multiple datasets. Prior work such as Model Soups [4] suggests that averaging a diverse set of well-aligned models (our top-K permutations) can improve performance. Consistent with this, our results indicate that accuracy generally increases with more permutations, but the gains saturate beyond K = 6. This suggests that using six permutations provides an effective balance between accuracy and computational cost. Accordingly, we fixed K = 6 for all experiments.\n\n| Traversal Permutation Number | CIFAR10-C (%) | CIFAR100-C (%) | ImageNet-C (%) |\n|------------------------------|---------------|----------------|----------------|\n| 2                            | 75.6          | 51.7           | 54.4           |\n| 4                            | 77.1          | 53.7           | 55.6           |\n| 6                            | 77.5          | 54.3           | 56.1           |\n| 8                            | 77.6          | 54.7           | 55.5           |\n\n**3. Is your method applicable to other vision tasks such as object detection or semantic segmentation?**\n\nThank you for this thoughtful question. We conducted additional experiments on segmentation tasks using various datasets, including Pascal VOC21 and Pascal Context 59. The results demonstrate that TRUST performs well in segmentation, outperforming methods like TENT. This further supports the generalizability and effectiveness of our approach beyond classification settings.\n\n### V21:\n\n| Corruption     | gaussian | shot  | impulse | defocus | glass | motion | zoom  | frost | snow  | fog   | brightness | contrast | elastic | pixelate | jpeg  | **Mean** |\n|----------------|----------|-------|---------|---------|-------|--------|-------|--------|--------|--------|-------------|----------|---------|-----------|--------|--------|\n| Source only    | 29.1     | 33.1  | 28.3    | 21.0    | 8.2   | 33.1   | 25.4  | 50.9   | 50.3   | 70.7   | 76.5        | 63.9     | 25.5    | 22.2      | 59.2   | **39.8** |\n| Tent           | 33.0     | 35.7  | 32.0    | 22.3    | 14.7  | 38.2   | 25.3  | 46.5   | 49.0   | 60.2   | 63.9        | 66.2     | 38.5    | 28.8      | 43.9   | **39.9** |\n| Trust          | 38.8     | 42.0  | 38.7    | 29.8    | 22.6  | 45.1   | 29.8  | 50.5   | 53.5   | 63.4   | 66.4        | 68.5     | 45.1    | 37.7      | 48.6   | **45.4** |\n\n\n### P59:\n\n| Corruption     | gaussian | shot  | impulse | defocus | glass | motion | zoom  | frost | snow  | fog   | brightness | contrast | elastic | pixelate | jpeg  | **Mean** |\n|----------------|----------|-------|---------|---------|-------|--------|-------|--------|--------|--------|-------------|----------|---------|-----------|--------|--------|\n| Source only    | 17.1     | 19.6  | 17.4    | 27.4    | 14.9  | 29.2   | 19.5  | 30.2   | 28.5   | 42.1   | 50.8        | 41.0     | 23.9    | 30.4      | 38.4   | **28.7** |\n| Tent           | 17.6     | 18.9  | 17.8    | 22.2    | 15.9  | 27.5   | 17.9  | 26.9   | 30.0   | 36.7   | 41.9        | 42.9     | 25.8    | 28.2      | 28.3   | **26.6** |\n| Trust          | 24.4     | 27.4  | 25.4    | 24.6    | 21.2  | 30.1   | 19.8  | 29.8   | 32.8   | 39.2   | 42.4        | 43.2     | 31.5    | 36.1      | 31.6   | **30.6** |\n\n\n\n[1] Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34, 22405-22418."}, {"review_id": "kfSYwm3mu1", "reviewer": "Reviewer_AAyK", "summary": "This paper introduces TRUST, a novel test-time adaptation (TTA) method specifically designed for the VMamba architecture. TRUST leverages the model's unique internal traversal mechanism by permuting the scanning order to generate diverse causal \"views\" of an input image. The model is then adapted on the most confident permutations, and the resulting weights are averaged to produce a single, more robust model for inference.", "questions": "- The experiment design only focuses on the simplest TTA setting. Testing TRUST under more challenging settings could further demonstrate its performance since the model architecture differs significantly from existing approaches—consider Continual TTA [7] or temporal correlation settings [6, 8, 9].\n- What's the latency when each batch requires all three phases (offline, adaptation, evaluation) to be performed?\n- Could the proposed method be applied to other tasks like segmentation?\n- I'm not familiar with Mamba—is it really necessary to develop a method specialized for it? Probably not a good question for this paper, but this might strengthen the paper's motivation if addressed.\n\nI'd be happy to raise my rating if these weaknesses and questions are thoroughly addressed in the rebuttal.", "limitations": "Though the author mentioned that limitations are in \" In the last part of the conclusion section.\" (Line 508), I failed to find it.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\nThe primary strength of this work is its novelty and clever use of the model's intrinsic properties. By treating the internal traversal paths as a form of implicit data augmentation, the method avoids the computational overhead of traditional augmentation techniques and is the first of its kind for State Space Models. The experimental results are impressive, showing consistent and significant improvements over well-established, general-purpose TTA methods across a variety of benchmarks.\n\n# Weaknesses\nMy main concerns with this paper relate to its specificity and experimental setup:\n\n- The core mechanism of TRUST is fundamentally tied to the VMamba architecture. The concept of permuting traversal paths cannot be applied to other popular architectures like CNNs or standard Vision Transformers, which limits the broader impact of the proposed method. It's an effective technique for one specific model family, not a general TTA paradigm. Moreover, the adaptation exclusively targets \"Mamba-specific state space parameters.\" The paper would benefit from justifying this choice—it's unclear if adapting other components (e.g., normalization layers, as in TENT) was explored and what the performance trade-offs might be.\n- For the VSS block, do you need to process the same image K times? This sounds even worse than data augmentation when the number of augmented samples is much smaller than K (e.g., 1). Moreover, are the traversal permutations fundamentally different from rotating the original image? For example, if we rotate the input image, would this be similar to changing from permutation (a) to (b) in Fig. 1? It seems like data augmentation methods require the model to process one image T times (T = number of augmented samples) while TRUST processes the same image K times with different traversal orders. Both methods require multiple forward passes per image, right? Also, for weight averaging approaches, there are works like [6] that achieve this without data augmentation (weight averaging prompt pools for the same image without multiple forward passes). A discussion of works in this direction is also necessary.\n- The baselines used are outdated. Stronger recent baselines [1-5] should be discussed to demonstrate performance. Also, [2, 4] are architecture-specific TTA methods (learning visual prompts for ViT) but could be extended to CNNs, whereas TRUST only works for VMamba. This makes the comparison potentially unfair since other methods are more architecture-agnostic.\n- The multi-stage process (generating permutations, ranking by entropy, separate adaptation for top-K paths, and final weight averaging) appears more complex and computationally intensive than single-pass methods like TENT, especially since it involves multiple backpropagation steps. A computation and memory comparison is needed.\n\n[1] Self-Bootstrapping for Versatile Test-Time Adaptation (ICML2025)\n\n[2] Test-Time Model Adaptation with Only Forward Passes (ICML2024)\n\n[3] Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation (ICML2025)\n\n[4] OT-VP: Optimal Transport-Guided Visual Prompting for Test-Time Adaptation (WACV2025)\n\n[5] Active Test-Time Adaptation: Theoretical Analyses and An Algorithm (ICLR2024)\n\n[6] DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation (ICML2025)\n\n[7] Continual test-time domain adaptation (CVPR2022)\n\n[8]. NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation (NeurIPS2022)\n\n[9]. Robust Test-Time Adaptation in Dynamic Scenarios (CVPR 2023)", "was_revised": true, "final_justification": "My concerns have been addressed, and I have raised my final score to 4.", "rebuttal": "We thank the reviewer for their thoughtful review. We are glad that the paper’s novelty and the use of VMamba’s intrinsic traversal mechanism were well appreciated. The recognition of our permutation strategy as a form of implicit augmentation, along with the strong empirical results, reinforces the effectiveness and originality of the proposed approach.\n\n**1. The experiment design only focuses on the simplest TTA setting. Consider Continual TTA [7] or temporal correlation settings [6, 8, 9].**\n\nWe would like to clarify that continual test-time adaptation and temporal correlation represents distinct subfields within test-time adaptation, with different assumptions and objectives, often requiring longer adaptation timelines and specialized setups. Additionally, most papers in the TTA literature do not compare their results against continual adaptation or temporally correlated test-time settings, such as those explored in [1, 2, 3]. Moreover, implementing and evaluating TRUST in this setting would require extensive design and tuning beyond the scope of a one-week rebuttal period. We will explore this in future works.\n\n**2. What's the latency when each batch requires all three phases (offline, adaptation, evaluation) to be performed?**\n\nOur method consists of three stages:\n**Offline phase**: We evaluate our pre-trained model under different traversal permutations and select the top-k permutations with the lowest entropy values, corresponding to the most confident predictions. Since no model adaptation occurs at this stage, the procedure is equivalent to a forward-pass evaluation and is computationally lightweight. For instance, on the PACS dataset, evaluating for a single permutation takes only **4 seconds** for all images in this dataset. Extending this across all 24 permutations results in a total evaluation time of **1.5 minutes**, which is a one-time cost. Our experiments show that the top-K permutations remain consistent across all test datasets, as visualized in Figure 2 of the supplementary material. This consistency stems from the similarity of these permutations to the original traversal order used during pre-training, as reflected in their high confidence scores. Therefore, this procedure can be performed only once on a single dataset to determine the appropriate Ks, and then used across all test datasets. \n**Adaptation stage**: We measured the latency for the batch size of 128 used in our experiments for forward and backward passes, which results in approximately **1.2 second** per permutation. \n**Evaluation stage**: In this phase, we first perform weight averaging, which involves a simple averaging of the model weights across selected permutations (**0.018 second**). The subsequent evaluation functions similarly to the offline stage, no adaptation is performed, and we simply test the model (**0.2 second**). Our overall per permutation is: 1.2 + 0.018 + 0.2 = 1.41 second.\n\n**3. Could the proposed method be applied to other tasks like segmentation?**\n\nWe conducted additional experiments on segmentation tasks using various datasets, including Pascal VOC21 and Pascal Context 59. The results demonstrate that TRUST performs well in segmentation, outperforming methods like TENT. This further supports the generalizability and effectiveness of our approach beyond classification settings.\n\n### V21:\n\n| Corruption  | gaussian | shot | impulse | defocus | glass | motion | zoom | frost | snow | fog | brightness | contrast | elastic | pixelate | jpeg | **Mean** |\n|-------------|----------|------|---------|---------|-------|--------|------|--------|------|-----|-------------|----------|---------|-----------|------|----------|\n|Source only|29.1|33.1|28.3|21.0|8.2|33.1|25.4|50.9|50.3|70.7|76.5|63.9|25.5|22.2|59.2| **39.8** |\n|Tent|33.0|35.7|32.0|22.3|14.7|38.2|25.3|46.5|49.0|60.2|63.9|66.2|38.5|28.8|43.9| **39.9** |\n|Trust|38.8|42.0|38.7|29.8|22.6|45.1|29.8|50.5|53.5|63.4|66.4|68.5|45.1|37.7|48.6| **45.4** |\n\n\n\n### P59:\n\n| Corruption     | gaussian | shot  | impulse | defocus | glass | motion | zoom  | frost | snow  | fog   | brightness | contrast | elastic | pixelate | jpeg  | **Mean** |\n|----------------|----------|-------|---------|---------|-------|--------|-------|--------|--------|--------|-------------|----------|---------|-----------|--------|--------|\n| Source only|17.1|19.6|17.4|27.4|14.9|29.2|19.5|30.2|28.5|42.1|50.8|41.0|23.9|30.4|38.4   | **28.7** |\n| Tent|17.6| 18.9| 17.8|22.2|15.9|27.5|17.9|26.9|30.0| 36.7|41.9|42.9|25.8|28.2|28.3| **26.6** |\n| Trust|24.4|27.4|25.4|24.6|21.2|30.1|19.8|29.8|32.8|39.2|42.4|43.2|31.5|36.1|31.6| **30.6** |\n\n\n**4. Is it really necessary to develop a method specialized for it?**\n\nAs we discussed in the introduction, Mamba is emerging as a powerful alternative to both CNNs and ViTs across a range of vision tasks. Unlike ViTs, which rely on self-attention and quadratic complexity with respect to sequence length, Mamba is built on SSMs that enable linear-time sequence modeling, making it highly efficient for long input sequences. This makes Mamba especially attractive in settings with limited compute resources. Despite being more lightweight, Mamba retains competitive or even superior performance in many tasks. Several recent papers [4, 5, 6] have shown that Mamba outperforms CNNs and ViTs on image classification, segmentation, and even video understanding, while using fewer parameters and less memory. \nPrevious test-time adaptation methods, such as TENT and SHOT, are not well-suited for Mamba-based models, as demonstrated in Table 2 of our paper. Due to the unique architecture of Mamba and favorable properties, such as linear-time complexity, we focused on designing a strategy specifically tailored for Mamba to effectively handle domain shifts during test time in vision applications.\n\n\n**5. The adaptation exclusively targets \"Mamba-specific state space parameters.\" How does the performance differ when other components (e.g., normalization layers, as in TENT) are adapted?**\n\nWe provided this experiment in table 2 of our supplementary material. The results represent that our method outperforms TENT in both settings, by 3.1% when using BatchNorm adaptation and by 11.8% with SS2D adaptation. We opt to proceed with SS2D adaptation, as traversal permutations directly influence Mamba-specific parameters encoded in the SS2D blocks. Updating these parameters is therefore essential to fully capture the effects of traversal-based modifications. \n\n### ImageNet-C:\n\n|Metric|Source only|TENT (BN)|TRUST (BN)|TENT (SS2D)|TRUST (SS2D)|\n|-------------|-------------|-----------|------------|-------------|--------------|\n|**Mean**|**38.7**|**41.7**|**44.8**|**44.3**|**56.1**|\n\n\n**6. Compare with the suggested baselines.**\n\nDue to architectural differences between VMamba and other backbones used in the baselines mentioned by the reviewer, a direct comparison would not be entirely fair. These methods, which rely on ViT-based architectures, will be included in a separate category in the final version of the paper. However, to provide a fair evaluation, we adapted the ViT-based method from [2] by replacing its backbone with VMamba and modifying it to be compatible with VMamba backbone. Specifically, we replaced the CLS token (absent in VMamba) with the mean of all tokens and adjusted the number of learnable tokens to match VMamba’s dimensionality. This ensures that both methods operate under the same architectural constraints. Our experimental results demonstrate that TRUST, which is specifically designed to exploit VMamba’s traversal mechanism, significantly outperforms the adapted baseline, highlighting its robustness under distribution shift.\n\n### ImageNet-C:\n\n| Corruption   | gaussian | shot  | impulse | defocus | glass | motion | zoom  | frost | snow  | fog   | brightness | contrast | elastic | pixelate | jpeg  | **Mean** |\n|--------------|----------|-------|---------|---------|--------|--------|-------|--------|--------|--------|-------------|----------|---------|-----------|--------|--------|\n| Source only| 24.3| 26.1| 25.1| 22.2| 23.2| 35.4| 43.2| 49.3| 48.4| 56.9| 70.0| 26.8| 45.1| 43.7| 41.4   | **38.7** |\n| Tent| 27.8| 30.0| 28.8| 24.9| 25.9| 38.0| 45.5| 51.0| 51.3| 59.1| 70.6| 30.0| 48.2| 47.8| 45.7| **41.7** |\n| Trust| 46.8| 49.4| 48.5| 42.8| 40.8| 57.1| 57.9| 57.3| 61.7| 66.8| 71.9| 54.9| 61.4| 63.6| 60.2| **56.1** |\n| FOA| 17.8| 19.4| 18.5| 15.1| 18.2| 22.66| 28.6| 38.69| 33.9| 44.3| 57.4| 22.7| 38.2| 40.9| 41.7| **30.5** |\n\n\n**7. Are the traversal permutations fundamentally different from rotating the original image?**\n\nWhile image rotation changes the spatial arrangement of tokens, it is not equivalent to the traversal permutations we apply within VMamba, as the specific traversal orders used in our method cannot be replicated through simple rotations. As shown in Figure 4 of our paper, replacing traversal permutations with standard image rotations leads to significantly lower performance. This confirms that the gains from TRUST arise specifically from manipulating the causal processing order through traversal permutations, rather than from altering the input appearance via rotation.\n\n\n[1] WATT: Weight average test time adaptation of CLIP. Advances in neural information processing systems, 37, 48015-48044.\n\n[2] Clipartt: Adaptation of clip to new domains at test time. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (pp. 7092-7101). IEEE.\n\n[3] Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35, 14274-14289.\n\n[4] Vmamba: Visual state space model. Advances in neural information processing systems, 37, 103031-103063.\n\n[5] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model.\n\n[6] LocalMamba: Visual State Space Model with Windowed Selective Scan. CoRR."}]

Author rebuttal:

Meta-review: Reviewers confirmed the novelty, thorough experiments and writing of this paper. After rebuttal and discussion, most of the concerns raised by reviewers are resolved the authors and all the ratings are positive.  I thereby recommend to accept this submission.
Decision: Accept (poster)