Title: TRUST: Test-Time Refinement using Uncertainty-Guided SSM Traverses
Venue: NeurIPS 2025
Accepted: yes
Keywords: Structured State-Space Models

VMamba

Test-Time Adaptation

Distribution Shift
Abstract: State Space Models (SSMs) have emerged as efficient alternatives to Vision Transformers (ViTs), with VMamba standing out as a pioneering architecture designed for vision tasks. However, their generalization performance degrades significantly under distribution shifts. To address this limitation, we propose TRUST (Test-Time Refinement using Uncertainty-Guided SSM Traverses), a novel test-time adaptation (TTA) method that leverages diverse traversal permutations to generate multiple causal perspectives of the input image. Model predictions serve as pseudo-labels to guide updates of the Mamba-specific parameters, and the adapted weights are averaged to integrate the learned information across traversal scans. Altogether, TRUST is the first approach that explicitly leverages the unique architectural properties of SSMs for adaptation. Experiments on seven benchmarks show that TRUST consistently improves robustness and outperforms existing TTA methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TRUST, a novel test-time adaptation (TTA) method specifically designed for the VMamba architecture. TRUST leverages the model's unique internal traversal mechanism by permuting the scanning order to generate diverse causal \"views\" of an input image. The model is then adapted on the most confident permutations, and the resulting weights are averaged to produce a single, more robust model for inference.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\nThe primary strength of this work is its novelty and clever use of the model's intrinsic properties. By treating the internal traversal paths as a form of implicit data augmentation, the method avoids the computational overhead of traditional augmentation techniques and is the first of its kind for State Space Models. The experimental results are impressive, showing consistent and significant improvements over well-established, general-purpose TTA methods across a variety of benchmarks.\n\n# Weaknesses\nMy main concerns with this paper relate to its specificity and experimental setup:\n\n- The core mechanism of TRUST is fundamentally tied to the VMamba architecture. The concept of permuting traversal paths cannot be applied to other popular architectures like CNNs or standard Vision Transformers, which limits the broader impact of the proposed method. It's an effective technique for one specific model family, not a general TTA paradigm. Moreover, the adaptation exclusively targets \"Mamba-specific state space parameters.\" The paper would benefit from justifying this choice—it's unclear if adapting other components (e.g., normalization layers, as in TENT) was explored and what the performance trade-offs might be.\n- For the VSS block, do you need to process the same image K times? This sounds even worse than data augmentation when the number of augmented samples is much smaller than K (e.g., 1). Moreover, are the traversal permutations fundamentally different from rotating the original image? For example, if we rotate the input image, would this be similar to changing from permutation (a) to (b) in Fig. 1? It seems like data augmentation methods require the model to process one image T times (T = number of augmented samples) while TRUST processes the same image K times with different traversal orders. Both methods require multiple forward passes per image, right? Also, for weight averaging approaches, there are works like [6] that achieve this without data augmentation (weight averaging prompt pools for the same image without multiple forward passes). A discussion of works in this direction is also necessary.\n- The baselines used are outdated. Stronger recent baselines [1-5] should be discussed to demonstrate performance. Also, [2, 4] are architecture-specific TTA methods (learning visual prompts for ViT) but could be extended to CNNs, whereas TRUST only works for VMamba. This makes the comparison potentially unfair since other methods are more architecture-agnostic.\n- The multi-stage process (generating permutations, ranking by entropy, separate adaptation for top-K paths, and final weight averaging) appears more complex and computationally intensive than single-pass methods like TENT, especially since it involves multiple backpropagation steps. A computation and memory comparison is needed.\n\n[1] Self-Bootstrapping for Versatile Test-Time Adaptation (ICML2025)\n\n[2] Test-Time Model Adaptation with Only Forward Passes (ICML2024)\n\n[3] Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation (ICML2025)\n\n[4] OT-VP: Optimal Transport-Guided Visual Prompting for Test-Time Adaptation (WACV2025)\n\n[5] Active Test-Time Adaptation: Theoretical Analyses and An Algorithm (ICLR2024)\n\n[6] DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation (ICML2025)\n\n[7] Continual test-time domain adaptation (CVPR2022)\n\n[8]. NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation (NeurIPS2022)\n\n[9]. Robust Test-Time Adaptation in Dynamic Scenarios (CVPR 2023)\n\nQUESTIONS:\n- The experiment design only focuses on the simplest TTA setting. Testing TRUST under more challenging settings could further demonstrate its performance since the model architecture differs significantly from existing approaches—consider Continual TTA [7] or temporal correlation settings [6, 8, 9].\n- What's the latency when each batch requires all three phases (offline, adaptation, evaluation) to be performed?\n- Could the proposed method be applied to other tasks like segmentation?\n- I'm not familiar with Mamba—is it really necessary to develop a method specialized for it? Probably not a good question for this paper, but this might strengthen the paper's motivation if addressed.\n\nI'd be happy to raise my rating if these weaknesses and questions are thoroughly addressed in the rebuttal.\n\nLIMITATIONS:\nThough the author mentioned that limitations are in \" In the last part of the conclusion section.\" (Line 508), I failed to find it.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTRUST addresses two key challenges faced by VMamba under distribution shifts: (1) the strong inductive bias introduced by fixed traversal scans, and (2) the accumulation of domain-specific artifacts in hidden states during sequential processing. To tackle these issues, TRUST introduces a test-time adaptation strategy specifically designed for vision-oriented state-space models.\n\nTo mitigate the first challenge, TRUST proposes Traversal Permutation, which involves training multiple versions of the model using different traversal orders. For the second challenge, TRUST merges the weight embeddings by averaging the parameters across all trained model variants, effectively reducing domain-specific biases.\n\nExperiments demonstrate that TRUST outperforms existing SOTA strategies and significantly improves VMamba's performance in cross-domain classification tasks.\n\nSTRENGTHS AND WEAKNESSES:\nstrength: \n\n1. The paper proposes a novel strategy for adapting VMamba to domain-specific data.\n\n2. The strategy does not rely on external source data, making it easily transferable across datasets from different domains.\n\n3. The proposed method significantly improves VMamba’s performance in cross-domain classification tasks and outperforms other adaptation methods applied to VMamba.\n\n4. The writing is clear and easy to follow.\n\nweakness:\n\n1. The paper would benefit from a more thorough explanation of how the simple weight averaging across multiple models leads to performance gains. A deeper analysis or ablation study could strengthen the justification.\n\nQUESTIONS:\n1. How is the value of K (i.e., the number of traversal permutations or model variants) determined across different images or datasets? Is K fixed, adaptive, or tuned per dataset or image? Clarifying this would help assess the generalizability and scalability of the method.\n\n2.  While the proposed strategy is evaluated on classification tasks, is it applicable to other vision tasks such as object detection or semantic segmentation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes TRUST, a TTA method that capitalizes on the architectural properties of VMamba, a vision-adapted State Space Model. It tackles the issue of performance degradation of SSMs under distribution shifts, which arises from strong inductive biases and the accumulation of domain-specific artifacts in hidden states due to fixed traversal paths. TRUST generates multiple traversal permutations of input images, which are then used to create diverse causal perspectives. By selecting the top-K permutations with the lowest predictive entropy and averaging the Mamba-specific parameters, the method mitigates the impact of noisy predictions and promotes robustness. The authors validate TRUST on various corruption-based and domain generalization benchmarks, showing consistent improvements over baseline TTA approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n * Novelty in SSM Adaptation: TRUST is presented as the first TTA approach explicitly designed for Mamba-based vision models, leveraging their internal traversal dynamics rather than relying on external data augmentation or auxiliary models. This is a significant contribution to the field of TTA for emerging architectures like SSMs.\n * Effective Use of Architectural Properties: The method effectively utilizes the unique traversal mechanism of VMamba to generate diverse causal perspectives of the input, which is a clever way to induce variability and explore flatter minima in the loss landscape without additional computational overhead from data augmentations.\n * Strong Experimental Results: The paper provides comprehensive experimental validation on seven standard benchmarks, including CIFAR10-C, CIFAR100-C, ImageNet-C, PACS, ImageNet-Sketch, ImageNet-V2, and ImageNet-R. TRUST consistently outperforms existing state-of-the-art TTA methods, demonstrating significant improvements in robustness and generalization under various distribution shifts.\n * Clear Ablation Studies: The paper includes \"TRUST naive\" as a baseline which adapts only Mamba-specific SS2D parameters, while \"TRUST\" further enhances robustness by averaging over multiple traversal permutations during adaptation. This effectively highlights the contribution of the proposed permutation and weight averaging strategy.\n\nWeaknesses\n * Complexity of Traversal Permutations: While novel, the concept of systematically generating and selecting traversal permutations based on entropy might introduce a non-trivial computational overhead during the offline phase, especially if the number of permutations or the size of the model increases. The paper states that \"All experiments were conducted using a single NVIDIA A6000 GPU\", but doesn't elaborate on the time taken for the offline phase. Further analysis on the scalability of this permutation generation and selection process would strengthen the paper.\n * Limited Generalizability to Other Architectures: The method is highly specific to the architectural properties of VMamba and its SS2D module. While this is presented as a strength, it also implies that the core mechanism of TRUST might not be directly transferable to other backbone architectures (e.g., CNNs or standard Transformers) without significant modifications. This limits its broad applicability across the wider machine learning community.\n * Detailed Explanation of \"Confidence Paths\": The paper mentions that TRUST \"averages model weights from the most confident paths\", and selects permutations \"with the lowest entropy, which are associated with more stable and domain-robust hidden states\". While the link to flat minima is made, a deeper theoretical or empirical dive into why these \"confident paths\" (low entropy permutations) consistently lead to better generalization beyond the current explanations would be beneficial.\n * Discussion on Hyperparameter Sensitivity: The method involves selecting the top-K permutations. A more detailed discussion on the sensitivity of the results to the choice of K, and how this hyperparameter is determined, would improve the completeness of the experimental analysis.\n\nQUESTIONS:\n* Could you provide a more detailed analysis of the computational overhead, particularly the time complexity, of the traversal permutation generation and selection process in the offline phase? How does the scalability of this component affect the practical application of TRUST, especially with larger models or an increased number of permutations?\n * Given that TRUST's core mechanism is highly specific to the architectural properties of VMamba and its SS2D module, could you elaborate on the potential challenges or modifications required to adapt the underlying principles of uncertainty-guided traversal to other neural network architectures, such as CNNs or standard Transformers?\n * Regarding the \"confidence paths\" and the selection of permutations with the lowest entropy: Could you provide a deeper theoretical or empirical explanation as to why these specific paths consistently lead to better generalization and more stable, domain-robust hidden states, beyond the current explanation of exploring flatter minima? Additionally, could you discuss the sensitivity of the results to the choice of the hyperparameter K (number of top permutations selected) and the methodology used to determine its optimal value?\n\nLIMITATIONS:\n* Dedicated Limitations Section: Authors should include a \"Limitations\" section to acknowledge the inherent constraints and potential drawbacks of their method. This could include:\n   * Computational Overhead: Acknowledge and quantify the potential computational cost associated with generating and selecting traversal permutations, especially in real-time or resource-constrained environments.\n   * Specificity to VMamba: Explicitly state that the method's reliance on VMamba's unique architectural properties limits its direct applicability to other model architectures. Discuss whether the core ideas could be generalized and what challenges might arise.\n   * Hyperparameter Sensitivity: Discuss the sensitivity of the results to key hyperparameters, such as the number of top-K permutations, and the robustness of the method across a wider range of hyperparameters.\n   * Theoretical Guarantees: Address the lack of strong theoretical guarantees for why \"confident paths\" (low entropy permutations) consistently lead to better generalization.\n * Discussion on Potential Negative Societal Impact: It's crucial to discuss any potential negative societal impacts, even if seemingly minor or indirect. This could involve:\n   * Bias Amplification: Consider whether the adaptation process could inadvertently amplify existing biases present in the training data, especially when applied to real-world, diverse datasets.\n   * Adversarial Robustness: While improving robustness to natural distribution shifts, discuss if the method inadvertently introduces new vulnerabilities to adversarial attacks or if its effectiveness in adversarial settings is yet to be explored.\n   * Ethical Considerations in Deployment: Briefly touch upon the ethical considerations when deploying such adaptive models, particularly in high-stakes applications where errors due to unaddressed biases or unforeseen shifts could have significant consequences.\n   * Environmental Impact: While not explicitly explored, a brief mention of the energy consumption associated with extensive training and adaptation (if applicable) could be considered, aligning with broader discussions on sustainable AI.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a method called TRUST, to adapt vision SSMs at test-time in out-of-distribution conditions. The core idea in TRUST is to leverage variations of the patch traversal mechanism in the VMamba style architectures. Specifically, the authors permute the order of traversals and pick top-k orders that have minimal entropy on the test-time distribution. Then, the authors increase the sharpness of the predicted distribution by self-training on the pseudo-labels assigned by each of the top-k traversal orders to obtain a weight per traversal order. Eventually, the authors average the weights across all the top-k permutations to get the network weights. Through experiments, the authors demonstrate that TRUST outperforms algorithms like TENT and SHOT, when applied to the base VMamba model. The authors demonstrate these results on ImageNet and CIFAR corruption datasets, and other OOD shifts including PACS, ImageNet-S/v2/R. The authors further conduct several ablations to establish the impact of batch-size, input augmentations, number of traversal permutations, adaptation iterations, ensembling methods, specificity to permutations and computational costs with increase in traversals. Overall, within the purview of the conducted experiments, the authors propose a useful TTA method specifically for VMamba style architectures.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe following points highlight the strengths associated with the submission.\n\n1. The paper is generally well-written and easy to follow. The authors do a good job of motivating the architecture specific TTA method with TRUST. There is novelty in the sense that authors explicitly exploit the traversal procedure of VMamba to improve generalization. That said, this comes with the cost of being limited to a specific architecture.\n2. The authors conduct thorough experiments to outline the method’s effectiveness. The controlled set of experiments across differing visual distribution shifts clearly demonstrate that varying the traversal mechanisms in itself can provide substantial utility over applying more architecture-agnostic methods like TENT or SHOT. The margins over the base algorithms are quite substantial in some cases, highlighting the usefulness of the proposed method.\n3. Further, to justify and assess the impact of different design decisions, the authors conduct thorough ablations – ranging from the impact of batch-sizes, to specificity to permutation orders. For future work attempting to build on top of TRUST, these ablations serve as useful starting points to iterate on design decisions.\n\n**Weaknesses:**\n\nI have the following weaknesses to point out in the submission.\n\n1. The model performance being sensitive to the traversal order (abcd > badc) seems odd. Ideally, at test-time, one doesn’t have access to ground-truth to infer the best performing permutation. The submission might benefit from a little discussion about this observation. This being an outcome despite weight averaging seems especially odd – the model is likely carrying over some bias from the source training dataset that TRUST isn’t able to overcome. Concretely, two things would help improve the submission in this regard – (1) an investigation about why this is happening? and (2) if it’s possible to find a reliable heuristic to choose the best test-time permutation.\n2. The method generally seems very computationally intensive – both in terms of memory and latency. A natural counter-opinion is to ask if similar performance gains can be squeezed out of existing methods (combining augmentations, finding the best TENT, SHOT TTA variants) instead of adopting such a computationally expensive method. Understanding the limits or the experimental settings in which it’s feasible to apply TRUST will help understand downstream use-cases.\n\nQUESTIONS:\nI think addressing the points outlined under weaknesses would help improve my rating of the submission. Specifically, the sensitivity to a particular traversal order and assessing the computational overhead vs performance gain trade-off (where best possible performance is squeezed out of a combination of strong augmentations and prior TTA methods).\n\nLIMITATIONS:\nYes. The authors do acknowledge the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}