Title: Feature-Based Instance Neighbor Discovery: Advanced Stable Test-Time Adaptation in Dynamic World
Venue: NeurIPS 2025
Accepted: yes
Keywords: Test time adaptation

domain adaptation

continual learning
Abstract: Despite progress, deep neural networks still suffer performance declines under distribution shifts between training and test domains, leading to a substantial decrease in Quality of Experience (QoE) for applications. Existing test-time adaptation (TTA) methods are challenged by dynamic, multiple test distributions within batches.  We observe that feature distributions across different domains inherently cluster into distinct groups with varying means and variances. This divergence reveals a critical limitation of previous global normalization strategies in TTA, which inevitably distort the original data characteristics. Based on this insight, we propose Feature-based Instance Neighbor Discovery (FIND), which comprises three key components: Layer-Wise Feature Disentanglement (LFD), Feature-Aware Batch Normalization (FABN) and Selective FABN (S-FABN). LFD stably captures features with similar distributions at each layer by constructing graph structures; while FABN optimally combines source statistics with test-time distribution-specific statistics for robust feature representation. Finally, S-FABN  determines which layers require feature partitioning and which can remain unified, thus enhancing the efficiency of inference. Extensive experiments demonstrate that FIND significantly outperforms existing methods, achieving up to approximately 30\% accuracy improvement in dynamic scenarios while maintaining computational efficiency. The source code is available at https://github.com/Peanut-255/FIND.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FIND, a method for dynamic test-time adaptation where test batches contain mixed distributions. FIND combines feature-based clustering (LFD), adaptive batch normalization (FABN), and selective application (S-FABN) to improve robustness under distribution shifts. Experiments show gains in accuracy and efficiency compared to existing TTA methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The paper is well written, with clear explanations that make the technical content accessible. The logical flow and precise language enhance the overall readability.\n2. The visualizations, including figures and tables, are carefully designed and effectively support the narrative. They provide clear and intuitive illustrations of the proposed method and its empirical results.\n\n\n## Weaknesses\n1. The problem setting considered in this work appears closely related to DDA [1]. While DDA is cited in the introduction, the paper does not provide a direct empirical comparison against DDA, nor does it offer a thorough discussion clarifying the differences between the two settings. This omission makes it difficult to precisely assess the novelty and contributions relative to existing work.\n2. The experimental validation is limited to ResNet-50. The lack of evaluation on a broader range of network architectures raises concerns regarding the generalizability and robustness of the proposed method across different model backbones.\n\n[1] Gao J, Zhang J, Liu X, et al. Back to the source: Diffusion-driven adaptation to test-time corruption[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 11786-11796.\n\nQUESTIONS:\n1. Comparison with DDA: The proposed setting seems closely related to DDA. Could the authors provide a direct experimental comparison with DDA under the same evaluation protocol? Additionally, a more detailed discussion distinguishing the problem settings and assumptions would help clarify the novelty of this work.\n2. Generality Across Architectures: The current experiments are limited to ResNet-50. Have the authors tested the proposed method on other network architectures such as ViTs or ConvNeXt? Demonstrating consistent performance improvements across diverse model families would enhance the method’s perceived robustness and practical utility.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes FIND, a novel test-time adaptation method designed for dynamic scenarios where test samples may come from multiple distributions. FIND addresses the limitations of existing normalization strategies by introducing a divide-and-conquer approach, comprising Layer-wise Feature Disentanglement (LFD) to partition features, Feature Aware Batch Normalization (FABN) to combine source and test-specific statistics, and Selective FABN (S-FABN) to enhance efficiency. Extensive experiments on benchmark datasets demonstrate FIND's superior performance, achieving significant accuracy improvements over state-of-the-art methods while maintaining computational efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper introduces a novel divide-and-conquer normalization approach tailored for dynamic scenarios, effectively addressing the issue of interference between different distributions and leading to a 30% accuracy improvement in dynamic scenarios compared to existing methods.\n2. FIND demonstrates robust performance across various architectures and datasets, achieving superior results on both convolutional networks and transformer-based models while maintaining consistent accuracy regardless of batch size.\n3. FIND achieves significant computational efficiency with low memory usage and fast inference speed, making it highly practical for real-world deployment.\n\nWeaknesses:\n1. The experiments are conducted on a limited set of benchmark datasets (CIFAR10-C, CIFAR100-C, and ImageNet-C). While these datasets are widely used, they may not fully capture the diversity of real-world distribution shifts. Evaluating FIND on a broader range of datasets could provide a more comprehensive assessment of its robustness and effectiveness.\n2. The cold-start phase, which involves computing the average sensitivity score for each layer, may introduce additional computational overhead at the beginning of the adaptation process. This could be a drawback in scenarios where rapid adaptation is required.\n3. How does the proposed method perform on larger models with high-dimensional feature spaces? Whether the computational complexity of the layer-wise feature disentanglement and selective normalization becomes prohibitive?\n4. More related methods should be discussed or compared in the manuscript for a comprehensive analysis, including but not limited to [A-D].\n\nMinor issues:\n1. In Eqn.(1), \\sum_L  F_{i;c;L} should be \\sum_{l=0}^L F_{i;c;l}.\n2. In line 193, F_{j;c;L} is not mentioned in any equation.\n\nReferences:\n[A] Ranked Entropy Minimization for Continual Test-Time Adaptation, ICML 2025.\n[B] Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting, TPAMI 2025.  \n[C] Test-Time Model Adaptation for Visual Question Answering With Debiased Self-Supervisions, TMM 2023.\n[D] COME: Test-time adaption by Conservatively Minimizing Entropy, ICLR 2025.\n\nQUESTIONS:\nPlease refer to the Weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors consider test-time adaptation under a more complicated real-world scenario, where multiple distinct distributions may emerge simultaneously. This impose new challenges to previous test-time adaptation methods. Motivated by this, authors propose to  exploit the per-lay feature map similarity among test samples in the same mini-batch to construct graph structures. Afterwards, batch normalization is applied under the combination of source batch normalization (SBN) statistics and test-feature-specific-normalization (TFN) statistics, which are computed on a per graph connected component basis. This captures domain-specific feature distributions and also alleviates statistical bias in TFN estimations with the introduction of SBN statistics. In addition, to improve the computation efficiency, authors propose to dynamically determine which layer should be handled by graph partition, thus reducing unnecessary computation for layers insensitive to domain shifts. Empirical evaluation on ResNet and ViT models further proves the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The test-time adaptation scenarios with multiple test distributions considered in the paper is well motivated and more closely-related to real-world settings. Authors identify the performance degradation of existing time-time adaptation methods under this setting, which naturally leads to the proposal of layer-wise feature disentanglement.\n* Authors also point out that not all the layers should be handled with similarity-based graph construction and feature-aware batch normalization. Instead, they propose to use the KL divergence to measure the shift between source and current target domains, which serves as the criterion for whether graph partition is needed for the current layer. This also leads to more efficient inference.\n\nWeaknesses:\n* Althought the selective feature aware batch normalization (S-FABN) is well motivated, the design of KL-divergence based criterion seems somehow problematic. Since graph partition is needed when there are multiple test distributions presented in a single mini-batch, then the criterion should reflect the distribution diversity among test samples, rather the difference between source and target domains.\n* Some experimental results are not thoroughly analyzed (please refer to subsequent questions for details). Besides, sensitivity analysis of $\\alpha$ and efficiency comparison against $\\alpha$-BN is missing.\n\nQUESTIONS:\n1. Section 2.4, Eq. (5) and Eq. (6). Based on my understanding, the feature aware batch normalization (FABN) is proposed to resolve the potential statistical bias in TFN estimations when the corresponding subset is small. Thus, the relative contribution of source batch normalization (SBN) should be dependent on the subset size, rather than fixed as a hyper-parameter $\\alpha$.\n2. Section 2.5, line 238. The selective feature aware batch normalization (S-FABN) relies on a cold-start phase to esimate the average score for each layer. What if the test distribution changes significantly after the cold-start phase? Will this leads to performance degradation?\n3. Sensitivity analysis of $\\alpha$, the trade-off coefficient between SBN and TFN, is missing.\n4. Section 3.6, Table 4. $\\alpha$-BN is not included in the comparison of memory and latency of different test-time adaptation methods.\n5. Section 3.7, Figure 7. The proposed method works well even under small batch size, which is quite interesting. For instance, with BS=1, the graph partition should have no effect completely, then why the proposed method still outperforms all the baseline methods?\n\nLIMITATIONS:\nYes, limitations are provided in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses test-time adaptation in dynamic scenarios. The authors propose FIND, a divide-and-conquer test-time normalization strategy that replaces existing batch normalization layers and is adaptable to various backbones. FIND partitions features at different layers, effectively avoiding interference between samples from different distributions. Experiments demonstrate that FIND achieves superior performance in the more complicated TTA scenarios for BN models. I believe that using the first-neighbor approach provides a sound solution to decouple features from multiple domains, which may inspire new methods for the TTA community. I thus recommend weak acceptance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. This paper addresses a challenging scenario where test data from multiple domains are mixed in the data stream, aiming to improve the robustness of TTA in the dynamic wild world.\n\n2. The structure of this paper is well-organized, introducing the method's design and underlying logic through observations, with a clear and well-articulated motivation.\n\n3. The proposed method, FIND, provides a sound and nice solution to decouple features from multiple domains, thereby achieving more precise normalization for samples from different domains. Experiments also verify the superiority of Find for calibrating the BN layer at test time.\n\n**Weaknesses**:\n\n1. Incomplete ablation study. First, it is suggested to provide the ablation and sensitivity analysis of $\\alpha$ to support the claim of mitigating the domain-specific feature perturbations. Second, it is recommended to include $\\alpha$-BN and IABN in Figure 7 for more rigorous comparisons.\n\n2. Lack of compatibility evaluation with the test-time tuning method. It is insightful to show whether the proposed method can be integrated with the test-time tuning approaches, such as Tent and ETA, to further boost the TTA performance.\n\n3. Some important related works are missing. Since FIND aims to decouple the TTA process of different domains, it is suggested to include discussions or comparisons with the domain-aware multi-modeling TTA techniques, such as DPCore [1], BeCoTTA [2], and CoLA [3].\n\n4. The introduction currently overlooks test-time adaptation methods built for truly “in-the-wild” settings. A brief overview of these approaches would anchor FIND within the broader TTA community, while underscoring that FIND tackles the stricter mixed-domain shift problem and is tailored for the BN models.\n\n[1] DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation. ICML 2025.\n\n[2] BECoTTA: Input-dependent Online Blending of Experts for Continual Test-time Adaptation. ICML 2024.\n\n[3] Cross-Device Collaborative Test-Time Adaptation. NeurIPS 2024.\n\nQUESTIONS:\n1.\tWhy is the performance of Find* omitted from Table 2?\n\n2.\tHow is accuracy measured for clustering algorithms in Table 3?\n\nLIMITATIONS:\nPlease refer to the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of test-time adaptation in dynamic setting, where is target domain is a mixture of multiple distributions. Under this setting, standard one-size-fits-all test-time normalization techniques suffer from cross-distribution averaging. To solve this problem, this paper proposes FIND, which can captures features with similar distribution and apply different strategy to each layer. The proposed algorithm has great performance on benchmarking datasets.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n- The solution is well-motivated. Mixture distribution is very common in real-world tasks. The author starts by observing the drop in performance, and leads to a \"divide and conquer\" normalization approach. \n- The proposed method has great performance in corruption benchmarks\n- From Figure 6, it looks like FABN outperforms TBN under a wide range of $\\gamma$ (basically TBN is the proposed method with $\\gamma=1$). \n\n# Weaknesses\n- The idea of interpolating between source and target statistics is very common in previous works, e.g., $\\alpha$-BN. Section 2.4 basically applies $\\alpha$-BN to each connected component. \n- The motivation of this paper is that TBN and relevant test-time normalization algorithms assume that the target domain is homogeneous (i.e., one-size-fits-all). However, the authors ignore the fact that there are already some test-time normalization papers consider mixture of multiple distributions. One relevant paper is [1], which fits multiple sets of running statistics for each normalization layer. However, the author didn’t mention this, or compare the proposed method to this paper. \n\n[1] Un-Mixing Test-Time Normalization Statistics: Combatting Label Temporal Correlation. ICLR 2024. \n\n# Minor\n- Equation 1: L is used for both spatial dimension (H*W) and its iterator. Though this does not affect understanding, typically one should use different notation. \n- $\\gamma$ is used both in Equation (7) and as the threshold of score in line 242. This might be a little bit confusing. \n- Line 271: detalis -> details\n\nQUESTIONS:\n- What is the dataset used in Figure 3, and what are the four domains? Are they four kinds of corruptions? How generalizable this conclusion is? \n- Figure 4(a): I am quite confused about the conclusion here. In line 161, the author mentioned that \"SBN maintains stable performance across these variables\". Although this is true, in Figure 4(a), we can see that TBN always outperforms SBN, and the variance looks smaller (especially for the dynamic setting). I don't see how this can demonstrate SBN's robustness. Figure 4(b) makes a lot of sense, although with SBN there is no adaptation of the model so it is expected to see a horizontal line of accuracy...\n- I assume that \"Source\" means no adaptation. Why the performance is different in three difference settings? For example, in Table 2, CrossMix IN-C the accuracy is 26.05% but on Random IN-C the accuracy is 27.39%. \n- Table 3 shows that the adaptation is much more effective with the proposed graph-based connecting component strategy, compared to other clustering methods. Could you provide some explanation of why this is the case? Especially given that FIND shares many similarity with some of the clustering algorithms, e.g., HDBSCAN. \n- In Figure 7, why the proposed algorithm have similar performance under batch size = 1 and batch size = 128? If there is only one sample in the batch, how to do clustering and normalization? I am very confused.\n\nLIMITATIONS:\nGenerally yes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}