Non-Parametric Neighborhood Test-Time Generalization: Application to Medical Image Classification

Authors: anonymous  institute
Venue: MICCAI_org_2024_Workshop_MSB
Published: 2024-06-27 16:10:51

Abstract: Reliable and stable performance is crucial for the application of computer-aided medical image systems in clinical settings. However, approaches based on deep learning often fail to generalize well under distribution shifts. In medical imaging, such distribution shifts can, for example, be introduced by changes in scanner types or imaging protocols. To counter this, test-time generalization aims to optimize a model that has been trained on single or multiple source domains to an unseen target domain. Common test-time adaptation methods fine-tune model weights utilizing losses with gradient-based optimization, a time-consuming and computationally demanding procedure. In contrast, our approach adopts a non-parametric method that is entirely feedforward and utilizes information from target samples to extract neighborhood information with dynamic voting. By doing so, we avoid fine-tuning or optimization procedures, enabling our method to be more efficient and achieve stable adaptation. We demonstrate the effectiveness of our approach by benchmarking it against different state-of-the-art methods with three backbones on two publicly available medical imaging datasets, consisting of fetal ultrasound and retinal images, and achieve classification accuracy improvements by up to 3.4\% and 1.1\%, respectively. Moreover, we also demonstrate the utility of our method in practical scenarios, proving efficiency in terms of computational runtime and handling of uncertainty.

Main paper: 
 introduction: 
 Computer-aided medical imaging systems have achieved significant progress in recent years, with a substantial part of this progress made possible by the advancements of deep learning models [18, 7, 19, 5] . However, a major limitation for their adoption in clinical environments is given by their restricted generalization capacity across unseen data distributions [29] . The reason for this is distribution shifts that can, for example, be caused by variations in scanner types or imaging protocols [20] . To address this, test-time adaptation and generalization arose as methods to optimize a trained source model on new incoming target data. Unlike domain adaptation and generalization techniques, test-time generalization can consecutively optimize the model on unlabelled data during the test phase without the requirement to access source data, fostering privacy-preserving adaptation to the target domain. Additionally, it allows the model to be optimized continuously without interrupting the inference process, proving especially beneficial in timesensitive applications where maintaining a flow of real-time decision-making is imperative. Furthermore, the capability of test-time generalization to process data in batches is reflective of real-world scenarios where medical data is also available serially. This aspect enhances its applicability in dynamic clinical workflows.As shown in Figure 1 (a), test-time generalization [24, 21, 15, 14, 17, 8 ] methods focus on fine-tuning of the source model based on source model predictions, surrogate models or task predictions. This optimization often involves computation of gradients with norm-based losses followed by finetuning of batch norm layers [23] , the whole model [14] or a linear classification layer [23] . A more recent approach [10] utilizes parameterized ensembles to optimize the last layers of the source model. Even though it is possible to only fine-tune the batch norm parameters [23] , gradient-based fine-tuning of model weights, in general, is resource and time-intensive. This increases computational costs and leads to slower adaptation processes, making such methods less practical for real-time applications such as dynamic contrast-enhanced imaging, real-time tumor classification, and rapid stroke identification. Gradient-based finetuning [23, 17, 10, 14] often relies on maximum a posteriori estimation to obtain the target model, which can lead to overfitting when adapting to target data that involves multiple distribution shifts. This reduces the model's generalizability, making it less robust to diverse shifts. Moreover, the potential to converge to local minima and a susceptibility to hyperparameter selections limits these methods efficacy.By the nature of their feedforward design, non-parametric methods bypass overfitting and negate the need for loss-based gradient finetuning, therefore offering clear advantages, Fig. 1(b ). T3A [9] computes class representations or prototypes based on the source model's weights and adjusts the classifier utilizing an entropy threshold. However, by relying only on the entropy of samples, information from the target domain is not fully utilized in the method. A more nuanced approach would be given by application of the source model to directly identify target samples with analogous characteristics to the source features. Such a method has the power to increase the utilization of target information and to align closely with the intrinsic data distribution.Building on these insights, our work introduces a novel, non-parametric method coined Test-time Non-parametric Neighbors (TNN). We leverage neighborhood information without the need for finetuning. In summary, our contributions are: Fig. 1 : Data and model interaction scheme. (a) Common test-time generalization techniques utilize norm-based losses for gradient-based finetuning of batch norm layers (β t and γ t ), full model (θ t ), or linear layers (Φ t ) to obtain target predictions y t . These methods feature memory and compute constraints and require precise hyperparameter selection with several rounds of backpropagation. (b) Non-parametric approaches obtain y t via techniques that operate on frozen source model predictions xt in a feed-forward manner. This neglects the need for additional computational resources and simplifies the generalization process.Through comprehensive experiments and ablation studies, we demonstrate the efficacy and potential of TNN in medical imaging contexts, an area where such non-parametric approaches have been underexplored. 
background: 
 Test-time domain generalization aims to generalize a model θ s trained on the source domains S to a unseen target domain T , with S usually consisting of several source domains {D s } S s=1 . T may also consist of several target domains {D t } T t=1 . Here, (x s , y s ) and (x t , y t ) denote the image and corresponding label pairs on the source {D s } S s=1 and target domain {D t } 1 t=1 , respectively. The objective of test-time domain generalization is to maximize the log-likelihood of the source model on the target data p(D t |θ s ), i.e., p(y t |x t , θ s ).Formulation of parametric methods. Due to distribution shifts between source and target domains, the source-trained model θ s is highly likely to fail on unseen target domains D t , causing unreliable predictions with high confidence [1, 27] . To prevent this, the source model must be generalized to the target domain at test time by transforming θ s to θ t . Most common parametric methods employ fine-tuning based on norm-based losses [10, 14, 23] . The log-likelihood of the target data is given by:EQUATIONwith the integration of the distribution p(θ t ) usually approximated by the maximum a posteriori (MAP) estimation. The final generalized MAP model θ * t is obtained by fine-tuning of the parameters with one or multiple rounds of backpropagation using a norm-based unsupervised loss function, like entropy minimization [23] , pseudo labeling [14] or task-specific losses [15, 21] . However, fine-tuning the model parameters through gradient optimization makes parametric methods time-consuming and computationally expensive while also being sensitive to hyperparameter settings.Non-parametric methods. To counter the above limitations, recent nonparametric methods such as [9] obtain class representations as prototypes, utilizing the weights of the source-trained linear classifier, i.e., without the need for MAP approximation or gradient-based optimization. Next, they obtain pseudo labels for the incoming target data based on the distance to those prototypes by applying entropy thresholds. After each incoming batch of target data, significant samples are selected, employing a threshold, and used to update the prototypes via simple adjustments to the classifier. 
method: 
 Source training. Recent studies have shown that utilization of empirical risk minimization (ERM) [22, 6] during source training enables models to generalize well under distribution shifts. Other methods, such as [26, 28, 1, 25] , included additional objectives to be minimized during source training. However, the requirement to interfere in the training procedure limits the applicability of such approaches. Therefore, we aim to develop a method that does not modify the source training procedure, making it applicable to any pretrained model without any additional requirements. Specifically, as in [22, 6] , on multiple source domains {D s } S s=1 , given a source model θ s , such as ResNet-18, and a loss function L, such as cross-entropy, the total risk is minimized via E (xs,ys)∼Ds [L(θ s (x s ), y s )].Test-time generalization via nearest neighbors. Our approach can be summarized by: at test-time, in a non-parametric way, we initially compute the source prototypes following [9] . Next, given a batch of target data, we obtain the nearest neighbors for classification and adjust the classifier weights as described below.Existence of nearest neighbors. We propose that for a sufficiently trained model able to separate classes reasonably well in the source domain, cases that are similar in the higher dimensional image domain will lie close to each other in the source learned lower dimensional embedding space. This is ensured by the Johnson-Lindenstrauss (JL) [11] lemma as stated below:Given a set of points {x i : i = 1, ..., M } in R m , the JL lemma states that if n ≥ cϵ 2 log M , with 0 < ϵ < 1, then there exists a linear map A : R m → R n such that for all i ̸ = j:1 -ϵ ≤ ∥A(x i ) -A(x j )∥ ∥x i -x j ∥ ≤ 1 + ϵ.At test-time, utilizing the above lemma, our approach first computes the source model prototypes for each class in lower-dimensional space. To do so, as in [9] , we initialize the class-specific prototypes by aggregating the weights of the sourcetrained linear classifier layer. When receiving new test-time data x t , we project it into lower-dimensional space using the source-trained model that preserves distances in the embedding space, as ensured by the JL lemma. In this embedding space, we assign x t the label of its nearest class prototypes based on a distance measure (see below). Finally, we update the class-specific prototypes with the new sample to reflect new target characteristics. This allows us to classify new samples without the need for extensive computation or any optimization schemes to find the nearest points to the prototypes.Selecting neighbors with dynamic voting. Since not all of the neighbors provide accurate information about the target data, i.e., some of them are noisy [4] , we calculate the distance between the initial prototypes and new classifier weights obtained from incoming neighbor samples at test time for the selection of valid neighbors. We utilized the cosine distance here, while in principle every other distance metric can be use. Next, we use dynamic voting to obtain the most useful neighbors, i.e. we aggregate each neighbor's prediction and calculate the mean of the obtained new weights to determine the final weights of the classifier.When a new batch of samples arrives, the pseudo labels are predicted based on these classifier weights. This process is repeated iteratively for each new incoming batch of data. 
datasets and implementation details: 
 Datasets. We validate the effectiveness of TNN on two publicly available datasets.Messidor [3] depicts a retinal image database collected from three independent medical centers, containing 1200 images of diabetic retinopathy, each labeled with a severity score ranging from 0 to 3. Therefore, it consists of 3 domains, 4 classes, and 1200 images in total. Fetal-8 [2] is a maternal-fetal ultrasound dataset that consists of eight classes representing different anatomical planes collected from imaging scanners of two different vendors. Hence, it consists of 2 domains, 8 classes and 12,058 images in total. We consider the problem as a test-time generalization setting, not domain adaptation. Therefore, we train our model on multiple source domains of the Messidor dataset instead of only one. Furthermore, we follow the leave-one out evaluation standard from [9, 13] and obtain the best source model following the training-domain validation split of [9] . For all the experiments, performance is reported on the target dataset with test-time adaptation [1, 9, 23] by utilizing accuracy as a metric.Implementational details. On both datasets, Messidor and Fetal-8, we evaluate the performance of our approach based on three backbones, DenseNet-121, ResNet-18 and ResNet-50. Further in-depth ablation experiments are then performed with the ResNet-18 model. As in common test-time methods [9, 6] , the backbones are pretrained on Imagenet. Baseline and state-of-the-art methods are also implemented for the two datasets and for all of the three backbones, utilizing the Domainbed library [6] with all hyperparameters set to default. The trainingvalidation split strategy of [9, 6] is used for selection of the best source model. At test-time, we perform just forward passes to perform the classification on the online target data. We evaluate the performance of methods following standard test-time generalization evaluation [9] . We utilize a small batch size of 32 for test-time generalization, reflecting real-world practical scenarios.We evaluate the performance of our model in reference to different state-of-the-art (SOTA) approaches and a source training strategy employing ERM minimization without adaption to the target domain as the baseline.State-of-the-art comparisons. We compare our approach to existing parametric and non-parametric state-of-the-art methods by re-implementing them on the two datasets for all three backbones. Parametric methods, as shown in Figure 1 , refer to techniques that finetune weights of the source-trained model, utilizing gradient optimization. Non-parametric methods refer to techniques that perform feedforward computation at test-time without any kind of finetuning, optimization, or usage of any external memory bank or an additional model.Table 1 shows the performance comparisons. TNN achieves the best results on both datasets. Parametric methods achieve comparatively lower performance than the ERM baseline in many cases. One reason for this can be given by the fact that the medical imaging datasets at hand only contain a fraction of the number of samples the parametric methods were designed for. Furthermore, differences between images of distinct classes in the medical domain are way more subtle than in the computer vision domain, with different classes in the retinopathy images of Messidor even depicting a severity score that changes only gradually. Therefore, it is likely that non-parametric methods achieve better performance due to the fact that they do not fine-tune the complete source model weights but rather act upon the source-trained embedding space that should be able to separate classes reasonably well. T3A [9] utilizes the entropy of the samples as a threshold to classify new cases. In contrast, we use detailed neighborhood information for classification. For the Fetal-8 dataset, utilizing the ResNet-18 backbone, the performance of all the generalization methods decreases with reference to the ERM baseline. Reason for this is most likely overfitting due to the small size of the model, but also the small dataset size at test-time. For all the remaining settings, TNN performs better than the other approaches.Table 1 : State-of-the-art comparisons using DenseNet-121, ResNet-18, and ResNet-50 as backbones. We re-implement the methods on the datasets and report the mean accuracy across all the domains. The best results are in bold. Our results are averaged over five runs. Our method performs the best consistently with the highest performance improvement. Addressing uncertain scenarios. Ensuring alignment between model output probabilities and the actual likelihood of events is crucial in uncertain scenarios [12] . To quantify this alignment, Table 2 presents the expected calibration error [16] (lower values indicate better calibration) for our approach compared to the Tent model [23] , utilizing a ResNet-18 backbone. Consistently, TNN demonstrates considerably better calibration scores across all domains on the Messidor dataset. Therefore, leveraging its entirely feedforward nature for classifier adjustments, TNN achieves a better-calibrated model.Computational cost. In Table 3 , we compare the number of parameters required to be trained at test-time and the number of floating point operations per second (FLOPS) consumed by the GPU for Tent [23] , T3A [9] and TNN. All of the methods, ours included, feature the same memory requirements of 1.4GB for the ResNet-18 model. However, as Tent optimizes the batch normalization layers of the target model at test time and, therefore, it requires more parameters to be trained than our approach. TNN and T3A are both non-parametric. Thus, they only perform a very limited amount of computational operations but do not need any additional computations to calculate the gradients on the GPU. Therefore, the measurement of TFlops is negligible in this case, considering the vast amount of computations required for weight optimization of parametric approaches. This is especially useful in limited resource settings.  
conclusion: 
 We propose the usage of a non-parametric-based neighborhood classification method for medical imaging tasks that involve distribution shifts as a novel test-time generalization method. By utilizing target information and neighbors in the embedding space we sequentially adjust the weights of the classifier, providing an efficient yet powerful generalization technique. As with common methods, our method requires a shared label space between domains for adaptation. We aim to address this limitation in future work by enabling the generation of classifiers for new categories at test-time, alongside existing classifier adjustments. Furthermore, we also provide additional experiments to demonstrate the utility of the method in uncertain scenarios and settings that require limited computational resources. 
 
 Appendix:  


Decision: The paper proposes a novel and potentially useful non-parametric test-time adaptation (TTA) method, TNN, for medical image analysis. The method achieves good performance on multiple datasets and outperforms current state-of-the-art approaches in terms of accuracy, runtime, and calibration error. The reviewers find the approach efficient, scalable, and easy to implement. They also acknowledge thorough evaluation on multiple datasets and architectures. Consider adding the ablation studies in future work to further strengthen the work. Also, evaluating the method on a wider variety of datasets, especially large-scale ones, would be beneficial. The authors have promised to improve the clarity of the TNN method description with pseudocode and visualizations and to add the standard deviation in the results for all models. Abstract and introduction needs to be strengthened with a clearer explanation in the final version. As discussed in the rebuttal, sufficient technical details must also be added to the final paper.

Reviews: review: main_review: main_review: To change my judgement, I would need more convincing on the novelty of the work. The paper is also missing ablation studies to explain why decisions were made when designing this approach, without this it is hard to understand why this method would be beneficial.
strength_weakness: strength_weakness: 1.	The novelty of this proposed method is questionable. Previous works, such as Jang, M. ‘Test-Time Adaptation Via Self-Training With Nearest Neighbor Information’ (2023), use a similar Prototype-based classification method. This includes using cosine similarity in the embedding space to class prototypes for classification and filtering out unreliable examples when calculating nearest neighbor-based pseudo labels. The paper mentioned also demonstrates the benefit of nearest-neighbor information in an optimization-free case in their ablation study (see TASH-N). The paper requires more explanation as to why this method is novel, other than being applied to new datasets.
2.	The work would benefit from `ablation studies to help explain the choices made when designing the method and to support the claim it is better than current state-of-the art methods. For example, the paper says, ‘we utilized the cosine distance here, while in principle every other distance metric can be use’. An ablation study would be useful to explain why using cosine similarity is better than using a different distance metrics (e.g. Euclidean distance, Mahalanobis distance etc) for this method.
3.	The methods section can be difficult to follow and understand. For example, the paper doesn’t really define what it means when it says it uses ‘dynamic voting’ and it doesn’t define some of the symbols used in equations (see section on JL Lemma).  A better approach might be to include more equations, as mathematical explanations can sometimes clarify the procedures more effectively.
review_summary: review_summary: Recognising that test-time generalization techniques which involve fine-tuning gradients or batch-norm statistics are computationally expensive (and hence less useful for real-time medical applications), the paper suggests a non-parametric method for test-time generalization that doesn’t require fine-tuning or optimization. The paper suggests a prototype-based method, which calculates prototype representations to represent each class in the embedding space, and then predict the label of an input as the class of the nearest prototype. The method continually updates the class prototypes with new samples to account for distribution shifts. The method is tested on two medical image datasets, retinal images and fetal ultrasound images, and show better test-time generalization accuracy than other parametric and non-parametric methods.
main_review: main_review: Even though the overall numeric improvement is not that significant compared to other available techniques, the proposed method is interesting and versatile, as it can be applied to any trained classifier.
strength_weakness: strength_weakness: The method's weaknesses include its dependence on pre-trained models and the assumption that the source model separates classes well. Scalability issues may arise with high-dimensional data or large datasets, and handling noisy data remains a challenge despite dynamic voting. The initial prototypes might be suboptimal if the source and target domains differ significantly, and the method is sensitive to the choice of distance metrics. The lack of end-to-end optimization during inference can limit adaptability, while iterative updates may increase computational overhead. Additionally, the dynamic voting mechanism adds complexity, and the method's success is heavily influenced by the quality of the source domain data.
Moreover, the imrpovement obtained compared to the baseline methods is not always significantly higher.
review_summary: review_summary: The authors propose a novel non-parametric test-time adaptation technique to improve generalisability of deep learning models affected by domain shifts. They apply their approach on three different backbones and validate it against several time test adaptation techniques from the state of the art on two publicly available datasets.
main_review: main_review: Multiple weaknesses were identified, such as an insufficient presentation of the methodology, arbitrary choices of competitors, and a lack of significance tests. By addressing these points, the submission could be substantially improved, and the paper could possibly be accepted, but not in its current state.
strength_weakness: strength_weakness: 1. The paper lacks sufficient detail in the description of the TNN method. Key aspects such as the algorithmic steps, mathematical formulations, and implementation specifics are not comprehensively covered. The dynamic voting mechanism for selecting neighbors needs more clarity. The criteria for neighbor selection and the weight adjustment process should be elaborated. Overall, this makes it difficult for readers to fully understand. This weakness, for example, could be solved by illustrating the method in sufficient detail, which could replace Fig. 1, as its information content is very limited.
2. Can the authors explain the direct relation of their method to the Johnson-Lindenstrauss lemma? What kind of assumptions are needed for the lemma to hold? Are there any constraints that need to be incorporated during training in order to ensure a stable local neighborhood? 
3. The selection of competitors for evaluation, especially for Tables 2 and 3, appears arbitrary. For example, the competitor in Table 2 (Tent) seems to achieve the lowest performance for the respective backbone. It is crucial to justify why specific methods were chosen and ensure a comprehensive comparison with the most competitive approaches when only considering single techniques, as in Table 2.
4. While using Messidor and Fetal-8 datasets is appropriate, the paper should include a wider variety of datasets to better demonstrate the proposed method's generalizability. For example, the paper lacks an evaluation of large-scale datasets and a discussion on scalability.
5. The results presented lack statistical analysis to support the claims. Including standard deviation, confidence intervals, or p-values would strengthen the evaluation.
review_summary: review_summary: The paper presents an approach to test-time generalization using a non-parametric method named Test-time Non-parametric Neighbors (TNN) as opposed to fine-tuning, thus offering computational efficiency.
main_review: main_review: Interesting paper that presents a simple but effective method for domain adaptation. Major claims are based on a thorough evaluation. Flaws can be considered minor.
strength_weakness: strength_weakness: - Table 3 is redundant and confuses more than it adds to the paper. I think the point was already made clear in the very beginning of the manuscript that the proposed method doesn't require any training. Further, the comparison is slightly unfair, as only _Model_ TFLOPS are compared. Still, TNN will use some FLOPS to work (likely not in the range of TFLOPS), but these are not reported.
- How is the calibration score estimated? Why is it not reported for T3A? Why only for ResNet-18 and not all baseline models? It looks like the evaluation of ECE is not as complete as it could be.
review_summary: review_summary: The authors propose TNN (Test-time Non-parametric Neighbors), a nearest-neighbor based parameter-free technique for the adaptation of neural network classifiers that doesn't require adaptation through training. During test time, source prototypes are generated for each class. Afterwards, the class is determined as argmax of the cosine distance between samples in embedding space. This method beats SOTA in terms of runtime and accuracy.

Comments: title: Rebuttal by Authors
Rebuttal by Authors
Rebuttal by Authors
Rebuttal by Authors
comment: We thank the reviewer for their constructive suggestions, which we believe have contributed to an improved manuscript.

**Novelty and comparison**

As stated in our introduction and also mentioned by the reviewer, TAST [10] uses neighborhood information to perform test-time adaptation. However, they utilize backpropagation for adaptation, a procedure that we are aiming to avoid due to the reasons discussed in Section 3 of our paper. Furthermore, multiple MLP ensemble modules are employed by TAST, which is not required by our model. Additionally, no theoretical foundation for their method is provided. TAST-N (from the ablation study of [10]) uses feedforward adaptation to obtain classifier weights via the usage of cosine similarity. However, our method uses subsets of neighbors and employs dynamic voting and obtain the final weights of the classifier. This ensures more robustness and a better-aligned target-specific model compared to TAST-N, which computes the weights directly. We will include this discussion in the paper.

  

**Ablation studies**

Thank you for suggesting new experiments to ablate the distance functions. However, we cannot add additional results due to MICCAI's review/rebuttal guidelines, and due to space limitations of the workshop format, we concentrated on the main comparisons to baseline models. We will certainly include insights from ablations in our future journal work.

  

**Additional details of our method**

For dynamic voting, we calculate the mean of obtained classifier weights through different sets of top neighbors that are chosen by their closeness, measured by the distance function, to predict the classification labels. We will include this discussion and pseudocode in the paper.

  

**Additional literature review**

Thank you for your feedback; while no specific papers were cited, we will expand our literature review with additional recent studies.
We thank the reviewer for their constructive suggestions, which we believe have contributed to an improved manuscript.
  

**Source model training and class separability**

The dependence on the utilization of a pre-trained model for source training is a common requirement of test time adaptation methods, including T3A, Tent, and SHOT. At test-time, our non-parametric approach explores neighborhood information and directly obtains the target model $\theta_{t}$ through classifier adjustments. This results in a target-specific model that utilizes target information more effectively than other methods and facilitates class separability.

  

**Prototypes for larger domain shifts**

For the computation of prototypes, we rely on source model predictions, such as T3A, and employ the distance function to find neighbors based on the JL Lemma. The JL Lemma asserts that points will remain close to each other even after random projections. In our case, our projections are better than random projections, which allows the distance function to identify neighbors. Significant domain shifts remain an open problem for all test-time adaptation methods.

  

**Why end-to-end optimization is not useful**

We avoid end-to-end optimization to obtain the target model, aside from increased computational usage, for two main reasons:

i) MAP Estimation: It is not reliable for adaptation. (As detailed in Methodology)

ii) Iterative Re-Training with gradient-based optimization: Since not all the predictions are accurate, retraining on these noisy predictions will lead to a misspecified model [1].

  

**Dynamic voting and performance**

Dynamic voting enables us to identify the most useful neighbors, which is crucial for the effective updating of prototypes across the streaming batches of data at test time. Without this, updating prototypes would be less beneficial. Across three different backbones, our method consistently achieves the best (in most scenarios) or second-best performance.

We would like to thank the reviewer for suggesting to depict an overview figure. As also suggested by Reviewer rJeC, we will revise and improve Figure 1 accordingly.

  

**References:**

[1] Ba, Cuimin. "Robust Misspecified Models and Paradigm Shifts." arXiv 2021.
We would like to thank the reviewer for their constructive suggestions, which we believe have contributed to an improved manuscript.

  

**1. Additional details of TNN**

We obtain neighbors utilizing the cosine distance function and further select appropriate neighbors via dynamic voting. For dynamic voting, we calculate the mean of obtained classifier weights through different permutation sets of top neighbors that are chosen by their closeness, measured by the distance function to predict the classification labels. We will add this clarification to the paper. To enhance reproducibility, we have open-sourced the code. We agree with the reviewer that replacing Figure 1 with an illustration will add more insights and will be added to the paper. Moreover, a pseudo-code reflecting the algorithmic steps of our model will be added to the paper.
    

  

**2. Further clarification about JL Lemma**

The JL lemma provides a theoretical foundation for our work. It does not impose any restrictions on our method. Specifically, the lemma ensures that if points lie close to each other in higher dimensional space, then, even when projected randomly, these points will also lie close to each other in a lower dimensional space [I]. In our case, the projections are better than random due to models being trained on the source domains.
    

Pandey et al., 2020 [II] propose an alternate lemma. In their work, they utilize gradient descent-based optimization to sample neighbors from the VAE latent space. Different from them and other methods, we obtain neighbors without any optimization, thus making the adaptation process at test-time entirely feedforward. We will include this discussion in the paper.

  
  

**3. Baselines comparison**

All baseline models chosen for comparison depict well-established SOTA approaches in terms of test time adaptation (TTA). Specifically, we have chosen baselines similar to T3A, as our model follows a similar direction, i.e., being non-parametric.
    

Tent also depicts a well-established TTA method. As detailed in Section 3, Tent obtains the target model by estimation of the MAP solution. This depicts an error-prone approach, as laid out in our paper. Moreover, Tent's sole reliance on entropy minimization typically necessitates a larger batch size and more samples in the target dataset, which is not feasible in medical imaging scenarios, leading to suboptimal performance. Nevertheless, we have chosen to include Tent as a baseline model to demonstrate its insufficiency when applied to medical imaging.

  
  

**4. Further clarification about extending to more datasets**

We agree with the reviewer that the inclusion of more datasets will provide even more insights into our approach’s generalizability. However, we cannot add additional datasets due to the review/rebuttal guidelines of MICCAI and plan to include more datasets in future journal work.
    

  
  

**5. Further clarification of our results**

We thank the reviewer for pointing out the missing standard deviation values. As mentioned in the paper, we report the mean values by utilization of 5 random seeds. Hence, as suggested, we will also include the standard deviations for the baselines in the main table. However, this did not alter our findings and we will add this to the paper.
    

  

**References:**

[I] Ghojogh, Benyamin, Mark Crowley, Fakhri Karray, and Ali Ghodsi. Elements of dimensionality reduction and manifold learning. Springer, 2023.  

[II] Pandey, Prashant, et al. "Unsupervised domain adaptation for semantic segmentation of NIR images through generative latent search.", ECCV 2020
Thank you for recognizing the strengths of our proposed method and your positive feedback.

  

**Weaknesses**  
  

**Further clarification of Table 3**

At test time, both parametric and non-parametric methods need resources for computation to compute gradients (parametric) or to adjust weights (non-parametric). We therefore believe that Table 3 is essential since it quantifies computational costs such as storage space (1.4 GB for all), number of trainable parameters, and TFlops. Our method performs simple matrix computations for which we will include the CPU cost consumed in the paper. Given MICCAI's focus this year on low-resource settings, we believe that specifying the exact requirements for our model to function effectively is necessary.

  

**Calibration error**

We report the ECE error by using the library developed in [I] and calculate the ECE error between the predicted labels and true labels, which is useful for uncertain environments. We mainly compared the ECE score for parametric methods and our method to quantify the calibration of models in uncertain scenarios, as demonstrated in [II]. However, we will also include the ECE error for T3A to the paper. Additionally, we will include ECE errors for additional backbones and methods in future work.

  

**Feedback**

  

**Including quantitative and additional benefits of our method in abstract**

We will strengthen the abstract by including the results of classification accuracy, runtime, and ECE error. Thank you for pointing this out.

  

**Neighborhood information**

We use neighborhood information of the source prototypes with the target data at test-time. We will add this in the introduction.

  

**Figure 1**

We will modify the Figure to include a comprehensive overview of the neighborhood exploration with the distance functions as suggested; thank you for this suggestion.

  

**Further explanation of prototypes**

We will include further details on how the prototypes are calculated at test time to the paper, within the space limitations.

  

References:

[I] Webpage: [https://torchmetrics.readthedocs.io/en/v0.8.0/classification/calibration_error.html](https://torchmetrics.readthedocs.io/en/v0.8.0/classification/calibration_error.html)

[II] Kumar, Ananya, Percy S. Liang, and Tengyu Ma. "Verified uncertainty calibration.", NeurIPS 2019