Title: Neighborhood Self-Dissimilarity Attention for Medical Image Segmentation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Digital Health Equity

Medical Image Segmentation

Prior Knowledge for AI

Efficient and Scalable Vision

Attention Mechanism
Abstract: Medical image segmentation based on neural networks is pivotal in promoting digital health equity. The attention mechanism increasingly serves as a key component in modern neural networks, as it enables the network to focus on regions of interest, thus improving the segmentation accuracy in medical images.  However, current attention mechanisms confront an accuracy-complexity trade-off paradox: accuracy gains demand higher computational costs, while reducing complexity sacrifices model accuracy. Such a contradiction inherently restricts the real-world deployment of attention mechanisms in resource-limited settings, thus exacerbating healthcare disparities. To overcome this dilemma, we propose a parameter-free Neighborhood Self-Dissimilarity Attention (NSDA), inspired by radiologists' diagnostic patterns of prioritizing regions exhibiting substantial differences during clinical image interpretation.  Unlike pairwise-similarity-based self-attention mechanisms, NSDA constructs a size-adaptive local dissimilarity measure that quantifies element-neighborhood differences. By assigning higher attention weights to regions with larger feature differences, NSDA directs the neural network to focus on high-discrepancy regions, thus improving segmentation accuracy without adding trainable parameters directly related to computational complexity.  The experimental results demonstrate the effectiveness and generalization of our method. This study presents a parameter-free attention paradigm, designed with clinical prior knowledge, to improve neural network performance for medical image analysis and contribute to digital health equity in low-resource settings. The code is available at [https://github.com/ChenJunren-Lab/Neighborhood-Self-Dissimilarity-Attention](https://github.com/ChenJunren-Lab/Neighborhood-Self-Dissimilarity-Attention).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel, parameter-free attention mechanism called Neighborhood Self-Dissimilarity Attention (NSDA) for medical image segmentation. The core idea is inspired by radiologists' diagnostic process, where they focus on regions that exhibit high contrast or difference from their immediate surroundings. Instead of computing pairwise similarity like in conventional self-attention, NSDA calculates an attention score for each feature element based on its dissimilarity to the aggregated representation of its local neighborhood. The dissimilarity is quantified using a complement of a Gaussian kernel. The method also introduces a Dynamic Neighborhood Scaling (DyNS) strategy to adapt the size of the local neighborhood across different layers of the network, preventing feature homogenization. The authors claim this approach improves segmentation accuracy without adding computational complexity (in terms of parameters and MACs), thus addressing the performance-complexity trade-off and promoting health equity in resource-limited settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. The core idea of a parameter-free, dissimilarity-based attention mechanism is highly original and significant. It is well-motivated by clinical practice and directly addresses the critical performance-complexity trade-off, making it highly practical for resource-constrained applications. \n\n2. The method is tested on multiple backbones and datasets against a wide array of baselines. Furthermore, strong ablation studies clearly demonstrate the contribution of each proposed component (DyNS and the dissimilarity measure). \n\n3. The paper is very well-written, clearly structured, and easy to follow, which significantly enhances the accessibility of its contributions. \n\nWeaknesses: \n\n1. The performance gains reported in Table 1 are impressive. Could the authors clarify if these results are from a single run or averaged over multiple runs? To strengthen the paper's claims, I would strongly recommend that the authors run the experiments for the main backbones (e.g., U-Net and TinyU-Net) with at least 3-5 different random seeds and report the mean and standard deviation for the Dice scores. This would provide crucial evidence of the statistical significance of the improvements over baselines. An update with this information would likely increase my overall score. \n\n2. The DyNS strategy uses a fixed scale factor S=8. The paper would benefit from a discussion on how this value was chosen. Could the authors provide a brief sensitivity analysis showing how performance (e.g., mean DSC on Synapse) varies with different values of S? This would help to better understand the robustness of this heuristic. \n\n3. The generalization experiments in Table 2 show that NSDA performs well on the natural image dataset VOCSeg. This suggests that \"dissimilarity\" might be a more general principle for attention, extending beyond its motivation as a \"clinical prior\". Could the authors elaborate on why they believe this principle generalizes so effectively outside of medical imaging, where the \"lesion vs. background\" intuition might be less directly applicable? \n\n4. The experiments integrate the attention module only at the terminal layer of the feature extractor. In U-Net like architectures, attention is often applied at multiple stages of the encoder and decoder. An investigation into how the performance of NSDA changes with different placement strategies would have provided deeper insights into its mechanism.\n\nQUESTIONS:\n1. The performance gains reported in Table 1 are impressive. Could the authors clarify if these results are from a single run or averaged over multiple runs? To strengthen the paper's claims, I would strongly recommend that the authors run the experiments for the main backbones (e.g., U-Net and TinyU-Net) with at least 3-5 different random seeds and report the mean and standard deviation for the Dice scores. This would provide crucial evidence of the statistical significance of the improvements over baselines. An update with this information would likely increase my overall score. \n\n2. The DyNS strategy uses a fixed scale factor S=8. The paper would benefit from a discussion on how this value was chosen. Could the authors provide a brief sensitivity analysis showing how performance (e.g., mean DSC on Synapse) varies with different values of S? This would help to better understand the robustness of this heuristic. \n\n3. The generalization experiments in Table 2 show that NSDA performs well on the natural image dataset VOCSeg. This suggests that \"dissimilarity\" might be a more general principle for attention, extending beyond its motivation as a \"clinical prior\". Could the authors elaborate on why they believe this principle generalizes so effectively outside of medical imaging, where the \"lesion vs. background\" intuition might be less directly applicable?\n\nLIMITATIONS:\nYes. The authors have adequately addressed the limitations in a dedicated paragraph. They correctly point out that while their attention module is lightweight, the overall deployment feasibility still depends on the size of the backbone network architecture, and they suggest future work on lightweight networks. This is a fair and honest assessment.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose a novel Neighborhood Dissimilarity Attention mechanism to aid medical image segmentation in resource-limited settings. Based on the way radiologists inspect neighboring pixels or elements in an image, Authors propose a weighting of spatial features based on the level of dissimilarity between each pixel and those of its neighborhood. Dynamic Neighborhood Scaling is introduced to limit the neighborhood at subsequent model depths, to avoid feature homogenization. Attention Fusion is also used, whereby features are scaled by the attention weights and an added to the original features via a residual - this avoid additional complexity usually added with an MLP. Authors demonstrate their method in 3 distinct imaging domains - CT, MRI, and Ultrasound - as well as natural images. Results demonstrate that NDSA achieves most significant performance gains under test, and is able to perform well across architectures and domains. The paper shows that dissimilarity-based attention yields better performance than similarity based attention.\n\nSTRENGTHS AND WEAKNESSES:\nThe authors set the context of their research well, and lay out a measured approach to understand how their proposed NDSA can enhance the performance of attention mechanisms in the low-resource setting.\n\nIn addition, Authors present a solid set of RQs, and demonstrate their approach on a diverse domains. Authors provide additional investigation by extending their analysis to the natural domain to assess generalization of their method. It is appreciated that Authors also compare against available resource-limited variants of attention mechanisms, given these are their main alternatives. Authors use appropriate metrics for analysis.\n\nAuthors have focused on the UNet architecture and its variants in this paper due to the resource-limited nature of their focus - it is known that alternative approaches such as transformer-based architectures can yield impressive results in image segmentation tasks. It would be beneficial for authors to compare their approach to other, more traditional attention mechanisms too.\n\nQUESTIONS:\nIn eq1, `S` is empirically set to 8 to help ensure the neighborhood of attention remains proportional to feature map scales across network depths. Could this be done more algorithmically using some aggregation of convolutional kernel size and stride etc.?\n\nLIMITATIONS:\nAuthors have not listed any limitations of their method in their Limitations section - instead, they have only noted that resource-limited setting need lower complexity approaches, which is what they address in their work. Authors should use this section to note actual limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a clinically inspired Neighborhood Self-Dissimilarity Attention (NSDA) mechanism, drawing on radiological diagnostic principles to address the long-standing performance–complexity trade-off in attention design through a parameter-free formulation. To further mitigate feature homogenization, the authors introduce a Dynamic Neighborhood Scaling (DyNS) strategy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n(1) This paper presents a parameter-free Neighborhood Self-Dissimilarity Attention (NSDA) framework that enables neural networks to focus on element-neighborhood differences, thereby boosting segmentation accuracy.\n(2) This paper introduces a Dynamic Neighborhood Scaling (DyNS) strategy in NSDA, which adaptively regulates NSDA’s neighborhood size across network hierarchies to prevent feature homogenization.\n(3) Experimental results show the effectiveness of the proposed model.\n\nWeaknesses:\n(1) While medical image segmentation typically relies on shape priors, multi-scale context, and boundary cues, NSDA instead emphasizes element-to-neighborhood dissimilarity. The authors do not sufficiently explain why such a “self-dissimilarity” mechanism offers better discriminative power than direct features, nor do they clarify its rationality and advantages in structured scenarios such as connected anatomical regions.\n(2) Although the authors claim to present the first parameter-free attention mechanism based on element–neighborhood dissimilarity, similar ideas have appeared in prior works such as SimAM and ULSAM, which leverage local statistics (e.g., mean, variance) or subspace reconstruction errors to generate attention weights. The paper does not adequately distinguish NSDA from these approaches, either conceptually or theoretically.\n(3) The dissimilarity computation is based on a fixed Gaussian kernel, rather than a learnable or adaptive metric. The authors do not provide justification for the selection or regulation of the variance term, which may lead to either over-smoothing or noise amplification. It also remains unclear why this fixed-kernel approach is preferable to alternative similarity or distance measures for segmentation.\n(4) The multi-scale attention fusion in NSDA lacks detail on how weights across scales are determined or balanced. Moreover, the absence of ablation experiments regarding fusion strategies weakens the empirical support for this design component.\n\nQUESTIONS:\n(1) In Fig. 3, the effectiveness of DyNS is demonstrated only on the Synapse dataset. However, its performance under small-object or densely structured targets remains untested, limiting conclusions about the generalizability and robustness of the DyNS design.\n\n(2) The neighborhood size in Equation (1) is heuristically determined using a fixed scaling factor (S = 8) and a compensatory constant (c), but the paper lacks sensitivity analysis to evaluate how this setting behaves across diverse network hierarchies and segmentation scenarios.\n\n(3) Although NSDA is described as “parameter-free,” the implementation involves per-pixel computation of local mean, variance, and exponential transformations, which introduce non-trivial computational cost and implicit hyperparameters. The paper does not provide an analysis of the resulting resource consumption or inference latency.\n\n(4) While Fig. 3 demonstrates the advantage of replacing static neighborhood sizes with DyNS, the paper does not report the performance of NSDA when DyNS is entirely removed, leaving the necessity of this component only partially evaluated.\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a different, parameter-free, attention mechanism, NSDA, that is more efficient than pairwise-similarity based mechanisms and shows promisiong results for segmentation tasks. As part of NSDA, DyNS is proposed to constrain NSDA to localized information throughout feature hierarchies present in deeper networks. The authors emphasize that both, NSDA and DyNS, are inspired by patterns present in how actual radiologists' workflows. The paper has strong experiments to demonstrate the efficacy of the method and support parameter choices. \nHowever, the major shortcoming of the paper is that it is poorly motivated in that a more efficient attention mechanism for medical image segmentation tasks doesn't actually address any of the health equity concerns the authors claim to care about.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The experiments are thorough and inspire confidence in the method's efficacy. \n\nWeaknesses:\n- The clinical motivation for this paper the authors present seems to be poorly suited for the task and model presented. What would the segmentation of lesions be needed for in a resource-scarce location? Who is taking the images and who is making the decision to biopsy after the segmentation is completed? The authors seem to think a focus on regions with scarce resources and health equity will get their paper more traction, but have not done any of the legwork necessary to ensure their method is applicable, or even interesting, for these situations. This weak motivation distracts from the main contribution of the paper and frankly, makes little sense. The brain MRI images in Figure 1 don't seem to represent a modality which would be available in a resource-scarce location. \n- The authors state that (paraphrased): regions with small ROIs are ignored by existing medical image segmentation methods. This is demonstrably untrue. See https://pmc.ncbi.nlm.nih.gov/articles/PMC9690845/ and https://link.springer.com/article/10.1007/s11432-021-3340-y and https://link.springer.com/article/10.1007/s12652-021-03358-8 for a few examples in breast imaging. \n- The authors Neighborhood Inspection and Difference Prioritization modules seem to be re-deriving traditional radiomic features. Localized variance and mean has been used to describe medical imaging texture for decades. Equation 3 seems to be conceptually similar to a Laplace of a Gaussian feature. \n- BUSI is a low-quality dataset in which about 30% of images contain measurement calipers, providing the model an ROI from which to segment. The authors would be better served testing their method on a cleaner dataset. See Figure 2 for an example lesion with measurement calipers which much be placed by an expert reader such as a sonographer. Which would be unavailable in the authors application area. \n- Again, CT and MRI are not going to be available in a resource-scarce locale. The authors contradict their motivation once again.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}