Keypoint-Augmented Self-Supervised Learning for Medical Image Segmentation with Limited Annotation
Authors: Zhangsihao Yang
Mengwei Ren
Kaize Ding
Guido Gerig
Yalin Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Keypoints
Medical image
self-supervised learning
transformer
segmentation
TL;DR: A pre-trianing method that utilize keypoints to help segmentatation on biomedical images.

Abstract: Pretraining CNN models (i.e., UNet) through self-supervision has become a powerful approach to facilitate medical image segmentation under low annotation regimes. Recent contrastive learning methods encourage similar global representations when the same image undergoes different transformations, or enforce invariance across different image/patch features that are intrinsically correlated. However, CNN-extracted global and local features are limited in capturing long-range spatial dependencies that are essential in biological anatomy. To this end, we present a keypoint-augmented fusion layer that extracts representations preserving both short- and long-range self-attention. In particular, we augment the CNN feature map at multiple scales by incorporating an additional input that learns long-range spatial self-attention among localized keypoint features. Further, we introduce both global and local self-supervised pretraining for the framework. At the global scale, we obtain global representations from both the bottleneck of the UNet, and by aggregating multiscale keypoint features. These global features are subsequently regularized through image-level contrastive objectives. At the local scale, we define a distance-based criterion to first establish correspondences among keypoints and encourage similarity between their features. Through extensive experiments on both MRI and CT segmentation tasks, we demonstrate the architectural advantages of our proposed method in comparison to both CNN and Transformer-based UNets, when all architectures are trained with randomly initialized weights. With our proposed pretraining strategy, our method further outperforms existing SSL methods by producing more robust self-attention and achieving state-of-the-art segmentation results. The code is available at https://github.com/zshyang/kaf.git.

Keypoint-Augmented Self-Supervised Learning for Medical Image Segmentation with Limited Annotation
Zhangsihao Yang" Arizona State University 2shyang116@@gai11 com
Mengwei Ren" New York University mengwei. ren@nyu. edu
Kaize Ding Northwestern University kaize. wwngeeorttwwetteeeeeo edu
Guido Gerig New York University
Yalin Wang Arizona State University y ylwangBasu.c edu
gerigSnyu. edu edu
Abstract
Pretraining CNN models (i.e., UNet) through self-supervision has become a power- ful approach to facilitate medical image segmentation under low annotation regimes. Recent contrastive learning methods encourage similar global representations when the same image undergoes different transformations, or enforce invariance across different image/patch features that intrinsically correlated. However, CNN- extracted global and local features are limited in capturing long-range spatial dependencies that are essential in biological anatomy. To this end, we present a keypoint-aud keypoint-augmentte augmenttdd fusion layer that extracts representations preserving both short- and Long-range self-attention. In particular, we augment the CNN feature map at multiple scales by incorporating an additional input that learns long-range spatial self-attention among localized keypoint features. Further, we introduce both global and local self- supervueee pretraining for the framework. At the global scale, we obtain global representations from both the bottleneck of the UNet, and by aggregating multiscale keypoint features. These global features are subsequently regularized through image-level contrastive objectives. At local scale, we define a distance-basse based criterion to first establish correspondences among keypoints and encourage similarity between their features. Through extensive experiments on both MRI and CT segmentation tasks, we demonstrate the architectural advantages our proposed method in comparison to both CNN and Transformer-baat when all architectures posed pretraining strategy, our method further outperforms existing SSL methods by producing more robust self-attention and achieving state-of-the-- art segmentation results. The code is available at https: : //g: github.... com/ zsshyang/// kaf, git
UNets,
trained with randomly initialized weights. With our pro-
1 Introduction
Large-scale and diverse source of training data has significantly empowered the generalization ability 34). However, in biomedical image analysis, manual delineation of images/volumes requires extensive domain knowledge and be extremely time- consuming and error-prone. Recently, self-supervised learning (SSL.) 99] 25 18] 45 72 (1) has
of supervised segmentation models
emerged in medical vision to pretrain a segmentation backbone (e.g. UNet) leveraging large scale unlabelled data, to serve as a better initialization before finetuning the model with limited annotation.
'Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

As one of the most successful learning paradigms in SSL. contrastive learning pulls closer the representations of positive pairs (i.e. the same image under different augmentations), while pushing apart the negative pairs (i.e. different images) p 28 I0
46 12 However, in medical images, strong anatomical similarities often exist across different images, resulting an increased number of false negative samples which is detrimental to the representation learning process. Therefore, heuristics about relationship across different images are utilized to constitute positive pairs, such as encouraging the global representations of 2D slices at similar volumetric positions 77] 72] to be close. To further benefit pixel-level tasks such as segmentation, local representation learning methods 7 learn distinctive local representations by attracting patch features at the same position from two augmented views 7 or across registered intra-subject volumes [51]. Nevertheless, the potential dependencies among patch features that spatially distant are not properly taken into account. Such region-awarenes, can be enhanced by attracting spatial features points with the same
25
semantics 35 27 38 75 1. However, their training requires segmentation ground truth, which
limits their applicability in medical vision when adequate annotation is unavailable.
In this paper, we incorporate the long-range spatial dependencies into the UNet backbone with plug-and-play keypoint-augmented fusion layer (KAF layer), accompanied by keypoinl-ehhaneed global and local objectives for pretraining the network with self-supervision. To do so, we first identify a set of sparse keypoint locations from the input image. On the output feature map of each UNet encoder block, the keypoint features are sampled from the convolutional feature map, and the attention among them are modeled through a Vision Transformer. The output is then scattered back to the feature grid, and gets fused to the original CNN feature, to explicitly provide clues the spatial long-range dependencies. Further, we propose the global and local self-supervised learning fusion layer -enhanced network. The global contrastive
objectives to pretrain the keypoint-augmentee loss is applied on both the UNet bottleneck feature, as well as obtained by aggregating the multiscale keypoint features. To further benefit pixel-level down-stream tasks, we identify the correspondence among keypoints across different image slices and maximize their feature similarities with local self-supervised objective.
global feature
Our contributions are threefold. (1) We develop a alug-and-ug that augments the comvolutional feature map with long-range dependencies among keypoint features. When trained with only limited annotation, our proposed KAF layer achieves noticeably better results than the CNN-only and/or Transformer- baasdd backbone. (2) We further propose the keypoint-aware global and local self-supervised learning objectives to pretrain our model before finetuning it with annotations. (3) We conduct extensive experiments three MRI and CT datasets, and achieve ofte-offtheear few-shot segmentation results, verifying both architecture advantage of the proposed layer and our pretraining strategies.
sappiitt-uugmeeted fusion layer
2 Related Work
Biomedical Image Segmentation is the process of dividing images into various segments or regions and isolating certain structures of interest modalities such as magnetic resonance imaging (MRI) and computed tomography (CT) [50 37.. UNet (52) is the pioneering method for addressing biomedical image segmentation tasks using deep learning. Subsequent research aims to enhance or apply the UNet model in various ways, such as extending to 3D volumes 44 64], auto-configuration (29].
altering skip connections [77 66
and replacing and/or combining the convolutional backbone 30 However, a common limitation shared by all end-to-end methods lies in the requirement of large-scale and high-quality biomedical annotations which are both time-consuming and requires expert knowledge
layers with Transformers 23
[52). This limitation underscores the need for novel methods that can learn effectively with fewer or without annotations.
Self-Supervised Learning (SSL) learns meaningful representations from unlabeled datasets with self-supervised objectives, and the pretrained network constitutes better initialization than random initialization when it is finetuned on downstream tasks with limited label supervision, e.g., classifica- tion or segmentation Existing SSL algorithms either define pretext tasks with heuristics, or employ self-supervised objectives to learn invariance and/or equivariance of image features. Early pex-text tasks based pretraining designs objective that does not require additional labels, such as rotation prediction [18), context restoration 48 reconstruction [32 among others [4
42
15 Recently,
contrastive learning has become prevailing paradigm in SSL and has shown great success both
natural image [6] 59 78 63 76 and medical vision 7 72 51 58 70 56 49
g 62
6 47
68)

---

It enforces similar representations between positive pairs and distinct representations between neg ative pairs. Typically, similarity is determined in an unsupervised way, i.e., the same image under different transformations are recognized as positive pairs, whose global representations should remain invariant, and features from different images are pulled apart. medical images, heuristics about relationships among different images are also utilized to define positives, e.g., 2D slices at similar volumetric positions 7 72), or local patches at the same location from intra-subject volumes [51].
Long-Range Dependency generally cannot be well captured in CNN due to its locality nature, thus neglecting the potential relationship across spatially distant local regions that possess high semantic consistency. To tackle the limitation, existing works focus on either (1) architectural design that injects long-range attention, or (2) SSL objectives that explicitly enforce regional similarity. For the first line of research, a series of non-local networks have been proposed. For instance, Vision Transformer (ViT) [14] enables self-attention on uniformly sampled image patches and has been
incorporated into CNN backbones for segmentation tasks [22 36 26
54
43
20
65 adopt
graph networks to model dependencies among context, keypoints or boundaries. The other line of research captures regional correlation with customized contrastive formulation objective, typically in a label-supervised manner [35 27 75 38), where local features within the same semantic label form the positíve pairs. In our setting, we eliminate the need of segmentation labels via establishing correspondence among detected keypoints and apply local self-supervision on the keypoint features.
Keypoint Descriptors have been successfully exploited in various domains to improve the perfor- mance on tasks such as pose estimation [9 image generation [57] and object detection [79 17]. Specifically, in biomedical imaging, keypoint descriptors have found applications in variety of areas, including anatomy object classification [31, medical image retrieval [74). segmentation tasks [69 61],
They aim to incorporate sparse and localized key points into the training process, in order to attend the network to the most important features within the image. This is typically done via first localizing sparse set of keypoint locations in the image, where keypoint features are obtained and their interactions are modeled. Finally, the feature is aggregated and used for downstream task prediction. In this work, we propose to inject learnable keypoint descriptors into the UNet building block so that the network is able to model complex long-range attention without significant increase in computational cost.
motion estimation of organs [53] and medical image registrations 21 3
3 Methodology
Fig. illustrates an overview of the proposed method. Built upon a segmentation backbone (i.e., UNet), we attach our proposed keypoint-augmented fusion layer (Sec. B.I after each convolutional block, to inject long-range self-attention into the original convolutional feature map. Further, we propose both global and local self-supervision (Sec. 3.2) pretrain the network before finetuning with limited annotation.
3.1 Keypoint-Augmenter Fusion (KAF) Layer
The proposed KAF layer is diagrammed in the red dashed box in Fig. 1 Given an input image X' E C we start with detecting keypoints K. E Nx2 using keypoint detector Dj RWx + N In particular, we employ the Scale-Invariant Feature Transform (SIFT) [41] We note that alternative keypoint detection methods can also be applied, and we have included the ablation in the appendix. At l-th layer of the 2D UNet, we acquire CNN features F E Nxxx
from the dense convolutional feature map F E WW'''H''x K". Note that at each scale, Ic' is acquired by re-scaling IC with the resolution ratio between the input image and the feature map. Then, we map the keypoint features to the embedding space with dimension E E with a projector ó x + Nx We use a single-layer MLP in our implementation as the projector. The embedded features are then fed into a Transformer T R N + Nx to learn the self-attention among keypoints. and we define the transformed feature as U² E NxE (which is further used as the keypoint feature at layer I for calculating the SSL. losses). Additionally, to properly propagate the keypoint features learned at the current scale to 1+1 within the convolutional UNet, U is scattered back into the feature grid indexed by the rescaled keypoint position K!! The output is sparse feature map Ft which retains information exclusively at the keypoint positions, and the rest of the feature map assigned with zeros.
based on the keypoint coordinates
3

---

UNet skip connection
Multiscale
Global Pooling
Global
KAt layer 2
RAE layer 31
keypoint feature
Loss
KAR layer
Conv
Conv
Conv
SIFT
Super
Local
Glue
Loss
keypoints
Input
Sample
Scatter
Projector
Transformer
CNN
Conv feature
Embedcling
keypoint
Sparse
Dense map
feature
feature maa
Keypoint-Augmented Fusion Layer (KAF Layer)
Figure I: Top: The overview of the proposed SSL framework that incorporates both local and global self-supervision with the keypoint guidance. The UNet decoder is omitted for better readability. Bottom: The proposed Keypoint-Augment spatial dependencies among localized keypoint features. We insert the KAF layer after each encoder block of the UNet backbone to augment the original convolutional features.
Fusion Layer (KAF layer), which learns long- aangrannge
We further attach two-layer CNN Dj diffuse the sparse keypoint feature map into dense image feature. Lastly, we concatenate the diffused keypoint feature map with the input feature map F!, constituting the final output of the proposed layer, represented F Concat(F", (F.).
3.2 Keypoint-Augmented Self-Supervised Learning
Furthermore, we design the global and local self-supervised learning algorithms to enhance the keypoint-augmeentd feature learning, thus the learned feature representations can be well generalized to different downstream tasks through finetuning.
that learns image wise feature similarity, we first aggregate multiscale keypoint features and extract a keypoint-enhancee global representation for each input image. Specifically, keypoint features from four encoder blocks are first concatenated as Concat (U' U U ,U*). which is then fed into a Multilayer Perceptron (MLP) get global feature Js. where i the index of the input image from the dataset. In our training, we adopt the global contrastive loss from PCL. 72 on the keypoint-enhance global feature, which assumes 2D slices within certain positional distance (within the 3D volume) threshold be anatomically similar and constitute similar pairs. Formally, the global self-supervised learning loss function can be formulated as:
Global SSL Loss. In order to apply a global contrastive loss (e.g.,
ssm (smi):99:)
L L
Σ log JEN
global
|22-1
link
k=1
where sim(:.)) is the cosine similarity between two vectors in the embedding space, indicates the temperature term. 5 is the set of positive images to the input Zt.
We additionally keep the global contrastive loss applied on the features from the last layer of the UNet
encoder, as commonly done existing SSL literature 772 9.. To do so, a global pooling layer to R Clx Afterward, an MLP projector is applied to obtain the global embedding for contrastive objectives, denoted c, Similar to Eq, [ the loss can be formulated by substituting 9i with e; and with e, in Eq. [ As we adopt PCL loss 72 to the global CNN feature, we denote this loss as LPCL in the following context.
transforms the feature Flast from TR wi Hlaxt xxlext
only considers image-wise similarity based on the slice positions, we claim that pixel-wise similarity across slices should also exploited to supervise the keypoint feature learning, enhancing ane-ggriied and localized control of the learned representation. We
Local SSL Loss. While 72

---

leverage the intrinsic structural correlation in 3D medical imaging, where nearby slices display strong local similarities. For instance, cardiac imaging, anatomical structures such as blood vessels and ventricles span across multiple 2D slices within the 3D volume. Therefore, keypoints within certain spatial distance from adjacent slices are likely to be semantically correlated and form positive pairs.
In order to locate the correspondence for each keypoint, we define two major criteria based on heuristics: (1) the spatial distance in the 2D plane between two correspondences should be within threshold; (2) the distance between their SIFT features should be similar. Correspondingly, we first sample two positive slices along with their keypoints (X, K.;) and (X,, K2), where the positional distance between X; and X fall within the positive threshold defined in [72]. For each keypoint E X3. we aim find its correspondence from X,. We do so by first filtering out a set of keypoints whose Manbattan distances with a is less than predefined threshold. Then, we find the keypoint b as the correspondence of a based on the closest 1.2 distance of the SIFT features. Once matched, (a, b) is added into the groundtruth match set M C AA x B. However, if no points in slice X; meet the criteria, the candidate point
treated as a negative sample and added to the unmatched set I S A. Similarly, for slice X,, the candidate point without match is added to the unmatched set J C B. M.I.J J are then used as the groundtruth for the keypoint feature matching detailed below.
We repurpose SuperGlue [54] as the backbone method to learn the correspondence between keypoint features on slice X; and keypoints on slice X,, supervised by the correspondence extracted above. We first compute the feature associated with keypoints U,- Concat(U? U/- U, U,)). and Uj- Concat(U), U3, UJJ Uf), along with their groundtruth matches M and unmatched sets I and J. Then, SuperGlue takes Uz, Uj. K,,, and K2 as inputs and computes an output P E [N (NV, through self- and cross- graph message passing. Here, Pa,b represents the probability that keypoint E K; matches with b E K.j. Given M, and J, we minimize the negative log- eeglikklihhod of the assignment P. formulated as follows:
Liocal =
Σ (a
log Pa, - Σ log Pa,N,+1 aES
Σ log PN,+1. BEJ
(2)
By establishing such localized correlation among keypoint features, we assume the network is able to better recognize local correspondence even when the input image undergoes various transformations. Such localized correlation ensures that the network can maintain a consistent understanding of the relationships between keypoint features, regardless of changes in the image appearance or orientation. Our assumption is empirically verified from the computed equivariant self-attention map shown in Fig. 3
Total SSL Loss. With the proposed
-uggentd fusion layer integrated into existing CNN backbone, the model can be pretrained with the above global and local losses, optionally with existing global contrastive loss (in our case, PCL 72).. The final objective is:
Leotal = w1 LPCL + W2 Lgtobol + wy Liocal
(3)
where the coefficients W1, W2, and Wy are employed balance the contributions of each term. In our experiments, we use a grid search method to optimize these coefficients.
4 Experiments
Datasets. We conduct experiments on two publicly accessible cardiac MRI datasets for the task of segmentation under limited annotation. For fair benchmarking, we follow the same preprocessing steps as 772.. For each pross-validation fold, Nimages are held-out for validation, and varying M images from the remaining images are used for the few-shot training. We additionally include results on a nan-car CT dataset Synapse' in the appendix illustrate the generalization of our method.
CHD (Congenital Heart Discase) 667 is a CT dataset consisting of 68 3D cardiac images, covering patent ages ranging from 1 month 21 year. It includes 14 types of congenital heart disease and the segmentation labels consists of seven distinct substructures: left ventricle (LV), right ventricle (RV), left atrium (LA), right atrium (RA), myocardium (Myo), aorta (Ao), and pulmonary artery (PA). For each cross-validation fold, N=18 images are used for validation, and M images are selected from the remaining 50 images for few-shot training.
https: http///wwwyaaps.eeee/e//e wwwppaaseerr//aee. htt4444aapppeppy111390//1/1111//
5

---

Table 1: Benchmark results on CHD and ACDC datasets under both random initialized weights, and pretrained weights from SSL. M is the number of patients used supervised training. We perform 5-fold cross- volsdvaaiiationn and the mean (standard deviation) dice scores are reported.
CHD (68 patients total)
Init.
Method
M=2
M=6
M=10
M=15
M=20
M=30
M=
UNer 52
0.184(00)
0.508(00)
0,584(005)
0.627(.05)
0.6581,04)
0.693(.04)
0.754(.02)
Swin-Unet 5
0.2911.07)
0.543(.07)
0.624(.04)
0.675(.05)
0.717(.04)
0.732(.05)
0.784(033
Random
SwinUNETR
33
0.345(.07)
0.565(.06)
0.638(00)
0.682(06)
0.711(,05)
0.725(066
0,785(,03)
Ours
0.344(00))
0.576(.07)
0.646(.03)
0.686(.03)
0.706(,03)
0.728,04)
0.778(003)
Rotation 18
0.171(.06)
0.488(.07)
0.575(.04)
0.625(.04)
0.651(,04)
0.691(004)
0.749(,03)
PIRL 445
0.196(,07)
0.504(.0)
0.617(.05)
0.658(003
0.674(.04)
0.714(.04)
0.761(.03)
SimCLR g
0.1921.06)
0.515(.06)
0.599(.06)
0.631(.05)
0.666(005
0.699(.05)
0.756(.03)
SSL
GLCL-gloobl
X
0,255(.10)
0.564(004)
0.646(.03)
0.669(00)
0,697(,04)
0.725(00))
0.766(03)
pretrain
GLCL-fill 7
0.286(006
0,555(.07)
0.614(.06)
0.666(044)
0.694.04)
0.733(004)
0,772(.03)
CAil) 56
0,265(.08)
0.581(.06)
0.647(004)
0.684(.04)
0.700(,04)
0.737(.04)
0.771(.02)
PCL
0.356(.08)
0.600(00)
0.661(.05)
0.686(.05)
0.716(,04)
0.735(005)
0.774(.03)
Ours
0.392(006
0.636(.06)
0.693(03)
0.712(.03
0.728(004)
0.754.04)
0.788(003
ACDC (100 patients in total)
Init.
Method
M=2
M=6
M=10
M=15
M=20
M=30
M=80
UNer 152
0.588(.07)
0.782(.03)
0.840.033
0.876(D1)
0.894.01)
0.909(01)
0.9284000
Swin-Unet [5]
0.181(.07)
0.483(.09)
0.610(00)
0.700(.04)
0.735(,02)
0.772(.11)
0,870(001)
Random
SwinUNETR
58
0,567(004)
0.740(.0)
0.799(.05)
0.850(.02
0.881(.01)
0.904(.01)
0,922(.00)
Ours
0.655(.05)
0.827(.05)
0.8716.02)
0.897.001)
0,901(.01)
0.915(000
0.927(.00)
Rotation 18
0,572(008)
0.809(03)
0. 0.868(002 02)
0.886(.01)
0.898(01)
0.910(011
0.925(.00)
PIRI. 45
0.492(.03)
0.823(.04)
0. 0.865(01)
0.880(02)
0.896(.02)
0.912(.01)
0.927(.00)
SimCLR g
0.352(.06)
0.725(.08)
0.824(00)
0.869(.02)
0.894(01)
0.913(001)
0,927(.00)
SSL
GLCLL-globvl ?
0.636(.05)
0.803(.04)
0.872(.01)
0.891(001)
0.902(001)
0.913(.01)
0,927(.01)
pretrain
GLCL-fill
0.6421.06)
0.802(.03)
0.877(01)
0.891(.01)
0,904,02)
0,9121.00)
0.927(.00)
CAiD [56]
0,483(111)
0.822(.02)
0,879(.0)
0.896(01)
0,905(.00)
0.914.00)
0,926(000
PCL 12
0.671(.06)
0,850(01)
0.885(01)
0,904(01)
0,909(.01)
0.919(00)
0,929(000)
Ours
0.741(.03)
0.873(.01)
0.895(.01)
0.908(.01)
0.915(.00)
0,921..00)
0.930.000
ACDC (Automatic Cardiae Diagnosis Challenge) 22 is a MRI dataset consists of cardiac images from 100 patients under several different pathologies. Manual expert segmentation of the RV, LV cavities, and Myo are conducted for the volumes from end-diastolic and end-systolic phase. For each cross-validation fold, N = 20 images are used as the validation set, and M images from the remaining 80 images are used for supervised finetuning.
Baselines and Evaluation Settings. We evaluate segmentation performance trained with limited annotation under both (1) random initialized network backbones (UNet v.s. Ours) to verify the architectural advantages of the proposed keypoint-augmented fusion layer. (2) pretrained weights from our self-supervision compared with various self-supervised pretraining methods including pre-text tasks [8] 45 and contrastive learning [9 77. We quantify the segmentation performance via
the Dice coefficient with five-fold cross validataion following the set up in (72). Additionally, conducted ablation studies to isolate each component in the pipeline to assess individual effects.
Implementation Details. For network configuration, we build our framework on 2D UNet, and set the starting number of channels of the network as 32 for CHD, and 48 for ACDC. Five convolutional blocks are included in the encoder and four blocks are used within the decoder. Each block contains two convolution layers, followed by batch normalization, and ReLU activation. Each encoder block downsample the feature map by factor of 2, while simultaneously doubling the number of channels. In between the first four encoder blocks, we further append the proposed leypoint-augmentte fusion layer, and concatenate the output to the original convolutional feature map (Fig. D Each Transformer inside KAF layer consists of six self-attention layers, and the comparison among different number of self-attention layers are presented in ablation study below. For pretraining,
assign loss weights W1. W2 W3 to 1.0, 1.0, and 0.01, respectively. We employ the SGD optimizer with a lcarning rate of 0.002 and batch sizes of 32 for CHD and ACDC. A cosine learning rate scheduler is utilized, with the minimum learning rate set to O. We pretrain the model for 50 epochs. For finetuning, we use
optimizer, with learning rates of to 5 x 10-5 for CHD, 5 10-4 for ACDC. The batch size is set to 10, and we finetune on CHD for 100 epochs and on ACDC for 200 epochs. Additional implementation details are provided in the appendix.
the standard cross-entropy loss with the Adam
33

---

Random Initialization
SSL pretrain Initialization
Input
(a) UNet
(b) SwinUNE
(c) Ours
(c) GLCL
(d) PCL
(e) Ours
(I) Reference
10.13
87.50
65.09
74.46
83.72
90.08
28. 15
54.04
72.24
74.84
43.95
79.63
CHD (M=15)
42.41
46.21
50.70
44.54
46.70
55.74
61.81
32
68.76
66.13
68.76
69.78
Few-shot segmentation results. Colomn (a)-(c) show a comparison among the UNet backbone, SwinUNETR (58), and our proposed backbone trained on limited dataset with random weight initialization. Our architecture allows for reduced false positive and improved segmentation accuracy. Column (c)-(e) compares our pretraining strategy with the existing state-of- of-the-art SSL methods 77 and (72). When finetuned with small number of labeled dataset, our method presents higher coherence with the reference image (column (f)). The number beneath each prediction
Figure 2:
represents the dice value for the displayed slice. For better readability, we scale the value by 100.
Segementation Results without Pretraining. To verify the benefits of the proposed layer, we start with analyzing the performance gain without any pretraining, isolate and evaluate the impact solely attributed to the architectural change on top of the UNet backbone. Specifically, we train the randomly initialized UNet and our backbone with varying limited number of training data, and the results under Init. Random. Compared with the UNet backbone with an identical number of convolutional blocks and network configurations, introducing KAF layer consistently and significantly improves the segmentation results on both datasets across various training sample sizes. The empirical findings strongly support our assumption that incorporating long-range dependencies into the segmentation backbone is crucial and helps better modeling and utilizing regional image information when only limited annotations are available. Notably, our network demonstrates superior performance over the UNet baselines when the training size is extremely small. For instance, when the sample size is reduced to two (M- = 2) on the CHD dataset, the inclusion of KAF layer leads to 87% performance gain in dice score over UNet. We also include Transformer- based UNets 5 588 which also consider long-range spatial dependencies. Our observation is consistent with the conclusion in [5), where Transformers may be more severely affected by the initialization and require large M in order to perform well, while method obtain much better results when only limited data is available.
are shown in Table [
Fig. 2 Random Initiall antiaaiiaat iizitiionn shows a visual comparison of the segmentation between UNet and Ours. While both methods suffer from sub-optimal generalization ability on the input image from the unseen validation set, our method largely reduces false positive prediction and presents higher similarity with the reference image on the right. Such improvement can be attributed to the utilization of keypoints, which enhances the feature learning process by directing attention to more constrained set of regions rather than the entire image space. we deduce that this focused awareness contributes to better localization of important features thus leading to better semantic segmentation.
Segmentation Results with Pretraining and Finetuning. To further enhance the segmentation performance and promote better generalization of our model, we pretrain the network with the proposed SS objectives and leverage the pretrained weights as the initialization for fine-tuning on the same amount of labeled dataset. Quantitative results are shown in Table | under the tab SSL pretrain Enitialization where benchmarking is conducted across existing state-of-the-art SSL pretraining methods. Notably, our pretraining technique yields significant improvements in few-shot
7

---

Table 2: Ablation study of our framework over (I) architecture design: the number of self-attention layers within the transformer (#T), scales insert KAF layer (1,,- 14); and (2) pretraining hyper- parameters (Wi, 12 Wz). Five-fold cross-validation results on both datasets are reported.
Architecture design
Pretraining
Dice
Init.
Exp
VT
1/
lg
W/
Wy
CHD (M=15)
ACDC (M==)
A
9
0
0
0
0
0.627(.05)
0.782(.03)
B
9
1
0.658(004
0.806(00)
9
1
0.677(.04)
0.817(04)
9
0
0.667(.04)
0.814(.04)
Random
F
9
0
1
0.666(.04)
0.811(00)
F
9
1
0. 0.686(003)
0.827( 05)
G
6
1
0.690(003
0.844.03)
3
0.677(.03)
0.814(.03)
9
1
0
0
0.699(.04)
0.867(00)
9
1
0
0.701(.03)
0.865(02)
SSL pretrain
9
0.711(.03)
0.865(01)
9
0.01
0.712(.03)
0.873(.01)
segmentation tasks compared to the random initialization. Moreover, in comparison to alternative pretraining strategies that do not incorporate long-range dependencies, method consistently achieves state-of-the-arr dice scores across different numbers of training subjects for both datasets, demonstrating the benefits of the proposed eeypoint-augmenti
SSL objectives. In Fig. 2 SSL initialization we present visual comparisons between our method and the best-performing SSL methods GLCL 7 and PCL 72.. Visually, we observe that GLCL tends to produce false positives or incorrectly labels the anatomy, as evidenced by the CHD result. Conversely, PCL tends to generate false negatives, observed in the ACDC example. contrast, our proposed method achieves the highest level of similarity with the reference image, accurately capturing the desired anatomical structures. These visual comparisons further reinforce the effectiveness and superior performance of our approach in comparison existing SSL methods.
Ablation Study. As our proposed framework consists of both architectural change on the UNet building blocks and pretraining strategy, we conduct ablation analysis over different architectural configurations and pretraining objectives, with results presented in Table 2 Additional ablations on keypoint detection method and threshold for identifying correspondence are provided in the appendix.
Architecture (Exp A-H). We start with architectural analysis by training different randomly initialized backbone networks under different configurations over the number of elff-attention (#T) within the KAF layer, , and the injection of KAF layer at different encoder scales ,,,4..
Exp A represents a UNet backbone without the use of KAF layer. With the same #T, Exp inserts four KAF layer after each of the encoder layer. A significant performance gain obtained on CHD by 0.06, and on ACDC by 0.04. To analyze layerwise effects, we remove one KAF layer each time from Exp B-E, and observe degradation in performance, indicating the importance of utilizing a multiscale setting. We observe that the last layer plays the most crucial role achieving superior segmentation results. In addition, Transformer used in our architecture in Exp F-H. We observe that the performance reaches a plateau with #T = 9, and when it reduces to 3, the performance further degrades. Nevertheless, Exp H still overperforms Exp A, indicating the benefits of introducing long-range self-attention.
investigate the effect of the number of self-attention within the
Pretraining Strategy (Exp I-L). On the architectures of Exp F, we further add the proposed pretrain- ing strategies, and isolate the different loss weights during the training. W1, We indicate the global PCL loss applied the convolutional feature and keypoint features respectively, w3 indicates the local loss weight (See Eq. 3). Compared with single global loss on either the global CNN feature (Exp or the keypoint feature (Exp J), a combination of the two terms (Exp K) shows improved results on CHD. The best performance is achieved with a combination of both global and local loss (Exp L). Additional weight tuning details are provided in appendix.
Qualitative Analysis of the Learned Self- Attentonnnnon. In Fig. we qualitatively compare the learned self-similarity from our pretraining. with the established PCL 772 (global), and GLCL 7 (global and local) pretraining. Given an input image r, we perform two random transformations T, and T2 to obtain simulated positive pairs in contrastive set up. A query point is randomly sampled from

---

(b)
PCL
GLCL
Ours
(((())
S(Fx(T2(x)
(x(S(F(())
((F())))))
S(F((T2())
T2(5(F((x))
Query point
Ft feature map from the I-th layer output
random data augmentation
Self-simllarity between query point image
Figure Learned multiscale self-similarity between the query point feature (star) and all other feature points within the same feature map from the UNet encoder. The comparison between (a) and (b), (d) and (e) indicates that our method maintains better invariance of the local feature self-similarity under different transformations. Additionally, the coherence between (b) and (c), (e) and (f) verify that our method learns better equivariance. Details are elaborated Sec.
the x and gets transformed to (z) and T2(2:) respectively (stars in Fig. 3). The image is fed into the pretrained network, and the feature map from the i-th layer is obtained. The self-similairty S is computed as the dot product between the query point feature and other feature points within the same map. Fig. 3 visualizes the self-attention from the third and fourth layer of the pretrained UNet (la, 14).
We observe two major differences between our method and PCL and GLCL: (1) our keypoint- augmented fusion layer helps the network to concentrate its attention on more constrained sub-regions. For example, both PCL and GLCL tend to learn high response values between the query point and background regions (e.g.. Fig. (b), (e)), whereas our method exhibits more refined attention mechanism, directing the focus of the query point towards anatomically relevant regions, which helps the model to capture and leverage important features within the desired regions, leading improved segmentation performance. (2) our proposed self-supervision helps maintain better local equivariance of the self-attention, i.e., with the same query point location, its interaction with other points remains identical no matter how the image is transformed. This is verified by comparison between Fig. (a) (b), and (d) v.s. (e), where our method achieves the best consistency of the local similarity when features are extracted from the image with different transforms.
Additionally, calculate the self-attention based the input image and transform the resulting similarity map using T2. The transformed similarity map is expected to exhibit high level of coherence with the similarity map obtained from the transformed input. Compared with PCL and GLCL, our method demonstrates stronger consistency between (b) and (c), as well as (e) and (f), which validates the advantage of introducing the -aapointtaugmentee self-supervised learning.
5 Discussion
In this work, we present a keypoint-augmeented fusion layer to incorporate long-range dependencies into the UNet-based segmentation framework, accompanied by global & local SSL objectives for pretraining. Open questions exist and will be included in future work. Currently, our backbone is built upon 2D UNet following [72] 77, while 3D UNet naturally serves as a better baseline in many biomedical segmentation tasks [51]. Although our method is generic and can be scaled up to 3D data, it may require different architectural configurations accommodate the increasing number of keypoints within the 3D volume. Besides, our assumption for identifying correspondence may fail in 3D volume with the sparse acquisition, when the neighboring slices no longer maintain semantically

---

similar structures. Therefore, more robust and complicated criteria for obtaining the correspondence required.
Acknowledgements
Co-authors Zhangsihao Yang and Yalin Wang are grateful R21AG065942, RO1EY032125, and R01DE030286.
References
[1] lñigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano, and Ana C. Murillo. Semi- supervised semantic segmentation with pixel-level contrastive learning from a class wise mem- ory bank. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8219-8228, October 2021.
[2] Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, et al. Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging, 25511)2554-2255, 2018.
[3] Alexander Bigalke, Lasse Hansen, and Mattias P Heinrich. Adapting the mean teacher for keypoint-based lung registration under geometric domain shifts. In Medical Image Computing and Computer Assisted Intervention-MICCAA 2022: 25th International Conference, Singapore, September 18-22 22, 2022, Proceedings, Part VI, pages 280-290. Springer, 2022.
[4] Guillem Brasó, Nikita Kister, and Laura Leal-Taixé. The center of attention: Center- Center-keypoint grouping via attention for multi- multi-person International Conference Computer Vision, pages 11853- 11853-11863, 2021.
person pose estimation. In Proceedings of the IEEE/CVF
[5] Hu Cao. Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. Swin-unet: Unet-like pure transformer for medical image segmentation. In European conference computer vision, pages 205-215 Springer, 2022.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self sulf-super vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9650-9660, 2021.
[7] Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, 33:1 2-21122661111 12558, 2020.
[8] Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu. Local contrastive loss with pscudo-label based self-training for semi-supervised medical image segmentation. Medical Image Analysis, 87:102792. 2023.
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[10] Ting Chen, Simon Korbblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi- comi-supervised learners. Advances neural information processing systems, 33:22243 22255, 2020.
[11] Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34, 2021.
[12] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 15750-15758, 15758, June 2021.
[13] Daniel Denone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pages 224-236, 2018.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
10

---

An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[15] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. Advances in neural information processing systems, 27, 2014.
[16] Mare Fischer, Tobias Hepp. Sergios Gatidis, and Bin Yang. Self-supervised contrastive learning with random walks for medical image segmentation with limited annotations. Computerized Medical Imaging and Graphics, 104:102174, 2023.
[17] Kun Fu, Jiamei Fu, Zhirui Wang, and Xian Sun. oriented ship detection in high-resolution and large-scale sar images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:11 162-11188, 2021.
network for
[18] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint
1801.0872722 2018.
[19] Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 1090-11100, 2022.
[20] Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu. Vision gnn: An image is worth graph of nodes. arXiv preprint arXiv:2206.0022
2022.
[21] Lasse Hansen and Mattias P Heinrich. Graphregnet: deep graph regularisation networks on sparse keypoints for dense registration of 3d lung cts. IEEE Transactions on Medical Imaging. 40(9):2246- 2257, 2021.
[22] Ali Hatamizadeh, Vishwesh Nath. Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In International MICCAI Brainlesion Workshop, pages 272-284. Springer, 2021.
[23] Ali Hatamizadeh, Ziyue Xu, Dong Yang, Wenqi Li, Holger Roth, and Daguang Xu. Unet- former: A unified vision transformer model and pre-training framework for 3d medical image segmentation. arXiv preprint arXiv:2204.006 10.31, 2022.
[24] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[25] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[26] Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. In European Conference on Computer Vision, pages 108-126. Springer, 2022.
(27] Hanzhe Hu, Jinshi Cui, and Liwei Wang. Region-aware contrastive learning for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 16291-16301, October 2021.
[28) Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. Boosting contrastive self-supervised learning with false negative cancellation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision pages 2785-2795 2022.
[29] Fabian Isensee, Paul Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein, nnu-net: self-configuring method for deep learning- based biomedical image segmentation. Nature methods, 1822)203-211, 2021.
[30] Yun Jiang, Yuan Zhang, Xin Lin, Jinkun Dong, Tongtong Cheng, and Jing Liang. Swinbts: A method for 3d multimodal brain tumor segmentation using swin transformer. Brain sciences, 12(6):797, 2022.
[31] Sumeer Ahmad Khan, Yonis Gulzar, Sherzod Turaev, and Young Suet Peng. A modified hsift descriptor for medical image classification of anatomy objects. Symmetry, 13(11):1987, 2021.
[32] Dahun Kim, Donghyeon Cho, Donggeun Yoo, and In So Kweon. Learning image representations by completing damaged jigsaw puzzles. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 793-802. IEEE, 2018.
11

---

[33] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[34]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXin:2304.02643 2023.
[35] Sangwoo Lee, Yejin Lee, Geongyu Lee, and Sangheum Hwang. Supervised contrastive embed- ding for medical image segmentation. IEEE Access, 9: 9:138403-13844 2021.
[36] Ailiang Lin, Bingzhi Chen, Jiayu Xu, Zheng Zhang, Guangming Lu, and David Zhang. Ds- transumet: Dual swin transformer u-net for medical image segmentation. IEEE Transactions on Instrumentation and Measurement, 71:1-15, 2022.
[37] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi,, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. A survey on deep learning medical image analysis. Medical image analysis, 42:60-88, 2017.
[38] Shikun Liu, Shuaifeng Zhi, Edward Johns, and Andrew Davison. Bootstrapping semantic segmentation with regional contrast. In International Conference on Learning Representations, 2022.
[39] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[40] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[41] David G Lowe. Distinctive image features from scale-invariant keypoints. International of computer vision, 60:91-110, 2004.
[42] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461-11471, 2022.
[43] Yanda Meng, Hongrun Zhang, Yitian Zhao, Xiaoyun Yang. Yihong Qiao, Ian JC MacCormick, Xiaowei Huang, and Yalin Zheng. Graph- region and boundary aggregation for biomedical image segmentation. IEEE transactions on medical imaging, 41(3):690-700 2021.
[44] Fausto Milletari, Nassir Navab, and Seyed- Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565-571. leee, 2016.
[45] Ishan Misra and Laurens der Maaten. Self-supervised learning of pretext-invariant rep- resentations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6707-6717, 2020.
[46] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 077800778..... 2018
[47] Cheng Ouyang, Carlo Biffi, Chen Chen, Turkay Kart, Huaqi Qiu, and Daniel Rueckert. Self- supervised learning for few-shot medical image segmentation. IEEE Transactions on Medical Imaging, 41(7):
2022.
[48] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536-2544, 2016.
[49] Jizong Peng, Marco Pedersoli, and Christian Desrosiers. Boosting semi-supervised im- age segmentation with global and local mutual information regularization. arXiv preprint arXiv:2103.0453 044.13,,, 2021.
[50] Dzung L Pham, Chenyang Xu, and Jerry L Prince. Current methods in medical image segmen- tation. Annual review of biomedical engineering, 2(1):315-337, 2000.
[51] Mengwei Ren, Neel Dey, Martin Styner, Kelly Botteron, and Guido Gerig. Local spatiotemporal representation learning for longitudnallly-coniisen neuroimage analysis. Advances Neural Information Processing Systems, 35:13541-13556 2022.
12

---

[52] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAN 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[53] Jan Rühaak, Thomas Polzin, Stefan Heldmann, Ivor JA Simpson, Heinz Handels, Jan Moder- sitzki, and Mattias P Heinrich. Estimation of large motion in lung by integrating regularized keypoint correspondences into dense deformable registration. IEEE transactions on medical imaging, 36(8): 1766))177677777,, 2017.
[54] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938-4947, 2020.
[55] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556, 2014.
[56] Mohammad Reza Hosseinzadeh Taher, Fatemeh Haghighi, Michael B Gotway, and Jianming Liang. Caid: Context-aware instance discrimination for self-vupe learning in medical imaging. In International Conference on Medical Imaging with Deep Learning, pages 535-551. PMLR, 2022.
[57] Hao Tang, Dan Xu, Gaowen Liu, Wei Wang, Nicu Sebe, and Yan Yan. Cycle in cycle generative adversarial networks for keypoint-guides image generation. In Proceedings of the 27th ACM international conference on multimedia, pages 2052-2002200 2060, 2019.
[58] Yucheng Tang, Dong Yang. Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20730-207400 2022.
[59] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Com- puter EssionEECCy 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 776-794. Springer, 2020.
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[61] Christian Wachinger, Matthew Toews, Georg Langs, William Wells, and Polina Golland. Key- point transfer for fast whole-body segmentation. IEEE transactions on medical imaging, 39(2):273-23) 282. 2018
[62] Bo Wang, Qian Li, and Zheng You. Self-supervised learning based transformer and convolution hybrid network for one-shot organ segmentation. Neurocomputing, 527:1-12, 2023.
[63] Zhaoqing Wang, Qiang Li, Guoxin Zhang, Pengfei Wan, Wen Zheng, Nannan Wang. Mingming Gong, and Tongliang Liu. Exploring set similarity for dense self-supervised representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16590-16599, 2022.
[64] Zhengyang Wang. Na Zou, Dinggang Shen, and Shuiwang Ji. Non-local u-nets for biomedical image segmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6315-6322, 2020.
[65] Zhanghao Wu, Paras Jain, Matthew Wright, Azalia Mirhoseini, Joseph E Gonzalez, and Ion Stoica. Representing long-range context for graph neural networks with global attention. Advances in Neural Information Processing Systems, 34:13266-1327, 2021.
[66] Xiao Xiao, Shen Lian, Zhiming Luo, and Shaozi Li. Weighted res-unet for high-quality retina vessel segmentation. 2018 9th international conference on information technology in medicine and education (ITME), pages 327-331. IEEE, 2018
[67] Xiaowei Xu, Tianchen Wang. Yiyu Shi, Haiyun Yuan, Qianjun Jia, Meiping Huang, and Jian Zhuang. Whole heart and great vessel segmentation in congenital heart disease using deep neural networks and graph matching. In Medical Image Computing and Computer Assisted InterventionMICCAA 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part II 22, pages 477-485 Springer, 2019.
13

---

[68] Ke Yan, Jinzheng Cai, Dakai Jin, Shun Miao, Dazhou Guo, Adam P Harrison, Youbao Tang, Jing Xiao, Jingjing Lu, and Le Lu. Sam: Self-supervised learning of pixel-wise anatomical embeddings in radiological images. IEEE IEEE Transactions on Medical Imaging, 41(10):2658-26)) 2022.
[69] Jingru Yi, Pengxiang Wu, Qiaoying Huang, Hui Qu, Bo Liu, Daniel J Hoeppner, and Dimitris N Metaxas. Multi-scale cell instance segmentation with keypoint graph based bounding boxes. In Medical Image Computing and Computer Assisted iireventtionMMICAA 2019: 22nd Inter- national Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part 22, pages 369-377, Springer, 2019.
[70] Chenyu You, Weicheng Dai, Yifei Min, Fenglin Liu, Xiaoran Zhang, Chen Feng, David A Clifton, Kevin Zhou, Lawrence Hamilton Staib, and James S Duncan. Rethinking semi- supervised medical image segmentation: A variance-reduction perspective. arXiv preprint MoKiv:2302.01755 2023.
[71] Chenyu You, Weicheng Dai, Lawrence Staib, and James S Duncan. Bootstrapping semi- supervised medical image segmentation with anatomical-awaan contrastive distillation. arXiv preprint Xiv: 02307, 2022.
[72] Dewen Zeng, Yawen Wu, Xinrong Hu, Xiaowei Xu, Haiyun Yuan, Meiping Huang, Jian Zhuang. Jingtong Hu, and Yiyu Shi. Positional contrastive learning for volumetric medical image segmentation. In Medical Image Computing and Computer Assisted InterventinnMICCAA. 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part 11 24, pages 221-230. Springer, 2021.
[73] Ruifeng Zheng, Ying Zhong. Senxiang Yan, Hongcheng Sun, Haibin Shen, and Kejie Huang. Msvrl: Self-supervised multiscale visual representation learning via cross-level consistency for medical image segmentation. IEEE Transactions on Medical Imaging, 42(1):91-102, 2022.
[74] Li-Jia Zhi, Shao-Min Zhang, Da-Zhe Zhao, Hong Zhao, and Shu-kuan Lin. Medical image retrieval using sift feature. In 2009 2nd International Congress on Image and Signal Processing,
pages 1-4. IEEE, 2009.
[75] Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. Pixel contrastive-comsisten semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 7273-7282, October 2021.
[76] Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. Pixel contrastive-consiiten semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7273-7282. 2021.
[77] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang, Unet++: Redesigning skip connections to exploit multiscale features in image segmentation. IEEE transactions on medical imaging, 39(6): 856-1867, 2019.
[78] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6002-6012, 2019.
[79] Tianpei Zou, Guang Chen, Zhijun Li, Wei He, Sanqing Qu, Shangding Gu, and Alois Knoll. Kam-net: Keypoint-aware and keypoint-matching network for vehicle detection from 2-d point cloud. IEEE Transactions on Artificial Intelligence, 3(2):277-217, 2021.
14

---

A Additional Experiments
More comparison with Transformers As we pointed out in the main text, Transformer-based architecture may be extremely sensitive to weight initialization, and may benefit from effective pretraining. Therefore, we further pretrain the SwinUNETR with two SSL strategies: PCL [72] and the self-supervised loss proposed in [22] (in 3). Both of these pretraining methods led to observable improvements over random initialization, while our proposed method still achieved the best performance.
Table 3: Comparison with Transformer-based methods on CHD and ACDC datasets under both random initialized weights, and pretrained weights from SSL. M is the number of patients used in supervised training. We perform 5-fold cross-validation and the mean (standard deviation) dice scores are reported.
CHD (68 patients in total)
Init.
Method
M=2
M=6
M=10
M=15
M=20
M=30
M=51
SwinUNETR-3d (58
01140(01)
0,189%.03 03)
0.352(.08)
0,420X (06]
0.472(,03)
0.504..5)
0.577(06)
Random
0.344(.05)
0.576(.07)
0.646(.03)
0.686(.03)
0.706(.03)
0.728(04)
0.778(003)
PCL (Swin- (Swin-Unet) 5
0.321(.05)
0,556 06)
0.658(003)
0.692(.04
0.722(.04)
0.747(044)
0.783,03)
SSL
PCL. (SwinUNET))
0.343(00)
0,576(077)
0.644(005)
0.692(.06
0.714(.05)
0.730X 05)
0.786(.03)
Ours
0.392(006)
0.636(.06
0.693(.03)
0.712(,03)
0.728(.04)
0.754(04)
0.788(003)
ACDC (100
patients in total)
Method
M=2
M=6
M=10
M=15
M=20
M=30
M=80
SwinUNETR-3d
0,391007)
0,466 05)
0,545(00)
0,603(033
0.629(033
0.679(003)
0.735(.11)
Random
Ours
0.655(.05)
0.827(.05)
(.871(002
0.897(001
0.901(.01)
0.915(.00)
0.927(.00)
PCL (Swin-Unet) 3
0.292(.09)
0,577(.07)
0,702(.04)
0.742(.03)
0.798(022)
0.825(.02) 02)
0,878(01)
PCL. (SwinUNETR)
58
0.547(.05)
0.759
0,808(.0)
0.850(.03)
0.882(.0)
0,904(01)
0.923.00)
Ours
0.741(.03)
0.873(01)
0.895(.01)
0.908(.01)
0.915(.00)
0.930(.00) (.921(.00)
Ablation keypoint detection method While use SIFT keypoint detector, it can be replaced with other learning based SoTA methods as well. We used a pretrained keypoint detection model alternative SIFT. Segmentation results CHD and ACDC dataset reported below comparing two methods, and the results indicate that our method is not sensitive to different keypoint detection algorithms. Overall, Superpoint leads to slightly lower results, and we speculate this was due to the pretraining was done on natural images (COCO dataset).
Superpoint [33
Table 4: Segmentation results (dice scores) with different keypoint detection methods (M=2).
Init.
Keypoint detector
CHD
ACDC
Random Init.
SuperPoint B
0.643(.04)
0.810K002)
SIFT
0.686(.03)
0.827(.05)
SSL Pretrain
SuperPoint 13
0.703(003
0.865(002)
SIFT
0.712(.03)
0.873(001)
Sensitivity of the keypoint correspondence To verify how sensitive the method is to the number of matching keypoints, we perform an ablation on different threshold values, to pretrain our model on the CHD dataset, and finetune it on M=15 labeled data. Global losses ( (W] = W2 = 0) are turned off isolate the effects the threshold for the local SSL loss. The dice scores are reported in Table indicating that our model is not sensitive to the threshold setting and remains robust across different values. We also visualize the correspondence and the number of matching keypoints under different threshold values in Fig
15

---

Table 5: Dice scores under different keypoint correspondence threshold values. Overall, we observe that the results are not sensitive to specific threshold values.
Threshold
Dice
5
0.689 (0.043)
10
0.690 (0.031)
0.684 (0.035)
20
0.689 (0.036)
25
0,684 (0,032)
30
0.685 (0.033)
0.690 (0.037)
40
0.687 (0.037)
5,
15,
Manhattan 20, Manhattan match
Manhattan
Figure 4: Sensitivity analysis of the Manhattan distance, illustrating the correspondence found under the specified Manhattan distance threshold.
Additional dataset We include experiments on a non-cardiac multi-organ CT segmentation dataset to evaluate the generalization of our method. Results reported below. With 2 training subjects, tested the performance the model under both random and SSL pretrained initialization. In both scenarios, method outperformed the existing works.
Table 6: Segmentation results on Synapse dataset, where finetuning is done on M-2 subjects out of
total of 18 subjects.
Init.
Method
Dice (M=2)
Unet
0.253(006
Swin Unet
0.198(.04)
Random
SwinUNETR
58
0.279(.06)
Ours
0.289(.06)
PCL. 72
0.306(.05)
Swin- Unet (with 72 )
0.210(.77)
SSL
SwinUNETR (with 72
0.304(06)
Ours
0.322(.06)
16

---

Ablation on UNet Channels (Double Channels). In our implementation, we augment the convolu- tional UNet features by concatenating them with features learned from KAF. Introducing KAF layers leads to an increase in the total number of parameters of the final KAF-enhanced UNet. Specifically, the number of channels for the second, third, and fourth blocks in the KAF-enhanced UNet become twice as large as the original UNet. For more fair comparison, we construct a non-KA non-KAF UNet ( ('UNet (c2) in Tab, 7 with the same amount of parameters in convolutional layers as our model by duplicating the features of the first, second, third, and fourth blocks from the UNet baseline ('UNet (c1)) in Tab. 7 and concatenated each with itself. The segmentation results are presented in Table 7 UÑet (c1) indicates the UNet backbone, and UNet (c2) denotes the larger UNet whose convolutional parameters matched our model. They indicate that widening the network 's architec- ture by increasing the input channel size can improve its performance. However, the performance enhancement is even more substantial with our modified layer. This suggests that incorporating features beyond simple convolution into the network architecture can further enhance the network's performance.
Table 7: Performance comparison among standard UNet (UNet (c1)), larger UNet with duplicated input channels (UNet (c2)), and UNet augmented with features derived from KAF (Ours). The results indicate that using a larger UNet slightly improves the segmentation performance. Our proposed KAF-enhanced UNet further boosts the performance significantly compared with UNet (c2).
Sample M
dataset
Method
mean/std
#params
15
CHD
UNet(c1)
0.627(.05)
7.8 M
15
CHD
UNet(c2)
0.646(.04)
27.9 M
15
CHD
Ours
0.712(.03)
71.7 M
6
ACDC
UNet(cl)
0.782(.03)
17.5
6
ACDC
UNet(c2)
0.796(.03)
62.8 M
6
ACDC
Ours
0,873(01)
106.8 M
Ablation on Correspondence Weights. To further investigate the contribution of the correspondence loss to the performance of the pretraining weights, we conducted a study on the various combinations of weights W1. W2, and W3, as defined in 3 and results are reported in Table X
First, we study the effect of varying wy by setting both wi and w2 to 1. We experimented with values of 0. 0.01, and 0.001 for wz. Among these, 0.01 yielded the best performance. In subsequent series of experiments, we set w1 to O, intending to exclusively investigate the impact of the local correspondence loss on the global KAF self-supervised learning (SSL) loss. When only w2 is set to 1, the model achieves an dice of 0.701. Upon varying W3 while keeping w2 fixed at 1, in most instances, incorporating Liocal yields performance better than with Lgiobal only. This implies that our local KAF correspondence SSL loss indeed offers superior local minimum for downstream tasks. When the weights are set to 1, 1, and O, our model's performance, 0.689, still surpasses that of model trained from scratch (0.686). This suggests that using only Liocal could assist in finding more optimal starting point for fine-tume tin-ttnnng. However, when compared having only local losses, global pretraining losses contribute more to enhancing performance.
Table 8: Additional assessment of weight of Liseat in pretraining to supplement Tab. 2 in the main text. The results are all from pretraining on CHD and finetuning at M 15.
W1
102
Dice
0.1
0.702(.04)
0.01
0.712(.03)
0.001
0.708(.04)
0.08
0.705(.03)
0.04
0.705(003
0 0.02
0.707(.03)
0 0.01
0.700(.03)
0
0.005
0.700(.03)
0.001
0.705(.03)
0.689(.04)
17

---

KAF Layer in FCN. To validate the general usefulness of the KAF layer, we injected KAF layer into a different segmentation backbone Fully Convolutional Network (FCN) 40 and compare the results between the original FCN versus the KAF-enhanced FCN. Note that in the original FCN implementation, VGG [55] was used as the encoder to gather multi-resolution features from different layers. However, given the small training size of our dataset and the relatively high complexity of VGG, we substitute the VGG encoder with a shallower CNN. Specifically, we replaced each block in VGG with two convolution layers, aiming create more efficient model that better suits our dataset and task.
The results of our experiment are reported in Table We conduct trials with different training sample sizes. The results verify that appending KAF layers to FCN boosts performance across all sample sizes, with an average improvement exceeding 1 %. Interestingly. we observed that FCN outperforms UNet tasks involving training from scratch. However, when we incorporated the KAF layer into FCN, it did not surpass the performance of our layer applied to UNet. This could potentially be attributed to our approach this experiment; we did not undertake hyperparameter optimization but instead directly added the layer we used in our main text to FCN. Therefore, compared to UNet, the performance improvement relatively smaller.
Table 9:
Segmentation results on CHD dataset from random initialized FCN backbone. The column 'With KAF' indicates whether the proposed KAF layer is inserted to the backbone. The results demonstrate that the integration of the KAF layer tends improve the mean values across different sample sizes, indicating an enhanced performance of the FCN when augmented with the KAF layer.
Sample M
With KAF
Fold 1
Fold 2
Fold
Fold 4
Fold 5
Mean/Std
0.2259
0.2133
0.3297
0.311
0.3516
0.286(.056)
2
V
0.2517
0.2133
0.3392
0.3034
0.3794
0.297(.059)
0.4441
0.5462
0.5427
0.5918
0,4854
0.522(.052)
6
/
0.4495
0,5603
0.5652
0.6286
0.5535
0.551(.058)
0.4866
0.5584
0.6393
0.6613
0.6094
0.591(.063)
10
/
0,5632
0.6209
0.6381
0.6718
0.6383
0.626(.036)
0.5938
0.6356
0.6702
0.6926
0.6499
0.648(.033)
15
0.6157
0.6163
0.7117
0.6936
0.6537
0.658(.039)
0.6318
0.6422
0.7102
0.7383
0.6356
0.672(.044)
20
>
0.6382
0.6541
0.7112
0.7309
0.6806
0.683(034)
0.6339
0.6558
0.7498
0.7701
0.6685
0.696(.054)
30
0.6841
0.6671
0.7507
0.7708
0.6973
0.714(.040)
0.7125
0.7287
0.7693
0.7755
0.7559
0.748(.024)
0.7087
0.7324
0.7711
0.7813
0.7661
0.752(.027)
Computation Analysis. As transformers (60] also incorporate long-range dependencies by learning self-attention among uniformly distributed patches within the image, we compare the computational differences of SwinTransforma
(39 and our model. Fig. displays a comparison of GFLOPs and GPU memory usage between our method and the SwinTransformer given two specific variations: the edge length of the input image and the number of self-attentions within each transformer.
In the left-side plot, the GFLOPs of our method vary by a constant value (around 130 GFLOPs) as the depth of the attention map escalates. Conversely, this metric grows exponentially in the SwinTransformer. The disparity arises because our method keeps the number of keypoints constant, meaning that even the edge length of the input image enlarges, the computation surge in the attention map remains consistent, In contrast, transformer- based methods like the Tranfforrfrrmrr require quadratic increase in computation to generate the attention map.
Our method demonstrates a slower growth rate in memory usage, particularly as the number of self- attentions increases. In the SwinTransformer the size of the attention map correlates quadratically with the edge length of the image. However, our approach maintains a fixed number of input keypoints, which stabilizes the attention map constant factor in memory usage.
18

---

o7
04.05
ACDD CHD
ACDC
CHD
200 059
1000 2300 3400
2003
1200 1800
1600
Figure 5: Comparison of GFLOPs and GPU memory usage between our method and the SwinTraas- former [39]. The x-axis indicates the size of the image. We test models with different numbers of self-attention blocks (20 or 100) within the transformer, represented by different colors. The results illustrate that the GFLOPs of our method vary minimally with increasing attention map depth, and the memory usage of our method increases at slower rate with growing number of self attennttens This highlights the efficiency of our method, especially with larger input image edge lengths and more complex attention maps. We also denote the actual input size of CHD dataset (512 512) and ACDC dataset (352 x 352), where our model more computationally efficient than using transformer in both the GFLOPs and memory consumption.
Comprehensive 5-Fold Results. Our detailed examination of the five folds in the CHD and ACDC datasets is exhibited in Table I0 We adhere strictly to the divisions described in the Positional Coding Learning (PCL) study 172,, ensuring consistency and reliable comparison results.
Table 10: The complete five-fold Dice results for CHD and ACDC.
Dataset
Sample
Fold 1
Fold
Fold
Fold
Fold
Mean/Std
CHD
0.3085
0.3292
0.4649
0.4405
0.4178
0.392(.622)
0.5370
0.6527
0.6707
0.7076
0.6119
0.636(.058)
10
0.6382
0.6797
0.7252
0.7326
0.6900
0.693(003)
15
0,6668
0.6892
0.7519
0.7458
0.6844
0.712(.035)
20
0.6738
0.7204
0.7629
0.7766
0.7051
0.728(038)
30
0.7291
0.7324
0.8001
0.8014
0.7093
0.754(.39)
51
0.7385
0.7594
0,8148
0.8234
0.8048
0.788(.033)
ACDC
0.7975
0.7027
0.7510
0.7097
0.7458
0.741,.034)
0.8827
0,8941
0,8620
0.8596
0.8682
0.873(013)
10
0.9101
0.9086
0.8919
0.8709
0.8914
0.895(014)
15
0.9175
0.9076
0.9140
0.8932
0.9091
0.908,008
20
0,9173
0.9152
0.9168
0.9101
0.9143
0.915(.003)
30
0.9224
0,9232
0,9252
0.9162
0.9187
0,921(.003)
80
0.9313
0.9285
0.9336
0.9255
0.9328
0.930(003)
Significance Tests We include statistical test and report p-values between results of our method versus each of the other methods in Table [I (each scalar indicates the p-value between our final method against others). We use the average dice score per slice to estimate the p-values. All values indicate that there is significant difference between our method and the existing methods.
Table 11: Significance tests.
CHD (M=15)
ACDC (M=6)
Unet random init.
52
0.001
0.001
Ours w/ random init.
0.001
0..01
GLCL-full pretrain
0.001
0.001
PCL pretrain
72
0.001
0.001
19

---

B Additional Implementation Details
Benchmark Results To acquire the benchmark results, we use the paper provided results whenever
possible. In Table results of UNet [52), Rotation [8]. PIRL [45, SimCLR [9, CLCL-global 7
PCL [72 are obtained from 772. and we reproduce the remaining results including Swin-Unet [5].
SwinUNETR [58], GLCL-local
77. CAiD [56] by training the model provided in their official reposi-
tories.
Keypoint Preprocessing Details. We compute the keypoint positions on the original image using SIFT. To obtain the keypoint positions augmented images, we propose three potential solutions: (1) Extract the translation matrix from the augmented image and apply this translation to the keypoint positions. (2) Recompute SIFT on the augmented image to determine the new keypoint positions. (3) Consider each keypoint position as a label, and augment these labels alongside the original image.
The first solution involves retrieving the translation matrix from the augmentation package, which is not easily accessible in popular data augmentation packages currently available. Moreover, this approach requires careful handling of the cropping of the translated keypoints. The second solution poses the challenge of translating the correspondence from the original image pair to the augmented image pair. The third solution, on the other hand, retains the index of the keypoints, thus preserving the correspondence from the original to the augmented image. Consequently, we choose the third solution as our preferred approach.
To claborate further, we initially assign a unique index to each detected keypoint. Subsequently, this index is attributed to the image space, resulting in a 2D matrix. In our implementation. the background is designated as 0, while the keypoint index starts from 1. This matrix is then transformed concurrently with the original image, resulting in translated keypoint index matrix. We further process the keypoints under the following two circumstances: (1) If a keypoint is present in the original image gets cropped out, simply disregard that keypoint. (2) The keypoints might project onto one or multiple nearby positions. In this scenario, we compute the mean of all these positions, and this average becomes the new keypoint position in the augmented image.
C Additional Results
Keypoints Detection Results. In Fig. 6nddd 7 we present the input images and detected keypoints using the Scale-Invariant Feature Transform (SIFT) [4T]. The results suggest that the majority of the detected keypoints are located in the foreground rather than the background. This helps the KAF layer to concentrate more on the regions that are crucial for segmentation tasks. Furthermore, we display the transformed keypoint positions after performing data augmentation on the input image in the third column. As described the preprocessing details in Sec. [B during the augmentation, we transform the keypoint positions from the original image to the augmented one with the transformation matrix.
we provide a visual representation of the detected image correspondence. The results indicate that our heuristic distance metrics effectively identify correspondences between neighboring slices in biomedical images. During data augmentation, rather than recomputing the correspondences between the augmented slices, we translate the found correspondences from the original slices the augmented ones, similar as the keypoint processing above.
Keypoint Correspondence on Images. In columns (b) and (d) of Fig. and Fig. 7
Additional Self-attention Map. In Fig. g we provide visualization of the self-similarity maps from two neighboring slices derived from various layers of our UNet. Similar to Fig. 3 in the main text, the similarity map of our method demonstrates better resilience augmentations and maintains localized consistency among keypoints, compared to other pretrained models such as PCL and GLCL 77].
Additional Segmentation Results. To supplement Sec. we present additional segmentation results comparing our method with other baselines in Fig. 9 Models trained and/or finetuned with different numbers of subjects from both ACDC and CHD datasets present.
20

---

(c) augmented
(d) augmented correspondence
(a) input
(b) correspondence
keypoints
Figure 6: Visualizatin of the detected keypoints and their correspondence on images sampled from CHD dataset. The green dots represent the keypoints detected by SIFT. column (a), illustrate two adjacent slices. In column (b), we showcase the correspondence between these two slices, applying the heuristic described in section 3.2 from the main text. For the actual training, we apply data augmentation into the input image, and examples are shown in (c) with the augmented keypoints.
We also display the transferred correspondence in column (d). Please zoom to view the details.
21

---

(c) augmented keypoints
(d) augmented
(a) input
(b) correspondence
correspondence
Figure 7: Visualization of slices, keypoints, and correspondences on the ACDC dataset.
22

---

Self-similarity
Self-similarity 1/8 resolution
1/4 resolution
(a)
(c)
(d)
T,(x,)
PCL
GLCL
T2(X2)
Ours
((F2(T1(x)) (T2(x)
((F)(T1(x))) ((F2(T2(x2)
Query point
F: feature map from the i-th layer output
Ti random data augmentation
Self-siimlarity between the query point and the image
Figure Learned self-similarity from two adjacent slices. Each map indicates the feature similarity between the query point feature (star) and other points within the image. We display similarity at two different scales within the UNet encoder. The comparison between (a) and (b), (c) and (d) indicates that ours is more resilient in maintaining the self-similarity of the features under various transformations.
23

---

Random Initialiizatio
SSL Initialization
Input
(a) UNet
(b) SwinUNETR
(c) Ours
(c) GI Cl
(d) PCL.
(e)Ours
(f) Reference
43.89
16:53
76:43
80:13
67.69
88.63
ACDC (222
22:31
51:13
58
71.96
71:03
72:66
72.07
55.49
34.35
48.64
75.27
81.39
82.01
83.21
25. 16
28 63
32.13
28.28
28.29
34 92
CHD (M=15)
22.66
27.29
31.29
25.54
28.36
31.74
26.59
37.
43.64
43.00
43.59
43.92
26.59
37.
43.64
43.00
43.59
43.55 92
Figure Additional segmentation results to supplement Fig. 2 in the main text.
24


Reviews:
[{"review_id": "F6V8aweJJL", "reviewer": "Reviewer_b18J", "summary": "The paper presents a self-supervised learning approach for medical image segmentation that uses matching keypoints to contrast local features that may be spatially distant from one another. The proposed model pretrains a segmentation network on unlabelled images, using three contrastive losses: a global loss on the keypoint-enhanced features, a global loss on pooled features in the last layer (as in PCL) and a local loss on features of matching keypoints (which are spatially close and have a high matching probability). This model exploits Keypoint-Augmented Fusion (KAF) layers which transforms the features of detected keypoints using self-attention transformer layers and then concatenates the transformed features to the original feature map (after constructing a sparse map that is diffused to a dense map via CNN layers). The SuperGlue network of Sarlin et al. is employed to compute the keypoints and their matching scores. The proposed approach is evaluated on two cardiac MRI segmentation datasets (ACDC and CHD) for which it shows superior performance compared to recent self-supervised approaches. \n\n", "questions": "* What is the threshold used to match keypoints?How sensitive is the method to the number of matching keypoints? \n\n* Why are the results reported for GLCL-full lower than those of the original paper?\n\n* Why not test the method on other segmentation tasks? For example, the GLCL paper by Chaitanya et al. also test on prostate segmentation.\n\n* In the ablation study (Table 2), why not test the case where w1=w2=0? This would be useful since the main novelty of the method lies in L_local.\n\n* In eq (2), does the index N+1 correspond to a no-match class?\n\nOther comments:\n\n* p5: \"based on the cloest L2 distance\" --> closest", "limitations": "Some limitations are mentioned in the discussion.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The idea of using keypoints in a local contrastive loss for self-supervised pre-training is novel (to my knowledge) and interesting. Although it relies on the accurate detection and matching of keypoints, it has the potential to overcome the limitations of current strategies, for instance, based on spatial proximity and transformation consistency (which can not be used across different images). \n\n* The method is sound and well presented.\n\n* Experiments are detailed and results show clear improvements over recent self-supervised approaches for segmentation in few shot settings. Likewise, ablation studies and visualization experiments are nicely done and showcase the proposed method.\n\n* The paper is quite well written.\n", "weaknesses": "* While the proposed method is compared against recent self-supervised approaches, only one of these approaches (GLCL) tries to learn a local representation, which makes the comparison a bit unfair. Yet, several approaches have been proposed for this purpose, for instance, see references below. As I understand, these approaches can also contrast local features that are semantically related (not necessarily spatially related), for example, based on clustering as in Peng et al, pseudo-labels as in Zhong et al., or super-pixels as in Chaitanya et al.\n\nWang, Zhaoqing, Qiang Li, Guoxin Zhang, Pengfei Wan, Wen Zheng, Nannan Wang, Mingming Gong, and Tongliang Liu. \"Exploring set similarity for dense self-supervised representation learning.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16590-16599. 2022.\n\nPeng, Jizong, Marco Pedersoli, and Christian Desrosiers. \"Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularization.\" Machine Learning for Biomedical Imaging 1, no. MIDL 2020 special issue (2021): 1-29.\n\nZhong, Yuanyi, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. \"Pixel contrastive-consistent semi-supervised semantic segmentation.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7273-7282. 2021.\n\nChaitanya, Krishna, Ertunc Erdil, Neerav Karani, and Ender Konukoglu. \"Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation.\" Medical Image Analysis 87 (2023): 102792.\n\n* Some of the reported results are not consistent with the literature. Namely, Table 1 reports a Dice of 0.642 for GLCL-full with M=2 in ACDC whereas the original paper reports a Dice of 0.789, which is higher than results of the proposed method. \n\n* A potential weakness of the proposed method is that it requires to find matching local features across different images. Typically, descriptors like SIFT focus on points corresponding to sharp edges or blobs, which are present in cardiac structures. I suspect such points would be harder to find in structures with lower contrast such as prostate in MRI. Thus, it would be necessary to test the method on a broader range of organs and even other modalities like CT.   \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Comparison with other local SSL methods.\n\nThank you for offering additional literature. We will add them in our paper.\n\nTo clarify the difference between our method and `Peng et al., Zhong et al.`, `Chaitanya et al.`: while they incorporate local losses, their frameworks are designed for **semi**-supervised segmentation purpose, which requires a set of labeled dataset for joint training. Our method is designed for more general purpose **self**-supervised representation learning, which can be further finetuned on a small set of annotations. We agree that the local losses in these frameworks may be applicable for self-supervised training, but properly repurposing them may require further investigation and remain interesting future directions. `Wang el al.` is relevant to our approaches and they establish the local set correspondence between two different views of an input image. Unfortunately, we were unable to find its public source code, and reproducing the method (originally designed for natural images) on medical images requires additional consideration. We will include the comparison with `Wang et al.` if the authors open source their code in the future.\n\nYet, for more comprehensive benchmarking purpose, we now add an additional *self-supervised* local SSL method as suggested by Reviewer `PaeV`: \"CAiD: Context-Aware Instance Discrimination for Self-supervised Learning in Medical Imaging\".\nResults are reported below. Our method outperforms CAiD in both datasets under different numbers of finetuning subjects (M). A full comparison will be attached into the paper.\n\n| | CHD | | ACDC | |\n|------------|:---:|------------|:----:|------------|\n| SSL Method | M=2 | M=15 | M=2 | M=6 |\n| *CAiD* | 0.265(.08) | 0.684(.04) | 0.483(.11) | 0.822(.02) |\n| Ours | **0.392(.06)** | **0.712(.03**) | **0.741(.03)** | **0.873(.01)** |\n\n>  Inconsistent Dice between GLCL-full and original paper\n\nThank you for pointing this out! We believe this is a miscommunication on our part. In the original GLCL [1] paper, experiments are conducted based on a single fold train/test split, and results are reported on a held-out test set. In our experiment, we followed the more recent PCL [2]paper and conducted 5-fold cross validation. Table 1 indicates an average dice across the 5 folds. Note that our reported result of the GLCL-global is consistent with the GCL results reported in the PCL paper Table 1 (where they adopted the global loss of the GLCL). We will clarify the evaluation differences in our paper.\n\n[1] Chaitanya et al., \"Contrastive learning of global and local features for medical image segmentation with limited annotations\"\n[2] Zeng et al., \"Positional Contrastive Learning for Volumetric Medical Image Segmentation\"\n\n> Test the method on other organs and other modalities\n\nThanks for your suggestion! We now add an additional multi-organ CT dataset Synapse [1], and the comparison with SoTA methods are below. Consistent with ACDC and CHD, our framework trained from randomly initialized weights or pretrained with SSL loss outperformed existing methods. We will supplement the full comparison in the paper.\n\n| Init. | Method | Dice (M=2) |\n|--------------|----------------------|------------|\n| Random Init. | Unet | 0.253(.06) |\n| | Swin-Unet | 0.198(.04) |\n| | SwinUNETR | 0.279(.06) |\n| | Ours | 0.289(.06) |\n| SSL pretrain | PCL | 0.306(.05) |\n| | Swin-Unet (with PCL) | 0.210(.07) |\n| | Ours | 0.322(.06) |\n\n[1] Synapse dataset: https://www.synapse.org/#!Synapse:syn3193805/wiki/217785\n\n> Threshold used to match keypoints and sensitivity\n\nThank you. We will specify the threshold. We used 20 in the paper to find the correspondence. To verify how sensitive the method is to the number of matching keypoints, we perform an ablation on different threshold values, to pretrain our model on CHD, before finetuning it on $M=15$ labeled data. We turned off the global losses ($w_1=w_2=0$) to isolate the effects of the threshold for the local SSL loss. Average dice scores are reported below. As indicated from the results, our model remains robust across different values and the number of matching keypoints. We also include a visualization in the attached pdf with the number of matching keypoints annotated under different threshold values.\n\n| Threshold | Dice |\n|:---------:|:-------------:|\n| 5 | 0.689 (0.043) |\n| 10 | 0.690 (0.031) |\n| 15 | 0.684 (0.035) |\n| 20 | 0.689 (0.036) |\n| 25 | 0.684 (0.032) |\n| 30 | 0.685 (0.033) |\n| 35 | 0.690 (0.037) |\n| 40 | 0.687 (0.037) |\n\n> w1=w2=0 ablation\n\nThank you. We have reported the results of $w_1=w_2=0$ and more ablations on the weight terms in our *appendix A. \"Ablation on Correspondence Weights\"*. Empirically, we find that only using the local loss for pretraining outperforms a randomly initialized UNet, which indicates that enforcing the localized representation across different slices offers a better initialization. However, a combined global and local loss during pretraining further increased the performance.\nThis is also a consistent observation as in [1] and [2], where it is necessary that the local loss is combined with a global loss and/or other regularizations to avoid degenerated representations.\n\n[1] Ren et al., \"Local Spatiotemporal Representation Learning for Longitudinally-consistent Neuroimage Analysis\",\n[2] Chaitanya et al., \"Contrastive learning of global and local features for medical image segmentation with limited annotations\".\n\n> In eq (2), index N+1\n\nYes. We follow [1] for the annotation. We will clarify this in the paper.\n[1] Sarlin et al., \"SuperGlue: Learning Feature Matching with Graph Neural Networks\"\n"}, {"review_id": "mNZvjXN7GJ", "reviewer": "Reviewer_9qHo", "summary": "The authors introduce an approach to improve the performance of the UNet model, specifically in the task of segmentation. They propose a keypoint augmented fusion layer to enhance the CNN-based encoder of UNet. This layer leverages self-attention to capture long-range spatial dependencies among keypoints extracted from multi-scale feature maps. In addition, the paper suggests the use of global and local contrastive losses to pretrain the model in a self-supervised manner. These losses help the model learn meaningful representations from unlabeled data, improving its performance. The proposed framework was evaluated on Cardiac CT and MRI datasets for segmentation tasks. The results demonstrate reasonable improvements over the state-of-the-art methods, particularly in scenarios with limited labeled data (few-shot settings).", "questions": "1) Keypoint Robustness - The manner of application of global contrastive loss depends on the effectiveness of keypoints. Because the other global loss is used off-the-shelf from [54] as PCL loss.\n\n2) As for local loss, the contribution appears to be more engineering (lines 175-200) i.e on how to use it among keypoints in adjacent slices. Here also, there is no ablation on the multiple thresholds selected in the form of spatial distance between keypoints or number of adjacent slices. \n\n3) Standard non-cardiac benchmark datasets like BTCV, SegTHOR have not been used. To establish it as a generic method, the authors\nshould definitely explore non-cardiac pan-organ datasets\n\n4) I also fail to understand what customization the authors have proposed in their architecture to make it work in few-shot settings with limited annotations. \n\n5) Can the authors please mention the unlabeled datasets used for SSL pretraining?", "limitations": "Not adequately addressed in the 'Discussion' section", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) This work addresses limitations of traditional UNet models, which rely on convolutional neural networks (CNNs) and may struggle to capture long-range dependencies among spatial positions in an image. In this study, the authors propose a more effective approach to handle this issue. At each scale of the encoder, the authors employ Scale-Invariant Feature Transform (SIFT) to detect key points within the CNN feature maps. By applying SIFT, the authors aim to identify salient locations within the feature maps that can capture important spatial information. To facilitate correlation and interaction among these detected key points, a transformer model is employed. \n\n2) Global contrastive loss is applied in an unique way on two types of representations - concatenated keypoint enhanced feature maps, as well as features  from last layer of UNet. \n\n3) instead of being used as a traditional image-wise similarity loss, here the local contrastive loss is proposed to exploit pixel-wise similarity (by measuring the correlation between keypoints within a spatial distance among adjacent slices)", "weaknesses": "1) It is unclear why this work ignores Transformer-based UNet architectures (TransBTS [1], UNetr[2], Swin-Unet [3], etc) which employ self-attention to better capture the correlations. Though the authors propose to use it on keypoints, I find it difficult to comprehend the advantages here, except maybe computational efficiency. There are many works which do it on entire feature maps - why are those suboptimal?\n\n2) Handcrafted features like SIFT/SURF to detect keypoints, though powerful, aren't the current SOTA. The performance can definitely be tested using finetuned object detection models (such as Faster RCNN [4])\n\nIn short - there should be ablation showing (1) advantage of using keypoints, (2) Different keypoint\ndetection strategies, and most importantly (3) comparison with more recent Transformer-based UNet architectures.\n\n[1] TransBTS: Multimodal Brain Tumor Segmentation Using Transformer\n[2] UNETR: Transformers for 3D Medical Image Segmentation\n[3] Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation\n[4] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Comparison with transformer-based UNets.\n\nThank you for your suggestions!\nWe now added two recent SoTA transformer-based unets including Swin-Unet[1] and Swin-UNETR[2]. To illustrate the effects of architectural changes, we train all models with random weight initialization, under limited annotation (number of training subjects $M=2$). Five-fold cross validation is performed on different datasets, and the average dice score with standard deviation is reported below. When trained with limited labels, Unet still outperformed transformer based architecture (except for Swin-Unet on CHD), and our method achieved the best results across the board. We note that our observation is consistent with the conclusion in [1], where transformers may be more severely affected by the initialization, and may benefit from large-scale pretraining. E.g., on Synapse, we observed an increased dice of SwinUNETR from 0.198(.04) to 0.210(.07) when further pretraining it with PCL. \n\n| | CHD | ACDC | Synapse |\n|--------------|------------|------------|------------|\n| Swin-Unet[1] | 0.236(.09) | 0.327(.10) | 0.198(.04) |\n| SwinUNETR[2] | 0.137(.09) | 0.501(.03) | 0.279(.06) |\n| Unet | 0.184(.06) | 0.588(.07) | 0.253(.06) |\n| Ours | 0.344(.05) | 0.655(.05) | 0.289(.06) |\n\n\n[1] Cao et al., \"Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation\".\n[2] Hatamizadeh et al., \"Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images\".\n\n\n\n\n> The advantage of using keypoints? The keypoint robustness? Other keypoint detection methods?\n\nThank you for the questions. We will try to clarify our intuition with additional experiments, to hopefully address the concerns.\n\n\n**Why use keypoints?**\n\nOur rationale for using keypoints is rooted in enhancing the conventional CNN layers with long-range dependencies, a characteristic often encountered in medical images. While the attention in transformers facilitates the learning of such long-range dependencies, it operates among all pairs of feature points, potentially resulting in both computational expense and a deficiency in targeting crucial regions. Empirically, our formulation leads to much better segmentation performance compared with CNN-only UNet and transformer-based UNet, while being less computational intensive.\n\n\n**Other keypoint detection methods?**\nWe agree. The keypoint detection can be replaced with other learning based SoTA methods as well. Given that finetuning Faster RCNN model for keypoint detection requires groundtruth labels (we do not have access to), we used a pretrained keypoint detection model Superpoint [1] as an alternative to SIFT. Segmentation results on CHD and ACDC dataset are reported below comparing two methods, and the results indicate that our method is not sensitive to different keypoint detection algorithms. Overall, Superpoint leads to slightly lower results, and we speculate this was due to the pretraining was done on natural images (COCO dataset).\n\n\n| Init. | | CHD | ACDC |\n|----------------|---------------------|--------------|-------------|\n| Random Init. | SuperPoint | 0.643(.04) |0.810(.02) |\n| | SIFT | 0.686(.03) |0.827(.05) |\n| SSL Pretrain | SuperPoint | 0.703(.03) |0.865(.02) |\n| | SIFT Pretrain | 0.712(.03) |0.873(.01) |\n\n\n[1] DeTone et al., \"SuperPoint: Self-Supervised Interest Point Detection and Description\".\n\n\n\n\n**Keypoint Robustness - No ablation on the thresholds.**\nTo verify how sensitive the method is to the number of matching keypoints, we perform an ablation on different threshold values, to pretrain our model on the CHD dataset, and finetune it on $M=15$ labeled data. Global losses ($w_1=w_2=0$) are turned off to isolate the effects of the threshold for the local SSL loss. The dice scores are reported below, indicating that our model is not sensitive to the threshold setting and remains robust across different values. We also include a visualization in the attached pdf with the number of matching keypoints annotated under different threshold values.\n\n\n| Threshold | Dice |\n|:---------:|:-------------:|\n| 5 | 0.689 (0.043) |\n| 10 | 0.690 (0.031) |\n| 15 | 0.684 (0.035) |\n| 20 | 0.689 (0.036) |\n| 25 | 0.684 (0.032) |\n| 30 | 0.685 (0.033) |\n| 35 | 0.690 (0.037) |\n| 40 | 0.687 (0.037) |\n\n\n\n\n> Non-cardiac pan-organ datasets\n\n\nWe now add a multi-organ CT segmentation dataset to evaluate the generalization of our method. Results are reported below. With 2 training subjects, we tested the performance of the model under both random and SSL pretrained initialization. In both scenarios, our method outperformed the existing works. We will supplement the remaining methods and different number of training subjects in the revised manuscript.\n\n\n| Init. | Method | Dice (M=2) |\n|--------------|----------------------|------------|\n| Random Init. | Unet | 0.253(.06) |\n| | Swin-Unet | 0.198(.04) |\n| | SwinUNETR |0.279(.06) |\n| | Ours | 0.289(.06) |\n| SSL pretrain | PCL | 0.306(.05) |\n| | Swin-Unet (with PCL) | 0.210(.07) |\n| | Ours | 0.322(.06) |\n\n\n\n\n> What customization in the architecture makes it work in few-shot settings? \n\nOur architectural customization involves integrating the KAF layer into every CNN block of the UNet. Leveraging the interplay among the features extracted from identified keypoints, this modification empowers the model to more effectively gather local information. Consequently, it facilitates enhanced representation learning even when working with limited data. These advantages have been substantiated through comprehensive empirical experiments conducted across diverse datasets.\n\n\n> The unlabeled datasets used for pretraining?\n\n\nWe follow common practice of medical SSL and use all unlabeled volumes on each dataset for our pretraining. On ACDC, we use all unlabeled volumes (100 patients, and each of them has ~15 volumes)l. On CHD, we use a total of 68 cardiac images.\n\n"}, {"review_id": "n8E7Ks8yal", "reviewer": "Reviewer_m84B", "summary": "SIFT keypoints are used in conjunction with a UNET to construct a transformer-like model for medical image segmentation. Global and local self supervised learning (SSL) losses are proposed. Correspondences identified using the SuperGlue method. Various experiments on two cardiac image datasets, including ablation studies, shows improves on other work in terms of segmentation dice score.\n\n", "questions": "- It would be good to hear a discussion regarding 3D keypoint methods, and how much this method would improve upon simple keypoint transfer segmentation as in [46]. \n\n- In lines 208 and 209 it is stated that images are pre-processed using the methodology described in [54]. Could the preprocessing stage be briefly described in the paper? Are there aspects without which this approach would not work, e.g. registration of training data?\n- Could the authors explain why the Manhattan distance is preferred over other distance measures for keypoint filtering (as mentioned in line 181)?\n- Would it be possible for the authors to provide the values of the similarity index in Figure 2? This would allow us to compare those values with the ones presented in the previous Table 1.\n- Can the authors provide more insight into the justification for the following claim? Clearly there are various random transforms which would scramble image information, to which SIFT is not invariant.\n\"(2) our proposed self-supervision helps maintain better local equivariance of the self-attention, i.e., with the  same query point location, its interaction with other points remains identical no matter how the image is transformed.\" \n", "limitations": "Several limitations exist: missing important details, computational complexity, description of data, transformer parameters (learning rate, batch size, attention head size, and input embedding size), statistical significance and comparison to based keypoint transfer segmentation.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The use of traditional keypoints as attention operator mechanism for UNet segmentation makes sense and is novel, to my knowledge.\n\nImproved segmentation results convince the reader that the approach has practical value.", "weaknesses": "The method presented is based on 2D keypoint matching, issues with multiple 2D slices, wheras anatomy is 3D. It would be interesting to know if 3D keypoints would improve results.\n\nTable 1 should mention that the numbers shown are dice scores.\n\nMinor spelling\nLine 182 “based on the cloest L2 distance” -> “closest”\nLine 213 “patent ages ranging from 1 month to 21 year” -> “21 years”\n\nIt would be interesting to know much the learning method here improves upon simple keypoint transfer segmentation as in [46]. \n\nObjective functions in equations (1) and (2) appear very similar to other recent keypoint matching work:\nChauvin, Laurent, et al. \"Efficient pairwise neuroimage analysis using the soft jaccard index and 3d keypoint sets.\" IEEE transactions on medical imaging 41.4 (2021): 836-845.\n\nThe description of the datasets used does not mention the size and resolution of the images. \n\nComputational complexity of the proposed method is not mentioned. \n\nThe paper lacks reproducibility. For instance, line 156 and line 181 allude to thresholds whose numerical values are not explicitly mentioned. Similarly, it is not clear what setup was used to train the Transformer block, including details such as learning rate, batch size, attention head size, and input embedding size.\n\nAdditionally, it would be valuable to include statistical tests to demonstrate the superiority of the proposed method compared to existing approaches.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> It would be interesting to know if 3D keypoints would improve results.\n\nThank you for the insightful point! As discussed in our limitations section, a promising future avenue is expanding our model to 3D. This direction, however, poses unique challenges. Firstly, 3D keypoint detection lacks established algorithms. Moreover, the inherent sparsity of 3D data adds complexity, making the transfer of features from sparse keypoint maps to dense feature maps a formidable task. Modifying this module presents an unexplored area requiring careful investigation.\nFurthermore, the increase in keypoints from $O(x^2)$ to $O(x^3)$ brings about the necessity for a novel sampling strategy to address memory constraints. While deploying our method on 3D data holds potential, we acknowledge the substantial learning curve and the need for extensive experimentation in this domain.\n\n> It would be interesting to know how much the learning method here improves upon simple keypoint transfer segmentation as in [46].\n\nWe fully agree that computing a registration based on keypoints and then transforming the segmentation task to the target image is an interesting idea when annotations are limited. However, we note that extending [46] for our task requires extensive research efforts that leads to a different project. Crucially, [46] registers key points across different subjects in order to transfer training segmentation during test time, whereas in our method, keypoint correspondence is detected across neighboring slices within the same subject, which is much simpler and trivial. Moreover, [46] uses keypoints for segmentation transfer, while our method utilizes keypoints for more general-purpose representation learning. Again, we acknowledge that comparing or even extending [46] under our task will be a promising future research direction.  \n\n> Objective functions in equations (1) and (2) appear very similar to other recent keypoint matching work.\n\nYes, we agree. In our approach, eq (1) is an established global contrastive loss, and the distinction of our method with others lies in the features used to compute similarity. Specifically, the global feature is now computed from our KAF layer in eq(1). Similarly, we follow SuperGlue in eq(2) for aligning the feature space between the matched correspondence features. \n\n[1] Sarlin et al.,  SuperGlue: Learning Feature Matching With Graph Neural Networks\n\n> The size and resolution of the images.\n\nWe mentioned the resolution of the image size in the caption of Figure 1 in our appendix, but we will make it more explicit in the main text.\n\n> Computational complexity.\n\nPlease kindly refer to our appendix A \"Computational analysis\".\n\n> Reproducibility: Threshold values in L156 and L181, learning rate, batch size, etc.\n\nThank you for raising the issues. We included learning rate, batch size, optimizers in L226 – L240, but we are happy to add more relevant details in the manuscript. Our code will be released for better reproducibility.\n\n> Statistical tests.\n\nThanks for the valuable suggestion. We include a statistical test and report p-values between results of our method versus each of the other methods below (each scalar indicates the p-value between our final method against others). We use the average dice score per slice to estimate the p-values. All values indicate that there is a significant difference between our method and the existing methods.\n\n| | CHD (M=15) | ACDC (M=6) |\n|----------------------|------------|------------|\n| Unet w/ random init. | < 0.001 | << 0.001 |\n| Ours w/ random init. | << 0.001 | << 0.001 |\n| GLCL-full pretrain | << 0.001 | << 0.001 |\n| PCL pretrain | << 0.001 | << 0.001 |\n\n> In lines 208 and 209 it is stated that images are pre-processed using the methodology described in [54]. Could the preprocessing stage be briefly described in the paper? Are there aspects without which this approach would not work, e.g. registration of training data?\n\nWe will add a description on the preprocessing. It mainly included intensity normalization and resampling, which is a commonly adopted preprocessing step in medical images. Further, our model does not assume registered training images and thus won’t be affected by such pre-processing.\n\n> Why is the Manhattan distance preferred over other measures?\n\nWe empirically used Manhattan distance as the criteria inspired by the PCL paper, but it is likely that other measures can also work, as long as the correspondence can be reasonably determined. This assumption was also supported by our ablation study for `9qHo` using different thresholds, where the number of matched correspondences and different threshold values had minimum effects on the final results.\n\n> Adding values of the similarity index in Figure 2.\n\nWe now added the corresponding Dice (scaled by 100)  into the revised figure attached. We observed that these individual values are consistent with the average scores in Table 1.\n\n> Insights on (1) SIFT is not invariant. (2) “our proposed self-supervision helps maintain better local equivariance of the self-attention.\"\n\nWe fully agree that SIFT may not be robust when strong image augmentations are applied on the images, and this is precisely the reason why we first detect keypoints on the **original** images before any augmentations are applied, and this makes sure that the keypoints are consistent under random augmentations. Further, the keypoint correspondences are also matched on the original images, and our local loss further encourages that the features between corresponding points remain similar even under strong augmentations. Such formulation ensures that our model is more robust in maintaining the identical interaction between keypoints under different image transformations, as shown in main text Figure 3 and appendix Figure 2.\n\n"}, {"review_id": "utpNPFYErx", "reviewer": "Reviewer_8g4p", "summary": "This paper focuses on improving medical image segmentation performance with limited labeled data. They (1) propose a modification to the standard UNet convolutional architecture by incorporating local features (at locations identified by SIFT) into the backbone, (2) show how their modification can be used to control self-supervised contrastive learning positive and negative pair selection, and (3) evaluate their proposed architectural change and self-supervised training strategy against relevant baselines. They find both the architectural change and self-supervised strategy improve segmentation performance. ", "questions": "My questions and suggestions are covered in the Weaknesses section.", "limitations": "Limitations and impact adequately discussed. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel architectural change that impacts network performance both with random initializations and with self-supervised contrastive learning. \n- Improves performance against relevant and recent self-supervised baselines. \n", "weaknesses": "- The GLCL baseline reported in Table 1 is much lower than the numbers reported in the GLCL original paper; the original GLCL paper actually reports performance better than the proposed method on the ACDC dataset. Please comment on why the reported GLCL performance in 14-18 points lower than the GLCL paper.\n- The proposed architectural change may increase compute requirements, but this is not discussed nor are any timing/computational load experiments reported.\n- There is limited dataset variety—both evaluation datasets are 3D cardiac anatomical segmentation tasks. Additional experiments, particularly including pathological segmentation, would strengthen the results and show the proposed method works for different segmentation targets.\n- Scaling the proposed solution to non-CNN architectures or 3d data may be difficult (as mentioned by the authors).\n- Minor: Table 1 caption needs to specify what metric you are reporting; Line 221 has a period where there should be a comma+”and.”\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The GLCL baseline reported in Table 1 is much lower than the numbers reported in the GLCL original paper; the original GLCL paper actually reports performance better than the proposed method on the ACDC dataset. Please comment on why the reported GLCL performance in 14-18 points lower than the GLCL paper.\n\n\n\n\nThank you for pointing this out! We believe this is a miscommunication on our part and we will clarify in our paper. In the original GLCL paper, experiments are conducted based on a single fold train/test split, and results are reported on a single held-out test set. In our experiment, we followed the PCL paper and conducted 5-fold cross validation. Results in Table 1 indicate an average dice score across the 5 folds. Note that our reported result of the GLCL-global is consistent with the GCL results reported in the PCL paper (where they adopted the global loss of the GLCL). We additionally performed 5 folds evaluation with the full losses and results are shown in Table 1.\n\n\n\n\n\n\n\n\n> The proposed architectural change may increase compute requirements, but this is not discussed nor are any timing/computational load experiments reported.\n\n\n\n\nThank you for your suggestion! Due to the page limit, we discussed the computational load in the supplementary materials. Please kindly refer to appendix A \"Computational analysis\" for more information.\nIn summary, our method demonstrates a slower growth rate in both memory usage and GFLOPs compared with transformer-based unet (i.e. SwinUNETR).\n\n\n\n\n> Limited dataset variety.\n\n\nWe agree. We now include results on an additional CT dataset for multi-organ segmentation. Results are reported in the Table below. With 2 training subjects, we tested the performance of the model under both random weight initialization and SSL pretrained initialization. In both scenarios, our method consistently outperformed the existing works. Due to the time constraint, we will supplement the remaining methods and different number of training subjects in the revised manuscript.\n\n\n| Init. | Method | Dice (M=2) |\n|--------------|----------------------|------------|\n| Random Init. | Unet | 0.253(.06) |\n| | Swin-Unet | 0.198(.04) |\n| | SwinUNETR | 0.279(.06) |\n| | Ours | 0.289(.06) |\n| SSL pretrain | PCL | 0.306(.05) |\n| | Swin-Unet (with PCL) | 0.210(.07) |\n| | Ours | 0.322(.06) |\n\n\n\n\n\n\n> Scaling the proposed solution to non-CNN architectures or 3d data may be difficult (as mentioned by the authors).\n\n\n\nThank you for the comment. It is worth mentioning that the current scope of our work is to enable the long-range dependencies into the CNN based segmentation architectures, and our results on the 2D medical image segmentation indicates the effectiveness of the proposed keypoint-based formulation. Our proposed KAF layer is generic, which can also be incorporated into recent transformer-based Unet architectures that integrate both transformer blocks and the convolutional blocks. We fully agree that such architectural modification or scaling up to 3D may require different consideration and further investigation, but these directions are interesting future works.\n\n\n\n\n> Table 1 and Line 221\n\n\n\n\nThanks for pointing that out, and we will mention that we use Dice as a metric and report its computed mean and standard deviation in Table 1.\nFor Line 221, we will change the dot to period. Thanks!\n\n"}, {"review_id": "0EOjQHKXEj", "reviewer": "Reviewer_PaeV", "summary": "This paper tackles self-supervised learning for medical image segmentation. The main contribution is the keypoint identification in input images in self-supervised pre-training to consider long-range spatial dependencies. Each UNet encoder layer is followed by a keypoint-augmented fusion (KAF) layer, where keypoints are identified by SIFT on the input image and rescaled with respect to the resolution of each layer. Convolutional features at each keypoint are processed by a Vision Transformer to incorporate self-attention, provided to additional convolutional layers for extracting dense features from transformer outputs and finally concatenated with the input feature map. Resulting KAF layer features are used for contrastive training of UNet for both global representation learning similar to PCL [54], as well as local similarity learning via the SuperGlue graph neural network [42].", "questions": "How are the thresholds selected for identifying positive/negative keypoints?\n", "limitations": "limitations and future work are discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Incorporating similarity learning from keypoints is interesting and novel in self-supervised medical image segmentation literature. Paper is well-written and the experiment setup is detailed. Cross-validation is used to compute error bounds in addition to performance metrics. Downstream segmentation performance with limited annotations outperforms several pre-training approaches across two benchmark datasets.   ", "weaknesses": "While contributions are novel, literature review is missing a lot of works on self-supervised medical image segmentation via global/local similarity/contrastive learning:\n\nTaher et al. \"CAiD: Context-Aware Instance Discrimination for Self-supervised Learning in Medical Imaging\" 2022\n\nYan et al. \"SAM: Self-supervised Learning of Pixel-wise Anatomical Embeddings in Radiological Images\" 2021\n\nZheng et al. \"MsVRL: Self-Supervised Multiscale Visual Representation Learning via Cross-Level Consistency for Medical Image Segmentation\" 2023\n\nOuyang et al. \"Self-supervised Learning for Few-shot Medical Image Segmentation\" 2022\n\nFischer et al. \"Self-supervised contrastive learning with random walks for medical image segmentation with limited annotations\" 2023\n\nXie et al. \"PGL: Prior-Guided Local Self-supervised Learning for 3D Medical Image Segmentation\" 2020\n\nWang et al. \"Self-supervised learning based transformer and convolution hybrid network for one-shot organ segmentation\" 2022: also incorporates Vision Transformer for global feature learning. \n\nExperimental comparisons against Wang et al. 2022 would further strengthen the novelty. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n> While contributions are novel, literature review is missing a lot of works on self-supervised medical image segmentation via global/local similarity/contrastive learning.\n> Experimental comparisons against Wang et al. 2022 would further strengthen the novelty.\n\n\nThank you for offering additional literature on self-supervised medical image segmentation! We will include them in our related works. Unfortunately, we were not able to find the publicly available source code for `Wang et al.`, but we are happy to include them as one of the comparisons when the authors release the code in the near future.\n\n\nAmong the list of papers provided, we were only able to find one available (`Taher et al.`) codebase, and we include them as an additional comparison.\nResults on CHD and ACDC dataset are reported below. Our method outperforms CAiD in both datasets under different numbers of subjects used for finetuning (M). A full comparison will be further attached into the revised paper.\n\n\n| | CHD | | ACDC | |\n|------------|:---:|------------|:----:|------------|\n| SSL Method | M=2 | M=15 | M=2 | M=6 |\n| *CAiD* | 0.265(.08) | 0.684(.04) | 0.483(.11) | 0.822(.02) |\n| Ours | **0.392(.06)** | **0.712(.03**) | **0.741(.03)** | **0.873(.01)** |\n\n\n\n\n\n\n> How are the thresholds selected for identifying positive/negative keypoints?\n\n\nAs there are two thresholds involved in our models, we clarify each of them below.\n- The first threshold is applied across different slices to determine the positive/negative slices, and we follow the optimal value found by the PCL paper [1] - 0.1 and 0.35 for CHD and ACDC.\n- The second threshold is applied among keypoint features between two positional positive slices to determine the correspondence among these points. In this step, we do not explicitly recognize any \"negative\" keypoints, as we only use the matched correspondence (\"positive\") in our local loss. By default, we set the threshold as 20. From our ablation on the threshold, and the attached visualization of the correspondence, our method is not sensitive to the threshold values and the number of the matched correspondence.\n\n\n[1] Zeng et al., Positional contrastive learning for volumetric medical image segmentation.\n\n"}]

Author rebuttal:

We thank the reviewers for their time and expert feedback.




As a summary of the reviews, we were happy to see that the submission was found to be well-written [`b18J,PaeV`], and it presents a sound and novel [`b18J,9qHo,m84B,8g4p,PaeV`] method for local self-supervised learning, with well formulated and detailed experiments [`b18J,PaeV`], which shows a significant advance [`b18J, m84B, 8g4p`].


Concerns raised included adding more related work [`b18J,PaeV`], comparisons with more local SSL methods [`b18J`] and transformer-based UNets [`9qHo`], experiments on non-cardiac datasets [`b18J, 8g4p`], ablations on key-point detection [`9qHo`], inconsistent GLCL results with the original paper [`b18J, 8g4p`], and scalability [`m84B,8g4p`].




To these ends, we address individual concerns below and will revise the paper accordingly. A few major responses are summarized as follows:




- As suggested by `b18J, PaeV`, we include one additional local SSL method (CAiD) to benchmark our method.
- As raised by `9qHo`, we now add comparison with SoTA Transformer-based unet architectures including Swin-Unet and Swin-UNETR as our benchmark.
- As mentioned by `b18J, 8g4p`, we now add results on an additional non-cardiac CT dataset (Synapse: the multi-organ segmentation benchmark).
- We clarified the difference between the GLCL results reported in the original paper (single fold train/validation/test) and ours (5-fold cross validation).
- As requested by `9qHo` and `b18J`, we provided more ablation analysis on key-point detection methods, as well as various thresholds for determining the correspondence, to evaluate the sensitivity to the keypoint detection and matching. We also attached an additional figure to visualize the detected correspondence. 
- We revised Figure 1 and attached the Dice score suggested by `m84B`.
- Typos, missing technical details, and related works will be revised in the paper.




Again, we deeply appreciate the feedback and are happy to receive any further questions, comments, and/or suggestions for improvements.



Meta-review: The paper introduces a new self-supervised pre-training mechanism based on keypoints to match spatially distant features for medical image segmentation. The method shows superior results compared to other self-supervised methods on two cardiac MRI segmentation benchmarks (ACDC and CHD). The paper received scores 5,6,6,6,3. The main strengths of the paper are: 1) novelty of using keypoints for self-supervised pre-training; 2) clarity; and 3) thorough experiments. The main weaknesses are: 1) unfair comparison with global representation approaches; 2) inconsistent results with literature; 3) potential disadvantage of the need to detect keypoints; 4) lack of comparison with recent UNet-based methods; 5) lack of complexity analysis; 6) challenging reproducibility; and 7) lack of significance tests. There was a lot of discussion about the paper and rebuttal, where many of issues raised by the reviewers were addressed. The authors should take into account these discussion points when preparing the final version of the paper.
Decision: Accept (poster)