Semi-Supervised Segmentation via Embedding Matching

Authors: weiyi  xie
nathalie  willems
tom  gibbons
daniele  de massari
n  xie
t  lessmann
d d gibbons
  massari
Venue: MIDL_io_2024_Conference
Published: 2024-01-26 22:14:23

Abstract: Deep convolutional neural networks are widely used in medical image segmentation but require many labeled images for training. Annotating three-dimensional medical images is a time-consuming and costly process. To overcome this limitation, we propose a novel semi-supervised segmentation method that leverages mostly unlabeled images and a small set of labeled images in training. 

Our approach involves assessing prediction uncertainty to identify reliable predictions on unlabeled voxels from the teacher model. These voxels serve as pseudo-labels for training the student model. In voxels where the teacher model produces unreliable predictions, pseudo-labeling is carried out based on voxel-wise embedding correspondence using reference voxels from labeled images.

We applied this method to automate hip bone segmentation in CT images, achieving notable results with just 4 CT scans. The proposed approach yielded a Hausdorff distance with 95th percentile (HD95) of 3.30 and IoU of 0.929, surpassing existing methods achieving HD95 (4.07) and IoU (0.927) at their best.

Main paper: 
 introduction: 
 Semantic segmentation is crucial in quantitative medical imaging analysis, such as patientspecific surgical planning for total hip arthroplasty based on accurate segmentation of target structures. With the advent of deep neural networks (Ronneberger et al., 2015; C ¸içek et al., 2016; Milletari et al., 2016; Isensee et al., 2021) , supervised approaches to semantic segmentation have seen significant advancements. However, obtaining large-scale annotated data can be prohibitively expensive in practice. To address this challenge, many attempts have been made towards semi-supervised semantic segmentation (Yu et al., 2019; Wang et al., 2022; Tarvainen and Valpola, 2017; Vu et al., 2019; Zhang et al., 2017; Verma et al., 2022; Chen et al., 2021; Ouali et al., 2020) , which learns from only a few labeled samples and numerous unlabeled ones. Consistency training is a widely-used approach in semi-supervised learning to enforce the model's outputs to be consistent for similar input data points (Tarvainen and Valpola, 2017) . Consistency training methods can be classified into four types, based on where the perturbations are applied. At the input level, uncertainty-aware mean teacher (Yu et al., 2019) , strong-to-weak consistency (Yang et al., 2022a) , and unsupervised data augmentation (Xie et al., 2020) have applied random noises and data augmentation techniques on input images. At the feature level, cross consistency training (CCT) (Ouali et al., 2020) enforces aligned outputs from perturbed and non-perturbed features. At the network level, Luo et al. (Luo et al., 2022) propose aligning the outputs from the CNN and transformer networks since different networks may extract varying information. At the task-level, dual-task consistency (DTC (Luo et al., 2021) ) proposed to align predicted segmentation maps with the predicted distance maps from the same network with two output heads.The other popular concept in semi-supervised segmentation is pseudo-labeling. The underlying idea is to assume that some model predictions are reliable enough to pseudolabel unknown voxels. Since model predictions may not always be reliable, several works use uncertainty (Yu et al., 2019; Wang et al., 2022) or confidence scores (Sohn et al., 2020; Yang et al., 2022b) to discard unreliable model predictions in pseudo-labeling. However, this exclusion leads to an information loss because some voxels in unlabeled images that have not been pseudo labeled are discarded as well, and therefore cannot be used in training supervision. Furthermore these voxels, probably with highly uncertain predictions from a trained network, may be areas critical for object segmentation (e.g., object boundaries). In this paper, we introduce a new strategy to pseudo-label unknown voxels with unreliable model predictions. Our method propagates labels from manual segmentations using voxelwise embedding correspondence. This correspondence is established when the embeddings generated by the student model for the unlabeled voxels match those produced by the teacher model for the labeled voxels. We evaluated this method on a challenging collection of CT images for segmenting hip bones. We demonstrated substantial improvements of our method for semi-supervised hip bone segmentation in comparison with state-of-theart methods (Chen et al., 2019; Ouali et al., 2020; Yu et al., 2019; Tarvainen and Valpola, 2017) . Our contribution manifests in two key aspects: first, the introduction of a novel labelpropagation schema through embedding correspondence; and second, an end-to-end semisupervised segmentation framework facilitating the pseudo-labeling of all regions within unlabeled images. 
overview: 
 Our framework, depicted in Fig. 1 , builds upon the uncertainty-aware mean teacher method (Yu et al., 2019) . We use two 3D U-Net (C ¸içek et al., 2016) models, the student and the teacher network, with identical architectures. To accommodate our computational budget, we reduced the number of convolution filters by half compared with the original 3D U-Net. The student network is trained using stochastic gradient descent, while the teacher model is updated by maintaining a moving average of the student network's weights.During training, our data are partitioned into labeled and unlabeled sets, denoted byD L = {(x i , y i )} N L i=1 and D U = {(x i )} N U i=1, respectively. N L and N U represent the number of labeled and unlabeled training images, respectively. The 3D input image is denoted as x i ∈ R H×W ×D , and the corresponding binary ground truth segmentation as y i ∈ {0, 1} H×W ×D , where foreground objects are hip bones (pelvis and femoral head) manually delineated on CT images (see section 3.1). For simplicity, we leave out i in the following notations. We randomly sample same amount of labeled and unlabeled images for each training batch. For training the student model, the loss computed on labeled images is a combination of cross-entropy and Dice loss. For unlabeled images, the teacher network is used to provide pseudo-labels, as illustrated in Fig. 1 , for computing the semi-supervised loss. To adjust the weight for each loss term, we use a Gaussian ramp-up function similar to (Laine and Aila, 2016) that gradually increases the importance of the semi-supervised loss:ramp up(T, s) = s • exp[-5.0(1.0 -T /T N ) 2 ] (1)Here,' s is a loss-specific scaling factor indicating the value at the ramp-up maturity, T is the current iteration, and the T N is the total number of training iterations.Uncertainty analysis aims to identify reliable predictions (ŷ t ) from the teacher network to train the student network on unlabeled images. Without manual correction, predictions from the teacher model may be inaccurate. To select only reliable predictions, we employ an uncertainty analysis technique based on Monte Carlo Dropout (Kendall and Gal, 2017) , as outlined in (Yu et al., 2019) . In detail, we perform M stochastic forward passes on the teacher model under random dropout and Gaussian input noise to measure predictive entropy for each image x. By averaging the predictive entropy at each forward pass m, we compute the dense uncertainty map H x t as the following:H x t = - M m=1 c∈0,1 p m t (y = c | x) log[p m t (y = c | x)]/M (2)where the subscript t indicates that H x t is measured based on the teacher model's predictions and p t (y = c|x) ∈ R H×W ×D is the dense softmax probability map from the teacher network for class c (c ∈ {0, 1}) given the input image x. The pseudo labels are produced from the teacher model predictions (i.e softmax probability map) as ŷt = argmax c (p t (y = c | x)). We apply the ramp-up procedure (Eq. 1) to compute the threshold for the current iteration T as λ T = [0.75 + ramp up(T, 0.25)] • ln(2) (Kendall and Gal, 2017) to determine which predictions of the teacher can be considered reliable. The final uncertainty analysis loss L U A for input x at training iteration T is the binary cross entropy (BCE) loss:L U A = [1(H x t < λ T ) • BCE(ŷ t , CN N s (x))] 1(H x t < λ T )taking the pseudo label ŷt and the student model's output CN N s (x) as the inputs, where the 1(•) is an indicator function used to mask out the unreliable predictions in the loss computation. Stronger random noise is added to the input to the student network compared to the teacher network, following weak-strong consistency training (Yang et al., 2022a) .Uncertainty analysis involves identifying and excluding unreliable pseudo-labels generated by the teacher network (refer to Section 2.2). However, this can result in losing coverage on many bone surface areas in the unlabeled images, where the model's uncertainty is typically high. Therefore, we propose a novel label propagation technique based on voxelwise embedding correspondence to pseudo-label these regions for training.Given a pair (x L , y) of a labeled image and its corresponding ground truth segmentation, we first find the object surface using the ground truth segmentation y. Subsequently, we randomly select k voxels to form two sets based on their distances from the object surface: the voxels near the surface inside the object (object voxels) and those near the surface outside object (background voxels). We use the dense feature map (before squeezing the channels with 1 × 1 × 1 convolutions at the output layer) from the teacher network to embed these voxels. We denote the embeddings for the object voxels and background voxels as f t L + ∈ R C×k and f t L -∈ R C×k , respectively, where C is the dimension of the embedding space.When presented with an unlabeled image x U and its corresponding voxel embeddings f s U ∈ R C×H×W ×D from the student network, we proceed to assign pseudo labels by comparing f s U with the labeled embeddings f t L + and f t L -in terms of their pairwise similarities. A voxel is assigned to the object class if its corresponding embedding is more similar to the object embeddings than to the background embeddings. This is equivalent to a k-nearest neighbor classifier in the embedding space. Interestingly, even when the labeled embeddings are on the wrong side of the true decision hyperplane, and the teacher network mislabels their corresponding voxels, we can still utilize the label of these voxels to guide the student training of other voxels in the unlabeled images given their embeddings similarity.We denote K as the similarity measurement kernel (cosine similarity). To obtain dense similarity mapsK(f t L + , f s U ) ∈ R H×W ×D and K(f t L -, f s U ) ∈ R H×W ×D , we first computepairwise similarity scores between f t L ± and f s U before reducing the scores across the k labeled embeddings by averaging. However, to avoid the contribution from potential outliers in the embedding space, we employ an ensemble of nearest neighbor classifiers by computing the dense similarity maps l times, where the results of each run are denoted asK l (f t L + , f s U ) and K l (f t L -, f s U ).At each run, we sample different sets of k voxels to generate f t L + and f t L -. The pseudo labels ŷ nn from the ensemble of nearest neighbor classifiers are obtained by averaging all l dense similarity maps. We added supplementary materials to showcase the computation of dense similarity maps is robust against the choice of similarity measurement kernels and reduce operations.These pseudo labels can be used to supervise training for the voxels at which the uncertainty measurements are higher than the predefined threshold, i.e., where the teacher model's predictions are not reliable enough to treat them as pseudo labels (see Sec. 2.2). We refer to this loss term as the nearest neighbor matching loss L N N , formulated as:L N N = [1(H x t ≥ λ T ) • BCE( ŷ nn , CN N s (x))] 1(H x t ≥ λ T ),EQUATION)where 1 is the indicator function. Note that the computation of ŷ nn is cut off from the gradient computation graph to avoid the situation where the student model is involved in both back-propagation and pseudo-label generation.Segmentation around the object surface is generally challenging, partly due to potential inconsistencies in manual labeling and blurry voxel intensities around the surface. This can lead to ambiguous model predictions and corresponding cluttered features in the embedding space. Therefore, it can be challenging to have a clear separation in dense similarity maps (as discussed in Sec. 2.3.1) such that K l (f t L + , f s U ) and K l (f t L -, f s U ) are distinguishable. To maximize the similarity map separation, we propose to add an entropy minimization loss term:L EN = -(p ⊕ log p ⊕ + p ⊖ log p ⊖ ) p ⊕ = e K(f t L + ,f s U ) /(e K(f t L + ,f s U ) + e K(f t L -,f s U ) ), p ⊖ = 1.0 -p ⊕ (4)p ⊕ and p ⊖ are predictive probabilities from the nearest neighbor classifier assigning labels to the object and background class, respectively. These can be computed by simply applying softmax on the dense similarity maps (K(f t L + , f s U ) and K(f t L -, f s U )) concatenated channelwise. The loss L EN is intended to minimize the predictive entropy of the nearest neighbor classifier and thus max-margin its classification decision boundary.The total loss L for training the student model on each mini-batch is the sum of the loss on labeled images L S and the loss on unlabeled images L U . L U is the weighted summation of L U A , L N N , and L EN . L and L U can be formulated as:L = L S + L U L U = w U A • L U A + w P S • (L N N + L EN ) (5)w U A follows a ramp-up procedure ramp up(T, 0.25), and w P S follows a ramp-up procedure ramp up(T, 0.125). The final weights for w U A and w P S are 0.25 and 0.125, respectively.  
data: 
 To evaluate the proposed semi-supervised learning framework, we conducted experiments on a cohort of 950 subjects. Hip bones in their CT scans were manually-segmented with proprietary software. Scans were randomly partitioned into 350 for training, 100 for validation, and 500 for testing. We further preprocessed CT scans in each split into 3D patches by sampling at different locations in the image. We ensure images to have isotropic spacing of 1 millimeter and normalized intensity values using the mean and standard deviation computed using 3D patches statistics. We collected 6266 3D patches for training, 1174 for validation, and 1200 for testing, where test set patches were sampled with minimal overlaps and chances from the same subject.For semi-supervised training, we divided the training set into five distinct subsets, each consisting of 50, 150, 300, 1000, and 2500 labeled patches, with the remaining patches being considered as unlabeled. These 3D patches were sampled from 4, 10, 21, 71, and 180 CT scans. Total 6266 training patches were sampled from 350 training CT scans. Each semisupervised subset was named after the number of labeled patches. For instance, 'Subset 50' comprised 50 labeled patches, while the remaining 6216 patches were considered unlabeled and utilized for semi-supervised training without their reference segmentations.The proposed semi-supervised segmentation approach was benchmarked against various SOTA methods, including the mean teacher (MT (Tarvainen and Valpola, 2017) ) method, consistency-training-based methods such as cross consistency training (CCT (Ouali et al., 2020) ), multi-tasking consistency (MTC (Chen et al., 2019) ) and dual-task consistency (DTC (Luo et al., 2021) ), pseudo-labeling based methods such as the uncertainty-aware mean teacher (UAMT (Yu et al., 2019) ). We specifically selected the MT and UAMT methods due to their direct relevance to our approach. Meanwhile, MTC, CCT, and DTC were chosen to represent diverse applications of consistency training and pseudo-labeling in weakly-supervised segmentation tasks. By comparing our method against these varied approaches, we aim to provide a comprehensive perspective on its strengths. Wilcoxon signed-rank tests were employed to assess whether the performance differences were statistically significant (p < 0.05). All models included in the comparison were based on the same 3D U-Net with a Dropout rate of 0.15 for all convolution layers except in the output layer. The baseline method is a single 3D U-Net, trained in a fully-supervised way using only labeled images from each subset.The model parameters were optimized using Adam optimizer (Kingma and Ba, 2014) with the initial learning rate set to 10 -4 and decay rate β 1 set to 0.9 and β 2 set to 0.99. The network parameters were initialized using He initialization (He et al., 2015) . The moving average decay was set to 0.99 for the teacher model. We used k = 16 in the nearest neighbor classification and set the number of classifiers in the ensemble to l = 5 (see Sec. 2.3). We ran M = 5 stochastic forward passes for uncertainty analysis to compute uncertainty (see Sec. 2.2). All models were trained for 40 epochs, reaching convergence as indicated by stable or increasing validation errors. And we selected the model with the lowest validation Hausdorff surface distance to evaluate the performance on the test set. The teacher input is subjected to weak Gaussian additive noise at a scale of 0.01, while the student input is exposed to more substantial Gaussian noise addictive at a scale of 0.02.Results in Table 1 show that the proposed method outperforms the other methods in subsets in overall segmentation performance. When training on only 50 3D patches, the proposed method reaches 3.30 (mm) in HD95 and 0.929 in IoU, substantially outperforms the baseline at 9.45 in HD95 and 0.894 in IoU, and the second best semi-supervised performance (4.07 in HD95 and 0.927 in IoU from DTC). The fully-supervised method (the baseline trained on all 6266 images) reaches 0.948 IoU and 1.92 HD95.This section shows the qualitative results of training the proposed semi-supervised method with 50, 150, 300, 1000, and 2500 labeled patches compared with the manual segmentation and the fully-supervised baseline. Figure 2 illustrates three challenging cases. We observe that the proposed method, only employing a limited number of labeled patches (50 and 150) in training, may produce the segmentation results (shown in the (3 rd and 4 th columns) deviating from the reference standard (1 st column), particularly in the presence of out of distribution samples (e.g., metal artifacts in the 2 nd -row case). The proposed semisupervised method overlooked metal artifacts (dark fan-shaped rays) caused by the inbody implants made of metal. This is because metal artifacts are uncommon in our data collection, so the chances of labeled examples containing metal artifacts are small.The proposed method trained with limited (≤ 300) labeled samples may fail to capture anatomical and pathological variations, seen as over-segmentations in the 3 rd row case, covering the gap between the acetabulum and the femoral head. These regions exhibit Figure 2 : Qualitative results of the proposed methods trained with 50, 150, 300, 1000, and 2500 labeled patches (from the 3 rd column to 7 th column) compared with the ground truth (1 st column) and the fully-supervised baseline (2 nd column). We show a 3D patch using its central axial slice. By adding more labeled examples, the proposed method converges to manual labels, although still keeping consistent errors in segmenting metal artifacts (2 nd row) and identifying local anatomical structures that are ambiguous in manual segmentation (3 rd row).significant anatomical and pathological variations, requiring sufficient supervision to learn powerful features. We saw that the proposed method with 1000 and more labeled images (the last two columns) performs substantially better in these regions. 
conclusion: 
 We present a novel semi-supervised segmentation method for segmenting hip bones (pelvis and femoral head) in CT images. Our key innovation is using embedding correspondence to transfer labels from labeled voxels to unlabeled ones via the proposed neighbor matching loss and entropy minimization regularization strategy. Unlike existing methods that typically exclude voxels with unreliable predictions during pseudo-labeling, our approach can leverage these voxels to have a full coverage in the pseudo labeling process, maximizing the potential to extract valuable information from large amount of unlabeled images.Our method outperforms other state-of-the-art techniques in hip bone segmentation from CT images, utilizing merely 50 labeled patches for training. Nonetheless, our findings highlight the necessity of a sufficient volume of labeled images to account for anatomical and pathological variations, as well as imaging artifacts. This is evidenced by the noticeable performance disparity between weakly-supervised approaches with limited labeled data and a fully-supervised method. Appendix C. Data collection details 950 subjects in our study underwent robotic-assisted total hip arthroplasty. All patients provided written consent allowing their data to be used for R&D purposes. Their preoperative CT scans were analysed with proprietary software and segmented by two expert human annotators. Annotator 1 generated an initial segmentation that was reviewed, and potentially updated, by annotator 2, resulting in the final segmentation ground-truth used in this study. The annotated CT scans were utilized for both training and evaluation of our algorithm. The scans were randomly partitioned into three sets, with 350 used for training, 100 for validation, and 500 for testing. Due to the large size of CT volumes, we preprocessed all images into smaller 3D patches by randomly sampling various locations in the image with minimal overlap. The number of 3D patches for each split can be found in Table 4 . All 3D patches included in the dataset were accompanied by corresponding reference segmentations. Each of these patches underwent scoring based on the surface deviation between the manual segmentations and the segmentations derived from statistical shape models. The surface deviation was measured by the Hausdorff surface distance (with 95 % percentile, HD95) between two surfaces. Note that the segmentations produced by the statistical shape models are often influenced by common shapes. Consequently, the surface deviation serves as an indicator of the anatomical variation present in relation to these common shapes, thereby reflecting the complexity of the segmentation. The score associated with surface deviation is referred to as the surface deviation score (SDS). SDS scores can further be categorized into six groups (ranging from 0 to 5). Table 4 provides an overview of the distribution of surface deviation scores in each group across the training, validation, and test sets. Specifically, the training set consists of 6266 3D patches, while the validation and test sets contain 1174 and 1200 3D patches, respectively. There is a relatively smaller number of 3D patches in the test set (n=1200). This is because we sampled 200 3D patches from each of the six pre-defined SDS groups. The mean and standard deviation of SDS scores from each group can be found in Table 4 , measured in HD95. We collected SDS scores to show that our dataset consists of many challenging cases as they have high degree of shape deformations, i.e., possibly either a heavily-diseased case or representing other complexities (e.g. metal artifacts) introduced by already-placed implants.  
 
 Appendix:  


Decision: The authors propose a novel semi-supervised segmentation method and apply it to hip bone segmentation. The reviewers found the idea of the paper innovative with strong mathematical modeling and promising results. Inclusion of ablation studies and statistical results in the rebuttal resulted in reviewers raising their ratings resulting in one strong accept and two weak accepts.

Reviews: review: paper_summary: paper_summary: This paper presents an innovative semi-supervised method for semantic segmentation of hip bones in CT images. The novelty lies in the pseudo labeling, with propagation from labeled to unlabeled images through voxel-wise matching of embeddings, providing a fresh perspective on utilizing pseudo labels even if the prediction from the teacher is unreliable. The authors assess the performance of their model on a private cohort of 950 patients with CT scans, outperforming state-of-the-art semi-supervised methods while not surpassing a fully supervised approach.
main_review: main_review: Authors have addressed most of the concerns raised in the first review with ablative studies and statistical tests. The methodology is interesting and contributes to the field, even if additional experiments/datasets would add value to the paper. I recommend weak accept.
strength_weakness: strength_weakness: - Even if the novelty is undeniable, one could question its magnitude since only subsection 2.3 is innovative. 2.2 is a simple repeat of [1], and the moving average is the same as [2].
- The performance is only assessed on one (private) dataset, with a very specific location and task. It would be interesting to see how the model behaves in other locations, and it is easy to test it with public datasets since the implementation is rather versatile.
- Source code sharing would be a plus for reproducibility.
- Basic statistical analysis with standard deviation for IoU and HD95 is required to truly assess the significance of the results.
- Ablative studies would have been really interesting: the entropy minimization loss makes sense, and its role is well explained, but the authors could have assessed its importance (or relevance?) by playing with a different weight from the NN matching loss. Anyway, some comments would have been appreciated about the relative importance of each component, the choice of ramp-up hyperparameters, the number of sampled voxels k, or the number of classifiers l (I am not referring to usual hyperparameters like initialization, weight decay or dropout rate). Such ablative studies could have been presented with curves or additional rows in the results table.
-	The choice of SOTA methods needs more justification since the author cites other papers in their review of the literature 
-	The main Figure 1 is not clear enough, I would suggest using different categories of boxes depending on the object (model, feature, loss, measure, image, …).
questions: questions: - The authors mention that they provide supplemental information to assess the choice of the similarity measurement and reduce operations, which is a very sound idea, but no such material is available.
- More detailed quantitative analysis of results should be tackled in the rebuttal, from ablative studies to benchmark choices and statistical significance. 
- Discussion about the robustness of the approach for other locations and tasks would be highly appreciated.
paper_summary: paper_summary: The paper introduces a semi-supervised segmentation method for hip bones in CT images, utilizing embedding correspondence and neighbor matching loss to propagate labels from labeled to unlabeled voxels. The authors claim superior performance over existing state-of-the-art methods, especially in scenarios with limited labeled data.
main_review: main_review: The authors addressed several concerns in their rebuttal. Based on these responses, I have changed the final rating. As written in the response, I still believe that the manuscript has limited practical relevance and could be evaluated on public datasets.
strength_weakness: strength_weakness: Supervised performance on 2500 patches across 180 CT scans is very similar to semi-supervised performance on the entire dataset (350 CT scans). Even for 71 CT scans the performance is possibly equivalent. So, it is unclear how much this approach is practically relevant.

The current evaluation lacks confidence intervals or statistical tests which makes it harder to understand the significance of the performance improvements.

It would be great to get more clarity on the challenges of segmenting hip bones to better estimate the significance of this work.

The paper appears to incorporate disparate approaches without establishing a cohesive motivation and narrative. The authors distinguish embeddings from object and background pixels, create an ensemble of nearest neighbor classifiers, add an entropy minimization loss term, and apply different noise for student and teacher. Why is each of these steps needed and what is the merit of the individual steps?
questions: questions: 1) Could you provide ablation studies that motivate the necessity of each of the individual steps?

2) What is the practical relevance of this approach given that on 180 CT scans, there seems to be no advantage compared to the semi-supervised approach? Segmenting 180 CT scans seems to be feasible, e.g., using prompt-based image segmentation methods like MedSAM.

3) Could you check for the statistical significance of your results with statistical tests?
paper_summary: paper_summary: The paper presents a semi-supervised segmentation scheme utilizing a teacher-student architecture combined with an embedding matching methodology. The proposed framework can sufficiently employ unlabeled images to enhance segmentation performance. In this methodology, the teacher’s segmentation results with high uncertainty are improved by a voxel-wise embedding correspondence. Evaluation experiments and comparisons indicate that the proposed method slightly outperformed the selected state-of-the-art methods.
main_review: main_review: The article presents a novel semi-supervised hip bone segmentation methodology utilizing a teacher-student architecture combined with an embedding matching strategy. The methodology and theoretical background are well described. The authors presented adequately the main contributions and comparisons against the state-of-the-art methods. Extensive evaluation experiments and ablation studies support the superiority of the method. Therefore, my recommendation is Strong Accept.
strength_weakness: strength_weakness: The proposed method’s results are quite close to the compared methods even for the 50 patches training (Table 1). This indicates that the model’s accuracy is approximately similar to the other state-of-the-art methods that were compared.
questions: questions: Please provide additional details on the choice of the number of epochs during experiments as stated in the “Questions” section. The authors should justify the choice of this number and if this number of epochs is adequate for the training of all models until convergence. Due to the variant methodologies to be compared, the same number of epochs may not be sufficient to train all the models leading to underperformance.

Comments: title: Oral or poster presentation
source code available
Please read and respond to author comments
Two ablation studies added, statistical tests were conducted.
Two ablation studies added, conducted statistical tests.
Ablation studies and statistical tests added
Ablation studies are added, minor changes are made as suggested by reviewers, details in our data collection are provided.
comment: Dear Program Chairs,

Thank you for your diligence and effort in the review process of our manuscript. Could you please inform us whether the paper has been accepted for an oral or a poster presentation?

Best, weiyi
we are agreed to share the source code upon acceptance.
I appreciate the detailed answers to my questions and the inclusion of the statistical test. Based on these updates, I have revised my rating to "weak accept." Although I understand the limitations of seeing improvements 
for only a small number of images, the appendix reveals a substantial availability of subjects, 950 to be exact, which would enable the training of a fully supervised algorithm that is comparable to the proposed approach. 
Consequently, in my view, the practical relevance remains limited.

The potential withholding of the source code creates significant obstacles for the scientific community to reproduce and delve into the intricacies of the algorithm. To address this issue, an evaluation on public datasets 
would be highly beneficial. It is possible that the private dataset is more complete or comprehensible, but I still believe that evaluation on a public dataset is indispensable, particularly if the code may not be made 
available.
Dear reviewers. The authors have posted responses to your reviews. Please take the time to read and respond before March 27.
We are grateful for the reviewer's comments on the strengths and weaknesses of our paper. In our revision, we have added two ablation studies to show how our algorithm performs with various hyper-parameters, as well as the combination of loss terms. Below, we provide detailed responses to each of the concerns raised. We start with the origin reviewer's comment, followed by our response.
- The proposed method’s results are quite close to the compared methods even for the 50 patches training (Table 1). This indicates that the model’s accuracy is approximately similar to the other state-of-the-art methods that were compared.

We have conducted statistical tests in our revision (mentioned in section 3.2) to demonstrate the significance of our improvement, especially when leveraging only 50 labeled images in training. 
- Please provide additional details on the choice of the number of epochs during experiments as stated in the “Questions” section. The authors should justify the choice of this number and if this number of epochs is adequate for the training of all models until convergence. Due to the variant methodologies to be compared, the same number of epochs may not be sufficient to train all the models leading to underperformance.

Thank you for your request for further clarification regarding our choice of the number of epochs in our experiments. In our study, we carefully monitored the convergence of all models throughout the training process. We determined that the number of epochs selected was sufficient to ensure that each model was well-trained and reached convergence. We stopped our training when the validation errors do not show changes or started increasing.  We added the discussion in section 3.2 in the revision.
We are grateful for the reviewer's constructive feedback. Below, we provide detailed responses to each of the concerns raised. We start with the origin reviewer's comment, followed by our response.

- Supervised performance on 2500 patches across 180 CT scans is very similar to semi-supervised performance on the entire dataset (350 CT scans). Even for 71 CT scans the performance is possibly equivalent. So, it is unclear how much this approach is practically relevant.

We thank the reviewer for their insightful comments on our performance comparison between semi-supervised and fully supervised approaches. Our study focuses on demonstrating the effectiveness of our method in enhancing segmentation with far fewer labeled examples, a critical asset given the dataset's complexity and the substantial labeling effort required. To underscore the dataset's challenging nature, we've included a table (Table 4) in the appendix detailing case complexity scores. This highlights the task's inherent difficulties, particularly in the context of total hip arthroplasty, where pathological changes and imaging artifacts make extensive labeling both costly and time-consuming. Thus, our method's ability to achieve improved segmentation with limited labeled data is not only significant but also highly relevant for practical applications.

- The current evaluation lacks confidence intervals or statistical tests which makes it harder to understand the significance of the performance improvements.
 
In the revision, a Wilcoxon signed-rank test was employed to assess whether the performance difference was statistically significant (p $<$ 0.05). We mentioned this in section 2.3.

- It would be great to get more clarity on the challenges of segmenting hip bones to better estimate the significance of this work.

To highlight the complexity of our dataset, we've added Table 4 to the appendix, presenting case complexity score distributions. This illuminates the inherent challenges in segmenting hip joints for total hip arthroplasty, where pathological variations and imaging artifacts, especially from pre-existing metal implants, significantly complicate the task. 

- The paper appears to incorporate disparate approaches without establishing a cohesive motivation and narrative. The authors distinguish embeddings from object and background pixels, create an ensemble of nearest neighbor classifiers, add an entropy minimization loss term, and apply different noise for student and teacher. Why is each of these steps needed and what is the merit of the individual steps?

In the revised manuscript, we've incorporated two ablation studies in the appendix (Table 2 and 3) to assess the impact of various loss terms and the hyper-parameters involved in computing dense similarity maps.

Questions To Address In The Rebuttal:
- Could you provide ablation studies that motivate the necessity of each of the individual steps? 

Added to the appendix.
- What is the practical relevance of this approach given that on 180 CT scans, there seems to be no advantage compared to the semi-supervised approach? Segmenting 180 CT scans seems to be feasible, e.g., using prompt-based image segmentation methods like MedSAM.

We sincerely appreciate your insightful inquiry regarding the practical relevance of our approach, particularly in the context of the results obtained from 180 CT scans. Our method aims to demonstrate the significant advantages of semi-supervised learning in scenarios where labeled data is scarce or costly to obtain. 
We acknowledge that prompt-based segmentation methods like MedSAM could be effective for some applications. However, these heavy methods are often computationally intensive, requiring at least 6GB of GPU memory for processing a single 2D image at 1024x1024 resolution, and the inference process is slow (> 6 seconds per 2D slice) on commonly available hardware based on our own experience. Moreover, applying SAM or MedSAM introduces new challenges, such as boundary ambiguity, which may necessitate multiple points or boxes prompts from the user to accurately define object contours, further increasing the time and effort required for prompting.

- Could you check for the statistical significance of your results with statistical tests? 

We've incorporated statistical test results for all the tables presented in the revised manuscript. Thank you for this valuable suggestion.
We are grateful for the reviewer's acknowledgment of our paper's novelty and appreciate the constructive feedback. Below, we provide detailed responses to each of the concerns raised. We start with the origin reviewer's comment, followed by our response.

- Even if the novelty is undeniable, one could question its magnitude since only subsection 2.3 is innovative. 2.2 is a simple repeat of [1], and the moving average is the same as [2].  

Indeed, the core innovation of our work is concentrated within section 2.3, where we introduce a novel approach that leverages embedding analysis for the pseudo-labeling of regions or voxels. We view this as a foundational contribution, offering a versatile framework that enriches the current landscape of semi-supervised learning methodologies.
- The performance is only assessed on one (private) dataset, with a very specific location and task. It would be interesting to see how the model behaves in other locations, and it is easy to test it with public datasets since the implementation is rather versatile. 

We understand and appreciate the reviewer's suggestion to validate our approach across multiple datasets to enhance the generalizability of our findings. However, our decision to focus on a single private dataset was driven by following considerations. 
Our dataset with curated manual segmentations, was carefully compiled to include a diverse array of pathologies, anatomical variations, and imaging artifacts in hip bone CT images, crucial for thoroughly testing the effectiveness and robustness of our segmentation approach. In the appendix (Table 4), we present the distribution of case complexity scores within our dataset to highlight the segmentation challenges, illustrating the diverse conditions our method is designed to address.

- Source code sharing would be a plus for reproducibility.

We are actively considering the possibility of sharing our source code upon acceptance and are currently in discussions regarding this matter. The final decision will adhere to our scientific dissemination review protocol, in line with the relevant company policies.

- Basic statistical analysis with standard deviation for IoU and HD95 is required to truly assess the significance of the results.

We employed a Wilcoxon signed-rank test (p < 0.05) to determine if the performance differences observed were statistically significant. We mentioned this in section 3.2.

- Ablative studies would have been really interesting: the entropy minimization loss makes sense, and its role is well explained, but the authors could have assessed its importance (or relevance?) by playing with a different weight from the NN matching loss. 

In the appendix, we've incorporated two ablation studies. The first assesses the significance of each loss term, including entropy loss, nearest neighbor matching loss, and uncertainty analysis loss. The second examines the effects of varying similarity measurement functions and reduction operations on dense similarity map computation. We consider these elements crucial to our model's design. Our ramp-up parameters follow the precisely defined ramp-up function from the uncertainty-aware mean teacher approach. The choice of the number of sampled voxels, k, is constrained by computational resources, with a larger k offering diminishing returns due to the computational demand and the randomness in sampling voxels.
 
- The choice of SOTA methods needs more justification since the author cites other papers in their review of the literature. 

MT and UAMT were selected because they are the most relevant to our method. MTC, CCT and DTC represents different way of using consistency training and pseudo labeling in weakly-supervised segmentation. We believe bringing methods in different lines of works in comparison provides comprehensive views of the strength of the proposed method. We added discussions in section 3.2.
- The main Figure 1 is not clear enough, I would suggest using different categories of boxes depending on the object (model, feature, loss, measure, image, …). 

We have updated Figure 1 to differentiate models and features through distinct color coding.

Questions:
- The authors mention that they provide supplemental information to assess the choice of the similarity measurement and reduce operations, which is a very sound idea, but no such material is available. 
Added.
- More detailed quantitative analysis of results should be tackled in the rebuttal, from ablative studies to benchmark choices and statistical significance.
Added.
- Discussion about the robustness of the approach for other locations and tasks would be highly appreciated.
Up to this point, our efforts have been concentrated on conducting a comprehensive analysis of the dataset presented in our study. We believe our future work will allow us to provide the empirical evidence necessary to fully demonstrate the versatility of our approach across various segmentation problems.
We sincerely thank the reviewers for their insightful and constructive feedback, which has undoubtedly helped improve the quality and clarity of our manuscript. We have carefully considered each comment and have made substantial revisions to address these concerns. Below, we summarize our responses to the main points raised, demonstrating how we have incorporated the feedback into our revised manuscript.

Ablation studies are added in the appendix of the revised manuscript.
 - The first ablation study (Table 2) shows that the proposed method w/o entropy loss, w/o nearest neighbor matching loss in combination with the uncertainty analysis loss. 
 - The second ablation study (Table 3) meticulously examines the performance of our method across different embedding similarity functions and reduction operators used to compute distance similarity maps. These maps are crucial for determining nearest neighbor matching loss and entropy loss, underscoring their importance in our methodology.

We have also conducted statistical analyses, specifically the Wilcoxon signed-rank test (p < 0.05) mentioned in section 2.3, across all reported results to rigorously determine the significance of the improvements observed.

We have provided a comprehensive explanation of our data collection protocols in the appendix (Table 4), detailing the acquisition of reference segmentations and the generation process for the 3D patches utilized in the training and validation phases of our method.

We have implemented minor revisions to enhance the manuscript's quality further. These adjustments, along with the clarifications provided, aim to fully address the reviewers' concerns and significantly elevate our contribution to the domains of semi-supervised learning and medical image segmentation. We are thankful for the chance to refine our work in light of the insightful feedback received and are hopeful that our revised manuscript aligns with the conference's acceptance criteria.