Reducing Uncertainty in 3D Medical Image Segmentation under Limited Annotations through Contrastive Learning

Authors: sanaz  karimijafarbigloo
reza  azad
amirhossein  kazerouni
dorit  merhof
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 11:18:53

Abstract: Despite recent successes in semi-supervised learning for natural image segmentation, 
applying these methods to medical images presents challenges in obtaining discriminative representations from limited annotations. While contrastive learning frameworks excel in similarity measures for classification, their transferability to precise pixel-level segmentation in medical images is hindered, particularly when confronted with inherent prediction uncertainty.
To overcome this issue, our approach incorporates two subnetworks to rectify erroneous predictions. The first network identifies uncertain predictions, generating an uncertainty attention map. The second network employs an uncertainty-aware descriptor to refine the representation of uncertain regions, enhancing the accuracy of predictions. Additionally, to adaptively recalibrate the representation of uncertain candidates, we define class prototypes based on reliable predictions. We then aim to minimize the discrepancy between class prototypes and uncertain predictions through a deep contrastive learning strategy.
Our experimental results on organ segmentation from clinical MRI and CT scans demonstrate the effectiveness of our approach compared to state-of-the-art methods.

Main paper: 
 introduction: 
 Medical image segmentation plays a pivotal role in the field of medical imaging, serving as a crucial step in the analysis and interpretation of complex visual data. This process involves the partitioning of images into meaningful and clinically relevant regions, allowing for a detailed examination of structures and abnormalities within the human body. One prominent approach to medical image segmentation is supervised learning. This paradigm involves training algorithms on labeled datasets, where each image is accompanied by annotations identifying the regions of interest. Despite its success in many applications, its main drawback lies in its dependency on large and accurate labeled datasets (Aljuaid and Anwar, 2022; Azad et al., 2023) . Creating such datasets for medical images requires considerable expertise and time. Additionally, supervised learning is susceptible to human error caused by manual segmentation and labeling.In response to these challenges, various strategies have been proposed to alleviate the dependency on meticulous labeling processes. Unsupervised learning operates without labeled data. Algorithms in this category identify inherent patterns and structures within medical images without prior knowledge of specific regions (Caron et al., 2018; Chen and Frey, 2020; Hamilton et al., 2022; Zhao et al., 2022; Feng et al., 2023; Omidi et al., 2024) . However, its limitations include the difficulty of distinguishing between normal and abnormal structures, hindering applicability in clinical settings that require precise identification. Interpretability of results is also challenging, as decisions rely solely on inherent data patterns. Transfer learning is another powerful strategy for enhancing medical image segmentation, leveraging pre-trained models to improve performance on tasks with limited labeled data (Kora et al., 2022; Araújo et al., 2022; Alhares et al., 2023) . However, its drawback stems from the assumption of similar distributions between the source and target domains. If this assumption is not met, pre-trained features may not capture the nuances of the target medical imaging data, potentially leading to suboptimal results. Self-supervised learning overcomes label scarcity by generating its own supervisory signals, enhancing model performance (Tang et al., 2022; Karimijafarbigloo et al., 2023; Ouyang et al., 2022; Kazerouni et al., 2023) . This approach, beneficial when labeled data is limited, faces the challenge of designing effective surrogate tasks. Ensuring these tasks capture pertinent information is crucial for the success of self-supervised methods. Furthermore, these approaches usually require additional supervisory signals derived from annotated data to be specifically directed toward the targeted task.To address these issues, semi-supervised learning (SSL) offers a promising solution. This approach involves training models with a limited number of labeled samples and a large number of unlabeled data, striking a balance between supervised and unsupervised methods. In medical image segmentation, it proves valuable in scenarios where obtaining a fully labeled dataset is impractical, providing a practical and cost-effective solution for training robust segmentation models (Luo et al., 2022b; Wu et al., 2022; Luo et al., 2022a; Wang et al., 2023) . SSL commonly employs two primary methods: pseudo-labeling and consistency regularization. Pseudo labeling involves generating "pseudo labels" for unlabeled data using a model's predictions. In a teacher-student (Tarvainen and Valpola, 2017; Wang et al., 2022; Xu et al., 2021) scenario, the teacher model, represented as the EMA of the student model, plays the role of pseudo labels generation. These pseudo labels are then integrated with the original labeled dataset to optimize accuracy and achieve cost-effective training (Lee et al., 2013; Xie et al., 2020) . However, it's important to acknowledge that while the pseudo-labeling approach offers advantages, the scarcity of labeled data raises concerns about the reliability of pseudo-label quality. Hence, this method may introduce the risk of inaccuracies in the training data, potentially affecting the final model's precision.To address the mentioned issue, current methods suggest adopting confidence score filtering for predictions (Zuo et al., 2021; Zou et al., 2021; Zhang et al., 2021; Sohn et al., 2020) . This means that only the predictions with high confidence scores are employed as pseudo-labels, while those that are uncertain are disregarded. Nevertheless, this approach is not perfect in removing inaccurate predictions, as some incorrect predictions might possess high classification scores, known as over-confidence or miscalibration (Guo et al., 2017) . Furthermore, setting a high threshold would significantly decrease the quantity of generated pseudo-labels, thereby restricting the efficacy of semi-supervised learning. Additionally, the potential problem of solely relying on presumably reliable predictions (which may instead comprise inadequate representations of certain classes or segments) may lead to an imbal-ance of the training data and ultimately compromise the model's performance, particularly for challenging and less frequent classes. For instance, when the model encounters difficulty in accurately predicting specific classes, generating accurate pseudo-labels for the corresponding pixels becomes problematic. In this respect, Lu et al. (Lu et al., 2023) suggested a technique that combines pseudo-labeling with dual consistency regularization, emphasizing its strong uncertainty awareness capability. This approach incorporates a cycle-loss regularization to enhance the accuracy of uncertainty estimation. Shen et al. (Shen et al., 2023) introduced the UCMT method for semi-supervised semantic segmentation. This approach consists of two main components: Collaborative Mean-Teacher (CMT) and an uncertaintyguided region mix. The CMT component aims to maintain model disagreement while enhancing the quality of pseudo-labels through collaboration. Specifically, UCMT generates a new image by replacing uncertain regions with certain ones and then utilizes a collaborative approach to ensure consistent predictions across different networks. However, a limitation of this method is that it does not explicitly modify the representation of related voxels to reduce uncertainty. Consequently, there is a need for a mechanism to re-represent these uncertain voxels with different localities, such as through deformable convolutions, which could significantly enhance the overall effectiveness of the approach.Acknowledging the reliability concerns linked with pseudo-labeling and the drawbacks of confidence score filtering, our method introduces a novel semi-supervised contrastive learning approach to address these challenges. In this context, we outline the following key contributions: 1) We propose a mechanism to recognize uncertain predictions as a means to refine network representation, aiming for improved overall representation quality. 2) To alleviate prediction uncertainty, we introduce an uncertainty-aware feature descriptor module. This module enhances contextual and semantic representation, contributing to a more robust and accurate prediction. 3) We design a deep contrastive supervision function to minimize discrepancies between class prototypes and uncertain predictions. 
method: 
 Current SSL algorithms that rely on consistency learning, such as Mean-teacher (Tarvainen and Valpola, 2017) and (Chen et al., 2021) , propose applying consistency regularization not within a single model but among the pseudo labels within a multi-model architecture. Nonetheless, throughout the training process, there is a tendency for the dual-network SSL framework to quickly reach a consensus, causing the co-training to degrade into selftraining (Kendall and Gal, 2017) . To address this issue, we introduce a setup consisting of a predictive model accompanied by an auxiliary model. The predictive model serves as a pseudo-label generator, directing the training of the other model. In our strategy, we work with two distinct datasets: D l , comprising labeled data pairs (x l i , y l i ), where i ranges from 1 to N l , and D u , a considerably larger unlabeled dataset denoted as {x u i } Nu i=1 , N l and N u indicate the number of samples in the labelled and unlabled dataset, respectively. The goal is to synergistically leverage the inherent potential of the unlabeled data to enhance feature representation, effectively training our semantic segmentation model by integrating a limited set of labeled data with an extensive repository of unlabeled data. For labeled data, both predictive and auxiliary models undergo optimization through supervised learning: where ŷl i , ỹl i denotes the predicted segmentation map of the predictive and auxiliary networks, respectively. B l indicates the batch of labeled data. When handling unlabeled data, two key factors come into play: 1) direct guidance from the predictive model to the auxiliary model to reduce uncertainty and 2) minimizing uncertain predictions through the definition of deep-supervisory contrastive learning. To accomplish the former, we first fed the unlabeled sample into the predictive model to generate the pseudo label ŷu i . Then using the pseudo label we calculate the prediction loss for the auxiliary models. To further enhance error correction, we incorporate a regularization loss L reg , which calculates the L 1 distance between the XOR predictions (Wang et al., 2023) of the predictive and the auxiliary models:L s = 1 |B l | (x l i ,y l i )∈B l ℓ ce (ŷ l i , y l i ) + Dice(ŷ l i , y l i ) + ℓ ce (ỹ l i , y l i ) + Dice(ỹ l i , y l i ) (1)EQUATIONwhere λ reg indicates the weight of the L reg . We additionally restrict the computation of the loss function to voxels with prediction confidence exceeding 0.7. The overall training loss (L) incorporates three components: the supervised loss (L s ), unsupervised loss (L u ), and the contrastive loss (L c ). Details of the contrastive loss are presented in subsection 2.2. The optimization objective revolves around minimizing the overall loss, expressed as:EQUATIONThe weights λ u and λ c determine the contribution of the unsupervised and contrastive losses, respectively. The overall architecture of the network is depicted in Figure 1 .In our design, we argue that inaccuracies can arise due to inadequate contextual information in the vicinity of voxels, which makes precise predictions challenging. To address this issue, we first input the sample into the predictive model to generate the segmentation map. Then, We create a binary uncertainty map by thresholding the softmax output of the network.In this process, pixels with confidence scores below a specified threshold are labeled as uncertain regions (assigned a value of 1), while pixels with confidence scores above the threshold are labeled as certain regions (assigned a value of 0). Finally, we enhance the representation of uncertain voxels on the auxiliary map using the MultiCURE module. MultiCURE, a key component in our proposed architecture, is designed to recalibrate context information across multiple scales by adaptively selecting receptive fields from global and local pathways. It effectively overcomes the limitations of fixed kernel sizes in traditional convolutions, reducing uncertainty in critical areas, especially along object boundaries, which inherently have the highest degree of uncertainty. Comprising three branches, two maintain fixed kernel sizes, enhancing the flexibility of receptive field sizes. However, grid misalignment issues persist, particularly along object boundaries (or uncertain regions), resulting in less optimal results. The third branch provides vital global information, refining boundary delineation, and significantly helps to improve overall performance.The MultiCURE employs a three-path split, incorporating convolution, Batch Normalization (BN), and ReLU activation function in two paths with 3 × 3 and 5 × 5 kernel sizes. The third path performs a 3 × 3 Deformable convolution (Dai et al., 2017) on the input z ∈ R H×W ×d . The deformable convolution in our design dynamically adjusts the receptive field for each feature map location by using an offset field to flexibly warp the sampling grid. This allows the model to handle varied object sizes in an image and gain a superior understanding of object regions. To further enhance feature representation within uncertain regions, we integrate (i.e., sum) an uncertain attention map with the output of the deformable convolution. This integration conditions the representation based on uncertain regions, serving as a mechanism to accentuate the representation of these areas using non-fixed resampling points. Consequently, this approach allows us to allocate increased attention to uncertain regions.To integrate information from all branches, a fusion step entails performing channelwise concatenation: q = [u 1 ||u 2 ||u 3 ] ∈ R H×W ×3d . The next step involves applying global average pooling (GAP) to q to condense the spatial information across the entire feature map. In this context, the vector q ′ ∈ R 3d undergoes a linear transformation denoted as F : q ′ → k, resulting in the vector k in R 3d/r . This transformation serves the purpose of dimensionality reduction, thereby enhancing computational efficiency. Subsequently, each path follows an independent linear layer to revert the transformed vector of dimension 3d/r back to its original dimensionality, denoted as d in the original input.By concatenating all paths within the channel dimension (K ∈ R 3×d ), a soft attention mechanism, specifically the SoftMax function, is deployed across channels. This adaptive approach facilitates the selective emphasis on the most pertinent feature scales. The resultant feature map, denoted as z ′ ∈ R H×W ×d , is attained by applying attention weights to kernels associated with the individual streamlines:z ′ = u 1 • K 1 + u 2 • K 2 + u 3 • K 3In our methodology, we propose deep contrastive supervision to refine the model's representation and enhance its discriminative capabilities. Initially, we identify high-confidence and uncertain predictions within the segmentation output. Leveraging the high-confidence predictions, we define class prototypes by extracting representations from multiple network levels, to capture both shallow and depth representation. To compute each prototype, we calculate the mean vector (c) of reliable voxel representations for class k by defining c k as:EQUATIONwhere f l:L (v r i ) represents the feature representation of the voxel corresponding to the reliable predictions (v r i , y i ) from different levels of the network and S k is the set of certain predictions for class k. Subsequently, for the set of uncertain predictions f (v u r i ), we resample candidates to align them with the corresponding class prototype, employing a contrastive learning algorithm for this purpose. By applying contrastive learning to feature sets extracted from various network blocks, we provide a deep supervisory signal for the network to contextually recalibrate the representation of uncertain pixels, aligning them with the class prototype. The contrastive loss is computed by aggregating their representations, and our objective is to minimize this loss. This approach leverages multi-level representations for deep supervision, enabling the refinement and improvement of the network's predictions to generate more discriminative features.EQUATIONwhere, L c k is the contrastive loss for class k, sim (v u i , c k ) is the similarity function measuring the cosine similarity between an uncertain voxel representation v r i and the class prototype c k , τ is a temperature parameter controlling the sharpness of the contrastive loss function. We augment the contrastive loss by adding an additional term that considers the distance between class prototypes. This extra term is incorporated to promote the representation space to actively separate the clustering spaces of different classes. 
experiments: 
 We implemented a two-stream pipeline in PyTorch, utilizing an RTX A5000 GPU. Following (Wang et al., 2023) , Vnet and Resnet were chosen for the predictive and auxiliary models, respectively. The training involved 6000 iterations with a batch size of 2, sampling randomly from supervised and unsupervised sources. SGD optimizer with parameters: decay factor 0.0001, momentum 0.9, and initial learning rate 0.01 (decayed by a factor of 10 every 2500 iterations). To manage hyperparameters, we set λ u = 1.0 and λ c = 0.1 * e 4(1-t/tmax) 2 for dynamic weighting, where t and t max denote current and maximum iterations. For evaluation, we follow (Wang et al., 2023) setting and use 5-fold and 4-fold cross-validation on the LA and Pancreas datasets, respectively.Left Atrial Dataset (LA): This dataset (Xiong et al., 2021) consists of 100 3D gadoliniumenhanced MR imaging volumes with non-uniform resolution (0.625 × 0.625 × 0.625 mm³) and manual annotations for the left atrial region. Following the pre-processing protocol from (Wang et al., 2023) , we normalized volumes to zero mean and unit variance. During training, random cropping used with dimensions of 112 × 112 × 80. For inference, we used a sliding window approach (112 × 112 × 80) with a stride of 18 × 18 × 4. NIH Pancreas Dataset: This dataset (Roth et al., 2015) consists of 82 abdominal CT volumes annotated for the pancreas. We preprocess it by applying soft tissue windowing (HU range: -120 to 240) and spatial alignment using a method from (Luo et al., 2021; Wang et al., 2023) . In training, we use random cropping for volumes, resulting in dimensions of 96×96×96. During inference, a stride of 16×16×16 is employed for efficient data processing.Table 1 illustrates the performance comparison between our proposed approach and the latest State-of-the-Art (SOTA) methods. Our method exhibits substantial enhancements across all metrics, excelling particularly in organ voxel detection, notably in Dice and Jaccard indices. This demonstrates that the incorporation of the uncertainty map along with the deep contrastive supervision can significantly enhance the efficacy of the model. In comparison to MCF, our technique is clearly superior, achieving a notable boost in DSC from 88.71 to 89.21 and a remarkable enhancement in the Jaccard index from 80.41 to 81.64, while simultaneously maintaining a superior 95HD metric. Furthermore, our method not only maintains a stable and reliable performance by minimizing variance but also excels in the visual comparison presented in Figure 3 , showcasing its superiority in left atrial segmentation. The visuals emphasize the increased alignment with ground truth labels and a noticeable reduction in false segmentations, signifying the nuanced details effectively captured by our innovative approach.Additionally, this robust performance also extends to the NIH Pancreas Dataset, as evidenced in the comprehensive results provided in Table 2 . As the pancreas is situated deep within the abdomen, it exhibits notable variations in size, location, and shape. Adding to the complexity, pancreatic CT volumes present a more intricate background compared to the relatively simpler background of left atrial MRI volumes. This inherent complexity makes pancreas segmentation a more challenging task than left atrial segmentation. More specifically, our method outperformed all SOTA methods across all performance metrics in Table 1 : Comparison of results using the LA dataset (average ± standard deviation).Dice(%)↑ Jaccard(%)↑ 95HD(voxel)↓ ASD(voxel)↓ MT (Tarvainen and Valpola, 2017) 85.89 ± 0.024 76.58 ± 0.027 12.63 ± 5.741 3.44 ± 1.382 UA-MT (Yu et al., 2019) 85.98 ± 0.014 76.65 ± 0.017 9.86 ± 2.707 2.68 ± 0.776 SASSNet (Li et al., 2020) 86.21 ± 0.023 77.15 ± 0.024 9.80 ± 1.842 2.68 ± 0.416 DTC (Luo et al., 2021) 86.36 ± 0.023 77.25 ± 0.020 9.02 ± 1.015 2.40 ± 0.223 MC-Net (Wu et al., 2021) 87.65 ± 0.011 78.63 ± 0.013 9.70 ± 2.361 3.01 ± 0.700 UCMT (Shen et al., 2023) 88.13 ± 0.000 79.18 ± 0.000 9.14 ± 0.000 3.06 ± 0.000 MCF (Wang et al., 2023) 88.71 ± 0.018 80.41 ± 0.022 6.32 ± 0.800 1.90 ± 0.187 Our Method 89.21 ± 00.18 81.64 ± 00.24 6.31 ± 0.842 1.92 ± 0.195 pancreas segmentation. Expanding on the segmentation outcomes, Figure 3 offers deeper insights, emphasizing the consequential influence of the proposed modules on elevating the overall segmentation quality. Notably, our method excels by producing sharper edges and achieving more precise boundary separation compared to the MCF and MC-Net methodologies. This emphasizes its effectiveness in enhancing the reliability of object boundary predictions and distinctly discerning the organ of interest from the background. Table 2 : Comparison of results using the NIH dataset (average ± standard deviation).Method Dice(%)↑ Jaccard(%)↑ 95HD(voxel)↓ ASD(voxel)↓ MT (Tarvainen and Valpola, 2017) 74.43 ± 0.024 60.53 ± 0.030 14.93 ± 2.000 4.61 ± 0.929 UA-MT (Yu et al., 2019) 74.01 ± 0.029 60.00 ± 3.031 17.00 ± 3.031 5.19 ± 1.267 SASSNet (Li et al., 2020) 73.57 ± 0.017 59.71 ± 0.020 13.87 ± 1.079 3.53 ± 1.416 DTC (Luo et al., 2021) 73.  
conclusion: 
 In summary, our approach tackles challenges in semi-supervised medical image segmentation through the integration of two subnetworks aimed at identifying and refining uncertain predictions. The model employs both an uncertainty attention map and an uncertaintyaware descriptor to enhance accuracy in pixel-level segmentation, particularly in scenarios with inherent prediction uncertainty. The efficiency of our method is substantiated by results obtained from both left atrial and pancreas datasets. 
 
 Appendix: In our strategic approach, we incorporated both the deep contrastive learning algorithm and the MultiCURE Module to mitigate prediction uncertainty. To assess the impact of this loss function in reducing uncertainty, we have illustrated the uncertainty map of the network predictions on two sample data in Figure 4 . Specifically, we conducted training with two settings: the first setting excludes the MultiCURE Module and deep contrastive learning, while the second setting includes them to address uncertainty. Figure 4 indicates that, during the inference process, the network tends to produce lower confidence scores in cases without utilizing the uncertainty aware modules. However, including the MultiCURE Module and deep contrastive modules leads to a more substantial increase in the models' prediction confidence. This observation suggests that incorporating these modules enhances the network's confidence in predicting uncertain voxels, thereby improving overall prediction performance. Additionally, Table 3 has The module's architecture allows the network to adapt its focus based on the characteristics of the input, providing a mechanism to selectively reduce uncertainty in regions where it is most crucial. This adaptability contributes to the observed improvement in the model's confidence, especially in predicting uncertain voxels. The results suggest that the MultiCURE Module enhances the network's ability to handle contextual uncertainties, leading to more confident and accurate predictions. Also, the contrastive loss is employed to guide the learning process. The results in Table 3 reveal that the absence of contrastive© 2024 S.K. , R.A. , A.K. & D.M. . 


Decision: This work innovates in integrating/utilizing uncertainty properties within the Mean-Teacher paradigm. The to key components are the use of a dedicated attention module and the design of a new contrastive loss. The performance of the method was favorably evaluated with state of the art methods on two different datasets (segmentation of the Left atrium from MRI and segmentation of pancreas from CT images).

The strengths of this work are:
1) the two methodological innovations that allow uncertainty to be exploited in the Mean-Teacher formalism
2) the ablation study, which proves the usefulness of each component
3) the comparison of the methods with 7 state of the art methods

The weaknesses of this work are:
1) the lack of comparison of the attention module with more conventional attention strategies
2) the description of the contrastive loss which should be improved
3) the proportion of label and unlabel data used during the experiments is not clearly mentioned in the article 

This study proposes interesting methodological innovations for the development of weakly supervised methods. The authors have responded positively to most of the comments made by the reviewers, which strongly improves the quality of their paper. 

For all these reasons, I have decided to accept this article.

Reviews: review: paper_summary: paper_summary: This paper proposes a semi-supervised contrastive learning approach using uncertainty, extending existing pseudo-labeling methods with confidence score filtering for semi-supervised training.
Two subnetworks are used: The first one identifies uncertain predictions, generating an uncertainty attention map. The second one employs an uncertainty-aware descriptor to refine the representation of uncertain regions, enhancing the accuracy of predictions.
main_review: main_review: The authors responded to most of my comments, except for those in the "Weaknesses" section (on the test cases of the challenge and the statistical tests). The overall quality of the paper was improved. My recommendation remains unchanged (weak accept).
strength_weakness: strength_weakness: The method could be better motivated (more details below).

There is important missing information on the splits.

Why not using the 54 test cases of the challenge to be able to compare against the participants ? Results in the leaderboard seem higher. 

No statistical test is performed.
questions: questions: My comments are rather minor and easy to address I think. 
A better motivation for the main part of reducing uncertainty would help, and fixing minor errors in the document and adding some missing information.
paper_summary: paper_summary: In their paper, the authors tackle semi-supervised learning for 3-D medical data using convolutional neural networks, focussing on improving the representation within the network with different components. First, they propose to use uncertainty-maps combined with a feature descriptor to provide better contextualization especially in the case of uncertain samples. Second, they utilize a contrastive loss function for multi-layer deep supervision within the network to better align the data along class prototypes. They report results on two different 3-D datasets (MRI, left ventricle segmentation and CT, pancreas segmentation) and report improved results compared to the state-of-the-art.
main_review: main_review: I again thank the authors for substantially clarifying multiple aspects in their rebuttal and in the revised version of the paper. The approach presents an interesting approach to force the network to pay more attention to uncertain regions in the image. While I would have liked to see a broader evaluation with different attention-based approaches, I raise my rating to a weak accept.
strength_weakness: strength_weakness: - Parts of the methodology is not clear to me, both from the point of motivation and from the way this is described (see details also below). Specifically I am missing a clear motivation of the need and the intricacies of the "MultiCURE" module and the benefit compared to a "simple" attention mechanism. 
- The mathematical notation and wording does not seem to be fully consistent, with multiple synonyms being used.
- Some comparisons seems to be missing, like the - at least in the introduction prominently mentioned - approach by Shen et al. 2023.
- The experimental setup is not clear: How was the data split between supervised and unsupervised data parts? Did the authors follow Wang et al. in their setup? Did they use the same data splits?
questions: questions: I'd like the authors to answer the following questions:
- what is the benefit of using the MultiCURE compared to an attention mechanism?
- clear out potential misunderstandings with regard to the underlying workings of the proposed method, especially with regard to the uncertainty map and the computation of the contrastive loss
- how exactly was the data employed for semi-supervised learning? How was train and test data handled?
paper_summary: paper_summary: This paper proposes a method to reduce uncertainty and improve 3D medical image segmentation through contrastive learning. They introduce two contributions: (i) an uncertainty-aware feature descriptor module that aims to improve contextual and semantic representations, and (ii) a contrastive learning function employed to reduce the differences between class prototypes and uncertain predictions. Their method is evaluated using k-fold cross validation in two tasks: left atrial and pancreas segmentation. They contrast their approach with various baseline methods and demonstrate a slight enhancement achieved.
main_review: main_review: This paper is well motivated, and the method is extensively evaluated. It is compared against several baselines using k-fold validation. However, code is not available. The availability of the code would strength the transparency and reproducibility of the results.
strength_weakness: strength_weakness: - Choice of datasets is not justified.
- Limitations of the method are not discussed.
- No code available. The availability of the code would strength the transparency and reproducibility of the results.
questions: questions: 1. Justify the choice of datasets. Is this method appropriate for any 3D medical image segmentation? How would it behave to segment small lesions?
2. Comment on the availability of the code.
3. Discuss foreseen limitations and ideas for future work.

Comments: title: respons
respond
Response
Response to Author Comments
Responses provided in order. Many thanks for all the comments.
Response
comment: Thank you for your suggestion. In our approach, we have already provided the mean and variance of our method for each metric, following the exact setting as presented in the MCF paper. This ensures consistency and enables direct comparison with the results reported in the MCF paper. Additionally, it's important to note that while we have not tested our method on the challenge test set, we anticipate that our method, under the same setting, would also produce acceptable results. This anticipation stems from the fact that our method does not rely on any prior conditions or dependencies specific to the dataset.
Thanks for the explanation and the corresponding changes. I do not have further questions.
#### While I understand that experiments on the LA dataset (MRI) and pancreas CT reflect different scenarios, this doesn't address my comment about small lesions (e.g. microbleedings in brain MRI) even when using convolutional neural networks. This could also be mentioned under limitations, if appropriate.
> many thanks for the feedback. In our strategy, we have focused on enhancing the network receptive field size, incorporating our MultiCURE module. This approach indeed extends the receptive field size, particularly beneficial for detecting organs requiring capturing long-range dependencies.
However, it's crucial to acknowledge that while this strategy proves effective for organs with extensive dependencies, it may encounter limitations in scenarios involving tiny lesions, such as microbleedings in brain MRI. These situations demand strong texture representation, which our current approach might not adequately address. o address this concern, we have revised the limitation section of the paper (see appendix) to underscore this aspect.
Thanks to the authors for the revised version.

While I understand that experiments on the LA dataset (MRI) and pancreas CT reflect different scenarios, this doesn't address my comment about small lesions (e.g. microbleedings in brain MRI) even when using convolutional neural networks. This could also be mentioned under limitations, if appropriate.
Thank you for the response and the modifications. 

Two comments that were in the "Weaknesses" section were: 

Why not using the 54 test cases of the challenge to be able to compare against the participants ? Results in the leaderboard seem higher.

No statistical test is performed.
We regret any inconvenience caused by the format of our rebuttal. We acknowledge your suggestion and commit to implementing it in our future rebuttals. Thank you once again for your comments. Below, we have provided our response and also updated the paper accordingly.


#### deformable convolutions: the authors may want add a reference to the paper on Deformable Convolutional Networks if this is the approach that they have used.
> Thank you very much for bringing up this point. We have now included the reference for the deformable convolution paper.

#### Global vs. local: Did the authors investigate the offsets predicted by the deformable convolution?

> In our experiments, we indeed replaced the deformable convolution with the regular convolution and observed a decrease in performance. Conversely, our method utilizing the deformable convolution consistently yielded superior performance. Additionally, we found that the deformable convolution tended to learn larger offsets compared to a 3x3 convolution, enabling better capture of global information. It's worth noting that the offset learned by the deformable convolution also depends on the characteristics of the data and the complexity of the patterns present within it. With this approach, we give freedom for the model to capture local to global information more effectively, rather than limiting the receptive field size.

#### Do I understand it correctly that the uncertainty map is a binary map? Do the authors then simply "add" a constant bias (of 1) for all locations the network is uncertain? I did not find a statement corresponding to "by assigning certain regions as zero ..." in the paper - could you kindly point me to it?
> Yes, you are correct. The uncertainty map is indeed a binary map. In our approach, we define uncertain regions as those where the network's confidence score falls below a specified threshold, assigning them a value of 1. Conversely, regions, where the confidence score exceeds the threshold, are labeled as certain and assigned a value of 0. We have updated section 2.1 to clarify this process.
I thank the authors for providing a revised version. 

I would like to mention that the way that the form of the rebuttal makes it somewhat difficult to follow the argumentation as they structured their reply along the detailed comments and not along the main points to be answered in the rebuttal. It further requires to side by side check the original comments, identify the corresponding answer and the revised manuscript, since many answers are very short. While I understand that OpenReview has a max. character number per message, this is not a restriction set by MIDL, and I have seen this solved in other cases on Open Review by simply posting a two-part response. 

Generally, the revised version of the paper clarifies many of my previously raised questions.

Some aspects the authors may want to further consider/clarify:
- *deformable convolutions*: the authors may want add a reference to the paper on Deformable Convolutional Networks if this is the approach that they have used.
- Global vs. local: Did the authors investigate the offsets predicted by the deformable convolution?
- Do I understand it correctly that the uncertainty map is a binary map? Do the authors then simply "add" a constant bias (of 1) for all locations the network is uncertain? I did not find a statement corresponding to "by assigning certain regions as zero ..." in the paper - could you kindly point me to it?
#### p. 3 "Ho ..
> We have revised the introduction section to further emphasize the limitation of the Shen et al. method in addressing the need for rectifying feature representations in uncertain regions.
#### After .? 
> We have incorporated the results of the Shen approach on the LA dataset into our manuscript. Upon comparison, it is evident that our method outperforms Shen's approach.
#### p. 4: "we ..?
> We have revised the equations accordingly and made the passage clearer for the reader. Specifically, we have introduced the terms "predictive" and "auxiliary" through the definitions to make it explicit.
#### Fig. 1..
> We have updated Figure 1 to ensure consistency with the text's notation throughout the manuscript. In the revised figure, the red and blue points now represent voxel representations belonging to the foreground and background classes, respectively. These representations are employed by the contrastive loss to promote the model's learning of discriminative features between voxels of two classes (e.g. LA and background). This clarification enhances the understanding of how the contrastive loss functions within our proposed method.
#### p. 4: I ..?
> What we intended to convey was not that false predictions generally arise from uncertainty. Instead, we aimed to highlight that inadequate contextual information in regional representation can lead to inaccuracies. These regions often lack sufficient contextual cues, posing challenges for precise predictions. By adaptively adjusting the context representation in these uncertain regions, our method aims to alleviate this uncertainty, thus minimizing false predictions.
#### p. 4: "we ..
> We have updated the paper with additional details. Specifically, we employed the threshold strategy and considered predictions with a confidence level below 0.7 to be uncertain.
#### p. 4/5: "Multi ..?
> Recalibration involves strategically refining the network's focus on crucial features through the adaptive adjustment of the sizes of the receptive fields it examines. The deformable convolution in the third path is designed to dynamically adjust the sampling grid, enabling the model to adaptively capture information from larger spatial extents based on the characteristics of the input data. This approach, especially effective on lower-dimensional feature maps, contributes to a better understanding of the global context. Hence, this is why we refer to it as contributing to global understanding. "Streamlines" refers to multiple pathways within the MultiCURE module that process data concurrently. We replaced "streamlines" with "paths" in the manuscript for improved clarity and readability.
#### p. 5: "To ..
> The uncertain attention map specifically targets areas within the input data where the model exhibits low prediction confidence. By assigning certain regions as zero and preserving the details of uncertain regions, this map focuses on enhancing the representation of areas needing further attention. Integration with the output of a deformable convolution is achieved through a straightforward summation process. In the architecture's first two pathways, the focus is primarily on the regions deemed certain, where the model's predictions are more reliable. However, the path involving the uncertain attention map diverges by prioritizing information considered uncertain. This approach intentionally amplifies the features within these uncertain regions, directing the model toward a better understanding of these areas.
#### p. 6: Eq ..?
> We have carefully reviewed all the equations and ensured that all notations are properly defined throughout the document. 
#### p.6: "first ..
> revised.
#### p. 6: co ..
> In our approach, we integrate the contrastive loss with a supervised loss function, which guides the contrastive loss toward better convergence. As a result, we did not encounter any significant fluctuations or divergence issues during the training process.
#### p. 7/8: Info..
> We have now clarified our evaluation setting regarding k-fold cross-validation.
#### p. 7/8: No 
> Our baseline models, following the MCF (Wang et al.) strategy and using VNet and 3D-ResNet architectures, show low performance with subset-supervised learning, as presented in the MCF paper.
#### Very minor
> Addressed. 
#### what .?
> MultiCURE aims to reduce prediction uncertainty through adaptive feature recalibration, especially in highly uncertain regions. Unlike standard attention mechanisms that enhance features based on relevance, we aim to extend attention not only to object regions but also to shift it to other areas where the model has difficulty classifying. MultiCURE adaptively selects kernels from two paths that focus on certain regions and one path from uncertain regions, all in a soft-attention manner. This approach offers adaptability to variations in organ shapes and ensures the precision required for boundary delineation.
#### clear ..
> see above.
####  how .?
> see above.
#### 1. Justify the choice of datasets. Is this method appropriate for any 3D medical image segmentation? How would it behave to segment small lesions?
> Thank you for bringing up this important aspect. Our method is intended to be a general approach applicable to a wide array of 3D medical imaging datasets, rather than being tailored to specific ones. Our experiments on both MRI (The LA dataset) and CT (Pancreas) volumes serve to demonstrate the effectiveness of our approach across different imaging modalities. These datasets represent challenging organ segmentation tasks in diverse anatomical regions and imaging modalities. By evaluating our method on such diverse datasets, we aimed to showcase its versatility and effectiveness across various modalities and challenging segmentation scenarios. It's worth noting that our method leverages convolutional layers with adjustable receptive field sizes, allowing for accurate localization of objects ranging from small to large sizes.

#### 2. Comment on the availability of the code.
> We have now included a link to the publicly available GitHub repository hosting our code.

#### 3. Discuss foreseen limitations and ideas for future work.
> We have included a new section "Limitation" in the appendix to address the limitations of our proposed method.
#### 1. As a general comment, I would motivate better why trying so much to reduce uncertainty. Border voxels are naturally uncertain, as reflected by inter- and intra-observer variability. The related works motivate the need for reducing uncertainty of pseudo-labels. And your motivation "However, the issue of enhancing model representation and rectifying feature representation in uncertain regions remains unresolved." is not evident to me.
> Thank you for your insightful comment. We have carefully revised the introduction section to better present the motivation behind our proposed method, specifically addressing the limitations of existing literature.

#### 2. "consists of 1003 3D" -> 100, not 1003 !
> Thank you for pointing that out. We have corrected the number of samples accordingly.

#### 3. "We employed k-fold cross-validation." Missing important information: what is k? Are the k-1 folds split into train/validation and with which percentage? I assume it is the same splits as in (Wang et al, etc.), but you need to clarify this.
> Thank you for highlighting this important point. We have now clarified our evaluation setting regarding k-fold cross-validation. Specifically, we follow the methodology of Wang et al. [1], utilizing 5-fold and 4-fold cross-validation for the LA and Pancreas datasets, respectively. Each fold is split into train/validation sets with the same percentages as described in the [1] method.

#### 4. Clarify if you report standard deviation or standard error in Tables 1 and 2. And report it also in Table 3.
> Thank you for the feedback. We have revised the captions of the tables and emphasized this by including "(average ± standard deviation)."

#### 5. Make sure to define all notations (eg S\_k, Eq. 5)
> We have carefully reviewed all the equations and ensured that all notations are properly defined throughout the document. 

#### 6. Some typos, eg "from different level"
> We thoroughly checked the manuscript to correct any typo errors.

#### 7. What is the link with you arxiv paper "Leveraging Unlabeled Data for 3D Medical Image Segmentation through Self-Supervised Contrastive Learning" ? Why are these results not included/compared with?
> Thank you for your inquiry. Our recent arXiv paper [2] proposes a distinct strategy for enhancing medical image segmentation on 3D medical images. As this paper has not been formally published yet, we have not directly compared it with our current approach. However, it's worth noting that our current method outperforms the approach proposed in [2] on the LA dataset and demonstrates comparable performance on the pancreas dataset. While we acknowledge the relevance of our arXiv paper, we focused our comparisons on published methods to ensure a comprehensive evaluation against existing state-of-the-art techniques.

References: 
###### [1] Zhiqiang Shen, Peng Cao, Hua Yang, Xiaoli Liu, Jinzhu Yang, and Osmar R Zaiane. Co-training with high-confidence pseudo labels for semi-supervised medical image segmentation. Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, 2023.
###### [2] Karimijafarbigloo Sanaz, Azad Reza, Velichko Yury, Bagci Ulas, Merhof Dorit. Leveraging Unlabeled Data for 3D Medical Image Segmentation through Self-Supervised Contrastive Learning. arXiv:2311.12617, 2023.