Active Learning with the nnUNet and Sample Selection with Uncertainty-Aware Submodular Mutual Information Measure

Authors: bernhard  föllmer
kenrick  schulze
christian  wald
sebastian  stober
wojciech  samek
marc  dewey
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 14:14:22

Abstract: Annotating medical images for segmentation tasks is a time-consuming process that requires
expert knowledge. Active learning can reduce this annotation cost and achieve optimal
model performance by selecting only the most informative samples for annotation. However, the eectiveness of active learning sample selection strategies depends on the model
architecture and training procedure used. The nnUNet has achieved impressive results in
various automated medical image segmentation tasks due to its self-configuring pipeline
for automated model design and training. This raises the question of whether the nnUNet
is applicable in an active learning setting to avoid cumbersome manual configuration of
the training process and improve accessibility for non-experts in deep learning-based segmentation. This paper compares various sample selection strategies in an active learning
setting in which the self-configuring nnUNet is used as the segmentation model. Additionally, we propose a new sample selection strategy for UNet-like architectures: USIM - Uncertainty-Aware Submodular Mutual Information Measure. The method combines
uncertainty and submodular mutual information to select batches of uncertain, diverse,
and representative samples. We evaluate the performance gain and labeling costs on three
medical image segmentation tasks with different segmentation challenges. Our findings
demonstrate that utilizing nnUNet as the segmentation model in an active learning setting is feasible, and most sampling strategies outperform random sampling. Furthermore,
we demonstrate that our proposed method yields a significant improvement compared to
existing baseline methods.

Main paper: 
 introduction: 
 Segmentation of tumors and surrounding anatomy is an important task in medical imaging for cancer diagnosis using CT and MRI (Hesamian et al., 2019) . However, training these models requires a large number of voxel-wise labeled images. Annotating such datasets is usually time-consuming and requires expert knowledge (Ren et al., 2022) . Active learning (AL) can address this issue by iteratively selecting only the most informative samples (batches) from the unlabeled dataset for annotation and training, while still achieving near-optimal model performance. However, the performance of AL sampling strategies depend heavily on the model architecture and training procedure used (Ji et al., 2023) . To analyze the gain of AL strategies, it is important to ensure robustness and reproducability under dierent experimental conditions (Munjal et al., 2022) . The performance gain and labeling cost reduction of AL strategies depend on the following factors. First, the training pipeline including data preprocessing (normalization, augmentation), model architecture (patch size, number of layers), and training procedure (batch size, learning rate scheduling). Second, the dataset characteristics such as dataset size, class imbalance, diculty of the learning task. Third, the labeling budget during each sampling round. The nnUNet (Isensee et al., 2021 ) is a self-conguring pipeline, which allows for the training of 2D and 3D UNet-like models (Ronneberger et al., 2015) without the need for manual adaptation of preprocessing, model architecture, or hyperparameters and has won many segmentation competitions (Antonelli et al., 2022) . The nnUNet identies robust design decisions based on multiple tasks (Isensee et al., 2021) , ensuring reproducibility and robustness in evaluating AL strategies (Munjal et al., 2022; Burmeister et al., 2022) by automatically conguring preprocessing, model architecture, and hyperparameters. This paper investigate whether the use of a self-conguring training pipeline in an AL setting can reduce annotation costs. This would also increase reproducibility and could facilitate the use of active learning for medical image segmentation. Most deep AL strategies that combine uncertainty, diversity and representativeness in their sampling objective have been developed for classication tasks, and less frequently for segmentation tasks (Saidu and Csató, 2021) . Novel AL strategies such as BADGE (Ash et al., 2020) select samples based on diverse gradients where gradient length captures the uncertainty. We explore the adaptation of gradient-based active learning methods to U-Net-like architectures and segmentation tasks with high-dimensional annotations. To eciently combine predictive uncertainty and gradient based sample representation, we propose USIM -Uncertainty-aware Submodular mutual Information Measure. This approach combines predictive uncertainty-based sampling with diversity and representative sampling in parameter space using submodular mutual information measures (Kothawade et al., 2021 (Kothawade et al., , 2022a)) . The novelty of our approach lies in the selection of the query set based on class-weighted predictive uncertainty using Monte Carlo Dropout and the estimation of representative gradient embeddings based on the bottleneck layer (USIMC) as well as an automated gradient embedding selection based on the Fisher information (USIMF). Our contributions are twofold: First, we evaluate various active learning strategies using the nnUNet pipeline as the segmentation model in an AL setting. Second, we propose and evaluate USIM, an AL strategy that combines predictive uncertainty with diversity and representativeness using a submodular mutual information measure. 
related work: 
 Two main principles for selecting batches of informative samples are uncertainty-based sampling and representation-based sampling. Various uncertainty-base approaches exist (Pratapa et al., 2011; Li and Alstrøm, 2020) , such as Monte Carlo dropout-based methods (Gal and Uk, 2016; Kendall et al., 2017) , Bayesian neural networks (Gal et al., 2017) , or ensemble methods (Chitta et al., 2018) . However, methods that rely solely on uncertainty are not suitable for large datasets with redundant information. Representation-based sampling methods model the representativeness and diversity of samples within a batch. For example, the Core-set approach (Sener and Savarese, 2018) estimates distances between samples modeled by the Euclidean distance between feature vectors. Uncertainty and representation-based methods have been used for classication and segmentation tasks (Burmeister et al., 2022) . Hybrid methods aim to combine uncertainty and diversity in their sampling objectives to select informative samples while avoiding redundancy (Yang et al., 2017; Nath et al., 2021) . These methods rely on compact image representation which are typically extracted from the last layer of networks for classication tasks such as BADGE (Ash et al., 2020) . Sreenivasaiah et al. investigated adaptation of this method for segmentation tasks (Aklilu and Yeung, 2022) . MEAL (Sreenivasaiah et al.) extracts embeddings based on Uniform Manifold Approximation to model representativeness of image patches for image segmentation. In addition, semi-supervised approaches oer additional solutions to reduce labeling costs (Mittal et al., 2023; Gaillochet et al., 2022) . Recent active learning strategies make use of the Fisher information ratio, Hessian, or similarity matrices but were mainly developed for classcation task (Kirsch and Gal, 2022; Kothawade et al., 2021; Ash et al., 2020 Ash et al., , 2021;; Liu et al., 2021) and seldom for segmentation tasks (Al, 2019; Yu et al., 2023) . We explicitly compare practical and user-friendly methods and exclude methods that require design of additional models to evaluate informativeness of unlabeled samples, such as VAAL (Sinha et al., 2019) or adaptation of the loss function. We further exclude methods that require additional sub-networks with trainable parameters that might inuence the automated architecture conguration of the nnUNet due to memory restrictions (Yoo and Kweon, 2019) . 
active learning and submodular subset selection: 
 Active learning is the process where the learning algorithm attempts to maximize a model's performance gain while annotating the fewest samples possible (Ren et al., 2022) .To evaluate AL sampling strategies for medical image segmentation, the dataset is split into training and test set. From the training data, we construct an unlabeled pool X U = {x (j) } U j=1 of patches x ∈ R C×H×W with patch width W , height H and number of channels C. We randomly select and label a small initial labeled dataset X L = {(x (j) , y (j) )} L j=1 with segmentation masks y ∈ R K×H×W (K -number of classes) to train the segmentation model. During multiple sampling rounds, we use one of the sampling strategies to select a batch X B ⊆ X U of B most informative samples, simulate annotation and add the annotated samples to the labeled dataset. Since retraining from scratch is very time consuming, we ne-tune the model in each AL round and evaluate the model using the test set.3.1.2. Submodular mutual information measure for subset selection Submodular functions are a class of functions that can be used to model guided (uncertaintyguided in our approach) data subset selection of representative subsets (Kothawade et al., 2022a) . The submodular mutual information (SMI) function is given byI f (X B ; X Q ) = f (X B ) + f (X Q ) -f (X B ∪ X Q ), where X Q is the query set (target set), X B is the selected subset of patches from the unlabeled data set and f : 2 X → R a submodular set function dened as f (X) = x i ∈Ω max x j ∈X S ij . The term S ij measures the similarity between the elements x i and x j . The instantiated submodular functionI f (X B ; X Q ) is used to select a subset that maximizes the submodular function X B ← argmax X B ⊆X U ,|X B |≤B I f (X B ; X Q ).Intuitively, SMI models the similarity between X Q and X B , and maximizing SMI will select points similar to X Q while being diverse. We use the mutual information function variant for the facility location (FL) function dened over X Q (FLQMI) shown in Equation 1. The variant was successfully applied in Kothawade et al. (2022a) and is memory and time ecient. The parameter η balances query-relevance and diversity. Kothawade et al. (2022b) and Beck et al. (2024) performed a qualitative an theoretical analysis with respect to querycoverage, query-relevance and diversity, and showed that as soon as η is increased, the summary produced by FLQMI becomes more query-relevant and less diverse.I f (X B ; X Q ) = x i ∈X Q max x j ∈X B S ij + η x i ∈X B max x j ∈X Q S ij (1)We further investigate the robustness of our method in a redundancy scenario to conrm the superiority of the submodular information measure for subset selection compared to sampling with a uniform distribution over the query set in Appendix E.We rst run the nnUnet preprocessing and planning function on the raw dataset to generate a dataset ngerprint and automatically congure model architecture and training procedure.Based on the automatic congured parameters (patch size, resampling parameter, etc.) we extract overlapping image patches to create the unlabeled pool of samples. We label a small randomly selected subset and run the self-conguring nnUNet pipeline again to re-congure a subset of parameters for resampling, intensity normalization and train the initial model. We used the trained model to select the most informative batch of samples using one of the compared sample selection strategies. After annotation, we rerun the nnUNet pipeline again to update the dataset ngerprint and planing conguration, and netune the model. Since the characteristics of the image data remain the same and only the annotations change, most model congurations, such as model architecture and learning rate, remain unchanged. The only changes that might occur are annotation resampling and intensity normalization. Sampling, annotation and self-congured model netuning is repeated in multiple AL round.The proposed AL framework is shown in Figure 1 . 3.3. USIM -Uncertainty-aware submodular mutual information measure for subset selectionWe propose USIM, a sample selection strategy that combines predictive uncertainty and submodular mutual information measure for subset selection with UNet-like architectures.To measure the informativeness of a sample, predictive uncertainty using Monte Carlo dropout is a powerful and simple method (Li and Alstrøm, 2020) . In our work, the uncertainty of the unlabeled samples is estimated using Monte Carlo dropout and the model θ k-1 trained in the previous round. A subset of Q uncertain samples X Q is sampled (with replacement) from the unlabeled dataset with sampling probability proportional to a weighted uncertainty p(θ k-1 , x) = c wc•U ncert(θ k-1 ,x,c) x∈X U c wc•U ncert(θ k-1 ,x,c) . The uncertainty of a foreground class c is weighted by w c which is the inverse of the number of annotated voxels of that class in the training set. The uncertainty-based weighting ensures that the performance is less biased towards classes where more data is presented. We analyze the inuence of query set size Q in Appendix D. In our experiments, Q is estimated using the elbow method (Thorndike, 1953) which plots the sample uncertainties in descending order and takes the elbow of the curve as the number of uncertain samples in the data set.To create an informative batch of samples, it is important to consider not only uncertainty but also diversity and representativeness. Since we select the query set X Q by sampling from the unlabeled dataset with a probability distribution proportional to the class-weighted uncertainty, the query set includes not only most uncertain but more diverse samples. To conrm that the proposed sampling strategy of the query set is more ecient than standard active learning (query set is the unlabeled dataset) (Kothawade et al., 2021) or sampling the most uncertain samples from the unlabeled dataset, we compare the performance gain for the methods in Appendix D. In our approach, we use mutual information function variant for the facility location (FL) function (FLQMI) show in Equation 1. The method requires a similarity matrix S uq ∈ R |X U |×|X Q | between samples of the unlabeled dataset and the query set and can be constructed in many ways, e.g. by computing the cosine distance (Kothawade et al., 2022a) , Euclidean distance (Neven and Goedemé, 2023) or Fisher information kernel. We analyze two variants of similarity matrices using cosine similarity between gradient embeddings of the bottleneck layer dened as USIMC and an approximation of pairwise inuence between samples using the Fisher kernel which is dened as USIMF.Details about the construction of the similarity matrices can be found in Appendix A. After construction of the similarity matrix, we instantiate the submodular function I f (X B ; X Q ) and use stochastic greedy method to select the batch X B for labeling. We use the stochastic greedy optimizer from SUBMODLIB (Kaushal et al., 2022) because it has a provably linear running time independent of the budget and is faster than the naive greedy approach. We summarize the proposed USIM method in Algorithm 1.Algorithm 1: USIM -Uncertainty-Aware Submodular Mutual Information Measure 1: Input: Initial labeled dataset: X L , unlabeled dataset: X U , initial self-congured nnUNet model parameter: θ 0 , batch size: B, number of sampling rounds:K 2: for k = 1, 2, ..., K do 3: Estimate sample uncertainty U ncert(θ k-1 , x) ∈ R for all unlabeled samples x ∈ X U 4:Sample a query subset X Q ⊆ X U with sample probability proportional to weighted uncertainty 5:Compute similarity matrix S uq with Equation 2 for USIMC or 4 for USIMF 6:Instantiate the submodular function I f (X B ; X Q ) based on S uq (Equation 1) 7:Use stochastic greedy method to select batch X B withX B ← argmax X B ⊆X U ,|X B |≤B I f (X B ; X Q ) 8:Query the oracle to obtain segmentation masks y(x), ∀x ∈ X B 9:X L ← X L ∪ X B ; X U ← X U \ X B 10: Train model on X L : θ k = arg min θ E X L [l(x, y; θ)]11: end for 12: return Final model θ K 
datasets: 
 We evaluate the AL sampling methods on three medical image segmentation datasets from the Medical Segmentation Decathlon (Antonelli et al., 2022) to ensure method evaluation on dierent target regions, modalities and challenging features. For our analysis, we choose the labeled training set of 1) Spleen dataset, as a small dataset of 41 CT scans with a large ranging foreground size, 2) Liver dataset, a large dataset of 131 CT scans with label imbalance between large liver class and small tumor class and 3) Hippocampus dataset consisting of 260 MRI scans with the challenge to segment two adjacent small structures with high precision 4.2. Active learning sampling strategies and implementation Sampling Strategies: We compare the following sampling method in our analysis: (1) Random Sampling, (2) Max Entropy (Shannon, 1948) , (3) Mean STD (Kendall et al., 2017) , (4) BALD (Gal et al., 2017) , (5) Core-Set (Sener and Savarese, 2018) , ( 6) BADGE(LL) (Ash et al., 2020; Aklilu and Yeung, 2022 ), (7) Stochastic Batches (Gaillochet et al., 2023) and ( 8) USIMC (ours), ( 9) USIMF (ours).Implementation: For our experiments, we used 2D nnUNet congurations because they have a higher training speed and are less prone to overtting. However, the proposed USIM method can easily be extended to 3D models and will be studied in further research. For all our experiments, we set η = 1.0 and analyzed the inuence of the hyperparameter in Appendix D. The used hardware congurations and self-congured hyperparameters can be found in Table 1 in Appendix F. The code, for training and evaluation is available at https://github.com/Berni1557/ALUNET. Further details about sampling strategies and implementation can be found in Appendix C.The performance gain in terms of dice score with respect to the number of annotated samples for all competing strategies is shown in Figure 2 .Spleen: All sampling strategies except Core-Set outperformed random sampling. However, the segmentation task is less challenging and the performance dierences between the strategies are therefore small. With less than 8% of annotated data, USIMF achieved a similar dice score as trained with fully annotated dataset.Liver: Uncertainty-based methods (BALD, Mean STD) performed better than random sampling. USIMF and USIMC outperformed most methods and reach near optimal performance with less than 5% of the data.Hippocampus: USIMC and USIMF outperformed all other sampling strategies and reach performance of the fully annotated dataset with roughly 30% of the annotated data. All other strategies outperformed or performed similar to random sampling. The pairwise penalty matrix (Ji et al., 2023) in Figure 2 (D) aggregates results over all conducted experiments. The overall performance Φ is measured by the column-wise average, where lower numbers indicate a higher-performing algorithm. The results show that USIMC and USIMF have the lowest column sum, indicating that they outperform other strategies. The Figures 10, 11, and 12 in Appendix G show examples of AL based segmentation results for all three datasets. Figure 3 shows a t-SNE plot of the USIMF gradient embeddings on the Liver dataset.  
discussion and conclusion: 
 In this paper, we have investigated the utility of the nnUNet as a self-conguring model in an AL setting to reduce labeling costs. We additionally proposed USIM, an AL strategy that combined predictive uncertainty and submodular mutual information measure to select informative, diverse, and representational batches with two similarity matrix variants (USIMF and USIMC). The experiments conrmed that using the self-conguring nnUNet pipeline in an active learning setting is an eective strategy for reducing labeling costs and facilitating AL by avoiding the cumbersome conguration of the training process. All methods performed equally or better than random sampling. We showed that methods based on uncertainty (BALD, STD MEAN) outperformed those based on representation (Core-Set). Our hybrid method outperformed the compared active learning methods, with USIMF performing slightly better than USIMC. The proposed method was evaluated only on 2D nnUNets, but it can be extended to 3D approaches. However, we refrained from conducting the evaluation due to longer training times, leaving it for future research. Further research is necessary to prove the eectiveness of active learning for medical image segmentation in real scenarios, rather than relying solely on simulations. To investigate the relationship between the segmentation error and the uncertainty measured by the gradient length and prediction uncertainty, we also performed a correlation analysis during the rst sampling round of the Hippocampus dataset. We analyzed the segmentation error (total number of misclassied pixels) and predictive uncertainty, as well as the gradient length (L2 norms) of the bottleneck layer on a randomly selected subset of the unlabeled dataset. The correlation coecients shows that for segmentation tasks with the nnUNet, the predictive uncertainty estimated using the Monte Carlo dropout correlates more strongly with the segmentation error (r = 0.84) than with the gradient length (r = 0.68), indicating that predictive uncertainty is a better measure of uncertainty. Query set size: To analyze the inuence of the size of the query set Q, we compare the performance gain with respect to Q in Figure 7 for USIMF. For the Hippocampus dataset, we set Q ∈ {200, 400, 1600, U } with U being the size of the unlabeled dataset and compare it with automated estimation of the query set size (USIMF). The query set size for the USIMF method which was estimated with the elbow method (Thorndike, 1953) was Q = 942 (942/7662) and Q = 1470 (1470/7562) for the rst and second round, respectively. For the Liver dataset, we set the query set size Q ∈ {200, 400, 800, 1600, 10000}. For the USIMF method, the estimated query size was automatically set to Q = 6606 (6606/104133) and Q = 6852 (6852/104033) for the rst and second rounds, respectively. As the query set is sampled with replacement, we assume that the overall uncertainty of the batch should remain similar. It can be observed that the performance gain is not signicantly aected by the size of the query set.  
 
 Appendix:  


Decision: The paper introduces an active learning (AL) strategy for medical image segmentation using the nnUNet model, focusing on reducing annotation costs. The approach incorporates submodular measures of mutual information for sample selection and leverages the pre-processing capabilities of nnUNet to optimize hyperparameters for each segmentation task automatically. While the approach shows promise in improving segmentation performance, reviewers raise concerns about methodological novelty, evaluation, and clarity in certain aspects.

While the paper presents a promising approach for reducing annotation costs in medical image segmentation, addressing the weaknesses highlighted by reviewers, such as conducting a more thorough evaluation, clarifying methodological choices, and comparing with existing approaches, would strengthen its contribution and impact in the field.

Reviews: review: paper_summary: paper_summary: The paper addresses the high annotation cost in image segmentation tasks through an active learning strategy. To this end, the paper proposes to
- evaluate the feasibility of a self-configuring nn-UNet  network.
- guide the active learning sampling strategy with a measure combining a monte-carlo uncertainty from (Gal 2016, 2017) and  the diversity and representative measures from  (Kothawade et al., 2021, 2022). The later is computed on parameter space with a submodular mutual information SMI measure.
main_review: main_review: Several aspects of the paper have been clarified, in particular the novelty. Although there are some limitations regarding the scope of the evaluation, new results presented after the rebuttal should ensure interesting discussions at MIDL
strength_weakness: strength_weakness: - The state of the art and evaluation do not discuss pseudolabeling strategies, which have been largely used in the field to tackle the cost of annotations.
- Appart of using gradient embeddings for segmentation, are there other novel aspects?
- The evaluation is done only with 2D models and single organ tasks.
- The arguments behind individual method choices are not clear and not always in phase with the introduction motivations.
- The proposed pipeline proposes to reconfigure the network each round of sampling which could raise a convergence issue.
- The practicality of Active Learning remains to be proven in real scenarios, specially how to determine a stopping criteria when GT is really missing?
questions: questions: Comments
- The number of initial samples seems realistic but going up to 900 or to 2000 as it is done for some experiments seems unreasonable for an AL scenario
- Gradient embeddings is mentioned to be promising as it works for classification, it was not discussed why is it interesting in the case of segmentation
- The stochastic greedy method in step 7 of the algorithm was not explained
- clearly state in the related work which methods have been used for segmentation and which for classification only.



Questions
- The method extracts overlapping patches. Why is this the case? can there be an overlap between a labelled and an unlabelled example?
- Why relying on Monte-Carlo dropout to compute the uncertainty which requires several evaluations of the method. Why not using the softmax predictions to compute the uncertainties?
- Why are the results in Fig 2 not complete for other methods? Only the curves of SMI methods go beyond a number of samples
- How do evaluated methods compare in terms of computational complexity and memory? 
 
Typos
“samples, thEIr method is using gradient embeddings”
“training of additional models to evaluate informativeness of unlabeled sampleS, such”
paper_summary: paper_summary: The paper presents a active learning (AL) approach for medical image segmentation based on the nnUNet model and sample selection with submodular measures of mutual information. This approach exploits the pre-processing and planning capabilities of nnUNet to select optimal hyperparameters for each segmentation task, in an automated way. The active learning strategy used in the approach follows the SIMILAR method (Kothawade et al., 2021), modifying this method in two ways: 1) applying it to segmentation instead of classification, 2) selecting the query set based on Monte-Carlo dropout uncertainty estimation. Experiments on three segmentation tasks/datasets (Spleen, Liver and Hippocampus) show the proposed method to outperform several recent AL algorithms.
main_review: main_review: The authors have added several ablation experiments showing the benefit of their query sample selection strategy, as well as the impact of query set size and hyper-parameter eta. Given these improvement, I upgrade my score to weak accept (however, not strong accept since I still believe that the proposed method is mainly an extension of SIMILAR to segmentation).
strength_weakness: strength_weakness: * One of the main weaknesses of the paper is its limited methodological novelty. The proposed method mostly adapts the SIMILAR algorithm to segmentation using nnUnet as base network. The uncertainty aware selection of the query set is interesting, however it can be seen as a variant of the one used in (Kothawade et al., 2021) for rare classes, and the usefulness of this strategy is not properly evaluated in the experiments.

* Given that one of the main differences is the selection of the query set, I expected a more thorough validation of this strategy. Currently, it is hard to tell if this strategy brings any benefit.
questions: questions: Clarify the novel contributions wrt SIMILAR and evaluated (if possible, else discuss) the impact of the query set selection strategy.
paper_summary: paper_summary: This paper introduces USIM (Uncertainty-Aware Submodular Mutual Information Measure), a novel sample selection strategy tailored for UNet-like models. USIM is designed to select batches of uncertain, diverse, and representative samples under active learning settings. The experiments confirms that the USIM approach outperforms traditional methods in three distinct medical image segmentation tasks.
main_review: main_review: The USIM method introduced in this paper appears to be innovative. Experiments demonstrate that USIM exceeds the performance of traditional methods in three separate medical image segmentation tasks. I believe that providing a clearer explanation of the problem mentioned earlier would enhance the overall presentation.
strength_weakness: strength_weakness: I understand the general concepts of the methods, but I struggle with some specifics of the model. The authors argue that their method excels by selecting batches of samples that are uncertain, diverse, and representative. Yet, I find a lack of both quantitative and qualitative evidence to demonstrate that the samples indeed meet these criteria.
questions: questions: 1. Why does maximizing the SMI function ensuresthe sampling of diverse and relevant samples with respect to the query set?

2. Why is Equation 1 submodular? Why does this function measure uncertainty?

3. Are there quantitative and qualitative evidence to demonstrate that the samples selected are uncertain, diverse, and representative?

Comments: title: Thank you very much for your constructive comments.
Thank you so much for your constructive comments, please see the detailed point-by-point response below.
Thank you very much for your detailed and constructive feedback.
comment: **Weaknesses:**

> I understand the general concepts of the methods, but..

In order to confirm our hypothesis that our selected samples are indeed uncertain, diverse, and representative, we conducted additional experiments in Appendix D.
To verify that the samples are uncertain, we analyzed the uncertainty estimated using Monte Carlo dropout. We compared the overall uncertainty of the selected batches between all sampling strategies in Appendix D, Uncertainty. 
One big advantage of our subset selection strategy is the robustness with respect to sample redundancy in the unlabeled dataset. To confirm that the samples are diverse and representative, we performed an experiment to analyze their behavior under strong sample redundancy in the unlabeled dataset in Appendix D, Redundancy.
As described in Kothawade et al., (2022), the FLQMI function can balance query-relevance on one hand and query-coverage and diversity on the other hand by adjusting the weighting parameter $\eta$. The influence of $\eta$ with respect to the performance gain was studied in Appendix D, Weighting parameter $\eta$, and we have clarified this in the text and updated equation (1).

> 1. In section 3.1.2, the authors claim... 

To confirm that the selected samples are indeed diverse, we study the performance gain under strong sample redundancy in the unlabeled dataset in Appendix D, Redundancy. It shows that the SMI function selects less redundant samples than uniform sampling from the query set, which is a major advantage of our method.
The FLQMI function has the ability to balance query relevance on the one hand and query coverage and diversity on the other hand by changing a weighting parameter $\eta$. We have clarified this in the text and updated Eq. (1) with the weighting parameter $\eta$. We also study the influence of the weighting parameter in Appendix D. We refer the interested reader to Beck et al. (2024) for a more theoretical analysis of the submodular mutual information measure and the FLQMI function. Diversity is further supported by sampling the query set with a probability distribution proportional to the sample uncertainty instead of sampling the most uncertain samples.

> 2. The authors seem to claim the Equation (1) ...

We apologize for the confusion in the text, and we restructured the section to present the general idea of submodular mutual information measure for subset selection in section 3.1.2 and highlight our adaptations and contributions in section 3.3.
The submodular mutual information (SMI) function is given by 
$I_f(X_B ; X_Q)=f(X_B)+f(X_Q)-f(X_B \cup X_Q)$, where $X_Q$ is the query set $X_B$ is the selected subset and $f: 2^{X} \rightarrow \mathbb{R}$ a submodular set function defined as
$f(X)=\sum_{x_i \in \Omega}\max_{x_j \in X} S_{i j}$.
The Equation (1) is a mutual information function variant (FLQMI) for the facility location (FL) function defined over the query set $X_Q$. Intuitively, the function measures the similarity between the query set $X_Q$ and the selected subset $X_B$. 
The function does not measure uncertainty. However, the query set consists of samples with high predictive uncertainty. These samples are sampled from an unlabeled dataset with a probability distribution proportional to predictive uncertainty. Therefore, we assume that samples selected in $X_B$ should also have high uncertainty. We confirmed this hypothesis by comparing all sampling strategies with respect to the overall uncertainty of selected batches in Appendix D. Kothawade et al. (2022) analyzed in detail several submodular information measures for guided subset selection.

> 3. The authors argue that their method excels by selecting batches of samples that are uncertain, diverse, and representative. Yet, I find a lack of both quantitative and qualitative evidence to demonstrate that the samples indeed meet these criteria.

We performed additional experiments to confirm that selected batches are uncertain, diverse, and representative. In Appendix D, we compare the batch uncertainty of the sampling methods. In addition, we analyzed the behavior under sample redundancy in the unlabeled dataset to demonstrate that the sampling methods selected samples that are diverse and representative. 
Moreover, as described in Kothawade et al., (2022), the FLQMI function has the ability to balance query relevance on the one hand and query coverage and diversity on the other hand by changing a weighting parameter $\eta$. We have clarified this in the text and updated Eq. (1) with the weighting parameter $\eta$. We study the influence of the weighting parameter in Appendix D.

Questions To Address In The Rebuttal: 

> 1. Why does maximizing the SMI function...?

See answer above.

> 2. Why is Equation 1 ...

See answer above.

> 3. Are there quantitative and qualitative evidence...

See answer above.
**Weaknesses:**
> One of the main weaknesses of the paper is its limited methodological novelty. The proposed method mostly adapts the SIMILAR algorithm to segmentation using nnUnet as base network. The uncertainty aware selection of the query set is interesting, however it can be seen as a variant of the one used in (Kothawade et al., 2021) for rare classes, and the usefulness of this strategy is not properly evaluated in the experiments. 

We appreciate the reviewer for bringing to our attention the need for clarification regarding the novelty of our method. 
Our contributions are twofold. Firstly, we select the query set based on the weighted predictive uncertainty using Monte Carlo dropout. To further evaluate this strategy, we performed additional experiments and compared our method with selecting the query set based on a uniform distribution over the unlabeled dataset (standard active learning). In addition, we compared our strategy to selecting the most uncertain samples from the unlabeled dataset as the query set. The results of the experiments are presented in Appendix D and show that our query set selection strategy is more efficient. 
Secondly, SIMILAR was developed and evaluated solely for classification tasks, using last layer gradients as feature embedding. However, for U-Net like architectures, the last layer gradients do not guarantee to provide an accurate representation for estimating similarities between samples. 
We analyzed two variants where the similarity matrix is estimated based on the cosine similarity of the gradients of the bottleneck layer, and a variant where the gradients of the decoder are selected based on the Fisher information measure.
We further evaluate our method in a redundancy scenario in Appendix E to confirm the superiority of the sampling method compared to uniform selection from the query set..

> Given that one of the main differences is the selection of the query set, I expected a more thorough validation of this strategy. Currently, it is hard to tell if this strategy brings any benefit.

We agree that the selection of the query set is novel. The query set was chosen by sampling from a probability distribution over the unlabeled dataset, proportionally to the weighted predictive uncertainty. In Appendix D, Query set selection, we conducted a more comprehensive validation by selecting samples with a uniform probability distribution over the unlabeled dataset (standard active learning) and choosing the most uncertain samples.

> p2: thr method --> our method ?

Thank you, we corrected the typo.

**Detailed Comments:**

> p2: "we propose and evaluate USIM, an AL strategy that combines predictive uncertainty with diversity and representativeness in the parameter space, using a submodular mutual information measure." Authors should mention explicitly the novel contributions w.r.t. SIMILAR

Thank you for your comment. We have clarified the novelty of our method in the introduction, as described above.

> What is the size of the query set, and does this size affect performance? What is the performance if using X_Q = X_U as in the default AL scenario of SIMILAR?

Thank you for your comment. In Appendix D, Query set size, we performed additional experiments to analyze the performance gain of our method with respect to the size of the query sets and could show that the query size does not affect the performance by a large margin. We also can assume that the overall uncertainty of the batch should remain similar since we select samples from the unlabeled dataset with replacement. In our experiments we estimated the size of the query set with the simple elbow method.

> Algorithm 1: Do you allow selection samples from X_Q. If so, isn't Eq (1) maximized by setting X_B ~= X_Q ? 

Yes, we allow selection of samples from $X_Q$ and we changed the text to make this clear. In our experiments, the size of the query set $X_Q$ was always larger than the sampling budget. We made this clear in the text.

> p6: encode was --> encoder was

Thank you, we corrected the typo.

**Questions To Address In The Rebuttal:**

> Clarify the novel contributions wrt SIMILAR and evaluated (if possible, else discuss) the impact of the query set selection strategy.

Thank you for your comment. We have clarified the novel aspects and contributions in the introduction and evaluated the influence of query set selection strategy through various experiments in Appendix D. These experiments analyzed the influence of the query set selection strategy and compare it with selecting the query set based on a uniform distribution over the unlabeled dataset (standard active learning) and selection of most uncertain samples. We further analyze batch uncertainty over sampling strategies, influence of the query set size and influence of the weighting parameter $\eta$.
We additionally analyze exemplary the robustness with respect to redundancies in the unlabeled dataset in Appendix E.
**Weaknesses**: 

> The state of the art and evaluation do not discuss pseudolabeling strategies...

We have revised the state-of-the-art section and added these strategies.

> Appart of using gradient embeddings...

We revised the manuscript and clarified the novelty of our method. 
Our novel contributions are twofold. 
Firstly, we select the query set based on the weighted predictive uncertainty using Monte Carlo dropout which can be seen as a variant of SIMILAR for rare classes. To evaluate this strategy, we compared it with selecting the query set based on a uniform distribution over the unlabeled dataset (standard active learning). In addition, we compared our strategy to selecting the most uncertain samples from the unlabeled dataset as the query set. The results of the experiments are presented in Appendix D and show that our query set selection strategy is more efficient. 
Secondly, SIMILAR was developed and evaluated solely for classification tasks, using last layer gradients as feature embedding. However, for U-Net like architectures, the last layer gradients do not guarantee to provide an accurate representation for estimating similarities between samples. We propose two variants where the similarity matrix is estimated based on the cosine similarity of the gradients of the bottleneck layer, and a variant where the gradients of the decoder are selected based on the Fisher information measure.

> The arguments behind individual method choices..

We added more experiments to analyse and explain individual method choices in the text.

> The practicality of Active Learning remains...

We agree that practicality of AL must be demonstrated in real scenarios in future reseacrch. Determining a stopping criteria remains an open question. We included the issue in the discussion.

**Detailed Comments:**

>  The evaluation is done only with 2D models ...

For our experiments, we used 2D nnUNet configurations because of higher training speed and less prone to overfit. However, the method can be extended to 3D models and will be studied in further research. We added this as a limitation and rephrased the text.

> The introduction and related work...

We apologize for the misunderstanding. The motivation of this work is to evaluate AL methods in combination with the nnUNet model. The nnUNet is used since it identified robust design decisions based on multiple tasks. We reformulated the introduction and state-of-the art section.
> “Sampling, annotation and self-configured...

Since only the annotations change during each sampling round, almost all model configurations except annotation resampling and intensity normalization remain the same. We clarified that. We did not encounter any convergence issues in our experiments.

**Questions To Address In The Rebuttal:**
> The number of initial samples...

The initial sample size of 100 is indeed realistic. However, we wanted to test the proposed strategy also for larger budgets to confirm its robustness with respect to the budget. We have clarified this in the text.

> Gradient embeddings is mentioned...

Gradient embeddings are a promising choice in AL for classification tasks by measuring how much a new data point influences changes in model parameters. Aklilu et al. (2022) adapted this idea for semantic segmentation tasks. However, the efficient extraction of gradient embeddings in U-Net like architectures remains an open question that we aim to address. We added this to the introduction section.

> The stochastic greedy method...

We utilized the stochastic greedy optimizer from the SUBMODLIB library developed to maximize the submodular function. We have clarified this in the text.

> clearly state in the related work...

We revised the related work section accordingly.

**Questions**

> The method extracts overlapping patches...

Overlaps between labeled and unlabeled patches exist. 
However, the nnUNet provides an “ignore” label to ensure that only annotated regions (patches) are used to train the model. 

> Why relying on Monte-Carlo dropout...

It has been shown that uncertainty estimation using Monte Carlo Dropout seem to outperform uncertainty estimation using softmax predictions for AL. Our experiments confirmed that findings. (see Fig. 2)

> Why are the results in Fig 2...

We only wanted to confirm that our proposed methods work well with larger budgets, and therefore did not show performance results for the other methods. We completed the curves for the other methods.

> How do evaluated methods...

Computational complexity and memory requirements were not analysed in detail. However, to ensure a fair comparison between USIMC and USIMF and to make the processing manageable, the gradient embeddings were truncated to a length of 10k elements. 
The complexity of submodular subset selection depends on kernel computation time O(Q\*U) and the complexity of the greedy algorithm O(Q\*U\*B). For more information we refer to Kothawade et al. (2021).