nnActive: A Framework for Evaluation of Active Learning in 3D Biomedical Segmentation
Authors: Carsten T. Lüth
Jeremias Traub
Kim-Celine Kahl
Till J. Bungert
Lukas Klein
Lars Krämer
Paul F Jaeger
Fabian Isensee
Klaus Maier-Hein
Venue: Accepted by TMLR ()

Abstract: Semantic segmentation is crucial for various biomedical applications, yet its reliance on large annotated datasets presents a significant bottleneck due to the high cost and specialized expertise required for manual labeling. Active Learning (AL) aims to mitigate this challenge by selectively querying the most informative samples, thereby reducing annotation effort. 
However, in the domain of 3D biomedical imaging, there remains no consensus on whether AL consistently outperforms Random sampling strategies. Current methodological assessment is hindered by the wide-spread occurrence of four pitfalls with respect to AL method evaluation. These are (1) restriction to too few datasets and annotation budgets, (2) training 2D models on 3D images and not incorporating partial annotations, (3) Random baseline not being adapted to the task, and (4) measuring annotation cost only in voxels.
In this work, we introduce nnActive, an open-source AL framework that systematically overcomes the aforementioned pitfalls by (1) means of a large scale study evaluating 8 Query Methods on four biomedical imaging datasets and three label regimes, accompanied by four large-scale ablation studies, (2) extending the state-of-the-art 3D medical segmentation method nnU-Net by using partial annotations for training with 3D patch-based query selection, (3) proposing Foreground Aware Random sampling strategies tackling the foreground-background class imbalance commonly encountered in 3D medical images and (4) propose the foreground efficiency metric, which captures that the annotation cost for background- compared to foreground-regions is very low. We reveal the following key findings: (A) while all AL methods outperform standard Random sampling, none reliably surpasses an improved Foreground Aware Random sampling; (B) the benefits of AL depend on task specific parameters like number of classes and their locations; (C) Predictive Entropy is overall the best performing AL method, but likely requires the most annotation effort; (D) AL performance can be improved with more compute intensive design choices like longer training and smaller query sizes. As a holistic, open-source framework, nnActive has the potential to act as a catalyst for research and application of AL in 3D biomedical imaging. Code is at: \href{https://github.com/MIC-DKFZ/nnActive}{https://github.com/MIC-DKFZ/nnActive}

Published in Transactions on Machine Learning Research (08/2025)
nnActive: A Framework for Evaluation of Active Learning in 3D Biomedical Segmentation
Carsten T. Li,th12,3* Jeremias 1,2,4 Kim-Celine Kahl Till Bungert Lukas 1.255 Lars Kraemer 1.2.3 Paul F. Jaeger Fabian Isensee Klaus Maier-Hein 1,2,,788
Cerman Cancer Research Center (DKFZ) Heidelbery, Division of Medical Image Compulingg Cermany Helmhollz Imaginng, German Cancer Research Center (DKFZ), Heidelberg, Germany
Faculty of Mathematics and Computer Science, University of Heidelberg, Germany
German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany Institute for Machine Learning, ETH Zürich, Switzerland
GGrrmn Cancer Research Center (DKFZ) Heidelberg, Interactive Machine Learning Group, Germany Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelbery University Hospital, Germany National Center for Tumor Discases (NCT) Heidelbery, Germany
(carsten. lueth, jeremias. atremias.traub/ Sdkfz-heiddlbbrr..c.m de */t: These authors contributed equally to this work.
Reviewed on OpenReview: https: // optp://eeeerveee...mm net/ formm? id= AJAnmRLJjJ
Abstract
Semantic segmentation is crucial for various biomedical applications, yet its reliance on large annotated datasets presents a sigmificant bottleneck due to the high cost and spe- cialized expertise required for manual labeling. Active Learning (AL) aims to mitigate this challenge by selectively querying the most informative samples, thereby reducing annotation effort. However, in the domain of 3D biomedical imaging, there remains no consensus whether AL consistently outperforms Random sampling strategies. Current methodological assessment. is hindered by the wide-spread occurrence of four pitfalls with respect to AL mothod evaluation. These are (1) restriction to too few datasets and annotation budgets, (2) training 2D models on 3D images and not incorporating partial annotations, (3) Random baseline not being adapted to the task, and (4) measuring annotation cost only in voxels. In open-source AL framework that systematically over- comes the aforementioned pitfalls by (1) means of a large scale study evaluating 8 Query Methods on four biomedical imaging datasets and three label regimes, accompanied by four large-scale ablation studies, (2) extending the state-of-the-art 3D medical segmentation method nnU-Net by using partial annotations for training with 3D patch-based query selec- tion, (3) proposing Foreground Aware Random sampling strategies tackling the foreground- background class imbalance commonly encountered in 3D medical images and (4) propose the foreground elficiency metric, which captures that the annotation cost for background- compared to foreground-regions is very low. We reveal the following key findings: (A) while im-
this work, introduce nnActive,
all AL methods outperform standard Random sampling, none reliably surpasses proved Foreground Aware Random sampling; (B) the benefits of AL depend on task specific parameters like number of classes and their locations; (C) Predictive Entropy is overall the best performing AL method, but likely requires the most annotation effort; (D) AL per- formance can be improved with more compute intensive design choices like longer training and smaller query sizes. As holistic, open-source framework, nnActive has the potential to act catalyst for research and application of AL in 3D biomedical imaging, Code is at: http:///gthuu..oo////C/DD///////

---

Published in Transactions on Machine Learning Research (08/2025)
1 Introduction
Semantic segmentation is vital for numerous biomedical applications, including the delineation and detection of structures and pathologies in computed tomography scans (CT), magnetic resonance imaging (MRI), and microscopy images. With the advent of deep learning, training-based approaches that require large annotated datasets have become the de facto standard for semantic segmentation. This trend is particularly evident in 3D medical imaging, where U-Net-like architoctures like mnU-Net [Isensee et al. 2021 have received siguificant emphasis.
While there is an increasing number of medical scans created each day and stored in large databases (Smith- Bindman et al. 2019), the cost of data annotation remains one of the main obstacles when solving specific 3D biomedical tasks, due to the high manual effort required to create segmentation masks. This is particularly drastic for biomedical images, as specialized personnel have to carry out the annotation.
The promise of Active Learning (AL) is to reduce this cost of annotation by only querying the most infor- mative samples to be annotated for the task. This reduction in annotation cost needs to be weighed against an increase in computational and setup costs stemming from the use of AL. Therefore, to justify a general recommendation for an AL method, it needs to reliably bring performance benefits over computationally cheap annotation strategies like Random sampling in multiple 'realistcc scenarios' that are substantial enough
to ensure amortization of its cost increases during application (Ltth et al. 2023 Munjal et al. 2022
Mittal
ct al. 2023)
While there are various studies on AL for 2D image and video segmentation (Mittal et al. 2023 Mackowiak et al. 2018), many open questions remain about its effectiveness in the 3D biomedical domain, largely due to fundamental differences between the 2D and 3D domains. Most importantly, the high annotation cost and the high redundancy in 3D image data, e.g. the similarity of neighboring areas, necessitate more efficient annotation strategies such partial annotations, in the form of slices or patches. In addition, 3D biomedical images commonly have background class that occupies most of the image, standing in stark contrast to the dense multiclass masks frequent in 2D natural image semantic segmentation tasks (Cordts et al. 2016).
The AL community lacks a common benchmark for the 3D biomedical domain as it is highly scattered in terms of evaluation practices (see table D making it nearly impossible to directly compare results across papers. Most importantly, there is no consensus on whether employing AL methods leads to reliable performance improvements over Random sampling. Many studies indicate that AL methods do not always outperform
Random sampling (Nath et al. 2021 Gaillochet et al. 2023a b Fölleer et al. 2024 Vepa et al.) 2024
Burmeister el al. (022)) and also commonly emphasize that it remains a surprisingly strong baseline (Nath et al. 2021 Burmeister et al.) (002).. Further, [Burmeister et al. (2022) concluded that AL methods do not reliably outperform advanced Random sampling strategics where the sampling is adapted to the 3D structure of the data. Crucially, this is the only work investigating improved Random baselines. When taking further into account that most studies neither make use of standardized segmentation models, proven to bring state- of-the-art performance, nor 3D models making explicit use of partial annotations during training. With both of these points potentially substantially reducing overall model performance, it is apparent that the current evaluation protocol does not allow for making practically relevant and generalizing assessments based on which a practitioner can make an informed decision whether to employ AL or not.
In response, this work introduces a novel framework for performance evaluation of 3D biomedical AL for semantic segmentation. It systematically addresses shortcomings in prior work, formalized as four pitfalls, by employing best practices for general AL cvaluation. These practices are extended to account for the specific properties of 3D biomedical segmentation, enabling potential practitioners and developers to better assess the expected performance improvements when utilizing AL in duse-to-producete scenario. Concretely, our contributions are:
1. We provide unActive, a highly configurable AL extension for nnU-Net using partial annotations in the form of 3D patches that ensure state-of segmentation performance and out-of-the-box adaptation to new segmentation tasks.
2. We introduce Foreground Aware Random sampling as a stronger, more realistic bascline, which tackles the class imbalance typically encountered in 3D images.
2

---

Published in Transactions on Machine Learning Research (08/2025)
3. We perform the largest study to date of AL methods with specific focus on uncertainty based Query Methods (QMs), encompassing over 7500 mnU-Net trainings on 12 dataset-settings from four different datasets with three respective Label Regimes for each dataset, alongside numerous ablations to allow a holistic view of AL performance benefits.
4. We propose Foreground Efficiency (FG-EII), a novel metrie measuring annotation efficiency which takes into account that annotating background has negligible annotation effort compared to foreground, setting apart from other metrics using voxels as proxy for annotation effort.
2 Requirements of Active Learning Evaluation
In its very foundation, AL represents wager where an expected reduction in anmotation cost is weighed against the additional experimental sctup and compute costs induced by it. The expected annotation cost reduction can be estimated from the annotation cost and the expected performance gains. Finding the best among multiple AL methods for a real-world use case entails spending the annotation budget multiple times, leading a net increase in annotation effort. Therefore, benchmarking AL in use-case scenario directly contradicts the purpose of using AL in the first place. This is also referred to as 'validation paradox', detailed in Lüth et al. (023
Therefore, the evaluation of AL methods must ensure that the measured performance gains are generalizable and practically relevant (Munjal et al., 2022; Zhang et al.) 2024 Mittal et al. 2019 Lüth et al.) 2023 Mittal
let al.
2023 Zhan et al. 2022). To that end, the following four requirements (RI-R4) need to be taken into
account ensuring:
R1 Generalization over datasets, annotation budgets, and query parameters.
R2
Performance gains in combination with orthogonal approaches increasing annotation efficiency e.g. Self- or Semi-Supervised Learning.
R3 Performance gains over computationally cheap methods, such as improved variants of Random sampling, Reduction in annotation effort AL methods is measured.
R4
The implementation of these requirements depends on the tasks (c.g. semantic segmentation or object recognition) and domains in which AL is applied. Each requirement maps to one pitfall and its implications are detailed for 3D biomedical imaging in section 3
Finally, to prevent overfitting on development datasets, the best-performing AL methods should also be evaluated on separate held-out test datasets that are independent of the development datasets as in a roll- out scenario, ensuring generalization gap to previously unseen datasets before widespread adoption.
3 Pitfalls and Solutions for a Systematic Validation of Active Learning Methods in 3D Biomedical Semantic Segmentation
Based on the requirements (RI-R4) stated in section necessary to build a generalizable AL method, wo discovered four respective pitsalls (P1-P4) in the evaluation protocols of the related literature for AL in the 3D biomedical domain, which hinder generalizable and reliable performance assessments. We emphasize that our goal here is not assign blame but to highlight the importance of evaluation, as inadequate assessment obscure which methods contact. with AL. In table D the prevalence of these pitfalls is shown in the related literature alongside important design parameters. We address these pitfalls by building the nnActive framework and performing a large scale empirical study following the design solutions we detail here.
truly the most effective, especially for potential practitioners getting first
P1: Evaluation is restricted to too few settings. Evaluating AL methods on a wide variety of datasets and multiple different annotation budgets is becoming common practice in AL for classification (Lüth) et al. 2023 Mittal et al.) 2023 2019 Zhang et al.) 2024) and also becomes incorporated into 2D Semantic Segmentation (Mittal al. 2023). Only by doing so
possible to obtain generalizing performance estimates, as the only benefit of an AL method lies in its ability to generalize novel scenarios during application. For example, in practice, it may not be clear what an adequate annotation budget to avoid
3

---

Published in Transactions on Machine Learning Research (08/2025)
Overview of the related work in AL for 3D biomedical image segmentation with regard to the described Pitfalls P1-P4 and key parameters. Retraining indicates whether a model is trained for each AL loop from a standard initialization. V/ indicates addressed, (V) partially addressed, and indicates unaddressed pitfalls. N/A is given, as in the experimental setup, this Pitfall can not occur. N.S. indicates unspecified value in the manuscript and code. A detailed description of our rating is given in appendix A
Table 1:
Query Design
P1
P2
P3
P4
# Datasets
Model
Retraining
#Seeds
Nath et al.
2021
3D Image
X
N/A
N/A
3D U-Net
yes
5
Burmeister et
2022
2D Slice
X
2D U-Net
no
3
Gaillocher et al.
2023a
2D Slice
X
X
2D U-Net
yes
5
Cachet et al.
2023b
2D Slice
X
X
2D U-Net
yes
5
Ma al. 2024
2D Slice
2D U-Net
N.S.
Follerr et al.
2024
2D Slice
X
3
2D nnU-Net
n1
2
Vepa et al. 2021
2D Slice
X
3
2D U-Net
yes
5
Shi et al.
2024
2D Slice
X
2D U-Net
n0
5
Ours
3D Patch
3D nuU-Net
yes
P1: Evaluation restricted too few settings
P2: Model Training does not incorporate partial annotations
Current State
Issue
Solution
Current State
Issue
Solution
Benchmark
2D Training And Querles
2D Training
3D Training
Unuued Context
Used Context
Gap in
Generelization
Ceneniizsion
30 Model
JD Mece Partil
Benchmark Guidelines
Practical Use-Case
Practical Use Case
Lover Model Performance
Higher Model Performance
P3: Random baseline is not adapted to 3D setting
P4: Annotation cost is only measured in voxels
Current State
Issue
Solution
Current State
Issue
Solution
3D Image
CANA
Anrolation
GM
GM
Random
Method
Foreground Aware
Random
Lubieled
Arsolattion
GM1
x00 Xgleap(-80
Random
Random artifiially disadverlgged
Oversamaling foreground palchis
Amotaainn Cued
annotation effort per vovel
Flavor methods
RQMM)>> OOMM
Label
Datacens
Partial Anmotations
3D
Metric
S
Low
Medium
High
Foneground
present:
No
Yos
FG-EIL.
Figure 1: Visualization of the four Pitfalls (P1-P4) alongside our solutions and how their presence hinders reliable performance assessments of AL methods for 3D biomedical imaging. For visualization purposes, we use 2D slices as partial annotations.
cold-start, indicated by AL methods being outperformed by Random sampling due to insufficient model performance (Gao et al. 2020). Only by evaluating AL over multiple different annotation budgets can the cold-start problem be characterized. State: Currently, the amount of 3D biomedical datasets used is severely limited, with more than half of the related works using less than two datasets (Nath et al.) 2021 Gaillochet al.) [2023b Ma et al. 2024). Further, this evaluation usually limited to one fixed annotation
budget (Nath et al.] 2021 Burmeister et al. 2022 Gaillochet et al. 2023a Ma et al. 2024 Föllmer al.,
2024 Shimizu et al. 2024 Only Vepa et al. (2024) and Gaillochet et al. 2023a evaluate multiple different
annotation budgets for at least one dataset.
Proposed solution: We translate the best practices for AL evaluation proposed by Lüth et al. 2023 into a framework for method development and benchmarking, locusing on a wide variely of medical imaging tasks, including multi-organ, tumor, fine-graine gnaiird pathological, and non-pathological segmentation tasks. On each of these datasots, we perform experiments on three separate annotation budgets termed Low-, Modium- and High-Label Regime to ensure a holistic view of the performance of AL methods. Further, we perform multiple ablations with regard to the query size and query patch size.
4

---

Published in Transactions on Machine Learning Research (08/2025)
P2: Model Training does not incorporate partial annotations. For 3D Images, image-based query selection is not, suitable, given the immense costs for annotating a whole image. In addition, biomedical datasets usually consist of only a few individual images. However, partial annotations, such as 2D slices or 3D patches (see appendix B for a definition), are remarkably rich in information w.r.t. the labels of neighboring areas. This is largely due to the inherent homogeneity present in 3D biomedical images. Explicitly using partial annotations for training the models while making use of unlabeled contextual information significantly decreases the amount of annotated data required to achieve performance comparable to training on the entire dataset without necessitating pretrained models often compute intensive semi-supervised training Gookkowki et al. 2024). State: Most related works train 2D models on slice-based queries (Burmeister et al.
2022 Gaillochet et al. 2023a b
Ma et al. 2024 Vepa et al. 2024 Shimizu et al. 2024). By training only on the 2D partial annotations, their surrounding context is discarded, which reduces the overall annotation efficiency. Two approaches to reducing annotation effort include Vepa et al. (0044), who train on 2D scribble annotations and use pretrained models, and Gaillochet et al. (2023b) who use Semi-Supervised pretraining. In addition, several other related works point out that using well-configured models is another simple way
2022 Mittal et al. 2023 2019). Given the dominance of nnU-Net in 3D biomedical imaging on both benchmarks and challenges (Isensee et al. 2021; 2024) we would like to emphasize the work by Föllmer et al. 20024) who proposed an AL integration for nn UneNNet that, however, only supports 2D training data and queries.
to reduce annotation effort (Lüth et al.) 2023 Munjal et al.
Proposed solution: We employ 3D nnU-Net models and train with the partial loss Gotkowski et al.) (2024) on our queried 3D partial annotations alongside a specific sampling method ensuring that the partial annotations are used during training with sufficient surrounding area as context. By utilizing the automatic configuration of nnU-Net, we further ensure that our models well configured for each dataset.
P3: Random Bascline is not adapted to 3D setting. In 3D biomedical image segmentation, many datasets have a background class and structure of interest (e.g. organ, tumor) that often occupies a small portion of the images and is located in a specific area. Based on this, the standard Random baseline is artificially disadvantaged when combined with partial annotations. It often queries image regions that are purely background, which require minimal annotation effort. This issue is already known for 2D slices (Ma) et al. 2024). Further, specific classes occupy smaller regions in the image than others, leading to a selection bias favoring large classes, which leads to a strong selection bias towards larger classes. Given that the primary challenge in manual voxel-wise annotation lies in delincating the structures rather than identifying their rough location, random image selection with oversampling of foreground areas by drawing random classes is a feasible approach in practice. Alternatively, using information about the inherent structure of 3D biomedical tasks allows for multiple improved Random strategies. State: Burmeister al. (2022) evaluate advanced Random strategies using stratified sampling (e.g. Strided) based on the 3D structure of the data and conclude based on its performance that Random and Strided sampling may be sulficient for many use cases. This is especially concerning given that all works acknowledge the surprising toughness of beating Random bascline (Nath et al., 2021) or many benchmarked AL methods underperforming and or being solely on par with Random (Gaillochet et al., 2023a Ma et al.) 2024 Föllmer et al., 2024). This leads to the question of how improved Random basclines would have changed the verdict of other works from 'AL is beneficial' into another direction.
Proposed solution: We employ additional Foreground Aware Random strategies, which simulate screening an image for random foreground class and ensure that foreground is present in a specific percentage of all queries. This also ensures that the class distribution of queries is diversified across classes. For more information, we refer to section
P4: Annotation Cost is only Measured in Voxels. Measuring annotation effort of two competing QMs purely based voxel based metrics does not capture that substantial differences can arise based on the structures with the queries. For example, a query completely consisting of background with minimal structure, requiring minimal annotation effort, has the same number of voxels as query containing multiple structures of interest that need to be delineated with a large amount of effort. Therefore evaluation methods purely using voxel based metrics mcasuring annotation effort can lead to a systematic bias favoring QMs which require a large annotation effort per queried voxel. State: To our knowledge, none of the related work takes this factor into account with any explicit measurement discusses this behavior as problematic.
5

---

Published in Transactions on Machine Learning Research (08/2025)
Proposed solution: We measure the amnotation efficiency by proxy of the amount of foreground annotation using the decay parameter 7, we term Foreground Efficiency (FG-Eff). It stems from an exponential decay fitted to the performance gap to a model trained on the entire dataset against the number of foreground voxels. Higher values indicate that a QM is more annotation efficient as it converges faster to the performance obtained when training on the entire dataset (example given in fig. D. Due to its nature, the FG-ENT only allows meaningful comparisons of QMS across a single Label Regime with identical training setups. As the number of foreground voxels represents proxy for annotation effort, the FG-Eff does not replace other performance metrics but should be seen as extension of them. For more details on the FG-Eff with mathematical definition and its interpretation, we refer to appendix D
4 nnActive Framework & Study Setup
The entire study is based on our proposed nnActive framework, an extension of nnU-Net for AL with 2D and 3D biomedical semantic segmentation enabling querying 3D patches with AL methods. We focus on 3D patch-based AL to keep the general framework versatile and because 3D patches can be annotated with multiple different strategies, e.g. dense or sparse with slice annotation. The design of nnActive allows it to be used directly in combination with the standard U-et- framework for both benchmarking and applica- tion. Hence, it allows easy implementation of future methodological developments in the benchmarking and application of AL since nnU-Net (Isensee et al. 2021 2024 is often extended with an ecosystem of projects built directly on top (Gotkowski ct al. 2024 Roy ct al. 2023
We now present the overall design of the nnAivvee framework, along with study-specific design choices made for our benchmark evaluation.
Model Architecture and Training Strategy Wc use nnU-Net [Isensee et al. 2021 a self-configuring deep learning framework, as our segmentation model. However, we enhanced the standard patch-based model trainer through region sampling, enriching the region observed by the model with additional unla- beled context from the rest of the image. Study Specific: We used the 3D full resolution configuration of nnU-Net and trained for 200 epochs. To increase model robustness, we used an ensemble of five mod- els trained via 5-fold cross-validation, as previous research has demonstrated that ensembles improve AT, performance by providing more reliable uncertainty estimates (Beluch et al.) [2018] Kahl et al.] [2024). Fur- ther, perform complete retraining of the models for each AL loop finetuning leads to reduced model performance (Beck et al. 2021: Ash & Adams, 2020) presumably due to the model getting stuck in local optimum. The training of the models themselves is not sceded, but all datasct-related parameters arc. All experiments were averaged over four seeds.
3D Query Methods We implemented the QMs for the 3D volumetric data in two steps. First, we draw a set of best patches with a maximum allowed overlap (o) with respect to previously drawn patches for each image using an uncertainty function, followed by an aggregation method. In the second step, the final query is drawn from the patches of the entire training & pool dataset. An example of uncertainty-basee QM is given in algorithm D Study Specific: We evaluate the following 8 QMs in our study, described in the following two paragraphs, with no allowed overlap (o 0) between patches.
AL Query Methods We implemented the following five uncertainty-based AL QMs 1) Predictive Entropy (Settles 2009), 2) Bayesian Active Learning by Disagreement (BALD) (Houlsby et al. 2011 Gal et al. 2017). 3) PowerBALD (Kirsch et al., 2023), 4) SoftrankBALD (Kirsch et al. (023), and 5) PowerPE (Kirsch) ee al. 2023 Both Predictive Entropy and BALD greedily select. the top-k uncertainty scores and are therefore referenced as 'Greedy'. PowerBALD, PowerPE, and SoftrankBALD use a selection mechanism with additional noise perturbations, which promotes the diversity of the samples and are therefore referenced as 'Noisy. Study Specific: We use for all QMs mean aggregation where the aggregation size is equal to the query patch size and set the 3-parameter for PowerBAL.D, SoftrankBALD and PowerPF, to 1 as proposed by Kirsch et al. (2023).
1 When extending our results we highlight the importance of using our exact nnU-Net version ensure compatibility
6

---

Published in Transactions on Machine Learning Research (08/2025)
Random Strategies We use three random strategies as baselines: 1) the standard Random sampling baseline and two more advanced Foreground Aware Random strategies, 2) Random 33% FG, and 3) Random 66% FG. The Random 66% FG baseline selects a completely random patch with a probability of 33%, while the remaining 66% of patches prioritize regions containing anatomical structures with foreground oversampling, Le. where half of the patches are centered on a randomly chosen foreground class and the other half are centered on the border of a foreground class. Similarly, the Random 33% FG baseline increases the proportion of fully random selections to 66%, while 33% of patches are drawn with the aforementioned fore- ground oversampling. These modifications ensure that Random baselines remain fair point of comparison by accounting for the natural biases present in medical imaging data.
Evaluation Metrics The general model performance is evaluated using the Mean Dice Score (per 3D image) (Dice, 1945). We use four different metrics, whereby the first three metrics allow relative comparisons of QMs within individual Label Regimes: 1) the Mean Dice score of the final AL loop (Final Dice), 2) the Area Under Budget Curve (AUBC) (Zhan et al., 2021 2022) aggregating the Mean Dice scores over all AL loops, 3) our proposed FG-Eff mcasure which is proxy for the annotation efficiency and 4) the Pairwise Penalty Matrix (PPM) (Ash et al., 2020), which assesses pairwise performance differences between QMs across multiple annotation budgets, based on a t-test with a significance level of a = 0.05. We argue that only a combination of these metrics provides a holistic assessment of AL by considering absolute performance (Final Dice, AUBC), relative performance (PPM), and annotation efficiency (FG-Eff). More details on our metrics and how we use them for our analysis are given appendix D
5 Empirical Study
5.1 Experimental Setup
Datasets and Preprocessing (challenge task 2) (Ji et al. 2022). Medical Segmentation Decathlon Hippocampus (Antonelli et al. 2022). KITS2021 (Heller et al. 2023), and ACDC (Bernard et al. 2018). Each image is resampled to the median datasot spacing with a training & pool split (75%) and test split (25%) which is identical across all soeds and experiments. Then the unU-Net "fingerprints" are created using the training & pool split, ensuring consistent input distributions across experiments. All following preprocessing steps were performed within the nnU-Net pipcline to maintain methodological consistency. More details are given in appendix EE]
Our study spans four prominent medical imaging datasets: AMOS2022
Query Design and Annotation Budget The selected query patch sizes for the datascts were seleeted laking into account the median image size and the size of the structures of interest for the corresponding datasets, leading to the following values: AMOS (32x 74)74,, KiTS (64x64x64), ACDC (4x40x40)) and Hippocampus (20x20x20) 20).. We assess the performance of QMs under three Label Regimes (Low-, Medium-, and High-Label) each corresponding to 5 AL loops that simulate real-world annotation constraints on all datasets. The entire annotation budget for the Low-, Medium- and High-Label Regimes correspond to: 150, 300 and 450 patches for ACDC; 200, 1000, 2500 patches for AMOS; 200, 1000, 2500 patches for KiTS; 100, 200, 300 patches for Hippocampus. We use starting budget and query size equal to 20% of the full annotation budget of each Label Regime. To ensure a representative starting budget, it is allocated to sample random foreground regions of each class, so that all classes are present in at least two patches. The rest of the starting budget is selected using the Random 33% FG strategy. More details on the dataset are given in appendix
5.2
Main Study
in fig. 3 all of which are aggregated over all experiments. Further, the ranking of all QMs with regard to AUBC, Final Dice, and FG-Eff for all Label Regimes of each datasct is shown in fig. 4 aloongiide a mcan ranking. Detailed results are shown in appendix F Using the aggregated results, we discuss the following five questions (Q1-Q5) with regard to the general performance of AL. methods:
The results of the main study are visualized in a PPM in fig. 2 and two
7

---

Published in Transactions on Machine Learning Research (08/202))
Pairwise Penalty Matrix (Main [%6]
BALD
17
3.3
1.7
18
37
22
27
50
PowerBALD
32
12
17
6.7
52
18
17
SoftrankBALLD
17
8.3
10
8.3
42
18
15
Predictive Entropy
12
22
13
2.3
38
27
27
Alganchmn autperforms
-30
PowerPE
25
0
10
17
52
15
12
Random
10
0
6.7
0
20
Random 33%
27
20
17
22
23
Random 66%
25
23
32
11
14
16
BALD
PowerBALD
SoftrankBADD
Predetive Entropy
PowerPE
Random
Random 33% FG
Random 6.6% FG
Algorithm outperformed by
Figure 2: PPM aggregated over all experiments the main study. each position (i,j) the values indicate the fraction of pairwise comparisons in % where method significantly outperformed method j.
Q1: How do AL methods compare against Random? We observe that all AL methods consistently outperform Random with regard to performance metrics comparing patch budgets. This is indicated by first both the rankings of the AUBC and Final Dice in fig. where it is consistently among the worst performing methods, especially for the Final Dice, and second all AL Strategies outperform it in over 37% of all evaluated budgets (fig. 3)).
However, we also observe that Random consistently draws the least amount of foreground voxels, indicated by its overall good ranking based on the FG-Eff metric, despite its bad ranking for Final Dice and AUBC (fig. 4.. This good FG-Fff performance makes unclear how much the annotation effort is actually reduced when employing AL methods over Random.
Q2: How does AL compare against Foreground Aware Random? We observe that Foreground Aware Random strategics often outperform AL methods. Further, they outperform Random across all measured metrics with the exception of FG-EIfT Random 33% FG generally performs slightly worse than most AL methods in terms of both AUBC and Dice (fig. 4 as well as in the mcan PPM (fig. 2 Mcanwhile, Random 66% FG seems to be the best overall method based on the AUBC mean rank (fig. 4 and the positive Win-/ Haa-/Lose-Raasd Ratio against all AL methods except for Predictive Entropy (fig. 3p). Measured by the mean PPM Random 66% FG is tied with PowerBALD and Predictive Entropy in the second place (fig. 2). With regard to the Final Dice, it is, however, apart from Random, the worst performing method as AL methods become better for later annotation budgets (fig. 4
In conclusion, Foreground Aware Random methods are much harder bascline than purely Random, and most AL methods have issues outperforming them reliably. This behavior demonstrates that the amount of foreground selected is important factor for the performance of QM.
8

---

Published in Transactions on Machine Learning Research (08/2025)
Win-/Lose-ratios Random
Win-rLose-ratios Random 66% FG
32
30
42
40
23
20
30
15
155
20
12
Frectio
Fraction
10
10
10
SoftrankBALD
PowerPE
SoftrankBALD
PowerBALD
Predictive Entropy
PowerBALD
Predictive Entropy
(a)
each OM:
Wins
Losses
(b)
Figure 3: A detailed view into the Vin-/Lose-rrttos of AL methods in the PPM (fig. 2 for the main study against Random (a) and Random 66%FG (b). All AL methods outperform Random substantially more often than being outperformed with Noisy QMs, showcasing no Lose-scenarios (a). However, only Predictive Entropy outperforms Random 66% FG slightly more often than it is outperformed (b).
Q3: Which AL method shows the best performance? Predictive Entropy demonstrates strong overall performance across multiple evaluation metrics. It achieves the best mean rank in both the AUBC and Final Dice score of all AL methods (fig.4) and the only AL method with a positive win-loss ratio against Random 66% FG (fig. 3p). With regard to mean PPM (fig. ( performance, it tied with PowerBAL.D and Random 66% FG in the second place. Generally, performance gains of Predictive Entropy are observed especially in the later stages of AL experiments, which is showcased by its rank w.r.t. Final Dice generally being better than its rank w.r.t. AUBC (fig. 4.. This behavior also leads to high variability in performance, particularly in low-label scenarios where selected queries are highly similar, and it negatively impacts its effectiveness, leading to it being outperformed by Random in some scenarios (fig. 3p). The queries of Predictive Entropy also commonly focus on foreground classes and thus query a lot foreground, resulting in relatively low FG-Eff compared to all other methods (fig. 4
Q4: How does the dataset influence AL performance gains? We observe strong differences across our datasets with regard to AL performance gains. When comparing against Random 66% FG on Hippocam- pus and KiTS, AL is beneficial, whereas the trend is more neutral for ACDC. Contrastingly on AMOS all AL methods are generally outperformed by Random strategies, as can be seen for the AUBC and Final Dice ranks in fig. We qualitatively discuss now these trends with the datasct-specific propertics.
The primary challenge ACDC lies in the anisotropic spacing and exact delineation of three spatially close cardiac structures. These structures are present in healthy and pathological conditions and most images are cropped to the chest area. The most challenging part is the exact delincation of structures, leading to Greedy QMs or Random 66% FG, which query large amounts of foreground, having good overall performance in AUBC and Final Dice.
For AMOS, the main challenge lies in correctly annotating 15 organs of varying sizes located a large area. Here we observe that the models trained with queries from Random and all AL methods have issues with reliably capturing specific small organs, such as adrenal glands, sometimes leading a Final Dice of For Random this is due to the small probability of drawing patches that contain these organs. For the AL methods, this is likely due to the redundancy of queries focusing specific classes. Consequently, this issue is less severe for Noisy QMs than for Greedy QMs. Random 66% and 33% FG do not exhibit this behavior (appendix
For the Hippocampus dataset, the main challenge lies in delineating the anterior from the posterior hip- pocampus, which why query methods focusing on borders and uncertain regions greedily, such as BALD and Predictive Entropy, perform well. As the dataset cropped the brain region, the ratio of foreground to background is relatively high, leading to Random being more competitive than the other datasets and
9

---

Published in Transactions on Machine Learning Research (08/2025)
Method Pank [AU (C)
2
1
Medium
High
Medium
Medium
Medium
High
Mean Rank
Methad Rank (Final Dice)
5
4
High
Medium
Medium
High
Methed Rank (FG Eff)
5
5
4
High
High
Medium
Medium
High
Hippocampus
BALD
SoftrankBALD
PowerPE
Random 33% FG
PowerBALD
Predictive Entropy
Random
Random 66% FG
Ranking of methods according to AUBC, Final Dice and FG-Eff for cach dataset and its Label Regimes (Low, Medium & High) alongside mean with standard deviations (bar).
Figure 4:
The trend across datasets with regard to the benefit of AT. differs over Foreground Aware Random strategies. On AMOS we observe no benefits when using across Label Regimes whereas on KiTS and Hippocam- pus AL methods lead performance improvements and a more neutral result for ACDC. Further, we observe trend with regard to different Label Regimes where Noisy QMs outperform their Greedy counterparts (e.g. PowerBALD and BALD) on the Low-Label Regime.
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: Do smaller query sizes improve AL performance? Kendall's t measuring correlation between smaller query size and Final Dice. Large values indicate that smaller query sizes lead performance improvements over larger query sizes, Dark colors indicate the significance of two-sided test (a 0.1).
Dataset
ACDC
AMOS
KITS
Label Regime
Low
High
Low
High
Low
High
Query Method
BALD
0.640
0.284
0.107
0.711
0.426
0.178
PowerBALD
0.142
0.213
0.320
-0.107
-0.071
0.604
SoftrankBALD
0.533
0.426
0.391
-0.036
0.142
0.249
Predictive Entropy
0.355
0.569
0.569
0.853
0.391
0.462
PowerPF
0.213
0.426
0.497
0.036
0.320
-0.391
even outperforming Random FG 66% the Medium- and High-Tabel Regime for AUBC and Final Dice (fig. 4h. Overall, the performance of models is close to the performance on the entire dataset (table) 6
For KiTS, the kidney structure and tumors are clustered together with the scan covering large surrounding areas and also large areas containing only air. Generally, foreground-aware strategies have many false positives in their scans due the queries covering mostly foreground but not all derivations of background (appendix F.2). This behavior is not observed for the AL methods due them querying exactly these background areas, leading the rankings generally favoring AL methods for AUBC, Final Dice, and also FG-Eff (fig. 4. Due the overall diversity of different structures, tendency for redundant queries by Greedy QMs can be observed, leading to Predictive Entropy and BALD being outperformed on the Low- Label Regime.
Q5: What is the influence of the annotation budget on AL Performance? Generally, observe that low annotation budgets are the most challenging setting for AL methods due to the potential redun- dancy of queries, which is especially strong for the Greedy Query Methods BALD and Predictive Entropy. Consequently, these are among the worst-performing methods on the Low-Tabel Regime on ACDC, AMOS, and KITS, especially when considering AUBC (fig. 4). As Noisy QMs query more diversified, they are more robust, lcading to them never being outperformed by Random (fig. 3p). However, on larger annotation budgets and later stages, the Noisy QMs do not perform as well as their Greedy counterparts. For example, ACDC, the difference in AUBC rankings between the Low- and High-Label Regime indicates that later stages are not as affected by the redundancy of queries (fig. 4
Our findings suggest that in early loops of AL and especially for low annotation budgets Noisy QMs are more reliable than Greody QMs.
5.3 Ablations
We give a short description of the experiment setup and a summary of our main findings and their analysis for each of four ablation studies. Detailed information and analysis for each of the four ablations are given in appendix
Query Size To evaluate the influence of the query size we conduct experiments on the ACDC, AMOS and KITS datasets the Low- and High-Label Regimes using three different settings of the query size based on the main study which is halved (QSx }), identical (QSx1) 1) and doubled (QSx2).
We observe the trend that smaller query sizes with more AL loops lcad to performance improvements over larger query sizes with fewer AL loops when measured with Kendall's 7 [Kendall 1948) (example in table 2 which is more pronounced for Greedy QMs than for noisy QMs. There is no setting in which a smaller query size leads to a significant performance decrease. Notably, these performance improvements can alter the method ranking substantially toward favoring AL QMs over Random strategies, especially from the ranking of QSx2 to QS*} 1 and QSx1 when measured with Kendall's While this indicates that smaller query sizes
11

---

Published in Transactions on Machine Learning Research (08/2025)
Table 3: Does longer training improve AL performance? AFinal Dice = (Final Dice Epochs) Final Dice(Precomputed)) x 100. Positive values indicate that longer training leads better queries even when accounting for performance differences stemming from longer training. Dark colors indicate the signif- icance of a two-sided t-test (a 0.1).
Dataset
AMOS
KiTS
Label Regime Query Method
Medium
High
Medium
High
BALD
0.98
0.87
1.73
1.64
PowerBALD
0.94
0.52
2.30
1.38
SoftrankBALD
1.00
0.32
1.41
1.04
Predictive Entropy
0.15
0.70
0.58
0.88
PowerPE
0.16
0.46
1.89
1.71
are preferable in practice, we want to highlight that this comes at the cost of increased computational cost, which scales inversely proportional to the query size. For the detailed analysis, we refer to appendix G.1
Training Length We evaluate the influence of the training length with three different settings: training the model for 500 epochs (500 Epochs), training the model for 200 epochs as in our main study (200 Epochs), and training the models for 500 epochs on the entire query trajectories from the models trained with 200 epochs (Precomputed). The experiments are performed on AMOS and KiTS Medium- and High-Label Regimes, as on these datasets, longer training leads to substantial performance differences when trained on the entire dataset.
We find that longer training leads to significantly better queries resulting in performance gains for AL methods, even when taking into account that longer trained models generally yicld higher Dice scores (i.c., comparing the Precomputed and 500 Epochs settings), as shown in table 3 Measuring the robustness of rankings with Kendall's т, we find the ranking of methods stays similar from shorter to longer-trainee models when AL methods already perform better than Random strategics (KiTS), whereas in scttings where AL methods do not outperform Random strategies (AMOS), there is general shift in ranking towards favoring AL. methods over Random strategies. This shift is stronger for the 500 Epoch setting than for the Precomputed setting, underlining the improved quality of queries for longer training. For the detailed analysis, we refer to appendix G.2
Noise strength in Noisy QMs Through an exemplary but systematic ablation of PowerBALD, we aim to understand the influence of the noise strength for the Noisy QMs. We assess the influence of the noise by performing experiments on the ACDC, AMOS, and KiTS datasets for the Low-, Medium- and High-Label Regime. In these experiments, we reduce the noise over 6 steps (controlled with the parameter 3) from the noise level used in main study from PowerBALD level (B 1) to BALD level (B = oo) without noise.
We observe as general trend that for the smaller annotation budgets, the best performance (in terms of AUBC and Final Dice) is obtained through stronger noise levels (3 1), while for larger annotation budgets, less noise is beneficial. For FG-Eff, we observe a decrcasing trend as we docrease the noise strength (fig. for exemplary results on KiTS). Both observations show that the hyperparameter of QMs can have a substantial impact on the performance. However, the optimal noise values vary greatly across datasets and are dependent upon a variety of different factors, s.a. query size, training length, data redundancy, query patch size and annotation budget. We believe more research is necessary to optimize it yet unseen datasets. For the detailed analysis, we refer to appendix G.3
Query Patch Size Unlike previous work that restricts queries to entire 3D volumes or 2D slices, our setup allows free 3D patch selection, introducing an additional hyperparameter, the query patch size. To systematically assess its effect, we repeat our entire primary benchmark across four datasets, halving the query patch size along each axis while maintaining the same number of queried patches per label regime. This setup enables fine-grained selection of annotation regions.
12

---

Published in Transactions on Machine Learning Research (08/2025)
shengh
60
65
22.5
20.0
60
Dice
17.5
50
15.0
12.5
45
50
10.0
45
5 10 20 40
5 10 20 40 A-Parameter
10 20 40 A-Parameter
#-eerameter
Low
Medium
High
Figure 5: How does the noise strength influence PowerBALD? AUBC, Final Dice and FG-Eff for different parameters of PowerBALDD the KiTS dataset across Low-, Medium- and High-Tabel Regime. Higher & leads to reduced perturbation of the rankings.
Contrary to our expectations, we observe no significant drop general AL method performance compared Random strategies from Patchx1 to Patchx 3, with the relative ranking according AUBC even improving, despite 8-fold reduction in absolute annotated voxels. Similarly, comparing the PPMs of Patchx1 (fig. 2 and Patchx (fig. 60, more AL methods have better mean rank than the Random strategies for Patch 2. Based on our examination of the ranking stability of the AUBC and Final Dice with Kendall's т, we find that depending on the dataset, rankings remain stable on KiTS and AMOS, whereas on Hippocampus and ACDC they are unstable. We observe a general trend that Noisy QMs perform better than Greedy QMs for the smaller patch size, indicated by PowerPE being the best method for the smaller patch size compared to Predictive Entropy for the larger patch size. In conclusion, while the relative performance of AL methods compared to Random strategies remains remarkably resilient w.r.t. changes in query size, their relative ranking is susceptible to change. To ensure optimal method selection, systematic evaluation of AL stralegies under varying patch sizes is necessary. For the detailed analysis, we refer to appendix G.4
Parsiise Penally Hetris (Patcec12 (S])
-55
Power BALD
17
12
13
45
20
SoftreikBALLD
6.7
1.7
3.3
3.3
52
25
20
Predictive Entropy
8.3
3.3
3.3
32
17
20
Alponthmm dpprrfemms
15
57
32
Randor
6.7
1.7
6.7
Bandum 53% PG
20
26
22
20
18
38
10
-27
Randium 56% PG
22
22
20
45
11
Rean
11
10
SALD
PowerBALLD
SutrendiALD
Protictive Ener
Preeuri
Randoo
Rundom elts Rd
Ransom 2ime
Figure 6: PPM for the Patchx 1 configuration aggregated over all settings. Mean results change pared to the Patchx1 (fig. 2).
13

---

Published in Transactions on Machine Learning Research (08/2025)
6 Discussion & Conclusion
We propose the nnActive framework for semantic segmentation in 3D biomedical imaging, AL extension of mnU-Net, which allows for measuring performance estimates of AL methods that are generalizing and practically relevant, which is crucial for real-world application. In addition, we conduct the largest to date empirical AL study in the 3D biomedical imaging domain, from which we obtain the following findings with regard to uncertainty-based AL methods:
AL vs. Random: All evaluated AL methods lead to substantial performance improvements pure Random sampling, but select substantially more foreground, which likely leads to higher annotation effort per query [section 5.2 Q1).
A new Baseline: Foreground Aware Random sampling is a trivial yet. hard to beat baseline. No AL method appears outperform it reliably [section 5.2 Q2).
Predictive Entropy is overall the best-performing AL method measured by AUBC, Final Dice and PPM, but its performance is highly variable, e.g., for small annotation budgets, and it has the worst overall FG-Eff, which indicates a high annotation effort per query [section 5.2 Q3).
Best AL Method:
AL genoralization: AL performance gains strongly depend on datasct and task propertics like the ratio of foreground to background and the number of structures to segment [section 5.2 Q4].
Noisy QMs: Noisy Query Methods like PowerPE are more reliable in earlier stages of AL and lead to better FG-Eff than Greedy Methods like Predictive Entropy (section 5.2 Q5).
Improving AL performance: AL method performance can be substantially increased with more compute-intensive settings like longer training and smaller query sizes |section 5.3 Query Size & Training Length].
AL hyperparameters: AL method hyperparameters, such as the noise strength, lead to substantial performance differences, but optimal hyperparameters differ between datasets and annotation budgets (section 5.3 Noise strength in Noisy QMs).
Guidelines Based on the findings listed above we provide the following guidelines for AL on 3D biomedical images: For Practitioners: 1) Basod on the strong performance of Foreground Aware Random strategies, we agree with Burmeister et al. (2022) that in many practical scenarios, improved Random strategies, that do not require iterative re-training, may be sufficient. 2) When employing AL, longer training and smaller query sizes represent ways to substantially reduce annotation effort at the cost of more compute. For developers: 1) Improvements over the naive Random baselines are not sufficient to give a recommendation for widespread use of AL. 2) Method evaluation can be performed using shorter trainings, as performance improvements through longer trainings are consistent across AL methods.
Relevance of our Framework & Benchmark We believe that the nnActive framework, in combination with our study, will serve as a catalyst for future method development by providing a reliable and unifying benchmark. This will lead to wide-spread adoption to the best practices laid out in sections 2 to 3 by overcoming key barriers w.r.t. their adoption which are the high implementation and computational costs required for integrating AL methods into state-of-the-art frameworks due to their complexity and evaluation of multiple AL methods and baselines.
Limitations Due to the depth and rigor of our evaluation, combined with several orthogonal improvements to the AL experiment design partial loss and queries in form of freely adaptable 3D patches, focus evaluation on imcortiinty-base AL methods, which are widely used and generally among the best-performing AL methods for 3D biomedical segmentation (Föllmer et al. 2024h whilst not requiring changes in model
architecture and training. We therefore did not evaluate methods like Learning Loss Active Learning YYo
& Kweon 2019), changing the training and diversity-based methods like Core-Set (Sener Savareso 2018). However, our selection of AL methods is still a comprehensive set that we believe to be representative of the current state-of-the-art for 3D biomedical AL.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Future directions Directly building top of our nnActive framework and study, the following directions are promising: 1) Scaling of diversity-based AL. methods like Vepa. et al. (2024) and Föllmer et al. (2024) to our performance optimized setting with 3D models and ensembles, as they are, as of now, not represented in our benchmark. 2) Incorporation using unU-Net due to the decreased time necessary for finetuning and better performance on low annotation budgets. 3) Extension of our proposed FG-Fff metric to a measure which more accurately measures annota- tion effort than number of foreground voxels, e-g. number of clicks for regions (Mackowiak et al. 2018). 4) Incorporation and benchmarking of methods for starting budget selection, as well-selected starting budget can increase AL performance (Gupte et al. 2024).
Foundation Models for 3D biomedical imaging into our benchmark
Acknowledgements
This work was funded by Helmholtz Imaging (HI), platform of the Helmholtz Incubator Information and Data Science. This work is supported by the Helmholtz Association Initiative and Networking Fund under the Helmholtz AT platform grant (ALEGRA (ZT-LPP--5-11)).
The authors gratefully acknowledge the computing time provided on the high-performance computer HoreKa by the National High- High-Performance Computing Center KIT (NHR@KIT). This center is jointly supported by the Federal Ministry of Education and Research and the Ministry of Science, Research and the Arts of Baden- Württembrr,,,, part of the National Righ--Performnce Computing (NHR) joint funding program wtwprriin.d/e/e//uuuuputrrree HoreKa is partly funded by the German Research Foundation
(DFG).
References
Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al. The medical segmentation decathlon. Nature communications, 1311):1128, 2022.
Jordan Ash and Ryan P Adams. On warm-starting neural network training, Advances in neural information processing systems, 33:3884-333 -8944, 2020.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. orXiv: 03671 [cs, stat], February 2020.
Nathan Beck, Durga Sivasubramanian, Apurva Dani, Ganesh Ramakrishnan, and Rishabh Iyer. Effective evaluation of deep active learning on image classification tasks. arXiv preprint arXiv:2106. 15324, 2021.
William H. Beluch, Tim Genewein, Andreas Nurnberger, and Jan M. Kohler. The Power of Ensembles for Active Learning in Image Classification. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9368 9377, Salt Lake City, UT, June 2018. IEEE. ISBN 978-1-5386-6420-. doi: 10. (0.100/CVVR.2018.000
Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, et al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE
transactions medical imaging, 37(11):2514-2523 2018.
Josafat-Mattias Burmeister, Marcel Fernandez Rosas, Johannes Hagemann, Jonas Kordt, Jasper Blum, Simon Shabo, Benjamin Bergner, and Christoph Lippert. Less More: A Comparison of Active Learning Strategics for 3D Medical Image Segmentation, July 2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiclo. The cityscapes datasct for semantic urban scene understand- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognétion, pp. 0213-3220
2016.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Lee R Dice. Measures of the amount of ecologic association between species. Ecology, 26(3):297-300 1945.
Bernhard Föllmer, Kenrick Schulze, Christian Wald, Sebastian Stober, Wojeiech Samck, and Marc Dewey. Active learning with the nnUNet and sample selection with uncertainty-aware submodular mutual infor- mation measure. In Medical Imaging with Deep Learning, 2024.
Mélanie Gaillochet, Christian Desrosiers, and Hervé Lombaert. Active learning for medical image segmen- tation with stochastic batches. Medical Image Analysis, 90:102958, December 2023a. ISSN 13618415. doi: :t..:0/6.mmciia22222100005.
Mélanie Gaillochet, Christian Desrosiers, and Hervé Lombaert. TAAL: Test-time Augmentation for Active Learning in Medical Image Segmentation, January 2023b.
Yarin Gal, Riashat. Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183 1192. PMLR, 2017.
Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan Ö. Arik, Larry S. Davis, and Tomas Pfister. Consistency- Based Semi-supervised Active Learning: Towards Minimizing Labeling Cost. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, volume 12355, pp. 510-526. Springer International Publishing, Cham, 2020. ISBN 978-3-030-58606-5 978-3-030-58607-22 30.
doi: 01117/997733000-550110
Karol Gotkowski, Carsten Lüth, Paul F Jäger, Sebastian Ziegler, Lars Krämer, Stefan Denner, Shuhan Xiao, Nico Disch, Klaus H Maier-Hein, and Fabian Isensee. Embarrassingl simple scribble supervision for 3D medical segmentation. arXiv preprint orXiv:2403. 1288244 2024.
Sanket Rajan Gupte, Josiah Aklilu, Jeffrey J. Nirschl, and Serena Yeung- Levy. Revisiting Active Learning in the Fra of Vision Foundation Models, January 2024.
Nicholas Heller, Fabian Isensee, Dasha Trofimova, Resha Tejpaul, Zhongehen Zhao, Huai Chen, Lisheng Wang, Alex Golts, Daniel Khapun, Daniel Shats, et al. The kits21 challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomeduullrry-paess ct. arxiv preprint arXi::2307.0198/, 2023.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian Active Learning for Classi- fication and Preference Learning. crXic:1112.5745 [cs, stat], December 2011.
Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Heit nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nat Methods, 18(2): 203 211, February 2021. ISSN 1548-7091, 1548-7105. doi: 10. 00.1038//115322000-000000
Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier-Hein, and Paul F Jaeger. Nnu-net revisited: A call for rigorous validation in 3d medical image segmentation. Tn International Conference on Medical Image Computing and Computer Assisted Intervention, pp. 488 498. Springer, 2024.
Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. Advances in neural information processing systems, 35:36722-36732, 2022.
Kim-Celine Kahl, Carsten T. Lüth, Maximilian Zenk, Klaus Maier-Hein, and Paul F Jaeger. VaIUES: A
framework for systematic validation of uncertainty estimation in semantic segmentation. In The Twelfth International Conference on Learning Representations, 2024.
Maurice George Kendall, Rank correlation methods. 1948.
Andreas Kirsch, Scbastian Farquhar, Parmida Atighehchian, Andrew Jesson, Frederic Branchaud-Chard and Yarin Gal. Stochastic Batch Acquisition: Simple Baseline for Deep Active Learning, September 2023.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Carsten Lüth, Till Bungert, Lukas Klein, and Paul Jaeger. Navigating the pitfalls of active learning evalu- ation: A systematic framework for meaningful performance assessment. Advances in Neural Information Processing Systems, 36:9789 9836, 2023.
Siteng Ma, Haochang Wu, Aonghus Lawlor, and Ruihai Dong. Breaking the Barrier: Selective Uncertainty- based Active Learning for Medical Image Segmentation, January 2024.
Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver Lange, and Carsten Rother. CEREALS -Cost-Effective REgion-based Active Learning for Semantic Segmentation. BMVC, October 2018.
Sudhanshu Mittal, Maxim Tatarchenko, Özgiin Çigek, and Thomas Brox. Parting with Illusions about Deep Active Learning. 1912.0900 [es], December 2019.
Sudhanshu Mittal, Joshua Niemeijer, Jörg P. Schäfer, and Thomas Brox. Best Practices in Active Learning for Semantic Segmentation, March 2023.
Prateek Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards Robust and Reproducible Active Learning Using Neural Networks. arXav:2002.009 00550440009964 [cs, stat), April 2022.
Vishwesh Nath, Dong Yang, Bennett A. Landman, Daguang Xu, and Holger R. Roth. Diminishing Uncer- tainty within the Training Pool: Active Learning for Medical Image Segmentation. IEEE Trons. Med. Imaging, 40(10))2534-255 2547, October 2021. ISSN 0278-0062, 1558-254X. doi: 10. 20000//T..0000000020000
Saikat Roy, Gregor Koehler, Constantin Ulrich, Michael Baumgartner, Jens Petersen, Fabian Isensee, Paul F Jacger, and Klaus H Maicr-Hein. Mednext: Transformer-driver scaling of convnets for medical image seg- mentation. In International Conference on Medical Image Computing and Computer-Assistee Intervention, pp. 405-415. Springer, 2023.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Maddoonn 2009.
Jun Shi, Shulan Ruan, Ziqi Zhu, Minfan Zhao, Hong An, Xudong Xue, and Bing Yan. Predictive accuracy- based active learning for medical image segmentation. In Proceedings of the Thirty- Third International Joint Conference on Artificial Intelligence, IJCAJ-24, pp. 4885-4893. International Joint Conferences on Artificial Intelligence Organization, August 2024. doi: 10.24963) 10.24963/ijcai.2000 /j////a...a222540...
Atsushi Shimizu, Xiaoou Cheng, Christopher Musco, and Jonathan Weare. Improved Active Learning via Dependent Leverage Score Sampling, May 2024.
Rebecca Shith-Binddaa Marilyn L Kwan, Emily C Marlow, Mary Kay Theis, Wesley Bolch, Stephanie Y Cheng, Erin JA Bowles, James R Duncan, Robert T Greenlee, Lawrence H Kushi, et al. Trends in use of medical imaging in US health care systems and in Ontario, Canada, 2000-2016. JAMA the journal of the American Medical Association, 322(9):843 856, 2019.
Arvind Murari Vepa, ZUKANG YANG, Andrew Choi, Jungseock Joo, Fabien Scalzo, and Yizhou Sun. Integrating deep metric learning with coreset for active learning in 3D segmentation. In The Thirly- Eighth Annual Conference on Neural Information Processing Systems, 2024.
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEFE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Xucying Zhan, Huan Liu, Qing Li, and Antoni B Chan. A comparative survey: Benchmarking for pool-based active learning, In IJCAI, pp. 4679 4686, 2021.
Xueying Zhan, Qingzhong Wang, Kuan-hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B. Chan. A Comparative Survey of Deep Active Learning, May 2022.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Jifan Zhang, Yifang Chen, Gregory Canal, Stephen Mussmann, Arnav M. Das, Gantavya Bhatt, Yinglun Zhu, Jeffrey Bilmes, Simon Shaolei Du, Kevin Jamieson, and Robert D. Nowak. LabelBench: A Compre- hensive Framework for Benchmarking Adaptive Label-Efficiet Learning, March 2024.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Author Contributions
Exact Details of Contributions
Core Contributors: Carsten T. Lüth, Jeremias Traub
Writing
First Draft: Carsten T. Lüth
Revising Text: Carsten T. Lüth, Jeremias Traub
Reviewing Text: Carsten T. Lüth, Jeremias Traub, Kim-Celine Kahl, Lars Krämer, Lukas Klein, Paul F.
Jacger, Fabian Isensee
Figure Creation: Lukas Klein & Carsten T. Lüth
Result. Visualization: Carsten T. Lüth & Jeremias Traub
Experiments & Framework:
Code Contributors: Carsten T. Lüth, Kim-Celine Kahl, Till Bungert, Fabian Isensee, Jeremias Traub Experiment Design and Concepts: Carsten T. Lüth, Fabian Isensee, Paul F. Jaeger, Jeremias Traub
Analysis Framework: Carsten T. Lüth
Running Experiments: Jeremias Traub, Carsten T. Lüth
Experiment Handling: Joremias Traub, Carsten T. Lüth
Releasing of the Framework: Jeremias Traub
Supervision: Klaus Maier-Hein, Fabian Isensee, Paul F. Jaeger
Leads: Carsten T. Lüth
Historical Description
This work performed over three years with multiple people contributing lot to the project, making the declaration of exact author contributions challenging.
In general, over the entire span of the time, Carsten T. Lüth acted as lead for the project with numerous and large contributions, especially from Kim-Celine Kahl, Till Bungert, Paul F. Jaeger, and Fabian Isensee. the last year the project, became apparent that the workload simply was too large for one person handle; therefore, Jeremias Traub joined the project full-time, at first just to support Carsten T. Lüth. His dedicated work, however, was much more than just pure support and his analytical thinking and flow of ideas led to many overall improvements of the work, increasing it greatly in quality. Therefore, Carsten and Jeremias agreed to share the first authorship.
As Paul F. Jaeger left the project half year before finalization and was therefore not present during the writing phase, he offered to concede his last author position.
19

---

Published in Transactions on Machine Learning Research (08/2025)
Appendix
Table of Contents
A Related Works
21
Task Description
24
Active Learning Framework
25
D Evaluation Metrics
26
D.1 Final Dice
26
D.2 AUBC
26
D.3 Pairwise Penalty Matrix
26
Foreground Efficiency
26
Dataset Details
28
Main Study Results
29
AMOS
29
F.2
KITS
34
G
Detailed Ablations
35
G.1
Query Size Ablation
................................
36
G.2
Training Length Ablation
................................
40
Noise strength in Noisy QMs Ablation
................................
46
G.4
Query Patch Size Ablation
49
H
Leave-One-Out Analysis of Rankings the Main Study
54
Model Prediction Visualizations
58
20

---

Published in Transactions on Machine Learning Research (08/2025)
A
Related Works
Pitfalls
We present detailed comparison of related works and their evaluation protocols in table
The scoring rules for our pitfalls criteria in table used for and a X if not addressed:
P1 A. Evaluate performance at least 3 datasets (only counting 3D biomedical).
B. Evaluate at least two different starting budgets and query sizes.
If A.
P2 Use 3D models with training optimized for partial annotations.
2D models:
pretrained, Semi-Supervised Training or partial annotations.
P3 Evaluate Random Basclines that take into account that for 3D Biomedical image datasets large areas of the images pure background and/or make use of the 3D structure of the data. V
P4 Use metrics that take into account that the effort to annotate background is very low compared to foreground.
Compared Literature
Nath et al. (2021) Contribution: Propose query samples from datasct pool without removing them
and enforce diversity with Mutual Information over histogramms.
Query Methods: BALD, BALD with Mutual Information Histogramms, Random.
Datasets: MSD Hippocampus and Pancreas
Evaluation Metric: Best Mean Dice (3D) over Experiment
Evaluated Query Sizes per dataset (max):
Evaluated Starting Budgets per dataset(max):
P1 biomedical datasets, no ablations for multiple annotation budgets. P2 Do use 3D models but partial annotations, also pretrained models 3D models in combination with partial annotations and also Data Augmentations.
P3 No improved random baseline.
P4 No Measurement taking into account the annotation effort.
Burmeister et al. 2022) Contribution: Evaluate Strided and Stratified Sampling Strategies.
Query Mothods: Least Confidence, Entropy, Distance-based representativennss sampling, Cluster-asse rep- resenlativeness sampling, Strided Random Sampling and Stratified Random Sampling. Additional experi- ments with label interpolation.
Datascts: MSD Hippocampus, Prostate and Hcart
Evaluation Metric: Mean Dice (3D) Plots.
Evaluated Query Sizes per dataset (max): 1 Evaluated Starting Budgets per dataset(max):
P1 biomedical datasets, not multiple annotation budgets per dataset.
P2 Does neither use pretrained models or 3D models in combination with partial annotations.
P3 Do use Strided and Stratified Random Sampling.
P4 No Measurement taking into account the annotation effort.
Gaillochet et al. (2023a) Contribution: Propose Stochastic Batches as Query Methods.
Query Methods: Stochastic Batches, Entropy, BALD, Test-Time Augmentations, Learning Loss, Core-Set,
Random.
Datasets: Prostate MR Image Segmentation (PROMISE) challenge 2012, MSD Hippocampus Evaluation Metric: Mcan Dice (3D) and Hausdorff Distance.
Evaluated Query Sizes per dataset (max): 3 (ablation one dataset)
Evaluated Starting Budgets per dataset(max): (ablation dataset)
21

---

Published in Transactions on Machine Learning Research (08/2025)
P1 biomedical datasets, not multiple annotation budgets per dataset; however, multiple annotation budget ablations for one dataset.
P2 Does neither use pretrained models or 3D models in combination with partial annotations.
P3 No Improved Random Basclines.
P4 No Measurement taking into account the annotation effort.
Gaillochet et al. (0023)) Contribution: Propose Test-Time Augmentations as Query Method.
Query Methods: Entropy, BALD, Best-Time Augmentations, Core-Set, Random.
Datasets: ACDC
Evaluation Metric: Mean Dice (2D and 3D). Evaluated Query Sizes per dataset (max):
Evaluated Starting Budgets per dataset(max, (max):
P1 biomedical datasets, not multiple annotation budgets per dataset, however, multiple annotation budget ablations for one dataset.
P2 Use 2D Supervsee models.
P3 No Improved Random Baselines.
P4 No Measurement taking into account the annotation effort.
Ma et al. (2024)
Contribution: Add target & boundary awareness to existing Query Methods.
Query Methods: Entropy (with and without Dropout), BALD, Margin Sampling, Teast Confidence. Datasets: MSD Spleen, BraTS
Evaluation Metric: Mean Dice (2D and 3D) - % required data to achieve fully annotated performance and peak performance.
Evaluated Query Sizes per dataset (max):
Evaluated Starting Budgets per (max))
P1 biomedical datasets, not multiple annotation budgets per dataset, however, multiple annotation budget ablations for one datasct.
P2 Does neither use pretrained models or 3D models in combination with partial annolations.
P3 No Improved Random Baselines.
P4 No Measurement taking into account the annotation effort.
Föllmer et al. (2024) Contribution: Propose Uncertainty-Awarr Subomdular Information Measure
(USIM) as Query Method.
Query Methods: USIME, USIMC, Mean STD, Core-Set, BADGE (LL), Stochastie Batches, Entropy, BALD, Random.
Datascts: MSD Spleen, Liver and Hippocampus
Evaluation Metric: Mean Dice (3D) Pairwise Penalty Matrix.
Evaluated Query Sizes per dataset (max): 1
Evaluated Starting Budgets per dataset(max):
P1 biomedical datasets, not multiple annotation budgets per dataset, however, multiple annotation budget ablations for one dataset.
P2 Use 2D Semi-Supervised models.
P3 No Improved Random Baselines.
P4 No Measurement taking into account the annotation effort.
Vepa et al. (024
Contribution: Propose Metric Learning Based Query Method building upon Core-Set
(Core-Metric). Query Methods: Core-Metric, Core-Set, Random, CoreGCN, TypiClust, Stochastic Batches, VAAL, Vari- ance Ratio, BALD.
Datasets: ACDC, CHAOS (Combined Healthy Abdominal Organ Segmentation), MS-CMR (Multi-sequence
Cardiac MR Segmentation Challenge) and DAVIS (Densely Annotated Video Segmentation
Not included in dataset count non-medical non-3D dataset.
22

---

Published in Transactions on Machine Learning Research (08/2025)
Evaluation Metric: Mean Dice (3D) Pairwise Penalty Matrix.
Evaluated Query Sizes per dataset (max): 2 (1 Pretrained and 1 Trained from Scratch) Evaluated Starting Budgets per dataset(max): 2 (1 Pretrained and Trained from Scratch)
P1 biomedical datasets and multiple annotation budget ablations for one dataset.
P2 Use 2D models, both pretrained and trained from random initialization.
P3 No Improved Random Baselines.
P4 No Measurement taking into account the annotation effort.
Shi et al. (2024)
Contribution: Propose Predictive Accuracy-based Active Learning (PAAL). Query Methods: Random, Entropy, Variation Ratio, Margin, KMeans, CoreSet, Entropy AB-
UNet, CEAL, LPL, PAAL
Datasets: ACDC, SegThor, MSD Brain, Liver OAR (in-house datasct)
Peculiarity: Use 1/5th of the data as a validation set used during training to determine whether the query step will be performed.
Evaluation Metric: Mean Dice (not specified whether 2D 3D in paper and not clearly described in code) Evaluated Query Sizes per dataset (max): 3
Evaluated Starting Budgets per dataset (max): 3
P1 4 biomedical datasets, no multiple annotation budgets per dataset.
P2 Does neither use pretrained models or 3D models in combination with partial annotations.
P3 No Improved Random Baselines.
P4 Show the number of slices for all classes for different QMs.
Ours Query Methods: BALD, Entropy, PowerBALD, SoftrankBAL.D PowerPE, Random, Random 66%FG, Random 33%FG. Datasets: ACDC, AMOS, Hippocampus, KITS Evaluation Metrics: Mean Dice (3D) Pairwise Penalty Matrix, Area Under Budget Curve, Final Mean Dice, Foreground Efficiency.
Evaluated Query Sizes per dataset (max): 3
Evaluated Starting Budgets per dataset (max):
P1 biomedical datasets with experiments on three different label regimes each with one query size and starting budget.
P2 Using 3D models that are trained with partial loss the annotated regions.
P3 Random 33%FG and Random 66% FG alleviate background selection issue of Random.
P4 We propose the dedicated measure named Foreground Efficiency (FG-Eff) (see appendix D.4 for details).
Table 4: Comparison of works in the field of Active Learning for 3D biomedical imaging. Notation: #Datasets!: Only counting 3D biomedical datasets; not: not specified in paper and not found in code; N.S.: not specified in paper and code.
Natuc
Seg. Modid
Full Retraising
Query Disign
#Dvitset
Novel QN
Eustuile
Sceds
Advunee Rasdoon
Traiting Strilggis
Dats Ausunettatiiee
mali-Net (c.1)
318
201
Truiing
313
Thining
77
101
Standari Thiising
376
331
Semi-8.
376
331
10.
Stanlurd Triiing
338
304
10
Thining
VIDEET
333
20
Sice
313
Thining
W.NeN (4:2)
70
y04
Partal Liwe
23

---

Published in Transactions on Machine Learning Research (08/2025)
B Task Description
In Active Learning (AL) for 3D biomedical image segmentation, acquiring full annotations for an entire volumetric sean is often infeasible due to the extensive time required. Instead, partial annotations allow for selective labeling of subregions within a 3D image, reducing annotation effort while still guiding model learning effectively. This section formalizes the task of querying and incorporating partial annotations in a 3D AL framework.
hematiiia Formulation Let X' denote the space of 3D volumetric images, where each sample is 3D image X RMx XxWx> with number of modalities M, height H width W and depth D. The corresponding dense ground-truth segmentation is given by Y E (0, C),,,., where C' is the number of classes.
In standard supervised learning setting, a model fo is trained using full annotations (X, Y) from a dataset D ((x(1) ) However, AT. with partial annotations, we define a Query Method that can select multiple subsets of the volume of a single image Q(X) spread over the entire datasct. For single image, the annotated subset denoted as:
Q()()), YCY
where y represents the annotated queries where only fraction of the full annotation is provided.
The unobserved regions remain unannotated and are ignored or used for weakly supervised training.
In this work, we focus on 3D patches for partial annotation. Thus, a partial annotation for one image is defined as y = (Yishp.uor www.dd d.d (h, w, d) E Sp}. with (hp, up dp) denoting the size of the 3D patch and Sp the set of patch locations.
Given dataset D (Xx,,, 1 where only ( is available for training, the loss function is adapted to account for missing labels:
N
((0) = Σ Σ
1=1
JESO
where s(i) denotes the queried (labeled) locations in image i
32D Slices represent a subset of 3D patches, defined by e.g. hp H. wp W.dp
24

---

Published in Transactions on Machine Learning Research (08/2025)
C Active Learning Framework
Algorithm Active Learning Patch Selection
Input:
Set of images (X(0) Jimi, query size labeled set L, Uncertainty function U, Aggregation function A, allowed overlap Output: Final query set Q
Initialize final query set Q 0
for each image X {X )== do
u U M) # compute uncertainty for image
MARK A(U) # aggregate uncertainties to patch-level 5: Oimage + 0 # initialize best patches for current image
6: for in sort (2 do # sort in descending order according to uncertainty
7:
if overlap(g, Qimage UL) I then # cnsure that
@image f Ormage {g}
end if 10: end for
11: + QU Oimage
12:
end for
13:
+ (ort(2)[:=1] # sort in descending according to uncertainty
14:
Return Q
To ensure that nnActive can be used both for benchmarking and in production, we perform all perturbations of the images inside of the nnU-Net dataset structure. More specifically, inside the nn also store loop_ XxXX.json files, which store all relevant information of the queried patches. This allows to change the labels of all images directly in-place. Changes in the nnUnet_raa raw folder are transferred to the preprocessed dataset used for training using the standard un UNet preprocserroeniinggggg step.
folder where
For the query stage we build it on the patchwise inference of unU-Net in a final stage after each image is predicted for all ensemble members. The algorithun used our framework for a lop-k uncertainty method (e.g., BALD Predictive Entropy) is outlined in algorithm [
To enrich the spatial context available to the model, enhanced the standard patch-based mnU-Net trainer through region sampling. Specifically, the final patch used for forward pass still contains at least one labeled voxel (based on random or class-specific sampling), but the patch is not centered on the annotated voxel (as for the standard nnU-Net trainer). Instcad, the annotated voxel is randomly located within the final patch, following a uniform distribution over the valid patch region. Since not all voxels in the imput patch are necessarily annotated, nnActive supports training with partial losses, applying the loss only where labels are available. Importantly, the patch size used during the model's forward pass is always determined by the nn U-ee plans and configurations, which is fixed for each dataset. The query patch size used in the nn Active experiment configuration is not necessarily identical to the nnU-Net patch size.
25

---

Published in Transactions on Machine Learning Research (08/2025)
D Evaluation Metrics
In our evaluation, we performed an analysis based on all of the metries described in this section.
In the analysis of our main study, focused on all metrics, whereas in ablations, we put special emphasis the AUBC and Final Dice as they allow casier direct comparisons of values. This is also visualized in the overview figure fig. g
Our newly proposed metric, FG-Eff, measuring the annotation efficiency by proxy of foreground voxels, is described in appendix D.4
D.1 Final Dice
We use the Final Dice value after the annotation budget is exhansted for evaluation, as it allows for easy interpretation and puts special emphasis on later stages of AL experiments.
D.2 AUBC
We compute the Area Under the Budget Curve (AUBC) for cach dataset and Label Regime based the Mean Dice to allow assessing the absolute performance each QM brings (see (Zhan et al. 2021 2022 for more details). It aggregates the results of one Label Regime using the trapezoid method, and higher values indicate better performance under all budgets of the label regime.
Our normalization of the AUBC set so that if all values on one Label Regime are equal to 0.8, the AUBC will return 0.8.
D.3 Pairwise Penalty Matrix
We employ the Pairwise Penalty Matrix (PPM) to assess whether one QM significantly outperforms others in terms of Mean Dice. This metric reflects how frequently a method yields statistically superior performance compared to another, based on a two-sided t-test with significance level of a 0.05 (sce (Ash al. (0200) for further details) and whether the mean performance of method is larger than that of method j and vice-versa. The PPM enables aggregation across multiple datasets and label regimes, though does not account for absolute performance differences.
In the final matrix, we show values in % where each row i represents the fraction of settings where method significantly outperforms other methods, whereas each column j shows the fraction of settings where another significantly outperforms method j.
D.4 Foreground Efficiency
Overview We mcasure the annotation efficiency by proxy of the amount of foreground annotation using the decay parameter we term Foreground Efficiency (FG-Eff) for an exponential decay fitted to the performance gap to a model trained on the entire dataset and the number of foreground voxels. allows for simpler interpretation of plots like the following: As the number of foreground voxels represents a proxy for annotation effort, the FG-Eff does not replace other performance metrics s.a. AUBC, Pairwise Pen but should be seen as an extension of them.
Mathematical Definition The formula for the fitted exponential decay is given in eq. 1 where values with a are estimated empirically based on the data prior to the fit of y and t is the mean % of annotated foreground voxels (therefore t E [o, 1]) and fy is on the starting budget while y is the performance (Mean Dice). youl the performance on the entire dataset using trainer with identical length trained on the entire dataset and g(fo) is the mean performance on the starting budgets.
v() (fo)) Drau) exp( (tpp->(t--0 - [o)) + Stull
(1)
26

---

Published in Transactions on Machine Learning Research (08/2025)
PywerPEty= 9:69)
Predictive Entropy (y=
Random (y=
0.65
0.60
0.55
0.50
10 Percentage Voxels Foreground [%]
30
Figure Visualization of fit for the FG-Eff on the KiTS Medium-Label Regime showing the QMs: Predictive Entropy, PowerPE and Random 66% FG. The points show the actual performance of all 4 seeds. The 7 (FG-EIT) values allow to capture that PowerPE requires much less foreground performance than Predictive Entropy and also merges the information that even though Random 66% FG and that while Predictive Entropy queries similar amount of foreground as Random 66% FG, the latter is much less performant.
achieve similar
Fit values: to = 0.028, Iaall = 0.705, (((o) = 0.472
Mathematical Assumptions
The bchavior can be modelled with an exponential decay.
v(t) < Dran'it [to, fmax]. Caveat y(1) Drull
Interpretation Higher values indicate that a QM is more annotation efficient as converges faster to the performance obtained when training on the entire dataset. As the number foreground voxels is a proxy for annotation effort, we also emphasize the importance evaluating the performance based on the AUBC, Final Dice, and PPM. In best-case scenario, QM has high FG-Eff and excels in the other metrics is among the better-performing methods.
Generally speaking, QM which has a high FG-Eff but very low performance based on all other metrics is not recommended as good method, as the metric potentially can also be hacked by simply querying a very small amount of foreground and a very steep increase in performance relative to the amount queried foreground.
The annotation efficiency a metric only meaningful when compared on precisely the same model and training with the starting budget and annotation budget because the estimated values 9(f)) and Urull change resulting values substantially. As the number of foreground voxels represents proxy for annotation effort, the FG-Eff does not replace other performance metrics but should be seen as an extension of them.
Limitation
27

---

Class names in order of labels (ascending): anterior hippocampus, posterior hippocampus Published in Transactions on Machine Learning Research (08/2025)
Table 5: Dataset descriptions and configurations for the main study,
Dataset
ACDC
AMOS
KTTS
Hippocampus
Classes w.o. Background
3
15
2
Median Shape
16. 26.6x237x206
237. 5x58822822582
526x5 12x5112
36x50x35
Used Spacing
2x0.6875xx)6687)
2s1.5625xx..66 1.625
221125x0.7812xxd0
1x1x1
Pool & Training
150
150
225
195
Validation
Budget: Low [# Patches] (% Voxels)
150 (0.75%)
200 (0.26%)
200 (0.16%)
100 (6,51%)
Budget: Medium |# Patches] (% Vorels)
300(1.50%)
1000 (1.30%)
1000 (0.80%
200 (13,02%)
Budget: High [# Patches] (% Voxels)
450(2.25%)
2500 (3.25%)
2500 (2.00%)
300 (19,54%)
Query Patch Size
4x40x400
32x74x74
60x64x64
20x20x20
Query Patch Size
0.0050%
0.0013%
0.0008%
0.06513%
Test set Mean Dice (1000 Epochs)
0.912
0.893
0.773
0.895
Test Mean Dice (500 Epochs)
0.912
0.883
0.751
0.895
Test set Mean Dice (200 Epochs)
0.910
0.860
0.705
0.895
E Dataset Details
Key dataset characteristics are shown in table 5
ACDC
Class names order of labels (ascending): right ventricle, myocardium, left ventricular cavity
Class names order of labels (ascending): spleen, right kidney, left kidney, gall bladder, esopha- gus, liver, stomach, aorta, postcava, pancreas, right adrenal gland, left adrenal gland, duodenum, bladder, prostate/uterus
AMOS
Hippocampus
KITS
Class names in order of labels (ascending): kidney, kidney-tumor, kidney-cyst
28

---

Published in Transactions on Machine Learning Research (08/2025)
F
Main Study Results
TLDR:
The Random baseline is clearly outperformed by all AL methods
No AL method substantially outperforms the Foreground Aware Random baseline (for 66% Fore- ground rate).
Predictive Entropy is the overall best performing AL method, but performance is highly variable, and has the lowest FG-Eff.
AL performance is strongly related to dataset properties and the task.
Noisy QMs are more reliable than Greedy counterparts, however their performance decreases for larger annotation budgets.
The overall design of the main study, alongside details of the ablation studies, is shown in fig. The detailed results with regard to AUBC, Final Dice and FG-ER for each dataset and Label Regime are shown table
Further, we show the aggregated PPMs for cach dataset scparately in fig.
Main Study
Ablation Studies
Experiment
Regime
Evaluation
Experiment
Regime
Evaluation
ACDC [1]
dimert Modbaity by uni Oroppad in Pansground Clamsen:
Query Marhhd
X Quty Merkedihh
AMOS [2]
Bag Radk Muil Organ Bodyy
Larggcoggh
Upowe
Query
Quary Mock
Foreground Classee:
Hpppocampus [S]
Hippormpue Modiait by utti Oroppad to Paneground Clasnes:2
Query Marhod
Paick
KITS
Bag, Taaks Rickey Tanne Redty
Varmgragh
Uptor
Query
Qenry
Foreground
Para
Flagime
Parumaters
Datenet Specti Phammederss
Quary Method
Acapling Duey Soe and durth
Luary Adact kuo loggi
cal
New PD 80
Pretecte Eimppy
Mediumi
Painiss Panall Musi (PPM
Foreground Elt ciencyy (FG
Final Diee
Figure 8: Systematic schema of our empirical study. is comprised of one Main Study, which focuses on the evaluation QMs, and four Ablation Studies analyzing the influence of specific design parameters on AL methods. Query Method HP's refers to the Noise strength in Noisy QMs ablation.
F.1 AMOS
We show visualization of the queried patches for Predictive Entropy, PowerPE, Random 66% FG and Random on the AMOS Low-Label Regime in fig. TO
An investigation of the performance of AL methods by the examples of Prodictive Entropy and PowerPE when compared Random and Random 66% FG are shown in fig. [1 It clearly shows that the main performance difference stems from subset of classes which get less well predicted when not queried frequent enough.
29

---

Published in Transactions on Machine Learning Research (08/2025)
Table 6: Finc-Graineh Results for the Main Study for cach datasct. Higher values are better and colorization goes from bright (best) to dark orange worst). AUBC and Final Dice are reported with factor (x 1000) for improved readability. AUBC, Final and Beta can only directly compared for each label regime on each dataset.
(a) ACDC
Dalanct
ACDC
Label Reglme
Mexitam Final Dise
High Final Dice
Metric
AUBC
Final Dice
FC-EM
AUBC
FG-EF
AUDC
FC-EE
Query Method
BALD
79.84
0.59
86.44
1: 0.96
20,50
3.11
85.85
0:45
89.02
0.15
21.85
5.10
87.74 1: 0.38
90.47
0.18
14.90
1.14
Powe
81.18
0.58
$6.46
0.55
46.30
13.
85.63
0.37
89.07
0.21
27.69
3,90
87.50 + 0.44
89,50
0.17
17.83
1.82
SoltaakBBLD
80.71
0.92
86.30
0.95
36.72
7.00
85.89
0.49
39.33
0.27
26.
4.97
87.28
1: 0.68
90.17
0.14
14.11
1.32
Precictive Eniropy
80.02
1.54
86.54
0.96
26.50
85.53
0.50
89.12
0.07
21
11
3.07
87
1: 0.27
90.52
0.06
15.50 1 1.22
Power
0.30
45,24
0.68
4.00
87.21
80.67
0.15
Random
26.65
0.81
50.04
1.64
19.28
54
$2.24
1924
83,46
0.87
10
8.15
84
01.06
96.228
08
21.45
+ 3.85
Random 33%
55.00
00.80
22
1:15
15.72
Random 66% PG
82,32
0.33
86.70
0.48
31.21
4.52
$6.16
0.44
88.62
0.52
18.92
2.12
87.86 =: 0.33
80.94
0.09
13.33 4: 0,50
(b) AMOS
Label Regine
Low
Mediuan Final Dioo
High Final Dice
Metrie
AUBC
Final Dice
EG-PI
AUBC
RG-RIF
AURC
FG-FI
Query Method
BALD
38
2.34
34.05
1.58
'22 66
$.50
52,50
2.74
59.26
1 2.73
1.54 0.22
09.38
0,70
74.93
2.38
-0 0.20
Power
3.65
14.36
1.47
73.02
2.00
17.86
N0.40
0.48
SoftronKBALD
44.40
1.56
0.46 45.15
-11.38
4.19
90.01
0.00
00:72
+ 0.65
5.70
0.10
75.29
1.46
$1.23
1.18
3,52 D.38
Entropy
1.78
0.17
$0.79
2.07
PowerPE
17 00
2,50
50:00
2,30
-9.80
12.14
66.74 1 2.80
73.68 1 0.92
18:00
77.92
0.29
80.52
0.36
8.87 0.30
0:37
0.62
Random 33% FC
58.05
+ 1.54
02.995
1.03
35.45
11.43
71.78 + 1.16
78.60
+ 0.37
96.58 + 2.09
79656
0.38
82.08
0.19
14:44
0.47
Random 60% FG
62.84
à: 1.88
71.11
1.42
40.63
9.82
74.87
0.64
80.72
0.54
32.62
6.15
80.98
0.19
$0.81
0.32
12:33
0.43
(c) Hippocampus
Dataset
Hippocampps Mediam Final Dice
Label Regime
Low Final Dice
High Final Dice
Metric
AUBC
FG-EE
AUBC
FC-DE
AUBC
PC-EF
Query Method
BALD
88.26 + 0.03
88.87
= 0.06
9.58
0.97
88.79 1 0.02
89.18 1 0.07
1.52
0.06
89.03 + 0.05
89: 1 0.05
3. 29 0.12
PowerBALD
SoftrankBAAI
88.41 1 0.11
88.93 + 0.18
9.61
0.98
72 1 0.08
80 12 0.022
3.90
0.05
89.03 + 0.06
$9.42 1 0.07
3.60 2: 0.12
Predictive Eatropy
PowerP
88 16
0:08
8S.
0.11
9.25
0.52
88.63
0:08
80.07 0.21
4.41
0.10
S1. 00
0.007
NO:
1:18
4.08
0.24
Random
58
0.08
8.76
0.47
025 0:11
N0.07 0.04
5.10
+
4.41
0.25
Random 33% FC
88
22
+ 0.16
88.70
0.18
9.00
0.81
NN-77
0:10
NO.02 0.14
6.20
0.17
88. 004
0.06
90:
0:10
0.15
Random
88
0.14
9.87
0.73
03
0002
50.02 0.04
4:21
0.03
N0. 20
4: 06
3.33
0.11
(d) KITS
Datamel
KITS
Reginee
Meeitum Final Diee
High Final Diee
Metric
AUBC
Fianl Dice
FC-EH
AUBC
EC-EF
AUDC
FC-EF
Query Mothod
BALD
40.58 0.75
44.13
3.18
7.56
0.82
55.06
1.30
61.97 + 1.49
6.32 + 0.14
62.33 + 0.84
67.57 1.72
9.35 +:555+100 0.46
PoWe
47.67
6.06
11.99
0.63
SoltoaBBALD
42.87 28 2.91
47.12
3.34
12.41
2:05
54.83
1.79
61.41
2: 2.02
7.00 4: 0.27
62.49 -: 0.74
67.00
0.97
9.82
1..22110
0.65
Precictive Eniropy
PowerPg
45.30 + 2.05
49.62 + 1.13
28.70
3.74
1:10 54.76
38.67 + 1:53
0:60 +: 0.27
60066 +: 0.66
03.62
9.60
0.51
Random
4.13
47.82
1.10
630.00
10.12
8.95
Random 35% PG
43.70 0.87
47.35
2.10
18.15
4.52
$1.50
.99
$4.08
2.76
3.28 +: 0.15
$5.30
1.26
56.29
L.02
1.51
0.04
Rondom 66% PC
44,07
46.83
2.53
11.25
M0.75
0.97
51.67
1.25
63.73
15.00
0.84
30

---

Published in Transactions on Machine Learning Research (08/2025)
Tubbe Naraal Matroe tMan ACBCC I&D
Parebee Fandlly Matr Mar AMCS I842
BALD
27
5.7
20
20
-16
BALD
FawerBALD
5.7
6.7
Setraanid AntrraniBALD
Wedidtive Stmme
13
27
60
20
Fredictive Entresy
13
0
6.7
Prwweet:
Rondom
9
Rardeen
40
0
27
Rancom 60%
20
27
11
11
10
73
4.
Tandon 20% FG
80
80
73
73
80
00
Mean
5.7
9.7
4.7
15
27
5.7
64
23
32
46
AdtrackBeeD
Perditionn Etirrge
Prowetif
Aanstom
Randimn 40% PM
BALD
Preer
Pestemee Krtsayy
Powertt
Axedooo
Reviaan 4409 Hi
Ransarn
Resiam
Agpertor suttefformed by
Aloe formed
ACDC
(b) AMOS
TewerBALD-
6.7
13
PoWeBALD
33
13
80
47
13
13
6.7
20
6.7
20
33
Predictive
13
Fredictive Entt
47
Agerttnm sateofamm
Prewertt
67
6
Fowerik
13
0
13
80
47
Kondron
Ransom P0%
9
5.7
6.7
6.7
fandon 134 PO
0
6.7
9
33
5.7
BAD
Profecionn Etergee
Rhwertt
Ransurm
Randam 336
BALD
BALD
Beudom JP6
Prodecive
Nensaas
Agertme sxtsenfored
Aloettees autorffemee
Hippocampus
(d) KiTS
Figure 9: Pairwise Penalty Matrix aggregated over all Label Regimes for each dataset. of the main study.
31

---

Published in Transactions on Machine Learning Research (08/2025)
(a) Predictive Entropy
(b) PowerPE
(c) Random 66% FG
(d) Random
Figure 10: Queries of the first AL loop on the Low-Label Regime on AMOS. Red colored areas are selected palches.
Best viewed on screen with Zoom.
Predictive entropy purely queries regions inside the body with a specific focus on some regions, whercas PowerPE also queries some regions at the borders and is more diverse overall. Random 66% FG queries from multiple regions of the body, but also queries from the outside, and Random queries from quite substantial amount of regions purely containing air.
32

---

Published in Transactions on Machine Learning Research (08/2025)
Niterseco Frrm trdiciiee trtres Kandien 2460 Lowe Regire
Netereee foom hoscciee ovires Rondor sels 6G AMMs Low Lalee legine
6.8
022
0.m
(a) Predictive Entropy
(b) Predictive Entropy
2effrrnce loweriE Random AMBS LowiLaba
Cifarente FoverfE Randam AMOS Low
0233
-5.00
06.0
0000
0086
0064
0.02
-..0
0..00
3.20
455
400
(c) PowerPE
(d) PowerPE
Visualization of the difference of the percentage of voxels for all classes alongside Final Dice performance on the AMOS Low-Label Regime from Predictive Entropy & PowerPE to Random and Random 66% FG. It shows that less data containing classes 11 & 12 (right left adrenal gland) is queried by Predictive Entropy and PowerPe (also Random) (5% less of the overall voxels of that class), which is strongly correlated with the Final Dice for these classes being O. For class (esophagus), similar behavior can be observed for Predictive Entropy, though not as pronounced. Compared to Predictive Entropy, this effect is weaker for PowerPE, which also queries more data from this class.
Figure 11:
33

---

Published in Transactions on Machine Learning Research (08/2025)
(a) Predictive Entropy
(b) PowerPE
(c) Random 66% FG
(d) Random
Figure 12: Queries of the first AL loop on the Low-Label Regime on KITS. Red colored areas are selected patches.
Best. viewed on screen with Zoom.
Predictive entropy purely queries regions inside the body with a specific focus on the kidneys. In contrast, PowerPE also covers different regions all over the body, still focusing on the kidney, but more diverse overall. Random 66% FG queries regions in the area of the kidney, but also covers the entire body with some queries containing purcly/ pooolyymmstyyyyooo air. Random queries from quite substantial number of regions purely containing air.
F.2 KiTS
We show visualization the queried patches for Predictive Entropy, PowerPE, Random 66% FG and Random the AMOS Low-Label Regime in fig. 12
34

---

Published in Transactions on Machine Learning Research (08/2025)
G
Detailed Ablations
Detailed analysis of the ablations can be found in the following subsoctions:
Ablation 1
Query Size: appendix
Ablation 2 Training Length: appendix G.2
Ablation 3 Noise strength in Noisy QMs: appendix G.3
Ablation 4 Query Patch Size: appendix G.4
An overview of all experiments of the main study and the ablations is given in fig.
35

---

Published in Transactions on Machine Learning Research (08/2025)
Do smaller query sizes improve the performance of QMs? Kendall's т correlations between smaller query size and performance measures. Higher values indicate that smaller query sizes tend to yield better performance. The correlation values range between and 1, where positive values suggest a beneficial effect of smaller queries, while negative values indicate the opposite. A two-sided test was performed with significance level of a = 0.1.
Table 7:
Colorscheme: Significant & positive correleation,
positive correlation,
negative correlation, significant
& negative correlation
(a) Query Size & AUBC
(b) Query Size Final Dice
ACDC
AMOS
KITS
ACDC
AMOS
KiTS
Label Regime
Low
High
Low
High
Low
High
Label Regimee
Low
High
Low
High
Low
High
Query Method
Query Method
BALD
0.004
-0.391
0.249
0,249
13:997
BALLD
0.284
0.107
0.711
0.425
0.178
PowerBALD
0.178
0.497
0.426
-0.036
0.071
0.533
PowerlLALD
0.142
0.213
0.320
-0.107
-0.071
0.604
SoftrankRAI.D
0.144
0.500
0.462
-0.071
0.178
0.462
Soft
0.426
0.391
0.142
0.240
Predictive Entropy
0.462
0.711
0.142
0.178
0.391
0.640
Prolictive Entropy
0.355
0. 0.509
0.589
0.833
0.391
06462
PowerPE
0.391
0,355
0,302
-0.142
0.320
-0.142
PowerPE
0.213
0.426
0.497
0.036
0.320
-0.391
G.1 Query Size Ablation
TLDR:
Smaller query sizes generally lead to better performance than larger query sizes for AL across all our AL QMs.
Smaller query sizes have a stronger impact on Greedy QMs than on Noisy QMs
Smaller query sizes can change ranking to favoring AL QMs from favoring Random strategies. As reducing the query size by a factor doubles the compute cost of AL we argue that 5 Loops for a Label Regime are sufficient compared to Loops when training models from scratch.
To assess the impact of query size on AL QMs, we conduct ablation studies using the same absolute anmo- tation budgets in our main experiments while varying the query sizes. We evaluate three different query sizes of twice the size (QSx2), identical size (QSx1) and half the size (QSx1/2) for one specific starting budget of our main study. This variation results in approximately half or double the number of AL loops, allowing us to analyze how different query sizes influence the performance of AT. QMs, separate from other factors. The evaluation based on two key metrics: the Final Dice score and the AUBC, which is computed only on the overlapping annotation budgets available across all three settings to ensure comparability across different query sizes.
These experiments are conducted on the AMOS, KiTS, and ACDC datasets for both Low- and High-Label Regimes to observe the behavior at the extreme settings.
By analyzing multiple datasets and annotation scales, we aim to gain a comprehensive understanding of how query size affects the performance of AL in different medical imaging contexts through answering the following questions regarding the influence of the query size:
Q1: Do AL QMs Benefit from Smaller query sizes? To investigate this, we analyze the correlation between query size and performance using a Kendall's 7 (Kendall, 1948) correlation test on AUBC and Final Dice values. The results, presented in table 7 indicate that smaller query sizes consistently improve perfor- mance of our benchmarked methods as across all evaluated QMs, we observe significant positive correlations and no significant negative correlations. Generally the effect of smaller query sizes have a strong positive impact on the Greedy QMs as they have three significant positive results for both AUBC and Final Dice.
Notably, in the ACDC and KiTS high-budget setting, fewer siguificant results are observed for Final Dice compared to AUBC, which is counterintuitive given that smaller query sizes are generally expected to provide cumulative benefits. We hypothesize that this occurs because, at high annotation budgets, substantial portion of the foreground structures in ACDC already annotated and the performance of the underlying segmentation model is already 'good' meaning that the decision boundaries does not travel high-density
36

---

Published in Transactions on Machine Learning Research (08/2025)
areas of potential queries. As result, this makes it less likely for larger query sizes to select multiple redundant samples. A similar effect, that for generally larger budgets the benefits of smaller query sizes tend to reduce, has been previously reported by Kirsch et al. (2023) for object recognition.
Q2: How does the Query Size influence rankings of annotation strategies? To investigate this, we analyze the ranking of all annotation strategies for both AUBC and Final Dice with Kendall's T for each Label Regime and Dataset between pairs of query sizes in table [ Generally, we observe that no ranking is negatively correlated and significant, and that over half of the results robust (positively correlated and significant).
The rankings for AMOS Low-Label Regime and KiTS High-Label Regime show very little change for both AUBC and Final Dice are robust across all compared query sizes. On the AMOS Low-Label Regime, Random FG strategics perform best for all query sizes (soe table 9c). and on the KiTS High-Label Regime, AL QMs like Predictive Entropy perform best (see table 10b).
Looking at the non-robust settings we observe for the correspouding datasets and Label Regimes, we will claborate these changes based on detailed results with rankings shown in appendix G.1
The strongest ranking perturbation is on ACDC where for the Low-Label Regime with QSx1/2 most AL QMs outperform all Random FG Strategies in teriss of AUBC and for the Final Dice leading especially for the AUBC to a strong difference in ranking since the Random FG has the best AUBC for QSx1 and QSx2 (see table Dal.
Similar behavior can be observed for the ACDC High-Label Regime, where, however, again a change in ranking occurs from smaller to larger query sizes, which favors AL QMs over Random and Random FG strategies in terms of AUBC (table 9b). For the Final Dice no such trend can be observed and the ranking remains stable.
On the AMOS High-Label Regime, the ranking perturbations stem from increased performance of the Pre- dictive Entropy, especially with regard to the Final Dice leads for smaller query sizes. These lead to its rankings being strongly influenced from the 2nd best ranked strategy for QSx1/2 to QSx2, the 2nd to worst ranked strategy for QSx2 in terms of Final Dice, with similar trends for all other AL QMs, which are more pronounced for the AUBC (table 9ch.
On the KITS Low-Label Regime the performance changes occur mostly from the QSx1/2 and QSx1 to the QSx2 ranking where for the smaller query sizes PowerPE leads to the best performance in terms of AUBC and Final DICE while for QSx2 is among the worst performing methods and outperformed by Random 66% FG (table 10b).
Overall, change in query size can lead substantial changes in the ranking of AL QMs relative to random strategies, with Greedy QMs especially being affected, swinging from among the best-performing to the worst-performing methods for larger query sizes.
For benchmarking purposes, we believe that reasonably chosen query sizes for given entire annotation budget, resulting in at least 4 annotation rounds, should suffice, as the correlation between QSx1/2 and QSx1 significantly positively correlated times out of 6. Especially considering that decreasing the QS by a factor of essentially doubles the compute cost of employing AL the returns are diminishing.
37

---

Published in Transactions on Machine Learning Research (08/2025)
Table 8: How robust are method rankings to changes in query size? Kendall's 7 corellations between rankings of QMs with different query sizes. high value indicates that the rankings between the two scttings are similar while lower values denote that they differ. two-sided test was performed with a significance level of = 0.1.
Colorscheme: Significant & positive correleation, positive correlation, negative correlation, significant
& negative correlation
(a) Ranking Correlation AUBC
(b) Ranking Correlation Final Dice
Datises
ACDC
AMOS
KITS
Datanet
ACDC
AMOS
KrTS
Label Regime
High
High
High
Regime
High
High
High
Selting
Setting
Q8x2 vs Q8el
0.571
0.571
0.857
0.557
0.300
0.786
Q2x2 vs Q8x1
0.786
0.711
0.786
0.571
0.641
0.929
Q8x2 vo Q2x1/2
0.214
0.286
0.786
0.929
0.300
0.785
Q8x2 vs Q8x1 /
0.286
0.643
0.786
0,500
0.429
0.857
Q8xl Q2x1/2
QSxl vs QSx1/2
0.357
0.786
1.000
0.613
0.929
Mean
0.143
0.521
0.857
0.857
0.619
0.837
Mean
0.714
0.857
Query Size Ablation Detailed Results
38

---

Published in Transactions on Machine Learning Research (08/2025)
Table 9: Fine-Grained Results for the query size ablation on ACDC and AMOS. AUBC and Final Dice are reported with factor (x100) for improved readability, Colors indicate the ranking, darker colors correspond to worse rankings.
(a) ACDC Low Label Regime
Dataset
ACDC Q8x1
Setting
Q8x1/2
Q8x2
Mecric
AUBC
Final Dice
AUBC
Final Dize
AUBC
Final Dice
Query Method
BALD
81 16 0.36
87 0.49
79.86
1.00
86.44
0.96
78 20
85.21
1.90
Power BALDD
70.55
0.76
85 31 0.90
80.43
0.25
96 46
0.55
79.08
0.54
85.25
0:31
SofD
Predictive Enttopp
80.70
+ 0.48
87 87:50- + 0.80
71.73
1.00
96.54
0.95
78.32
2.10
86.33 + 0.46
Po
0.50
4.
$6.56
0.40
78.81
85.88
Raadom
76.21
1.07
80:61 +: 2.13
75.06
4 1.29
50.34
1.64
70:97
1.56
81.32
1.61
Raudoon 33% PG
78.
1:30
83 1. 0.70
80.00
0.95
$5.09
1.14
79.70
1.01
85
0.75
Racdoon 00% PG
80.25
0.56
86 66
0.74
81.16
0.34
56.70
0.48
80.74
0.92
86.52
0.79
(b) ACDC High Label Regime
Daizsel
ACDC Q8xl
Setting
Q8xL/2 /2
Q8x2
Mesric
AUBC
Final Dice
AUBC
Final Dice
AUBC
Final Dice
Query Method
BALD
87.73 = 0.55
90:52 + 0.10
87.54 + 0.28
90.47 + 0.18
86.72 = 0.38
90.38 + 0.14
PAWD
87.36
01:45
89.96
87.26
87.45 = 0.35
50:18
+ 0.10
ST.14
0.04
90.17
0.14
56.46 = 0.45
80.05
0.43
Predictive Entropy
87.95
0.25
90.00 1: 0.12
87.61
0.35
90.52
0.06
86.82 0.52
90.29
0.19
PowerPE
87
0:47
0.10
56.09
0.64
80.67
0.15
86.35
89.71
0.21
Random
84
1.10
86-05 + 1.30
84.55
0.83
96.28
1.08
84.32
0.98
85.60
1.17
30% Randoo 66% KG
0.88
80.00
96271
87
00000
80. 20.78
0.201
87.34 0.25
90.04
0.00
87.34
0.34
90.13 0.20
(c) AMOS Low Label Regime
Dalseel
AMOS Q8x1
Setting
Q8x1/2 /2
Q8k2
Metzie
AUDC
Finel Dice
AUBC
Final Dise
AUDC
Finel Dice
Query Method
RALD
0..1
35.74 + 2,82
38.86 2.67
34.05
1,58
41 1M
2.89
35-97
5.10
PoweBBALD
$1.99
2.05
16.18
SoLD
11:00 2.27
15:10 1 5.10
10.6 0129
25.33
0.95
11:33
2.29
15:15
5.29
Predictive Eniroop
PowerPR
51 08.38-0
3.18
64:40 + 2.1
47.94 + 9.16
50.04
+ 2.30
46.38
3.15
49:41
+ 628
Random
43.47
3.25
40:30
$2.24
96.36
2.92
40.85
20
Ramdon 39% PG
$4.84
1 2.83
58.72 4: 2.01
57.64 4: 1.56
62.96
1.03
56.45
= 1.39
03.51
3.30
Randonn 68% FG
62.21 1.51
70.90 1: 0.53
62.04 1.57
71.11
1.12
61.96
2.59
71.90 1: 1.58
(d) AMOS High Label Regine
Datsent
AMOS Q8xl
Setting
Q8x1/2 /2
Q8x2
AUBC
Final
AUBC
Final
Query Method
72.87
2.38
BALD
77-51 = 0.22
80:12 + 0.37
77.61 + 0.25
MI.48
0.48
77.43
0.43
80.55 + 0.37
75.12
$1.23
1.18
Predictive Estropy
75.22
2.08
83.05 4 0.26
72.06
1.50
$0.79
2.07
73.85
71.95
-: 2.09
PowerP
77.43 + 0.67
80.48 + 0.16
77.96
0.26
SI1.22
0.16
77.00
0227
80,20
0.62
Razdoom
15.48
0.37
Raadoon 33% FG
78.97 - 0.22
82.37 1 0.33
79.00 J. 0.32
$2.68
0.19
79.21
0.33
82.57 - 0.12
Raneoon 66%
0.09
0.32
39

---

Published in Transactions on Machine Learning Research (08/2025)
Table 10: Fine- Grainer Results for the query size ablation on KITS. AUBC and Final Dice are reported with factor (x100) for improved readability. Colors indicate the ranking, darker colors correspond to worse rankings.
(a) KiTS Low Label Regime
Dalasel
KITS
Setting
Q8x1/2
Q8x2
Metrie
AUBC
Finnl Dice
AUBC
Final Dise
AUBC
Final Dine
Queryy Method
BALD
45.15 2.91
44.03 + 2.12
60.43 5.15
43.03 3,18
36.47
2.89
35,62
5.95
PoweBBLD
11.18
3.02
47.71 1 4.15
44.29 2.61
47.67
5.63
13.70
2.63
18.15
2.91
SofD
2.10
2.20
41.12
Predictive Entropy
12:09 1 3.58
47:44 1 1.65
20.17 1 2.76
05.53
3.57
09:20
2.77
11.81
5.50
PowerP
45.19
Random
36.10 2.22
36 + 3.48
38.71
3.46
W0.10
4.13
37.76
3:10
37.98
2.05
Random 33% PC
43.89
0.63
4: 2:20
45.00
41.45
2.10
43.97
48.17
2:022
Rauddon 68% PG
41.48
1.89
48.01 1: 0.90
44.32 4: 2.08
06.83
2.53
03.63
1.58
47.38
0.90
(b) KITS High Label Regime
Dataset
KITS Q8c1
Setting
Q8x1/2
Q8x2
Metric
AUDC
Final Dice
AUBC
Final Dioe
AUDC
Final Dice
Query Method
BALD
62.85 1 0.80
68 1 0.58
61.96 1 1.22
07.57 1: 1.72
61.54
0.39
68.10 1 0.38
POW
0.57
SoftzaniHBA
61.29
1.04
66.78
+ 1.17
61.46
0.50
67.00
0.97
59:91
0.54
66-02
0.51
Predictive Entrops
63.65
0.31
03.04
0.61
63.03
0.70
68.74
0.65
62.10
0.27
PowaPE
29.84
0.28
63:48 1 0.81
59.59
0.00
65.02
1.19
00
0.84
64.44 2: 1.01
Raudlon
53.17
1.01
54 1 1.11
53.65
0.61
55.12
1:27
51..99
0.59
55,41
1.79
Random 33% PG
54 95
36
66
54.80
0.83
56.
1.02
77
Random 66% PC
03.36
1.11
06,20 + 1.18
13,82
1,23
15,00
0.84
60.06
0.74
56,00
1.79
G.2 Training Length Ablation
TLDR:
Longer training leads to better queries for AL QMs.
Gains of using AL persist from shorter trained models to longer trained models
Uncertainty based QMs are more likely to oupterform Random strategies when training longer.
It is possible to reduce AL compute cost by using shorter training for AL with final longer training performance gains over Random can translate.
To assess the impact of the training length on AL QMs we conduct ablation studies using the same setup in our main study whilst varying the training length. Concretely we evaluate the following three settings of training the model for 500 epochs (500 Epochs), training the model for 200 epochs as in our main study (200 Epochs) and training the models for 500 epochs but using the query trajectories from the models trained with 200 epochs (Precomputed). This design allows us to investigate the effect of longer training while also separating the effects of extended training from its influence on query selections.
The Precomputed experiments are particularly useful in distinguishing whether performance differences arise from the query selection process itself or from the increased training duration.
We performed the experiments on the KiTS and AMOS dataset as ACDC and Hippocampus did not show improvements in Mean Dice when training for more than 200 Epochs on the entire dataset. Our focus is especially on the Medium and High Label Regimes as longer training typically mostly leads to improvements for larger datascts.
By comparing these experimental conditions, aim to answer the following two questions regarding the relationship between query effectiveness, model training duration, and overall segmentation performance:
Q1: Does longer training lead to better queries? We investigate this by comparing the AUBC and Final Dice for all AL QMs of the Precomputed and 500 Epochs settings by computing their differences and testing for statistical significance with t-test in table 11 We observe that the performance metrics for
40

---

Published in Transactions on Machine Learning Research (08/2025)
Table 11: Does an increased training length of the model lead to better queries?
AMetric= peos Metrio((recomputee) for the Training Length Ablation with all models trained for 500 epochs. Larger values show that the queries when training the model for longer are better than from shorter trained model. Significance comparison performed with two-sided t-test using significance level a 0.1. Colorscheme: Significant & positive difference, positive difference, negative difference
negative difference, significant &
(a) AAUBC
(b) A Final Dice
Dataset
AMOS
KiTS
Dataset
AMOS
KiTS
Label Regime
Medium
High
Medium
High
Label Regime
Medium
High
Medium
High
Query Method
Query Method
BALD
0.98
0.87
1.73
1.64
BALD
1.66
1.12
2.75
2.04
BALD
0.94
0.52
2.30
1.38
PowerBALD
1.45
0.57
3.92
1.68
SoftrankBALD
1.00
0.32
1.41
1.04
SoftrankBALD
1.43
0.45
1.49
0.84
Predictive Entropy
0.15
0.70
0.58
0.88
Predictive Entropy
1.76
0.76
2.06
0.44
PowerPE
0.16
0.46
1.89
1.71
PowerPE
0.20
0.55
2.18
2.30
the 500 Epochs models are higher in all cases than for the Precomputed models and with the exception of Predictive Entropy at least. in 3 out of 4 settings statistically significant. This indicates that when performance increases with longer training uncertainty based QMs query data more effectively leading to performance improvements even when correcting for performance differences arising from training length.
Ranking based analysis To evaluate how each of our three training settings influences the ranking of our annotation strategies we perform Kendall's 7 (Kendall 1948) correlation test for the AUBC and Final Dice each Label Regime and dataset between two settings, the results shown in table 12 We deem a ranking as stable when it is positively correlated and significant and will not discuss it except for change where AL QMs outperform Random strategies where they previously did not or the other way around.
Q2: Do gains obtained by using AL persist when training the queried dataset for longer?
Generally, the method rankings between 200 Epochs and Precomputed are stable in 3 out of 4 cases, shown in table 12 with the exception of the AMOS High-Label Regime. We observe on the KiTS dataset that the rankings are stable and the trend that AL outperforms Random and Random FG strategics for both settings. Generally the performance gains of using AL persist from 200 Epochs to Precomputed but decrease in absolute value for the longer trainings on identical queries. This indicates that the results of our ranking for models trained with 200 epochs are likely hold also for longer trained models on KiTS.
On the AMOS dataset the ranking is stable for the Medium but not for the High Label Regime. Generally observable is a large jump in performance for the models with the AI. QMs from 200 Epochs to Precomputed (larger than for Random FG strategies) which we trace back to the Dice score of specific classes that are hard for the models to learn jumping from 0 to 0.5 for the longer training (see fig. 13 For 200 Epochs, Random 33% and 66% FG do not exhibit this behavior of individual classes having a Dice score 0, presumably because they sample more data from these classes (sce appendix F.1) On the Medium-Label Regime, PE and BALD have strong increase in the AUBC, leading them to outperform Random for Precomputed, which they did not do for 200 Epochs, but otherwise, big changes in ranking, On the High-Label Regime, for the AUBC Predictive Entropy and BALD increase from being outperformed by Random to outperforming Random with longer training and the Predictive Entropy and Softrank BALD outperform Random 33% FG which t they did not for shorter training (table 13b So, generally, longer training is beneficial for the AL QMs even when the queries are not performed with longer trained models.
Concluding, the gains obtained with AL QMs over Random strategies seem to translate from shorter trained to longer trained models for shorter time and the performance losses seem to decrease.
41

---

Published in Transactions on Machine Learning Research (08/2025)
Q3: How does training length influence the ranking of strategies? For this question the rank- ing differences between 500 Epochs and 200 Epochs and 500 Epochs and Precomputed from table 12 are evaluated.
Generally, the rankings between 500 Epochs and Precomputed showed higher correlation and were more stable than between 500 Epochs and 200 Epochs, being again robust in out of cases for both AUBC and Final Dice, with the exception of AMOS on the High-Label Regime.
For the KITS dataset there are no changes with respect to the rankings of AL QMs and Random strategies on all Label Regimes and Metrics. The only unstable ranking appears on KiTS Medium for the AUBC comparing the 200 and 500 Epoch Setting, which is mostly due to the inter AL QMs ranking changing with Random and Random FG stralegies occupying the worst three ranks in terms of AUBC (lable 13c
Mcanwhile, for the AMOS dataset, the trend is that AL QMs perform better for longer training, which is reasonable as the models guiding the query selection are much better fitted onto the datasct. Most noteworthy for the 500 Epoch setting in the High-Label Regime Predictive and BALD are the only QMs to outperform Random 66% FG in terms of Final Dice which leads to large ranking differences between 500 Epochs and 200 Epochs (which is the only negative corrclation) as well as Precomputed (table 13a On the Mediu Regime, a similar trend can be observed, though not as pronounced, as only Random 33% FG becomes outperformed in terms of Final Dice in the 500 Epochs Setting (table 13b
In conclusion, the overall results of the main study with 200 epochs extend to a large degree to 500 epoch settings, indicating that they also should hold for longer training lengths. On the AMOS dataset, this is, however, not the case, as apparently the short training of 200 epochs leads to systematic disadvantage for the uncertainty-based QMs against the Foreground Aware Random strategies. An optimal AL QM should, however, be able to work under a variety of training settings.
Q4: Can the compute cost of AL by reduced using shorter trainings and a final long training? As training is a significant cost factor, this question asks whether we can reduce the training cost while still keeping the gains of AL over Random Strategies? Recalling the Analysis from Q2, seems that in the scenarios where we obtain large gains from utilizing AL, they should persist while potential performance losses should reduce for the final long training.
However, in QI we showed that significant performance differences arise from queries of shorter to longer trained models even when accounting for performance differences due to training length.
In Q3 we observed on AMOS that these differences in query quality can cause the difference between a per- formance increase over Foreground Aware Random with queries from longer trained models to a performance loss compared to Foreground Aware Random.
For the KiTS dataset, we observed that even though ranking differences among AL methods appeared, the general trend of performance benefits over Random strategies was persistent.
Given this evidence, we suspect that it is likely feasible to perform AL experiments with shorter training. However, one must make sure, by means of validation, that the shorter trained models approximate the task 'well enough" when compared to longer trained models.
42

---

Published in Transactions on Machine Learning Research (08/2025)
Table 12: How does the training length influence method ranking? Kendall's т correlation coef- ficients comparing rankings under different training setups on the AMOS and KiTS for the Medium- and High-Label Regime. Larger values mean rankings are consistent across experiments.
Colorscheme: Significant & positive correlation, positive correlation, negative correlation, significant
& negative correlation
(a) Ranking Correlation AUBC
(b) Ranking Correlation Final Dice
Dataset
AMOS Medium
KiTS
Dataset
AMOS Medium
KrTS
Label Regime
High
Medium
High
Label Regime
High
Medium
High
Setting
Setting
Precomputed 500 Epochs
0.714
Precompnted 500 Epocls
0.619
0.524
200 Epochs & 500 Epocls
0.810
-0.143
0.524
0.905
200 Epochis & 500 Epocis
0.524
-0.143
1.000
0.810
200 Epochs & Precompated
0.929
0.500
0.857
0.857
200 Epochs & Procompatee
1,000
0,500
0.857
0.929
Training Length Ablation Detailed Results
We show detailed results for the training length ablation focusing on the ranking in table 13
AMOS Training length We observe especially strong performance increase for longer trained models AMOS across all AL methods and Random compared to Random 66%FG, which is discussed in ap- pendix F.1 We observe that the longer training leads to substantial performance improvements on classes 11 & 12.
43

---

Published in Transactions on Machine Learning Research (08/2025)
Table 13: Fine-Grained Results for the training length ablation. AUBC and Final Dice are reported with factor (x100) for improved readability. Colors indicate the ranking, darker colors correspond worse rankings.
(a) AMOS Medium Label Regime
Dataset
AMOS Precomputtdd
Setting
200 Epochs
500 Epocia
Mecric
AUBC
Final Dice
AUBC
Final Dize
AUBC
Final Dice
Query Method
BALD
52 36
2.74
30:26 2.73
76.75
1.97
80.01
2.07
15
1:20
81.67
2:40
Power BALDD
66:11
1.47
73.02 + 2.01
78.93
0.08
92.51
0.42
79.87
0.33
83.96
0.41
SofD
82.12
Predictive Enttopp
56.30
1.78
62:07 + 1.30
75.06 0.61
$1.64
0.50
77:21
0.53
83:40
0.41
Po
06.74
2,80
S0.15
0.45
79:27
0.96
Ramdom
54.65
2.82
56:22 + 4.61
72.51
1.31
15.46
0.94
72.81
= 1:31
Y5.46
+: 0.91
Raudonn 33% PG
71.78
1.16
78.00 1 0.37
79.63
0.53
58.70
0.37
79.63
0.53
83.70
0.37
Racdonn 06% PG
74.87 0.64
80.72 4: 0.54
81.31 0:39
54.04
0.46
81.31
0.30
84.94
0.46
(b) AMOS High Label Regime
Daizsel
AMOS Precomputed
Setting
Eporha
Epocha
Mesric
AUBC
Final Dice
AUBC
Final Dice
AUBC
Final Dice
Query Method
BALD
60 = 0.70
74.05
t 2.38
$3.50 0.17
96.66
0.00
84.37
0.10
87.18 + 0.07
PAWD
77.50
0.14
80:48
0.48
86.78
84.50
75.22 1:48
81.23 + 1.18
$4.25 4 0.11
$6.39
: 0.10
84.00
= 0.26
80.83 - 0.16
Predictive Entropy
27 1.52
80.79 1 2.07
$1,00
0.14
80.77
0.13
81.70
0.03
87.52
0.05
PowerPE
77.92 0.29
80.52
0.16
53.86
0.16
55.09
0.19
84.52
0.32
86.25
0.19
Random
72 0150
75.48
0.37
81.31 0.41
91.73
0.06
81.31
0.41
82.73
0.06
30% Randoo 66% KG
79333
0.38
96.28
0.14
80.98
0.19
83.81 + 0.32
85.06 0.10
86.98
0.13
86.06
0.10
86.98
0.13
(c) KITS Medium Label Regime
Dalseel
KITS Precomputed
Setting
200 Epodhs
500 Epoda
Metzie
AUDC
Finel Dice
AUBC
Final Dise
AUDC
Finel Dice
Query Method
BALD
55/06-
1.20
61.37 + 1.49
61.96
U.66
07.84
0.71
63 0..6
70:60 + 0.55
PoweBBALD
SoLD
54.83 + 1.79
61.44 1 2.02
62.44 1 1.22
08,12 4 0.38
63.85
1.01
09.80 1 0.50
Predictive Eniropy
57.42
65
09.77
1.10
PowerPR
54.76
1:10
06.07
1.69
62.22
0.82
06:53
0.30
64.11
0.80
68:71 0150
Random
1.84
1.22
16.08
0.91
Raadon 39% PG
31
1.97
54.08 1: 2.78
59.33
2:33
82.50
3.22
50.33
2.33
02.55
5.22
Raudonn 68% PG
50.78
0.97
51.67
2.51
03
01.27
1.72
58 03
61.27 1.72
(d) KITS High Label Regine
Datsent
KITS Precomputed
Setting
200 Epochs
200 Epachs
AUDC
Final
AUBC
Final
AUBC
Final
Query Method
62.53 0.81
72.47
BALD
61.24
0.57
65.04 + 0.81
09.44 + U.47
71.47
+ U.54
09:52 0.50
73.15
0:48
SoLD
02.49
0.74
67
0.97
0.23
73.44
0.49
70.17 0.62
74.25
Predictive Estropy
64.00
0.15
68.74 4 0.65
69.77
0.46
13.78
0.87
70.65 = 0.31
74.11 2 0.14
PowerP
00:00
03002 -: 1:10
0864
0107
69:01 0.25
72.78
Razdoom
00
0.68
85,00
0.94
150.00
Raadoon 33% FG
35.30
26
56.79 1.02
05.27
155
08.81
1.08
05.27 1.39
08.81 1.08
Raneoom 66%
55.
2.52
44

---

Published in Transactions on Machine Learning Research (08/2025)
Wediitie Erienge 4MGS Lediumiatel Negane
Weferenee from Pelicee beteeyy Randon ses Fo Arros Hed Lied Regi
4.11
6.39
(a) 200 Epochs (absolute)
(b) 200 Epochs (difference)
Pridicive Entrepy AMDS Mediumn Rog
Diferrnce Fraditiva Driropy Random 65% AMOS Prccmmoned Medum
6.30-
4.5-
5.8-
0.39
0.00
2,091
-0.0
4.199
(c) Precomputed (absolute)
(d) Precomputed (difference)
Figure 13: Visualization of absolute values and the difference the percentage of voxels for classes alongside Final Dice performance on the AMOS Medium-Labee Regime from Predictive Entropy. shows that less data containing classes 11 12 (right & left adrenal gland) is queried by Predictive Entropy (also Random) (5% less of the overall voxels of that class), which is strongly correlated with the Final Dice for these classes being 0 for the 200 Epochs results (a) whereas it is al ~ 0.7 for the Precomputed models on the exact same data just trained for 500 epochs (c). This substantially reduces the performance gap compared to Random 66% FG as can be seen in (b &d).
45

---

Published in Transactions on Machine Learning Research (08/2025)
G.3 Noise strength in Noisy QMs Ablation
TLDR:
Given more data the optimal noise for a Noisy QM decreases.
Preemptively optimizing the noise parameter remains an open research question.
Our aim is to understand the influence of the noise strength for the Noisy QMs (PowerBALD, SoftrankBAL.D, PowerPE) in the experimental setup of our main study by an exemplary systematic ablation for PowerBALD.
For PowerBALD 3 is the parameter which perturbs the ranking of the BALD scores SBALD on a logarithmic scale with Gumbel noise follows:
PALD log(SBALD) +E
where c Gumbel(O, B-1). The standard deviation of is proportional to 8-1, meaning that smaller values of B introduce greater randomness in query selection, while larger values preserve the original ranking. As B + oo, the ranking remains unchanged after adding noise, whereas as 3 + O, query selection becomes entirely random. By varying 3, can control the balance between exploration and exploitation in the selection process. It has already been noted by Kirsch et al. (2023) that in later stages of training the correlation of queries for Greedy Methods due to top-k sampling decreases. We suspect therefore that the optimal choice of 3 will differ across our experiments leaving room for method improvement from the standard setting 8 = 1 [Kirsch et al. (2023)) we used in our main study.
To assess the influence of data distribution and label regime we perform experiments on the ACDC, AMOS and KiTS dataset for the Low-, Mediun- and High- Label Regime whilst varying the parameter 3 = (1, 5, 10, 20, 40, } with 3 = oo being identical to BALD. Generally we only analyze larger values of B as our implementation adds Gumbel noise on the mean aggregated scores leading to the standard deviation of aggregated values naturally being smaller than for singular values.
Using this experimental setting with the results shown in fig. 14 we aim to answer the following two questions (Q1-Q2):
Q1: How is optimal 3 influenced by amount of data? When evaluating the results on each dataset separately, observe that the optimal parameter of 3 with regard to the AUBC and Final Dice generally increases from Low to Medium to High Label Regime. This aligns with our broader observations that noise-perturbed QMs generally outperform their greedy counterparts in the early stages of AL but are often overtaken in later loops as training progresses. With regard to foreground efficiency, we observe a steady decrease for higher values of 3, converging toward the FG-Eff of BALD across all label regimes, indicating that the reduction in queried foreground voxels greater than the difference in performance.
Generally, the optimal B is therefore strongly correlated with the amount of data and increases with more data.
Q2: How to select optimal B preemptively Despite the observation from Q1, we do not identify a single, universally optimal value range of B across all datasets, as they differ greatly across the different datasets. On AMOS, optimal values range from to 5, with a sharp decline in performance for higher values. In ACDC, the optimal range shifts to 5-40, while KiTS, it spans 1-40. This indicates that dataset properties play an important role the optimal selection of this paramcter, such as but not limited to the number of classes and their diversity. Furthermore, we hypothesize the following design decisions of the AL Pipeline to be important: Training length, Query Method (uncertainties and aggregation function) and query patch size.
Based on this, we conclude that setting this value preemptively remains an open question.
46

---

Published in Transactions on Machine Learning Research (08/2025)
ACDC
AMOS
KiTS
60
AUBC
85
60
50
40
40
80
Final Dice
60
40
FG-EIf
20
10-
8-Parameter
B-Parameter
B-Parameter
Medium
High
Figure 14: The 3-parameter for PowerBALD plotted against AUBC, Final DICE and FG Eff. for the Low-; Medium- and High-Label Regimes. 8-values leading to the best AUBC and Final DICE tends to increase for higher budgets. This indicates that for higher budgets less ranking perturbations perform better. At the same time the FG Eff. decreases which shows that the reduction in perturbation means that more FG is queried.
Noise Strength Detailed Results
47

---

Published in Transactions on Machine Learning Research (08/2025)
Table 14: Ablating the influence of the noise parameter for PowerBALD. AUBC and Final Dice are reported with factor (x100) for improved readability. The values leading to the highest AUBC and Final Mean Dice increase for larger budgets across all datasets.
(a) ACDC
Dataee
ACDC Moliim
Label Regine
Low
High
Query Mechod
Powe
PoweBBLLD (b=5)
80.89
11
87.29 1 0.31
36 03
6:21
80.21
0.39
89.53
0.35
28
16
1 6.55
87. 15
0.51
90.40 0.34
18.31 1.85
(b=10)
122
PALLD (b-20)
81.45
1.11
86.01
0.79
33.60
8.96
$5.63
0.31
89.00
0.14
21
3.06
S7.02
0.46
90.28
0.20
14.25
1.35
PowerBALD (b=40)
80,00
09
86.31
0.34
88
4.46
85,39
0.75
89.43
0.28
19.28
3.48
$7.00
90.51
0.19
13.05
0.64
PAWeBBLLD (b=zcc
79.81
0.39
01
0.96
26.90
$5.83
0.45
89.62
0.15
21.85
87.74
0.38
90.17
0.18
10.90
1.14
(b) AMOS
Datnert
AM08
Regime
HEgh Fmnal Dice
Metric
AUBC
Final Dine
FG-ER
AUBC
Final Dise
FG-EH
AUBC
FG-EE
Query Method
PAW.D (b-1)
50.34
3.00
56.18
1.24
3.65
14.56
66. + 1.47
73.02
2.01
18.28
D:44
77 86
+ 0.14
S0.28
0.48
9.90
0.08
PowD (b=8)
47.72
1.70
18.61
1.31
4.72
3.70
60:87
0:97
86.76
76.11
1.65
82.58
0.60
4.56
0.57
PoweBALD (b=10)
12.89
1:229
12:32
3.03
13.01
0.50
50
03.99
0.06
1.06 0.17
39
1.09
$0.71
2,03
1.74
0.19
PowrBALD (b=20)
02.08
2.58
30,
2.02
-16.5
4.56
68
62
61.04
2.21
3.40
70.05
1.00
03.05
1.15
-1.22
0:27
(b-40)
3.50
37.08
6.50
02.37
1.00
PowerBALD (b=oo)
38.00
2.34
34.05
1,58
-2..66
8.50
62.56 + 2.74
50.26 2.73
1.54 4: 0.22
00 38
4: 0.70
74.95
2.38
-0.45
0.20
(c) KITS
Label Regime
Low Final
Medium Final
Tligh
AUBC
FG
AUBC
AUBC
Query Method
PowD
45.10
47
54.63
61
BOLe (b-5)
43.03
3.65
46.20
4.98
14,30
$4.80
1.30
63.02
1.43
7.89
0.24
62.62 + 1.09
68.17
0.36
10.56
0,39
BALD (b=10)
42.05
3.72
46.00
3.58
11.53
54.73
63.63
1.06
7.44
63.19
66.34
0.57
9.54
0.15
PowerBALD (b=20)
21.06
1:13
11:30
6.96
10.96
55,26
1.03
63.12
0.22
7.20
0.17
62.80
0.57
67.95 2:
0.70
9.06
0.28
ToweBALD (b=40)
30.31
1.30
43
4.69
6.93
02.01
1.46
6.54
0.14
62.51
0.63
08:25
0.61
9.21
0.39
BALD (b-no)
40.58
2.75
7,96
0.52
1.72
0.35
48

---

Published in Transactions on Machine Learning Research (08/2025)
4.4 Query Patch Size Ablation
TLDR:
1. AL methods are resilient with regard to smaller Query Patch Sizes, even though absolute annotated voxels are greatly reduced.
2. The Query Patch Size can induce substantial changes in method rankings, making it important to systematically evaluate the benefits of AL methods.
Here we aim to understand the inftuence of the query patch size parameter on our AL experiments.
The query patch size is hyperparameter of our AL pipeline, setting our work apart as are the first to allow completely free 3D Patch selection, differentiating our experimental setup from related work, which uses either 2D slice or 3D image queries.
To evaluate its influence, we repeat our entire main study with all four datasets with the respective query patch size halved along cach axis whilst koeping the mumber of patches for each label regime identical. We motivate these design decisions as we are interested in seeing whether a more fine-grained selection of areas helps AL methods and the annotation effort for smaller patches does not necessarily decrease linearly with the voxel size.
As the changes with regard to the query patch size make experiments across Label Regimes incomparable, we compare instead across the dataset mean ranking and the overall mean ranking. To do so, we first perform bootstrap sampling to obtain a mcan mothod ranking for each label regime of each dataset, which we then aggregate to the dataset and overall level. These mean aggregated rankings are then compared using Kendall's r (Kendall 1948) and significance test; the results are shown table 15 and the mean ranking values are shown in appendix G.4
With this setup, evaluate the following questions:
Q1: Does the Query Patch Size influence AL Performance? When comparing the Average Mean rank for Patchx 1 and Patchx 1. it appears that AL has improved Performance compared to Random strate- gies (appendix G.4.. especially with regard to the AUBC. Even though the absolute annotated voxels are reduced by a factor of 16 for Patchx 2. the trend indicates that the AL methods perform better compared to the Foreground Aware Random strategies on all datasets with the exception of AMOS where for both Patch Sizes the Foreground Aware Random strategies perform best.
When comparing the mean PPMs, observe that (fig. 15 similar trends also with the Predictive Entropy being the method with the best win/lose-ratio against Random 66% FG.
In conclusion, we observe that AL methods are surprisingly resilient with regard to the Query Patch Size.
Q2: How does the Query Patch Size influence the ranking? The mean rankings of the Final Dice across all datascts are stable, with Predictive Entropy being the best performing method, followed by most other AL methods, with Random FG 66% mixed in between, followed by Random FG 33%, and finally Random as the worst performing. For the AUBC we observe a change in trend for the smaller Query Patch Size where all Noisy QMs are outperformed by their Greedy counterparts. We hypothesize that there are two reasons for this behavior: the reduced amount of training data and/or the higher chance of highly similar patterns in the dataset, resulting in high uncertainty values.
On the dataset level, the trend is that for AMOS and KiTS the rankings across Query Patch Sizes are stable, whereas they are less so for Hippocampus and almost completely unstable for ACDC. On ACDC BALD and its derivatives perform better for the smaller than the larger Query Patch Size in terms of AUBC and Final Dice (table 18). On Hippocampus BALD and SoftrankBALD also perform better for the smaller than the larger Query Patch Size in terms of AUBC and Final Dice, PowerBALD less so presunably due to the noise parameter being too large (table 20
We conclude that different Query Patch Sizes can load to substantial differences in the ranking of QMs.
49

---

Published in Transactions on Machine Learning Research (08/2025)
ACDC
AMOS
Hippocampus
KiTS
Average
AUBC
0.357
0. 786
0.571
0.793
0.143
Final Dice
0.036
0.764
0.5
0.837
0.714
Table 15: How does the Query Patch Size influence method benchmarking? High values indicate that method rankings are consistent across different query patch sizes. Kendall's 7 correlation coefficients comparing the mean rankings for all datasets and each dataset separately with different patch sizes. two-sided test was performed with a significance level of a = 0.1. Colorscheme: Significant positive correlation, positive correlation, negative correlation, & negative correlation
significant
10
1.
15
BALD.
17
1.7
18
6.7
6.7
1.7
3.3
3.3
52
25
20
17
8.3
10
8.3
42
15
Agp Th Frpe-boo
Alp ten 00g
Powere
15
19
13
5.3
5/
23
17
Fowertt
25
0
10
17
52
15
12
Kanosn 2016
20
20
3G
10
3B
10
Fandeen 37% FG
27
20
17
22
23
45
Meun
Maan
18
BADD
HaveRALDD
Hesdictee Btsrpge
Raserf
Aanmi PD
BALO
PownBBB
Randum 25%
Westiche
Nansout
Patchx1/2 1/
(b) Patchx1
Figure 15: PPM aggregated over all Label Regimes for cach datasct for the Patch Size Ablation with size Patchx1/2 and Patchxl (Main Study).
Query Patch Size Ablation Detailed Results
50

---

Published in Transactions on Machine Learning Research (08/2025)
Table 16: FinneGrained Results for the patch ablation with setting Patchx for each dataset. Higher values are better and colorization goes from bright (best) to dark wrange Final Dice is reported with a factor (x100) for improved readability. AUBC, Final and Beta can only directly compared for each label regime on each dataset.
(a) ACDC
Dattaott
ACDC Medium Fisal Dise
Label Regime
Low
High Final Dise
Metric
AUBC
Final Dice
FC-EI
AUBC
FG-EI
AUBC
IF C-EEI
Query Method
BALD
68 03:223-
+ 2:31
77.72
1.45
200.04
445 25
75.80 =1 1.10
82.50
+ 1.24
140 13
452.06
70.50
1.05
53.90
+ 0.85
75 63
PoweBALLD
68 00.90011
5.61
75.24
2:32
306.70
94
77.07
1 1.21
60.27
1.39
$4.54
1.25
121 75
191.80
SoftrankBALD
3.68
73.98
240.78
76,84
0.47
1.56
Prodictive Entrropy
66:27
2.45
75.79
2.45
185.36
200.07
74.67
1.26
81.18
1.32
110.05
160.70
79.26
0.95
83.58 + 1.33
85.12 + 103.40
PowerP'
65
3.90
74.45
2.28
301.75
891.9
76.26
2,36
16
2.15
21 11
118.00
70.85
1.51
84. 1.55
133.05 05
271
9..66
481179
70.30
710.00
2652
Random 335 FC
67 08.88
+ 1.51
77.43
0.27
2166.09 + 07.30
70.08
1.38
81.65 + 1.01
127 45
64.58
70.55
1.12
64.44 + 0.32
85.072
Random 66% FG
64.33
1.17
73.77
0.55
101.88 + 47.06
74.00
01228
82.18
1.52
71.88
20.74
80.33
01.66
65.88
0.64
16.000 + 8.47
Table 17: AMOS
Dataset Regime
AMOS
Mediun Final Dice
Metrie
AUDC
Final Dice
FG
AUBC
FG-EE
AUBC
Final Dise
Queyy Method
BALD
-149.85
1.72
11..1
Powe
14.44
= 2.70
11.74
= 2.59
-247 20
763.03
21.71 + 1:28
25
2.25
8.77
16.48
40.14 = 1,86
42.30
1.47
8.16
3.06
RAI.D
13.63
2.60
11.30
1.68
-127 01
335.48
19.95 + 1.55
23
3.19
+0.92 8.31
36.13
2.40
30.37
-3.29
5.42
Predictive Entropy
1.98
PowerPE
15.18 4 2.35
13.00 = 4.63
-210.99
25 1 1.26
27.05
2,03
14.75 8.90
44.211
1.75
41.34
3.06
18.40
3.35
Random
12:78
2.02
8.80
1.91
-157 1457-99-
+ 1770012
16:14
1:02
16.00
3.32
-01.07
168 PM
37.00
1.57
37.28
3.44
-141
22.20
1,66
10.78
Random 65% PG
31.10 1 2.39
39.20 =: 034
135.25 1: 00.74
42 1 0.88
00.25
0.45
94.96 J: 28.61
02.07
0.73
70.71
0.00
51.43
8.30
(a) Hippocampus
Dataset
Hippocampus Final Dise
Regime
High Final Dice
Metric
AUBC
Final Diee
FG-E#
AUBC
FC-EI
AUBC
FG-
Query Method
BALD
58.
62
0.47
87.85
1 0.15
72.53 1 176.66 66.66
87.64 + 0.17
88 13
0.19
15:03
19
87.99
0.18
88.76
1: 0.08
12:35
1.38
PoWD
56.07
0.35
87.45
19.38
09
87.82
0.04
88.47
17.41
SoLD
50.44
0.33
87.06
0.32
73.37
1 156.28
87.54 1 0.17
58.27 - 0.10
16.64
19
67.92
0.07
85.66
0.14
15:25
1.73
Predictive Eatropy
86
0.22
87.69
63
130.1
87/43
1 0.11
88.41
0.09
13:37
22
67
0.14
88.74
0.09
12
1.77
PowerPP
87.26
87.43
0.11
19.52
87.00
85.43 0.15
Random
85 02
0.65
M6.74
0:31
118.84
225.57
87.16
+ 0.21
87.70
0.10
5.49 25.06
87
0.15
83.13
0.15
26.30 3.24
50% FG
Randon 60% FG
86.3
26.21-2
0.13
87.54
0.15
57.33
56.
87.49 1 0.21
88.2 27
0.10
1192
1.
87.85 0.21
88.54 1 0.17
12:31 0.66
(b) KITS
Label Regine
Low
Medium Final Diee
High Final Dice
Metric
AUBC
Final Dice
FG-EE
AUDC
FG-EH
AUBC
FC-EF
Query Mechod
RALD
25.10
0.65
31.76
4.51
87.05
38.56
3:27
43.25
3.79
$1.77
9.34
48.46
1.19
53.50
1.28
24.11
PowDBALD
27.91
L.74
29:39
1.50
185.27
580.70
41.70
1.09
45.50
1.40
77.07
13:37
49.80
0.96
54.00
1.25
43.11 1 17412
31.47
00.05
41.08
1.00
45.78
49.08
Predictive Entrupy
24.08
166
29.07
5.82
25.47 11.91
40.99
3.00
06.80
3.63
25.06
2.82
50.22
1.42
35.79
1.07
14.55
1.08
PowerP:
Random
22.00
1.62
22.55
100.75
$5.14
2.00
$7.95
90.63
136.75
42.15
1.05
41.35
1.60
47.22
74.17
Random 33% PG
23.88
43
83
1.46
19
31 52
37
0.74
1:29
15:15
11:00
42.28
11
19
.311
3161
0.15
7.79
31.12
0.26
40.24
42.55
0,69
51

---

Published in Transactions on Machine Learning Research (08/2025)
Table 18: ACDC Mean Ranks
Rank Mean Dice AUBC Main Patchx1-2
Rank Mean Dice Final Main
Setting
Patchx1-2
Query Method
BALD
3.83
3.25
2.50
2.92
PowerBAL.D
3.67
2.67
4.58
2.92
SoftrankBALD
3.92
3.17
3.25
4.33
Predictive Entropy
4.50
6.08
2.17
5.75
PowerPE
5.58
3.83
4.42
4.50
Random
8.00
8.00
8.00
8.00
Random 33% FG
5.42
4.08
7.00
3.58
Random 66% FG
1.08
4.92
4.08
4.00
Table 19: AMOS Mean Ranks
Rank Mean Dice AUBC
Rank Main
Mean Dice Final
Setting
Main Patchx1-2
Patchx1-2
Query Method
BALD
7.75
6.92
7.58
7.17
PowerBALD
3.58
4.50
4.08
4.92
SoftrankBALD
5.00
6.42
4.42
5.83
Predictive Entropy
6.92
4.83
5.42
4.00
PowerPE
3.42
3.33
4.08
3.50
Random
6.33
7.00
7.42
7.58
Random 33% FG
2.00
2.00
2.00
2.00
Random 66% FG
1.00
1.00
1.00
1.00
Table 20:
Hippocampus Mean Ranks
Rank Mean Dice AUBC
Rank Mean Dice Final Main
Setting
Main Patchx1-2
Patchx1-2
Query Method
BALD
2.33
1.67
2.67
1.17
PowerBALD
4.75
5.83
4.83
5.50
SoftrankBALD
3.33
2.67
3.00
3.08
Predictive Entropy
1.08
3.00
1.83
2.25
PowerPE
6.17
3.92
5.92
4.58
Random
6.83
7.92
6.83
8.00
Random 33% FG
5.17
7.00
4.33
7.00
Random 66% FG
6.33
4.00
6.58
4.42
Table 21: KITS Mean Ranks
Rank Mean Dice AUBC Main
Rank Mean Dice Final Main
Setting
Patchx1-2
Patchx1-2
Query Method
BALD
3.92
4.83
3.83
3.83
PowerBALD
3.50
1.92
3.58
3.75
SoftrankBALD
3.58
3.58
3.00
2.33
Predictive Entropy
2.75
3.58
2.58
2.67
PowerPE
3.50
1.67
3.67
3.08
Random
7.83
7.08
8.00
7.17
Random 33% FG
5.42
6.25
5.17
6.00
Random 66% FG
5.50
7.08
6.17
7.17
52

---

Published in Transactions on Machine Learning Research (08/2025)
Table 22: Average Mean Ranks over all datasets
Rank Mean Dice AUBC Main
Rank Mean Dice Final Main
Setting
Patchx1-2
Patchx1-2
Query Method
BALD
4.46
4.17
4.15
3.77
PowD
3.88
3.73
4.27
4.27
SoftrankBALD
3.96
3.96
3.42
3.90
Predietive Entropy
3.81
4.38
3.00
3.67
PowerPE
4.67
3.19
4.52
3.92
Random
7.25
7.50
7.56
7.69
Random 33% FG
4.50
4.83
4.62
4.65
Random 66% FG
3.48
4.25
4.46
4.15
Tatwass Fanaly Vatti (PaccloLL ACOC (MC)
Painwiss Ponalty Mabti (Pa:dedk/2 AMOS (SD)
BALD
6.7
6.7
20
-16
BALD
S.7
FowerBALD
67
27
6.7
PowerDALD
27
20
6.7
20
SehransBALD
Prvdiiivr Srivey
6.7
0
0
40
13
Predictive Entreay
20
0
13
13
5.7
ggnt euturuume
Bundan
5
Bardon
27
0
1.5
5.7
Rancom 65%
6.7
6.7
6.7
53
6.7
Tandom 20% FG
80
80
8D
80
80
Plar
3,7
3.7
55
4.7
13
Mican
26
35
25
22
3.7
BADD
AdtraciBBeD
derakiivee Eleerpyy
floeerif
Nenstom
Randion 40% PD
BALD
SatvenrBALD
Prodccie Entrary
Peweett
Axedino
Reviln 44% Hi
Kaniero
Resiam
Alyne the fommeee
ACDC
(b) AMOS
4.7
6.7
FowerBALD
PowerBALD
6.7
13
35
0
73
53
Pesditive
Fredictive Fitt
Agetten extpafcrm
Agrrtann eupoffum
Powwrre
6.7
20
27
38
13
0
13
13
80
60
Ransom 30%
0
6.77
Tandon 23% PC
8
20
20
Mean
BeD
Protictoo Eteepp
Rhwerf
Renstum
Ranszer VRG
BALD
DeweDBALD
Prodictoe Fewerrty
Bauddm 3P6
Nensaas
Woothee s.toefforet
Hippocampus
(d) KiTS
Figure 16: Pairwise Penalty Matrix aggregated over all Label Regimes for each dataset for the Patch Size Ablation with size Patchx1/2.
53

---

Published in Transactions on Machine Learning Research (08/2025)
H Leave-One-Out Analysis of Rankings on the Main Study
We additionally analyze the results of the main study shown in appendix F by means of computing the rankings for AUBC and Final Dice in leave-one-out fashion based on experimental seeds.
Results Alternative versions of the main overview figure (shown in fig. which are obtained by means of aggregating the mean rank for each scenario from the 4 leave-one-out rankings, are shown for the AUBC fig. 17 and for the Final Dice in fig.
Detailed results showing also the distribution of the four obtained rankings shown for the AUBC in fig. 19 and for the Final Dice in fig. 20
Tway: General groups of ranking performance of QMs can be observed in all scenarios where certain groups of QMs are better than others. Overall, based on this analysis, little overall changes compared to the ranking shown in fig. are observed.
Details Each experiment is performed with different seeds, therefore each ranking is obtained times.
54

---

Published in Transactions on Machine Learning Research (08/2025)
8
Mechod Rank (AUBC)
2-
High
High
Modium
High
ACDC
AMOS
Hippocampus
KITS
BALD
PowerPE
Random
Random 66% FC SoftrankBALD
Prodictive Entrapy
Random
Figure 17: Leave-One-One Overview of Main Study Overview for AUBC. Ranking of methods according to AUBC for each dataset and its Label Regimes (Low, Medium & High) alongside mean with standard deviations (bar).
0-
Method Rank (Prnl))
Low
Medium
High
High
Low
Medium
High
Medium
High
ACDC
ANOS
Hippocampus
KITS
Random
66% SoftrankBALD
PowerBALD
Predictive Entropy
Random 33% PC
Figure 18: Leave-One-Out Overview of Main Study for Final Dice. Ranking of methods according to Final Dice for each dataset and its Label Regimes (Low, Medium & High) alongside with standard deviations (bar).
55

---

Published in Transactions on Machine Learning Research (08/2025)
ADDC
Hippocampus
14
Figure 19: Leave-One-Out Detailed Results of Main Study for AUBC. Ranking of methods according to AUBC for each dataset and its Label Regimes (Low, Medium & High). A specific colored field of height at x-axis x denotes that for one of the four seeds the method corresponding to this color obtained in the leave-one-out (seed based) ranking place x.
56

---

Published in Transactions on Machine Learning Research (08/2025)
ACDC
Hippocampus
12
KITS
Figure 20: Leave-One-Out. Detailed Results of Main Study for Final Dice. Ranking of methods according to AUBC for each dataset and its Label Regimes (Low, Medium & High). A specific colored field of height 1 at x-axis denotes that for one of the four seeds the method corresponding to this color obtained in the leave-one-out (seed based) ranking place x.
57

---

Published in Transactions on Machine Learning Research (08/2025)
Model Prediction Visualizations
We provide exemplary visualizations of the predicted segmentation masks for different QMs for ACDC (fig.
21
AMOS (fig, 22 Hippocampus (fig. 23 and KiTS (fig. 24 of the Main Study. We selected the
following model configurations:
ACDC (fig. 21 Low-Label sctting of the main study (annotation budget: 150, query patch size: 4 40
40); seed: 12347
AMOS (fig. 2 Low-Label setting of the main study (annotation budget: 200, query patch size: 32
74 x 74); seed: 12347
Hippocampus (fig. 23:: Low-Label setting of the main study (annotation budget: 100, query patch size:
20 x 20 x 20); seed: 12345
KiTS (fig. 24): Low-Labol sctting of the main study (annotation budget: 200, query patch size: 64 x 64
64); seed: 12347
58

---

Published in Transactions on Machine Learning Research (08/2025)
GT
loop 3
loop
BALD
PowerBALD
PowerPE
Predictive Entropy
Random 33% FG
Random 68% FG
Random
SoftaanBBADD
Figure 21: Exemplary Model Predictions on ACDC for different QMs. Column 1: Ground Truth
(GT) segmentation masks; Column 2-6: predicted segmentations after each AL loop.
59

---

Published in Transactions on Machine Learning Research (08/2025)
GT
loop
loop 2
loop 3
loop
loop 5
BALD
PowerBALD
PowerPE
Predictive Entropy
Random 33% FG
Random 66% FG
Random
SoftrankBALD
Figure 22: Exemplary Model Predictions on AMOS for different QMs. Column 1: Ground Truth
(GT) segmentation masks; Column 2-6: predicted segmentations after each AL loop.
60

---

Published in Transactions on Machine Learning Research (08/2025)
GT
loop
loop 2
loop 3
loop
loop
BALD
PowerBALD
PowerPE
Predictive Entropy
Random 33% FG
Random 66%
Random
SoftranBBLD
Figure 23: Exemplary Model Predictions on Hippocampus for different QMs. Column 1: Ground
Truth (GT) segmentation masks; Column 2-6: predicted segmentations after each AL loop.
61

---

Published in Transactions on Machine Learning Research (08/2025)
GT
1000
loop 2
loop 3
loop
loop 5
BALD
PowerBBLD
PowerPE
Predictive Entropy
Random 33% FG
Random 66% FG
Random
SoftrankBALD
Figure 24: Exemplary Model Predictions on KiTS for different. QMs. Column 1: Ground Truth
(GT) segmentation masks; Column 2-6: predicted segmentations after each AL loop.
62


Reviews:
[{"review_id": "j6UPHkxLLi", "reviewer": "Reviewer_Y74B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors effectively solve an interesting and important problem: the lack of standardized AL benchmarks in 3D biomedical imaging is a genuine gap.\n2. With nnUNet, a widely adopted baseline, enhances usability and promotes reproducibility.\n3. The framework is tested across several datasets and active learning strategies, showcasing its flexibility and extensibility.\n4. The code is promised to be made publicly available, which aligns with the goals of community benchmarking.\n\nWeaknesses:\n1. The core contribution is a well-structured benchmark framework, but not a new algorithm or methodological innovation in active learning or segmentation; thus, the novelty is limited in methodology.\n2. While various AL strategies are implemented, the paper lacks deep theoretical or empirical exploration of why certain strategies perform better.\n3. While the framework simulates AL, it’s unclear how well it maps to real clinical workflows with human-in-the-loop feedback. An experiment with real annotators would strengthen the claims.\n4. The improvement margins of AL over random sampling are relatively small in several experiments, which may raise doubts about the practical benefit of AL in these settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents nnActive, a unified evaluation framework for active learning (AL) in 3D biomedical image segmentation. Built upon the established nnUNet pipeline, nnActive introduces a modular and reproducible setup for simulating various AL scenarios. It includes support for uncertainty-based sampling, retraining strategies, and multiple datasets. The paper aims to standardize benchmarking in AL for biomedical segmentation by providing a comprehensive and extendable platform.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Include additional statistical analysis (e.g., confidence intervals, significance tests) to substantiate empirical claims.\n2. Provide more guidance or recommendations based on the experiments, e.g., when is AL beneficial and when not?\n3. Discuss the computational overhead of retraining strategies and their implications in clinical settings.\n4. A qualitative analysis (e.g., visualizations of queried samples, segmentation errors) could offer more insight into what the model learns via AL.\n5. If feasible, add a real-world experiment with radiologists or annotators to validate simulation realism.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tpOZSpGi55", "reviewer": "Reviewer_aV1L", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper has the following metrics:\n\n1. Comprehensive related works and comparison with the previous method to show the benefits of this paper.  For example, the section on the requirement of Active learning evaluation and the analysis of different related baselines undoubtedly demonstrates the unique advantage of this paper in its detailed evaluation across various aspects.  The author further proposes solutions to address the limitations point by point \n\n\n2. This paper is well organized. The figures make it easier to understand the key concepts and compare with the previous method. Even the appendix is quite detailed and well-organzied. \n\n3. The experiments conducted are pretty comprehensive and convincing. The main study answers the performance of the current AL methods compared to the random baseline and the foreground-aware random baseline; the latter one is quite inspiring for the community's following direction, as current AL methods fail to outperform this strong baseline consistently. This paper also analyzes how the budget and dataset will influence the AL performance,  which also makes the results more convincing the a more generalized manner. Sufficient ablation has been provided to supplement the main study. \n\n\n\nWeakness.\n\nAs the author mentions, though this paper aims to be comprehensive, the mainly covers the uncertainty-based AL methods and does not cover the methods that change the architecture or the loss function. Therefore, I would like the author to reflect this in the title and introduction. The way it is organized now may express the wrong information.\n\nThe main takeaways in the final section can be more organized according to the topics. Right now it is a little bit scattered in different aspects.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Active learning is an important field for applying medical image segmentation methods in the medical domain, especially given that the annotation can be quite expensive and limited. In this paper, the author proposes nnActive, a highly configurable AL extension for nnUnet, utilizing 3D patch-level partial annotation. The other contributions are as follows:\n\n1. Investigate the foreground-aware sampling method as a stronger and more realistic baseline in AL, which tackles the class imbalance in the 3D images. This proves to be a strong baseline. \n\n2. Perform a large-scale analysis and experiments of different AL methods over 7500 nnU-Net trainings for more than 12 dataset settings. \n\n3. To evaluate the AL performance, FG-Eff is proposed as a novel metric to measure the annotation efficiency. Such a metric takes into account that the annotation of the background actually takes negligible effort compared to the foreground.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the above", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "K4I2Zj8nbK", "reviewer": "Reviewer_By9h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Very comprehensive experimentation that exhibits familiarity and respect of the medical imaging domain.\n- Provides several insights and conclusions that can be taken away by the reader. I can imagine this paper being useful for practitioners who want to deploy AL setups in real-world contexts.\n- Clear writing and easy to understand\n\nWeaknesses:\n- Regarding foreground-related arguments (including foreground-aware random sampling and FG-Eff), do the authors assume that the foreground can be readily extracted? I can imagine scenarios where the foreground is not clearly delineated. E.g. suppose one wants to segment structures (e.g. tumors) in one organ within an abdominal MRI scan. Can the authors comment on this?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work pinpoints four recurring pitfalls of AL methods in medical imaging: too few datasets/budgets are tested, 2-D models are trained on 3-D data without exploiting partial labels, random baselines are ill-suited to the foreground–background imbalance, and annotation “cost” is measured only in voxels rather than in true human effort.\n\nThe authors propose a framework for AL evaluation, including baselines, metrics, and benchmarks.\n\nThe authors run their framework on a suite of experiments and find that foreground-aware random sampling is tough to beat. Other insights were on best query methods, noisy vs. greedy selection, improvements with longer training, etc.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- QM abbreviation is used but never defined.\n- What does the model see during a single forward pass? Is it a cropped patch centered around a label? The authors say, \"However, we enhanced the standard patch-based model trainer through region sampling, enriching the region observed by the model with additional unlabeled context from the rest of the image.\" Concretely, does this mean including x number of voxels of the input patch about every dimension of the label patch?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)