Integrating Deep Metric Learning with Coreset for Active Learning in 3D Segmentation
Authors: Arvind Murari Vepa
ZUKANG YANG
Andrew Choi
Jungseock Joo
Fabien Scalzo
Yizhou Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: Active Learning
Medical Imaging
Segmentation
Deep Metric Learning
Primary area: active_learning
TL;DR: We propose a novel metric learning framework for Coreset for active learning in 3D medical image segmentation.

Abstract: Deep learning has seen remarkable advancements in machine learning, yet it often demands extensive annotated data. Tasks like 3D semantic segmentation impose a substantial annotation burden, especially in domains like medicine, where expert annotations drive up the cost. Active learning (AL) holds great potential to alleviate this annotation burden in 3D medical segmentation. The majority of existing AL methods, however, are not tailored to the medical domain. While weakly-supervised methods have been explored to reduce annotation burden, the fusion of AL with weak supervision remains unexplored, despite its potential to significantly reduce annotation costs. Additionally, there is little focus on slice-based AL for 3D segmentation, which can also significantly reduce costs in comparison to conventional volume-based AL. This paper introduces a novel metric learning method for Coreset to perform slice-based active learning in 3D medical segmentation. By merging contrastive learning with inherent data groupings in medical imaging, we learn a metric that emphasizes the relevant differences in samples for training 3D medical segmentation models. We perform comprehensive evaluations using both weak and full annotations across four datasets (medical and non-medical). Our findings demonstrate that our approach surpasses existing active learning techniques on both weak and full annotations and obtains superior performance with low-annotation budgets which is crucial in medical imaging. Source code for this project is available in the supplementary materials and on GitHub: https://github.com/arvindmvepa/al-seg.

Integrating Deep Metric Learning with Coreset for Active Learning in 3D Segmentation
Arvind Murari Vepa
Zukang Yang
Andrew Choi Horizon Robotics asj chwihuccl....ooooooooooooacaawae edu
UCLA amvepa@ucla.co edu
UC Berkeley zukangy@berkeley.coo edu
Jungseock Joo UCLA jjoo8comm..o ucla. edu
Fabien Scalzo UCLA fabθcs. www..uclllleoo. edu
Yizhou Sun UCLA yzsunčcs. ucla.edillee
Abstract
Deep learning has seen remarkable advancements in machine learning, yet it often demands extensive annotated data. Tasks like 3D semantic segmentation impose substantial annotation burden, especially in domains like medicine, where expert annotations drive up the cost. Active learning (AL) holds great potential alleviate this annotation burden in 3D medical segmentation. The majority of existing AL methods, however, are not tailored to the medical domain. While weakly-supervised methods have been explored to reduce annotation burden, the fusion of AL with weak supervision remains unexplored, despite its potential to significantly reduce annotation costs. Additionally, there little focus on slice-based AL for 3D segmentation, which can also significantly reduce costs in comparison to conventional volume-based AL. This paper introduces a novel metric learning method for Coreset to perform slice-based active learning in 3D medical segmentation. By merging contrastive learning with inherent data groupings medical imaging, we learn metric that emphasizes the relevant differences in samples for training 3D medical segmentation models. We perform comprehensive evaluations using both weak and full annotations across four datasets (medical and non-medical). Our findings demonstrate that our approach surpasses existing active learning techniques on both weak and full annotations and obtains superior performance with low-annotation budgets which is crucial in medical imaging. Source code for this project is available in the supplementary materials and GitHub: https:
com/ com/arvini arvindmvepa/all.. all-eg.
Introduction
In the field of 3D medical segmentation, manual annotation of entire volumes, despite being laborious and time-consuming has been the gold standard. Annotating single 2D image take minutes to hours depending on the complexity of the image (75 73 15 71 (), and 3D medical volume, containing up 200 slices, can require significant amount expert labor. Annotating full dataset not only imposes a significant time burden on medical experts but also incurs high costs. Therefore, active learning (AL) urgently needed to optimize annotation efforts.
Surprisingly, the potential AL in the context of 3D medical segmentation has not been extensively explored. Traditional AL techniques typically focus either diversity model uncertainty, often neglecting relevant groupings within the data. For example, slices from the same patient or volume tend to show consistent characteristics. We propose deep metric learning strategy that identifies and utilizes these similarities to better highlight diversity the active learning process. Diverse samples
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

help the model learn a wide variety of patterns and features, which can be crucial for generalization. While medical imaging has natural groupings which we can leverage, our approach extends to a wide range of real-world datasets, including video segmentation.
Several other strategies may also help in reducing costs. Most current AL approaches use volume- based AL (39) rather than slice-based AL for 3D segmentation, which tends to be more costly and less efficient, Alternatively, weakly supervised methods, which require simpler annotations like scribbles
(34 29 M. bounding boxes [33 83), points (82), or semi-automated techniques (5) 44 62] 56), have
been shown to perform comparably to fully supervised methods (72 40 54
B
26 However,
combining AL with weak supervision, especially in medical settings, remains unexplored. In our
research, we explore both slice-based AL and the integration of AL with weak supervision as potential cost-redueing measures.
In this paper, we present our contributions to AL for 3DD medical segmentation:
1. The first work to integrate deep metric learning with Coreset during active learning for 3D medical segmentation. Our approach shows superior performance across four datasets (medical and non non-medical) with low annotation budgets.
2. The first work to comprehensively compare new and existing algorithms for slice-based active learning for 3D medical segmentation utilizing both weak and full annotations.
2 Related work
Active learning AL methods can be broadly classified into 1) uncertainty- based and diversity-
based methods (76). Uncertainty- methods include deep bayesian methods (21) 45| 38) (65.
deep ensembles 314 52 32 20), contrastive learning methods (85 41 87
36). and geometry-
methods 199) Diversity- based methods include coreset-base -baed methods (64 7). clustering-
methods (33), variational adversarial learning methods (66) (37,, and random sampling methods (48).
A limitation of previous methods is that they are too general and fail to utilize common data groupings found in real-world datasets. Prior methods (67 69) have tried to incorporate groupings but do not utilize domain information to generate them, which can be suboptimal. Recent research has started integrating domain -ppeifi-ppecific data groupings into AL algorithms with some success(28] (7)), but these methods are designed for specific uses and lack broad upplicability. While other methods have adapted Coreset (31 33),, they are not specifically tailored to 3D medical segmentation as our method is.
Deep metric learning Metric learning is focused on developing methods to measure similarity between data points and used in many applications. Recently, metric learning has focused on deep learning-based feature representations for data points (46). Contrastive losses are popular for metric learning, including Triplet Loss (63) and NT-Xent loss (67). Several non-contrastive approaches have been proposed based on center loss T6), proxy-based methods 70: 27), and LLM guidance (61). One interesting approach is ensemble deep metric learning which combines embeddings from methods by factorizing the network training based on differerent objectives (74). However, these approaches can be computationally expensive and narrowly tailored to specific applications. Additionally, non- contrastive approaches often require class supervision to perform well. While some non-contrastive approaches do not (88 [8), they are also narrowly tailored to specific applications.
an ensemble of encoders (1 43 53 600 80). Recent work has improved on ensemble-bar
In contrast, contrastive learning is often used in self-supervised settings in diverse applications 10 2 (55). The NT-Xent loss specifically outperformed other contrastive losses self-supervised zhto-shot image classification and outperformed fully-supervised ResNer-50 (12). However, prior work (including in active learning (85) 23 (3)) does not leverage any inherent data groupings in the contrastive loss, which can be useful weak supervision. Additionally, recent work in active learning can be computationally expensive because they retrain the feature encoder after each active learning round (85).
Active learning in medical segmentation There has been some notable research on AL in medical segmentation. Earlier work utilized bootstrapping to estimate sample uncertainty (8I). In another work, the researchers built a mutual information-based metric between the labeled and unlabeled pools to improve diversity (49). However, both of these approaches are computationally expensive

---

dø (.)
Coreset
d
Unlabeled Samples
dø (I1, x22)
Previously Labeled Samples Su
Newly Labeled Samples S S. Metric Space
Unlabeled Sample
Embedding Distance
Contrastive Learning- based Encoder Je
L
Labeled Sample
22
Figure Overview our active learning pipeline
and not scalable for large datasets. There were also inefficiencies in previous studies, such as choosing whole volumes instead of individual slices 3D segmentation, which increased costs (39). Random sampling proved effective in some cases (6) and calculating uncertainty using stochastic batches was also effective (20). None of the prior approaches leverage the data groupings inherent medical 3D data and also do not focus on active learning with weak annotations (c.g., scribbles). While another method also utilizes groupings in medical data (86), their groupings are assumed be quite large and it's unclear how they would extend their group classification approach to when there are large number of patients, volumes, and adjacent slice groups like with our datasets.
Methodology
3.1 Problem Formulation
In this section, we formally describe the problem of active learning for segmentation. Let X' C RRxxxxd be the set of 3D volumes and y C (0,1)xxxxxxx be the set of 3D masks where h, w. correspond the height, width, and depth (number of slices) 3D volume and k refers to the number of classes. In 3D segmentation, we learn mapping F' -.
Consider loss function L F 3 R where J is the range of model prediction probabilities (F= = [0, 1| (xxxxdx)), We consider the dataset D a large collection of data points which are sampled i.i.d. over the space Z X as (x,, y: Jej pz. We additionally consider partially labeled subset s c. D. Thus, active learning for 3D segmentation can be formulated as follows:
argmin Exyx<pZ (0(x,p([((F,))) [C(
(1)
AsC AeCD,
where ÿ are the model predictions F(V) (where optimal requested labeled set, the active learning budget, and is the deep learning method learned from L As. The Coreset approach is method of solving Equation (64). However, there are two distinguishing factors in our current work: I) slice-based active learning for 3D segmentation and 2) deep metric learning. In blice-based 3D segmentation, we learn mapping f from X' + y" where X' C hxx ) C {0, 1] 1 hxxwxx and F(V) = [f (slicee)), .... fsslice)). Additionally, D is collection of slices which are sampled i.i.d. over the space Z' X" x )" as {x/, y'lie(n) pzr.
..., slicea] is the volume), As is the
In the original Coreset paper (64) 8 As is the set cover of D with radius d. However, the Euclidean metric used to calculate the radius does not utilize task-relevant information and may be sub-optimal. This motivates to consider deep metric learning to utilize more task-relle task-nelevant information, where do (21, x2) 2 is some parameterized metric. In the the original paper (64), the authors provide theorem

---

which bounds the loss from Equation I based on the radius ò. We show a very similar bound where 8 max min
(2) (we defer the precise bound and proof to the Appendix B). This leads
xDED #2 EsE
to our Coreset optimization formulation:
argmin max AxCR,JAA|ER FIED agEaUds
min
do(11, x22
The above formulation can be intuitively defined as follows; choose & additional center points such that the largest distance between data point and its nearest center is minimized (64).
3.2 Metric learning
The parameterized metric dø (01, x2) is defined as:
dø (11 x2)
go (z2))
where & is the Euclidean metric. Our goal is to learn Jo. or feature representation, that emphasizes task-relevant similarities and differences in the data for selecting diverse samples for Coreset. We do this by training a contrastive learning-based encoder with a unique Group-based Contrastive Learning (GCL) which utilizes inherent data groupings specific medical imaging to generate the feature representation. dø is then used by the Coreset algorithm to select the optimal set of slices. A flow chart illustrating our pipeline can be seen in Figure The annotations for the collected slices are then used to train segmentation model.
3.2.1 Group-based Contrastive Learning for feature representation in metric learning
While there may be several task-relevant groupings in 3D medical imaging, it is not immediately apparent which of these would be useful for feature representation for metric learning for Coreset. For the ACDC dataset, we note the mean pairwise absolute deviation of the normalized training slice pixel values within different groups averaged over the dataset: 0.217 over the entire dataset, 0.159 within patient groups, 0.166 within volume groups, and 0.115 within adjacent slice groups. Note that the volume groups and the adjacent slice groups are the most and least diverse respectively. While intuitively the most diverse group would have most important features for diversity, this does not necessarily indicate what combinations of groups would be helpful as well.
Instead, we propose a general group contrastive loss based on NT-Xent loss (67) The NT-Xent loss focuses on generating and comparing embeddings for image pairs and their augmentations. promotes similarity in embeddings for the same image and its augmentation while encouraging dissimilarity for different images. In the same vein, in our group- based loss, we promote similar embeddings for slices from the same group and dissimilar embeddings for slices from different groups, enhancing group-level representation. We define "group" as set of 2D slices associated with one patient. The formula as follows:
N
cxp(sim(z,,2;)/1
Ggroup
log
NG
Σx ((lxd))((lovaa)MMM
cxp (xpsimi/ 2x)/T)
1=1
j
in which i E {1, ..., N} the indices for the standard batch slices, E {N + 1, + 2,.,, 2N} are the indices for the augmented batch slices, 9: refers to the group associated with slice in the batch, ps refers the patient associated with slice in the batch, G represents the average number of slices per group (calculated as the total number of unaugmented slices divided by the total number of groups). is the embedding for slice in the batch, is temperature parameter, and sim is similarity function which was cosine similarity in this study. The formula shares similarities with NT-Xent loss but introduces some modifications:
1. There are two summations which reflects all the group slices for a particular data point.
2. In the numerator of the logarithmic term, the group slices for particular data point are considered similar, encouraging the development of similar embeddings for these slices.

---

3. The denominator excludes patient slices for particular data point that are not part of the same group
Excluding uon-group patient slices ensures that the model does not promote dissimilarity between non-group slices from the same patient. This ensures that we can sum multiple group losses together without counteracting their effects. In our study, we considered patient, volume, adjacent slice group contrastive losses in addition to NT-Xent loss. Our overall loss formulation is as follows:
Leontrastive LNT-Xent + X paticet group + Xa Colume group + À3 grou
(5)
where puiiet group Lvllume group, fisise group are the group contrastive losses associated with patient, volume, and adjacent slice groups respectively and (X1, )2, )3) are constants. The summation of
different contrastive losses in our overall loss Loontrattvve is the focus of the ablation study in Section
4.6
Batch sampler We employ Leontristvee to train a SimCLR network (35) using a ResNet-18 encoder 24), which is our dø- In practice, standard random data loading would yield minimal impact from Ccontrative due to the low probability of randomly selecting two slices from the same group (e.g., same patient, same volume, or adjacent slices) within batch. To address this, introduce a batch sampler designed to increase their occurrence. The batch sampling process can be summarized as follows: 1. create singular slice group for all the dataset slices, 2. for each group, randomly include a slice from the same patient for each of the groupings used (e.g., patient and volume), 3. combine groups from different patients to form batch. An epoch consists of all the dataset groups: thus, more contrastive loss groups result in larger epochs. The batch sampler would be reset every epoch to ensure randomness during training. Please see Appendix [o more details.
Training the SimCLR network with this batch sampler eliminates the need for complex algorithmic adjustments to accommodate multiple contrastive loss groups, challenge in other AL contrastive learning methodologies (85). We conduct the training over 100 epochs using an ADAM optimizer, with learning rate of 3e-4, a weight decay of 1.0c-6, and batch size of 8 for or three groups and 9 for two groups.
3.2.2 Segmentation model training
Once we have Is. we can calculate dø and collect annotations for unlabeled slices train segmentation model. Our AL evaluation consists of several rounds of annotation collection from an oracle. After each round of annotation collection, the model's test score with the highest validation score. We repeat the AL experiment five times for each algorithm, each time with different random seed, and report the average model test score per round. For veakly-supervised segmentation, we train a Dual-Branch Network with Dynamically Mixed Pseudo Labels Supervision (DMPLS) (40), which reported strong metrics the ACDC dataset using weak supervision. For full supervision, we train UNet (58) which is a frequently used fully-supervised segmentation bascline model. We calculate the 3D DICE score on each 3D volume and report the average on all the volumes in the test set. For full supervision, we also provide results using a pre-trained segmentation model with a ResNet-50 backbone (42) (pre-trained on medical images (42) for medical datasets and ImageNetV2 for non-medical datasets). We do not report weakly-superviseer results on pre-trained architectures because the veakly-supervised architectures cannot easily utilize pre-trained backbones.
train five segmentation models and record
4 Experiments
4.1 Datasets
ACDC (Automated Cardiac Diagnosis Challenge) The ACDC dataset (4) consists of 200 short- axis cine-MRI scans from 100 patients the training set and 100 scans from 50 patients in the test set. Acquired data was fully anonymized and handled within the regulations set by the local ethical committee of the Hospital of Dijon (France). Each patient has two annotated end-diastolic (ED) and end-systolic (ES) phase scans. Annotations are available for three structures: the right ventricle (RV), myocardium (Myo), and left ventricle (LV). Additionally, scribble annotations have been provided for each scan by a previous study (72). The training set size consists of 1448 slices.
5

---

CHAOS (Combined Healthy Abdominal Organ Segmentation) The CHAOS dataset (33) com- prises of abdominal CT images from 20 subjects in the training set. primarily used for liver and vessel segmentation. The anonymized dataset was collected from the Department of Radiology, Dokuz Eylul University Hospital, Izmir, Turkey and the study was approved by the Institutional Review Board of Dokuz. Eylul University. Each patient's CT scans contains approximately 144 slices. Binary segmentation masks for the liver are provided. We resampled, cropped, and normalized the images, following the process described in previous study (72). We partitioned the training set into training (75%), validation (10%), and test (15%) subsets. The training set size consists of 2351 slices.
MS-CMR (Multi-sequence Cardiac MR Segmentation Challenge) The MS-CMR dataset (22
89 78) contains late gadolinium enhancement (LEG) MRI images from 45 patients who underwent cardiomyopathy. The data has been collected from Shanghai Renji hospital with institutional ethics approval and has been anonymized. These images were multi-slice, acquired in the ventricular short-axis views. We obtained realistic and manual scribble annotations from prior study (84). Similar to this study (84). we split the data such that 25 patients were assigned to train, and 20 to test, resulting in 382 slices in the training set. For data processing, we resampled the images into a resolution of 1.37x1.37 mm, and then they were cropped or padded to a fixed size of 212 x 212 pixels. During training, the image pixel values were normalized to zero mean and unit variance.
validation,
DAVIS (Densely Annotated Video Segmentation) The DAVIS dataset (50 51 is densely annotated video dataset associated with the 2016 DAVIS and 2017 DAVIS Challenge. The dataset collection was partially funded by SNF and human subject data was collected ethically, to the best of our knowledge. In our study, we utilized the train and val sets associated with the 2016 DAVIS Challenge which contained 30 and 20 videos respectively and all frames with 480p resolution. While we used the 2016 DAVIS train set, we created the val and test set by further splitting the 2016 DAVIS val set into 5 and 15 videos respectively. Our train set consisted of 2079 densely annotated frames.
For additional generalizability of our approach. we evaluated our methodology on both weak and full supervision, depending data availability. Because the CHAOS, MS-CMR, and DAVIS dataset do not contain any hierarchical organization of multiple volumes/videos, we did not use the patient group loss. For video segmentation, in our contrastive loss treated each video "volume" and each video frame as a "slice"; thus, we considered both the volume and adjacent slice group loss.
4.2 Experimental settings
When conducting experiments with pre-trained segmentation models, for ACDC, CHAOS, and MS-CMR we collect annotations in cumulative increments of 2%, 3%, 4%, 5%, 10%, 15% 20% and 40% in each round. Because segmentation networks require more training data for natural images, for the DAVIS dataset we collect annotations in cumulative increments of 10%, 20% 30%, and 40% in each round. When conducting experiments with pre- pre-trained segmentation models, because of the benefit of prior pre-training, we collect annotations in cumulative increments of 1%, 2%, 3%, 4%, and 5% in each round.
Because solving the Coreset problem NP-Hard, we utilized the K-Center Greedy algorithm for our Coreset implementation (48), which is 2 OPT solution (64) and produces very competitive results in comparison to other more computationnally-ineeesss solutions. We compared our approach to vanilla Coreset (K- (K-Center Greedy) (64). Random Sampling, CoreGCN (7), TypiClust (23) (39), Stochastic Batches (using Deep Ensembles with Entropy) (20). VAAL (66). Deep Ensembles (utilizing Variance Ratio scoring) (3), and Bayesian Deep Learning (utilizing the BALD score) (21). To ensure a fair comparison, all approaches were evaluated using the same experimental settings.
All of our experiments were primarily conducted with a single Tesla V100 GPU on an internal cluster. Our contrastive learning- based encoder and segmentation models consumed approximately 3 GB of GPU memory while training. The contrastive encoder's 's training speed was approximately 40 slices/second which resulted in a training time of 100 minutes, 200 minutes, and 300 minutes on ACDC for one, two, and three group losses respectively. One single AI. experiment for our method on ACDC (eight rounds with five models trained per round) took approximately 24 hours and used about 400 MB of storage.

---

Table I: DICE scores for ACDC, MS-CMR, and CHAOS
Weakly-supervised
Fully supervisd
2%
3%
4%
5%
40%
2%
3%
4%
5%
40%
ACDC
BALD (1
44.8
54.8
61.4
66.2
86.4
53,8
66.0
67.9
71.7
89.3
Variance Ratio (3
43.3
45.0
54.2
61.4
85.9
52.6
62.2
66.6
69.1
86.9
Random 48
44.9
45.7
59.2
66.9
86.8
66.3
76.9
79.3
80.1
90.2
VAAL (66)
41.8
47.8
66.1
72.1
86.4
63.2
75.2
79.5
81.3
89.9
Coresct (64
45.2
49.8
68.7
70.1
86.9
58.9
69.3
75.7
81.9
90.2
TypiClust (23 39
44.8
45.6
67.6
73.1
85.7
66.6
75.8
79.5
81.7
89.5
Stochastic Batches 20
36.9
39.5
53.7
57.4
85.8
60.3
69.8
74.1
75,4
89.3
CoreGCN 7
40.8
49.3
69.1
74.1
86.3
67.7
74.9
79.0
80.7
89.6
Ours
52.3
59.8
73.3
76.1
86.4
70.9
77.4
81.6
82.5
90.2
Weakly-superviser
Fully-supervised
2%
3%
5%
40%
2%
3%
4% CHAOS
5%
40%
MS-CMR
BALD
37.0
48.9
57.4
60.0
85.8
79.6
80.7
80.4
81.1
95.8
Variance Ratio
41.3
46.8
52.8
55.3
84.3
74.9
72.9
76.6
76.2
92.8
Random
39.7
55.0
61.0
61.5
85.7
80.7
81.7
84.2
85.1
96.2
Coreset
28.7
53.6
56.9
58.7
86.7
80.0
80.4
81.2
88.1
96.5
Stochastic Batches
38.5
56.2
59.8
60.5
86.2
77.2
82.9
83.8
84.7
96.1
CoreGCN
27.0
48.7
57.1
57.2
85.5
67.8
77.7
77.8
74,4
94.3
Ours
44.3
53.3
63.4
63.5
86.3
80.5
82.5
85.9
90.3
96.3
Table DICE scores for DAVIS
Fully-supervised
10%
20%
30%
40%
Mean
BALD
43.6
42.0
43.4
43.8
43.1
Variance Ratio
36.1
31.2
34.9
40.7
35.6
Random
39.6
40.5
47.4
48.5
45.5
Coreset
31.7
39.4
42.2
42.1
41.2
Stochastic Batches
40.3
41.5
45.1
47.6
44.7
Ours
42.8
45.2
45.5
46.6
45.8
4.3 Results
Tables and 2 summarize the results from our experiments on weak and full annotations for the ACDC, MS CMR, CHAOS, and DAVIS datasets when trained from scratch. Our method excels in low annotation budget scenarios (2%-5% on the ACDC dataset, outperforming other methods by up to 10% in some cases. This advantage is vital in the medical field where annotation costs often high. On the MS-CMR, CHAOS, and DAVIS datasets, our method remains highly competitive,
Table 3: Mean DICE scores over all annotation datapoints with pre-trained weights
Fully-supervised
ACDC
CHAOS
DAVIS
Random
78.4
94.9
74.9
Stochastic Batches
77.3
95.0
75.1
Coreset
78,6
95.1
74.6
Ours
79,5
95.2
75.1

---

80
Dice
Ours (wcaklly-supervised 2D slices)
Ours (fully-supervvieed 2D slices)
40
Random (fully-supervised 3D volumes)
10
35
Annotation
Figure 2: Describes the relationship between model performance and annotation time for our method utilizing weakly and fully-supervised 2D slices and random sampling of fully-supervised 3D volumes on the ACDC dataset. Annotation % is measured as the percentage of the fully-labeled ACDC training data. For weak supervision, we extrapolate the percentage fully-labeled data based on equivalent annotation time (we follow prior work which assumes that annotators annotate scribbles 15x as fast as the full masks (72)). The dashed green line represents the performance our method using
2D slices with 40% of the ACDC training data.
Our method
CoreGCN
2%
3%
4%
5%
Coreset
Figure 3: Qualitative comparison our method, CoreGCN, and Coreset. Blue indicates agreement between model predictions and groundtruth masks and red indicates disagreement.
consistently achieving the highest close the highest performance for particular annotation level.
Additionally, algorithm consistently demonstrates superior performance in both weak and full annotation scenarios, unlike other top-performing methods which struggle one these settings. Compared to other algorithms, method shows superior performance different datasets well.
note that clinically acceptable DICE scores for similar medical segmentation tasks range from 0.5-0.9, depending the task (68) 25 75 3. However, even lower DICE scores can be clinically useful, especially for particular volumes with higher scores in conjunction with semi-automated segmentation methods (57).

---

Table Ablation study based on the mean DICE scores for the 2-5% weak annotation datapoint
Coreset
NT-Xent
Patient
Volume
Slice
mDICE
58.5
59.5
60.0
60.7
58.1
61.5
60.9
62.6
65.4
64.1
61.1
(a) NT-Xent Loss
(b) Our Loss
Figure 4: t-SNE visualization of dataset clusters generated by different go
Our sully-supervised experiments with pre-trained models are in Table 3 We saw improvements with our method in comparison to the best performing comparison methods. Please refer to Appendix [ for comprehensive results across all datasets, including calculated bootstrapping standard errors.
4.4 Relationship between model performance and annotation time
In Figure 2 we provided graph which describes the relationship between model performance and annotation time for our method utilizing weakly and sully-sue 2D slices and random sampling of supev-sue 3D volumes different methods, we do not incorporate any the results from the pre-trained segmentation models. For the 3D results, similar to the 2D U-Net, train 3D U-Net from seratch. Given comparable annotation time, our methods trained on both weakly and fully-supervised 2D slices far exceed the performance of random sampling of 3D volumes and achieve 3D volume maximum performance (with the given budget) with much less annotation time.
the ACDC dataset. To ensure a fair comparison between the
4.5 Comparison with related work
Of the diversity- based comparison methods (Coreset, VAAL, TypiClust, Random), the performance for these three are generally worse than our method. For example, our method achieves the best
performance on out of comparison points (Tables 02 3. The next closest Random sampling, which achieves the best performance on 5 out of 27 comparison points. Of the entropy-based methods (BALD, Variance Ratio, Stochastic Batches), Stochastic Batches achieves the best performance on out of 27 comparison points, the best out of the group.

---

Coreset and CoreGCN share similarities with our method. However, on almost all the comparison points, they perform much worse. In Figure E there is a qualitative comparison of our method, CoreGCN, and Coreset on difficult slice in the volume after several weak annotation rounds. With more requested annotations, our method is able to reduce errors even difficult slices. With 5% weak annotation, while Coreset and CoreGCN still retain large error artifacts, our method has minimally visible errors.
4.6 Ablation study
In T we see our ablation experiments. The first three sections represent our experiments with one, tWO, and three more contrastive losses respectively. We note that all the contrastive loss experiments perform better than vanilla Coreset. Additionally, generally the larger combination of losses tend outperform the smaller combination of losses. The best loss combination involves the volume group loss, patient group loss, and NT-Xent.
One interesting observation is that the loss associated with the volume group the most diverse group- - tends to produce the best additive performance and the loss associated with the the adjacent slice group the least diverse group tends have the worst additive performance. We see this trend in the one loss experiments as well in the two loss and three or more loss experiments, where the combinations with the volume group loss tend to produce the best performance and the combinations with the adjacent slice group loss tends to produce the worst performance.
In order to visualize how effective the learned de trained with different losses are for Coreset, we applied k-means clustering to the generated dataset features with a contrastive encoder trained with NT-Xent and trained with optimal loss and visualized the quality of cluster labels using t-SNE plot, which can be seen in Figure 4 We note that the clusters from our loss show good cohesion (tightly grouped), separation between clusters, and are easy to differentiate. However, the clusters from NT-Xent show much less cohesion (points are spread over more space) and the separation is less defined. Higher quality clustering emphasizes points well separated which leads to better performance when trying to find representative points using Coreset.
5 Conclusion
In our research, we introduced novel metric learning method for Coreset to perform slice-based active learning 3D medical segmentation. By leveraging diverse data groups in our feature representation, we were able to learn a metric that promoted diversity and our Coreset implementation was able to outperform all existing methods on medical and non-medical datasets weak and full annotation scenarios with low annotation budget. Due to limited computational resources, restricted the number of experiment runs and models we trained. We also acknowledge that we did not fully consider training set bias which can result in unfair outcomes for underrepresented groups. In future work, we hope to remedy some of these issues and focus more robustly on the applications of our approach in diverse domains.
Acknowledgments
This work was partially supported by NSF 2211557, NSF 1937599, NSF 2119643, NSF 2303037, NSF 2312501, NASA, SRC JUMP 2.0 Center, Amazon Research Awards, and Snapchat Gifts.
10

---

References
[1] Aziere, N., Todorovie, S.: Ensemble deep manifold similarity learning using hard proxies. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7299-7307 (2019)
[2] Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems 33, 12449-12460 (2020)
[3] Beluch, W.H., Genewein, T., Nürnberger, A., Köhler, J.M.; The power of ensembles for active learning in image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp.
(2018)
[4] Bernard, O., Lalande. A., Zotti, C.. Cervenansky, F., Yang, X., Heng, P.A.. Cetin, I., Lekadir, K., Camara, O., Ballester, M.A.G., al.: Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging (7)11), 2514-2525 (2018)
[5] Bianconi, F., Fravolini, M.L.,, Pizzoli, S., Palumbo, I., Minestrini, M., Rondini, M., Nuvoli, S., Spanu, A., Palumbo, B.: Comparative evaluation of conventional and deep learning methods for semi-automated segmentation of pulmonary nodules on Quantitative imaging in medicine and surgery 11(7), 3286 (2021)
[6] Burmeister, J.M., Rosas, M.F., Hagemann, J., Kordt, J., Blum, J., Shabo, S., Bergner, B., Lippert, C.: Less is more: A comparison of active learning strategies for 3d medical image segmentation. arXiv preprint (tX.ix:22770008444 (2022)
[7] Caramalau, R., Bhattarai, B., Kim, T.K.: Sequential graph convolutional network for active learning. In: Proceedings of the IEEE/CVF conference computer vision and pattern recogni- tion. pp. 9583-9592 (2021)
[8] Cardenas, C.E., McCarroll, R.E., Court, I.F.,, Elgohari, B.A., Elhalawani, H., Fuller, C.D., Kamal, M.J., Meheissen, M.A., Mohamed, A.S., Rao, A., et al.: Deep learning algorithm for auto-delineation of high-risk oropharyngeal clinical target volumes with built-in dice similarity coefficient parameter optimization function. International Journal of Radiation Oncology* Biology* Physics 101(2), 468-478 (2018)
[9] Chaitanya, K., Erdil, E., Karani, N., Konukoglu, E.: Contrastive learning of global and local fea- tures for medical image segmentation with limited annotations. Advances in neural information processing systems 33, 12546-12533 12558 (2020)
[10] Chen, L., Bentley, P., Mori, K., Misawa, K., Fujiwara, M., Rueckert, D.:
Self- supersuper learning for medical image analysis using image context restoration. Medical image analysis 58, 101539 (2019)
[11] Chen, Q. Hong, Y:: Scribble2d:: Weakly-supervised volumetric image segmentation via scribble annotations. In: International Conference Medical Image Computing and Computer- Assisted Intervention. pp. 234-243. Springer (2022)
[12] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: simple framework for contrastive learning of visual representations, In: International conference machine learning. pp. 1597-1607. PMLR (2020)
[13] Chibane, J., Engelmann, F,, Anh Tran, T., Pons-Moll, G.: Box2mask: Weakly supervised 3d
semantic instance segmentation using bounding boxes. In: European Conference on Computer Vision. pp. 681-699. Springer (2022)
[14] Chitta, K., Alvarez, J.M., Lesnikowski, A.: Large-scale visual active learning with deep probabilistic ensembles. arXiv preprint trXiv:1811.0355 (2018)
[15] Çiçek, Ö., Abdulkadir, A., Lienkamp. S.S., Brox, T., Ronneberger, 3d u-net: learning dense volumetric segmentation from sparse annotation. In: Medical Image Computing and Computer-Assist Intervention-MIICAA 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19. pp. 424-432. Springer (2016)

---

[16] Deng, J., Guo, J., Liu, T., Gong, M., Zafeiriou, S.: Sub-center areface: Boosting face recognition by large-scale noisy web faces. In: Computer Vision ECCV 2020: 16th European Conference. Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16. pp. 741-757. Springer (2020)
[17] Deng. J., Guo, J., Xue, N., Zafeiriou, S.: Areface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4690 4699 (2019)
[18] Dutta, U.K., Harandi, M., Shekhar, C.C.: Semi-supervised metric learning: A deep resurrection. In: Proceedings of the AAAI Conference on artificial intelligence. vol. 35, pp. 7279-7287 (2021)
[19] Franco, I.,, Mandica, P., Kallidromitis, K., Guillory, D., Li, Y.T., Galasso, F: Hyperbolic active learning for semantic segmentation under domain shift. arXiv preprint arXiv:2306.J 1180 (2023)
[20] Gaillochet, M., Desrosiers, C., Lombaert, H.: Active learning for medical image segmentation with stochastic batches. arXiv preprint arXiv:2301.00 1.06700 (2023)
[21] Gal, Y., Islam, R., Ghahramani, Z.: Deep bayesian active learning with image data. In: Interna- tional conference on machine learning, pp. 1183- 1192. PMLR (2017)
[22] Gao, S., Zhou, H., Gao, Y.. Zhuang, X.: Bayeseg: Bayesian modeling for medical image segmentation with interpretable generalizability (2023)
[23] Hacohen, G., Dekel, A., Weinshall, D.: Active learning on budget: Opposite strategies suit high and low budgets. arXiv preprint wrXiv:220.002999 (2022)
[24] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 (2016)
[25] Hoebel, K.V.: Domain and User-Centered Machine Learning for Medical Image Analysis. Ph.D. thesis, Massachusetts Institute of Technology (2023)
[26] Huang, Z., Guo, Y., Zhang, N., Huang, X., Decazes, P., Becker, S., Ruan, S.: Multi-scale feature Aimilarity-basee weakly supervised lymphoma segmentation in pet/ct images. Computers in Biology and Medicine 151, 106230 (2022)
[27] Jawade, B., Mohan, D.D., Ali, N.M., Setlur, S., Govindaraju, V.: Napreg: nouns proxies reg-
ularization for semantically aware cross-modal embeddings. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1135-1144 (2023)
[28] Ji, W., Liang, R., Zheng, 7.,, Zhang, W., Zhang, S., Li, J., Li, M., Chua, T.s.: Are binary annotations sufficient? video moment retrieval via hierarchical uncertainty- based active learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23013-23022 (2023)
[29] Ji, 7., Shen, Y., Ma, C., Gao, M.: Scribble-based hierarchical weakly supervised learning for brain tumor segmentation. In: Medical Image Computing and Computer Assisted Intervention- MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Pro- ceedings, Part III 22. pp. 175-183. Springer (2019)
[30] Jin, Q., Yuan, M., Qiao, Q., Song, Z.: One-shot active learning for image segmentation via contrastive learning and diversity-based sampling. Knowledge-Bard -aade Systems 241, 108278 (2022)
[31] Ju. Jung, H., Oh, Y., Kim, J.: Extending contrastive learning to unsupervised coreset selection. IEEE Access 10, 7704-7715 (2022)
[32] Jung, S., Kim, S., Lee, J.: A simple yet powerful deep active learning with snapshots ensembles. In: The Eleventh International Conference Learning Representations (2022)
[33] Kavur, A.E., Selver, M.A., Dicle, O., Barış, M., Gezer, N.S.: CHAOS Com-
bined (CT-MR) Healthy Abdominal Organ Segmentation Challenge Data (Apr 2019). http::/itt.orry/1..2222/moooo... http:///mooo...222....ttts/pp// org/ 5281/2e0od0oooo 33324224400033322844
12

---

[34] Ke, T.W. Hwang, JJJ, Yu, S.X.: Universal weakly supervised segmentation by pixel-to- contrastive learning, arXiv preprint arXiv:2105.00951 (2021)
[35] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C.,
Krishnan, D.: Supervised contrastive learning, Advances in neural information processing
systems 33, 18661-18673 (2020)
[36] Kim, J., Kim, J., Hwang, S.: Deep active learning with contrastive learning under realistic data pool assumptions. arXiv preprint arXiv:2303. 14433 (2023)
[37] Kim, K., Park, D., Kim, K.I., Chun, S.Y.: Task-aware variational adversarial active learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8166-8177 8175-817540 (2021)
[38] Kirsch, A., Van Amersfoort, J., Gal, Y.: Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems 32 (2019)
[39] Liu, H., Li, H., Yao, X., Fan, Y., Hu, D., Dawant, B.M., Nath, V., Xu, Z., Oguz, I.: Colossal: A benchmark for cold-start active learning for 3d medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assiin Assisted Intervention. pp. 25-34. Springer (2023)
[40] Luo, X., Hu, M., Liao, W., Zhai, S., Song, T., Wang, G., Zhang, S.: Scribble-supervised medical image segmentation via dual- mual-branch network and dynamically mixed pseudo labels supervision. In: International Conference on Medical Image Computing and Computer- Intervention. pp. 528-538. Springer (2022)
[41] Margatina, K., Vernikos, G., Barrault, L., Aletras, N.: Active learning by acquiring contrastive examples. arviv preprint arXiv:2109.03764 (2021)
[42] MH Nguyen, D., Nguyen, H.,, Diep, N., Pham, T.N., Cao, T., Nguyen, B., Swoboda, P., Ho, N., Albarqouni, S., Xie, P., al.: Lvm-med: Learning large-scale self-superrised vision models for medical imaging via second-orded graph matching. Advances in Neural Information Processing Systems 36 (2024)
[43] Michael, O., Georg, W. Horst, P., Horst, B.; Deep metric learning with bier: Boosting independent embeddings robustly. IEEE TPAMI 42(2), 276-290 (2018)
[44] Militello, C., Rundo, L., Dimarco, M., Orlando, A., Conti, V., Woitek, R., D' D'Angelo, I., Bartolotta, T.V., Russo, G.: Semi-automated and interactive segmentation of contrast-enhancing masses on breast dce-mri using spatial fuzzy clustering. Biomedical Signal Processing and Control 71, 103113 (2022)
[45] Mohamadi, S., Amindavar, H.: Deep bayesian active learning, a brief survey on recent advances. arXiv preprint arXiv:201 2.000044 (2020)
[46] Mohan, D.D., Jawade, B., Setlur, S., Govindaraju, V.: Deep metrie learning for computer vision: brief overview. Handbook of Statistics 48, 59-79 (2023)
[47] Movshovitz-AAtias, Y., Toshev, A., Leung, T.K., Ioffe, S., Singh, S.: No fuss distance metric learning using proxies. In: Proceedings of the IEEE international conference computer vision. pp. 360-368 (2017)
[48] Munjal, P., Hayat, N., Hayat, M., Sourati, J., Khan, S.: Towards robust and reproducible active
learning using neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 223-232 (2022)
[49] Nath, V., Yang, D., Landman, B.A., Xu, D., Roth, H.R.: Diminishing uncertainty within the training pool: Active learning for medical image segmentation. IEEE Transactions on Medical Imaging 40(10), 2534 2547 (2020)
[50] Perazzi, F., Pont-Tuset J., McWilliams, B., Van Gool, L., Gross, M., Sorkine-Hornund A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 724-732 (2016)
13

---

[51] Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Corkine-Hornung A., Van Gool, L.: The 2017 davis challenge on video object segmentation. arXiv preprint 1.XXxx1704.00677 (2017)
[52] Pop, R., Fulop, P.: Deep ensemble bayesian active learning: Addressing the mode collapse issue in monte carlo dropout via ensembles. arXiv preprint arXiv: 1.03897 (2018)
[53] Qian, Q.. Shang, L., Sun, B., Hu, J., Li, H., Jin, R.: Softuriple loss: Deep metric learning without triplet sampling. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6450-6458 (2019)
[54] Qian, Z., Li, K., Lai, M., Chang, E.I.C., Wei, B., Fan, Y., Xu, Y:: Transformer based multiple instance learning for weakly supervised histopathology image segmentation. In: International Conference on Medical Image Computing and Computer-Assiste Intervention. pp. 160-170. Springer (2022)
[55] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748-8763. PMLR (2021)
[56] Ren, H., Zhou, L., Liu, G., Peng, X., Shi, W., Xu, H., Shan, F., Liu, L.: An unsupervised semi-automated pulmonary nodule segmentation method based on enhanced region growing. Quantitative Imaging in Medicine and Surgery 10(1), 233 (2020)
[57] Rhee, D.J., Cardenas, C.E., Elhalawani, H., McCarroll, R., Zhang, L., Yang. J., Garden, A.S., Peterson, C.B., Beadle, B.M., Court, L.E.: Automatic detection of contouring errors using convolutional neural networks. Medical physies 46(11), 5086-5097 (2019)
[58] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assiste Intervention-MIC MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. pp. 234-241. Springer (2015)
[59] Rossetti, S., Zappia, D., Sanzari, M., Schaerf, M., Pirri, F: Max pooling with vision trans- formers reconciles class and shape in weakly supervised semantic segmentation. In: European Conference on Computer Vision. pp. 446-463. Springer (2022)
[60] Roth, K., Brattoli, B., Ommer, B.: Mic: Mining interclass characteristics for improved metric learning. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8000-8009 (2019)
[61] Roth, K., Vinyals, O., Akata, Z.: Integrating language guidance into vision-based deep metric learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16177-16189 (2022)
[62] Sakinis, T., Milletari, F., Roth, H., Korfiatis, P., Kostandy, P., Philbrick, K., Akkus, Z., Xu, Z., Xu, D., Erickson, B.J.: Interactive segmentation of medical images through fully convolutional neural networks. arXiv preprint 19Xiv:1903.8822 (2019)
[63] Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: A unified embedding for face recognition and clustering, In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 815-823 (2015)
[64] Sener, O., Savarese, S.: Active learning for convolutional neural networks: A core-set approach. arXiv preprint 1rXiv:
(2017)
[65] Siddhant, A., Lipton, Z.C.: Deep bayesian active learning for natural language processing: Results of a large-scale empirical study. arXiv preprint arXiv:1808 0068800577778 (2018)
[66] Sinha, S., Ebrahimi, S., Darrell, T.:
Variational adversarial active learning. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 5972-5981 (2019)
[67] Sohn, K.: Improved deep metric learning with multi-class n-pair loss objective, Advances in neural information processing systems 29 (2016)
14

---

[68] Sun, Y., Wang, Y., Gan, K., Wang, Y., Chen, Y., Ge, Y., Yuan, J., Xu, H.: Reliable delineation of clinical target volumes for cervical cancer radiytherapy on ct/mr dual-modality images. Journal of Imaging Informatics in Medicine pp- 1-14 (2024)
[69] Taleb, A., Loetzsch, W., Danz, N., Severin, J., Gaertner, T., Bergner, B., Lippert, C.: 3d self- supervised methods for medical imaging. Advances in neural information processing systems 33, 18158-18172 (2020)
[70] Teh, E.W. DeVries, T., Taylor, G.W.; Proxynca++: Revisiting and revitalizing proxy neigh- borhood component analysis. In: Computer Vision-ECCV 2020: 16th European Conference,
Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIV 16. pp. 448 448-464. Springer (2020)
[71] Tsai, D.M., Fan, S.K.S., Chou, Y.H.: Auto-annotated deep segmentation for surface defect detection. IEEE Transactions Instrumentation and Measurement 70, 1-10 (2021)
[72] Valvano, G., Leo, A., Tsaftaris, S.A.: Learning to segment from scribbles using multi-scale adversarial attention gates. IEEE Transactions on Medical Imaging 40(8), 1990-2001 (2021)
[73] Vepa, A., Choi, A., Nakhaei, N., Lee, W., Stier, N., Vu, A., Jenkins, G., Yang. X., Shergill, M., Desphy, M., et al.: Weakly-supervisedd convolutional neural networks for vessel segmentation in cerebral angiography. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 585-594 (2022)
[74] Wang, C., Zheng, W., Li. J., Zhou, J., Lu, J.: Deep factorized metric learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7672-7682 (2023)
[75] Wang. S., Li, C., Wang. R., Liu, Z., Wang, M., Tan, H., Wu, Y., Liu, X., Sun, H., Yang, R., et al.: Annotation-efficien deep learning for automatic medical image segmentation. Nature communications 12(1), 5915 (2021)
[76] Wang. T., Li, X., Yang. P., Hu, G., Zeng, X., Huang, S., Xu, C.Z., Xu, M.: Boosting active learning via improving test performance. In: Proceedings of the AAAI Conference Artificial Intelligence. vol. 36, pp. 8566-8574 (2022)
[77] Wen, Y., Zhang. K., Li, Z., Qiao, Y.: A discriminative feature learning approach for deep face recognition. In: Computer Vici 2016: 14th European conference, amsterdam, the netherlands, October 11 11-14, 2016, proceedings, part VII 14. pp. 499-515 Springer (2016)
[78] Wu, F., Zhuang, X.: Minimizing estimated risks unlabeled data: A new formulation for semi-supervised medical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(5), 6021-6036 (2022)
[79] Xie. B., Yuan, L., Li. S., Liu, C.H., Cheng. X., Wang, G.: Active learning for domain adaptation: An energy-based based approach. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp.
(2022)
[80] Xuan, H., Souvenir, R., Pless, R.: Deep randomized ensembles for metric learning. In: Proceed- ings of the European conference computer vision (ECCV). pp. 723-734 (2018)
[81] Yang, L., Zhang, Y., Chen, J., Zhang, S., Chen, D.Z.: Suggestive annotation: deep active learning framework for biomedical image segmentation. In: Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 1 11-13, 2017, Proceedings, Part III 20. pp. 399-3999-- Springer (2017)
[82] Zhang, B., Xiao, J., Jiao, J., Wei, Y., Zhao, Y.: Affinity attention graph neural network for weakly supervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(11), 8082-8096 (2021)
[83] Zhang, H., Burrows, L., Meng, Y., Sculthorpe, D., Mukherjee, A., Coupland, S.E., Chen, K., Zheng, Y.: Weakly supervised segmentation with point annotations for histopathology images via contrast-ban bontea variational model. In: Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15630-15640 (2023)
15

---

[84] Zhang, K., Zhuang, X.: Cyclemix: A holistic strategy for medical image segmentation from scribble supervision. arXiv preprint arXiv:2203.01475 (2022)
[85] Zhang, Y., Zhang, X., Xie, L., J., Qiu, R.C., Hu, H., Tian, Q.: One-bit active query with contrastive pairs. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9697-9705 (2022)
[86] Zheng, H., Han, J.. Wang, H., Yang. L., Zhao, Z., Wang, C., Chen, D.Z.: Hicrarchical self- supervised learning for medical image segmentation based on multi-domain data aggregation. In: Medical Image Computing and Computer Assisted InterventionMMICAAI 2021: 24th International Conference, Strasbourg, France, September 27-October 2021, Proceedings, Part I 24. pp. 622-632 Springer (2021)
[87] Zhu, Y., Xu, W., Liu, Q. Wu, S.: When contrastive learning meets active learning: A novel graph active learning paradigm with self-supervision. arXiv preprint arXiv:2010.16091 (2020)
[88] Zhuang, F., Moulin, P.: Deep semi-supervised metric learning with mixed label propagation. In: Proceedings of the IEEE/VV Conference Computer Vision and Pattern Recognition, pp. 1429-3433 (2023)
[89] Zhuang, X.: Multivariate mixture model for myocardial segmentation combining multi-soures images. IEEE transactions on pattern analysis and machine intelligence 41(12), 2933-2946 (2018)
16

---

A Loss weights
In experiments and ablation study, a 1.0 weight was applied whenever the NT-Xent loss was used. Additionally, a 1.0 weight was applied if only one group contrastive loss was used. In the ablation study on the ACDC dataset, for the experiments with multiple group contrastive losses, tried different combinations weights for the group contrastive losses and reported the best results. We will utilize the formulation from Equation for the loss weights and report them as a four-tuple (a, b, c, d) which corresponds to (Ao, Aι X2, A3) (in which Ag is 1 if the NT-Xent loss was used and O if is not). The combinations we tried
follow (with the best bolded):
patient and volume group loss without NT-Xent loss: (0, 0.125, 0.875, 0), (0, 0.50, 0.50, 0)
volume and slice group loss without NT-Xent loss: (0, 0, 0.125, 0.875), (0, 0, 0.50, 0.50)
patient and volume group loss with NT-Xent loss: (1,0. 10, 0.35, 0), (1, 0.117, 0.233, 0), (1, 0.05, 0.35, 0) patient, volume, and slice group loss without NT-Xent loss: (0, 0.05, 0.25, 0.7), (0, 0.33, 0.33, 0.33)
patient, volume, and slice group loss with NT-Xent loss: (1, 0.10, 0.20, 0.05), (1, 0.05, 0.35, 0.025)
We utilized the best loss/weight combination for our ACDC experiments. For the CHAOS, MSCMR, and DAVIS datasets, because there is only one volume per patient and thus no difference between the patient and volume loss, we tested two loss/weight combinations:
volume loss with weight 0.35 with NT-Xent loss
volume loss with weight 0.10 and slice loss with weight 0.30 with NT-Xent loss
Both weight combinations performed better than other comparison methods though volume loss with weight 0.35 with NT-Xent loss performed slightly better than the other combination.
For the pre- pre-trained weights, found that best results the patient group without the NT-Xent loss. We tested different combinations of groups but found the results were worse. We used the same weight setting for CHAOS and DAVIS
obtained on the ACDC dataset with just
B Theoretical bounds for loss
First, we assume that the expectation over the data distribution of the colume-based loss and slice- based loss are equivalent. Formally, this is described as follows:
Ex Exxyypzz (evatume y) = Exe y' [Lstice y')]
where Lvolume : X x y + R and Lalice : X' x y + R are the volume based and slice- based loss respectively. For the rest of the proof, Latsce (-) is referred to by Z(-). Following the derivation provided in the original Coreset paper (64) we have:
1 Σ L(S' .Y)
E ('''''')' e' IC(S'''
C(s'
8E(m)
jEH
Generalization Error
Training
L(s Is
1
c(S'''))
(m(m)
JEB
Core-Ser Lass
L(S
D(''''')
xE(m]
JES
Cure- Loss
17

---

We get the last line of the inequality because we assume the Training Error is zero and, similar to the original Coreset paper, we assume the Generalization Error zero, (which is reasonable assumption because most CNNs have very small generalization enror) (64)
Thus, active learning objective can be re-defined as:
argmin L(9', s s S
y')
Σ c(y', JEM
(6)
S
RE(m)
We will now present the following theorem:
Theorem 1. Given n. i.i.d. samples drawn from pz. Yihelay and set of points loss function L(y', y") X -Lipschitz continuous for all y',y' and bounded by L, segmentation function (x(x') ply' clx') is X X continuous for all x' D! and E Y" is cover of Jihiel and 1( (i = Vj E [m]; with probability at least %
Σ ((
C S 8(X + X" 122
log(1/7)
1 E
2n
Similar (64 approach to 64
start our proof with bounding (nt v1). Following similar we get:
Eegooix ((G))))))))))))))))))))))))))))) & 5( AT
Again, following (64) we can use the Hoeffding's Bound conclude the rest of the proof.
C Batch sampler
The pseudocode for the batch sampler can be in Algorithm 1 This implementation assumes the use of all three group contrastive losses. If the adjacent slice group is removed, then remove line 6. the volume group removed, then remove line 7. the patient group removed, then remove line An illustration on how the batch sampler works can be seen in Figure 5
Batch
Patient 1
Group
Group
Valurnie
Patient 2
Batch
Dataset
Phase 1: Group Creation
Phase 2: Batch Creation
Figure 5: Overview of the batch sampler for Group-based Contrastive Learning
18

---

Algorithm Batch sampler for one epoch
Input: List of patient groups Pi, P2. PL. Each patient group i contains a list of volume
groups Vi, V2 ...... Vi- Each volume group for patient contains a list of slices s 52 T
2 y
Batch size is M.
1: PG <- list()
2: for all Pi. P2.... PL do
3: groups x-
4: for all Vi. V V do
5: for all s 82
ST do
6:
+- Random Cholloomhhiceeeepe &
7:
v +- RandomCChoice (U)44 - RmndonChaiclUvveiievv////
8:
9:
group + | s, v,
10:
Add group to groups
11: end for
12: end for
13: Add groups to PG
14:
end for
15:
batches f- (
while |PG| 2 M do
17: batch e- list()
18:
S
S+()
19:
while |batch| < M do
groups + RandomChoiiees RandomChoiie(geerraass group + Random ChaoooCCChhiiiagaaoaaaaa
21:
groups < groups' Srappss grouu
23:
if groups = 0 then
PG + PG' groups
25:
else
Add groups to 27: end if
26:
Extend batch with group end while
29:
30:
Add batch to batches
31:
end while
32:
return batches
D Additional results
We have included all collected results on all the datasets. We also report twice the bootstrap standard for the means the different experiment runs. Our process follows. We generate 1000 bootstraps. For each bootstrap. we generate resampling with replacement the test set volumes and generate the mean of the test metrics (average 3D Dice score over the test set) over the different experiment runs at the desired evaluation point. We then calculate the sample standard deviation of the bootstrapped means and multiply by two.
Because the CHAOS and DAVIS have fewer volumes (and the bootstrapping errors are artifically inflated), rather generating resampling with replacement the test set volumes, generate resampling over the slices.
19

---

Table 5: DICE scores for ACDC
Weakly-supervised
2%
3%
4%
5%
BALD
4..8+1.9
1.981119
6..411..8
1.6
Variance Ratio
43.3+2.1
45.0+:2.2
54.2++2.44
61.4:2.11
Random
44.9+2.3
45.7+22.
59.24221
66.9#1.9
VAAL
41.8+2.3
47.8+2.2
66.1+2.1
1..1
Coreset
45.2+22.1
49.8+22.0
68.7+:2.0
7..1
TypiClust
44.8+22.2
45.6++2.00
67.6+
1.6
Stochastic Batches
36.9+2.2
39.5+-2.0
53.7++1.8
1.5
CoreGCN
40.
49.3+2.00
1..1
1.7
Ours
55.6
61.4
73.7
17.5
10%
15%
20%
40%
BALD
1..
81.6+112
83.3
86.4:10.9
Variance Ratio
1.
13.
84,4
85.9+0.9
Random
1..
83.0+1..
84.1+1.1
86.7+0.99
VAAL
1..6#1.4
82.4+1.2
84.4+1.1
86.4+0.9
Coreset
10.
83.6+1.2
85.3+1.11
8.
TypiClust
1..
82.0+1.3
83.6++1.3
8..
Stochastic Batches
6.
7.3
81,9+1.2
1.1
CoreGCN
1..4+1..
8..
83.9+1.1
86.3+0.9
Ours
10.1
83.8
84.3
1..4
Fully-supervised
2%
3%
4%
5%
BALD
53.8113.7
66.012.7
67.9+12.5
71.7+2.2
Variance Ratio
52.6+E3.3
62.2433.
66.6
69.1133.3
Random
66.3133.
76.91330
79.3
80.1
VAAL
63.2+22.6
75.2+:2.3
79.5+2.11
81.3+1.8
Coreset
58.9+3.77
69.3+3.3
75.7=3.66
81.9+2.7
TypiClust
66.6+3.4
75.8+3.1
79.542.4
81.7+2.1
Stochastic Batches
60.3+3.0
69.8+228
74.1
75.4+2.5
CoreGCN
67.7+22.6
74.9+-2.4
79.0+2.2
80.7+2.1
Ours
70.9+22.7
77.4+2.4
81.6+2222
82.5+
10%
15%
20%
40%
BALD
81.6+1.88
1.2
85
89.
Variance Ratio
76,413.2
80.743.0
83.6+22.4
1.
Random
86.8+1.6
88.1+1.3
8..
90.2
VAAL
15
88.0+1..3
1.
1.
Coreset
85.8+2.2
87.7+1.3
88.8+1.11
90.2+1.0
TypiClust
8..
87.1+1.2
88.0+1.00
89.5+0.9
Stochastic Batches
81.6+2.4
84.6+1.4
86.5+1.3
89.3+1.1
CoreGCN
85,9+1.3
87.9+1.2
88.5+1.2
89.6+1.0
Ours
86.8+188
87.5+1.44
88.5+1.3
90.2+1.0
20

---

Table 6: DICE scores for MSCMR
Weakly-supervised
2%
3%
4%
5%
BALD
37.0=2.8
48.9
57.4
60.0113.0
Variance Ratio
41.3=+5.0
46.8+5.1
52.8++3.5
55.3:13.8
Random
39.7#3.7
55.0=4.0
61.0+t.11
61.5+3.1
Coreset
28.7=5.00
53.6+44.5
56.9:4411
58.7+3.6
Stochastic Batches
38.51:..6
56.2.14.2
59.8
60.5413.6
CoreGCN
27.0
48.745.3
57.1
37.2413.6
Ours
44.3+-.2
53.3+449
63.4
63.5
10%
15%
20%
40%
BALD
77.0
79,5+322
80.1
85.8
Variance Ratio
65.8
72.9=3.1
78.412.9
84.3:2.6
Random
76.0=2.9
82.443.1
83,2+3.00
85.742.8
Coreset
74.4=2.8
81.4+2.1
83.1
86.7+119
Stochastic Batches
76.1#3.2
80.6+2.8
83.1+2.2
86.2+
CoreGCN
71.0=3.6
80.9#3.7
83.0
85.5+3.8
Ours
77.2+2.9
82.4+2.3
83.6#2.11
86.3+2.3
Table 7: DICE scores for CHAOS
Fully-supervised
2%
3%
4%
5%
BALD
79.6
8.8
8..4
81.1+119
Variance Ratio
74.9+2.5
72.9+1..7
76.6412.6
76.24:2.6
Random
80.7#1.9
81.7=1.7
84.2+11.
85.1+1.3
Coreset
80.0=1.8
80.4+2.0
1..2
88.1+1.0
Stochastic Batches
77.242.3
82.9+1.9
83.8+1.8
84.7+2.0
CoreGCN
67.8+2.11
77.7+2.11
7.2
74.4+1.5
Ours
8..5+1.8
82.5+1.5
85.9+
90.3++1.3
10%
15%
20%
40%
BALD
92.6+
94.5++0.5
94.9++0.5
95.8++0.4
Variance Ratio
82.6+2.44
85.1
87.2+1.
92.8+112
Random
92.710.8
94.3110.6
94,8+10.6
96.210.3
Coreset
92.2-10.9
94.9110.5
95.8110.4
96.5-1.22
Stochastic Batches
92.1
93.0+11.0
94.110..7
96.110.3
CoreGCN
85.9=1.0
93.3+00.6
94.110.55
94.3+0.3
Ours
92.
94.4+00.7
95.6+00.
96.310.3
Table 8: DICE scores for DAVIS
Fully-supervised
10%
20%
30%
40%
BALD
43.6:10.6
42.0-10.5
43.4-10.4
43.8:10.6
Variance Ratio
36.55
31.2+10.4
34.9-10.6
40.7+L0.3
Random
39.6
40.5+.0.7
47.410.7
48.5+10.5
Coreset
31.7+10.7
39.4+0.6
42.210.6
42.1
Stochastic Batches
40.3+0.6
41.540.8
45.110.6
47.6+0066
Ours
42.8+0.77
45.240.6
45.5+0.7
46.6+00.7
21

---

Table 9: DICE scores for ACDC (pretrained)
Fully-supervieed
1%
2%
3%
4%
5%
Random
6..9=188
7..8+1.4
11.
85.0:10.9
85.6+10.9
Stochastic Batches
55.2=1.7
1..
8.2
83.4#1.0
86.4=1.0
Coreset
64.0=1.5
78.3+1.4
81.7#1.3
83.7#1.1
84.0#1.2
Ours
66.1=1.9
79.4+1.44
82.0#113
84.3+1.0
85.9
Table 10: DICE scores for CHAOS (pretrained)
I %
2%
3%
4%
5%
Random
91.8 0.5
94.6110.4
95.9+10.3
96.2.10.2
96.4:10.2
Stochastic Batches
9.54
94.5+0..3
95.5+10.3
95.9.10.3
96.3-10.3
Coreset
9..
94.8+10.4
95.7++0.2
96.3++0.2
96.4:10.2
Ours
93.4=0.3
95.1
95.4+0.2
96.1+0.2
96.1
Table 11: DICE scores for DAVIS (pretrained)
Fully supervseee
%
2%
3%
4%
5%
Random
71.0=0.5
71.2:10.5
76.7+0.6
76.2+00.7
79.3+10.7
Stochastic Batches
69.1-0.6
73.5+0.7
76.9+0.77
76.8+0.7
78.9+0.6
Coreset
68.5+0.6
73.2+0066
76.9+0.66
76.9+0.7
77.4+00.
Ours
71.1=0.6
73.5++0.7
76.4+00.6
77.2+00.7
77.2+00.5
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes they do.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Experimental limitations are discussed in Section Computational require- ments discussed in Section 4.2
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
We provide full set of assumptions and complete proof in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All implementation and experiment details are discussed in Section 3.2.1 ??. and 4.2 and Appendix. We have also provided our code in the supplementary materials and the data is publicly available. We will also post our code to Github.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided code and links to data in the supplementary materials. The code will also be posted to Github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic/guudees/Cdeettbii.ss cosssiinnooyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppclcccuui.eeeeeecccuea witeeee//ooSSSeSes...ee.a www.ssssiicggsooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, this information is provided Section 3.2.1 and 4.2 Appendix, and can also be viewed in the code in the supplementary materials or on Github.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we report bootstrapping standard errors for our results and describe our approach in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
We discuss this in Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpts://///et http://wwwpppppoooooooo/ccctcaaa
Answer: [Yes]
Justification: Because we used existing datasets with human subjects, Section 4.1 we discussed whether the dataset owners followed data collection regulations and whether human consent was obtained.
Guidelines:
The answer NA
that the authors have reviewed the NeurIPS Code of Ethics. If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: While positive social impacts are discussed throughout the paper (especially in the introduction), we also briefly discussed that we did not consider issues of bias which can have unfair outcomes for underrepresented groups in Section 5
26

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our models do not have a high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
For the datasets we used, no license information was available and we were unable to reach out to the asset creators. However, have properly credited the original owners of all the datasets and followed all the owners' directions for research use of the datasets. The license and terms of use for the segmentation model repository both followed and provided within the ws14mis subdirectory in the code submitted in the supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
27

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The only assets we are providing is our code, which is provided in the supplementary materials and on Github. The license information as well as documentation how the code is provided within the code directory. Other relevant details can also referenced the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: We used existing datasets that contain human subject data. As mentioned before, in Section 4. we discussed whether the dataset owners followed data collection regulations and whether human consent obtained.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
28

---

Answer: [No]
Justification: We used existing datasets that contain human subject data. While we were not able obtain all of this information, mentioned before in Section [4. we discussed whether the dataset owners followed data collection regulations and whether human consent obtained.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "wqvt4tI4fA", "reviewer": "Reviewer_gm1Z", "summary": "The paper presents an active learning framework based on deep metric learning for 3D medical image segmentation. The proposed approach extends the CoreSet algorithm for active learning by computing distances on features learned via a contrastive loss. This contrastive loss exploits low-cost labels to group samples (2D slices) based on their spatial position and corresponding subject. Additionally, a weakly-supervised version of the method, based on scribbles, is presented. In this version, scribbles are used in an auxiliary distance term measuring the intensity difference for labelled pixels. Experiments are conducted on three medical imaging dataset and one video segmentation dataset to evaluate the method's performance. Results shows the method to outperform recent active learning approaches and baselines in the majority of test cases.", "questions": "* Can authors clarify the contribution of their work with respect to previous work on active learning based on a contrastive technique?\n\n * What is the meaning/definition of h_{S0} in Eq (3) ?\n\n* Do you assume that pixels x^i_1 and x^i_2 correspond to the same anatomical structure in the scribble-weighted distance (i.e., images are registered)? Why compute the distance on pixel intensities?\n\n* Why is the method worse than random selection on the CHAOS dataset (2% fully-supervised) ?\n\n* Can you compare your method against random/maxEntropy selection on a segmentation model pre-trained using the SAME contrastive loss as your model?", "limitations": "* The last section of the paper mentions relevant limitations of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper introduces a simple yet efficient way to incorporate contrastive learning in the CoreSet algorithm for active learning in segmentation.\n\n* The extension of active learning to a weakly-supervised setting is also interesting and could have useful applications.\n\n* Experiments are comprehensive. The proposed method is tested on four datasets and compared against several related approaches. Results show clear improvements in most cases.\n\n* The paper is well written, and the method can be understood easily.", "weaknesses": "* Despite its novelty, the board idea of the proposed method is similar to previous work using contrastive learning for core set selection, for instance:\n\nJu J, Jung H, Oh Y, Kim J. Extending contrastive learning to unsupervised coreset selection. IEEE Access. 2022 Jan 13;10:7704-15.\n\nJin Q, Yuan M, Qiao Q, Song Z. One-shot active learning for image segmentation via contrastive learning and diversity-based sampling. Knowledge-Based Systems. 2022 Apr 6;241:108278.\n\nWhile these previous approach do not use group labels as the proposed method, a similar strategy was employed for self-supervised segmentation of medical images:\n\nChaitanya K, Erdil E, Karani N, Konukoglu E. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in neural information processing systems. 2020;33:12546-58.\n\n* The comparison against other approaches might be unfair since the contrastive (pre) training, which is used to select the core set, also boosts the performance of the segmentation model (regardless of which labelled samples are used for training). This can be seen in the work of Chaitanya et al (see reference above), where a segmentation trained with the same contrastive loss achieves a performance near full supervision with very few samples.\n\n* The scribbled weighted distance presented in Section 3.2 lacks motivation. It is unclear to me how the pixel-wise difference in intensity gives useful information for non-registered images (potentially of different subjects).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their great feedback. We hope to address the main concerns for the reviewer here.\n\n***Method novelty***\n\nThank you for your feedback related to this. We have addressed several of your concerns related to method novelty in the general response when discussing how our approach, unlike others [1-2], is uniquely tailored to the 3D medical segmentation task. We note that the focus of the other contrastive learning work in 3D medical segmentation [3] is self-supervised pre-training. Thus, the generated contrastive features may not be optimal for the Coreset-based active learning objective, as shown in the ablation results (Table 3 of our paper) when comparing the generated contrastive features from a standard contrastive loss and our proposed loss. While they utilize local and global features, they do not utilize domain information to generate the groupings which we have demonstrated is beneficial for the active learning task. Additionally, self-supervised pre-training is outside the scope of this work and we do not dispute that it is likely there may be additional performance benefits with self-supervised pre-training. We also compare our method to TypiClust [4], another contrastive learning-based active learning method, and demonstrate that our medically-tailored approach produces better results (Table 1 of our paper). We will discuss these references and differences from our method in the final camera-ready version.\n\n***No self-supervised pre-training for fair comparison***\n\nThank you for your feedback related to this. In order to ensure a fair comparison between all the active learning methods, the segmentation model does not utilize the pre-trained contrastive encoder. There are some practical engineering benefits if the active learning and segmentation models are decoupled. We hypothesize that there may be more performance improvements if the pre-trained contrastive encoder is utilized by the segmentation model. \n\n***Scribble-weighted distance***\n\nThank you for your feedback related to this. The ACDC challenge data is roughly registered and others have utilized the correspondence between the local image regions for self-supervised pre-training on the ACDC dataset [3]. We compared several different approaches of computing the scribble-weighted features (including segmentation model features as well as segmentation model predictions) and found that the best performance was obtained with the pre-processed pixel intensities.\n\n***Additional questions***\n\n1. We hopefully clarified the contribution of our work related to similar work. Please let us know if there are any additional questions.\n2. h_{S0} in Eq (3) refers to the scribble-weighted distance. The {S0} part is technically unnecessary but we wanted to emphasize that the scribble-weighted distance is calculated based on the annotations that are currently available.\n3. As mentioned prior, we do assume that the images are roughly registered and we compare the pre-processed pixel intensities because it obtained the best performance during the validation testing. This may be because of the anatomical region correspondence between images.\n4. Our method does perform worse than the random selection on the CHAOS dataset for the 2% fully-supervised data point. We want to emphasize that random selection is still considered an extremely competitive baseline in active learning [5]. However, in our paper, we demonstrate that our method achieves the best performance on 19 out of 24 comparison points in comparison to the diversity-based methods (Coreset, VAAL, TypiClust, Random) and 22 out of 24 comparison points in comparison to the entropy-based methods  (BALD, Variance Ratio, Stochastic Batches) from Table 1 and 2 in our paper.\n5. As mentioned prior, we do not perform self-supervised pre-training for fair comparison between the methods.\n\nThank you again for reading our paper and your help and feedback!\n\n[1] Ju, J., Jung, H., Oh, Y., & Kim, J. (2022). Extending contrastive learning to unsupervised coreset selection. IEEE Access, 10, 7704-7715.\n\n[2] Jin, Q., Yuan, M., Qiao, Q., & Song, Z. (2022). One-shot active learning for image segmentation via contrastive learning and diversity-based sampling. Knowledge-Based Systems, 241, 108278.\n\n[3] Chaitanya, K., Erdil, E., Karani, N., & Konukoglu, E. (2020). Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in neural information processing systems, 33, 12546-12558.\n\n[4] Hacohen, G., Dekel, A., & Weinshall, D. (2022). Active learning on a budget: Opposite strategies suit high and low budgets. arXiv preprint arXiv:2202.02794.\n\n[5] Munjal, P., Hayat, N., Hayat, M., Sourati, J., & Khan, S. (2022). Towards robust and reproducible active learning using neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 223-232).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mafha5pVyp", "reviewer": "Reviewer_ff5T", "summary": "The authors propose a new metric learning approach to improve the Coreset method (an optimization framework to seek the most diverse samples for model learning in active learning) with applications for slice-based active learning in 3D medical segmentation. The main idea is to utilize data groups extracted from 2D slides in 3D volumes and build group-based contrastive learning to enhance feature representations. The authors conduct experiments on four datasets and compare them with several baselines in active learning. The experiment shows good overall performance.", "questions": "My questions include:\n\n(a) can the author explain the 'group' definition in Section 3.1? It is confusing whether this 'group' refers to a set of similar 2D slides inside a 3D volume of one patient or it refers to a set of 2D slides of different patients. Reviews did not understand what the author wrote in lines 123-128.\n\n(b) how the algorithm 1 be extended if each patient has different data modalities like CT & MRI?", "limitations": "As mentioned in the weakness section, this paper needs to make a major improvement by either adding more experiments (with another architecture) or further improving the novelty in the way it formulates deep metric learning with contrastive ideas. The writing also need to be improved, especially when explaining the 'group' concept to avoid confusing readers.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "One of the main strengths of this paper is that the authors compared diverse baselines and benchmarks with some of the latest methods, for e.g., [19,37]. Also, their method achieves good performance with significant improvements on four datasets. The author also provides ablation studies to analyze the contributions of key components in Table 3.", "weaknesses": "There are two main critical concerns of Reviewers:\n\n(a) The way the author improves the Coreset with deep metric learning using group-based contrastive learning is limited in its novelty. In particular, group-based contrastive learning has already been introduced in prior work [65], and several advanced versions have been introduced. Although the authors modify with two sums in Equation (4), this is a minor point.  In short, given the claim that deep metric learning is the core contribution, authors should further tailor contrastive learning for 3D medical tasks, for e.g., making it robust under domain shift or considering other factors like hierarchical relations [1,2]. Also, fusing different metric learning using contrastive or generative learning is also a promising direction. \n\n(b) Although experiments are promising, they are tested only with a single architecture, which is not enough to conclude whether the method is generalized to different architectures. As can be seen in Table 2 (DICE scores for DAVIS), model accuracy is still under an acceptable threshold; this suggests that further evaluation of current foundation models [3] (using SAM) or advanced pre-trained medical models [4] (has versions for both ResNet and ViT) is necessary to boost results further. In summary, it's important to provide further experiments with another architecture (preferred large-pre-trained models) to validate the robustness of the proposed active learning. Besides 3D volumes, considering one or two test cases with video is also interesting, where the group of similar 2D slices becomes a group of similar frames. \n\n[1] Taleb, Aiham, et al. \"3d self-supervised methods for medical imaging\", NeurIPS 2020\n\n[2] Zheng, Hao, et al. \"Hierarchical self-supervised learning for medical image segmentation based on multi-domain data aggregation\" MICCAI 2021\n\n[3] Ma, Jun, et al. \"Segment anything in medical images.\" Nature Communications 2024\n\n[4] MH Nguyen, Duy, et al. \"Lvm-med: Learning large-scale self-supervised vision models for medical imaging via second-order graph matching\", NeurIPS 2023\n\n[65] Sohn, Kihyuk. \"Improved deep metric learning with multi-class n-pair loss objective.\" NeurIPS 2016", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their great feedback. We hope to address the main concerns for the reviewer here.\n\n***Method novelty***\n\nThank you for your feedback related to this. We addressed your concerns related to the method novelty in the general response and here further elaborate our novelty compared to the three suggested references. While group based contrastive learning has been proposed in the literature, as acknowledged in our paper [1], our overall framework with Coreset and contrastive learning and the application to the medical domain is very novel. \n\nTo our knowledge, no other method has a comparable approach which is tailored to the medical domain. It is also easy to extend our approach to different groupings. In the paper, we have also provided an intuition on what groupings will generate useful features (i.e. groupings with larger intra-group differences are more likely to be useful). We also have provided our batch sampling technique, which is a nontrivial addition to our methodology because standard random data loading would yield minimal impact from our loss due to the low probability of randomly selecting two slices from the same group.\n\nSpecifically, our group-based contrastive loss is very different from the multi-class n-pair loss [1]. We use explicit medical groupings in the data to generate positive and negative samples for a particular batch. While the other approach combines samples from different classes, we define hierarchical groups in an unsupervised setting and adapt the loss to address batch samples in the same hierarchical group. Generally in the loss slices that are not part of the same group would be negative samples. However, if there is a hierarchical relationship between slices, slices that are not part of the same subgroup but are part of the same more general group are ignored when computing the batch loss. This is seen in the denominator of Equation 4, which excludes patient slices for a particular data point that are not part of the same group. As mentioned in the paper, excluding non-group patient slices ensures that the model does not promote dissimilarity between non-group slices from the same patient. This ensures that we can sum multiple group losses together without counteracting their effects.\n\nWhile the other referenced hierarchical method also utilizes groupings in medical data [3], their groupings are assumed to be quite large and it’s unclear how they would extend their group classification approach to when there are a large number of patients, volumes, and adjacent slice groups. Our contrastive-based approach and our batch sampling technique addresses these problems and can also be extended to incorporate larger groups as well.\n\nIn the other referenced method [4], while their approach is interesting they do not explicitly utilize any medical groupings in their method and it is not clear whether their method could be easily extended to incorporate that information as well.\n\nWe will discuss these references and differences from our method in the final camera-ready version.\n\n***Empirical evaluation using large pre-trained segmentation model***\n\nThank you for your feedback related to this. As discussed in the general response, we have additionally evaluated our methodology with your suggested methodology using a ResNet-50 backbone [2]. For the medical tasks, we utilize a backbone pre-trained on medical images [2] and for the video segmentation task we utilize a backbone pre-trained on ImageNetV2. The results can be seen in Table 8 and improved the overall segmentation model performance.\n\n***Additional questions***\n\n1. For our proposed method and the paper implementation, the definition of “group” refers to a set of 2D slices associated with one patient. In some cases, this group can also be associated with one volume: for example, for the volume or slice-adjacency group. However, this may not be the case for the patient group, in which it is possible for slices from different volumes to be part of the same group if they are associated with the same patient. However, the definition of group can be extended to include 2D slices associated with multiple patients if a more general hierarchical group is used. One example is a pathology group which may contain slices from different patients.\n2. Unfortunately, we did not have time to thoroughly consider utilizing multiple modalities like CT and MRI. However, it is an interesting question. Assuming we used the same segmentation model for both modalities, preliminarily, it seems like we would add an intermediate hierarchical level for modality in between volume (the more specific group) and patient (the more general group). We will look into this in future work. \n\nThank you again for reading our paper and your help and feedback!\n\n[1] Sohn, K. (2016). Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29.\n\n[2] MH Nguyen, D., Nguyen, H., Diep, N., Pham, T. N., Cao, T., Nguyen, B., ... & Niepert, M. (2024). Lvm-med: Learning large-scale self-supervised vision models for medical imaging via second-order graph matching. Advances in Neural Information Processing Systems, 36.\n\n[3] Zheng, H., Han, J., Wang, H., Yang, L., Zhao, Z., Wang, C., & Chen, D. Z. (2021). Hierarchical self-supervised learning for medical image segmentation based on multi-domain data aggregation. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24 (pp. 622-632). Springer International Publishing.\n\n[4] Taleb, A., Loetzsch, W., Danz, N., Severin, J., Gaertner, T., Bergner, B., & Lippert, C. (2020). 3d self-supervised methods for medical imaging. Advances in neural information processing systems, 33, 18158-18172.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ogi8XPqIB2", "reviewer": "Reviewer_aT8e", "summary": "The paper introduces a novel active learning approach based on group-based contrastive learning, aiming to optimize data selection and model efficiency in medical image segmentation tasks. By combining NT_Xent loss with various group contrastive losses, it addresses the Coreset problem in active learning, particularly in handling the diversity and complexity of medical image data. The paper provides detailed descriptions of the method's construction and training process, validated through experiments on multiple classical and novel datasets to demonstrate the model's superiority and robustness under different annotation budgets and data complexities. The research findings hold significant academic and practical significance for enhancing workflows in medical image analysis and have profound implications for advancing machine learning applications in medicine.", "questions": "1. The main point of the authors' discussion is the time-consuming and labor-intensive nature of medical annotation. However, they added a dataset in a natural scenario, which seems somewhat off-topic. Is there a reason for this?\n\n2. Why are fully supervised and semi-supervised methods compared using different annotation increments? Fully supervised data is entirely annotated. How does this relate to the primary focus of the paper on semi-supervised scenarios?\n\n3. The paper introduces different loss functions, but what are the weight values for these different loss functions under different experimental settings? The paper does not clarify this point.\n\n4. The ACDC dataset includes both weakly supervised and fully supervised experiments. Why does the MS-CMR dataset only have weakly supervised experiments, while the CHAOS and DAVIS datasets have fully supervised experiments?", "limitations": "1. Annotation costs and efficiency: Although the paper discusses the potential of active learning to reduce annotation costs, it would be beneficial to explicitly discuss the trade-offs between annotation quality and the computational expense of implementing the proposed methodology. This discussion could provide insights into practical considerations for real-world deployment.\n\n2. Section 4.5 ablation study focuses solely on the combination of loss functions. Consider other ablation experiments, such as the scribble-weighted distance.\n\n3. Due to limited computational resources, the researchers selected a single architecture for each type of annotation and limited the number of experimental runs and model training. This may result in an incomplete evaluation of other architectures and models' performance.\n\n4. Video segmentation models typically require extensive pre-training to achieve good metrics. However, due to resource constraints, the researchers did not conduct such experiments, potentially leading to suboptimal performance in video segmentation tasks.\n\n5. The paper does not adequately address the issue of training set bias, which may result in unfair outcomes for minority groups. Future work should consider this issue more comprehensively and focus on the method's applicability across different domains.\n\n6. The proposed method mainly targets 3D segmentation tasks in the medical field. Further validation and adjustments may be necessary for applications in other fields.\n\n7. Was the selection of annotated data random? Did it consider data distribution bias? This could result in unfair outcomes for minority groups.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Firstly, by introducing a group-based contrastive learning approach, the paper significantly enhances model performance in medical segmentation tasks. Secondly, it innovatively combines NT_Xent loss with various group contrastive losses, effectively addressing the Coreset problem in active learning and tackling challenges posed by data diversity and complexity in medical image segmentation. The methodology and experimental design are of high quality, providing clear descriptions of contrastive learning loss applications and their integration within the active learning framework. Experimental results demonstrate the model's strong performance under varying annotation budgets and data complexities, highlighting its relevance for medical image analysis and its potential value to the NeurIPS community.", "weaknesses": "1. The description is confusing. For instance, the first chapter mentions two main contributions, but the abstract and methods sections also discuss the loss function, which is not explained in the contributions of the first chapter.\n\n2. The paper's description mainly focuses on solving the task of medical image annotation, yet it also includes experiments on natural data. I believe the use of video data does not align well with the primarily medical theme described in the paper.\n\n3. Even if the method is shown to be extendable to natural data, using only one natural dataset is insufficient. To enhance its applicability and impact across broader domains within the NeurIPS community, I suggest using more varied types of data (e.g., natural images, video sequences).\n\n4. Writing details. There are inconsistencies in figure captions, with some having periods and others not. The numbers in Table 1 exceed the table lines. Some tables comparing with other methods lack proper citations. The colors in the figures are too varied; please use black, white, and gray tones to highlight professionalism.\n\n5. The explanation of the annotated data volume for weak supervision and full supervision is unclear.\n\n6. Although the text mentions experiments conducted under low annotation budgets (2%-5%) and fully annotated scenarios, comparing the performance of different methods, it does not clearly explain why a 2%-5% annotation data volume is used in the fully annotated scenarios.\n\n7. In practical experimental design, the reason for using a 2%-5% annotation data volume under full supervision is usually to compare the performance of weak supervision and full supervision methods on the same data volume basis. However, weak supervision and full supervision are different concepts, and using a partial annotation data volume for both does not align with the concept of full supervision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their great feedback. We hope to address the main concerns for the reviewer here.\n\n***Model performance and annotation time trade-offs for weakly and fully-supervised 2D slices and 3D volumes***\n\nThank you for your feedback related to this. As stated in the general response, in Figure 5 of the attached PDF we have provided a graph that describes the relationship between model performance and annotation time for our method utilizing weakly and fully-supervised 2D slices and random sampling of fully-supervised 3D volumes on the ACDC dataset. This will help ensure a more fair comparison between the different active learning methods. \n\n***Performance on Video Segmentation Dataset***\n\nThank you for your feedback related to this. As stated in the general response, we additionally evaluated a segmentation model with a ResNet-50 backbone [1] pre-trained on ImageNetV2 on the video segmentation task and generated results which can be seen in Table 8 in the attached PDF. We can see that there is significant improvement on the scores for the video segmentation dataset, which is much more comparable to current methods. \n\nWhile the focus of our work is 3D medical segmentation, we wanted to demonstrate that our approach can be extended to other types of datasets. We picked video segmentation because there are several frames in a video, which is similar to having several slices in a volume. In future work, we will evaluate our model on additional video segmentation datasets.\n\n***Writing Quality***\n\nThank you for your feedback related to this. We will make sure to improve on this when revising.\n\n***Additional questions***\n\n1. As mentioned before, we wanted to demonstrate that our approach can be extended to other types of datasets. We picked video segmentation because there are several frames in a video, which is similar to having several slices in a volume. \n2. Fully supervised and weakly-supervised methods are compared using the same proportion of data annotated. We have also extrapolated the performance vs. annotation time of the different methods in Figure 5 in the attached PDF. Please let us know if we misunderstood your question.\n3. We have provided information about the loss weights in the general response. Please let us know if you would like any additional information.\n4. Unfortunately, the fully supervised labels are unavailable for the MS-CMR training set. We have reached out to the dataset owners and have not received a reply. There are no manually generated scribbles for the CHAOS and DAVIS datasets. We have explored several automated methods to generate scribbles from the full labels, including skeletonization and random walk methods as specified in prior work [2]. However, we were unable to generate reasonable segmentation results with the automated methods.\n\n***Discussion on limitations***\n\n1. In Figure 5 in the attached PDF, we have provided performance and annotation time trade-offs for weakly and fully-supervised slice-based active learning and fully-supervised volume-based active learning\n2. The last line of the ablation study includes the impact of scribble-weighted distance. The title “Scribble” may have been misleading and will be fixed in the camera-ready version.\n3. We have provided video segmentation results using a stronger pretrained model in Table 8 of the attached PDF and the results are much more comparable to other video segmentation methods.\n4. Unfortunately, we did not have the opportunity to consider training set bias which may lead to unfair outcomes to minority groups. The demographic data was unavailable for us to consider this. In future work, we will incorporate datasets which have this information to better address this issue.\n5. We agree with this. In future work, we will consider evaluation on more varied datasets.\n6. We agree that this could be an issue. Related to #4, in future work we will utilize datasets with demographic data to ensure that we are able to address this issue.\n\nThank you again for reading our paper and your help and feedback!\n\n[1] MH Nguyen, D., Nguyen, H., Diep, N., Pham, T. N., Cao, T., Nguyen, B., ... & Niepert, M. (2024). Lvm-med: Learning large-scale self-supervised vision models for medical imaging via second-order graph matching. Advances in Neural Information Processing Systems, 36.\n\n[2] Valvano, G., Leo, A., & Tsaftaris, S. A. (2021). Learning to segment from scribbles using multi-scale adversarial attention gates. IEEE Transactions on Medical Imaging, 40(8), 1990-2001.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hF0Fjqs5Vm", "reviewer": "Reviewer_4vB1", "summary": "This paper is concerned with the task of 3D segmentation in medical images. Motivated by the high cost of annotations by radiologists, the authors propose a novel active learning method based on the coresets. Here, instead of using the euclidean metric for defining the similarity between different samples, the authors propose to learn a metric via a contrastive loss. The authors propose a group contrastive loss based on the NT-Xent loss which leverages intrinsic similarities in the data (different patients, different volumes of the same patient, adjacent slices) and evaluate their method both on weakly-supervised (with scribble annotations) and fully-supervised tasks and using three datasets from the medial domain and one video segmentation dataset.", "questions": "- In  L. 582 the authors write that they did not include the slice group loss in the optimal formulation of their loss but in L. 217-220 it seems they included this term in their experiments for all datasets (also if the slice group loss was not included in training of CHAOS, MS-CMR, DAVIS) this would mean that the authors only used the volume group contrastive loss for all these experiments? Can the authors please clarify this in the manuscript?\n- How were the individual group losses weighted?\n- In L. 224 the authors mention that segmentation requires more training data for natural images. Could you please provide more details on this or a reference?\n- Can you better explain the different groups you have, in particular the patient groups? I suspect that these are different pathology subgroups + healthy group (e.g. in the ACDC dataset we would have 5 patient groups)\n- On which dataset is the ablation study in Tab. 3 performed?\n- Were reference methods trained for the same number of update steps and on the same amount of training data as the proposed method?\n- For the random baseline, did the authors choose samples randomly or weigh samples on a per-volume/per-patient level? Otherwise patients for which more volumes/slices are acquired would overrepresent in the coreset, right?", "limitations": "The authors discuss limitations of their method in Sec. 5. Maybe the authors can also discuss the possibility of including other groups (e.g. pathologies, patient age/sex/..., ...), in the future? I would also appreciate a more honest discussion of the limitations and implications of slice-based segmentation (see **3D vs. 2D segmentation models** above)", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is well written and methods to improve medical image segmentation with limited annotations are of high interest to the community. The related work is comprehensive and the authors motivate their work well. Overall methods are sound and the evaluation is enough to support the claims.", "weaknesses": "**3D vs. 2D segmentation models**\n- To the best of my knowledge 3D models (i.e., *not* slice-based models) generally outperform 2D models in most medical image segmentation tasks [a,b]. While I understand the authors' motivation for using 2D (i.e., slice-based) methods, I find their statements (e.g., L. 36-38, 82-83)\n    > There were also inefficiencies in previous studies, such as choosing whole volumes instead of individual slices in 3D segmentation, which increased costs [37]. (L. 82-83)\n    \n    surrounding this choice somewhat misleading. I would appreciate a more honest discussion where the authors phrase their focus on slice-based methods less as an advantage and more as a limitation of the present work. They could also briefly discuss potential extensions of their work to 3D segmentation. I suppose that generally, this is possible when the adjacent slice group contrastive loss is removed (potentially one could still consider adjacent 3D patches or something similar). The ablation studies also suggest that the adjacent slice group loss doesn't really help much and the authors did not include it in their final loss formulation (L. 582).\n\n- The restriction to slice-based methods and in particular the slice group contrastive loss imply that the proposed methods would assume the annotations acquired during the active learning to also be slice-based. However, for medical image segmentation it might be much easier to acquire annotations for $N$ slices within the same volume than it is to acquire annotations for $N$ slices across different volumes because readers would usually use some interactive segmentation tool and segmentations are similar for adjacent slices. \n\n**Ablation studies**\n\nIn Tab. 3 the authors present an ablation study where they study the effect of different group losses on the overall performance. I find this study very interesting, however:\n- It seems that ablation studies where only conducted on a single dataset. If so, on which dataset? \n- How comparative are the results, given that including more contrastive losses lead to larger epochs and more update steps by design of the batch sampler? Can the authors clarify?\n- How did the authors choose the weighting of each loss term here?\n\n**Typos and minor comments:**\n\n- \"NT_Xent loss\" -> \"NT-Xent loss\"\n- L. 246: \"Training speed for for the [...]\"\n- Fig. 2 could be improved by just adding a column ground truth segmentation and marking the segmentation by the model in a single color. The blue and red colors can be somewhat confusing. Also, which dataset is this from? I suspect the ACDC dataset? Then, shouldn't there be three classes (RV, LV, Myocardium)?\n\n---\n[a] Avesta A, Hossain S, Lin M, Aboian M, Krumholz HM, Aneja S. Comparing 3D, 2.5D, and 2D Approaches to Brain Image Auto-Segmentation. Bioengineering. 2023; 10(2):181. https://doi.org/10.3390/bioengineering10020181\n\n[b] Wasserthal, J., Breit, H. C., Meyer, M. T., Pradella, M., Hinck, D., Sauter, A. W., Heye, T., Boll, D. T., Cyriac, J., Yang, S., Bach, M., & Segeroth, M. (2023). TotalSegmentator: Robust Segmentation of 104 Anatomic Structures in CT Images. Radiology. Artificial intelligence, 5(5), e230024. https://doi.org/10.1148/ryai.230024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their great feedback. We hope to address the main concerns for the reviewer here.\n\n***3D vs. 2D segmentation models***\n\nThank you for your feedback related to this. It’s true that generally 3D models outperform 2D models. However, based on our empirical results, 3D models seem to require more training data. In Figure 5 in the attached PDF, we provide performance and annotation time trade-offs for weakly and fully-supervised slice-based active learning and fully-supervised volume-based active learning. However, in order to provide a fair comparison with the 2D U-Net network, we utilize a simple 3D U-Net work without any advanced features. However, a more advanced 3D segmentation model may provide better performance. In future work, we can look into this more thoroughly. \n\nAs you mentioned, we can also adapt our methodology to intelligently annotate 3D volumes vs. 2D slices. Thus, we would likely not utilize the volume and slice-adjacency grouping and, instead, consider patient grouping and other possible groupings like pathology, patient height, patient weight, or modality.\n\nYou also mentioned another important point that annotation cost is not uniform among all slices and slices for the same volume tend to be lower cost. We unfortunately did not have the opportunity to consider this in our present work. However, in future work we can consider varied costs in our active learning objective.\n\n***Ablation study***\n\nThank you for your feedback related to this. We performed the ablation study on the ACDC dataset. The training time for the contrastive learning encoder scaled based on the number of groupings. Thus, the training time for each subgroup in the ablation study was roughly similar. We have also included the loss weights for the ablation study in the general response. Please let us know if there are any additional questions.\n\n***Typos and comments***\n\nThank you for your feedback related to this. We will fix the typos in our camera-ready version. Figure 2 is from the ACDC dataset. Initially, we did not want to have too many colors in the diagram but will make sure all three classes are indicated in our revision.\n\n***Additional questions***\n\n1. We have discussed this in the general response in the “Loss weights” section. Please let us know if there are any additional questions.\n2. We have discussed this in the general response in the “Loss weights” section. Please let us know if there are any additional questions.\n3. Generally, natural images have much more variability than medical imaging which is why more training data tends to be needed [1]. That being said, utilizing pre-trained models can significantly reduce the need for training data, as can be seen with our results in Table 8 in the attached PDF.\n4. In our paper, we primarily focused on three groups, starting from most general to most specific: patient, volume, and adjacent slice group. The patient group is all the slices associated with volumes pertaining to a particular patient. The volume group is the slices associated with exactly one volume. The adjacent slice group are all the pairs of slices which are adjacent to one another in a particular volume. \n5. The ablation study is performed on the ACDC dataset.\n6. Yes, all training methods were trained for the same number of update steps and on the same amount of training data as the proposed method.\n7. So, yes, in random sampling we chose samples randomly rather than weigh samples on a per-volume/per-patient level. However, patients had roughly the same number of volumes/slices so this was not a significant issue for us.\n\n***Discussion on limitations***\n\n1. For our proposed method and the paper implementation, the definition of “group” refers to a set of 2D slices associated with one patient. However, the definition of group can be extended to include 2D slices associated with multiple patients if a more general hierarchical group is used. If this group is valuable, we could incorporate it into our group contrastive loss. In the denominator of the loss, rather than ignoring non-group slices from the same patient we would ignore non-group slices from the same more general group. We also would need to slightly adapt our batch sampler to address utilizing this group. We can provide more details in the camera-ready version of our paper.\n2. We discussed 3D vs. 2D segmentation models in the previous section. Please let us know if there are any additional issues or questions.\n\nThank you again for reading our paper and your help and feedback!\n\n[1] Xu, Y., Shen, Y., Fernandez-Granda, C., Heacock, L., & Geras, K. J. (2024). Understanding differences in applying DETR to natural and medical images. arXiv preprint arXiv:2405.17677.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you all for reading our paper and your great feedback. I will address some of your general questions in the following response. Please note that we have attached a PDF with an additional table and figure that we reference in our response.

***Method novelty***

We want to emphasize the unique application area of our proposed method, which is slice-based active learning for 3D medical segmentation. Our proposed method is specifically tailored to utilizing individual slices for 3D medical segmentation, which is more cost-effective than using full volumes. We propose a unique Group Contrastive Learning (GCL) framework which leverages patient, volume, and adjacent slice relationships which are groupings inherent to medical imaging. Furthermore, we provide ablation results which emphasize the benefit of these groupings for active learning in 3D medical segmentation and also provide intuition on what groupings will generate useful features (i.e. groupings with larger intra-group differences are more likely to be useful). Additionally, in Figure 5 in our attached PDF we provide a graph which demonstrates the cost-effectiveness of our approach in comparison to volume-based active learning.

The other Coreset-based methods which use contrastive features [1-2] are not tailored to efficiently learning 3D medical segmentation models. For example, the volume group in our method specifically reduces the number of redundant slices annotated from similar volumes.  The other approaches do not specifically focus on reducing slice annotations per volume. Additionally, in our ablation study, we demonstrate that the Coreset algorithm with contrastive features generated from our proposed method produces much better performance than the Coreset algorithm with standard contrastive features not tailored to the 3D medical segmentation task.

***Empirical evaluation using large pre-trained segmentation model***

We additionally evaluated our method and the top three comparison methods using a large segmentation model with a ResNet-50 backbone [3], pretrained on medical images [3] and ImageNetV2. We noted that there was a performance improvement for the fully-supervised task and strong performance even with 1% of the data. Results are in Table 8 in the attached PDF. We do not report weakly-supervised results because the weakly-supervised architectures cannot easily utilize pre-trained backbones.

***Model performance vs. annotation time for different methods***

As some reviewers suggested, in Figure 5 in the attached PDF we provided a graph which describes the relationship between model performance and annotation time for our method utilizing weakly and fully-supervised 2D slices and random sampling of fully-supervised 3D volumes on the ACDC dataset. We follow prior work which assumes that annotators can annotate scribbles 15x as fast as the full masks [4]. To ensure a fair comparison between the different methods, we do not incorporate any of the results from the pre-trained segmentation models. For the 3D results, similar to the 2D U-Net we train a 3D U-Net from scratch. Given comparable annotation time, our methods trained on both weakly and fully-supervised 2D slices far exceed the performance of random sampling of 3D volumes and achieve 3D volume maximum performance (with the given budget) with much less annotation time.

***Loss Weights***

In our experiments and ablation study, a 1.0 weight was applied whenever the NT_Xent loss was used. Additionally, a 1.0 weight was applied if only one group contrastive loss was used. In the ablation study on the ACDC dataset, for the experiments with multiple group contrastive losses, we tried different combinations of weights for the group contrastive losses and reported the best results. We will utilize the formulation in Appendix A for the loss weights and report them as a four-tuple (a,b,c,d) which corresponds to ($\lambda_1$,$\lambda_2$,$\lambda_3$,$\lambda_4$). A weight of 0 indicates the loss was not used. The combinations we tried are as follow (with the best bolded):

- patient and volume group loss without NT_Xent loss: (0, 0.125, 0.875, 0), **(0, 0.50, 0.50, 0)**
- volume and slice group loss without NT_Xent loss: (0, 0, 0.125, 0.875), **(0, 0, 0.50, 0.50)**
- patient and volume group loss with NT_Xent loss: (1,0. 10 , 0.35, 0), (1, 0.117, 0.233, 0), **(1, 0.05, 0.35, 0)**
- patient, volume, and slice group loss without NT_Xent loss: (0, 0.05, 0.25, 0.7), **(0, 0.33, 0.33, 0.33)**
- patient, volume, and slice group loss with NT_Xent loss: (1, 0.10, 0.20, 0.05), **(1, 0.05, 0.35, 0.025)**

We utilized the best loss/weight combination for our ACDC experiments. For the CHAOS, MSCMR, and DAVIS datasets, because there is only one volume per patient and thus no difference between the patient and volume loss, we tested two loss/weight combinations:
- volume loss with weight 0.35 with NT_Xent loss
- volume loss with weight 0.10 and slice loss with weight 0.30 with NT_Xent loss

Both weight combinations performed better than other comparison methods though volume loss with weight 0.35 with NT_Xent loss performed slightly better than the other combination.

[1] Ju, J., Jung, H., Oh, Y., & Kim, J. (2022). Extending contrastive learning to unsupervised coreset selection. IEEE Access, 10, 7704-7715.

[2] Jin, Q., Yuan, M., Qiao, Q., & Song, Z. (2022). One-shot active learning for image segmentation via contrastive learning and diversity-based sampling. Knowledge-Based Systems, 241, 108278.

[3] MH Nguyen, D., Nguyen, H., Diep, N., Pham, T. N., Cao, T., Nguyen, B., ... & Niepert, M. (2024). Lvm-med: Learning large-scale self-supervised vision models for medical imaging via second-order graph matching. Advances in Neural Information Processing Systems, 36.

[4] Valvano, G., Leo, A., & Tsaftaris, S. A. (2021). Learning to segment from scribbles using multi-scale adversarial attention gates. IEEE Transactions on Medical Imaging, 40(8), 1990-2001.

Meta-review: While this work initially received mixed scores, authors addressed positively most of the reviewers comments, which led to an overall increase of the recommendation scores for this submission. I strongly recommend the authors to consider the reviewer comments in the camera-ready version, and include requested changes, such as i) deeper details about the contrastive supervised learning tailored for 3D volumes (Fig 1) to highlight the contributions this work, ii) experiments with pre-trained models, as well as other ablation studies used to address the raised concerns.
Decision: Accept (poster)