Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels
Authors: Jizong Peng
Ping Wang
Christian Desrosiers
Marco Pedersoli
Venue: NeurIPS 2021 Poster (2021)
Keywords: self-supervised learning
semi-supervised learning
medical image segmentation

Abstract: The contrastive pre-training of a recognition model on a large dataset of unlabeled data often boosts the model’s performance on downstream tasks like image classification. However, in domains such as medical imaging, collecting unlabeled data can be challenging and expensive. In this work, we consider the task of medical image segmentation and adapt contrastive learning with meta-label annotations to scenarios where no additional unlabeled data is available. Meta-labels, such as the location of a 2D slice in a 3D MRI scan, often come for free during the acquisition process. We use these meta-labels to pre-train the image encoder, as well as in a semi-supervised learning step that leverages a reduced set of annotated data. A self-paced learning strategy exploiting the weak annotations is proposed to further
help the learning process and discriminate useful labels from noise. Results on five medical image segmentation datasets show that our approach: i) highly boosts the performance of a model trained on a few scans, ii) outperforms previous contrastive and semi-supervised approaches, and iii) reaches close to the performance of a model trained on the full data.

Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels
Jizong Peng"
Ping Wang
ETS Montreal
ETS Montreal
jizong peng 10etsmtl.nes net
ping wang 10ens etamtl.ca
Christian Desrosiers
Marco Pedersoli
ETS Montreal
ETS Montreal
christian. www.ddeessssierettmtt.....
marco pecersold wetooolleeetmtlll.mmmmmmmmmm ca
Abstract
The contrastive pre-training of recognition model large dataset of unlabeled data often boosts the model's performance on downstream tasks like image classifi- cation. However, in domains such medical imaging, collecting unlabeled data can be challenging and expensive. In this work, we consider the task of medical image segmentation and adapt contrastive learning with meta-label annotations to scenarios where additional unlabeled data is available. Meta-labels, such the location a 2D slice in a 3D MRI scan, often come for free during the acquisition process. We use these meta-labels to pre-train the image encoder, well as in a semi-supervised learning step that leverages a reduced set of annotated data. self-paced learning strategy exploiting the weak annotations is proposed to further help the learning process and discriminate useful labels from noise. Results on five medical image segmentation datasets show that our approach: i) highly boosts the performance of model trained on few scans, ii) outperforms previous contrastive and semi-supervised approaches, and iii) reaches close to the performance of a model trained on the full data.
Introduction
Since the emergence of deep learning [26]. there has been active debate on the importance of pre-training neural networks. Precursor works [17] showed that pre-training a convolutional neural network with an unsupervised task (e.g.. denoising autoencoders (49) could lead to better performance in the final supervised task. As the amount of labeled training data increased, thanks to large datasets like ImageNet [33, it was however found that pre-training could actually hinder performance [38). This makes
in light of recent studies showing, for instance, that symmetries large networks induce many equivalent local minima [16] 35 47 which pre-trained model can get stuck. Recently, contrastive learning has renewed the interest in unsupervised pre-training B7]. Several works [8] TO
IT 20 62) have found that pre-training a model with a contrastive loss can improve its performance on a subsequent supervised training task, often outperforming network with supervised pre-training pre-training, it offers little help for domains where data is scarce such as medical imaging. In medical imaging, not only are labels expensive since they come from highly- experts like radiologists, but images are also hard obtain due the need for costly equipment (e.g., MRI or CT scanner) and privacy regulations.
ImageNet. While this has reopened the debate the benefit of
Over the last years, breadth of semi-supervised learning approaches have been proposed for medical image segmentation, including methods based on attention 32 adversarial learning (60].
"Corresponding author
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

co-training [40, 63], data augmentation [6 61 and transformation consistency [5]. The common principle of these approaches to add an unsupervised regularization loss using unlabeled images, which is optimized jointly with a standard supervised loss on a limited set of labeled images. Despite reducing significantly the amount of labeled data required for training, current semi-suuerviin learning methods still suffer from important drawbacks which impede their use in various applications. Thus, a large number of unlabeled images is often necessary to properly learn the regularization prior. As mentioned before, this may be impossible in medical imaging scenarios where data is hard to obtain. Morcover. these methods also need a sufficient amount of labeled data, otherwise the learning may collapse (36.
temporal ensembling 12 55
In a recent work, Chaitanya et al. 7 showed that unsupervised pre-training can be useful to learn a segmentation task with very few samples, by leveraging the meta information of medical images (e.g., the position of 2D image in the 3D volume). While achieving impressive accuracy with as few as two volumes, this work has significant limitations. First, it relies on the strong assumption that the global or local representations of 2D images are similar if their locations within the volume or feature map are related. This assumption does not always hold in practice since volumes may not be well aligned, or due to the high variability of structures to segment. Second, it requires dividing the 2D images of 3D volume in an arbitrary number of hard partitions that are contrasted, while the structure to segment typically varies gradually within the volume. Third, they do not exploit the full range of available meta data, for instance the patient ID cycle phase of cardiac cine MRI, nor evaluate the benefit of combining several types of meta information in pre-training. Last, their approach leverages meta data only in pre-training, however this information could further boost performance used while learning the final segmentation task, a semi-supervised setting.
Our work addresses the limitations of current semi superisuere and self-supervised approaches for segmentation by proposing novel self- self-paced contrastive learning method, which takes into account the noisiness of weak labels from meta data and exploits this data jointly with labeled images in a semi cemi-supervised setting. The detailed contributions of this paper are as follows:
We propose, our knowledge, the first self-paced strategy for contrastive learning which dynami- cally adapts the importance of individual samples in the contrastive loss. This helps the model deal with noisy weak labels that arise, for instance, from misaligned images or splitting 3D volume in arbitrary partitions.
We demonstrate the usefulness of a contrastive loss on meta-data for improving the performance of a final task, not only in pre- pre-training but also as an additional loss in semi-supervisee training. contrastive learning framework
We show that combining multiple meta-labels in our can improve performance on the final task, compared to using them independently. Our results also demonstrate the benefit of combining contrastive learning with temporal ensembling to further boost performance.
We empirically validate our contributions on five well-known medical imaging datasets, and show the proposed approach to outperform the contrastive learning method of 7 as well as several state-of- the-art semi-supervised learning methods for segmentation [4] 43
50 58 60). In the results, our
approach obtains a performance close to fully supervised training with very few training scans.
2 Related work
We focus our presentation of previous works on two machine learning sub-fields that are most related to our current work: self-supervision, which includes contrastive learning, and self self-paced learning.
Self-superviiion and contrastive learning Self-supervision is a form of unsupervised learning where pretext task is used to pre-train model so to better perform a downstream task. Examples of
pretext tasks are learning to sort sequence 288
[44, predicting rotations [9 [8] solving a jigsaw
puzzle (33) and many others [4] [5] 25
46 59 Most of these methods can improve performance
on the downstream task when labeled data for training scarce. However, when large and general-
enough dataset of labeled images like ImageNet [13 is available, a simple supervised pre-training may sometimes provide better results 22 34]. Recently, unsupervised contrastive learning [8] 20 [0]
was shown to boost performance even when learning a downstream task on a large dataset, and to improve over a model pre-trained in supervised manner on large dataset.
This approach is based on the simple idea of enforcing similarity in the representation of two examples from the same class (pesitive pairs), and increase representation dissimilarity on pairs from different classes (negative pairs) (37 48). In image analysis tasks, positive pairs are typically defined two

---

transformed versions of the same image, for instance using a geometric or color transformation, while negative ones are any other pair of images
[8] 10
20
In a recent work, Khosla et al. [4 showed that, when combined with true semantic labels, con- trastive learning based "pre-train and fine-tune" pipeline performed surprisingly well, outperforming conventional training with cross-entropy in some cases. The work in [62] extended this idea to pixel-level tasks like semantic segmentation, clustering the representations of pixels in an image according their labels. Applying a similar strategy medical image segmentation, |Chaitanya ct al. leveraged meta-labels from 3D scans in a local and global contrastive learning framework to improve performance when training with limited data. However, positive and negative pairs in their contrastive loss are defneed using noisy "weak" labels, arising for example from misaligned images or an arbitrary partitioning of 3D volumes, which may lead to learning sub-optimal representations. The work in (56) mitigated this problem by imposing a maximum distance along the z axis between slices forming positive pairs. Moreover, existing approaches only exploit meta-labels in pre-training, instead of considering them jointly with labeled images in semi-supervised strategy. Our work extends these approaches with a self-paced learning method that adapts the importance of positive pairs dynamically during training, and focuses the learning on the most reliable ones. In contrast to 77. we also exploit contrastive learning as regularization loss in eumi-supereieed training and show that further improvements can be achieved when combining it with a temporal ensembling strategy like Mean Teacher [12 [55] A recent approach by Chen al. [10] also performs contrastive learning for image recognition in a semi supervised setting. In this approach, a large teacher model is trained with unsupervised contrastive loss and then fine-tuned with small fraction of labeled data. student model is trained afterwards using a knowledge distillation technique. Hence, unlike our method, there is no joint optimization of the supervised and contrastive objectives. In this work, we show that significant improvements can be achieved by jointly optimizing these two objectives.
Self-paced learning A sub-category of curriculum learning (CL.) 22, self-pacer learning (SPL.) is inspired by the learning process of humans that gradually incorporates easy to hard samples in training [27] The effectiveness of such strategy has been validated in various computer vision tasks [52 23 57]. Jiang et al. (23] proposed an SPI. method considering both the difficulty and diversity of training examples, which outperformed conventional SPL methods that ignore diversity. [Zhang cct al. [57] incorporated SPL in a DNN fine-tuning process for object detection, to cope with data ambiguity and guide the learning in complex scenarios. The usefulness of SPL. when training with a limited budget or when the training data is corrupted by noise was also studied in recent work [53]. So far, the application of SPL to image segmentation remains limited. Wang et al. [52] presented an SPL method for lung nodule segmentation, where the uncertainty of each sample prediction in the loss is controlled by the SPL regularizer. Similarly, a self-paced co- co-training method was proposed in [1 for the supissupee segmentation of medical images. Related to our work, Wang et al. 31] proposed a margin preserving contrastive learning framework for domain adaptation that uses a self-paced strategy in self- self-training. Compared to this approach, which uses SPL outside the contrastive loss to select confident pseudo-labels for self- training, our method incorporates it within the loss via importance weights that are learned jointly with network parameters.
3 Proposed method
In this section, we present our self-paced contrastive learning approach for segmentation that leverages intrinsic meta information extracted from medical volumetric images. Our method effectively pre- trains segmentation model with self-paced variant of contrastive learning that is more robust to noisy annotations. The same approach is also used to further boost the segmentation accuracy in a semi-supervised setting, where very limited pixel-wised annotations are used. In the following subsections, we detail the formulation of the proposed approach.
3.1 Contrastive learning with meta-labels
Given batch of N images from a dataset D. of unlabeled images, unsupervised contrastive learning approaches [ 21 (77 aim at finding a feature extractor f(-) that gives similar representations for two augmented instances of the same image and different ones for those of two separate images, regardless of their true classes. This can be achieved by creating an augmented set of samples indexed by I = 1, I,.... 2N, with two augmented samples for each original image in the batch. The

---

4
following loss is then optimized:
2N exp (z. 2 (1))
LunsupCon
Σ 2N
1
log
(I)
EaeA()) exp (z]za/? Za/T)
In this loss, z, =
(x; is the Lz-normalized representation of an image x, in the augmented batch
f(x;)
(i.e., the anchor) and i is the index of the other augmented sample from the same image (i.c., the positive). A(i) I\ (i) contains all indexes of the augmented set except i, and has size of 2NN -1. Finally, 7 is a small temperature factor that helps gradient descent optimization by smoothing the landscape of the loss.
This approach works well when large dataset of unlabeled images is available. However, the number of available images is small in our case. To alleviate this problem, our contrastive learning framework also leverages meta-labels arising from the structure of the data. Following 77, we consider the 2D slices of a given set of M volumetric scans as our training data, and extract various meta-labels for each 2D image (e.g., patient ID, position of the slice in the volume, etc). More generally, we suppose that each image x, has set of K meta-labels denoted as y E (l..... C+}. where Cj is the number of class labels for meta information & {l...., K}. The contrastive loss for the meta-labo & is then defined as
2N
exp (z|zz/T)
c.om
Σ
- log
2N
(k(
EoEA() exp (z) Zu/T
(EP
where ( =f are the indexes augmented samples with same label as X,, or coming from the same original image. By minimizing this loss, the feature extractor learns to group together representations with the same class and push away those from different ones.
In medical image segmentation, encoder-decoder based networks such as U-Net [5] and its variants are widely utilized thanks to their symmetric design and appealing performance on various dataset. Such network F()) decomposes in two parts, an encoder E(() that summarizes the global context of input 2D slice into a low-dimensional representation, and a decoder D(.) that takes input the representation and gradually recovers its spatial resolution using side information such as skip connections or pooling indexes. Previous work contrastive learning showed that pre-training both the encoder and decoder separately helped the downstream segmentation task 77. In preliminary experiments, we found that pre-training the decoder gave marginal improvements and thus focused our method on the encoder. Specifically, we consider the features of the encoder as a single vector E(x) E Rd and use a shallow non-linear projector 9(-) called head to obtain the final normalized embedding z, [[]])(
(xE()))
3.2 Self-paced learning to mitigate noisy meta-labels
The supervised contrastive loss of Equ. can actually hurt the learning of representations in pre- training if the positive pairs are obtained with "weak" or noisy labels. For instance, if using patient ID as meta-label, we will force the encoder to cluster the representations of all 2D slices in 3D volume, including those containing mainly background noise. Likewise, grouping together the slices in the same partition of two volumes hinders pre- pre-training if the volumes are not fully aligned and/or their partitions cover different regions of the structure segment.
To overcome this problem, we propose self self-paced strategy for contrastive learning which assigns an importance weight Way E [0, 1] to the specific loss of each positive pair (i, defined as by in Equ. (2). A self-paced regularization term (wiy), controlled by the learning pace parameter 7, is added to give a greater importance (i.e., larger Wij) to pairs that are more confident (i.e., smaller lij), and vice-versa. The learning pace 7 is increased over training so that high-confidence pairs are considered the beginning and then less-confident ones are gradually added as training progresses. We achieve this by defining the following self- self-paced contrastive loss optimized over both encoder parameters and importance weights:
2N
Σ jEP
wiy lij (y(ss)
(3)
2N
We omit the nonlinear projector head for the sake of simplification.

---

Following standard SPL approaches [23], we define the regularizer R2 such that the weights are monotone decreasing with respect the loss lyj (i.e., harder examples are given less importance) and monotone increasing with respect to the learning pace (i.e., a larger y increases the weights). In this work, consider two regularizer functions, based hard thresholding and linear imputation:
Rhard (wsy) = -7 Waj Re
Rlinear (Wiy) - 7 (2 wii).
by optimizing alternatively with respect to the encoder parameters 05 or importance weights Wij. while keeping the other fixed. With fixed wij. we update 0g via stochastic gradient descent where the gradient given by:
Optimization process
We minimize the loss in Equ. (3)
2N
Σ JEP
Wis Vegly.
(5)
SP-con
2N
(i)
As can be seen, the gradient of low-confidence pairs (i,j) will be scaled down by their weight Wij. thus these pairs will contribute less to the learning. Then, given a fixed 0g we compute the optimal weights wiy by solving the following problem:
= arg min Way ly + (wis)
(6)
wiy E|0.1]
The following proposition gives the optimal solution for the hard and linear SPL regularization strategies.
Proposition 1. Given the definitions of Rhard and Rlinear in Equ. (4 the closed-form solutions to Equ. O0) are given by
hard W
{b),
1,
by ST
lincar
L (15 0).
0,
else
max
Proof. For the hard regularizer Reqard the problem becomes
min E
Wiy lip - 7 Whi -li - y)
(8)
l, 7 2 O, since we are minimizing, the optimum is obviously wi O. Else, if by - < 0, the
minimum is achieved for wij- = 1. Combining these two results gives the hard threshold of Equ. 7
A similar approach is used for the linear regularizer Ihhard In this case, the problem to solve is
min Wiy Lij + (H wis) 3 Bwy + (eig - 7) wij
M0y [0
la 2 y, since way 2 O, the minimum is reached for Wij = 0. Else, if lij < 7, we find the optimum by deriving the function w.r.t. Wij and setting the result to zero, giving
Wij =
4gj
(10)
Since both and Ey are non- negative, we have that way E [0, 11, hence it is a valid solution. Consider- ing both cases simultaneously, therefore get the linear rule of Equ.
These update rules in Equ. only considers confident pairs with by S and ignores the others. In contrast, the linear rule weighs each pair proportionally to and the inverse of lij. emphasizing more confident ones.
be explained intuitively. For given 7, the hard threshold rule
Selecting the learning pace parameter One of the main challenges in self-paced learning methods is selecting the learning pace parameter 7. I is too small, all pairs will ignored and there will be no learning. Conversely, if is too large, all pairs will be considered regardless of their confidence, which corresponds having no self-pad celf-paced learning. The following proposition provides insights on how to set this parameter during training.
Proposition 2. The loss lyg related to a given pair (i.j) in the SPL objective of Equ. bounded by log 2(N-2NN)))) -1) 2/T < lig I log 2N + 2/T, where N is the batch size.

---

Proof. We start by rewriting by equivalently as
bg = log EoEA(1) exp (2/7a/ exp (z) 2/(2
exp 2n/22/T
log (+
(1I)
exp 2/Zz/T)
Since the representation vectors z; are Ln-normalized, their dot product is cosine similarity falling in the range [-,, To minimize lij, then need to minimize the dot product in the numerator inside the sum and maximize the one in the denominator. Using (A(i) \ = 2N -2, we get
log (1 (2
log (1 2(N 3/7) 2 log 2(N-1) -/TT. (12)
e\/r
Similarly, we maximize lis by doing the opposite:
f log (1+ (2N-2) -//=)
log (1 222 2(NN-1)
1 log 2N 2/T. (13)
This proposition tell that using Y e4) guarantees that all pairs used in the loss, for both the hard and the linear SPL regularizers. Additionally, when using the hard regularizer Rhard, y gmin the minimum learning pace so that at least one pair can be selected.
The complete SPL loss To exploit the information in all available meta-labels, our final loss combines the contrastive losses C Spp-con for meta-labels l = 1,..., K:
K
Sp--cnn
Σ
A Ck
Sp-con
(14)
Here, A 2 is coefficient controlling the relative importance of the 1tth meta-label in the final loss, which is determined by grid search on a separate validation set.
3.3 Semi-supervised segmentation with contrastive learning
In previous work 77, contrastive learning has mostly been used for pre-training the model. Here, we show that can further boost results semi-supervised setting, where training is performed with limited set of samples. In this setting, in addition to the unlabeled images D,, small amount of www.dwiweeanntateee images DD also available. To incorporate the knowledge from meta information semi-supervised setting, we modify our self- contrastive loss as
Luemi-sup
Laup
Areg freg Asp Lsp-cony
(15)
where Loup is the loss computed labeled data (cross-entropy loss in work), freg is the regularization loss normally used in semi-supervised approaches (in our experiments we use Mean Teacher) and Lse-con is our self- celf-paced contrastive loss. Last, Areg and Aap are weights balancing the different loss terms which are determined by grid search.
Experimental setup
To assess the performance of the proposed self- contrastive learning, we carry extensive experiments on five benchmark datasets with different experimental settings. In this section, we briefly describe these datasets and give implementation details for our method. For further information, the reader can refer the Supplementary Material.
4.1 Datasets
Five clinically-relevantt benchmark datasets for medical image segmentation are used for our exper- iments: the Automated Cardiac Diagnosis Challenge (ACDC) dataset (3) the Prostate MR Image Segmentation 2012 Challenge (PROMISEI2) dataset (29), and Multi-Modality Whole Heart Seg- mentation Challenge (MMWHS) dataset (64), well as the Hippocampus and Spleen segmentation

---

datasets from [.. These datasets contain different anatomic structures and present different acquisi- tion resolutions. For the contrastive loss, we exploit meta-labels on slice position and patient identity, Additionally, for ACDC, we consider the cardiac phase (i.e., systole or diastole) as a third source of meta-data. For all datasets, we split images into training, validation and test sets, which remain unchanged during all experiments. We train the model with only a few scans of the dataset as labeled data (the rest of the data is used without annotations as in semi-supervised setting) and report results in terms of 3D DSC metric [4] on the test set. Details on the training set split, data pre-processing augmentation methods and evaluation metrics can be found in the Supplementary Material.
For all datasets, we report the segmentation performance by varying the number of labeled scans across experiments. For the ACDC dataset, this number ranges from to 4, representing 0.5% to 2% of all available data. For PROMISEI2, we use 3 to 7 scans, representing 6% to 14% of the whole data. For MMWHS, we use 1 and annotated scans, corresponding 10% and 20% of the training data. We use 1 to 4 scans as annotated data for the Hippocampus dataset, representing 0.5% to 0.2% of the whole data, and 2 4 scans for the Spleen dataset, which corresponds to 5.7% to 11.4% of the whole available training data. Note that once randomly selected, those labeled volumes are fixed across the different experiments. Selecting labeled scans per experiment yielded significant variances (up to 1 11.25% in term of 3D DSC), as shown in the Supplementary Material. We include the segmentation results for both Hippocampus and Spleen datasets in the Supplementary Material.
4.2 Network architecture and optimization parameters
We use PyTorch [39 as our training framework and, following 77. employ the U-Net architecture 455 as our segmentation network. This 2D-based networks often works well for data with anisotropic acquisition resolutions. Moreover, it has a lower computational cost and require less GPU memory than its 3D counterparts. Network parameters are optimized using stochastic gradient descent (SGD) with a RAAdm optimizer [0) We provide the detailed training hyper-parameters in the Suppl. Material. For the pre- pre-training process, we obtain representations by projecting the encoder's output to a vector of size 256, using a simple MLP network with one hidden layer and LeaklyReLU activation function, following ( Our proposed self- self-paced contrastive learning objective, defined in Equ. (.)
involves a learning pace parameter "y set as
cur_epoch P
Nstart + (%end Vsttrt)) x
(16)
max_epoch
where Ystart, Tend are hyper-parameters controlling the importance weights in the beginning and the end of training, and cur_ _epoch max_ max_epoch are the current training epoch and the total number of training epochs respectively. p controls how fast ? increases during the optimization procedure.
5 Results
In this section, we first compare the hard and linear regularization strategy for SPL on ACDC. Then, we evaluate all components of our method in comprehensive ablation study with reduced set of training data on the different datasets. Finally, we compare our method with the most promising approaches for semantic segmentation in medical imaging, with reduced training data.
5.1 Hard vs. linear self-paced regularization
Table I reports the validation 3D DSC score for the hard and linear SPL, while training with different p in Equ. (16), and different numbers of annotated scans on the ACDC dataset. We observe that both SPL strategies (IRlardd and Nliuear) effectively help improve performance, however the linear strategy always leads to a higher improvement. This is because the hard strategy only employs binary weights, i.e., Wiy E {0, 1), whereas the linear strategy gradually increases Wiy and therefore provides a smoother optimization.
In Fig. [ (a), we plot the value of 7 over epochs for different values of and show in (b) the corresponding expectation of way for all positive pairs. We observe that, for large p, Y tends to be small for most of the training and mainly increases in the very end of the process, resulting in small wy for positive pairs. In contrast, when p 1/2, see rapid increase of weights way during training, which results in higher segmentation scores. This observation is inline with the findings from 44] and 42] on different tasks, where raising rapidly the self-paced learning rate in the first

---

Table I: 3D DSC Performance ACDC for hard and linear SP strat- egy and different values of
Figure 1: Self-paced strategy for y. (a) Evolution of y (b) Expectation of Wis over training epochs.
ACDC
SP type
1
scans
scans
Baseline
37.53%
67.06%
75.64%
0N.1
1/a 74.40%
80.34%
81.86%
Lincar
T2.06%
79.54%
81.03%
59.72%
70.36%
80.05%
1/a 64.42%;
78.26%
80.07% 80.24 %
Hard
72.01 %.
79.80%
71.86%
72.14%
79.19%
(a)
(b)
half of the training benefits the generalization performance. In the Suppl. Material, we also present a concrete evaluation of Wij for three different scans during model optimization, corresponding the four orange star markers in Fig. (b). Since we found that Rljnear strategy works better than Rhard we will use Rlinear for all following experiments.
5.2 Ablation study
Table summarizes the 3D DSC performance on test set for three datasets (ACDC, PROMISE12 and MMWHS) with very limited labeled data. At the top of the table, report the number of labeled scans used and, for every result, also give in parenthesis the standard deviation computed with different random seeds for parameter initialization. In the second and third columns of the table, we provide the loss used for the pre-training, any, and the loss for the downstream training.
Upper and lower bounds We present results for a Baseline which uses only the annotated scans with cross-entropy as standard supervised loss Lsup, and for Full Supervision where the same loss is used with all available data and associated annotations (175 for ACDC, 40 for Prostate and 10 for MMWHS). These two rows represent lower and upper bounds on the expected performance of the different variants of our approach.
Unsupervised contrastive loss We evaluate the performance pre-training the network encoder with Unsupervised Contrastive loss as in 8, where two augmented versions of the same image are considered as positive pair. In all datasets. this loss improves the improvement is limited because the amount of unlabeled data available still reduced compared
baseline model, although
to the settings of previous work on unsupervised contrastive learning [8] 20 10 62) We also add our self self-paced learning strategy on top of this contrastive loss, and call this modified model Unsupervised Contrastive + SP. As meta-labels may be noisy, performance is increased in almost all experiments, especially when fewer labels available.
Pre-training contrastive loss on meta-data We report the performance of model pre-trained with a Contrastive loss on meta-labels. The meta-labels are 3D slice location C patient identity 2 and cardiae phase L3 (only for ACDC). We find that that slice position always gives the highest accuracy among all leta-labels, and largely outperforms the unsupervised contrastive loss. While all meta-labels increase performance compared to unsupervised contrastive loss, their combination leads to the best results in most cases.
Pre-training self-pacee contrastive loss on meta-data Next, we evaluate the model pre-trained Contrastive loss meta-labels SP-Con (pre-train)). As with the unsupervised con- trastive loss, the self-paced approach also successfully improves segmentation quality compared to treating all positive pairs equally.
with
Semi-supervised We report the performance of model without any pre pre-training, but using the unlabeled data during training with our proposed self celf-paced contrastive loss (SP-Con (semi-sup)). It can be seen that performance is inferior to Contrastive pre-training on meta-data, however the improvement still quite relevant and, most cases, superior to unsupervised pre-training.
Pre-trained and semi-supervised Our next subsection reports results for the combination of Self-Paced Contrastive learning used for both pre-training and semi-supervised training (SP-Con

---

Table 2: 3D DSC performance (and standard deviation) for different components and approaches on three medical image datasets with few labelled scans.
ACDC
PROMISR12
Method
Pretain
Train
1 scam
2 scans
scams
3 accuns
scan
scans
scam
Baseline
Geup
[57 (1..58
67.06 (0.88) (0.13)
35.02 (2.59) 59,08 (1.25) 72.81 (1.8)
70.94 (0.84) (0.38)
(all labels)
Full Supervision
Coup
88.06 (0.20)
89. 10.70 (0.51)
88.27 (1 (1.25)
Czsup. Con.
Unsup Con, SP
Chilaa.
Cuup
Coop
66. (2.55)
(1.96)
75.11 (0.92)
12.88 (3.00)
(6.56(1.34)
76.87 (0.84)
40.21 (6.63)
(8.77)
67.3 (0.99)
60.99 (5.20)
75.28
75.14 (0.50)
74.12 (1.45) 80.57 (2.50)
74.30 (1 80.71 (0.83)
70.57 (0.96)
78.59 (0.79)
(0.69
55..4 (4.89)
75:21 (1.94)
80.02 (1.28)
76.53 (1.79) 83.05 (2.65)
Contrastive
Goop
63.63 (1.80)
13.50 (1.25)
76.83 (0.91)
(3.83)
69.95 (1.06)
78.95 (0.63)
74.71 (0.29)
82.41 (0.33)
76.81 (1.52
(0.97)
77.66 (0.56)
SP-Com (pre-traie)
Cy
Cup
69.28 (1.69)
73.99 (1.27)
76.34 (0.60)
81.01 (1 (1444)
82.83 (0.26)
[5888 (2.35)
$6.80
75.28
69.75 (0.47)
(1.49)
79.02 (0.18)
80.71 (1.27)
77.20 (0.87) 82.87 (0.59)
76.67 (0.45)
(1.50
63.18 (1.50)
79.05 (.026)
81 (0.16)
SP-Con (semi-sup)
C2
67.34 (0.74)
13.74 (0.51)
77.27 (0.12)
(1.53)
70.49
76.95 (0.81)
23.82 (0.68) 81 (0.39)
60.82 (0.95)
68.06 (2))
(0.35)
41.67 (1.59)
61.04((146)
75.98 (0.98)
73.43 (1.33)
78.08 (1.88)
62.52 (0.46) Co
68.39 (0.26)
77.24 (0.17)
SP-Con
(buok)
Eesp
De
C
NU.37 (5.66
70.47 (0.93)
76.58 (0.45)
82:35
78.39 (0.22)
56.68 (2.54)
(8..79
72.31
77.38
77.28 (1.86)
80.55 (0.75)
76.58 11000) 82.69 (0.39)
75.33 (0.62)
82.39 (0.27)
70.08 (0.96)
78.70 (0.51)
80.19 (0.28)
28.76 (0.26)
92.14 19)
84.42 (0.18)
(1.3))
(0.50
84.14 (0.35)
28.82 (0.34)
$4.90 (0.58)
SP-Con (hoth) Mesa Teacher
Cump
75. 30 (0.68)
79.67 26)
82.65 (0.32)
61. (1.33)
77.74
83.92 (0.39)
75.94 (0.00)
84.57 (0.61)
Co
73.94 (0.54)
81.29 (0.09)
83.21 (0.05)
4---
79.50 (0.33)
83.20 (0.25)
84.84 (0.15)
74.47 (0.36) 83.78 (0.30) (0.17)
78.97 (0.52)
(0.11)
(both)). Although the loss is the same, its use during pre-training and as additional regularization in a semi-supervised setting brings additional improvements.
Pre-trained and semi-supervised with Mean Teacher We then evaluate the model using both pre- pre-training and semi-supervised (as the previous setting) but with an additional cemi-supervision (SP-Con (both) Mean-Teucher). By combining our approach with a simple Mcan- Mean-Teacher method, our results on all datasets are further boosted, approaching the performance of fully supervised training but using a very low number of annotated scans. This is the model that is used in the comparison with the state-of-the-art.
for
5.3 Comparison with the state-of-the-art
Table 3: 3D DSC performance (and standard deviation) of our method and other approaches on three medical image datasets with few labelled scans. Bold red-colored values are the best performing methods, underlined blue-colored ones correspond to the second best performing method.
ACDC
PROMISE12
MMWHS
Method
1 scan
2 scans
scans
3 scans
5 scans
7 scans
sean
scans
Entropy Min.
50
60.47 (1.03)
69.81 (0.99) 76.19 (1.21)
53.47 (5.70)
65.66 (0.42)
73.52 (2.71)
72.28 (0.58) 78.39 (1.54)
Mix-up 58
60.87 (1.28)
67.45 (1.04) 76.18 (0.49)
41.38 (2.80)
64.55 (1.93)
73.56
71.50 (0.54)
80.12 (0.84)
Adv. Training 60
63.05 (0.80) 70.68 (0.27) 75.89 (0.94)
61.58 (2.10) 71.00 (1.20)
81.05 (1.34)
73.47 (1.42)
80.40(0..33) (0.93)
Mean Teacher
62.85 (0.67) 72.84 (0.22) 79.12 (0.08)
$2.96 (1.97) 68.38 (2.04) 77.37 (0.87)
72.36 (1.35)
81.01 (0.57)
Discrete MI [1
69.27 (1.41) 77.74 (0.42) 80.06 (0.24)
47.77 (3.58)
68.29 (2.35)
77.63 (1.13)
72.38 (1.04)
82.45 (1.36)
Contrastive t
20.05 (2.66) 79.11 (2.02)
81.25 (2.15)
61.15 (2.95)
74.62 (1.69)
80.08 (1.39)
76.45 (0.62)
82.93 (0.42)
Our Method
79.80 (0.33) 83.20 (0.25) 84.84 (0.15) 74.47 (0.36) 83.78 (0.30)
84.52 (0.17)
78.97 (0.52) 84.87 (0.11)
We compare our method with other approaches that aim to improve training with few annotated images/scans. Table 3 presents results in terms of 3D DSC score for approaches based on data augmentation (58). pre- -rainiain the weights on both encoder and decoder of the model [7] and various semi-supervised learning methods (50] 60 43
As with the ablation study, we report results for the ACDC, PROMISE12 and MMWHS datasets. A detailed explanation of the experimental setup of each method and results for the other two datasets can be found in the Supplementary Material.
To have fair comparison, for all methods, we used grid search on the validation set tune the hyper-parameters. For most methods, the improvement with respect to the baseline trained with only the supervised loss is quite limited and varies depending on the dataset and the number of annotated scans used. For instance, Adversarial training performs quite well on the PROMISE12 datasct (scans in this dataset exhibit more variability in terms of intensity contrast), but not well

---

10
on ACDC. Likewise, Mean-Teacher does not perform well for 1 or 2 annotated scans in ACDC but, when increasing the seans to it outperforms most of the other methods. The global and local Contrastive loss using meta-data manages to obtain an excellent improvement on all datasets. However, approach still yields substantial improvements with respect that method. This is due to our proposed self- self-paced learning strategy, as well as the combined use of the contrastive loss for pre-training and semi-supervised learning.
6 Discussion and conclusion
In this paper, we proposed a technique based on contrastive loss with meta-labels that can highly improve the performance of a medical image segmentation model when training data is scarce. It was shown that, with reduced amount of unlabeled images, unsupervised contrastive loss is not very effective. Instead, in the context of medical images, additional meta-data is freely available and, if properly used, can greatly boost performance. We presented results on five well-known medical image datasets and have shown that the accuracy of the contrastive loss with meta-labels can be boosted by the use of self-paced learning. Our self- self-paced contrastive learning method can well regularization loss during semi-supervised training, and the combination of the two can further boosts results. Finally, we have compared our approach with the state-of-the-ee in semi-super learning, and have shown that the simple combination of our approach with multiple meta-data and a simple semi-supervised approach as Mean Teacher more effective than previous approaches. While using few scans, our method can approach fully supervised training.
be used during pre-training
Social impact and limitations The proposed method can have an effective and practical impact in terms of medical imaging analysis in hospitals and health centers. As shown in our experiments, it produces an accurate medical image segmentation with very reduced set of annotated data. This has the potential of helping radiologists and other clinicians using medical images, which can in turn contribute to better diagnosis and reduced costs. While our empirical evaluation has shown excellent results with very limited data, using fewer annotated images also increases chances of over-fittiigg potential outliers in the data that may lead to erroneous or misleading results. A further study on the reliability of medical image segmentation with reduced images is therefore recommended.
7
Acknowledgment
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC Grant No. RGPIN-2018 (4425) and Fonds de recherche du Québec Nature et technologies (FRQNT Grant No. B2X 276565). This research was also enabled in part by support provided by Calcul Québec (www.cacculquebec.co and Compute Canada (www. www.commutecanadaa.a)
References
[1] M. Antonelli, A. Reinke, S. Bakas, K. Farahani, B. A. Landman, G. Litjens, B. Menze, O. Ron- nebrgerr, R. M. Summers, B. van Ginneken, et al. The medical segmentation decathlon. arXiv preprint arXiv:2
2021.
[2] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pages 41-48, 2009.
[3] O. Bernard, A. Lalande, C. Zotti, F. Cervenansky. X. Yang, P.-A. Heng, I. Cetin, K. Lekadir, O. Camara, M. A. G. Ballester, al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE Transactions on Medical Imaging, 2018.
[4] J. Bertels, T. Eelbode, M. Berman, D. Vandermeulen, F. Maes, R. Bisschops, and M. B. Blaschko. Optimizing the dice score and jaccard index for medical image segmentation: Theory and practice. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 92-100. Springer, 2019.
[5] G. Bortsova, Dubost, I.. Hogeweg, I. Katramados, and M. de Bruijne. Semi-supervised medical image segmentation via learning consistency under transformations. In International Conference on Medical Image Computing and Conputer-Assisted Intervention, pages 810-818. Springer, 2019.

---

11
[6] K. Chaitanya, N. Karani, C. F. Baumgartner, A. Becker, O. Donati, and E. Konukoglu. Semi- supervised and task driven data augmentation. In International Conference on Information Processing in Medical Imaging, pages 29-41. Springer, 2019.
[7] K. Chaitanya, E. Erdil, N. Karani, and E. Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. In Advances in Neural Information Processing Systems, volume 33, pages 22546-12555 2020.
[8] T. Chen, S. Korbliihh, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, pages 1597-1607, 2020.
[9] Chen, S. Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
[10] T. Chen, S. Korbblith, K. Swersky, M. Norouzi, and G. Hinton. Big eelfssuee models are strong semi-supervised learners. arXiv preprint arXiv:2006. /0029, 2020.
[11] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint wrXXv:2003.04222 2020.
[12] W. Cui, Y. Liu, Y. Li. M. Guo, Y. Li, Li, T. Wang, X. Zeng. and C. Ye. Semi-supervised brain lesion segmentation with an adapted mean teacher model. In International Conference on Information Processing in Medical Imaging, pages 554-565. Springer, 2019.
[13] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In Computer Vision and Puttern Recognition, 2009. CVPR 2009. IEEE Conference on, pages
Ieee, 2009.
[14] C. Doersch and A. Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2051-20666 2017.
[15] A. Dosovitskiy, Springenberg, M. Riedmiller, and Brox. Discriminative unsupervised feature learning with convolutional neural networks. In Advances in Neural Information Processing Systems, volume 27, 2014.
[16] DuD J. Lee, H. Li, Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. In K. Chaudhuri and R. Salakhutdinov, editors. Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 1675-1685, 2019.
[17] D. Erhan, Y. Bengio, A. Courville, P.-.. Manzagol, P. Vincent, and S. Bengio. Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res., 11:625-660 Mar. 2010. ISSN 1532-4435.
[18] Z. Feng, C.
Xu, and D. Tao. Self-supervised representation learning by rotation feature decoupling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10364-10374, 2019.
[19] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. ArXiv, abs/1803.07725 2018.
[20] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9729-9738. 2020.
[21] R. D. Hjelm, A. Fedorov, S. Y. Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint
K. Grewal, P. Bachman, A. Trischler, and
(at.:::008.66770. 2018
[22] M. Huh, P. Agrawal, and A. A. Efros. What makes imagenet good for transfer learning? arXiv preprint arXiv: 1608 08614 2016.
[23] L. Jiang, D. Meng. S. Yu, Z. Lan, S. Shan, and A. Hauptmann. Self-paced learning with diversity. In Advances in Neural Information Processing Systems, pages 2078-2086, 2014.

---

12
[24] P. Khosla, P. Teterwak, C. Wang. A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and
D. Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. 11362, 2020.
[25] M. Kim, J. Tack, and S. J. Hwang. Adversarial self-supervised contrastive learning. arXiv preprint 201.::0000.778899 2020.
[26] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep con- volutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Wein- berger, editors, Advances in Neural Information Processing Systems. volume 25. Curran Associates, Inc.. 2012. URL https: https::/ p/roceedinnsscsgss neurips wwwprrisscccppppr//aaaaaaa pt/pppper/0122/11ll (ttp:6222bbbb6666633666638888882 Pappe pdf
[27] M. P. Kumar, Packer, and D. Koller. Self-paced learning for latent variable models. In NIPS, volume I, page 2, 2010.
[28] H.-Y. Lee, J.-B. Huang, M. Singh, and M.-H. Yang. Unsupervised representation learning by sorting sequences. In Proceedings of the IEEE International Conference on Computer Vision, pages 667-676, 2017.
[29] G. Litjens, R. Toth, W. van de Ven. C. Hocks, S. Kerkstra, B. van Ginneken, G. Vincent, G. Guillard, N. Birbeck, J. Zhang, et al. Evaluation of prostate segmentation algorithms for MRI: the PROMISE12 challenge. Medical image analysis, 18(2):359-333 -7335,9-37331 2014.
[30] I.. Liu, H. Jiang, P. He, W. Chen, X. Liu, Gao, and J. Han. On the variance of the adaptive learning rate and beyond. arXiv preprint MrXiv:1908.03265 2019.
[31] Z. Liu, Z. Zhu, S. Zheng, Y. Liu, Zhou, and Y. Zhao. Margin preserving self-paced con- trastive learning towards domain adaptation for medical image segmentation. arXiv preprint arXiv:2103.088 08455, 2021.
[32] S. Min and X. Chen. A robust deep attention network to noisy labels in sumi-supervised biomedical segmentation. arXiv preprint urXiv:1807.11711 2018.
[33] I. Misra and L. d. Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1707-6717, 2020.
[34] M. A. Morid, A. Borjali, and G. Del Fiol. A scoping review of transfer learning research on medical image analysis using imagenet. Computers in Biology and Medicine, page 104115, 2020.
[35] Q. Nguyen and M. Hein. The loss surface of deep and wide neural networks, 2017.
[36] A. Oliver, A. Odena, C. A. Raffel, E. D. Cubuk, and I. Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems 31, pages 3239-3250. 2018
[37] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 0000110770074 2018
[38] L. Paine, P. Khorrami, W. Han, and S. Huang. An analysis of unsupervised pre-training in light of recent advances. arXiv preprint arXiv: 1412.697, 2014.
[39] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. VeVito, Z. Lin, Desmaison,
L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.
[40] J. Peng, G. Estradab, M. Pedersoli, and C. Desrosiers. Deep co-training for semi-supervised image segmentation. arXiv preprint 1rXiv:1903.1123: 2019.
[41] J. Peng, M. Pedersoli, and C. Desrosiers. Boosting semi-supervised image segmentation with global and local mutual information regularization. arXiv preprint arXiv:2103.04813 2021.
[42] G. Penha and C. Hauff. Curriculum learning strategies for ir: An empirical study on conversation response ranking. arXiv preprint arXiv:1912.08555. 2019.

---

13
[43] C. Perone and J. Cohen- Adad. Deep semi- supervised segmentation with weight-averaged consistency targets. In Deep learning in medical image analysis and multimodal learning for clinical decision support, pages 12-19. Springer, 2018.
[44] E. A. Platanios, O Stretcu, G. Neubig, B. Poczos, and T. M. Mitchell. Competence-based curriculum learning for neural machine translation. arXiv preprini arXiv:1903 09840, 2019.
[45] O. Ronneberger, P. Fischer, and I. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234 241, 2015.
[46] Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. cime-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 1134-1141. IEEE. 2018
[47] D. Soudry and Y. Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks, 2016.
[48] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. arXiv preprint arXiv: 190.v:996.00884 2019.
[49] P. Vincent, H. Larochelle, Y. Bengio, and P.-A Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning, page 1096-1103, 2008.
[50] T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2517-2526, 2019.
[51] P. Wang, J. Peng, M. Pedersoli, Y. Zhou, C. Zhang, and C. Desrosiers. Self-paced and self- consistent co-training for semi-supervised image segmentation. Medical Image Analysis, 73: 102146, 2021.
[52] W. Wang, Y. Lu, B. Wu, T. Chen, D. Z. Chen, and J. Wu. Deep active self-pane self-paced learning for accurate pulmonary nodule segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 723-731. Springer, 2018.
[53] X. Wu, F. Dyer, and B. Neyshabur. When do curricula work? In International Conference on Learning Representations, 2021. /RLL: www.:neefforii tiw-:mmmdddttttinmmmm
[54] Y. Xiong, M. Ren, W. Zeng. and R. Urtasun. supe-super representation learning from flow equivariance. arXiv preprint vxXiv:2011005555 2021.
[55] L. Yu, S. Wang, X. Li, C.-W. Fu, and P.-A. Heng. nncertainty-awar self-ensembling model for semi- supervise 3d left atrium segmentation. In International Conference on Medical Image Computing and Computer-Assiste
Intervention, pages 605-613. Springer, 2019.
[56] D. Zeng, Y. Wu, X. Hu, X. Xu, H. Yuan, M. Huang. J. Zhuang. J. Hu, and Y. Shi. Positional con- trastive learning for volumetric medical image segmentation. arXiv preprint arXiv:2106.0913 2021.
[57] D. Zhang, D. Meng, L. Zhao, and J. Han. Bridging saliency detection to weakly supervised object detection based on self-paced curriculum learning. arXiv preprint 17X3v:170..0/20) 2017.
[58] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. arXiv preprint
2017.
[59] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In European conference on computer vision, pages 649-666. Springer, 2016.
[60] Y. Zhang, L. Yang, J. Chen, M. Fredericksen, D. P. Hughes, and D. Z. Chen. Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 408-416. Springer, 2017.

---

14
[61] A. Zhao, G. Balakrishnan, F. Durand, J. V. Guttag, and A. V. Dalca. Data augmentation using learned transformations for one-shot medical image segmentation. Proceedings of the IEEE conference on computer vision and patlern recognition, pages 8543-8553, 2019.
[62] X. Zhao, R. Vemulapalli, P. Mansfield, B. Gong, B. Green, L. Shapira, and Y. Wu. Contrastive learning for label-efficient semantic segmentation. arXiv preprint 20ix:20122006833 2020.
[63] Y. Zhou, Y. Wang, P. Tang, S. Bai, W. Shen, E. Fishman, and A. Yuille. Semi-supervised 3D abdominal multi- multi-organ segmentation via deep mulli-planar co- vo-training. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 121-140. IEEE, 2019.
[64] X. Zhuang and J. Shen. Multi-scale patch and multi-modality atlases for whole heart segmenta- tion of MRI. Medical image analysis, 31:77-87, 2016.


Reviews:
[{"review_id": "wuGy_Xwu6A", "reviewer": "Reviewer_G3GS", "summary": "This work proposes the use of meta labels and self paced learning to improve performance of medical image segmentation when using contrastive pre-training. Through results on three different medical image segmentation tasks, the authors demonstrate the efficacy of the proposed method.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nWhile the paper is mostly built on top of https://arxiv.org/pdf/2006.10511.pdf, the addition of self paced learning and meta labels are interesting improvements on top of the prior work.\n\nQuality and clarity:\nThe paper is mostly well-written and easy to follow. The authors demonstrate good experimental rigor with experiments on multiple datasets and good ablations\n\nSignificance:\nIts challenging and expensive to obtain labels for medical segmentation. Efforts like this which aim to improve the sample efficiency for medical segmentation are great research directions in general\n\nDetailed Comments:\n\n> However, thanks to large datasets like ImageNet [11], the amount of labeled data for training has increased and, in such setting, pre-training often hinders performance [36].\n\nThis comment is a bit confusing. You may want to clarify that while supervised pre-training on Imagenet is still the norm for developing many vision models, this comment is for unsupervised pre-training\n\n> Contrastive training as a regularization loss in semi-supervised learning. \n\nPlease cite related work - https://arxiv.org/abs/2102.06605\nThere is a lot of recent work on self supervised learning for medical imaging and segmentation. Related work section is missing those papers.\nEq 2 is pretty much an instantiation of the supervised contrastive objective. Please cite this in the text\n\n>  These update rules in Equ. (7) can be explained intuitively. For a given γ, the hard threshold rule only considers confident pairs with Lij ≥ γ and ignores the others\n\nThere seems to be an error here and i think this should be Lij < y\n\n>  contrastive learning has mostly been used for pre-training the model. Here, we show that it can further boost results in a semi-supervised setting.\n\nPlease cite https://arxiv.org/abs/2006.10029 here\n\n> How is gamma in eq 16 related to the gamma derived from equation 2? It is not clear they relate.\n\nPlease substantiate the relationship here.\n\nIt is difficult to read the results in table 3. Please bold the best ones\n\n> Please add more details as to how the self paced objective was in the semi-supervised setup without the contrastive objective. \n\nAssume the loss in this case would be wrt individual training examples?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Major concern regarding the work is \n- why the experimentation setup doesn't exactly match https://arxiv.org/pdf/2006.10511.pdf? The papers seem to use different number of training examples\n- it would be good to see the demonstration of self paced learning beyond segmentation or medical tasks. Without the results, it is difficult to ascertain the generality of the proposed method(s)\n\nIf the authors can address these concerns satisfactorily, will be happy to reconsider my scores.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "aEYWItERbXo", "reviewer": "Reviewer_HQGh", "summary": "Annotation in the clinical dataset is hard to prepare, so it is important if self-learning algorithms can be applied to medical data which can do few-shot learning. In this paper, authors put forward a method to do segmentation which makes use of pretraining with contrastive learning with metadata, and then better fine tuning with self-paced learning. The second step is necessary to mitigate meta data noise. This paper shows experiments on three public datasets. With a few annotated data, the algorithms still show comparable performance with the fully supervised segmentation algorithm.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The major strength of this paper is the soundness of empirical methodology used to validate the empirical approach. Authors showed intensive experiments and persuasive results to demonstrate the advance of the proposed methods compared with prior works (table 2 and table 3).\nThe idea of this paper is also with great clinical impact: right now, fully supervised segmentation methods although show good performance, depends heavily on the expensive annotations. It is great to explore the self-/semi- supervised learning algorithms to reduce such strict requirement on data acquisition.\n\nIn this paper, authors discussed about the previous related work about self-supervision, contrastive learning and self-paced learning, and later clearly discussed how the proposed method was built upon the prior work.\n\nThis work is reproducible. They used public datasets to train and evaluate the algorithm, and the authors promised to make the codes public available if the paper got accepted.\n\nPaper Strength:\n+ A new self-paced strategy is proposed for contrastive learning by assigning importance weights to positive sample pairs. With self-paced strategy, the model learns from easy samples in the beginning and gradually add noisy/hard samples.\n+ Meta-labels are incorporated in the contrastive learning process, which is reasonable and effective. Using all three meta-labels achieves the best performance.\n+ Experiments show that the proposed method outperforms several existing methods on 3D medical image segmentation in the semi-supervised setting. Ablation experiments also demonstrate the effectiveness of the proposed self-paced learning strategy and the use of meta-label in contrastive learning.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One weakness is in the experiment design. In this paper, table 2, authors showed the ablation study of all combination of loss functions on all three datasets, and selected “SP-Con (both) + Mean Teacher” as the best model to report. It would be better if this ablation study and model selection was done only with one of the three datasets, and only report the best model’s performance on the rest, so that it can follow the best practice of model selection. In table 1 (section 5.1) the selection of hard vs. linear self-paced regularization is a good example of parameter selection.\nAnother limitation for the current submission is that authors did not discuss whether all organs can be segmented with the proposed algorithm design. Right now, from the dataset point of view, it works with prostate (PROMISE12) and heart (ACDC and MMWHS). Further experiments and discussion with different organs could better support the idea that proposed method could have a good clinical impact.\n\nOverall, this paper is well written. However, it could be better if an algorithmic workflow figure is provided. Right now, although this paper put much effort to discuss about the methodology, it is still not straight forward enough for readers to understand how this algorithm works from input to contrastive learning, and to the self-paced learning. \nIn table 1, performance on ACDC is shown with percentage. It would be better, if in the table title, the authors could explain what are these percentage numbers: are they dice or auc or any other metrics.\n\nMore graphics in the main body, instead of putting them in the supplementary material, could better help readers understand the idea. For example, choosing one row from Figure 1, Figure 2, and Figure 3 in the supplementary material respectively as a new figure, and moving it to the main body as examples of “meta-labels”. A selection of Figure 6 in the supplementary material could also be shown in the main body, to visually show the segmentation performance.\n\nAdditionally, \n- There are many hyper-parameters used in the paper, which are determined using grid search on the validation set. But they are not specified, including the loss weights in Equation (14) and (15). How is the temperature factor set in Equation (2)?\n- The output of the encoder should be a 2D feature map. Why use a non-linear head to project it to 256 dimensional vector? Is the vector reshaped to a 2D feature map before fed to the decoder?\n- It appears that both the training and the inference are done for 2D slices individually. The 3D continuity information between slices is lost. Such information is important for 3D medical image segmentation, although it might be outside the scope of this paper. It would be better to include a 3D segmentation model for comparison.\n- The experiments show that the proposed method is useful when there are few training data (1 to 7 scans). How does the proposed method work when more labeled data are available? It should be studied since they are available in the three datasets. Besides, the training scans are randomly selected from the available labeled data. How does the random selection affect the performance?\n- Table 1 does not have the baseline results for reference.\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "OT0FE1L7ru", "reviewer": "Reviewer_UfTw", "summary": "In this work, the authors show how to use weak labels to boost segmentation of medical datasets.\nThe idea hinges on a combination of contrastive learning with a new self-paced learning approach that gives preference to more confident labels (and therefore not being mislead by label noise or irrelevant weak labels).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading the paper. It is written very clearly, addresses very relevant issues (learning with limited data, utilisation of weak labels), and convinced me by presenting a solid set of experiments as well as ablations/baselines.\n\nThe theoretical contributions are, as they are presented, useful and novel. Contributions might not be seen as earth shattering, are likely not the start of a new paradigm, but can still be labeled as solid. The main text points at all hyper-parameters and how good values have been found (grid search, mostly). The range for sensible $\\gamma$ values, one of the more critical parameters to choose, are discussed in detail, and presented understandably.\n\nWhile I'm fairly confident that I understand the manuscript very well, I want to point out that I do not know the related work very well and would not know if some work exists that was not mentioned/cited by the authors.\n\nIn summary, the manuscript presents a solid method that shows excellent performance wrt. to competing methods, even if only very limited training data is available. Contributions are good, presentation excellent.\n\nMinor comments:\nline 144 -- I would have preferred to find a more complete list of used meta-labels.\nline 159 -- What is g(.)? Could the authors be more concrete about it?\nline 170 -- Superfluous \"A\".\nline 182 -- missing \"y\" in \"by\".\npage 5 -- l_{i,j} is used with inconsistent font\nline 198 -- shouldn't it be l_{i,j} <= \\gamma ???\nline 218 -- I was wondering how critical is it at the end to find the right \\lambda_k? (How fine do you need to grid search, how far can you be off before results drop significantly in quality?)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A social impact and limitations sections exists and I agree with the authors comments in it.", "review_text": "", "extra_scores": {"ethical_concerns": "Nothing that goes beyond the concerns one could voice about virtually all deep learning submissions these days.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "about 5 hours in 3 sessions distributed over one day"}, "extra_text": {}}, {"review_id": "1oyucy20C1P", "reviewer": "Reviewer_n24V", "summary": "The authors propose using “metal labels” in pertaining image encoders on unlabeled datasets. They use the metal-labels in pertaining the encoder, and also as an auxiliary task in training. They show their method works on multiple medical datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The value of clustering 2D slices in a 3D volume seems not as helpful as just learning the model on 3D data in the first place (a 3D implementation of the U-net exists). \n\nMore information could be provided on what meta-labels were included. It’s possible that manually viewing them with some domain expertise could already select those that are unlikely to contribute any signal.\n\nBest performance is only achieved by combining everything, compared with the baselines that quite possibly focus on a single innovation. Compared with the other individual methods tried, the best performance is achieved by the baselines in some cases. So the paper becomes more about how much one can push performance than introducing a single innovation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are short - I’ve mentioned a number above, but these are not discussed in the limitations section.\n\nThe social impact of the problems tackled is clear, as the authors have focused on medical use cases. I don’t think it is as hard to collect unlabeled medical data as they are claiming (there are many large open source datasets), but the quality of labels is poor so the problem remains an important one.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers broadly agreed that the overall motivation for the work (reducing the dependency on labels in medical image segmentation) and the novelty and thorough evaluation of the proposed method are of general interest to the community. There were a few minor clarity points brought up that should be addressed in the revised version, and the authors should also include the additional experimental results from the discussion.  
Decision: Accept (Poster)