Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels
Year: 2021

Abstract:
The contrastive pre-training of a recognition model on a large dataset of unlabeled data often boosts the model’s performance on downstream tasks like image classification. However, in domains such as medical imaging, collecting unlabeled data can be challenging and expensive. In this work, we consider the task of medical image segmentation and adapt contrastive learning with meta-label annotations to scenarios where no additional unlabeled data is available. Meta-labels, such as the location of a 2D slice in a 3D MRI scan, often come for free during the acquisition process. We use these meta-labels to pre-train the image encoder, as well as in a semi-supervised learning step that leverages a reduced set of annotated data. A self-paced learning strategy exploiting the weak annotations is proposed to further help the learning process and discriminate useful labels from noise. Results on five medical image segmentation datasets show that our approach: i) highly boosts the performance of a model trained on a few scans, ii) outperforms previous contrastive and semi-supervised approaches, and iii) reaches close to the performance of a model trained on the full data.

heading: 1 Introduction
text: Since the emergence of deep learning [26], there has been an active debate on the importance of pre-training neural networks. Precursor works [17] showed that pre-training a convolutional neural network with an unsupervised task (e.g., denoising autoencoders [49]) could lead to a better performance in the final supervised task. As the amount of labeled training data increased, thanks to large datasets like ImageNet [13], it was however found that pre-training could actually hinder performance [38]. This makes sense in light of recent studies showing, for instance, that symmetries in large networks induce many equivalent local minima [16, 35, 47] in which a pre-trained model can get stuck. Recently, contrastive learning has renewed the interest in unsupervised pre-training [37]. Several works [8, 10, 11, 20, 62] have found that pre-training a model with a contrastive loss can improve its performance on a subsequent supervised training task, often outperforming a network with supervised pre-training on ImageNet. While this has reopened the debate on the benefit of pre-training, it offers little help for domains where data is scarce such as medical imaging. In medical imaging, not only are labels expensive since they come from highly-trained experts like radiologists, but images are also hard to obtain due to the need for costly equipment (e.g., MRI or CT scanner) and privacy regulations.
Over the last years, a breadth of semi-supervised learning approaches have been proposed for medical image segmentation, including methods based on attention [32], adversarial learning [60],
∗Corresponding author
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
2 temporal ensembling [12, 55], co-training [40, 63], data augmentation [6, 61] and transformation consistency [5]. The common principle of these approaches is to add an unsupervised regularization loss using unlabeled images, which is optimized jointly with a standard supervised loss on a limited set of labeled images. Despite reducing significantly the amount of labeled data required for training, current semi-supervised learning methods still suffer from important drawbacks which impede their use in various applications. Thus, a large number of unlabeled images is often necessary to properly learn the regularization prior. As mentioned before, this may be impossible in medical imaging scenarios where data is hard to obtain. Moreover, these methods also need a sufficient amount of labeled data, otherwise the learning may collapse [36].
In a recent work, Chaitanya et al. [7] showed that unsupervised pre-training can be useful to learn a segmentation task with very few samples, by leveraging the meta information of medical images (e.g., the position of a 2D image in the 3D volume). While achieving impressive accuracy with as few as two volumes, this work has significant limitations. First, it relies on the strong assumption that the global or local representations of 2D images are similar if their locations within the volume or feature map are related. This assumption does not always hold in practice since volumes may not be well aligned, or due to the high variability of structures to segment. Second, it requires dividing the 2D images of a 3D volume in an arbitrary number of hard partitions that are contrasted, while the structure to segment typically varies gradually within the volume. Third, they do not exploit the full range of available meta data, for instance the patient ID or cycle phase of cardiac cine MRI, nor evaluate the benefit of combining several types of meta information in pre-training. Last, their approach leverages meta data only in pre-training, however this information could further boost performance if used while learning the final segmentation task, in a semi-supervised setting.
Our work addresses the limitations of current semi-supervised and self-supervised approaches for segmentation by proposing a novel self-paced contrastive learning method, which takes into account the noisiness of weak labels from meta data and exploits this data jointly with labeled images in a semi-supervised setting. The detailed contributions of this paper are as follows:
– We propose, to our knowledge, the first self-paced strategy for contrastive learning which dynamically adapts the importance of individual samples in the contrastive loss. This helps the model deal with noisy weak labels that arise, for instance, from misaligned images or splitting a 3D volume in arbitrary partitions.
– We demonstrate the usefulness of a contrastive loss on meta-data for improving the performance of a final task, not only in pre-training but also as an additional loss in semi-supervised training.
– We show that combining multiple meta-labels in our self-paced contrastive learning framework can improve performance on the final task, compared to using them independently. Our results also demonstrate the benefit of combining contrastive learning with temporal ensembling to further boost performance.
We empirically validate our contributions on five well-known medical imaging datasets, and show the proposed approach to outperform the contrastive learning method of [7] as well as several state-ofthe-art semi-supervised learning methods for segmentation [41, 43, 50, 58, 60]. In the results, our approach obtains a performance close to fully supervised training with very few training scans.

heading: 2 Related work
text: We focus our presentation of previous works on two machine learning sub-fields that are most related to our current work: self-supervision, which includes contrastive learning, and self-paced learning.
Self-supervision and contrastive learning Self-supervision is a form of unsupervised learning where a pretext task is used to pre-train a model so to better perform a downstream task. Examples of pretext tasks are learning to sort a sequence [28, 54], predicting rotations [19, 18], solving a jigsaw puzzle [33] and many others [14, 15, 25, 46, 59]. Most of these methods can improve performance on the downstream task when labeled data for training is scarce. However, when a large and generalenough dataset of labeled images like ImageNet [13] is available, a simple supervised pre-training may sometimes provide better results [22, 34]. Recently, unsupervised contrastive learning [8, 20, 10] was shown to boost performance even when learning a downstream task on a large dataset, and to improve over a model pre-trained in a supervised manner on a large dataset.
This approach is based on the simple idea of enforcing similarity in the representation of two examples from the same class (positive pairs), and increase representation dissimilarity on pairs from different classes (negative pairs) [37, 48]. In image analysis tasks, positive pairs are typically defined as two
3 transformed versions of the same image, for instance using a geometric or color transformation, while negative ones are any other pair of images [8, 10, 11, 20].
In a recent work, Khosla et al. [24] showed that, when combined with true semantic labels, a contrastive learning based “pre-train and fine-tune” pipeline performed surprisingly well, outperforming conventional training with cross-entropy in some cases. The work in [62] extended this idea to pixel-level tasks like semantic segmentation, clustering the representations of pixels in an image according to their labels. Applying a similar strategy to medical image segmentation, Chaitanya et al. [7] leveraged meta-labels from 3D scans in a local and global contrastive learning framework to improve performance when training with limited data. However, positive and negative pairs in their contrastive loss are defined using noisy “weak” labels, arising for example from misaligned images or an arbitrary partitioning of 3D volumes, which may lead to learning sub-optimal representations. The work in [56] mitigated this problem by imposing a maximum distance along the z axis between slices forming positive pairs. Moreover, existing approaches only exploit meta-labels in pre-training, instead of considering them jointly with labeled images in a semi-supervised strategy. Our work extends these approaches with a self-paced learning method that adapts the importance of positive pairs dynamically during training, and focuses the learning on the most reliable ones. In contrast to [7], we also exploit contrastive learning as regularization loss in semi-supervised training and show that further improvements can be achieved when combining it with a temporal ensembling strategy like Mean Teacher [12, 55]. A recent approach by Chen et al. [10] also performs contrastive learning for image recognition in a semi-supervised setting. In this approach, a large teacher model is trained with a unsupervised contrastive loss and then fine-tuned with a small fraction of labeled data. A student model is trained afterwards using a knowledge distillation technique. Hence, unlike our method, there is no joint optimization of the supervised and contrastive objectives. In this work, we show that significant improvements can be achieved by jointly optimizing these two objectives.
Self-paced learning A sub-category of curriculum learning (CL) [2], self-paced learning (SPL) is inspired by the learning process of humans that gradually incorporates easy to hard samples in training [27]. The effectiveness of such strategy has been validated in various computer vision tasks [52, 23, 57]. Jiang et al. [23] proposed an SPL method considering both the difficulty and diversity of training examples, which outperformed conventional SPL methods that ignore diversity. Zhang et al. [57] incorporated SPL in a DNN fine-tuning process for object detection, to cope with data ambiguity and guide the learning in complex scenarios. The usefulness of SPL when training with a limited budget or when the training data is corrupted by noise was also studied in recent work [53]. So far, the application of SPL to image segmentation remains limited. Wang et al. [52] presented an SPL method for lung nodule segmentation, where the uncertainty of each sample prediction in the loss is controlled by the SPL regularizer. Similarly, a self-paced co-training method was proposed in [51] for the semi-supervised segmentation of medical images. Related to our work, Wang et al. [31] proposed a margin preserving contrastive learning framework for domain adaptation that uses a self-paced strategy in self-training. Compared to this approach, which uses SPL outside the contrastive loss to select confident pseudo-labels for self-training, our method incorporates it within the loss via importance weights that are learned jointly with network parameters.

heading: 3 Proposed method
text: In this section, we present our self-paced contrastive learning approach for segmentation that leverages intrinsic meta information extracted from medical volumetric images. Our method effectively pretrains a segmentation model with a self-paced variant of contrastive learning that is more robust to noisy annotations. The same approach is also used to further boost the segmentation accuracy in a semi-supervised setting, where only very limited pixel-wised annotations are used. In the following subsections, we detail the formulation of the proposed approach.

heading: 3.1 Contrastive learning with meta-labels
text: Given a batch of N images from a dataset Du of unlabeled images, unsupervised contrastive learning approaches [8, 21, 37] aim at finding a feature extractor f(·) that gives similar representations for two augmented instances of the same image and different ones for those of two separate images, regardless of their true classes. This can be achieved by creating an augmented set of samples indexed by i ∈ I ≡ 1, . . . , 2N , with two augmented samples for each original image in the batch. The
4 following loss is then optimized:
LunsupCon = 1
2N 2N∑ i=1 − log exp
( zᵀi zj(i)/τ )∑ a∈A(i) exp ( zᵀi za/τ
) (1) In this loss, zi =
f(xi) ‖f(xi)‖ 2 is the L2-normalized representation of an image xi in the augmented batch (i.e., the anchor) and j(i) is the index of the other augmented sample from the same image (i.e., the positive). A(i) ≡ I \ {i} contains all indexes of the augmented set except i, and has a size of 2N−1. Finally, τ is a small temperature factor that helps gradient descent optimization by smoothing the landscape of the loss.
This approach works well when a large dataset of unlabeled images is available. However, the number of available images is small in our case. To alleviate this problem, our contrastive learning framework also leverages meta-labels arising from the structure of the data. Following [7], we consider the 2D slices of a given set of M volumetric scans as our training data, and extract various meta-labels for each 2D image (e.g., patient ID, position of the slice in the volume, etc). More generally, we suppose that each image xi has set of K meta-labels denoted as yki ∈ {1, . . . , Ck}, where Ck is the number of class labels for meta information k ∈ {1, . . . ,K}. The contrastive loss for the meta-label k is then defined as
Lkcon = 1
2N 2N∑ i=1
1 |Pk(i)| ∑
j∈Pk(i)
− log exp
( zᵀi zj/τ )∑ a∈A(i) exp ( zᵀi za/τ
)︸ ︷︷ ︸ `ij
(2)
where Pk(i) = {j ∈ I | ykj = yki } ∪ {j(i)} are the indexes of augmented samples with same label as xi, or coming from the same original image. By minimizing this loss, the feature extractor learns to group together representations with the same class and push away those from different ones.
In medical image segmentation, encoder-decoder based networks such as U-Net [45] and its variants are widely utilized thanks to their symmetric design and appealing performance on various dataset. Such network F (·) decomposes in two parts, an encoder E(·) that summarizes the global context of an input 2D slice into a low-dimensional representation, and a decoder D(·) that takes as input the representation and gradually recovers its spatial resolution using side information such as skip connections or pooling indexes. Previous work on contrastive learning showed that pre-training both the encoder and decoder separately helped the downstream segmentation task [7]. In preliminary experiments, we found that pre-training the decoder gave marginal improvements and thus focused our method on the encoder. Specifically, we consider the features of the encoder as a single vector E(x) ∈ Rd and use a shallow non-linear projector g(·) called head to obtain the final normalized embedding zi =
g(E(xi)) ‖g(E(xi))‖ .

heading: 3.2 Self-paced learning to mitigate noisy meta-labels
text: The supervised contrastive loss of Equ. (2) can actually hurt the learning of representations in pretraining if the positive pairs are obtained with “weak” or noisy labels. For instance, if using patient ID as meta-label, we will force the encoder to cluster the representations of all 2D slices in a 3D volume, including those containing mainly background noise. Likewise, grouping together the slices in the same partition of two volumes hinders pre-training if the volumes are not fully aligned and/or their partitions cover different regions of the structure to segment.
To overcome this problem, we propose a self-paced strategy for contrastive learning which assigns an importance weight wij ∈ [0, 1] to the specific loss of each positive pair (i, j), defined as lij in Equ. (2). A self-paced regularization term Rγ(wij), controlled by the learning pace parameter γ, is added to give a greater importance (i.e., larger wij) to pairs that are more confident (i.e., smaller `ij), and vice-versa. The learning pace γ is increased over training so that high-confidence pairs are considered in the beginning and then less-confident ones are gradually added as training progresses. We achieve this by defining the following self-paced contrastive loss optimized over both encoder parameters and importance weights:
LkSP-con = 1
2N 2N∑ i=1
1 |Pk(i)| ∑
j∈Pk(i)
wij `ij + Rγ(wij) (3)
2We omit the nonlinear projector head for the sake of simplification.
5 Following standard SPL approaches [23], we define the regularizer Rγ such that the weights are monotone decreasing with respect to the loss lij (i.e., harder examples are given less importance) and monotone increasing with respect to the learning pace (i.e., a larger γ increases the weights). In this work, we consider two regularizer functions, based on hard thresholding and linear imputation:
Rhardγ (wij) = −γ wij ; Rlinearγ (wij) = γ ( 1 2w 2 ij − wij ) . (4)
Optimization process We minimize the loss in Equ. (3) by optimizing alternatively with respect to the encoder parameters θE or importance weights wij , while keeping the other fixed. With fixed wij , we update θE via stochastic gradient descent where the gradient is given by:
∇θELkSP-con = 1
2N 2N∑ i=1
1 |Pk(i)| ∑
j∈Pk(i)
wij ∇θE `ij . (5)
As can be seen, the gradient of low-confidence pairs (i, j) will be scaled down by their weight wij , thus these pairs will contribute less to the learning. Then, given a fixed θE we compute the optimal weights w∗ij by solving the following problem:
w∗ij = argmin wij∈[0,1] wij `ij + Rγ(wij) (6)
The following proposition gives the optimal solution for the hard and linear SPL regularization strategies.
Proposition 1. Given the definitions of Rhardγ and Rlinearγ in Equ. (4), the closed-form solutions to Equ. (6) are given by
whardij = { 1, if `ij ≤ γ 0, else ; w linear ij = max ( 1− 1 γ `ij , 0 ) . (7)
Proof. For the hard regularizer Rhardγ the problem becomes
min wij∈[0,1]
wij `ij − γ wij = (`ij − γ)wij (8)
If `ij − γ ≥ 0, since we are minimizing, the optimum is obviously wij = 0. Else, if `ij − γ < 0, the minimum is achieved for wij = 1. Combining these two results gives the hard threshold of Equ. (7).
A similar approach is used for the linear regularizer Rhardγ . In this case, the problem to solve is
min wij∈[0,1]
wij `ij + γ ( 1 2w 2 ij − wij ) = γ2w 2 ij + (`ij − γ)wij (9)
If lij ≥ γ, since wij ≥ 0, the minimum is reached for wij = 0. Else, if `ij < γ, we find the optimum by deriving the function w.r.t. wij and setting the result to zero, giving
wij = 1 − 1
γ `ij . (10)
Since both γ and `ij are non-negative, we have that wij ∈ [0, 1], hence it is a valid solution. Considering both cases simultaneously, we therefore get the linear rule of Equ. (7).
These update rules in Equ. (7) can be explained intuitively. For a given γ, the hard threshold rule only considers confident pairs with `ij ≤ γ and ignores the others. In contrast, the linear rule weighs each pair proportionally to γ and the inverse of `ij , emphasizing more confident ones.
Selecting the learning pace parameter One of the main challenges in self-paced learning methods is selecting the learning pace parameter γ. If γ is too small, all pairs will be ignored and there will be no learning. Conversely, if γ is too large, all pairs will be considered regardless of their confidence, which corresponds to having no self-paced learning. The following proposition provides insights on how to set this parameter during training. Proposition 2. The loss `ij related to a given pair (i, j) in the SPL objective of Equ. (3) is bounded by log 2(N−1)− 2/τ ≤ `ij ≤ log 2N + 2/τ , where N is the batch size.
6 Proof. We start by rewriting lij equivalently as
`ij = log
∑ a∈A(i) exp ( zᵀi za/τ ) exp ( zᵀi zj/τ ) = log(1 + ∑ a∈A(i)\j exp ( zᵀi za/τ ) exp ( zᵀi zj/τ )) (11) Since the representation vectors zi are L2-normalized, their dot product is a cosine similarity falling in the range [−1, 1]. To minimize lij , we then need to minimize the dot product in the numerator inside the sum and maximize the one in the denominator. Using |A(i) \ j| = 2N−2, we get
`minij = log ( 1 + (2N−2) e −1/τ
e1/τ
) = log ( 1 + 2(N−1)e−2/τ ) ≥ log 2(N−1) − 2/τ. (12)
Similarly, we maximize `ij by doing the opposite:
`maxij = log ( 1 + (2N−2) e 1/τ
e−1/τ
) = log ( 1 + 2(N−1)e2/τ ) ≤ log 2N + 2/τ. (13)
This proposition tell us that using γ = `maxij guarantees that all pairs are used in the loss, for both the hard and the linear SPL regularizers. Additionally, when using the hard regularizer Rhardγ , γ = ` min ij is the minimum learning pace so that at least one pair can be selected.
The complete SPL loss To exploit the information in all available meta-labels, our final loss combines the contrastive losses LkSP-con for meta-labels k = 1, . . . ,K:
LSP-con = K∑ k=1 λk LkSP-con (14)
Here, λk ≥ 0 is a coefficient controlling the relative importance of the kth meta-label in the final loss, which is determined by grid search on a separate validation set.

heading: 3.3 Semi-supervised segmentation with contrastive learning
text: In previous work [7], contrastive learning has mostly been used for pre-training the model. Here, we show that it can further boost results in a semi-supervised setting, where training is performed with a limited set of samples. In this setting, in addition to the unlabeled images Du, a small amount of pixelwise-annotated images Dl are also available. To incorporate the knowledge from meta information in a semi-supervised setting, we modify our self-paced contrastive loss as
Lsemi-sup = Lsup + λreg Lreg + λSP LSP-con, (15)
where Lsup is the loss computed on labeled data (cross-entropy loss in our work), Lreg is the regularization loss normally used in semi-supervised approaches (in our experiments we use Mean Teacher) and LSP-con is our self-paced contrastive loss. Last, λreg and λsp are weights balancing the different loss terms which are determined by grid search.

heading: 4 Experimental setup
text: To assess the performance of the proposed self-paced contrastive learning, we carry out extensive experiments on five benchmark datasets with different experimental settings. In this section, we briefly describe these datasets and give implementation details for our method. For further information, the reader can refer to the Supplementary Material.

heading: 4.1 Datasets
text: Five clinically-relevant benchmark datasets for medical image segmentation are used for our experiments: the Automated Cardiac Diagnosis Challenge (ACDC) dataset [3], the Prostate MR Image Segmentation 2012 Challenge (PROMISE12) dataset [29], and Multi-Modality Whole Heart Segmentation Challenge (MMWHS) dataset [64], as well as the Hippocampus and Spleen segmentation
7 datasets from [1]. These datasets contain different anatomic structures and present different acquisition resolutions. For the contrastive loss, we exploit meta-labels on slice position and patient identity. Additionally, for ACDC, we consider the cardiac phase (i.e., systole or diastole) as a third source of meta-data. For all datasets, we split images into training, validation and test sets, which remain unchanged during all experiments. We train the model with only a few scans of the dataset as labeled data (the rest of the data is used without annotations as in a semi-supervised setting) and report results in terms of 3D DSC metric [4] on the test set. Details on the training set split, data pre-processing, augmentation methods and evaluation metrics can be found in the Supplementary Material.
For all datasets, we report the segmentation performance by varying the number of labeled scans across experiments. For the ACDC dataset, this number ranges from 1 to 4, representing 0.5% to 2% of all available data. For PROMISE12, we use 3 to 7 scans, representing 6% to 14% of the whole data. For MMWHS, we use 1 and 2 annotated scans, corresponding to 10% and 20% of the training data. We use 1 to 4 scans as annotated data for the Hippocampus dataset, representing 0.5% to 0.2% of the whole data, and 2 to 4 scans for the Spleen dataset, which corresponds to 5.7% to 11.4% of the whole available training data. Note that once randomly selected, those labeled volumes are fixed across the different experiments. Selecting labeled scans per experiment yielded significant variances (up to 11.25% in term of 3D DSC), as shown in the Supplementary Material. We include the segmentation results for both Hippocampus and Spleen datasets in the Supplementary Material.

heading: 4.2 Network architecture and optimization parameters
text: We use PyTorch [39] as our training framework and, following [7], employ the U-Net architecture [45] as our segmentation network. This 2D-based networks often works well for data with anisotropic acquisition resolutions. Moreover, it has a lower computational cost and require less GPU memory than its 3D counterparts. Network parameters are optimized using stochastic gradient descent (SGD) with a RAdam optimizer [30]. We provide the detailed training hyper-parameters in the Suppl. Material. For the pre-training process, we obtain representations by projecting the encoder’s output to a vector of size 256, using a simple MLP network with one hidden layer and LeaklyReLU activation function, following [9]. Our proposed self-paced contrastive learning objective, defined in Equ. (3), involves a learning pace parameter γ set as
γ = γstart + (γend − γstart)× ( cur_epoch max_epoch )p (16)
where γstart, γend are hyper-parameters controlling the importance weights in the beginning and the end of training, and cur_epoch, max_epoch are the current training epoch and the total number of training epochs respectively. p controls how fast γ increases during the optimization procedure.

heading: 5 Results
text: In this section, we first compare the hard and linear regularization strategy for SPL on ACDC. Then, we evaluate all components of our method in a comprehensive ablation study with a reduced set of training data on the different datasets. Finally, we compare our method with the most promising approaches for semantic segmentation in medical imaging, with reduced training data.

heading: 5.1 Hard vs. linear self-paced regularization
text: Table 1 reports the validation 3D DSC score for the hard and linear SPL, while training with different p in Equ. (16), and different numbers of annotated scans on the ACDC dataset. We observe that both SPL strategies (Rhardγ and R linear γ ) effectively help improve performance, however the linear strategy always leads to a higher improvement. This is because the hard strategy only employs binary weights, i.e., wij ∈ {0, 1}, whereas the linear strategy gradually increases wij and therefore provides a smoother optimization.
In Fig. 1 (a), we plot the value of γ over epochs for different values of p, and show in (b) the corresponding expectation of wij for all positive pairs. We observe that, for a large p, γ tends to be small for most of the training and mainly increases in the very end of the process, resulting in small wij for positive pairs. In contrast, when p = 1/2, we see a rapid increase of weights wij during training, which results in higher segmentation scores. This observation is inline with the findings from [44] and [42] on different tasks, where raising rapidly the self-paced learning rate in the first
8
half of the training benefits the generalization performance. In the Suppl. Material, we also present a concrete evaluation of wij for three different scans during model optimization, corresponding to the four orange star markers in Fig. 1 (b). Since we found that Rlinearγ strategy works better than R hard γ , we will use Rlinearγ for all following experiments.

heading: 5.2 Ablation study
text: Table 2 summarizes the 3D DSC performance on test set for three datasets (ACDC, PROMISE12 and MMWHS) with very limited labeled data. At the top of the table, we report the number of labeled scans used and, for every result, also give in parenthesis the standard deviation computed with 3 different random seeds for parameter initialization. In the second and third columns of the table, we provide the loss used for the pre-training, if any, and the loss for the downstream training.
Upper and lower bounds We present results for a Baseline which uses only the annotated scans with cross-entropy as standard supervised loss Lsup, and for Full Supervision where the same loss is used with all available data and associated annotations (175 for ACDC, 40 for Prostate and 10 for MMWHS). These two rows represent lower and upper bounds on the expected performance of the different variants of our approach.
Unsupervised contrastive loss We evaluate the performance of pre-training the network encoder with Unsupervised Contrastive loss as in [8], where two augmented versions of the same image are considered as a positive pair. In all datasets, this loss improves over our baseline model, although the improvement is limited because the amount of unlabeled data available is still reduced compared to the settings of previous work on unsupervised contrastive learning [8, 20, 11, 10, 62]. We also add our self-paced learning strategy on top of this contrastive loss, and call this modified model Unsupervised Contrastive + SP. As meta-labels may be noisy, performance is increased in almost all experiments, especially when fewer labels are available.
Pre-training contrastive loss on meta-data We report the performance of a model pre-trained with a Contrastive loss on meta-labels. The meta-labels are 3D slice location L1con, patient identity L2con and cardiac phase L3con (only for ACDC). We find that that slice position always gives the highest accuracy among all meta-labels, and largely outperforms the unsupervised contrastive loss. While all meta-labels increase performance compared to unsupervised contrastive loss, their combination leads to the best results in most cases.
Pre-training self-paced contrastive loss on meta-data Next, we evaluate the model pre-trained with a Self-Paced Contrastive loss on meta-labels (SP-Con (pre-train)). As with the unsupervised contrastive loss, the self-paced approach also successfully improves the segmentation quality compared to treating all positive pairs equally.
Semi-supervised We report the performance of a model without any pre-training, but using the unlabeled data during training with our proposed self-paced contrastive loss (SP-Con (semi-sup)). It can be seen that performance is inferior to Contrastive pre-training on meta-data, however the improvement is still quite relevant and, in most cases, superior to unsupervised pre-training.
Pre-trained and semi-supervised Our next subsection reports results for the combination of Self-Paced Contrastive learning used for both pre-training and semi-supervised training (SP-Con
9
(both)). Although the loss is the same, its use during pre-training and as additional regularization in a semi-supervised setting brings additional improvements.
Pre-trained and semi-supervised with Mean Teacher We then evaluate the model using both pre-training and semi-supervised (as the previous setting) but with an additional Mean-Teacher for semi-supervision (SP-Con (both) + Mean-Teacher). By combining our approach with a simple Mean-Teacher method, our results on all datasets are further boosted, approaching the performance of fully supervised training but using a very low number of annotated scans. This is the model that is used in the comparison with the state-of-the-art.

heading: 5.3 Comparison with the state-of-the-art
text: We compare our method with other approaches that aim to improve training with few annotated images/scans. Table 3 presents results in terms of 3D DSC score for approaches based on data augmentation [58], pre-training the weights on both encoder and decoder of the model [7] and various semi-supervised learning methods [50, 60, 43, 41]. As with the ablation study, we report results for the ACDC, PROMISE12 and MMWHS datasets. A detailed explanation of the experimental setup of each method and results for the other two datasets can be found in the Supplementary Material.
To have a fair comparison, for all methods, we used grid search on the validation set to tune the hyper-parameters. For most methods, the improvement with respect to the baseline trained with only the supervised loss is quite limited and varies depending on the dataset and the number of annotated scans used. For instance, Adversarial training performs quite well on the PROMISE12 dataset (scans in this dataset exhibit more variability in terms of intensity contrast), but not so well
10
on ACDC. Likewise, Mean-Teacher does not perform well for 1 or 2 annotated scans in ACDC but, when increasing the scans to 4, it outperforms most of the other methods. The global and local Contrastive loss using meta-data manages to obtain an excellent improvement on all datasets. However, our approach still yields substantial improvements with respect to that method. This is due to our proposed self-paced learning strategy, as well as the combined use of the contrastive loss for pre-training and semi-supervised learning.

heading: 6 Discussion and conclusion
text: In this paper, we proposed a technique based on contrastive loss with meta-labels that can highly improve the performance of a medical image segmentation model when training data is scarce. It was shown that, with a reduced amount of unlabeled images, unsupervised contrastive loss is not very effective. Instead, in the context of medical images, additional meta-data is freely available and, if properly used, can greatly boost performance. We presented results on five well-known medical image datasets and have shown that the accuracy of the contrastive loss with meta-labels can be boosted by the use of self-paced learning. Our self-paced contrastive learning method can be used during pre-training as well as a regularization loss during semi-supervised training, and the combination of the two can further boosts results. Finally, we have compared our approach with the state-of-the-art in semi-supervised learning, and have shown that the simple combination of our approach with multiple meta-data and a simple semi-supervised approach as Mean Teacher is more effective than previous approaches. While using a few scans, our method can approach fully supervised training.
Social impact and limitations The proposed method can have an effective and practical impact in terms of medical imaging analysis in hospitals and health centers. As shown in our experiments, it produces an accurate medical image segmentation with a very reduced set of annotated data. This has the potential of helping radiologists and other clinicians using medical images, which can in turn contribute to a better diagnosis and reduced costs. While our empirical evaluation has shown excellent results with very limited data, using fewer annotated images also increases chances of over-fitting potential outliers in the data that may lead to erroneous or misleading results. A further study on the reliability of medical image segmentation with reduced images is therefore recommended.

heading: 7 Acknowledgment
text: We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC Grant No. RGPIN-2018- 04825) and Fonds de recherche du Québec – Nature et technologies (FRQNT Grant No. B2X 276565). This research was also enabled in part by support provided by Calcul Québec (www.calculquebec.ca) and Compute Canada (www.computecanada.ca).

Review summaries:
The authors propose using “metal labels” in pertaining image encoders on unlabeled datasets. They use the metal-labels in pertaining the encoder, and also as an auxiliary task in training. They show their method works on multiple medical datasets.

Annotation in the clinical dataset is hard to prepare, so it is important if self-learning algorithms can be applied to medical data which can do few-shot learning. In this paper, authors put forward a method to do segmentation which makes use of pretraining with contrastive learning with metadata, and then better fine tuning with self-paced learning. The second step is necessary to mitigate meta data noise. This paper shows experiments on three public datasets. With a few annotated data, the algorithms still show comparable performance with the fully supervised segmentation algorithm.