Title: Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: Long-tailed Medical Image Segmentation

Contrastive Learning

Variance Reduction

Imbalanced Learning

Semi-Supervised Learning
Abstract: For medical image segmentation, contrastive learning is the dominant practice to improve the quality of visual representations by contrasting semantically similar and dissimilar pairs of samples. This is enabled by the observation that without accessing ground truth labels, negative examples with truly dissimilar anatomical features, if sampled, can significantly improve the performance. In reality, however, these samples may come from similar anatomical features and the models may struggle to distinguish the minority tail-class samples, making the tail classes more prone to misclassification, both of which typically lead to model collapse. In this paper, we propose $\texttt{ARCO}$, a semi-supervised contrastive learning (CL) framework with stratified group theory for medical image segmentation. In particular, we first propose building $\texttt{ARCO}$ through the concept of variance-reduced estimation, and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks with extremely limited labels. Furthermore, we theoretically prove these sampling techniques are universal in variance reduction. Finally, we experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings, and our methods consistently outperform state-of-the-art semi-supervised methods. Additionally, we augment the CL frameworks with these sampling techniques and demonstrate significant gains over previous methods. We believe our work is an important step towards semi-supervised medical image segmentation by quantifying the limitation of current self-supervision objectives for accomplishing such challenging safety-critical tasks.

Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective

Abstract

For medical image segmentation, contrastive learning is the dominant practice to improve the quality of visual representations by contrasting semantically similar and dissimilar pairs of samples. This is enabled by the observation that without accessing ground truth labels, negative examples with truly dissimilar anatomical features, if sampled, can significantly improve the performance. In reality, however, these samples may come from similar anatomical regions and the models may struggle to distinguish the minority tail-class samples, making the tail classes more prone to misclassification, both of which typically lead to model collapse. In this paper, we propose ARCO , a semi-supervised contrastive learning (CL) framework with stratified group theory for medical image segmentation. In particular, we first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks with extremely limited labels. Furthermore, we theoretically prove these sampling techniques are universal in variance reduction. Finally, we experimentally validate our approaches on eight benchmarks, i.e. , five 2D/3D medical and three semantic segmentation datasets, with different label settings, and our methods consistently outperform state-of-the-art semi-supervised methods. Additionally, we augment the CL frameworks with these sampling techniques and demonstrate significant gains over previous methods. We believe our work is an important step towards semi-supervised medical image segmentation by quantifying the limitation of current self-supervision objectives for accomplishing such challenging safety-critical tasks. 1

1 Introduction

Model robustness and label efficiency are two highly desirable perspectives when it comes to building reliable medical segmentation models. In the context of medical image analysis, a model is said to be robust if (1) it has a high segmentation quality with only using extremely limited labels in longtailed medical data; (2) and fast convergence speed [1, 2, 3]. The success of traditional supervised learning depends on training deep networks on a large amount of labeled data, but this improved segmentation/model robustness often comes at the cost of annotations and clinical expertise [4, 5, 6]. Therefore, it is difficult to adopt these models in real-world clinical applications.

Recently, a significant amount of research efforts [7, 8, 9, 10] have resorted to unsupervised or semi-supervised learning techniques for improving the segmentation robustness. One of the most effective methods is contrastive learning (CL) [11, 12, 13, 14]. It aims to learn useful representations

by contrasting semantically similar (positive) and dissimilar (negative) pairs of data points sampled from the massive unlabeled data. These methods fit particularly well with real-world clinical scenarios as we assume only access to a large amount of unlabelled data coupled with extremely limited labels. However, pixel-level contrastive learning with medical image segmentation is quite impractical since sampling all pixels can be extremely time-consuming and computationally expensive [15]. Fortunately, recent studies [16, 17] provide a remedy by leveraging the popular strategy of bootstrapping, which first actively samples a sparse set of pixel-level representation (queries), and then optimize the contrastive objective by pulling them to be close to the class mean averaged across all representations in this class (positive keys), and simultaneously pushing apart those representations from other class (negative keys). The demonstrated imbalancedness and diversity across various medical image datasets, as echoed in [18], show the positive sign of utilizing the massive unlabeled data with extremely limited annotations while maintaining the impressive segmentation performance compared to supervised counterparts. Meanwhile, it can lead to substantial memory/computation reduction when using pixel-level contrastive learning framework for medical image segmentation.

Nevertheless, in practical clinical settings, the deployed machine learning models often ask for strong robustness, which is far beyond the scope of segmentation quality for such challenging safetycritical scenarios. This leads to a more challenging requirement, which demands the models to be more robust to the collapse problems whereby all representations collapse into constant features [14, 13, 19] or only span a lower-dimensional subspace [20, 21, 22, 23], as one main cause of such fragility could be attributed to the non-smooth feature space near samples [24, 25] ( i.e. , random sampling can result in large feature variations and even annotation information alter). Thus, it is a new perspective: how to sample most informative pixels/voxels towards improving variance reduction in training semi-supervised contrastive learning models . This inspires us to propose a new hypothesis of semi-supervised CL. Specifically, when directly baking in variance-reduction sampling into semi-supervised CL frameworks for medical image segmentation, the models can further push toward state-of-the-art segmentation robustness and label efficiency.

In this paper, we present ARCO , a semi-supervised str A tifed g R oup Co ntrastive learning framework with two perspectives ( i.e. , segmentation/model robustness and label efficiency ), and with the aid of variance-reduction estimation, realize two practical solutions - Stratified Group (SG) and Stratified-Antithetic Group (SAG) - for selecting the most semantically informative pixels. ARCO is a group-based sampling method that builds a set of pixel groups and then proportionally samples from each group with respect to the class distribution. The main idea of our approach is via first partitioning the image with respect to different classes into grids with the same size, and then sampling, within the same grid, pixels semantically close to each other with high probability, with minimal additional memory footprint .

Subsequently, we show that baking ARCO into contrastive pre-training ( i.e. , MONA [17]) provides an efficient pixel-wise contrastive learning paradigm to train deep networks that perform well in long-tailed medical data. ARCO is easy to implement, being built on top of off-the-shelf pixel-level contrastive learning framework [13, 14, 26, 27, 28], and consistently improve overall segmentation quality across all label ratios and datasets ( i.e. , five 2D/3D medical and three semantic datasets).

Our theoretical analysis shows that, ARCO is more label efficient, providing practical means for computing the gradient estimator with improved variance reduction. Empirically, our approach achieves competitive results across eight 2D/3D medical and semantic segmentation benchmarks. Our proposed framework has several theoretical and practical contributions:

We propose ARCO , a new CL framework based on stratified group theory to improve the label efficiency and model robustness trade-off in CL for medical image segmentation. We show that incorporating ARCO coupled with two special sampling methods, Stratified Group and Stratified-Antithetic Group, into the models provides an efficient learning paradigm to train deep networks that perform well in those long-tail clinical scenarios.

To our best knowledge, we are the first work to show the benefit of certain variancereduction techniques in CL for medical image segmentation. We demonstrate the unexplored advantage of the refined gradient estimator in handling long-tailed medical image data.

We conduct extensive experiments to validate the effectiveness of our proposed method using a variety of datasets, network architectures, and different label ratios. For segmentation robustness/accuracy, we show that our proposed method by demonstrating superior segmentation accuracy (up to 11.08% absolute improvements in Dice). For label efficiency, our

method trained with different labeled ratios - consistently achieves competitive performance improvements across all eight 2D/3D medical and semantic segmentation benchmarks.

Theoretical analysis of ARCO shows improved variance reduction with optimization guarantee. We further demonstrate the intriguing property of ARCO across the different pixel-level contrastive learning frameworks.

2 Related work

Medical Image Segmentation. Contemporary medical image segmentation approaches typically build upon fully convolutional networks (FCN) [29] or UNet [30], which formulates the task as a dense classification problem. In general, current medical image segmentation methods can be cast into two sets: network design and optimization strategy. One is to optimize segmentation network design for improving feature representations through dilated/atrous/deformable convolutions [31, 32, 33], pyramid pooling [34, 35, 36], and attention mechanisms [37, 38, 39]. Most recent works [40, 41, 6] reformulates the task as a sequence-to-sequence prediction task by using the vision transformer (ViT) architecture [42, 43]. The other is to improve optimization strategies, by designing loss function to better address class imbalance [44] or refining uncertain pixels from high-frequency regions improving the segmentation quality [45, 46, 47, 48, 49]. In contrast, we take a leap further to a more practical clinical scenario by leveraging the massive unlabeled data with extremely limited labels in the learning stage. Moreover, we focus on building model-agnostic , label-efficiency framework to improve segmentation quality by providing additional supervision on the most confusing pixels for each class. In this work, we question how medical segmentation models behave under such imbalanced class distributions and whether they can perform well in those challenging scenarios through sampling methods.

Semi-Supervised Learning (SSL). SSL aims to train models with a combination of labeled, weaklylabeled and unlabelled data. In recent years, there has been a surge of work on semi-supervised medical segmentation [8, 9, 50, 48, 16, 51, 52, 17, 10, 53, 54], which makes it hard to present a complete overview here. We therefore only outline some key milestones related to this study. In general, it can be roughly categorized into two groups: (1) Consistency regularization was first proposed by [55], which aims to impose consistency corresponding to different perturbations into the training, such as consistency regularization [56, 57], pi-model [58], and mean-teacher [59, 60]. (2) Self-training was initially proposed in [61], which aims at using a model's predictions to obtain noisy pseudo-labels for performance boosts with minimal human labor, such as pseudo-labeling [7, 62], model uncertainty [8, 63], confidence estimation [64, 65, 66], and noisy student [67]. These methods usually lead to competitive performance but fail to prevent collapse due to class imbalanceness. In this work, we focus on semi-supervised medical segmentation with extremely limited labels since the medical image data is extremely diverse and often long-tail distributed over anatomical classes. We speculate that a good medical segmentation model is expected to distinguish the minority tail-class samples and hence achieve better performance under additional supervision on hard pixels.

Contrastive Self-Supervised Learning. Self-supervised representation learning is a subclass of unsupervised learning, but with the critical distinction that it incorporates 'inherent' supervision from the input data itself [68]. The primary aim of self-supervised representation learning is to enable the model to learn the most useful representations from the large amount of unlabelled data for various downstream tasks. Self-supervised learning typically relies on pretext tasks, including predictive [69, 70, 71], contextual [72, 73], and generative [74] or reconstructive [75] tasks.

Among them, contrastive learning is considered as a popular approach for self-supervised representation learning by pulling the representations of similar instances closer and representations of dissimilar instances further apart in the learned feature space [11, 12, 13, 14]. The past five years have seen tremendous progress related to CL in medical image segmentation [50, 23, 76, 48, 16, 17, 77], and it becomes increasingly important to improve representation in label-scarcity scenarios. The key idea in CL [11, 12, 13, 14] is to learn representations from unlabeled data that obey similarity constraints by pulling augmented views of the same samples closer in a representation space, and pushing apart augmented views of different samples. This is typically achieved by encoding a view of a data into a single global feature vector. However, the global representation is sufficient for simple tasks like image classification, but does not necessarily achieve decent performance, especially for more challenging dense prediction tasks. On the other hand, several works on dense contrastive

Global Instance

Discrimination Loss

Global Instance

Discrimination Loss

𝐹

!

𝐹

"

𝐹

"

Local Instance

Discrimination Loss

Local Instance

Discrimination Loss

Supervised

Loss

Ground-truth

Unlabeled Image

L

Labeled Images

Augment.

from unlabeled set

U

U

𝐹

!

𝐹

"

Supervised

Loss

Unlabeled Image

L

Labeled Images

U

EMA

Shared

Weights

EMA

Ground-truth

Unsupervised Loss

Nearest Neighbor

Loss

(a) Relational Semi-supervised Pre-training

(b) Anatomical Contrastive Reconstruction Fine-tuning

Contrastive Loss

Augment.

Augment.

Augment.

Augment.

learning [50, 23], aim at providing additional supervision to capturing intrinsic spatial structure and fine-grained anatomical correspondence, while these methods may suffer from class imbalance issues. Particularly, very recent work [16, 17] for the first time demonstrates the imbalancedness phenomenon can be mitigated by performing contrastive learning yet lacking stability. By contrast, a key motivation of our work is to bridge the connection between model robustness and label efficiency, which we believe is an important and under-explored area. We hence focus on variance-reduced estimation in medical image segmentation, and show that certain variance-reduction techniques can help provide more efficient approaches or alternative solutions for handling collapse issues, and improving model robustness in terms of accuracy and stability. To the best of our knowledge, we are the first to provide a theoretical guarantee of robustness by using certain variance-reduction techniques.

3 Methodology

In this section we set-up our semi-supervised medical segmentation problem, introduce key definitions and notations and formulate an approach to incorporate stratified group theory. Then, we discuss how our proposed ARCO can directly bake in two perspectives into deep neural networks: (1) model robustness , and (2) label efficiency .

3.1 Preliminaries and setup

Problem Definition. In this paper, we consider the multi-class medical image segmentation problem. Specifically, given a medical image dataset ( X , Y ) , we wish to automatically learn a segmentator, which assigns each pixel to their corresponding K -class segmentation labels. Let us denote x as the input sample of the student and teacher networks F ( · ) 2 , consisting of an encoder E and a decoder D , and F is parameterized by weights θ s and θ t .

Background. Contrastive learning aims to learn effective representations by pulling semantically close neighbors together and pushing apart other non-neighbors [11]. Among various popular contrastive learning frameworks, MONA [17] is easy-to-implement while yielding the state-of-the-art performance for semi-supervised medical image segmentation so far. The main idea of MONA is to discover diverse views ( i.e. , augmented/mined views) whose anatomical feature responses are homogeneous within the same or different occurrences of the same class type , while at the same time being distinctive for different class types .

Hereinafter, we are interested in showing that certain variance-reduction techniques coupled with CL frameworks are particularly beneficial in long-tail pixel/voxel-level segmentation tasks with extremely limited labels. We hence build our ARCO as a simplification of the MONA pipeline [17], without additional complex augmentation strategies, for deriving the model robustness and label efficiency proprieties of our medical segmentation model. Figure 1 overviews the high-level workflow of the proposed ARCO framework. Training ARCO involves a two-phase training procedure: (1) relational semi-supervised pre-training, and (2) anatomical contrastive fine-tuning. To make the discussion self-contained, we defer the full details of ARCO to the appendix E.

3.2 Motivation and Challenges

Intuitively, the contrastive loss will learn generalizable, balanced and diverse representations for downstream medical segmentation tasks if the positive and negative pairs correspond to the desired latent anatomical classes [50, 16, 17]. Yet, one critical constraint in real-world clinical scenarios is severe memory bottlenecks [15, 16]. To address this issue, current pixel-level CL approaches [16, 17] for high-resolution medical images devise their aggregation rules by unitary simulators , i.e. , Naïve Sampling (NS), that determines the empirical estimate from all available pixels. Despite the blessing of large learning capacity, such aggregation rules are unreliable 'black boxes'. It is never well understood which rule existing CL models should use for improved model robustness and label efficiency ; nor is it easy to compare different models and assess the model performance. Moreover, unitary simulators, especially naïve sampling, often incur high variances and fail to identify semantically similar pixels [24], limiting CL stability. As demonstrated in Figure 3, regions of similar anatomical features should be grouped together in the original medical images, resulting in corresponding plateau regions in the visualization of the loss landscape. This is consistent with the observations uncovered by the recent empirical findings [79, 80].

(1) Naïve Sampling

(NS)

(2) Stratified Group (SG)

Sampling

(3) Stratified-Antithetic Group

(SAG) Sampling

If we take a unified mathematical perspective, the execution of simulation can be represented either through an adaptive rule, or by a unitary simulation. To tackle the two critical issues, we look back at adaptive rules. We hence propose two straightforward yet effective techniques - Stratified Group (SG) and StratifiedAntithetic Group (SAG) - to mitigate the undesirable high-variance limitation, and turn to the following idea of sampling the most representative pixels from groups of semantically similar pixels. In particu- lar, our proposed solution is based on stratified group simulation to adaptively characterize anatomical regions found on different medical images. This characterization is succinct, and regions with the same anatomical properties within different medical images are identifiable. In practice, we first partition the image with respect to different classes into grids with the same size, and then sampling, within the same grid, the pixels semantically close to each other with high probability, with minimal additional memory footprint (Figure 2).

In what follows, we will theoretically demonstrate the important properties of such techniques ( i.e. , SG and SAG), especially in reduced variance and unbiasedness. Here the reduced variance implies more robust gradient estimates in the backpropagation, and leads to faster and stabler training in theory, as corroborated by our experiments (Section 4). Empirically, we will demonstrate many practical benefits of reduced variances including improved model robustness, i.e. , faster convergence and better segmentation quality, through mitigating the collapse issue.

3.3 Stratified Group Sampling

To be consistent with the previous notation, we denote an arbitrary image from the given medical image dataset as x ∈ X , and P as the set of pixels. For arbitrary function h : X ×P → R , we define

0.8

0.6

0.4

0.4

0.2

0.2

0.0

(1) ACDC

(2) LiTS

(3)MMWHS

8888

0.8

0.6

0.4

0.2

0.0

(4) LA

(5) MP-MRI

the aggregation function H 3 as:

As a large cardinality of P prevents efficient direct computation of H , an immediate approach is to compute H ( x ) by first sampling a subset of pixels D⊆P according to certain sampling strategy, and then computing ̂ H ( x ; D )= ∑ p ∈D h ( x ; p ) / |D| . SG sampling achieves this by first decomposing the pixels into M disjoint groups P m satisfying ∪ M m =1 P m = P , and then sampling D m ⊆ P m so that D = ∪ M m =1 D m . The SG sampling can then be written as:

SAG, built upon SG, adopts a similar form, except for an additionally enforced symmetry on D m : ∀ m, ∃ c m ∈ P m , such that for any p ∈ D m ,

Here c m denotes the center of the group P m 4 . The implementation of SG and SAG involves two steps: (1) to create groups {P m } M m =1 , and (2) to generate each D m ⊆ P m . For the latter, we consider independent sampling within and between groups, i.e. , D m ⊥ ⊥ D m ′ for m = m ′ , and p ⊥ ⊥ p ′ ∀ p , p ′ ∈ D m , where the variance of SG sampling is as follows.

̸

Lemma 3.1. Suppose in SG sampling, for each m , D m is sampled from P m with sampling variance σ 2 m and sample size |D m | = n m . Then the variance satisfies Var[ ̂ H SG ] = ∑ M m =1 σ 2 m n m /n , and SAG with the same sample size satisfies Var[ ̂ H SAG ] ≤ 2 Var[ ̂ H SG ] .

To ensure the unbiasedness property, we adopt the setting of proportional group sizes [85, 86], i.e. , |D m | ∝ |P m | for all m . It turns out that such setting also enjoys the variance-reduction property.

Theorem 3.2 (Unbiasedness and Variance of SG) . SG with proportional group sizes is unbiased, and has a variance no larger than that of NS. That is: E [ ̂ H SG ( x )] = H ( x ) , and

The last term is the intra-group variance, which captures the discrepancy between the pixel groups {P m } M m =1 . Theorem 3.2 guarantees that the variance of SG is no larger than that of NS, and SG

has strictly less variance than NS as long as all the pixel groups do not share an equal mean over h ( x ; p ) , which is almost-sure in medical images (See Figure 3). For SAG, Lemma 3.1 guarantees its variance is of the same magnitude as that of SG, and at worst differs by a factor of 2. Since the pixel/voxel-level contrastive loss L contrast is an aggregation function over pixels by definition (E.2), it benefits from the variance-deduction property of SG/SAG. In Section 4.1, we will see that such variance reduction allows ARCO to achieve better segmentation accuracy, especially along the boundary of the anatomical regions (Figure 4).

Training Convergence. We further demonstrate the benefit of variance reduction estimation in terms of training stability. Specifically, leveraging techniques from standard optimization theory [87, 88, 89], we can show that variance-reduced gradient estimator through SG sampling leads to faster training convergence. Suppose we have a loss function L ( θ ) with the model parameter θ , and use stochastic gradient descent (SGD) as the optimizer. A gradient estimate g ( θ ) ≈ ∇L ( θ ) is computed at each iteration. It is well-known that the convergence of SGD depends on the quality of the estimate g ( θ ) [87]. Specifically, we make the common assumptions that the loss function is smooth and the gradient estimate has bounded variance (More details in Appendix A.3), which can be formulated as below:

Under these two assumptions, the average expected gradient norm of the learned parameter satisfies the following:

For general non-convex loss function, the above implies convergence to some local minimum. Importantly, the slow rate ( σ g / √ T ) depends on standard deviation σ g , indicating a faster convergence can indeed be achieved with a more accurate gradient estimate. This indicates our proposed sampling techniques demonstrate universality in variance reduction, as they can be applied to a wide range of scenarios that involve pixel/voxel-level sampling (See Appendix A.3). In Figure 5, we observe that using SG enables faster loss decay with smaller error bar, showing that it outperforms other methods in both convergence speed and stability. See Section 4.2 and Appendix A.3 for more details.

4 Experiments

In this section, we present experimental results to validate our proposed methods across various datasets and different label ratios in Appendix B. We use 2D UNet [30] or 3D VNet [78] as our backbones. Further implementation details are discussed in Appendix C. 5

4.1 Main Results

In this subsection, we first examine whether our proposed ARCO can generalize well across various datasets and label ratios. Then, we investigate to what extent ARCO coupled with two samplers can realize two essential properties: (1) model robustness ; and (2) label efficiency . The quantitative results for all the compared methods on eight popular datasets: (1) Medical image segmentation tasks: three 2D benchmarks ( i.e. , ACDC [81], LiTS [82], MMWHS [83]), two 3D benchmarks ( i.e. , LA [84], in-house MP-MRI) under various label ratios ( i.e. , 1%, 5%, 10%) are collected in Table 1, Table 5 (Appendix G), Table 6 (Appendix H), and Table 9 (Appendix I and J), respectively; (2) General computer vision tasks: To further validate the effectiveness, we experiment on three popular segmentation benchmarks ( i.e. , Cityscapes [97], Pascal VOC 2012 [98], indoor scene segmentation dataset - SUN RGB-D [99]) in the semi-supervised full-label settings. We follow the identical setting [100] to sample labelled images to ensure that every class appears sufficiently in our three datasets, ( i.e. , CityScapes, Pascal VOC, and SUN RGB-D). The results are collected in Appendix Section K. Several consistent observations can be drawn from these extensive evaluations with eighteen segmentation networks.

❶ Superior Performance Across Datasets. We demonstrate that ARCO achieves superior performance across all datasets and label ratios. In specific, our experiments consider three 2D benchmarks

ACDC

1 Labeled (1%)

3 Labeled (5%)

7 Labeled (10%)

Method

Average

RV

Myo

LV

Average

RV

Myo

LV

Average

RV

Myo

LV

UNet-F

[30]

91.5/0.996

90.5/0.606

88.8/0.941

94.4/1.44

91.5/0.996

90.5/0.606

88.8/0.941

94.4/1.44

91.5/0.996

90.5/0.606

88.8/0.941

94.4/1.44

UNet-L

40.3/22.7

29.0/25.4

43.6/15.3

48.2/27.5

51.7/13.1

36.9/30.1

54.9/4.27

63.4/5.11

79.5/2.73

65.9/0.892

82.9/2.70

89.6/4.60

EM

[90]

43.1/18.1

38.7/23.1

42.0/12.0

48.7/19.4

59.8/5.64

44.2/11.1

63.2/3.23

71.9/2.57

75.7/2.73

68.0/0.892

76.5/2.70

82.7/4.60

CCT

[91]

48.6/19.2

38.7/28.0

49.2/14.8

57.9/17.0

59.1/10.1

44.6/19.8

63.2/6.04

69.4/4.32

75.9/3.60

67.2/2.90

77.5/3.32

82.9/4.59

DAN

[92]

48.9/17.5

45.4/19.7

41.0/8.88

60.4/23.8

56.4/15.1

47.1/21.7

58.1/11.6

63.9/11.9

76.5/3.01

75.7/2.61

73.3/3.11

80.5/3.31

URPC

[57]

43.0/21.1

38.6/22.8

41.7/14.4

48.6/26.0

58.9/8.14

50.1/12.6

60.8/4.10

65.8/7.71

83.1/1.68

77.0/0.742

82.2/

0.505

90.1/3.79

DTC

[9]

51.7/17.5

39.3/23.3

54.6/9.12

61.3/20.2

56.9/7.59

35.1/9.17

62.9/6.01

72.7/7.59

84.3/4.04

83.8/3.72

83.5/4.63

85.6/3.77

DCT

[93]

49.7/16.4

42.4/20.4

48.8/10.6

57.9/18.2

58.5/10.8

41.2/21.4

63.9/5.01

70.5/6.05

78.1/2.64

70.7/1.75

77.7/2.90

85.8/3.26

ICT

[95]

42.1/21.0

36.5/18.5

43.4/11.1

46.3/33.5

59.0/6.59

48.8/11.4

61.4/4.59

66.6/3.83

80.6/1.64

75.1/0.898

80.2/1.53

86.6/2.48

MT

[59]

42.9/15.1

32.5/21.9

46.2/8.99

50.1/14.7

58.3/11.2

39.0/21.5

58.7/7.47

77.3/4.72

80.1/2.33

75.2/1.22

79.2/2.32

86.0/3.45

UAMT

[8]

36.9/15.2

32.5/21.9

46.2/8.99

50.1/14.7

48.3/9.14

37.6/18.9

50.1/4.27

57.3/4.17

81.8/4.04

79.9/2.73

80.1/3.32

85.4/6.07

SASSNet

[94]

42.6/24.8

29.8/34.7

45.4/13.3

52.5/26.6

57.8/6.36

47.9/11.7

59.7/4.51

65.8/2.87

84.7/1.83

81.8/0.769

82.9/1.73

89.4/2.99

CPS

[62]

51.5/15.3

41.1/17.7

52.0/7.27

61.4/21.0

61.0/2.92

43.8/2.95

64.5/2.84

74.8/2.95

78.8/3.41

74.0/1.95

78.1/3.11

84.5/5.18

GCL

[50]

59.7/14.3

49.5/25.3

60.9/6.28

68.8/11.5

70.6/2.24

56.5/1.99

70.7/1.67

84.8/3.05

87.0/

0.751

86.9/

0.584

81.8/0.820

92.5/0.849

MC-Net

[96]

53.4/17.17

43.0/25.3

51.2/7.41

60.8/15.11

62.8/2.59

52.7/5.14

62.6/0.807

73.1/1.81

86.5/1.89

85.1/0.745

84.0/2.12

90.3/2.81

SS-Net

[53]

63.4/2.94

64.7/3.32

57.0/1.81

68.4/3.70

65.8/2.28

57.5/3.91

65.7/2.02

74.2/0.896

86.8/1.40

85.4/1.19

84.3/1.44

90.6/1.57

ACTION

[16]

81.0/3.45

76.9/3.09

78.4/2.07

87.5/5.17

86.6/1.20

85.2/0.734

84.7/0.909

89.8/1.97

87.2/1.47

86.1/0.976

85.7/1.11

89.7/2.33

MONA

[17]

82.6/1.43

80.2/1.57

79.9/1.10

87.8/1.43

86.9/1.07

84.7/1.01

85.4/0.731

90.6/1.48

87.7/1.33

86.9/0.687

85.7/1.70

90.5/2.14

•

ARCO-SAG

(ours)

84.9/1.47

81.7/1.98

81.9/0.903

90.9

/1.52

87.1/0.848

85.6/

0.414

85.1/0.930

90.6/

1.20

88.5/1.40

87.1/0.635

86.2/1.04

92.2/2.53

◦

ARCO-SG

(ours)

85.5

/

0.947

81.8

/

1.19

83.8

/

0.801

90.9

/

0.853

88.7

/

0.841

88.2

/0.618

85.9

/

0.673

91.9

/

1.23

89.4

/0.776

90.2

/0.701

86.5

/0.787

91.6

/

0.839

MC-Net

GCL

SS-Net

ACTION

MONA

Ground Truth

ARCO-SAG (ours)

5% Labeled Data

10% Labeled Data

Right  Ventricle (RV)

Myocardium (Myo)

Left  V entricle (L V)

1% Labeled Data

ARCO-SG (ours)

( i.e. , ACDC [81], LiTS [82], MMWHS [83]), two 3D benchmarks ( i.e. , LA [84], in-house MP-MRI), and different label ratios ( i.e. , 1%, 5%, 10%). As shown in Table 1, Table 5 (Appendix G), Table 6 (Appendix H), and Table 9 (Appendix I and J), we observe that our methods consistently outperform all the compared SSL-based methods by a considerable margin across all datasets and label ratios, which validates the superior performance of our proposed methods in both segmentation accuracy and label efficiency. For example, compared to the second-best MONA , our ARCO-SG under {1%, 5%, 10%} label ratios achieves {2.9% ↑ , 1.8% ↑ , 1.7% ↑ }, {3.3% ↑ , 1.8% ↑ , 1.8% ↑ }, {4.1% ↑ , 2.0% ↑ , 1.8% ↑ }, {0.3% ↑ , 0.3% ↑ , 0.5% ↑ }, {2.2% ↑ , 0.8% ↑ , 0.4% ↑ } in average Dice across ACDC, LiTS, and MMWHS, MP-MRI, and LA, respectively. Our ARCO-SAG achieves {84.9%, 87.1%, 88.5%}, {64.1%, 67.3%, 69.4%}, {86.1%, 88.6%, 89.3%}, {91.5%, 92.5%, 92.6%}, {73.2%, 86.9%, 89.1%} in averaged Dice across ACDC, LiTS, MMWHS, MP-MRI, and LA. These results indicate that our methods can generalize to different clinical scenarios and label ratios.

❷ Across Label Ratios and Robustified Methods. To further validate the label efficiency property of our ARCO , we evaluate our ARCO-SG and ARCO-SAG with limited labeled training data available ( e.g. , 1% and 5%). As demonstrated in Table 1, Table 5 (Appendix G), Table 6 (Appendix H), and Table 9 (Appendix I and J), our models under 5% label ratios surpass all the compared SSL methods by a significant performance margin. For example, compared to MONA , we observe our methods to

push the best segmentation accuracy higher by 0.3% ∼ 2.0% in Dice on ACDC, LiTS, and MMWHS, MP-MRI, and LA, respectively. For example, the best segmentation accuracy on MMWHS rises from 87.3% to 89.3%. This suggests that our SSL-based approaches - without compromising the best achievable segmentation results - robustly improve performance using very limited labels, and further lead to a much-improved trade-off between SSL schemes and supervised learning schemes by avoiding a large amount of labeled data.

Similar to our results under 5% label ratio, our ARCO-SG and ARCO-SAG trained with 1% label ratio demonstrate sufficient performance boost compared to MONA by an especially significant margin, with up to 0.3% ∼ 4.1% relative improvement in Dice. Taking the extremely limited label ratio ( i.e. , 1%) as an indicator: (1) on 3D LA, ARCO-SG achieves 2.2 % higher average Dice, and 6.64 lower average ASD than the second best MONA ; (2) on LiTS, ARCO-SG achieves 3.3 % higher average Dice, and 4.7 lower average ASD than the second best MONA ; and (3) considering the more challenging clinical scenarios ( i.e. , 7 anatomical classes), ARCO-SG achieves 5.1% higher average Dice, and 2.26 lower average ASD than the second-best MONA on MMWHS. It highlights the superior performance of ARCO is not only from improved label efficiency but also credits to the superior model robustness.

❸ Qualitative Results. We provide qualitative illustrations of ACDC, LiTS, MMWHS, LA, MP-MRI in Figure 4, Figure 6 (Appendix G), Figure 7 (Appendix H), Figure 8 (Appendix I), and Figure 9 (Appendix J), respectively. As shown in Figure 4, we observe that ARCO appears a significant advantage, where the edges and the boundaries of different anatomical regions are clearly more pronounced, such as RV and Myo regions. More interestingly, we found that in Figure 6, though all methods may confuse ambiguous tail-class samples such as small lesions, ARCO-SG and ARCO-SAG still produces consistently sharp and accurate object boundaries compared to the current approaches. Wealso observe similar results for ARCO on MMWHSinFigure 7 (Appendix H), where our approaches can regularize the segmentation results to be smooth and shape-consistent. Our findings suggest that ARCO improves model robustness mainly through distinguishing the minority tail-class samples.

4.2 Ablation Studies

In this subsection, we conduct various ablations to better understand our design choices. For all the ablation experiments the models are trained on ACDC with 1% labeled ratio.

Importance of Loss Components. Weanalyse several critical components of our method in the final performance and conduct comprehensive ablation studies on the ACDC dataset with a 1% label ratio to validate their necessity. First , at the heart of our method is the combination of three losses: L contrast for tailness , and L nn for diversity (See Section 3 for more details). We deactivate each component and then evaluate the resulting models, as shown in Table 2. As is shown, global contrastive loss L contrast and nearest neighbor loss L nn can boost performance by a large margin. Moreover, incorporating our methods ( i.e. , SG and SAG) consistently achieve superior model robustness gains compared to naïve sampling ( i.e. , NS), both of which suggests the importance of these components. Second , we compare the impact of different loss function ( e.g. , unsupervised loss/ L unsup, global instance discrimination loss/ L global inst , and local instance discrimination loss/ L local inst ). As shown in Table 3, using each loss function consistently achieves significant performance gains, which demonstrates positive contribution of each component to performance gains.

0

25

50

75

100

125

150

175

200

Epochs

0.017

0.018

0.019

0.020

0.021

0.022

0.023

0.024

contrast

NS

SG

Importance of Augmentation Components. Wefurther investigate the impact of data augmentation on the ACDC dataset with a 1% label ratio. As is shown in Table 4 (in Appendix), employing each data augmentation strategy consistently results in notable performance improvements, underscoring the efficacy of these data augmentations. Of note, ARCO-SAG / ARCO-SG using all three augmentations and ARCO-SAG / ARCO-SG using no augmentation are considered as the upper bound and the lower bound for the performance comparison. These results show that each augmentation strategy systematically boosts performance by a large margin, which suggests improved robustness.

Stability Analyses. In Figure 5, we show the stability analysis results on ARCO over different sampling methods. As we can see, our SG and SAG sampling facilitates convergence during the training. More importantly, SG sampling has stable performance with small standard derivations, which aligns with our hypothesis that our proposed sampling method can be viewed as the form of variance regularization. Moreover, loss landscape visualization of different loss functions (Figure 3) reveals similar conclusions.

Extra Study. More investigations about (1) generalization across label ratios and frameworks in Appendix L; (2) final checkpoint loss landscapes in Appendix M; (3) ablation on different training settings are in Appendix N.

5 Conclusion and Discussion of Broader Impact

In this paper, we propose ARCO , a new semi-supervised contrastive learning framework for improved model robustness and label efficiency in medical image segmentation. Specifically, we propose two practical solutions via stratified group theory that correct for the variance introduced by the common sampling practice, and achieve significant performance benefits. Our theoretical findings indicate Stratified Group and Stratified-Antithetic Group Sampling provide practical means for improving variance reduction. It presents a curated and easily adaptable training toolkit for training deep networks that generalize well beyond training data in those long-tail clinical scenarios. Moreover, our sampling techniques can provide pragmatic solutions for enhancing variance reduction, thereby fostering their application in a wide array of real-world applications and sectors. These include but are not limited to 3D rendering, augmented reality, virtual reality, trajectory prediction, and autonomous driving. We hope this study could be a stepping stone towards by quantifying the limitation of current self-supervision objectives for accomplishing such challenging safety-critical tasks.

Broader Impact. Defending machine learning models against inevitable variance will have the great potential to build more reliable and trustworthy clinical AI. Our findings show that the stratified group theory can provide practical means for improving variance reduction, leading to realistic deployments in a large variety of real-world clinical applications. Besides, we should address the challenges of fairness or privacy in the medical image analysis domain as our future research direction.

References

Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya Nori, and Antonio Criminisi. Measuring neural net robustness with constraints. In NeurIPS , 2016.

Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE symposium on security and privacy (sp) , 2017.

Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin Vechev. Fast and effective robustness certification. In NeurIPS , 2018.

Hayit Greenspan, Bram Van Ginneken, and Ronald M Summers. Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging , 2016.

Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In NeurIPS , 2019.

Chenyu You, Ruihan Zhao, Fenglin Liu, Siyuan Dong, Sandeep Chinchali, Ufuk Topcu, Lawrence Staib, and James Duncan. Class-aware adversarial transformers for medical image segmentation. In NeurIPS , 2022.

Wenjia Bai, Ozan Oktay, Matthew Sinclair, Hideaki Suzuki, Martin Rajchl, Giacomo Tarroni, Ben Glocker, Andrew King, Paul M Matthews, and Daniel Rueckert. Semi-supervised learning for network-based cardiac mr image segmentation. In MICCAI , 2017.

Lequan Yu, Shujun Wang, Xiaomeng Li, Chi-Wing Fu, and Pheng-Ann Heng. Uncertainty-aware self-ensembling model for semi-supervised 3d left atrium segmentation. In MICCAI , 2019.

Xiangde Luo, Jieneng Chen, Tao Song, and Guotai Wang. Semi-supervised medical image segmentation through dual-task consistency. In AAAI , 2020.

Yicheng Wu, Zongyuan Ge, Donghao Zhang, Minfeng Xu, Lei Zhang, Yong Xia, and Jianfei Cai. Mutual consistency learning for semi-supervised medical image segmentation. Medical Image Analysis , 2022.

Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR , 2006.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , 2020.

Ke Yan, Jinzheng Cai, Dakai Jin, Shun Miao, Dazhou Guo, Adam P Harrison, Youbao Tang, Jing Xiao, Jingjing Lu, and Le Lu. Sam: Self-supervised learning of pixel-wise anatomical embeddings in radiological images. IEEE Transactions on Medical Imaging , 2022.

Chenyu You, Weicheng Dai, Lawrence Staib, and James S Duncan. Bootstrapping semi-supervised medical image segmentation with anatomical-aware contrastive distillation. In IPMI , 2023.

Chenyu You, Weicheng Dai, Fenglin Liu, Haoran Su, Xiaoran Zhang, Lawrence Staib, and James S Duncan. Mine your own anatomy: Revisiting medical image segmentation with extremely limited labels. arXiv preprint arXiv:2209.13476 , 2022.

Zeju Li, Konstantinos Kamnitsas, and Ben Glocker. Overfitting of neural nets under class imbalance: Analysis and improvements for segmentation. In MICCAI , 2019.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML . PMLR, 2021.

Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation in self-supervised learning. In ICCV , 2021.

Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348 , 2021.

Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICML . PMLR, 2021.

Chenyu You, Ruihan Zhao, Lawrence H Staib, and James S Duncan. Momentum contrastive voxel-wise representation learning for semi-supervised volumetric medical image segmentation. In MICCAI , 2022.

Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Improving contrastive learning on imbalanced data via open-world sampling. In NeurIPS , 2021.

Zhengfeng Lai, Chao Wang, Henrry Gunawan, Sen-Ching S Cheung, and Chen-Nee Chuah. Smoothed adaptive weighting for imbalanced semi-supervised learning: Improve reliability against unknown distribution data. In ICML , 2022.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurIPS , 2020.

Ajinkya Tejankar, Soroush Abbasi Koohpayegani, Vipin Pillai, Paolo Favaro, and Hamed Pirsiavash. Isd: Self-supervised learning by iterative similarity distillation. In ICCV , 2021.

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 , 2021.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR , 2015.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI , 2015.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE TPAMI , 2017.

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoderdecoder with atrous separable convolution for semantic image segmentation. In ECCV , 2018.

Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In CVPR , 2017.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR , 2017.

Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueckert. Drinet for medical image segmentation. IEEE Trans. Med. Imaging , 2018.

Yunhe Gao, Rui Huang, Ming Chen, Zhe Wang, Jincheng Deng, Yuanyuan Chen, Yiwei Yang, Jie Zhang, Chanjuan Tao, and Hongsheng Li. Focusnet: Imbalanced large and small organ segmentation with an end-to-end deep neural network for head and neck ct images. In MICCAI . Springer, 2019.

Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 , 2018.

Dong Nie, Yaozong Gao, Li Wang, and Dinggang Shen. Asdnet: Attention based semi-supervised deep networks for medical image segmentation. In MICCAI , 2018.

Ran Gu, Guotai Wang, Tao Song, Rui Huang, Michael Aertsen, Jan Deprest, Sébastien Ourselin, Tom Vercauteren, and Shaoting Zhang. Ca-net: Comprehensive attention convolutional neural networks for explainable medical image segmentation. IEEE Trans. Med. Imaging , 2020.

Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. In MICCAI , 2021.

Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. arXiv preprint arXiv:2103.10504 , 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2020.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV , 2017.

Yuan Xue, Hui Tang, Zhi Qiao, Guanzhong Gong, Yong Yin, Zhen Qian, Chao Huang, Wei Fan, and Xiaolei Huang. Shape-aware organ segmentation by predicting signed distance maps. arXiv preprint arXiv:1912.03849 , 2019.

Gonglei Shi, Li Xiao, Yang Chen, and S Kevin Zhou. Marginal loss and exclusion loss for partially supervised multi-organ segmentation. Medical Image Analysis , 2021.

Zhengfeng Lai, Chao Wang, Sen-ching Cheung, and Chen-Nee Chuah. Sar: Self-adaptive refinement on pseudo labels for multiclass-imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4091-4100, 2022.

Chenyu You, Yuan Zhou, Ruihan Zhao, Lawrence Staib, and James S Duncan. Simcvd: Simple contrastive voxel-wise representation distillation for semi-supervised medical image segmentation. IEEE Transactions on Medical Imaging , 2022.

Chenyu You, Weicheng Dai, Yifei Min, Lawrence Staib, and James S Duncan. Implicit anatomical rendering for medical image segmentation with stochastic experts. In MICCAI , 2023.

Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. In NeurIPS , 2020.

Zhengfeng Lai, Chao Wang, Zin Hu, Brittany N Dugger, Sen-Ching Cheung, and Chen-Nee Chuah. A semi-supervised learning for segmentation of gigapixel histopathology images from brain tissues. In International conference of the IEEE engineering in Medicine & Biology Society (EMBC) , 2021.

Zhengfeng Lai, Chao Wang, Luca Cerny Oliveira, Brittany N Dugger, Sen-Ching Cheung, and Chen-Nee Chuah. Joint semi-supervised and active learning for segmentation of gigapixel pathology images with cost-effective labeling. In ICCV , 2021.

Yicheng Wu, Zhonghua Wu, Qianyi Wu, Zongyuan Ge, and Jianfei Cai. Exploring smoothness and class-separation for semi-supervised medical image segmentation. In MICCAI , 2022.

Chenyu You, Weicheng Dai, Yifei Min, Lawrence Staib, Jas Sekhon, and James S Duncan. ACTION++: Improving semi-supervised medical image segmentation with adaptive anatomical contrast. In MICCAI , 2023.

Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In NeurIPS , 2014.

Gerda Bortsova, Florian Dubost, Laurens Hogeweg, Ioannis Katramados, and Marleen de Bruijne. Semisupervised medical image segmentation via learning consistency under transformations. In MICCAI , 2019.

Xiangde Luo, Wenjun Liao, Jieneng Chen, Tao Song, Yinan Chen, Shichuan Zhang, Nianyong Chen, Guotai Wang, and Shaoting Zhang. Efficient semi-supervised gross target volume of nasopharyngeal carcinoma segmentation via uncertainty rectified pyramid consistency. In MICCAI , 2021.

Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In NeurIPS , 2016.

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS , 2017.

Simon Reiß, Constantin Seibold, Alexander Freytag, Erik Rodner, and Rainer Stiefelhagen. Every annotation counts: Multi-label deep supervision for medical image segmentation. In CVPR , 2021.

Henry Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Trans. Inf. Theory , 1965.

Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In CVPR , 2021.

Tanya Nair, Doina Precup, Douglas L Arnold, and Tal Arbel. Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Medical image analysis , 2020.

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In ICML , 2015.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML , 2016.

Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NeurIPS , 2017.

Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In CVPR , 2020.

Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In NeurIPS , 2014.

Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV , 2015.

Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In ICCV , 2017.

Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV , 2016.

Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV , 2016.

Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In ECCV , 2016.

Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR , 2016.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR , 2022.

Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu. Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation. arXiv preprint arXiv:2112.09645 , 2021.

Quan Quan, Qingsong Yao, Jun Li, and S. kevin Zhou. Information-guided pixel augmentation for pixel-wise contrastive learning. arXiv preprint arXiv:2211.07118 , 2022.

Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV . IEEE, 2016.

Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. Learning deep bilinear transformation for fine-grained image representation. In NeurIPS , 2019.

Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. In NeurIPS , 2021.

Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, et al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE Transactions on Medical Imaging , 2018.

Patrick Bilic, Patrick Ferdinand Christ, Eugene Vorontsov, Grzegorz Chlebus, Hao Chen, Qi Dou, ChiWing Fu, Xiao Han, Pheng-Ann Heng, Jürgen Hesser, et al. The liver tumor segmentation benchmark (lits). arXiv preprint arXiv:1901.04056 , 2019.

Xiahai Zhuang and Juan Shen. Multi-scale patch and multi-modality atlases for whole heart segmentation of mri. Medical image analysis , 2016.

Zhaohan Xiong, Qing Xia, Zhiqiang Hu, Ning Huang, Cheng Bian, Yefeng Zheng, Sulaiman Vesal, Nishant Ravikumar, Andreas Maier, Xin Yang, et al. A global benchmark of algorithms for segmenting the left atrium from late gadolinium-enhanced cardiac magnetic resonance imaging. Medical Image Analysis , 2021.

William G Cochran. Sampling techniques . John Wiley & Sons, 1977.

Søren Asmussen and Peter W Glynn. Stochastic simulation: algorithms and analysis , volume 57. Springer, 2007.

Dimitri P Bertsekas and John N Tsitsiklis. Gradient convergence in gradient methods with errors. SIAM Journal on Optimization , 10(3):627-642, 2000.

Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review , 60(2):223-311, 2018.

Zeyuan Allen-Zhu. How to make the gradients small stochastically: Even faster convex and nonconvex sgd. In NeurIPS , 2018.

Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR , 2019.

Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with crossconsistency training. In CVPR , 2020.

Yizhe Zhang, Lin Yang, Jianxu Chen, Maridel Fredericksen, David P Hughes, and Danny Z Chen. Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In MICCAI , 2017.

Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan Yuille. Deep co-training for semi-supervised image recognition. In ECCV , 2018.

Shuailin Li, Chuyu Zhang, and Xuming He. Shape-aware semi-supervised 3d semantic segmentation for medical images. In MICCAI , 2020.

Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. In IJCAI , 2019.

Yicheng Wu, Minfeng Xu, Zongyuan Ge, Jianfei Cai, and Lei Zhang. Semi-supervised left atrium segmentation with mutual consistency training. In MICCAI , 2021.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR , 2016.

Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJCV , 2015.

Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR , 2015.

Shikun Liu, Shuaifeng Zhi, Edward Johns, and Andrew J Davison. Bootstrapping semantic segmentation with regional contrast. arXiv preprint arXiv:2104.04465 , 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, highperformance deep learning library. In NeurIPS , 2019.

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR , 2017.

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc V Gool. Revisiting contrastive methods for unsupervised learning of visual representations. In NeurIPS , 2021.

Since |D m | = n m , it follows that

Appendix to

Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective

Sec. A provides additional theoretical analysis.

Sec. B provides additional details on the training datasets.

Sec. C provides additional details on the implementation.

Sec. D provides details on 2D/3D methods in comparision.

Sec. E provides additional framework details.

Sec. F provides model architecture details.

Sec. G provides more experimental results on the 2D LiTS [82] dataset.

Sec. H provides more experimental results on the 2D MMWHS [83] dataset.

Sec. I provides more experimental results on the 3D LA [84] dataset.

Sec. J provides more experimental results on the 3D MP-MRI dataset.

Sec. K provides more experimental results on three semantic segmentation benchmarks (Cityscapes [97], Pascal VOC 2012 [98], SUN RGB-D [99]).

Sec. L forms a better understanding of generalization across label ratios and frameworks.

Sec. M provides final checkpoint loss landscapes.

Sec. N provides additional details on the hyperparameter selection.

A Theoretical Results Details

A.1 Proof of Lemma 3.1

Proof. By definition of the estimate ̂ H , since the sampling from each group is independent, the variance can be written as

Note that σ 2 m is the variance of the h ( x ; p ) where p ∼ P m uniformly.

Analysis of SAG. We now consider SAG. Recall that, compared to SG, SAG first samples n m / 2 pixels from P m 6 from each m = 1 , · · · , M in the same way as SG. SAG then deterministically picks the rest n m / 2 pixels to be the reflection pixels of the first half (see Figure 2(3)). Let's choose arbitrary group m and denote by p and p ′ the sampled pixel and its reflection from that group. Note that the variance of p satisfies Var SAG [ h ( x ; p )] = σ 2 m since p is sampled in a same way as in the case of SG. Observe that, by symmetry, the variance of p and p ′ should be the same Var SAG [ h ( x ; p )] = Var SAG [ h ( x ; p )] = σ 2 m . It follows that:

where the second step holds because the correlation between h ( x ; p ) and h ( x ; p ′ ) is at most 1. It follows that:

A.2 Proof of Theorem 3.2

Proof of Theorem 3.2. Denote w m = n m /n , where n m = |D m | . The unbiasedness is straightforward: by proportional sampling, the probability that an arbitrary pixel in group P m being chosen is equal to |D m | / |P m | . Since |D m | = |P m | , the probability of being chosen is equal across all pixels, and hence an arbitrary pixel p ∈ D = ∪ m D m is equally likely to be any pixel in the population P . As a result, we have

where the first equality is by the construction of ̂ H SG ( x ; D ) , the second equality is by symmetry of p ∈ D , and the last inequality is by the above conclusion that p is equally possible to be any pixel in P . This finishes the proof that ̂ H SG is unbiased.

The variance of ̂ H SG is equal to

where σ 2 m is the variance of the f ( x ; p ) when p ∼ D m uniformly. Under the case of proportional sampling, i.e., n m ∝ |D m | for all m = 1 , · · · , M (for simplicity we assume such n m 's are all integers), the variance becomes

We now consider NS (i.e. naïve sampling). By definition, the random sampling samples pixels uniformly from the set P of all pixels. Since n is the total number of pixels sampled, the variance of the ̂ H NS is given as Var[ ̂ H NS ] = σ 2 /n where σ 2 is the sampling variance of h ( x ; p ) given p ∼ P uniformly. To determine σ 2 , we apply a variance decomposition trick via conditioning. Specifically, the sampling from NS is a two-step process: (1) sample the group index m from [ M ] , (2) sample the pixel uniformly from P m . Applying the law of total variance, we have

As a result, we conclude that, for any image x , the sampling variance of SG estimate and NS estimate satisfies

which finishes the proof.

A.3 Further Details on Convergence

We first make the following assumptions about the loss function and the gradient estimate in our learning problem.

Assumption A.1 (Smoothness) . The objective function L ( · ) is L -smooth, i.e., L is differentiable and ∥∇L ( θ ) -∇L ( θ ′ ) ∥ 2 ≤ L ( ∥ θ -θ ′ ∥ 2 ) for all θ , θ ′ in the domain of ∇L .

Assumption A.2 (Bounded variance) . There exists some σ 2 g > 0 such that for all θ in the domain,

These assumptions are standard in optimization theory and in various settings [87, 88, 89]. The first assumption allows us to characterize the loss landscape of our learning problem, and the second assumption ensures the gradient estimate does not deviate largely from the truth, which is essential for the training to converge. With these two assumptions, it is guaranteed that

Proposition 1. Suppose the step sizes in SGD are taken to be min { 1 /L, α/ ( σ g √ T ) } where T is the number of steps. Then under Assumption A.1 and A.2, there exists some constant C depending on L , α , and L , such that

Proposition 1 shows that, in expectation, the convergence rate of the average gradient norm consists of a fast rate 1 /T and a slow rate σ g / √ T , with the slow rate being the dominant term. Importantly, the slow rate depends on the variance of the gradient estimate. This suggests that a variance reduced gradient estimate allows SGD to reach an approximate local minimum with less iterations, leading to faster convergence of the training loss. This intuition is corroborated by Fig. 5, where we directly visualize the training trajectory of contrastive loss versus epoch for three sampling methods. We observe that SG features a faster loss decay and a narrower error bar, showing that it outperforms other methods in both the convergence speed and the stability. Furthermore, Proposition 1 indicates that our proposed sampling techniques are universal, since it applies to all scenarios as long as the mild assumptions A.1 and A.2 are satisfied. In other words, when it comes to scenarios that involve pixel/voxel-level sampling on 2D/3D images, utilizing SG/SAG instead of naive sampling can lead to enhanced stability and decreased variance.

B Datasets

Our experiments are conducted on five 2D/3D representative datasets in semi-supervised medical image segmentation literature, including 2D benchmarks ( i.e. , ACDC [81], LiTS [82], and MMWHS [83]) and 3D benchmarks ( i.e. , LA [84] and in-house MP-MRI).

The ACDC dataset was collected from MICCAI 2017 ACDC challenge [81], consisting of 200 3D cardiac cine MRI scans with 3 classes - left ventricle (LV), myocardium (Myo), and right ventricle (RV). Following [53, 9, 10], we utilize 120, 40, and 40 scans for training,

validation, and testing 7 . Note that 1%, 5%, and 10% label ratios are the ratio of patients. The splitting details are in the supplementary material. For pre-processing, we normalize the intensity of each 3D scan into [0 , 1] by using min-max normalization, and re-sample images and segmentation maps to 256 × 256 pixels.

The LiTS dataset was collected from MICCAI 2017 Liver Tumor Segmentation Challenge [82], consisting of 131 contrast-enhanced 3D abdominal CT volumes with 2 classes - liver and tumor. Note that 1%, 5%, and 10% label ratios are the ratio of patients. We utilize 100 and 31 scans for training and testing, with random order. The splitting details are in the supplementary material. For pre-processing, we follow the setting in [6] by truncating the intensity of each 3D scan into [ -200 , 250] HU for removing irrelevant and redundant features, normalizing each 3D scan into [0 , 1] , and re-sample images and segmentation maps to 256 × 256 pixels.

The MMWHS dataset was collected from MICCAI 2017 challenge [83], consisting of 20 3D cardiac MRI scans with 7 classes - left ventricle (LV), left atrium (LA), right ventricle (RV), right atrium (RA), myocardium (Myo), ascending aorta (AAo), and pulmonary artery (PA). Note that 1%, 5%, and 10% label ratios are based on the ratio of patients. Following [17], we utilize 15 and 5 scans for training and testing. The splitting details are in the supplementary material. For pre-processing, we normalize the intensity of each 3D scan into [0 , 1] by using min-max normalization, and re-sample images and segmentation maps to 256 × 256 pixels.

The LA dataset [84] was a representative 3D benchmark, consisting of 100 gadoliniumenhanced MRI scans with one class - left atrium (LA), with an isotropic resolution of 0 . 625 × 0 . 625 × 0 . 625 mm 3 . Note that 1%, 5%, and 10% label ratios are the ratio of patients. The fixed split ( i.e. , 5%, and 10% ) [8, 53] uses 80 and 20 scans for training and testing 8 , and 1% label ratio is randomly split. The splitting details are in the supplementary material. For pre-processing, we crop all the scans at the heart region, normalize the intensities of each 3D scan into [0 , 1] , and randomly crop all the training sub-volumes into 112 × 112 × 80 mm 3 .

The Multi-phasic MRI (MP-MRI) dataset was an in-house 3D dataset, consisting of 160 multi-phasic MRI scans with one class - liver, each of which includes T1 weighted DCE-MRI images at three-time points ( i.e. , pre-contrast, arterial phase, and venous phases). Three images are mutually registered to the arterial phase images, with an isotropic voxel size of 1 . 00 × 1 . 00 × 1 . 00 mm 3 . The dataset is randomly divided into 100 scans for training, 40 for validation, and 20 for testing. Note that 1%, 5%, and 10% label ratios are the ratio of patients. The splitting details are in the supplementary material. For pre-processing, we normalize the intensity of each 3D scan into [0 , 1] by using min-max normalization, and re-sample images and segmentation maps to 256 × 256 pixels.

C Implementation Details

In our experiments, all of our evaluated methods have been trained using similar settings for simplicity in reproducing our results. All experiments are conducted with PyTorch [101] on an NVIDIA RTX 3090 Ti. We adopt an SGD optimizer with momentum 0.9 and weight decay 10 -4 . The initial learning rate is set to 0.01. We use the 2D UNet [30] or 3D VNet [78] backbones as the segmentation network under different labeled ratio settings ( i.e. , 1%, 5%, 10% labeled ratios). Following [17], we adopt the feature pyramid network ( FPN ) [102] architecture as the representation head ψ r with a separate 512-dimension output layer. The momentum hyperparameter is set to 0.99.

For pre-training, the networks are trained for 100 epochs with a batch size of 6. We apply the mined views with d =5 , following [17]. As for fine-tuning, the networks are trained for 200 epochs with a batch size of 8. The learning rate decays by a factor of 10 every 2500 iterations during the training. We apply the temperature with τ t =0 . 01 , τ s =0 . 1 , and τ =0 . 5 , respectively. The size of the memory bank is set to 36. For the CL training, we use the implementation from [17] and leave all parameters on their default settings, e.g. , we apply the hyperparameters with λ 1 =0 . 01 , λ 2 =1 . 0 , and λ 3 =1 . 0 . For other hyper-parameters in all the evaluated methods, we adopt the suggested settings in the original papers because they are not of direct interest to us.

For 2D medical segmentation, we follow the same data augmentations [9, 53] to the teacher 's input and the student 's input, respectively. The augmentations include random rotation, random cropping, and horizontal flipping. For 3D medical segmentation, we follow the same data augmentations [8, 9, 96, 53] to the teacher 's input and the student 's input, respectively. The augmentations include random rotation and random flipping. We evaluate our methods on 3D segmentation results with two classical metrics: (1) Dice coefficient (DSC) and (2) Average Symmetric Surface Distance (ASD). Note that, for all the evaluated methods, we make no additional modifications during the training process for fair evaluations. We run all our experiments in the same environments with fixed random seeds (Hardware: Single NVIDIA GeForce RTX 3090 GPU; Software: PyTorch 1.10.2+cu113, and Python 3.8.11).

D 2D/3D Methods in Comparison

2D Medical Segmentation : For experiments, we use 2D UNet [30] as backbone, and compare ARCO with previous state-of-the-art medical segmentation methods: (1) Baseline ( i.e. , UNet-F / UNet-L [30]) using both fully-supervised and limited-supervised settings; and (2) SSL-based : EM [90], CCT [91], DAN [92], URPC [57], DTC [9], DCT [93], ICT [95], MT [59], UAMT [8], SASSNet [94], CPS [62], GCL [50], MC-Net [96], SS-Net [53], ACTION [16], and MONA [17]. Note that among all the above evaluated methods, several methods use a contrastive objective, including GCL [50], SS-Net [53], ACTION [16], and MONA [17].

3D Medical Segmentation : For experiments, we use 3D VNet [78] as backbone, and compare ARCO with previous state-of-the-art medical segmentation methods: (1) Baseline ( i.e. , VNet-F / VNet-L [78]) using both fully-supervised and limited-supervised settings; and (2) SSL-based : EM [90], CCT [91], DAN [92], URPC [57], DTC [9], DCT [93], ICT [95], MT [59], UAMT [8], SASSNet [94], CPS [62], GCL [50], MC-Net [96], SS-Net [53], ACTION [16], and MONA [17]. Note that among all the above evaluated methods, several methods use a contrastive objective, including GCL [50], SS-Net [53], ACTION [16], and MONA [17].

E Framework Overview

In the following, we provide a concise overview of ARCO , consisting of two training phases: (1) relational semi-supervised pre-training, and (2) anatomical contrastive fine-tuning. Note that, in this paper, sampling strategies in pixel-level CL framework are of direct interest, so we use a simplification of MONA [17] without using any additional complicated augmentation strategies.

(1) Relational Semi-Supervised Pretraining. Given an unlabeled sample, ARCO has two ways to define augmented and mined views: (1) ARCO augments the samples to be x 1 and x 2 as augmented views, with two separate data augmentation operators; and (2) ARCO randomly samples d mined views ( i.e. , x 3 ) from the unlabeled dataset with additional augmentation. The pairs [ x 1 , x 2 ] are then processed by [ F s , F t ] , and in a similar way x 3 is processed by F t (See Figure 1(a) in main paper), outputting three global features [ h 1 , h 2 , h 3 ] after E and their local features [ f 1 , f 2 , f 3 ] after

1% Labeled

5% Labeled

10% Labeled

Method

Average

Liver

Lesion

Average

Liver

Lesion

Average

Liver

Lesion

UNet-F

[30]

68.2/ 16.9

90.6/8.14

45.8/25.6

68.2/16.9

90.6/8.14

45.8/25.6

68.2/16.9

90.6/8.14

45.8/25.6

UNet-L

50.8/30.4

76.5/15.2

25.0/45.6

63.4/30.4

90.5/9.84

36.4/50.9

64.6/28.3

88.4/18.6

40.9/37.9

EM

[90]

56.6/38.4

86.4/26.3

26.9/50.5

61.2/33.3

87.6/9.47

34.7/57.1

62.9/38.5

87.4/21.3

38.3/55.7

CCT

[91]

52.4/52.3

79.1/42.0

25.7/62.6

61.4/26.1

84.6/12.3

38.2/39.9

63.4/23.0

88.8/7.64

38.0/38.5

DAN

[92]

57.1/28.3

84.5/19.2

29.6/37.4

62.3/25.8

88.6/9.64

36.1/42.1

63.2/30.7

87.3/15.4

39.1/46.1

URPC

[57]

55.5/34.6

83.0/27.3

28.0/42.0

60.9/21.4

85.5/8.11

36.3/34.6

62.5/24.8

84.9/12.9

40.1/36.7

DTC

[9]

39.3/37.5

68.1/10.7

10.5/64.3

59.2/18.6

85.0/7.54

33.3/29.7

62.5/24.8

84.9/12.9

40.1/36.7

DCT

[93]

57.7/38.5

87.0/22.4

28.3/54.6

60.8/34.4

89.2/12.6

32.5/56.2

61.9/31.7

86.2/19.3

37.5/44.1

ICT

[95]

58.3/32.2

86.5/22.8

30.1/41.5

60.1/39.1

86.8/12.6

33.3/65.6

62.5/32.4

88.1/16.7

36.9/48.2

MT

[59]

54.7/24.7

83.1/10.2

26.3/39.1

60.9/23.7

87.5/

6.34

34.4/41.1

62.3/23.7

88.5/9.32

36.1/38.1

UAMT

[8]

55.5/34.6

83.0/27.3

28.0/42.0

61.5/24.7

84.5/10.6

38.6/38.8

62.9/23.6

87.4/7.78

38.4/39.6

SASSNet

[94]

39.6/42.7

69.0/14.7

10.3/7.06

60.4/25.3

86.1/11.6

34.7/39.0

62.4/21.1

86.4/8.31

38.3/33.9

CPS

[62]

57.7/39.6

87.0/22.4

28.3/54.6

59.5/26.3

84.0/9.01

34.9/43.5

62.1/30.8

88.6/18.3

35.6/43.3

GCL

[50]

59.3/29.5

88.6/14.2

30.0/44.9

63.3/20.1

90.7/9.46

35.9/30.8

65.0/37.2

91.3/10.0

38.7/64.3

MC-Net

[96]

60.9/32.1

87.1/17.8

34.8/46.5

61.6/19.8

86.3/8.21

36.9/31.4

63.4/29.9

89.0/13.1

38.0/46.8

SS-Net

[53]

55.0/35.9

89.6/19.8

20.5/51.9

59.1/24.6

87.5/11.2

30.6/38.1

63.4/19.8

91.1/7.33

35.8/32.2

ACTION

[16]

61.0/24.5

89.8/16.9

32.2/32.3

66.3/23.6

93.0/6.41

39.5/40.8

67.2/20.4

92.8/5.08

41.6/35.8

MONA

[17]

62.2/24.9

91.3/13.9

34.0/35.9

66.6/16.6

93.1/7.74

40.1/25.4

68.3/18.0

93.4/8.88

43.3/27.0

•

ARCO-SAG

(ours)

64.1/20.6

91.5/7.63

36.8/

33.5

67.3/13.2

93.0/7.10

41.7/19.3

69.4/14.9

93.3/7.60

45.5/22.2

◦

ARCO-SG

(ours)

65.5

/

20.2

92.5

/

5.59

38.4

/34.7

68.4

/

11.3

93.7

/6.63

43.0

/

16.0

70.1

/

13.5

94.1

/

5.01

46.1

/

22.1

MC-Net

GCL

SS-Net

ACTION

MONA

Ground T ruth

ARCO-SAG (ours)

5% Labeled Data

10% Labeled Data

Liver

Lesion

1% Labeled Data

ARCO-SG (ours)

Figure 6: Visual results on LiTS with 1%, 5%, 10% label ratios. As is shown, ARCO-SG and ARCO-SAG consistently produce more accurate predictions on anatomical regions and boundaries compared to all other SSL methods.

D . These features are fed to the two-layer non-linear projectors for outputting global and local embeddings v g and v l .

To alleviate the collapse issues [26, 27, 16], we make the architecture asymmetric between the student and teacher pipeline by further feeding both the global and local embeddings v with respect to the student branch into the non-linear predictor, producing w in both global and local manners 9 . After passing through the nonlinear projectors and predictor, the relational similarities between augmented and mined embeddings are computed using the softmax transform, which can be formulated as:

temperature parameters. The unsupervised instance discrimination loss ( i.e. , Kullback-Leibler divergence KL ) can be defined as:

The parameters of the teacher model are the exponential moving average (EMA) of the student model parameters that are updated by the stochastic gradient descent. For pretraining, the entire loss consists of the global and local instance discrimination loss, and supervised segmentation loss L sup ( i.e. , equal combination of Dice loss and cross-entropy loss), i.e. , L = L global inst + L local inst + L sup .

(2) Anatomical Contrastive Finetuning (ACF). We use the pre-trained network weights as the initialization for subsequent fine-tuning (See Figure 1(b) in Main Paper). For semi-supervised training, we follow two principles described in [17] 10 : (1) tailness : giving more importance to tail class hard pixels; and (2) diversity : ensuring anatomical diversity in the set of different sampled images.

Following the abovementioned principles, we employ a two-step routine: (1) Tailness : we first perform anatomical contrastive formulation. Specifically, we additionally attach the representation head ψ r 11 , and generate a higher n -dimensional dense representation with the same spatial resolution as the input image. A pixel-level contrastive loss is designed to pull queries r q ∈R to be similar to the positive keys r + k ∈R , and push apart the negative keys r -k ∈R . The semi-supervised contrastive loss L contrast is defined as:

where C is a set including all available classes in the current mini-batch, and τ is a temperature hyperparameter. We refer to R c q , R c k , r c, + k as a query set including all representations within this class c , a negative key set including all representations whose labels is not in class c , and the positive key which is the c -class mean representation, respectively. Consider P is a set including all pixel coordinates with the same resolution as R , these queries and keys are then defined as:

̸

(2) Diversity : we leverage the first-in-first-out (FIFO) memory bank [13] to search for K -nearest neighbors, and use the semi-supervised nearest neighbor loss L nn in a way that maximizing cosine similarity, to exploit the inter-instance relationship.

For fine-tuning, the total loss includes contrastive loss L contrast, nearest neighbors loss L nn, unsupervised cross-entropy loss L unsup and supervised segmentation loss L sup : L sup + λ 1 L contrast + λ 2 L unsup + λ 3 L nn. See Appendix N for the ablation of hyperparameters.

F Model Architecture

Figure 1 provides an overview over our approach. Our semi-supervised segmentation model F takes an 2D/3D medical image x as input and outputs the segmentation map and the representation map. We leverage MONA pipeline [17] including two stages: (1) relational semi-supervised pre-training: on labeled data, the student network is trained by the ground-truth labels with the supervised loss L sup; while on unlabeled data, the student network takes the augmened and mined embeddings from the EMA teacher for instance discrimination L inst in the global and local manner, (2) our proposed anatomical contrastive reconstruction fine-tuning: on labeled data, the student network is trained by the ground-truth labels with the supervised loss L sup; while on unlabeled data, the student model takes the representation maps and pseudo labels from the EMA teacher to give more importance to tail class L contrast, exploit the inter-instance relationship L nn, and compute unsupervised loss L unsup . See Appendix M for details of the visualization loss landscapes.

G More Experiment Results on LiTS

Figure 6 and Table 5 show qualitative and quantitative results, where our ARCO-SG and ARCO-SAG provide better segmentation results than all other methods. This clearly demonstrates the superiority of our models.

MC-Net

GCL

SS-Net

ACTION

MONA

Ground Truth

ARCO-SAG (ours)

5% Labeled Data

10% Labeled Data

1% Labeled Data

ARCO-SG (ours)

Right  Ventricle (RV)

Left Atrium (LA)

Left  V entricle (L V)

Left  V entricle (L V)

Right Atrium (RA)

Myocardium (Myo)

Left Atrium (LA)

1% Labeled Data

5% Labeled Data

10% Labeled Data

MC-Net

GCL

SS-Net

ACTION

MONA

Ground Truth

ARCO-SAG (ours)

ARCO-SG (ours)

H More Experiment Results on MMWHS

We run the baselines and our methods on the third medical image segmentation dataset ( i.e. , MMWHS [83]) under various label ratios ( i.e. , 1%, 5%, 10%), reporting results in Table 6 and Figure 7. This clearly demonstrates the effectiveness of our models.

I More 3D Experiment Results on LA [84]

We run the baselines and our methods on the fourth 3D medical image segmentation dataset ( i.e. , LA [84]) under various label ratios ( i.e. , 1%, 5%, 10%), reporting results in Table 9 and Figure 8. This clearly demonstrates the robustness of our models.

J More 3D Experiment Results on MP-MRI

We run the baselines and our methods on the five 3D medical image segmentation dataset ( i.e. , MP-MRI) under various label ratios ( i.e. , 1%, 5%, 10%), reporting results in Table 9 and Figure 9. This clearly shows the superiority of our models.

Liver

MC-Net

GCL

SS-Net

ACTION

MONA

Ground T ruth

ARCO-SAG (ours)

ARCO-SG (ours)

1% Labeled Data

5% Labeled Data

10% Labeled Data

K More Experiment Results on Semantic Segmentation

To further validate the effectiveness, we experiment on three popular segmentation benchmarks ( i.e. , Cityscapes [97], Pascal VOC 2012 [98], indoor scene segmentation dataset - SUN RGB-D [99]) in the semi-supervised full-label settings. We follow the identical setting [100] to sample labelled images to ensure that every class appears sufficiently in our three datasets, i.e. , CityScapes, Pascal VOC, and SUN RGB-D. Specifically, we aim to have the least frequent class appear in at least 5, 15, and 50 images in each dataset, respectively. In CityScapes, we have at least 12 semantic classes represented in our labeled images, while in Pascal VOC and SUN RGB-D we have at least 3 and 1 semantic classes, respectively. We compare our ARCO-SG and ARCO-SAG under various grid settings ( i.e. , 9, 16, 25) to baselines (supervised and semi-supervised ReCo ). Table 7 shows the mean IoU validation performance. We can see that for all cases, ARCO-SG and ARCO-SAG consistently improve performance, compared to ReCo , in all the semi-supervised settings. For example, under the fewest label setting in each dataset, compared to ReCo , applying stratified group sampling (SG) can improve results by an especially significant margin, with up to 2.4 - 7.8% relative gains. As shown in Pascal VOC2012 [98] - 60 labels (Figure 10), 200 labels (Figure 11), 600 labels (Figure 12), Cityscapes [97] - 20 labels (Figure 13), 50 labels (Figure 14), 150 labels (Figure 15), SUN RGB-D [99] - 50 labels (Figure 16), 150 labels (Figure 17), 500 labels (Figure 18), ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to ReCo . All those clearly demonstrate the superiority of our models.

L Generalization across Label Ratios and Frameworks

Besides generalizing well across different datasets and diverse labeled settings, we additionally evaluate the performance of SG and SAG sampling coupled with different CL frameworks ( i.e. , MoCov2 [103], k NN-MoCo [104], SimCLR [14], BYOL [26], ISD [27], VICReg [28]) on ACDC under

various label ratios ( i.e. , 1%, 5%, 10%). In this work, we mainly study the state-of-the-art CL frameworks from the computer vision domain hereinafter for ablations, considering their superior performance in the task of image classification. Note that, for each fair comparison, we strictly follow the default setting in these CL frameworks [103, 104, 14, 26, 27, 28] for pretraining, and fine-tune each networks using the same settings in Appendix C. Full experimental details are in Appendix C.

Since they work orthogonally to our pretraining strategy, we conduct a comprehensive comparison of these CL-based frameworks in Table 8. Clearly, our proposed SG and SAG sampling help significantly in improving the segmentation performance across all the CL-based frameworks and is capable of being integrated with previous frameworks for further enhanced model robustness . Moreover, it is important to observe that the performance benefits of our methods increase significantly with a lower label setting. This observation augments the necessity of our proposed methods while training networks with high label efficiency .

M Final Checkpoint Loss Landscapes

From visualizations in Figure 19, ARCO-SG converges to much flatter loss valleys, which evidences their robustness in learning anatomical features.

N Ablation on Different Training Settings

Hyperparameter Selection. For grid search, we detail the tuning steps here. The tuning is done in sequential order. λ 1 is chosen from { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 1 . 0 } , and λ 2 , λ 3 are chosen from { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 , 2 . 0 , 10 . 0 } . We use the validation set to search over hyperparameters and find the best hyperparameter on ACDC with 1% labeled ratio. As shown in Figure 20, with a carefully tuned hyperparameters λ 1 =0 . 01 , λ 2 =1 . 0 , and λ 3 =1 . 0 , such setting achieves superior performance compared to others.

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix

ARCO-SAG (16 Grid) + ClassMix

ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix

ARCO-SAG (16 Grid) + ClassMix

ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Case 1 (60 Labels)

Case 2 (60 Labels)

Aeroplane

Bicycle

Background

Bird

Boat

Bottle

Bus

Car

Cat

Chair

Cow

Dog

Horse

Table

Motorbike

Person

Plant

Sheep

Sofa

Train

Monitor

Invalid

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix

ARCO-SAG (16 Grid) + ClassMix

ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix

ARCO-SAG (16 Grid) + ClassMix

ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Aeroplane

Bicycle

Background

Bird

Boat

Bottle

Bus

Car

Cat

Chair

Cow

Dog

Horse

Table

Motorbike

Person

Plant

Sheep

Sofa

Train

Monitor

Invalid

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix

ARCO-SAG (16 Grid) + ClassMix

ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix

ARCO-SAG (16 Grid) + ClassMix

ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Aeroplane

Bicycle

Background

Bird

Boat

Bottle

Bus

Car

Cat

Chair

Cow

Dog

Horse

Table

Motorbike

Person

Plant

Sheep

Sofa

Train

Monitor

Invalid

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix

ARCO-SAG (16 Grid) + ClassMix

ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix

ARCO-SAG (16 Grid) + ClassMix

ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Case 1 (20 Labels)

Case 2 (20 Labels)

Sidewalk

Building

Road

Wall

Fence

Pole

Traffic Light

Traffic Sign

Vegetation

Terrain

Person

Rider

Sky

Car

Truck

Bus

Train

Motorcycle

Bicycle

Undefined

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix

ARCO-SAG (16 Grid) + ClassMix

ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix

ARCO-SAG (16 Grid) + ClassMix

ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Case 1 (50 Labels)

Case 2 (50 Labels)

Sidewalk

Building

Road

Wall

Fence

Pole

Traffic Light

Traffic Sign

Vegetation

Terrain

Person

Rider

Sky

Car

Truck

Bus

Train

Motorcycle

Bicycle

Undefined

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix

ARCO-SAG (16 Grid) + ClassMix

ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix

ARCO-SAG (16 Grid) + ClassMix

ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Case 1 (150 Labels)

Case 2 (150 Labels)

Sidewalk

Building

Road

Wall

Fence

Pole

Traffic Light

Traffic Sign

Vegetation

Terrain

Person

Rider

Sky

Car

Truck

Bus

Train

Motorcycle

Bicycle

Undefined

Case 1 (50 Labels)

Case 2 (50 Labels)

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix ARCO-SAG (16 Grid) + ClassMix ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix ARCO-SAG (16 Grid) + ClassMix ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Floor

Picture

Wall

Shelves

Bathtub

TV

Curtain

Blinds

Cabinet

Bed

Chair

Sofa

Table

Door

Window

Bookshelf

Counter

Desk

Dresser

Pillow

Mirror

Clothes

Ceiling

Books

Fridge

Paper

Towel

Bath Curtain

Box

Whiteboard

Person

Sink

Nightstand

Toilet

Lamp

Bag

Underdefined

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix ARCO-SAG (16 Grid) + ClassMix ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Ground Truth

Supervised

Reco+ClassMix

ARCO-SAG (9 Grid) + ClassMix ARCO-SAG (16 Grid) + ClassMix ARCO-SAG (25 Grid) + ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Case 1 (150 Labels)

Case 2 (150 Labels)

Floor

Picture

Wall

Shelves

Bathtub

TV

Curtain

Blinds

Cabinet

Bed

Chair

Sofa

Table

Door

Window

Bookshelf

Counter

Desk

Dresser

Pillow

Mirror

Clothes

Ceiling

Books

Fridge

Paper

Towel

Bath Curtain

Box

Whiteboard

Person

Sink

Nightstand

Toilet

Lamp

Bag

Underdefined

Ground Truth

Supervised

Reco+ClassMix

ARCO-SG (9 Grid) + ClassMix

Ground Truth

Supervised

Reco+ClassMix

ARCO-SG (25 Grid) + ClassMix

ARCO-SG (16 Grid) + ClassMix

ARCO-SG (9 Grid) + ClassMix

Floor

Picture

Wall

Shelves

Bathtub

TV

Curtain

Blinds

Cabinet

Bed

Chair

Sofa

Table

Door

Window

Bookshelf

Counter

Desk

Dresser

Pillow

Mirror

Clothes

Ceiling

Books

Fridge

Paper

Towel

Bath Curtain

Box

Whiteboard

Person

Sink

Nightstand

Toilet

Lamp

Bag

Underdefined

MP-MRI

LA

1% Labeled

5% Labeled

10% Labeled

1 Labeled (1%)

4 Labeled (5%)

8 Labeled (10%)

3D Method

Liver

Liver

Liver

Left Atrium (LA)

Left Atrium (LA)

Left Atrium (LA)

VNet-F

[78]

93.1/5.73

93.1/5.73

93.1/5.73

91.5/1.51

91.5/1.51

91.5/1.51

VNet-L

68.6/33.4

81.6/12.5

87.9/7.55

40.0/21.2

52.6/9.87

82.7/3.26

EM

[90]

73.2/30.1

86.0/15.8

91.9/6.89

48.3/21.3

81.1/4.68

82.7/4.77

CCT

[91]

74.2/24.5

86.0/15.8

90.9/9.54

40.3/13.8

70.8/8.31

82.0/5.25

DAN

[92]

69.4/31.4

88.3/10.2

91.1/8.76

38.5/22.0

78.8/6.53

80.2/5.37

URPC

[57]

72.7/29.9

89.8/9.00

91.7/7.41

65.0/8.97

80.2/5.48

83.1/4.57

DTC

[9]

78.6/18.8

89.6/10.1

90.5/11.2

36.2/11.7

83.6/2.81

87.1/2.23

DCT

[93]

74.1/31.1

88.2/12.4

90.1/11.1

42.9/19.1

80.1/9.06

80.4/9.18

ICT

[95]

72.3/30.5

88.6/11.2

91.1/8.46

47.7/16.0

78.4/6.96

85.4/4.14

MT

[59]

73.8/29.4

87.7/12.8

92.0/7.15

58.1/17.8

77.0/8.15

82.8/5.90

UAMT

[8]

71.6/31.2

87.1/12.8

91.3/9.71

60.3/11.3

82.3/3.82

87.8/2.12

SASSNet

[94]

78.8/19.6

88.4/13.1

88.7/13.1

51.5/14.6

81.6/3.58

87.5/2.59

CPS

[62]

80.0/17.1

89.2/10.8

91.0/9.16

45.1/22.0

79.7/9.28

80.7/5.16

GCL

[50]

78.9/14.2

87.9/8.29

90.4/5.68

52.6/12.8

75.5/7.60

84.8/4.22

MC-Net

[96]

79.7/20.2

90.1/8.27

92.4/

4.34

44.3/14.1

83.6/2.70

87.6/1.82

SS-Net

[53]

88.0/8.93

90.9/9.94

92.0/5.67

43.4/14.8

86.3/2.31

88.6/1.90

ACTION

[16]

86.5/13.6

90.3/12.3

90.9/10.0

71.1/6.23

86.6/2.24

88.7/2.10

MONA

[17]

91.3/

5.31

92.2/9.46

92.3/8.16

72.8/10.7

87.0/2.81

89.5/2.40

•

ARCO-SAG

(ours)

91.5/6.82

92.5

/6.95

92.6/7.54

73.2/6.47

86.9/2.73

89.1/2.30

◦

ARCO-SG

(ours)

91.6

/6.60

92.5

/

6.31

92.8

/6.64

75.0

/

4.06

87.8

/

1.66

89.9

/

1.47

ACDC

0.6

(1)

(2)

(3)

(4)

(5)

(6)

612113

0.6

LiTS

(1)

(2)

(3)

(4)

(5)

(6)

MMWHS

0.6

0.6

(1)

(2)

(3)

(4)

(5)

(6)

0.6

0.6

(1)

(2)

(3)

(4)

(5)

(6)

MR-MRI

2 1 2 2 2

(1)

(2)

(3)

(4)

(5)

(6)

(a)

(b)

(c)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEdit: Updating score from 6 t o7 based on the discussions.\n\nExtracting useful representations from unlabelled data for label efficient training of medical image segmentation task is a widely studied problem. This work approaches learning useful representations using contrastive learning (CL) within a semi-supervised setting. Strategies for obtaining variance reducing pixel partitions for CL are presented, along with theoretical analysis that show their variance reduction properties. The variance reduction estimation is also used to improve training stability and convergence. Comprehensive experiments on multiple medical imaging and computer vision datasets are performed showing strong performance improvements compared to other SSL methods. The authors show their method is label efficient across all the datasets.\n\nSTRENGTHS:\n* Focusing on variance reduction guarantees to extract contrastive samples for pixel level training is a strong contribution of this work. \n\n* The theoretical analyses showing their unbiasedness and use of variance reduction techniques to improve training stability can have important applications in related domains like self-supervised learning.\n\n* The  experimental evaluation is extensive, with strong performance improvements on multiple datasets.\n\n* Contrastive loss landscape visualisation in Fig 3 is insightful; that the contrastive learning holds across datasets is quite convincing.\n\nWEAKNESSES:\n* **Robustness**: This work makes two main claims about the usefulness of their CL framework. Firstly, and convincingly so, about label efficiency. There are several places in the paper where model robustness is alluded to, or strong claims made, without any evidence. This could be because the authors view robustness simply to be good performance across multiple datasets? If so, this should be clarified. Currently, the claims about model robustness are misleading. See [1,2,3] for different robustness analyses of deep neural networks.\n\n* **Assumption in Th. 3.2**: The guarantees of $Var[H_{SG}] < Var[H_{NS}]$ only holds if different $P_m$ do not have the same expected value over the aggregation function h(x;p). How is this ensured? In medical images, there are scenarios where the differences between classes are small in both intensity and feature spaces? How does the variance reduction guarantees hold in such situations? \n\n* **Aggregation functions**: Aren't the aggregation functions some type of a distance measure? And why is it expensive to compute these on dense pixel grids (L213)? \n\n* **Main method in Appendix**: While I appreciate all the details presented in this paper, moving the main method to the Appendix is not a good idea. Several of the important details are in the Appendix and does not serve the purpose of what Appendices are supposed to be.\n\n* **Literature overview**: I was curious as to why the authors refrained from discussing self-supervised representation learning both when motivating the work, and also in their general discourse. \n\n[1] Bastani, Osbert, et al. \"Measuring neural net robustness with constraints.\" Advances in neural information processing systems 29 (2016).\n\n[2] Carlini, Nicholas, and David Wagner. \"Towards evaluating the robustness of neural networks.\" 2017 ieee symposium on security and privacy (sp). Ieee, 2017.\n\n[3] Singh, Gagandeep, et al. \"Fast and effective robustness certification.\" Advances in neural information processing systems 31 (2018).\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nAuthors do not discuss any limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present ARCO, a novel semi-supervised contrastive learning framework that employs a stratified group sampling strategy (i.e. **SG** and **SAG**) to compute gradient estimators with reduced variance, thereby enhancing representation learning in dense contrastive learning. By improving dense contrastive learning, ARCO addresses issues related to class imbalancedness and enhances the performance of semi-supervised segmentation, particularly in scenarios with long-tail distributed anatomical classes. The authors provide theoretical evidence demonstrating the effectiveness of the proposed sampling techniques in reducing the variance of the aggregation function, specifically the contrastive loss. The efficacy of the proposed sampling technique is validated on eight 2D/3D benchmark datasets with different label settings, further reinforcing its effectiveness and practical applicability.\n\nSTRENGTHS:\nThe motivations are clear, and the method is reasonable. This study shows me a new insight/direction to consider medical image segmentation. The following are my detailed comments.\n\n---\n\n**Empirical contribution**\n\nPixel/voxel-wise sampling constitutes a critical facet of contrastive learning at the pixel/voxel level. With the aid of variance-reduction estimation, the authors proffer two pragmatic approaches - Stratified Group (SG) and Stratified-Antithetic Group (SAG), tailored for pixel/voxel-level segmentation tasks with exceedingly scarce labels.  \n\n* The authors introduce a novel framework termed ARCO (strAtifed gRoup COntrastive learning) devised for multi-class segmentation tasks. This framework appears to be both intriguing and efficacious. The authors undertake a rigorous validation of the proposed methodologies across eight benchmark datasets, encompassing three 2D medical image segmentation, two 3D medical image segmentation, and three semantic segmentation benchmarks.  \n\n* The empirical results, both quantitative and qualitative, attest to the efficacy of the proposed model across all label ratios and datasets. For instance, the model demonstrates a marked enhancement in segmentation accuracy (up to 4.1% absolute improvements in Dice coefficient) on the challenging multi-class MMWHS dataset under a 1% label setting.  \n\n* The authors conduct comprehensive ablation studies to substantiate that the proposed mechanisms merit consideration. These studies encompass eight benchmark datasets, diverse network architectures, and varying label ratios to validate the efficacy, model-agnostic nature, and label efficiency of the proposed methodology.  \n\n* Lastly, the proposed methodologies are not only facile to implement but also boast of universal applicability. For instance, they can be seamlessly integrated into any scenario necessitating pixel/voxel sampling.  The paper presents a robust and versatile framework for pixel/voxel-level contrastive learning, which is empirically validated through extensive experiments and ablation studies. The methodologies are characterized by ease of implementation and broad applicability, making them a valuable contribution to the field of image segmentation.\n\n---\n**Theoretical contribution**\n\n* The proposed methodologies, SG (Stratified Group) and SAG (Stratified-Antithetic Group), have exhibited remarkable efficacy in the experimental study. Consequently, it is intriguing to ascertain whether theoretical insights can elucidate this enhancement in performance. To this end, the paper furnishes a cogent theoretical analysis of the methodologies, revealing that the variance-reduction attribute of the two sampling methods is instrumental to their performance.\n\n* First and foremost, in Section 3.3, the paper meticulously delineates the SG and SAG sampling methodologies through lucid mathematical equations. SG is executed by segregating pixels into mutually exclusive groups, followed by uniform sampling of a specified number of pixels from each group. SAG, which is predicated on SG, imposes an additional constraint of symmetry among the sampled pixels within each group.\n\n* Subsequently, the sampled pixels are amalgamated through an aggregation function, which acts as an estimator for the target quantity. In the realm of image segmentation, this quantity could be, for instance, the contrastive loss function. It is posited that an optimal balance must be struck in the sample size; an overly diminutive sample size may fail to encapsulate the salient information from the underlying image, whereas an excessively large sample size would entail high computational complexity. Thus, the ideal scenario would be for SG to capture the crux of the image information through a relatively modest sample size. The paper demonstrates that SG possesses this attribute by establishing that it achieves reduced variance in comparison to the naïve sampling method (i.e., uniform random sampling from all pixels). Specifically, Theorem 3.2 establishes that SG is an unbiased sampling methodology, with a variance that does not exceed that of naïve sampling. More precisely, the variance of SG can be decomposed into the variance of naïve sampling minus a non-negative term. This non-negative term is conjectured to be almost certainly greater than zero, as it would be zero only if the expectation of the aggregation function within each group is identical to the expectation over the entire image, which is highly improbable. Thus, Theorem 3.2 suggests that SG is likely to consistently outperform naïve sampling.\n\n* Figure 5 reveals that SG/SAG exhibits marginally expedited training convergence compared to naïve sampling. This observation is theoretically substantiated in the concluding paragraph of Section 3, which is commendable.\n\n* In my assessment, the theoretical analysis presented in the paper is inextricably linked to the empirical component and provides a persuasive rationale for the empirical performance augmentation of SG vis-à-vis naïve sampling, thereby bringing the empirical narrative full circle.\n---\n\n**To sum up**\n\n**1. Clarity** \n\nThe manuscript is eloquently composed, proffering a lucid and cogent progression of information. The authors adeptly elucidate the procedural framework, facilitating the readers' comprehension of the proposed methodologies, namely, the two instance sampling methods - Stratified Group (SG) and Stratified-Antithetic Group (SAG). In Section 3.3, the SG and SAG sampling methods are meticulously delineated through precise mathematical formulations. The discourse within the Methodology section (Section 3) furnishes an exhaustive exposition of the innovations introduced by the study, adeptly accentuating the distinct contributions of the research to the scholarly domain.\n\n**2. Novelty** \n\nThis work is the first work, empirically and theoritically, to validate the variance-reduction approach within the context of pixel/voxel-level contrastive learning for semi-supervised medical image segmentation, particularly in scenarios characterized by a paucity of labels.\n\n**3. Experimental Comprehensiveness** \n\nThe authors undertake a comprehensive suite of experiments encompassing eight medical datasets, which include both 2D and 3D modalities, as well as semantic segmentation benchmarks. Additionally, a diverse array of contrastive learning frameworks and varying label ratios are employed to rigorously assess the efficacy, model-agnostic properties, and label efficiency of the proposed methodology. This extensive experimental evaluation substantiates the robustness and versatility of the technique in the domain of medical image segmentation.\n\n**4. Theoretical Implication** \n\nThe authors furnish a cogent and meticulously articulated theoretical analysis of the proposed approach, elucidating the underlying principles with clarity and precision. This analytical exposition contributes to a deeper understanding of the methodology's foundations and its implications.\n\nWEAKNESSES:\nThe proposed method is very interesting. There is no obvious weakness in the proposed ARCO. However, I do have a few questions. See the following section.\n\nQUESTIONS:\n* The proposed two sampling techniques are intriguing. I wonder if you could kindly provide insights on the applicability of these sampling techniques to different tasks. Such information would prove beneficial not only to the current domain but also to other related domains and scenarios. Understanding the potential applications of these techniques would undoubtedly contribute to the advancement of various fields.\n \n* Lemma 3.1 demonstrates that the variance of SAG is at most twice of that of SG, which implies that the variance reduction magnitude of SAG and SG are roughly at the same level. Could the authors further elaborate on when SAG be preferred to SG?\n\nLIMITATIONS:\nThe authors have addressed the potential broader impact in clinical scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper proposes two new sampling strategies, SG and SAG, for contrastive learning in semi-supervised frameworks. Compared with randomly sampling pixels for contrastive learning, pixels are grouped into several subsets, and then pixels are sampled from each subset. The proposed method is proven to reduce the variance of sampled pixels. Solid experiments are conducted to support the above claim.

STRENGTHS:
1. Solid experiments and good performance.
2. Simple yet effective method that benefits contrastive learning for semi-supervised frameworks.
2. Sound theoretical analysis.

WEAKNESSES:
1. It would be better to also compare NS with the proposed SG/SAG in natural image datasets.
2. What is the application scenario of SAG? SG seems to have better performance and stability than SAG.
3. Line 222-223, what is the meaning of "p is orthogonal to p'"?

QUESTIONS:
See weaknesses.

LIMITATIONS:
The authors have included a discussion on the potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a sampling strategy to improve contrastive-learning-based medical image segmentation performance with limited labeled data and training stability. The sampling strategy is used in conjunction with a previously-published contrastive semi-supervised training strategy. The main contributions include this sampling strategy, some theoretical support that the strategy should improve training stability, and experiments evaluating the method.\n\nSTRENGTHS:\n- The baseline comparison experiments are thorough.\n- The authors address a real applied problem (that most medical image segmentation datasets have limited data) with a new method and theoretical support, resulting in a well-rounded paper.\n\nWEAKNESSES:\nMajor weaknesses\n- The paper’s writing is a major limitation. The prose is difficult to understand, which limits the entire paper—it is hard to clearly understand the motivation, the proposed method, the contributions, or the benefits. The work would benefit from more rounds of grammatical revision; it is hard to parse what the author is trying to communicate a lot of the time.\n- Partly due to the writing issues, it is unclear how the proposed sampling strategy differs from previous dense contrastive learning-based image segmentation strategies; it seems that the methodological contributions here are minor, if existent.\n- The method is complex, consisting of two backbones (connected via EMA), global/local instance discrimination losses, augmentations, supervised losses, nearest neighbor loss, global contrastive loss, unsupervised loss… with all of these components, a very thorough ablation section is needed to understand how much the novel component (a pixel-level sampling strategy) is contributing to performance. The existing ablation section is not so thorough. As a result, the paper does not contribute much understanding about the strengths/weaknesses of different components of this complex pipeline.\n\nMinor weaknesses \n- MONA is not a well-known training strategy; it would be useful to provide a longer overview on what MONA does and how the proposed approach differs. This discussion could go in an appendix.\n- The same concept is often referred to using different words: model “convergence,” “robustness,” and “stability.” I’m not sure if you’re always talking about the same concept, or if you are using different words to refer to the same idea. If the latter, it helps the reader to always use the same word.\n- This is a minor stylistic note, but the use of bold and italics is often distracting.\n\nQUESTIONS:\nQuestions and suggestions discussed above\n\nLIMITATIONS:\nLimitations and impacts adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}