Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective
Authors: Chenyu You
Weicheng Dai
Yifei Min
Fenglin Liu
David A. Clifton
S Kevin Zhou
Lawrence Hamilton Staib
James s Duncan
Venue: NeurIPS 2023 poster (2023)
Keywords: Long-tailed Medical Image Segmentation
Contrastive Learning
Variance Reduction
Imbalanced Learning
Semi-Supervised Learning
TL;DR: We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for 2D/3D medical image segmentation.

Abstract: For medical image segmentation, contrastive learning is the dominant practice to improve the quality of visual representations by contrasting semantically similar and dissimilar pairs of samples. This is enabled by the observation that without accessing ground truth labels, negative examples with truly dissimilar anatomical features, if sampled, can significantly improve the performance. In reality, however, these samples may come from similar anatomical features and the models may struggle to distinguish the minority tail-class samples, making the tail classes more prone to misclassification, both of which typically lead to model collapse. In this paper, we propose $\texttt{ARCO}$, a semi-supervised contrastive learning (CL) framework with stratified group theory for medical image segmentation. In particular, we first propose building $\texttt{ARCO}$ through the concept of variance-reduced estimation, and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks with extremely limited labels. Furthermore, we theoretically prove these sampling techniques are universal in variance reduction. Finally, we experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings, and our methods consistently outperform state-of-the-art semi-supervised methods. Additionally, we augment the CL frameworks with these sampling techniques and demonstrate significant gains over previous methods. We believe our work is an important step towards semi-supervised medical image segmentation by quantifying the limitation of current self-supervision objectives for accomplishing such challenging safety-critical tasks.

Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective
Chenyu You'
Weicheng Dai
Yifei Min
Fenglin Liu
David A. Clifton
Kevin Zhou
Lawrence Staib
James S. Duncan
1 Yale University
2 University of Oxford
3 University of Science and Technology of China
Abstract
For medical image segmentation, contrastive learning the dominant practice to improve the quality of visual representations by contrasting semantically similar and dissimilar pairs of samples. This is enabled by the observation that without accessing ground truth labels, negative examples with truly dissimilar anatomical features, sampled, can significantly improve the performance. reality, however, these samples may
from similar anatomical regions and the models may struggle to distinguish the minority tail-class samples, making the tail classes more prone misclassification, both of which typically lead to model collapse. In this paper, we propose ARCO, semi-supervised contrastive learning (CL) framework with stratified group theory for medical image segmentation. In particular, we first propose building ARCO through the concept of variance-reduced estimation and show that certain pariance-reduction techniques are particularly beneficial in pixel/voxcl-leve segmentation tasks with extremely limited labels. Furthermore, we theoretically prove these sampling techniques are universal variance reduction. Finally, we experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings, and our methods consistently outperform state-of-the-ate semi-supervised methods. Additionally, we augment the CL frameworks with these sampling techniques and demonstrate significant gains over previous methods. We believe our work is an important step towards semi-supervised medical image segmentation by quantifying the limitation of current self-supervision objectives for accomplishing such challenging safety-critical tasks.
Introduction
Model robustness and label efficiency are two highly desirable perspectives when it comes to building reliable medical segmentation models. In the context of medical image analysis, model is said to be robust if (1) has a high segmentation quality with only using extremely limited labels in long- tailed medical data; (2) and fast convergence speed [1, 3]. The success of traditional supervised learning depends on training deep networks on a large amount of labeled data, but this improved segmentation/modell robustness often comes the cost of annotations and clinical expertise [4, 6]. Therefore, is difficult to adopt these models in real-world clinical applications.
Recently, significant amount of research efforts [7, 9, 10] have resorted to unsupervised or semi-supervised learning techniques for improving the segmentation robustness. One of the most effective methods is contrastive learning (CL) [11, 12, 13, 14], It aims learn useful representations
Codes available here.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

by contrasting semantically similar (positive) and dissimilar (negative) pairs of data points sampled from the massive unlabeled data. These methods fit particularly well with real-world clinical scenarios as we assume only access to a large amount of unlabelled data coupled with extremely limited labels. However, pixel-level contrastive learning with medical image segmentation is quite impractical since sampling all pixels can be extremely time-consuming and computationally expensive [15]. Fortunately, recent studies [16, 17] provide a remedy by leveraging the popular strategy of bootstrapping, which first actively samples a sparse set of pixel-level representation (queries), and then optimize the contrastive objective by pulling them to be close to the class mean averaged across all representations in this class (positive keys), and simultaneously pushing apart those representations from other class (negative keys). The demonstrated imbalancedness and diversity across various medical image datasets, as echoed in [18], show the positive sign of utilizing the massive unlabeled data with extremely limited annotations while maintaining the impressive segmentation performance compared supervised counterparts. Meanwhile, it can lead to substantial memory/ccomputatinn reduction when using pixel-level contrastive learning framework for medical image segmentation.
Nevertheless, in practical clinical settings, the deployed machine learning models often ask for strong robustness, which is far beyond the scope of segmentation quality for such challenging safety- critical scenarios. This leads to more challenging requirement, which demands the models to be more robust to the collapse problems whereby all representations collapse into constant features [14, 13, 19] or only span a fragility could be attributed to the non-smooth feature space near samples [24, 25] (i.e., random sampling can result in large feature variations and even annotation information alter). Thus, it is a new perspective: how to sample most informative pixels/voxels towards improving variance reduction in training semi-supervised contrastive learning models. This inspires to propose new hypothesis of semi-supervised CL. Specifically, when directly baking in variance-reduction sampling into semi-supervised CL frameworks for medical image segmentation, the models can further push toward state-of-the-an segmentation robustness and label efficiency.
subspace [20, 21, 22, 23), as main cause of such
In this paper, we present ARCO, a semi-supervised strAtifed gRoup Contrastive learning framework with two perspectives (i.e., segmentation//model robustness and label efficiency), and with the aid of suriance-reduction estimation, realize two practical solutions Stratified Group (SG) and Prratifid-Antitheen Group (SAG) - for selecting the most semantically informative pixels. ARCO is group-based sampling method that builds set of pixel groups and then proportionally samples from each group with respect to the class distribution. The main idea of our approach is via first partitioning the image with respect different classes into grids with the same size, and then sampling, within the same grid, pixels semantically close to each other with high probability, with minimal additional memory footprint.
Subsequently, we show that baking ARCO into contrastive pre-training (i.e, MONA [17]) provides an efficient pixel-wise contrastive learning paradigm to train deep networks that perform well in long-tailes sailed medical data. ARCO is easy to implement, being built on top of off-the-shelf pixel-level contrastive learning framework [13, 14, 26, 27, 28], and consistently improve overall segmentation quality across all label ratios and datasets (i.e., five 2D/3D medical and three semantic datasets).
Our theoretical analysis shows that, ARCO is more label efficient, providing practical means for computing the gradient estimator with improved variance reduction. Empirically, our approach achieves competitive results across eight 2D/3D medical and semantic segmentation benchmarks. Our proposed framework has several theoretical and practical contributions:
We propose ARCO, a new CL framework based on stratified group theory to improve the label efficiency and model robustness trade-off in CL for medical image segmentation. We show that incorporating ARCO coupled with two special sampling methods, Stratified Group and
Group, into the models provides an efficient learning paradigm to train deep networks that perform well in those long-tail clinical scenarios.
To our best knowledge, we are the first work to show the benefit of certain variance- reduction techniques in CL for medical image segmentation. We demonstrate the unexplored advantage of the refined gradient estimator in handling long-tailed medical image data.
We conduct extensive experiments to validate the effectiveness of our proposed method using variety of datasets, network architectures, and different label ratios. For segmentation robustnesssaccuracyy we show that our proposed method by demonstrating superior segmen- tation accuracy (up to 11.08% absolute improvements in Dice). For label efficiency, our
2

---

method trained with different labeled ratios consistently achieves competitive performance improvements across all eight 2D/3D medical and semantic segmentation benchmarks.
Theoretical analysis of ARCO shows improved variance reduction with optimization guarantee. We further demonstrate the intriguing property of ARCO across the different pixel-level contrastive learning frameworks.
2
Related work
Medical Image Segmentation. Contemporary medical image segmentation approaches typically build upon fully convolutional networks (FCN) [29] or UNet [30], which formulates the task as a dense classification problem. In general, current medical image segmentation methods can be cast into two sets: network design and optimization strategy. One is to optimize segmentation network design for improving feature representations through dillated/atroousddeooomabbb convolutions [31, 32, 33], pyramid pooling [34, 35, 36], and attention mechanisms [37, 38, 39j. Most recent works [40, 41, 6] reformulates the task as a sequence-to- to-ttt--sseee to-sequence prediction task by using the vision transformer (ViT) architecture [42, 43J. The other is to improve optimization strategies, by designing loss function to better address class imbalance [44] or refining uncertain pixels from high-frequency regions improving the segmentation quality [45, 46, 47, 48, 49]. In contrast, we take leap further more practical clinical scenario by leveraging the massive unlabeled data with extremely limited labels in the learning stage. Moreover, we focus on building model-agnostic, label- to improve segmentation quality by providing additional supervision on the most confusing pixels for each class. In this work, question how medical segmentation models behave under such imbalanced class distributions and whether they can perform well in those challenging scenarios through sampling methods.
framework
Sepi-Superrised Learning (SSL). SSL aims to train models with a combination of labeled, weakly- labeled and unlabelled data. recent years, there has been surge of work on semi-supervised medical segmentation [8, 9, 50, 48, 16, 51, 52, 17, 10, 53, 54], which makes it hard to present complete overview here. We therefore only outline some key milestones related to this study. In general, it can be roughly categorized into two groups: (1) Consistency regularization was first proposed by [55], which aims to impose consistency corresponding to different perturbations into the training, such consistency regularization [56, 57]. pi-model [58], and mean-teacher [59, 60]. (2) Self-training was initially proposed in [61], which aims at using a model's predictions to obtain noisy pseudo-labels for performance boosts with minimal human labor, such pseudo-labeling [7. 62], model uncertainty [8, 63), confidence estimation [64, 65, 66), and noisy student [67]. These methods usually lead to competitive performance but fail to prevent collapse due to class imbalanceness. In this work, we focus on semi-supervised medical segmentation with extremely limited labels since the medical image data is extremely diverse and often long-tail distributed over anatomical classes. We speculate that a good medical segmentation model is expected to distinguish the minority tail-class samples and hence achieve better performance under additional supervision on hard pixels.
Contrastive Sell-Supervised Learning. Self-supervised representation learning is subclass of unsupervised learning, but with the critical distinction that it incorporates "inherent" supervision from the input data itself [68]. The primary aim of eeffssupevvised representation learning is to enable the model to learn the most useful representations from the large amount of unlabelled data for various downstream tasks. Self-supervised learning typically relies on pretext tasks, including predictive [69, 70, 71], contextual [72, 73], and generative [74] or reconstructive [75] tasks.
Among them, contrastive learning is considered as a popular approach for self-supervised repre- sentation learning by pulling the representations of similar instances closer and representations of dissimilar instances further apart in the learned feature space [11, 12, 13, 14]. The past five years have seen tremendous progress related CL in medical image segmentation [50, 23, 76, 48, 16. 17, 77], and it becomes increasingly important to improve representation in label-scarcity scenarios. The key idea in CL [11, 12, 13, 14] to learn representations from unlabeled data that obey similarity constraints by pulling augmented views of the same samples closer in representation space, and pushing apart augmented views of different samples. This is typically achieved by encoding view of a data into single global feature vector. However, the global representation is sufficient for simple tasks like image classification, but does not necessarily achieve decent performance, especially for more challenging dense prediction tasks. On the other hand, several works on dense contrastive
3

---

(n Adiziaa sxpanaed PaOcEEa
Anstzrzia Doerartio Reccarsccion flarcaaa
Gaed
casd levver
Hurrs tepler
Figure 1: Pipeline overview. Our semi semi-suuereised segmentation model takes a 2D/3D medical image as input and outputs the segmentation map and the representation map. We leverage a simplification of MONA pipeline [17] which is composed of two stages: (1) relational semi- supervised pre-training:
labeled data, the student network is trained by the ground-truth labels with the supervised loss Lup: while on unlabeled data, the student network takes the augmened and mined embeddings from the EMA teacher for instance discrimination Linst in the global and local manner, (2) anatomical contrastive reconstruction fine-tuning; on labeled data, the student network is trained by the ground-truth labels with the supervised loss Lup: while unlabeled data, the student network takes the representation maps and pscudo labels from the EMA teacher to give more importance to tail class Coontrast exploit the inter-instance relationship Lan- and compute unsupervised loss Lunsup- See Appendix M for details of the visualization loss landscapes.
learning [50, 23], aim at providing additional supervision to capturing intrinsic spatial structure and fine-grained anatomical correspondence, while these methods may suffer from class imbalance issues. Particularly, very recent work [16, 17] for the first time demonstrates the imbalancedness phenomenon can be mitigated by performing contrastive learning yet lacking stability. By contrast, a key motivation of work bridge the connection between model robustness and label efficiency, which we believe an important and under-explored area. We hence focus on variance-reduced estimation in medical image segmentation, and show that certain variance-reduccion techniques can help provide more efficient approaches or alternative solutions for handling collapse issues, and improving model robustness in terms of accuracy and stability. To the best of our knowledge, we are the first to provide a theoretical guarantee of robustness by using certain variance-reduction techniques.
Methodology
In this section set-up our semi-supervised medical segmentation problem, introduce key definitions and notations and formulate an approach to incorporate stratified group theory. Then, we discuss how our proposed ARCO can directly bake in two perspectives into deep neural networks: (1) model robustness, and (2) label efficiency.
3.1 Preliminaries and setup
Problem Definition. In this paper, we consider the multi-class medical image segmentation problem. Specifically, given medical image dataset (X, wish automatically learn segmentator, which assigns each pixel to their corresponding K K-class segmentation labels. Let us denote x as the input sample of the student and teacher networks F(-) consisting of an encoder and a decoder D, and F' is parameterized by weights 0g and 0,.
Background. Contrastive learning aims to learn effective representations by pulling semantically close neighbors together and pushing apart other non-neighbors [11]. Among various popular contrastive learning frameworks, MONA [17] is easy-to-implement while yielding state-of-thh-arr performance for semi-supervised medical image segmentation so far. The main idea of MONA is to discover diverse views (i.e., augmented/mined views) whose anatomical feature responses are homogeneous within the same or different occurrences of the same class type, while at the same time being distinctive for different class types.
The student and teacher networks both adopt the 2D UNet [30] 3D VNet [78] architectures.

---

Hereinafter, we are interested in showing that certain variance-reduction techniques coupled with CL frameworks are particularly beneficial in long-tail pixcl/voxel-level segmentation tasks with extremely limited labels. We hence build our ARCO as simplification of the MONA pipeline [17], without additional complex augmentation strategies, for deriving the model robustness and label efficiency proprieties of our medical segmentation model. Figure overviews the high-level workflow of the proposed ARCO framework. Training ARCO involves a two-phase training procedure: (I) relational semi-supervised pre-training, and (2) anatomical contrastive fine-tuning. To make the discussion self-contained, we defer the full details of ARCO to the appendix
3.2 Motivation and Challenges
Intuitively, the contrastive loss will learn generalizable, balanced and diverse representations for downstream medical segmentation tasks if the positive and negative pairs correspond the desired latent anatomical classes [50, 16, 17J. Yet, one critical constraint in real-world clinical scenarios severe memory bottlenecks [15, 16]. To address this issue, current pixel-level CL approaches [16, 17] for high-resolution medical images devise their aggregation rules by unitary simulators, Le., Natve Sampling (NS), that determines the empirical estimate from all available pixels. Despite the blessing of large learning capacity, such aggregation rules are unreliable "black boxes". It is never well understood which rule existing CL. models should use for improved model robustness and label efficiency; nor is it easy to compare different models and assess the model performance. Moreover, unitary simulators, especially naïve sampling. often incur high variances and fail to identify semantically similar pixels [24], limiting CL stability. As demonstrated in Figure 3, regions of similar anatomical features should be grouped together the original medical images, resulting in corresponding plateau regions in the visualization of the loss landscape. This is consistent with the observations uncovered by the recent empirical findings [79, 80].
If we take unified mathematical per- spective, the execution of simulation can be represented either through an adaptive rule, or by a unitary simulation. To tackle the two critical issues, we look back at adaptive rules. We hence propose two straightforward yet effective techniques - Stratified Group (SG) and Stratified- Antithetic Group (SAG) to mitigate the undesirable high- variance limitation, and turn the following idea of sampling the most representative pixels from groups of semantically similar pixels. In particu- lar, our proposed solution is based on stratified group simulation to adaptively characterize anatomical regions found on different medical images. This characterization is succinet, and regions with the same anatomical properties within different medical images are identifiable. In practice, we first partition the image with respect to different classes into grids with the same size, and then sampling, within the same grid, the pixels semantically close to each other with high probability, with minimal additional memory footprint (Figure 2).
A
*
f+
*
100
100
*
...
Native Sampling (NS)
Stratified Group (SG) (3) Stratified -nttiihhtt Group Sampling
(SAG) Sampling
Figure 2: Overview of three sampling methods.
(1) Naive Sampling, (2) Stratified Group Sampling, and Group Sampling.
(3)
In what follows, we will theoretically demonstrate the important properties of such techniques (Le., SG and SAG), especially in reduced variance and unbiasedness. Here the reduced variance implies more robust gradient estimates in the backpropagation, and leads to faster and stabler training in theory, as corroborated by our experiments (Section 4). Empirically, we will demonstrate many practical benefits of reduced variances including improved model robustness, i.e., faster convergence and better segmentation quality, through mitigating the collapse issue.
3.3 Stratified Group Sampling
To be consistent with the previous notation, we denote an arbitrary image from the given medical image datasct as x X, and P as the set of pixels. For arbitrary function h X' x P - R, we define
5

---

(1) ACDC
[2) LiTS
(3) MMW11S
(4) LA
(5) MP-MRI
Figure Loss landscape visualization pixel-wise contrastive loss Ceontrast with ARCD-SG. Loss plots are generated with same original images randomly chosen from ACDC [81]. LiTS [82]. MMWHS [83], LA [84], and MP-MRI, respectively. z-axis denotes the loss value each pixel. For each example of the five benchmarks, the left subplot indicates that similar anatomical features are grouped together in the original medical images, as shown by different anatomical regions different colors.
the aggregation function H as:
H(x)
1
Σ h(x;p).
(3.1)
[P] PEP
As large cardinality of P prevents efficient direct computation of H, immediate approach is to compute (xx by first sampling a subset of pixels DCP according to certain sampling strategy, and then computing H D): Eped [(xpp) SG sampling achieves this by first decomposing the pixels into M disjoint groups Pm satisfying P P, and then sampling Dm S Pm so that D LMM m= Dm- The SG sampling can then be written as:
M
Aso D) =
Σ h(x:p).
M m=1
D m
PED
SAG, built upon SG, adopts similar form, except for an additionally enforced symmetry on Dm: m, 3 Cm Pm, such that for any p E D
p pp' Cm, for some p' DD--
Here Cm denotes the center of the group P The implementation of SG and SAG involves two steps: (1) to create groups (P.m. } M
mm=1, and (2) to generate each D,,, S Pm. For the latter, we consider independent sampling within and between groups, i.e., Dm IL Dm for m + m', and p p' p.
p' E Dm. where the variance of SG sampling as follows.
Lemma 3.1. Suppose in SG sampling, for each m, Dm is sampled from Pm with sampling variance o and sample size (Dmil = Tem- Then the variance satisfies Var[Hsg] = EM= o am am//m/ m/n,,, and SAG with the same sample size satisfies Var [as[] < 2 Var[[[sc).
To ensure the unbiasedness property, we adopt the setting of proportional group sizes [85, 86), i.e., [Dml [Pml for all m. It turns out that such setting also enjoys the variance-reduction property. Theorem 3.2 (Unbiasedness and Variance of SG). SG with proportional group sizes is unbiased, and has variance no larger than that of NS. That is: [[Hs = H Hxx), and
Var Vasol Var[āiss]
1
Σ
E P
[h(x;
EE ( [hxx))))
The last term the intra-group variance, which captures the discrepancy between the pixel groups [Pmm JMM Theorem 3.2 guarantees that the variance of SG is no larger than that of NS, and SG
'The pixel-level contrastive loss Leomtn is an example of aggregation function (up to normalizing constant) according to Eqn. (E.2). 'The choice of c.. is flexible. For example, if the convex hull of the pixels in P.,. form circle, then e. can be taken as the geometric center.

---

has strictly less variance than NS as long as all the pixel groups do not share an equal mean over h(x;p), which almost-sure in medical images (See Figure 3). For SAG, Lemma 3.1 guarantees its variance is of the same magnitude as that of SG, and at worst differs by a factor of 2. Since the pixel/voxel-level contrastive loss Leuntrast an aggregation function over pixels by definition (E.2), it benefits from the variance-deduction property of SG/SAG. In Section 4.1, we will see that such variance reduction allows ARCO to achieve better segmentation accuracy, especially along the boundary of the anatomical regions (Figure 4).
Training Convergence. We further demonstrate the benefit of variance reduction estimation in terms of training stability. Specifically, leveraging techniques from standard optimization theory [87, 88, 89), we can show that variance-reduced gradient estimator through SG sampling leads to faster training convergence. Suppose we have loss function L(0) with the model parameter 0, and use stochastic gradient descent (SGD) as the optimizer. A gradient estimate g(0) R ((() is computed at each iteration. is well-known that the convergence of SGD depends on the quality of the estimate g(0) [87]. Specifically, we make the common assumptions that the loss function is smooth and the gradient estimate has bounded variance (More details in Appendix A.3), which can be formulated as below:
|V 22(00)22 s L[I10 1(12)-0'l2), E [lla(0)
S og
Under these two assumptions, the average expected gradient norm of the learned parameter satisfies the following:
T [ sC
og
For general non-convex loss function, the above implies convergence to some local minimum. Importantly, the slow rate (o,/vT) depends on standard deviation dg. indicating a faster convergence can indeed be achieved with more accurate gradient estimate. This indicates our proposed sampling techniques demonstrate universality in variance reduction, as they can be applied to a wide range of scenarios that involve pixel/voxel-level sampling (See Appendix A.3). Figure 5, we observe that using SG enables faster loss decay with smaller error bar, showing that it outperforms other methods in both convergence speed and stability. See Section 4.2 and Appendix A.3 for more details.
4 Experiments
In this section, we present experimental results to validate our proposed methods across various datasets and different label ratios in Appendix B. We use 2D UNet [30] or 3D VNet [78] as our backbones. Further implementation details are discussed in Appendix C.
4.1 Main Results
In this subsection, we first examine whether our proposed ARCD can generalize well across various datasets and label ratios. Then, we investigate to what extent ARCO coupled with two samplers can realize two essential properties: (1) model robustness; and (2) label efficiency. The quantitative results for all the compared methods on eight popular datasets: (1) Medical image segmentation tasks: three 2D benchmarks (i.e., ACDC [81], LITS [82], MMWHS [83]). two 3D benchmarks (i.e.,
LA [84], ho-house MP-MRI) under various label ratios (i.e,, 1%, 5%, 10%) are collected in Table 1, Table (Appendix G), Table 6 (Appendix H), and Table 9 (Appendix and J), respectively; (2) General computer vision tasks: To further validate the effectiveness, we experiment on three popular segmentation benchmarks (i.e., Cityscapes [97], Pascal VOC 2012 [98], indoor scene segmentation dataset SUN CG (99]) in the supervipe full-label settings. We follow the identical setting [100] to sample labelled images to ensure that every class appears sufficiently in our three datasets, (i.e., CityScapes. Pascal VOC, and SUN RGB-D). The results collected in Appendix Section K. Several consistent observations can be drawn from these extensive evaluations with eighteen segmentation networks.
Superior Performance Across Datasets. We demonstrate that ARCO achieves superior perfor- mance across all datasets and label ratios. In specific, our experiments consider three 2D benchmarks
Codes are available here.
7

---

Table I: Quantitative comparisons (DSC[%) f ASD[voxel] 4) across the three labeled ratio settings (1%, 5%, 10%) on the ACDC benchmark. All experiments are conducted as [30, 90, 91, 92, 93, 8. 94, 9, 57, 95, 62, 50, 59, 96, 53, 16, 17] in the identical setting for fair comparisons. Best and second-best results are coloured blue and red, respectively. UNet-F (fully-supervided) and U (semi-supervided)
considered as the upper bound and the lower bound for the performance comparison, Note that, Right Ventricle + RV, Myocardium - Myo, Left Ventricle + LV. We adopt the identical data augmentation (i.e., random rotation, random cropping, and horizontal flipping) for fair comparisons.
MC3C
(1
Labeled (5%)
LLabeldd
Aserage
Ausrage
Mgo
Averig
Myn
Uiees of [o]
91,500506
90,590.506
$3.00.911
94.41144
91.50.996
90.580.06
85,80,941
94,411.4
91.50.99
90.573.00
$5.3/2.941
96 06.4/1.4
USet-L
40 40.3/227
29.023.4
45.615.3
46.2/22.5
$1.713..
36.963011
$4.904.2
63.45.11
79.5/2.3
65.900.92
$2.92.70
$0.644.00
11.92.57
75.7/273
[P1]
58 7725
40.3 10.2/148
$1.517.0
59./10.3
666/19.8
07.22.00
(3.94.59
DAP (92)
48 115
45 03.41977
41.08.55 58
60.423.8 6
55.41533
<7 27.1/1..
56 SL1/1
63.9T1
76.53.01
75.70.61
73.35 13.301 11
50.53311
UAPC (17)
43:021111
58 82222
41.714.4
46.6/2600
$4.005 14
50.172.6
60.30 10
65.67771
$3.01.68
77.0039422
82 22.248.56 565
32333
61.3/21
55,07720
19919
01
72-17.59
$4.394.4
83.20122
831
45.6/10
195)
56. 51..5
43
46.34555
59.00,55
08 08.011.4
85
$0.0114
508
10.311
66 08
[99]
47 5815
53. 5211 5
46. 1135 59
50 50.07-- 7
54. 11.3113 1:3
58 47
17. 143 73
80.072.35
75.24 77
79. 00.2033 37
16
UAIT [S]
$2.52.55
46.28.99
50.17--7
48.39.14
37/01399
50.144.2
57.344.17
41.5/4.04
79.02.73
50.13332 32
55.4
BASDINs [94]
29
45 413.3
<7.911127
65.872.87
34.71183
51.9727199
51.5/15.3
41.1/1522
51.421.0
61.02.91
75.1/311
14.5%
99. 2254.14 3
09. 52555
50 10.95.55 18
56.9911
20.60.54
96.
TO
$4.6/.05
K7.001751
16.909994
$1300.530
07. 10.08.50 549
NC-Mer 1961
11.4/17 17
431025 1
11.3741
10811111
61.00.8 00
50 14
62.00-807
96.971.99
65. 12.15445
$400211 17
90 32
67-42.44
46 76.40770
9/191
02
895
36.55 45.40140
85 19
$4.41 44
WIN1 57
ACTICN
81,03445
7EL
78 42.00
87-55.17
96.60
85.2/0734
900
5/197
87.20 47
84
50.00
86.90.667
55.711
50.5211
AROZ- SAG
54.903.00
81: 721
900003
96.91155 525
100 885
98.414
00.930
90.57.30
58.571400
5T. 103.555
26.211 04
92.32255
AICD 52 (SU))
95.5409977
61.81119
83
88.70844
88.200618
85.399673
91.91135
$0.401 776
98.200 101
86.30.757
91A00689
"pocardem Phol
Le Cenaar
PAg 7441
MCNK
Si-Ncc
ACTION
MONA
ANCOSAD (um))
AKCOAG roent
Figure 4: Visual results on ACDC with 1%, 5%, 10% label ratios. ARCO consistently produce more accurate predictions on anatomical regions and boundaries compared to all other SSL methods.
(i.e,, ACDC [81], LITS [82]. MMWHS [83]), two 3D benchmarks (i.e., LA [84], in-house MP-MRD, and different label ratios (i.e., 1%, 5%, 10%). As shown Table 1, Table (Appendix G). Table (Appendix H), and Table 9 (Appendix and J), we observe that our methods consistently outperform all the compared SSL-based methods by a considerable margin across all datasets and label ratios, which validates the superior performance of our proposed methods in both segmentation accuracy and label efficiency. For example, compared to the second-best MONA, our ARCO under (1% 5%, 10%) label ratios achieves (..9% 1.8%.. 1.7%tt. (3.3%1, 1.8%1. 1.8%1). (4.1% 2.0% 1.8% 1. 1. 0.5%+). 1..2% 0.8%+, 0.4%^ in average Dice across ACDC, LiTS, and MMWHS, MP-MRI, and LA. respectively. Our ARCO-SAG achieves (84.9%, 87.1%, 8..5%). (64.1%, 67.3%, 69.4%). (86.1% %, 88.6%, 89.3%), J. (91.5%. 92.5%, 92.6%), ). (73.2%, 86.9%, 89.1%) in averaged Dice across ACDC, LITS, MMWHS, MP-MRI, and LA. These results indicate that our methods can generalize to different clinical scenarios and label ratios.
@ Across Label Ratios and Robustified Methods. To further validate the label efficiency property of our ARCO, we evaluate our ARCO-SG and ARCO-SAG with limited labeled training data available (e.g., 1% and 5%). As demonstrated in Table 1, Table 5 (Appendix G), Table 6 (Appendix H), and Table 9 (Appendix and J), our models under 5% label ratios surpass all the compared SSL methods by significant performance margin. For example, compared to MONA, we observe our methods to
8

---

Table 2: Ablation on component aspect: (1)
Table 3: Ablation on loss function: (I) unsu- pervised loss/ Cunsupi (2) global instance dis- crimination loss/ Limua and (3) local instance
Luilness/Coomane (2) diversity/Cmm
Method
DSC[&]]
ASD[voxel]
discrimination loss/ Clse!
Vanilla
49.3
7.11
ARCO-SAC (ours)
84,9
1.47
Method
DSC[$]]
ASD[voxel]
w/o Sai
60.9
4.11
ARCD-SAG (ours)
84,9
1.47
w/o diversity
78.6
1.68
w/o Lunsp
81.2
1.87
ARCO-SG SG (ours)
85.5
0.947
w/o Calibal
84.0
2.64
w/o SG
60.9
4.11
w/o Linal
83.3
2.63
w/o diversity
79.3
1.26
ARCO-SG SG (ours)
85.5
0.947
ARCO-NS
82.6
1.43
w/o Lussup
81.9
1.04
w/o tailness-NS
60.9
4.11
w/o Cakiial
84.1
2.10
w/o diversity
75.2
2.07
w/o Linal
83.8
2.11
push the best segmentation accuracy higher by 0.3%~~2.00 in Dice on ACDC, LiTS, and MMWHS. MP-MRI, and LA, respectively. For example, the best segmentation accuracy MMWHS rises from 87.3% to 89.3%. This suggests that our SSL-based approaches - without compromising the best achievable segmentation results robustly improve performance using very limited labels, and further lead to a innpovidp trade-off between SSL schemes and supervised learning schemes by avoiding large amount of labeled data.
Similar to our results under 5% label ratio, our ARCO-SC and ARCO-SAG trained with 1% label ratio demonstrate sufficient performance boost compared to MONA by especially significant margin, with up to -..1% relative improvement in Dice. Taking the extremely limited label ratio (i.e. 1%) as indicator: (1) on 3D LA, ARCO-SG achieves 2.2% higher average Dice, and 6.64 lower average ASD than the second best MONA; (2) on LiTS, ARCO-SE achieves 3.3% higher average Dice, and 4.7 lower average ASD than the second best MONA; and (3) considering the more challenging clinical scenarios (i.e., 7 anatomical classes), ARCO-SG achieves 5.1% % higher average Dice, and 2.26 lower average ASD than the second- best MONA MMWHS. It highlights the superior performance of ARCO is not only from improved label efficiency but also credits to the superior model robustness.
Qualitative Results. We provide qualitative illustrations of ACDC, LITS, MMWHS, LA, MP-MRI Figure 4, Figure (Appendix G), Figure (Appendix H), Figure 8 (Appendix 1), and Figure 9 (Appendix J), respectively. As shown in Figure 4. observe that ARCO appears a significant advantage, where the edges and the boundaries of different anatomical regions are clearly more pronounced, such as RV and Myo regions. More interestingly. we found that in Figure 6, though all methods may confuse ambiguous tail-class samples such as small lesions, ARCO-SG and ARCO-SAG still produces consistently sharp and accurate object boundaries compared to the current approaches. We also observe similar results for ARCO on MMWHS in Figure 7 (Appendix H), where our approaches can regularize the segmentation results to be smooth and shape-consistent. Our findings suggest that ARCO improves model robustness mainly through distinguishing the minority tail-class samples.
4.2 Ablation Studies
In this subsection, conduct various ablations to better understand design choices. For all the ablation experiments the models are trained on ACDC with 1% labeled ratio.
Importance of Loss Components. We analyse several critical components of our method in the final performance and conduct comprehensive ablation studies on the ACDC dataset with a 1% label ratio validate their necessity. First, at the heart of our method is the combination of three losses: Coontrast for tailness, and L. for diversity (See Section 3 for more details). We deactivate each component and then evaluate the resulting models, as shown in Table 2. As is shown, global contrastive loss Cuontrast and nearest neighbor loss L.n can boost performance by a large margin. Moreover, incorporating our methods (i.e., SG and SAG) consistently achieve superior model robustness gains compared to naive sampling (i.e., NS), both of which suggests the importance of these components. Second, we compare the impact of different loss function (e.g., unsupervised loss/Cunsup- global instance discrinination loss/ C and local instance discrimination loss/ Clocal). As shown in Table 3, using each loss function consistently achieves significant performance gains, which demonstrates positive contribution of each component performance gains.

---

0.024
SG
0.023
SAG
0.022
0.021
0.020
0.010
0.017
25
50
75
100
125
150
175
200
Epochs
Figure 5: Visualization of training trajectories given by Coontast vs. epochs ACDC under 10% label ratio. The proposed ARCO is compared in terms of different sampling methods: Naive Sampling (NS), Stratified Group (SG) Sampling, and Stratifedd-Antihhetie Group (SAG) Sampling. The solid line and shaded area of each sampling method denote the mean and variance of test accuracies over independent trials. Clearly, we observe SG sampling consistently outperforms the other sampling methods in convergence speed and training stability. SAG slightly outperforms NS.
Importance of Augmentation Components. We further investigate the impact of data augmentation on the ACDC dataset with 1% label ratio. As is shown in Table 4 (in Appendix), employing each data augmentation strategy consistently results in notable performance improvements, underscoring the efficacy of these data augmentations. Of note, ARCO-SAG/AAA S000-SA///RCD-SS using all three augmentations and ARCO-S RAROOOSSG/ARCO---- using no augmentation are considered as the upper bound and the lower bound for the performance comparison. These results show that each augmentation strategy systematically boosts performance by a large margin, which suggests improved robustness.
Stability Analyses. In Figure 5, show the stability analysis results on ARCO over different sampling methods. As we can see, our SG and SAG sampling facilitates convergence during the training. More importantly, SG sampling has stable performance with small standard derivations. which aligns with our hypothesis that our proposed sampling method can be viewed as the form of variance regularization. Moreover, loss landscape visualization of different loss functions (Figure 3) reveals similar conclusions.
Extra Study. More investigations about (1) generalization across label ratios and frameworks in Appendix L; (2) final checkpoint loss landscapes in Appendix M; (3) ablation on different training settings are in Appendix N.
5 Conclusion and Discussion of Broader Impact
In this paper, we propose ARCO, a new semi-supervised contrastive learning framework for improved model robustness and label efficiency in medical image segmentation. Specifically, we propose two practical solutions via stratified group theory that correct for the variance introduced by the common sampling practice, and achieve significant performance benefits. Our theoretical findings indicate Stratified Group and Stratiffed-Antichttic Group Sampling provide practical means for improving variance reduction. It presents curated and easily adaptable training toolkit for training deep networks that generalize well beyond training data in those long-tail clinical scenarios. Moreover, our sampling techniques can provide pragmatic solutions for enhancing variance reduction, thereby fostering their application in wide array of real-world applications and sectors. These include but are not limited to 3D rendering, augmented reality, virtual reality, trajectory prediction, and autonomous driving. We hope this study could be stepping stone towards by quantifying the limitation of current self-supervision objectives for accomplishing such challenging safety-critical tasks.
Broader Impact. Defending machine learning models against inevitable variance will have the great potential build more reliable and trustworthy clinical AI. Our findings show that the stratified group theory can provide practical means for improving variance reduction, leading to realistic deployments in large variety of real-world clinical applications. Besides, we should address the challenges of fairness privacy in the medical image analysis domain our future research direction.
10

---

References
[1] Osbert Bastani. Yani Ioannou. Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya Nori, and Antonio Criminisi, Measuring neural net robustness with constraints. NeurlPS, 2016.
[2] Nicholas Carlini and David Wagner. Towards evaluating the robustness neural networks. In IEEE symposium on security and privacy (sp). 2017.
[3] Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Plschell, and Martin Vechev. Fast and effective robustness certification. In NeurlPS, 2018.
[4] Hayit Greenspan, Bram Van Ginneken, and Ronald M Summers. Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging, 2016.
[5] Maithra Raghu, Chiyuan Zhang. Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In NeurIPS, 2019.
[6] Chenyu You, Ruihan Zhao, Fenglin Liu, Siyuan Dong. Sandeep Chinchali, Ufuk Topeu, Lawrence Staib, and James Duncan. Class- aware adversarial transformers for medical image segmentation. In Neur!PS, 2022.
[7] Wenjia Bai, Ozan Oktay, Matthew Sinclair, Hideaki Suzuki. Martin Rajchl, Giacomo Tarroni. Ben Glocker. Andrew King, Paul M Matthews, and Daniel Rueckert. Semi-supervised learning for network-based cardiac mr image segmentation. In MICCAI, 2017.
[8] Lequan Yu, Shujun Wang, Xiaomeng Li, Chi-Wing Fu, and Pheng-Ann Heng. Uncertainty-awar self-ensembling model for semi-supervised left atrium segmentation. In MICCAI, 2019.
[9] Xiangde Luo, Jieneng Chen, Tao Song, and Guotai Wang. Semi-supervised medical image segmentation through dual-task consistency. In AAAI, 2020.
[10] Yicheng Wu, Zongyuan Ge, Donghao Zhang, Minfeng Xu, Lei Zhang, Yong Xia, and Jianfci Cai. Mutual consistency learning for semi-supervised medical image segmentation. Medical Image Analysis, 2022.
[11] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In CVPR, 2006.
[12] Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 180,v0/807 2018
[13] Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[14] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In JCML, 2020.
[15] Ke Yan, Jinzheng Cai, Dakai Jin, Shun Miao, Dazhou Guo, Adam P Harrison, Youbao Tang, Jing Xiao, Jingjing Lu, and Le Lu. Sam: Supf-super learning of pixel-wise anatomical embeddings in radiological images. IEEE Transactions on Medical Imaging, 2022.
[16] Chenyu You, Weicheng Dai, Lawrence Staib, and James S Dunean. Bootstrapping semi-supervised medical image segmentation with anatomical-aware contrastive distillation. In IPMI, 2023.
[17] Chenyu You, Weicheng Dai, Fenglin Liu, Haoran Su, Xiaoran Zhang, Lawrence Staib, and James S Duncan. Mine your own anatomy: Revisiting medical image segmentation with extremely limited labels. arXiv preprint 134467200.13376, 2022.
[18] Zeju Li, Konstantinos Kamnitsas, and Ben Glocker. Overfitting of neural nets under class imbalance: Analysis and improvements for segmentation. In MICCAI, 2019.
[19] Jure Zboutar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML. PMLR, 2021.
[20] Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrela- tion in self-supervised learning. In ICCV, 2021.
[21] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint 1rX.v.2110.09348, 2021.
11

---

[22] Yuandong Tian. Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICML. PMLR, 2021.
[23] Chenyu You, Ruihan Zhao, Lawrence H Staib, and James S Duncan. Momentum contrastive voxel-wise representation learning for semi-supervised volumetric medical image segmentation. In MICCAI, 2022.
[24] Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Improving contrastive learning on imbalanced data via woold sampling. In NewrIPS, 2021.
[25] Zhengfeng Lai, Chao Wang, Henrry Gunawan, Sen-Ching S Cheung, and Chen-Nee Chuah. Smoothed adaptive weighting for imbalanced semi-supervised learning: Improve reliability against unknown distribution data. In JCML, 2022.
[26] Jean-Basten Ban--Bateen Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In Neur/PS, 2020.
[27] Ajinkya Tejankar, Soroush Abbasi Koohpayegani, Vipin Pillai, Paolo Favaro, and Hamed Pirsiavash. Isd: Self-supervised learning by iterative similarity distillation. In ICCV, 2021.
[28] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: self-supervised learning. arXiv preprint arXiv:2105.04900 2021.
regularization for
[29] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmen- tation. In CVPR, 2015.
[30] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. MICCAI, 2015.
[31] Liang-Chich Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE TPAMI, 2017.
[32] Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
[33] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In CVPR, 2017.
[34] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi. Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
[35] Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueckert. Drinet for medical image segmentation. IEEE Trans. Med. Imaging, 2018
[36] Yunhe Gao, Rui Huang, Ming Chen, Zhe Wang, Jincheng Deng, Yuanyuan Chen, Yiwei Yang, Jic Zhang, Chanjuan Tao, and Hongsheng Li. Focusnet: Imbalanced large and small organ segmentation with an end-to-end deep neural network for head and neek images. In MICCAI. Springer, 2019.
[37] Ozan Oktay, Jo Schlemper. Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh. Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprini 1rXiv:1804.03999 2018.
[38] Dong Nic, Yaozong Gao, Li Wang, and Dinggang Shen. Asdnet: Attention based semi-supervised deep networks for medical image segmentation. In MICCAI, 2018.
[39] Ran Gu, Guotai Wang, Tao Song. Rui Huang, Michael Acrtsen, Jan Deprest, Sébastien Ourselin, Tom Vercauteren, and Shaoting Zhang. Ca-net: Comprehensive attention convolutional neural networks for explainable medical image segmentation. IEEE Trans. Med. Imaging. 2020.
[40] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. In MICCAI, 2021.
[41] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. arXiv preprint arXiv:2103.1 100.00044 2021.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
12

---

[43] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition scale. In ICLR, 2020.
[44] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[45] Yuan Xue, Hui Tang. Zhi Qiao, Guanzhong Gong, Yong Yin, Zhen Qian, Chao Huang, Wei Fan, and Xiaolei Huang. Shape-aware organ segmentation by predicting signed distance maps. arXiv preprint arXiv:1912.038499 2019.
[46] Gonglei Shi, Li Xiao, Yang Chen, and S Kevin Zhou. Marginal loss and exclusion loss for partially supervised multi-organ segmentation. Medical Image Analysis, 2021.
[47] Zhengfeng Lai, Chao Wang, Sen-ching Cheung, and Chen-Nee Chuah. Sar: Self-adaptive refinement pscudo labels for multiclass-ibaalanee Conference on Computer Vision and Pattern Recognition, pages 4091-4100, 2022.
supervisd learning. In Proceedings of the IEEE/CVF
[48] Chenyu You, Yuan Zhou, Ruihan Zhao, Lawrence Staib, and James S Duncan. Simevd: Simple contrastive voxel-wise representation distillation for semi-supervised medical image segmentation. IEEE Transactions on Medical Imaging, 2022.
[49] Chenyu You, Weicheng Dai, Yifei Min, Lawrence Staib, and James S Duncan. Implicit anatomical rendering for medical image segmentation with stochastic experts. In MICCAI, 2023.
[50] Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. In NeurlPS, 2020.
[51] Zhengfeng Lai, Chao Wang, Zin Hu, Brittany N Dugger, Sen-Ching Cheung, and Chen-Nee Chuah. A semi-supervised learning for segmentation of gigapixel histopathology images from brain tissues, In International conference of the IEEE engineering Medicine & Biology Society (EMBC). 2021.
[52] Zhengfeng Lai, Chao Wang, Luca Cemy Oliveira, Brittany N Dugger, Sen-Ching Cheung, and Chen-Nee Chuah. Joint semi-supervised and active learning for segmentation of gigapixel pathology images with cost-effective labeling. In ICCV, 2021.
[53] Yicheng Wu, Zhonghua Wu. Qianyi Wu, Zongyuan Ge, and Jianfei Cai. Exploring smoothness and class seasaaaepa for semi-supervised medical image segmentation. In MICCAI, 2022.
[54] Chenyu You, Weicheng Dai, Yifei Min, Lawrence Staib, Jas Sekhon, and James S Duncan. ACTION++: Improving semi-supervised medical image segmentation with adaptive anatomical contrast. In MICCAI, 2023.
[55] Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pecudo-ensembles. In NeurlPS, 2014.
[56] Gerda Bortsova, Florian Dubost, Laurens Hogeweg, Ioannis Katramados, and Marleen Bruijne. Semi- supervised medical image segmentation via learning consistency under transformations. In MICCAI, 2019.
[57] Xiangde Luo, Wenjun Liao, Jieneng Chen, Tao Song. Yinan Chen, Shichuan Zhang, Nianyong Chen, Guotai Wang, and Shaoting Zhang. Efficient semi-supervised gross target volume of nasopharyngeal carcinoma segmentation via uncertainty rectified pyramid consistency. In MICCAI, 2021.
[58] Mchdi Sajjadi, Mchran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised lcarning. In NeurIPS, 2016.
[59] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NewrlPS, 2017.
[60] Simon Reiß, Constantin Seibold, Alexander Freytag, Erik Rodner, and Rainer Stiefelhagen. Every annotation counts: Multi-lahel deep supervision for medical image segmentation. In CVPR, 2021.
[61] Henry Scudder. Probability of enror of some adaptive pattern-recoonitioon machines. IEEE Trans. Inf. Theory, 1965.
[62] Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In CVPR, 2021.
[63] Tanya Nair, Doina Precup, Douglas L Amold, and Tal Arbel. Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Medical image analysis, 2020.
13

---

[64] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In ICML, 2015.
[65] Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
[66] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NeurIPS, 2017.
[67] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Lc. Self-training with noisy student improves imagenet classification. In CVPR, 2020.
[68] Alexey Dosovitskiy. Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In Neur/PS, 2014.
[69] Carl Doerseh, Abhinav Gupta, and Alexci A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
[70] Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual Iearning. In ICCV, 2017.
[71] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCVV 2016.
[72] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
[73] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In ECCV, 2016.
[74] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
[75] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[76] Krishna Chaitanya, Ertune Erdil, Neerav Karani, and Ender Konukoglu. Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation. arXiv preprint arXiv:2112.09645, 2021.
[77] Quan Quan, Qingsong Yao, Jun Li, and S. kevin Zhou. Information guided pixel augmentation for pixel-wise contrastive learning. arXiv preprint 2rXiv:2211.07/11 2022.
[78] Fausto Millctari, Nassir Navab, and Seyed- Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV. IEEE, 2016.
[79] Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. Learning deep bilinear transformation for fine-grained image representation. In NeurlPS, 2019.
[80] Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. In NeurlPS, 2021.
[81] Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang. Pheng-Ane Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, et al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE Transactions on Medical Imaging, 2018
[82] Patrick Bilic, Patrick Ferdinand Christ, Eugene Vorontsov, Grzegorz Chlebus, Hao Chen, Qi Dou, Chi- Wing Fu, Xiao Han, Pheng-Ann Heng, Jürgen Hesser, et al. The liver tumor segmentation benchmark (lits). arXiv preprint arXiv: 04056, 2019.
[83] Xiahai Zhuang and Juan Shen. Multi-scake patch and multi- multi-modality atlases for whole heart segmentation of mri. Medical image analysis, 2016.
(84) Zhaohan Xiong, Qing Xia, Zhiqiang Hu, Ning Huang, Cheng Bian, Yefeng Zheng, Sulaiman Vesal, Nishant Ravikumar, Andreas Maier, Xin Yang, al. A global benchmark of algorithms for segmenting the left atrium from late gadolinuu-eehanneed cardiac magnetic resonance imaging. Medical Image Analysis, 2021.
[85] William Cochran. Sampling techniques. John Wiley Sons, 1977.
[86] Søren Asmussen and Peter W Glynn. Stochastic simulation: algoritimss and analysis, volume 57. Springer. 2007.
14

---

[87] Dimitri P Bertsekas and John N Tsitsiklis. Gradient convergence in gradient methods with errors. SIAM Journal Optimization, 10(3):627-642, 2000.
[88] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223 11133,1122220111 2018.
[89] Zeyuan Allen-Zhu. How to make the gradients small stochastically: Even faster convex and nonconvex sgd. In NeurIPS, 2018.
[90] Tuan-Hung Vu, Himalaya Jain, Maxime Rucher, Matthicu Cord, and Patrick Pérez, Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, 2019.
[91] Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross- consistency training. In CVPR, 2020.
[92] Yizhe Zhang, Lin Yang, Jianxu Chen, Maridel Frederieksen, David P Hughes, and Danny Z Chen. Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In MICCAI, 2017.
[93] Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan Yuille. Deep co-training for semi-supervised image recognition. In ECCV, 2018.
[94] Shuailin Li, Chuyu Zhang, and Xuming He. Shape-aware semi-supervised 3d semantic segmentation for medical images. In MICCAI, 2020.
[95] Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. In IJCAI, 2019.
[96] Yicheng Wu, Minfeng Xu. Zongyuan Ge, Jianfei Cai, and Lei Zhang. supe-super left atrium segmentation with mutual consistency training. In MICCAI, 2021.
197] Marius Cordts, Mohamed Omran, Scbastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Be- nenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. CVPR, 2016.
[98] Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: retrospective. IJCV, 2015.
[99] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. CVPR, 2015.
[100] Shikun Liu, Shuaifeng Zhi, Edward Johns, and Andrew J Davison. Bootstrapping semantic segmentation with regional contrast. arXiv preprint arXiv:2104.04466 2021.
[101] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NewrlPS, 2019.
[102) Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In CVPR, 2017.
[103] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint 2xXiv:2003.04297, 2020.
[104] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue v Gool. Revisiting con- trastive methods for unsupervised learning of visual representations. In NeurlPS, 2021.
15

---

sampled in same Appendix to Rethinking Semi-Supervised Medical Image Segmentation:
A Variance-Reduction Perspective
Sec. A provides additional theoretical analysis.
Sec. B provides additional details on the training datasets.
Sec. C provides additional details on the implementation.
Sec. D provides details 2D/3D methods in comparision.
Sec. E provides additional framework details.
Sec. F provides model architecture details.
Sec. G provides more experimental results the 2D LiTS [82] dataset.
Sec. H provides more experimental results on the 2D MMWHS [83] dataset.
Sec. provides more experimental results on the 3D LA [84] dataset.
Sec. J provides more experimental results the 3D MP-MRI dataset.
Sec. K provides
experimental results three semantic segmentation benchmarks (Cityscapes [97], Pascal VOC 2012 [98], SUN RGB-D [99]).
Sec. L forms better understanding of generalization across label ratios and frameworks.
Sec. M provides final checkpoint loss landscapes.
Sec. N provides additional details on the hyperparameter selection.
A Theoretical Results Details
A.1 Proof of Lemma 3.1
Proof. By definition of the estimate H. since the sampling from each group is independent, the variance can be written as
Varl
1
Σ
m
n²
pm
Since |Dmll nm, it follows that
M
2 o
Var [ar]
Σ
n2 7m
Note that of is the variance of the ((x;p) where Pm uniformly.
Analysis of SAG. We now consider SAG. Recall that, compared to SG, SAG first samples nmm pixels from P., 6 from each m = 1, 1....,............... ....... .M in the same way as SG. SAG then deterministically picks the rest n2m2 / pixels to be the reflection pixels of the first half (see Figure 2(3)). Let's choose arbitrary group m and denote by and p' the sampled pixel and its reflection from that group. Note that the variance p satisfies VarsAg[M(x; p)] = o since way as in the case of SG. Observe that, by symmetry, the variance of and p' should be the same Varsao (x; p)] VarsAo(/((x;
o It follows that:
Varsag[M(x:/) + (xx:'')
Varsaa (x(x; p)] + Varsaa[h(x;p 2CovsAG (x(x p), h(x:p'))
< 40
FFor simplicity, ignore the rounding issue.
16

---

where the second step holds because the correlation between p) and h(x; is at most 1. It follows that:
M
Var Var[HsAG]
Σ
m
Var
Σ Mxxip)
IP]
1Dm
2
PEDm
M
n
Varlh(xx:) p) + h(x:p))
nta
n
===1
p.p'
M
mm
1lm 40 2
M
2m n²
0² m
m
2 Var Har]
A.2 Proof of Theorem 3.2
Proof of Theorem 3.2. Denote Wm mm/nn, where 7.m |Dml The unbiasedness is straightfor- ward: by proportional sampling, the probability that an arbitrary pixel group P... being chosen equal to //ml//Pmll Since [Dm = 1. the probability of being chosen is equal across all pixels. and hence an arbitrary pixel D Um Dm is equally likely to be any pixel in the population P. As result, we have
E ( D) = E Σ
h(x:p)/ Ep(h((: p)]
PED
Hx),,
where the first equality is by the construction of Hx; D), the second equality is by symmetry of p E D, and the last inequality is by the above conclusion that p is equally possible be any pixel in P. This finishes the proof that AsG unbiased.
The variance of Asg is equal to
M
Var [Asa]
Σ
m n
=
num
where o is the variance of the f(x: p) when p ~ Dm uniformly. Under the case of proportional sampling, i.e. n'm & [Dmll for all m 1,,,,, (for simplicity assume such 72m 's are all integers), the variance becomes
M (Wm n)
M
Var
Wm
n²
Wm
m=1
m==1
We now consider NS (i.e. naive sampling). By definition, the random sampling samples pixels uniformly from the set of all pixels. Since n is the total number of pixels sampled, the variance of the As is given as Var [ar[ = =02/n where o² the sampling variance of h(x; given P uniformly. To determine 2² apply variance decomposition trick via conditioning. Specifically, the sampling from NS is a two-step process: (1) sample the group index m from [M]. (2) sample the pixel uniformly from Pm. Applying the law of total variance, have
Var pm
[1(x; p]]
= [[Va((((((:p))) p)llp E Pmn] + Var[B[M((;p))p E Pm]]
M
M
02
+
Σ
E
[x(x;p)] - E
pmu [h(x::)]]
p
Pm
17

---

As result, we conclude that, for any image x, the sampling variance of SG estimate and NS estimate satisfies
Var (NNss = -Var (HsG)
E wi [h(x:P)]
IE un pMx:
n
which finishes the proof.
A.3
Further Details on Convergence
We first make the following assumptions about the loss function and the gradient estimate in our learning problem.
Assumption A.1 (Smoothness). The objective function L(-) is L-smooth, i.e, is differentiable and [[V - VL(6")l22 S L(| (2) for all 0, 6' in the domain of V..
Assumption A.2 (Bounded variance). There exists some og > 0 such that for all 0 in the domain,
E [l((0) VC(() S o
These assumptions are standard in optimization theory and in various settings [87, 88, 89]. The first assumption allows to characterize the loss landscape of our learning problem, and the second assumption ensures the gradient estimate does not deviate largely from the truth, which is essential for the training to converge. With these two assumptions, it is guaranteed that
T EE S (
T
0 (TV
Proposition 1. Suppose the step sizes in SGD taken to be min { a where T is the number of steps. Then under Assumption A.l and A.2, there exists some constant C depending on C. a, and L, such that
1 T
og T VT
Proposition shows that, in expectation, the convergence rate of the average gradient norm consists of fast rate 1/T and slow rate og / with the slow rate being the dominant term. Importantly, the slow rate depends on the variance of the gradient estimate. This suggests that variance reduced gradient estimate allows SGD reach an approximate local minimum with less iterations, leading to faster convergence of the training loss. This intuition is corroborated by Fig. 5, where we directly visualize the training trajectory of contrastive loss versus epoch for three sampling methods. We observe that SG features faster loss decay and a narrower error bar, showing that it outperforms other methods in both the convergence speed and the stability. Furthermore, Proposition indicates that our proposed sampling techniques are universal, since applies all scenarios as long the mild assumptions A. and A.2 are satisfied. In other words, when comes to scenarios that involve pixel/voxel-level sampling on 2D/3D images, utilizing SG/SAG instead of naive sampling can lead to enhanced stability and decreased variance.
B Datasets
Our experiments conducted on five 2D/3D representative datasets in semi-supervised medical image segmentation literature, including 2D benchmarks (i.e., ACDC [81], LiTS [82], and MMWHS [83]) and 3D benchmarks (i.e., LA [84] and in-house MP-MRI).
The ACDC dataset was collected from MICCAI 2017 ACDC challenge [81], consisting 200 3D cardiac cine MRI scans with 3 classes left ventricle (LV),, myocardium (Myo), and right ventricle (RV). Following [53, 10], we utilize 120, 40, and 40 scans for training,
18

---

validation, and testing" 7 Note that 1 1%, 5%, and 10% label ratios are the ratio of patients. The splitting details are in the supplementary material. For pre-processing, we normalize the intensity of each 3D scan into [0. 1] by using min-max normalization, and re-sample images and segmentation maps to 256 x 256 pixels.
The LiTS dataset was collected from MICCAI 2017 Liver Tumor Segmentation Challenge [82], consisting of 131 contrast-enhanced 3D abdominal CT volumes with 2 classes - liver and tumor. Note that 1%, 5%, and 10% label ratios are the ratio of patients. We utilize 100 and 31 scans for training and testing, with random order. The splitting details are in the supplementary material. For pre-pree mre-proesssing we follow the setting [6] by truncating the intensity of each 3D scan into [-200, 250] HU for removing irrelevant and redundant features, normalizing each 3D sean into [0, 1], and re-sample images and segmentation maps to 256 x 256 pixels.
The MMWHS dataset was collected from MICCAI 2017 challenge [83], consisting of 20 3D cardiac MRI scans with 7 classes- left ventricle (L)), left atrium (LA), right ventricle (RV), right atrium (RA), myocardium (Myo), ascending aorta (AAo), and pulmonary artery (PA). Note that 1%, 5%, and 10% label ratios are based on the ratio of patients. Following [17], we utilize 15 and 5 scans for training and testing. The splitting details are in the supplementary material. For pre-processing, we normalize the intensity of each 3D sean into [0. 1] by using min-max normalization, and re-sample images and segmentation maps to 256 x 256 pixels.
The LA dataset [84] was a representative 3D benchmark, consisting of 100 gadolinium- enhanced MRI scans with one class left atrium (LA), with an isotropic resolution of 0.625 x 0.625 x 0.625mm². Note that 1%, 5%, and 10% label ratios are the ratio of patients. The fixed split (i.e.. 5%, and 10% ) [8, [3] uses 80 and 20 scans for training and testing $ and 1% label ratio randomly split. The splitting details are the supplementary material. For pre-processing, we crop all the scans at the heart region, normalize the intensities of each 3D scan into [0, 1]. and randomly crop all the training sub-volumes into 112 x 112 x 80mm².
The Multi-phasi MRI (MP-MRI) dataset was an in-hous 3D dataset, consisting of 160 multi-phasie MRI scans with one class - liver, each of which includes T1 weighted DCF-MRI images at three-time points (i.e. pre-contrast, arterial phase, and venous phases). Three images are mutually registered to the arterial phase images, with an isotropic voxel size of 1.00 x 1.00 x 1.00mm². The dataset is randomly divided into 100 scans for training. 40 for validation, and 20 for testing. Note that 1 1%, 5%, and 10% label ratios are the ratio of patients. The splitting details are in the supplementary material. For pre-proe preprrcessing, we normalize the intensity of each 3D scan into 00, 1J by using min-max normalization, and re-sample images and segmentation maps to 256 x 256 pixels.
C Implementation Details
In our experiments, all of our evaluated methods have been trained using similar settings for simplicity in reproducing our results. All experiments are conducted with PyTorcc [101] on an NVIDIA RTX 3090 Ti. We adopt an SGD optimizer with momentum 0.9 and weight decay 10-4. The initial learning rate is set to 0.01. We use the 2D UNet [30] or 3D VNet [78] backbones as the segmentation network under different labeled ratio settings (i.e., 1%, 5%, 10% labeled ratios). Following [17], we adopt the feature pyramid network (FPN) [102] architecture as the representation head v; with a separate 512 512-dimension output layer. The momentum hyperparameter is set to 0.99.
For pre- pre-training, the networks are trained for 100 epochs with a batch size of 6. We apply the mined views with d= 5, following [17]. As for fine-tuning, the networks are trained for 200 epochs with batch size of 8. The learning rate decays by factor of 10 every 2500 iterations during the training. We apply the temperature with t. = 0.01, 7. = 0.1, and 0.5,, respectively. The size of the memory bank is set to 36. For the CL training, we use the implementation from [17] and leave all parameters on their default settings, e.g., we apply the hyperparameters with A = 0.01, Ag = 1.0, and A = 1.0. For other pyper-parameters in all the evaluated methods, we adopt the suggested settings in the original papers because they are not of direct interest us.
www.tooaaoooeeceecee http://ia..aaaas/.....s/////eaaa hattps://githin wwt.c//ww//////////// http:///s////eeettte//aaa/tttttt
19

---

Table 4: Ablation on data augmentation strategies: (1) random rotation; (2) random cropping: and (3) horizontal flipping, compared to our methods with two settings (i.e., w/ no augmentation and w/ all three augmentation). Note that we use the identical data augmentation strategies (i.e., random rotation, random cropping, and horizontal flipping), as [30, 90, 91, 92, 93, 8, 94, 9, 57, 95, 62, 50, 59, 96, 53, 16, 17] for fair comparisons.
Method
DSC[
ASD[voxel]]
ARCO-SAG (ours)
84,9
1.47
w/o random rotation
84.1
wlo random cropping
84.4
3.47
wo horizontal flipping
73.7
6.70
no augmentation
70.8
9.83
ARCO-SE SG (ours)
85.5
0.947
w/o random rotation
83.8
3.27
w/o random cropping
84.6
1.62
w/o horizontal flipping
78.8
4.15
no augmentation
76.2
7.74
For 2D medical segmentation, we follow the same data augmentations [9, 53] to the teucher's input and the student's input, respectively. The augmentations include random rotation, random cropping. and horizontal flipping. For 3D medical segmentation, we follow the same data augmentations [8, 9, 96, 53] to the teacher's input and the student's s input, respectively. The augmentations include random rotation and random flipping. We evaluate our methods on 3D segmentation results with two classical metrics: (1) Dice coefficient (DSC) and (2) Average Symmetric Surface Distance (ASD). Note that, for all the evaluated methods, we make no additional modifications during the training process for fair evaluations. We run all our experiments in the same environments with fixed random seeds (Hardware: Single NVIDIA GeForce RTX 3090 GPU; Software: Py 1.10.2+cul 13, and Python 31).
D 2D/3D Methods in Comparison
2D Medical Segmentation: For experiments, we use 2D UNet [30] as backbone, and compare ARCO
with previous state-of-the-art medical segmentation methods: (1) Buseline (i.e., UNet-F/UNet-L
[30]) using both fully-super and suitee--uperviied settings; and (2) SSL-based: EM [90], CCT [91], DAN [92], URPC [57]. DTC [9], DCT [93], ICT [95], MT [59]. UANT [8]. SASSNet [94], CPS [62], GCL [50], MC-Net [96], SS-Net [53], ACTION [16]. and MONA [17]. Note that among all the above evaluated methods, several methods use a contrastive objective, including GCL [50], SS-Net [53], ACTION [16], and MONA [17].
3D Medical Segmentation: For experiments, we use 3D VNet [78] as backbone, and compare ARCO with previous state-of-the-art medical segmentation methods: (1) Buseline (i.e., WNet-F/VNet-L (78)) using both fully-supervised and supervd- settings; and (2) SSL-based: EM [90], CCT [91], DAN [92]. URPC [57], DTC [9], DCT [93], ICT [95], MT [59], UAMT [8], SASSNet [94], CPS [62], GCL [50], MC-Net [96], SS-Net [53], ACTION [16]. and MONA [17]. Note that among all the above evaluated methods, several methods use contrastive objective, including GCL [50], SS-Net [53]. ACTION [16], and MONA [17].
E Framework Overview
In the following, we provide concise overview of ARCO, consisting of two training phases: (1) relational semi-supervised pre-training, and (2) anatomical contrastive fine-tuning, Note that, in this paper, sampling strategies in pixel-level CL framework are of direct interest, so we use simplification of MONA [17] without using any additional complicated augmentation strategies.
(1) Relational Semi-Supervised Pretraining. Given an unlabeled sample, ARCO has two ways to define augmented and mined views: (1) ARCO augments the samples to be x and x² as augmented views, with two separate data augmentation operators; and (2) ARCO randomly samples mined
views (i.e., x3) from the unlabeled dataset with additional augmentation. The pairs [x xx] are
then processed by [F,, Fi). and in similar way x3 is processed by F, (See Figure 1(a) in main
paper), outputting three global features [h', h² h3] after E and their local features [f] f2 f3] after
20

---

Table 5: Quantitative comparisons (DSCI%) / ASD[voxel] 4) across the three labeled ratio settings (1%, 5%. 10%) on the LiTS benchmark. All experiments are conducted as [30, 90, 91, 92, 93, 8, 94, 9, 57, 95, 62, 50, 59, 96, 53, 16, 17] in the identical setting for fair comparisons. Best and second-best results are coloured blue and red, respectively. UNet-F (fully-superviied)) and U (semi-supervided)
considered as the upper bound and the lower bound for the performance comparison. Please refer to the text for discussion. We adopt the identical data augmentation (i.e., random rotation, random cropping, and horizontal flipping) for fair comparisons.
LITS
Labeled
Labeled
Labeled
Method
Average
Liver
Lesion
Average
Liver
Losion
Average
Liver
Loaion
UKes-P [30]
68.2/ 16.9
90.6/8
8/25.6
68.20 08.2/16.9
90.6/8.14
8/25.6
68 68.2/16.9
90.678.14
8/2566
Utlet-L.
50.8/30.4
76.5/15.2
25.0/45.6
63.4/30.4
90.5/9.84
36.4/50.9
64.6/28.3
88.4/18.6
40.9/37.9
EM 1901
56.6/38.4
86.4/26.3
26.9/50.5
61.2/33.3
87.6/9.47
34.7/57.1
62.9/38.5
87.4/21.3
38.3/55.7
OCT [91]
52.4/52.3
79.1442.0
15.7662.6
61.4/26.1
84.6/12.3
38.2/39.9
63.4/23.0
88.8/7.64
38.0/38.5
DAU [921
1/.1/2
84.5/19.2
29.6/37.4
62.3/25.8
88.6/9.64
36.1/42.1
63.2/30.7
87.3/15.4
19.1/46.1
URPC
55.5/34.6
83.0/27.3
28.0/42.0
60.9/21
85.5/8.11
36.3/34.6
62.5/24.8
84,9/12.9
40.1/36.7
DTC [9]
39.3/37.5
68.1/10.7
10.5/64.3
59.2/18.6
85.077,54
33.3/29.7
62.5/24.8
84,9/12.9
40. 1/36/
DCT 1931
57.7/38.5
87.0/22.4
28.3/54.6
60.88/34.4
89.2/12.6
32.5/56.2
61.9/31.7
86.2/19.3
37.5/44.1
ICT [95]
58.3/32.2
86.5/22.8
30. 1:415
60.1/29.1
86.8/12.6
33.3/65.6
62.5/32.4
88.1/16.7
36.9/48.2
NT [59]
54.7/24.7
83.1/10.2
26 31391
60.9/23.7
37,511634
34,4141,1
62.3/23.7
88.5/9.32
36, 1/38.1
UAMT [8]
55.5/34.6
83.0/27.3
28 28.0/42.0
61.5/24.7
84.5/10.6
18.6/38.8
62.9/23.6
87.4/7.78
SASallet [94]
39.642.7
69.0V14.7
10.377
60.4/25.3
86,1/11.6
34.7/39.0
62.4/21.1
86,418.31
38. 3/3399
CPS [62]
57.7/39.6
87.0/22.4
28.3/54.6
59.5/26.3
84.0/901
34,9/43.5
62.1/30.8
88.6/18.3
35 6/4,3
CCL [50]
59.3/29.5
88.6/14.2
30.0/44.9
63.3/2.11
90.7/9.46
35.9/30.8
65.0/37.2
91.3/10.0
18, 7/64.1
MC-Ket [96]
60.9/32.1
87.1/17.8
8/46.5
61.6/198
86,318.11
36.9/31.4
63.4/29.9
89.0/13.1
0/4.8
SS- [53]
55.0/35.9
89.6/19.8
20.5/51.9
59.1/24.6
87.5/11.2
30.6/38.1
63.4/19.8
91.1/7.33
8538/32.2
ACTION [16]
61.0/24.5
89.8/16.9
32.2/32.3
66.3/23.6
93.066.1
39.5/40.8
67.2/20.4
92,8/5,08
41 6/35.8
XOKA [17]
62.2/24.9
91.3/13.9
34.0/35.9
66.6/16.6
93,1/7,74
40.1/25,4
68.3/18.0
93,47888
43.3/27.0
ARCD-SAC (ours)
1/20/
91.577.63
36 8/33.5
67.
93.077.10
41.7/19.3
69. 4/14.9
93,377.60
5/22,2
ARCO-SG (ours)
65.5/20.2
92.515.59
38.4/3,77
68.4/T1.3
93.7/6.63
43.0/16.0
70.1/13.5
94,1/5.01
46.1/22.1
Ground Trudh
GCL
MCN
NN
ACTION
MONA
AACO-SAG (ours)
ARCOSSG (ouns)
Figure 6: Visual results on LITS with 1 1%, 5%, 10% label ratios. As is shown, ARCO-SG and ARCD-SAC consistently produce more accurate predictions on anatomical regions and boundaries compared to all other SSL methods.
D. These features are fed to the two-layer non-linear projectors for outputting global and local embeddings Vg and Vi.
To alleviate the collapse issues [26, 27, 16], we make the architecture asymmetric between the student and teacher pipeline by further feeding both the global and local embeddings with respect the studens branch into the non-linear predictor, producing w in both global and local manners" " After passing through the nonlinear projectors and predictor, the relational similarities between augmented and mined embeddings are computed using the softmax transform, which can be formulated as:
exp sim( w .v²) /r,)
exp(sim( ( w? vv'))/)))
u, = log > exp( sim (w'.va) /T.) U = log دل exp(sim sim (w2.va) va /T.)) where Ta and Te are different temperature parameters. The unsupervised instance discrimination loss (i.e., Kullback-Leebler divergence K.)) can be defined as:
Lisst = KCC(u.llu)..
(E.1)
We omit details of local instance discrimination setting for simplicity in following contexts.
21

---

The parameters of the teacher model are the exponential moving average (EMA) of the student model parameters that are updated by the stochastic gradient descent. For pretraining, the entire loss consists of the global and local instance discrimination loss, and supervised segmentation loss Cxup (iee equal combination of Dice loss and cross-entropy loss), i.e., C
inst + Cinsal
+ Ssup
(2) Antomiica Contrastive Finetuning (ACF). We use the pre-trained network weights as the initialization for subsequent fine-tuning (See Figure ((b) in Main Paper). For sumi-supereise training, follow two principles described in [17] 10, (I) tailness: giving more importance to tail class hard pixels; and (2) diversity: ensuring anatomical diversity in the set of different sampled images.
Following the abovementioned principles, we employ a two-step routine: (1) Tailness: we first perform anatomical contrastive formulation. Specifically, we additionally attach the representation head u/, II and generate a higher n-dimensional dense representation with the same spatial resolution as the input image. A pixel-level contrastive loss is designed to pull queries r, E to be similar to the positive keys r E R. and push apart the negative keys rig E.. The semi-supervised contrastive loss Leuntrast is defined as:
exp(r, /T)
Gonntratt
-Σ - log GEC GEC Rg
(E.2)
exp(ry Fx 1 //// + Ei. exp(r, rT//T)
where C is set including all available classes in the current mini-batch, and T is a temperature hyperparameter. We refer to Rg Rgg rf. class c, a negative key set including all representations whose labels is not in class and the positive key which is the c-class mean representation, respectively. Consider coordinates with the same resolution as Rt these queries and keys are then defined as:
query set including all representations within this
set including all pixel
RS- = U -c) =c)
Rg U py ())
F(i.)) Ik c.t
Σ Tg. P,ERS
(E.3)
R 9
[AJEAA
[5.]EA
(2) Diversity: we leverage the first-in-firt-out (FIFO) memory bank [13] to search for K-nearest neighbors, and use the semi-super nearest neighbor loss L.n in way that maximizing cosine similarity, to exploit the inter-instance relationship.
For fine-tuning, the total loss includes contrastive loss Ccontrast nearest neighbors loss Lun, unsuper- vised cross-entropy loss Lunsup and supervised segmentation loss Saap Loup + X Ccontrass Aunsuppp Aghan- See Appendix N for the ablation of hyperparameters.
F Model Architecture
Figure provides an overview over our approach. Our semi-supervised segmentation model F takes an 2D/3D medical image x as input and outputs the segmentation map and the representation map. We leverage MONA pipeline [17] including two stages: (1) relational semi-supervised pre-training: on labeled data, the student network is trained by the ground-truth labels with the supervised loss Sapp while on unlabeled data, the student network takes the augmened and mined embeddings from the EMA teacher for instance discrimination Linst in the global and local manner, (2) our proposed anatomical contrastive reconstruction fine-tuning: on labeled data, the student network trained by the ground-trute -tuth labels with the supervised loss Lupp while on unlabeled data, the student model takes the representation maps and pseudo labels from the EMA teacher give more importance to tail class Ccontrast exploit the inter-instance relationship Lan- and compute unsupervised loss Lumap- See Appendix M for details of the visualization loss landscapes.
G More Experiment Results on LITS
Figure 6 and Table 5 show qualitative and quantitative results, where our ARCO-SG and ARCO-SAG provide better segmentation results than all other methods. This clearly demonstrates the superiority of our models.
IIn this paper. sampling strategies in pixel-level contrastive learning frameworks are of direct interest, so we use a simplification of MONA [17]- - without using any additional complicated augmentation strategies.
The representation head is only applied during training, and is removed during inference
22

---

night Vortnol
Arnum
Aight (UA))
Myocardum (Myo)
Accending (AA)
Lef Sherncle (IV)
GroundTruth
GCL
HCNe
MONA
(ours)
(ours)
Figure 7: Visual results on MMWHS with 1%, 5%, 10% label ratios. As is shown, ARCO-SG and ARCO-SAG consistently produce more accurate predictions on anatomical regions and boundaries compared all other SSL methods.
Labeled Dars
GroundTrut Trush
ARCO-SG
Figure 8: Visual results on LA with 1%, 5%, 10% label ratios. As is shown, ARCO- SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
H More Experiment Results on MMWHS
We the baselines and our methods on the third medical image segmentation dataset (i.e., MMWHS [83]) under various label ratios (i.e., 1 %, 5%, 10%), reporting results in Table 6 and Figure 7. This clearly demonstrates the effectiveness of models.
I More 3D Experiment Results on LA [84]
We run the baselines and our methods on the fourth 3D medical image segmentation dataset (i.e., LA [84]) under various label ratios (i.e., 1%, 5%, 10%). reporting results in Table 9 and Figure 8. This clearly demonstrates the robustness of our models.
J More 3D Experiment Results on MP-MRI
We run the baselines and our methods the five 3D medical image segmentation dataset (i.e., MP-MRT) under various label ratios (i.e., %, 5%, 10%), reporting results in Table and Figure This clearly shows the superiority of our models.
23

---

LNz
and sooee 200
Ground Truth
GCL
MCN
Net
ACTION
MONA
ARCIDSAG (ours)
ARDO-SG (ours)
Figure 9: Visual results on MP-MRI with 1%, 5%, 10% label ratios. As is shown, ARCO-SO SG and BAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
K More Experiment Results on Semantic Segmentation
To further validate the effectiveness, we experiment on three popular segmentation benchmarks (i.e., Cityscapes [97], Pascal VOC 2012 [98], indoor scene segmentation dataset SUN RGB-D [99]) in the semi-supervised full-label settings. We follow the identical setting [100] to sample labelled images to ensure that every class appears sufficiently our three datasets, i.e., CityScapes, Pascal VOC, and SUN RGB-D. Specifically, we aim to have the least frequent class appear in at least 5, 15, and 50 images in each dataset. respectively. represented in our labeled images, while in Pascal VOC and SUN RGB-D we have at least 3 and semantic classes, respectively. We compare our ARCO-SG and ARCO-SAG under various grid settings (i.e., 9, 16, 25) to baselines (supervised and cemi-supervised ReCo). Table 7 shows the mean IoU validation performance. We can see that performance, compared to ReCo, in all the semi-supervised settings. For example, under the fewest label setting each dataset, compared to ReCo, applying stratified group sampling (SG) can improve results by an especially significant margin, with up to 2.4 7.8% relative gains. As shown in Pascal VOC 2012 [98] - 60 labels (Figure 10), 200 labels (Figure 11), 600 labels (Figure 12), Cityscapes [97] 20 labels (Figure 13), 50 labels (Figure 14), 150 labels (Figure 15), SUN RGB-D [99] 50 labels (Figure 16), 150 labels (Figure 17), 500 labels (Figure 18), ARCO-SG and ARCO-SA4 SAG consistently
CityScapes, we have least 12 semantic classes
cases, ARCO-SG and ARCO-SAG consistently improve
yield more accurate and sharper boundaries compared to ReCo. All those clearly demonstrate the superiority our models.
L Generalization across Label Ratios and Frameworks
Besides generalizing well across different datasets and diverse labeled settings, we additionally evaluate the performance SG and SAG sampling coupled with different CL. frameworks (i.e., MoCov2 [103]. kNN-MoCo [104], SimCLR [14], BYOL [26], ISD [27], VICReg [28]) ACDC under
24

---

various label ratios (i.e,, 1%, 5%, 10%). In this work, we mainly study the state- fftte-of-thee ooftthe-rt art CL frameworks from the computer vision domain hereinafter for ablations, considering their superior performance in the task of image classification. Note that, for each fair comparison, we strictly follow the default setting in these CL. frameworks [103, 104, 14, 26, 27, 28] pretraining, and fine-tune each networks using the same settings in Appendix C. Full experimental details are in Appendix C.
Since they work orthogonally to our pretraining strategy, we conduct a comprehensive comparison of these CL-based frameworks in Table 8. Clearly, our proposed SG and SAG sampling help significantly improving the segmentation performance across all the CL-based frameworks and is capable of being integrated with previous frameworks for further enhanced model robustness. Moreover, it important observe that the performance benefits of our methods increase significantly with lower label setting. This observation augments the necessity of our proposed methods while training networks with high label efficiency.
M Final Checkpoint Loss Landscapes
From visualizations in Figure 19, ARCD-SG converges to much flatter loss valleys, which evidences their robustness in learning anatomical features.
N Ablation on Different Training Settings
Hyperparameter Selection. For grid search, we detail the tuning steps here. The tuning is done in sequential order. Aι is chosen from {0.001, 0.005, 0.01, 0.05, 0.1, 1.0]. and X2 A3 are chosen from {0.1, 0.2, 0.5, 1.0, 2.0, 10.0). We use the validation set to search over hyperparameters and find ACDC with 1 % labeled ratio. As shown in Figure 20, with carefully
the best hyperparameter tuned hyperparameters =1=0.01, A2=1.0, and Ay=1.0, such setting achieves superior performance compared to others.
25

---

Table 6: Quantitative comparisons (DSCI&) ASD[voxel] 4) across the three labeled ratio settings (1%, 5%, 10%) on the MMWHS benchmark. All experiments are conducted as [30, 90, 91, 92, 93, 8, 94, 9, 57, 95, 62, 50, 59, 96, 53, 16, 17] in the identical setting for fair comparisons. Best and second-best results are coloured blue and red, respectively. UNet-F (fully-supervided) and U (semi-supervided) are considered as the upper bound and the lower bound for the performance comparison. Please refer the text for discussion. Note that, Left Ventriele - LV, Right Ventricle
+ RV, Left Atrium + LA, Right Atrium - RA, Myocardium + Myo, Ascending Aorta - AA, Pulmonary Artery + PA. We adopt the identical data augmentation (i.e., random rotation, random cropping, and horizontal flipping) for fair comparisons.
Method (1% Labeled)
Average
L.V
RV
LA
RA
Myn
AA
PA
Ullet [30]
85.8/8.01
87.044.11
79.5/14.7
92.7/4.96
81.6/13.1
83.9/9.32
95.0/3.33
$1.1/6.46
UllttL
58.3/33.9
73.5/28.2
57.9/37.6
74,9/30.2
47.2/65.4
61.9/27.8
74.0/18.7
18.6/29.4
EM [90]
59.5/63.2
71.7/53.0
64.7/26.3
66.6/51.5
61.7/39.4
56.8/6600
81.9/8888
12.9/157.2
OCT [9]]
62.8/27.5
78.1/18.3
62.8/45.1
83.0/18.4
45.3/69. 9
67.0/18.9
76.3/10.6
26.7/1 1.2
DAN [92]
63.8/39.0
76.3/15.4
62.2/58.3
68.0V25.3
52.348.2
57.0/54.3
89.0/28.0
42.0/43.4
URPC [57]
65.7/29.7
77.0/29.5
1/33.4
87.6/18.5
52. 1/56.3
65.8/28.5
8/.5/22.7
27. 27.3/29.7
DTC [9]
62.9/32.3
78.0/29.6
63.6/29.4
74.6/30.8
49,4/53.5
67.3/28.1
89.0/10.0
18.2/44.4
DCT [93]
60.0/35.3
72.1/316
$5.6/42.9
79.0/24.0
54.6/65.7
62.7/31.4
67.3/26.7
29.2/35.3
1951
59,9/32.8
77.1/15.5
53.4/418
79.7/25.4
44.1/69.0
62.9/28.5
1/.1/20.8
28.4/28.8
HT [59]
61.3/36.0
70.7/37.5
62.8/29.0
74.0/49.6
$2.8/58.5
58.7/27.0
85.3/13.0
24.9/37.5
UANT
61.1/27.6
75.1/319
60.3/49.2
79.5/30.7
50.8/11
62.5/33.1
76.6/11.7
23.0/25.0
SASStlt [94]
62.5/33.9
76.6/27.66
62.8/29.0
74.0/49.6
$2.8/58.5
58.7/27.0
85.3/13.0
24.9/37,5
CPS [62]
69.2/22.6
17.9/20.1
67.9/25.3
87.8/18.0
67.6/27.7
66.7/14.2
91.5/20.3
8/3.52
GCL [50]
71.6/20.3
83.5/10.9
66.0/31.7
91.2/55.0 29
58 9/48.0
73.2/16.9
89.8/4,59
38. 2/2.8
NC-Net: [96]
65.4/56.9
67.699.00
59.6/45.8
68.8/29.6
61.7/22.8 8
53.8/56.8
84.4/80.5
61.9163.6
S8-Ket [53]
60.6/30.7
67.5/15.5
66.0/42.5
73.2/25. 1
$3.7/52.9
59.5/15.0
81.0/29.5
23.2/34.5
ACTION [16]
78.2/10.5
88.2/3.95
71.0/11.3
89/4/3300
1/325
79.08.52
83.2/3.82
62.3/0.2
MOEA [17]
#2.2/8.05
89.2/3.64
75.8/6.98 98
89 89.9/3.31
77.6/27
79.5/3.70
94,4/2.72
6/.4/861
ARCD-BAG (ours)
1/8.78
89.9/3.54
80.9/8.79
92. 1/3.22
83. 1/24.1
84.07.36
94.8/5.41
77.5/9:02
ARCO-SG (ours)
87.N5.79
90.7/3.43
52.5/5.58
92.9/3.13
86.2/15.0
85.6/2.89
95.2/2.85
77.9/7.62
Method (5% Labeled)
Average
LV
RV
LA
RA
Myo
AA
PA
Ulet-FF [30]
85.8/8.01
87.0/4.11
79.5/14.7
92.7/4.90
81.6/13.1
83,99.32
95.0/3.3
81.1/6.46
Utiet-L
1,
78.5/31.2
69.0/25.6
78.
57.0/45.3
69.4/34.5
90.2/14.5
63,9/42.5
EM [90]
80.6/17.3
82.0/22.7
75.3/26.5
87.9/19.2
74.8/19.1 72.8/19.4
94. 1/6.52
77.6/7.68
OCT [9]]
79.0/21.9
82.9/15.3
73.5/31.6
83.3/14.4
75.5/34.0
77.9/24.3
92.7/17.6
67.0/1..1
DAN [92]
79. 79.4/22.7
80.1/37.0
77.2/30.0
83.1/12.3
74.4/13.9
78.9/27.4
92.2/28.9
69.5/22.7
URPC [57]
76.3/25.5
84.2/18.8
71.3/25.2
78.5/18.2
63.4/39.4
72.0/24.9
93.5/17.33
71.034.9
DTC [9]
76.4/21.3
82.3/17.2
72.4/24.4
76.1/15.2
65.0/31.8
75.2/20.9
92.8/13.1
70.8/26.7
DCT [93]
80.8/23.0
84.0/45.4
75.7/26.0
87.9/12.1
73.9/31.7
77.2/34.1
94.6/5.26
72.5/6.45
ICT [95]
77.9/18.8
84.1/13.2
76.7/26.3
74.2/14.0
66.5/24.9
1/18/8
94.2/6.21
70.3/28.3
MT [59]
77.5/24.2
83.5/15.4
72.8/29.0
78.0/16.2
68.9/39.2 2
74.7/24.4
93.3/11.8
71.1/33.4
UAMT [8]
16.2/21.1
83,4/20.7
71.5/24.4
77.014.66
62.8/30.6
75.8/22.1
93.0/8.91
69.7/26.0
SASSNet [94]
75.2/25.0
80.9/25.0
70.8/31.2
80.0/17.0
61.4/4..0
70.0/31.3
92.4/18.9
71.0/21.9
CPS [62]
78.3/22.5
83.0/29.6
68.8/27.7
85.0/20.4
1/15.5
71.9/35.2
94.7/9.05
71.9/20.2
GCI. [50]
83.5/7.41
86.4/4.72
78.5/9.79
6/4.44
8/.8/12.1
81.4/8.07
93.5/3.91
76.4/8.95
[96]
78.5/23.9
83.7/23.8
74.4/25.7
819/16.8
70.8/38 4
74.4/23.9
93.3/14.2
71.1/25.0
S8-Ket [53]
78.0/25.2
83.0/7.76
74.8/32.0
82.3/2.3
69 6/42.6
11.1/
92.4/25.5
73.2/32.1
ACTION [16]
85.4/6.71
88.2/3.09
78.8/9.66 66
90.5/2.84
80.6/15.6
84,4/7.37
94.0/2.56
81.3/5.86
NOKA [17]
87.3/6.62
90,2/2.92
80.9/9.22
92.8/2.65
82.5/15.2
87.0%8.3
95.3/1.86
82.7/5.60
ARCD SAG (ours)
88.6/6.73
9/.1/2.70
92.9/2.84
84.3/16.1
89.044.88
95.9/1.57
33.3/6.19
ARCO-8G (ours)
89,3/4.8
91.2/2.70
84.6/8.30
93.7/2.49
85.6/10.4
89.2/3.41
96.0/1.42
84.7/4.95
Method (10% Labeled)
Averago
LV
RV
LA
RA
Myo
AA
PA
UNet-F [30]
85.8/8.01
87.04.11
79.5/14.7
92.7/4.96 96
81.6/13.1
83.9/932
95.0/3.33
81. 1/6./6
Ullet-I.
77.8/19.7
82.8/8922
77.3/16.1
75.9/22.8
74.6/24.7
75.3/11.6
90.8/21.6
67.8/32.3
EM [90]
82.1/15.1
86.7/19.8
78.4/24.5
88. 18.1/77.4
77.6/15.9
75.8/25.1
95.0/4.13
73.2/8.86
[91]
79.4/16.3
85.4/5.65
73.5/30.0
89.1/7.10
68.2/51.2
70.2/24.3
92.4/5.90
77.1/9.77
DAN [92]
80.2/15.0
81.6/22.6
74.2/21.2
85.0/10.1
75.5/17.1
76.9/20.2
94.0/4.40
74.4/9.15
URPC [57]
81.9/2.3
11.1/941
68.3/20.7
18.1/6.73
76.6/14.3
80.4/19.6
94.5/4.26
77.2/11.0
DTC [9]
79.5/20.6
82.8/10.8
75.8/18.7
85.9/13.8
75 15.15
74.4/13.9
90.7/25.1
71.7/20.1
[93]
82.8/12.5
85.4/111.6
78.0/23.3
89.0/4.30
79.0/16.5
7
94.3/4.42
78.4/8.00 08
ICT [95]
82.2/12.0
88.4/5.11
75.0/13.55
89 98
75. 2/26./
79.6/20.4
94.9/4,29
73.3/7.30
[59]
79.4/19.8
80.4/24.1
70.3/21.3
86.0/18.0
80.0/17.0
73.3/28.7
92.3/20.9
73.8/8.0 92
UAMT [8]
83.7/14.2
86.7/12.3
80.3/20.6
89.6/8.10
79.5/19.2
79.2/19.6
93.9/10.3
73.8/8.97
SASStet [94]
81.8/15.5
84,9/8.01
78.3/15.9
84.4/12.5
79. 3/27.3
79.0/14.6
93.4/8.30
3/.3/22.3
CPS [62]
1
84.4/9.85
78.5/21.1
85.9/6.61
81.0/18.7
76.4/18.3
93.2/7.04
74.9/10.3
OCL [50]
86.7/8.76
90.5/2.95
81.3/19.6
90.4/4.31
83.1/18.1
86.7/5.84
94,8/2.00
80.3/8.55
3C-Fet [96]
81.9/15.4
85.4/5.78
80.1/17.2
81.5/11.1
79.7/34.1
79.8/10.9
93.1/6.28
73.7/22.4
SS Se-Net [53]
82.3/13.9
85.7/8.80
79.5/17.6
84.1/12.1
80.2/20.0
81.0/14.9
93.6/8.60
72.0/16.1
ACTION [16]
86.1/5.95 93
88.9/3.25
81.3/6.99
89.4/3.13
81.6/14.1
87.8/3.76
94.4/2.53
79.4/7.78
NOEA [17]
87.6/6.83
90.7/2.89
82.8/8 99
91.8/3.48
85.2/15.7
87.2/5.32
94.9/4.32
S0. 10.4/7.13
ARCO-SAG (ours)
89.3/4.42
91.1/2.97
84.8/6.30
94.1/2.38
86. 1/9.3
89.2/2.89
96.0/1.68
83.6/5002
ARCO-SG (ours)
89,444,80
91.6/2.56
88.0/6.79 79
93.9/2.53
86,3/11.
89.6/2.71
95.9/1.66
83,6/5.88
26

---

Table 7: Quantitative comparisons (Intersection Over Union (IoU) 1) for Pascal VOC. CityScapes and SUN RGB-D datasets. All experiments are conducted as [100] the identical setting for fair comparisons. Best and second-best results are colored blue and red, respectively.
Piscal VOC
CityScapts
SUN BGB D
Methed
labels
lobels
labels
all labels
Libels
lobels
150 bhols
labels
501atels
labels
labels
all lahels
Segervised
Classii
57.1
AUUSS-GA (9 Cridy Checddic
58.3
70.5
75.4
50.2
00.2
06.5
31.5
40.5
15.7
1800 SAO Grd) ClassMix
38.7
70.9
15.1
02.1
02.5
63.3
37.8
40.2
63.7
AUD3--5AI Gri) ClooMis
39.1
70.9
74.9
49.8
00.6
65.7
38.5
00.3
05.5
AA20-99 Gndi ClaoMix
32.1
CheeMii
Table Ablation different contrastive learning frameworks with respect to various labeled ratio settings (1%, 5%, 10%). Experiments are conducted on ACDC using UNet [30] as the backbone. Here we report the segmentation performance in terms of DSC[% % and ASD[mm]. On all three labeled settings, incorporating our methods (i.e., SG and SAG) consistently achieve superior model robustness gains compared to naive sampling across different state-of the-thhe-r CL frameworks.
1% Labeled
5% Labeled
10% Labeled
Category
Method
DSC f
ASD 4
DSC
ASD
DSC
ASD
NS
76.4
5.64
83.3
3.78
83.8
3.17
McCov2 [103]
SAG
80.9
4.16
84.7
3.53
84.6
3.84
SG
81.4
4.01
85.1
1.48
84.9
3.08
NS
78.3
4.54
83.8
3.74
2.97
ANN-MoCo [104]
SAG
81.8
4.08
84.6
3.41
84.7
2.58
SG
83.9
3.17
85.4
3.17
85.0
2.47
NS
74.9
4.89
80.9
3.19
84.1
2.78
SimCLR [14]
SAG
78.5
4.01
83.8
2.68
85.8
2.01
SG
3.49
84.3
2.31
86.0
1.76
NS
77.3
5.01
82.9
3.14
84.8
1.67
BYOL [26]
SAG
79.8
4.12
85.0
2.81
85.7
1.36
SG
80.2
3.79
85.8
1.67
85.9
1.13
NS
3.54
81.2
2.86
85.6
1.90
ISD [27]
SAG
81.4
3,50
83.1
2.53
86.3
1.61
SG
82.2
2.04
83.7
1.97
86.7
1.34
NS
64.0
10.6
79.1
4.18
82.9
3.89
VICReg [28]
SAG
81.1
3.49
83.8
3.02
86.3
2.14
SG
81.6
3.12
84.1
2.78
86.8
2.01
NS
82.6
1.43
86.9
1.07
87.7
1.33
ARCO (ours)
SAG
84,9
1,47
87.1
0,848
88.5
1,40
SG
85.5
0.947
88.7
0.841
89.4
0.776
27

---

Ground Truth
Supervised
Reco+ClassMix
Case (60 Labels)
ARCO-SAG (9 Grid)
ClassMix
ARCO-SAG (16 Grid)
ClassMix
ARCO-SAG (25 Grid)
ClassMix
ARCO-SG (9 Grid)
+ ClassMix
ARCO-SG (16 Grid) + ClassMix
ARCO-SG (25 Grid)
ClassMix
Ground Truth
Supervised
Class
Case 2 (60 Labels)
ARCO-SAG (9 Grid) + ClassMix
ARCO-SAG (16 Grid) ClassMix
ARCO-SAG (25 Grid) + ClassMix
ARCO-SG (9 Grid) ClassMix
ARCO-SG (16 Grid) ClassMix
ARCO-SG (25 Grid) ClassMix
Background
Aeroplane
Bicycle
Bird
Boat
Bottle
Bus
Car
Cat
Table
Dog
Horse
Motorbike
Porson
Plant
Shoop
Sofa
Train
Chair
Cow
Monitor
Invalid
Figure 10: Visual results on Pascal validation set with 60 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared all other SSL methods.
28

---

Ground Truth
Supervised
Reco+ClassMi
Case (200 Labels)
ARCO-SAG Grid) ClassMix
ARCO-SAG (16 Grid) ClassMix
ARCO-SAG (25 Grid)
ClassMix
ARCO-SG (9 Grid)
ClassMix
ARCO-SG (16 Grid) ClassMix
ARCO-SG (25 Grid)
ClassMix
Ground Truth
Supervised
Reco+ClassMa
Case 2 (200 Labels)
ARCO-SAG Grid) + ClassMix
ARCO-SAG (16 Grid) ClassMix
ARCO-SAG (25 Grid)
ClassMix
ARCO-SG (9 Grid) ClassMix
ARCO-SG (16 Grid) ClassMix
ARCO-SG (25 Grid) ClassMix
Background
Acroplane
Bicycle
Bird
Bottle
Bus
Car
Car
Table
Dog
Horse
Motorbike
Person
Plant
Sheep
Sofa
Train
Monitor
Invalid
Figure 11: Visual results on Pascal validation set with 200 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
29

---

Ground Truth
Supervised
Recs+ClassMix
Case (600 Labels)
ARCO-SAG (9 Grid) + ClassMix
ARCO-SAG (16 Grid) ClassMix
ARCO-SAG (25 Grid)
ClassMix
ARCO-SG (9 Grid) +
ClassMix
ARCO-SG (16 Grid) + ClassMix
ARCO-SG (25 Grid)
ClassMix
Ground Truth
Supervised
ReasMChassix
Case 2 (600 Labels)
ARCO-SAG (9 Grid)
ClassMix
ARCO-SAG (16 Grid) ClassMix
ARCO-SAG (25 Grid)
ClassMix
ARCO-SG Grid) + ClassMix
ARCO-SG (16 Grid) + Classix
ARCO-SG (25 Grid) ClassMix
Background
Acroplane
Bicycle
Bird
Boat
Bottle
Bus
Car
Cat
Table
Dog
Horse
Motorbike
Person
Plant
Sheep
Sofa
Train
Chair
Cow
Monitor
Invalid
Figure 12: Visual results Pascal validation with 600 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
30

---

Ground Truth
Supervised
Reco+Claassix
Case (20 Label))
ARCO-SAG (9 Grid)
CassMix
ARCO-SAG ChasMix
ARCO-SAG Grid) ClassMix
ARCO-SG Grid)
ClassMbc
ARCO-SG (16 Grid)
ClassMix
ARCO-SG ClassMox
Ground Truth
Super vised
Resss
Gare (20 Labol))
ARCO-SAG
ARCO-SAG ClassMix
ARCO-SAG (25 Grid)
ClassMix
ARCO-SG (9 Grid) ClassMax
ARCO-SG (16 Grid) ClassMix
ARCO-SG (25 Grid) ClassMox
Sidewalk
Building
Wall
Fence
Traffic
Sign
Vegeration
Person
Rider
Car
Truck
Bus
Train
Matorcycle
Bicycla
Undoffned
Figure 13: Visual results on Cityscapes validation set with 20 labels. As is shown, ARCO-SG and ARCO-SAC SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
31

---

Ground Truth
Supervised
Reco+ChassMi
Case (50 Labels)
(9 Grid) ClasMix
Gridj
(25 Grid)
ARCO-SG Grid)
ClassMbc
ARCO-SG (16 Grid) ClassMox
ARCO-SG (25 Grid)
ClassMix
Ground Truch
Supervised
Reco+ClassMx
Cuse 2 (50 Labob)
ARCO-SAG (9 Classixx
ARCO-SAG ClasMix
ARCO-SAG (25 Grid) ClssMix
ARCO-SG (9 Grid) ChasMix
ARCO-SG (16 Grid) ClasMix
ARCO-SG (25 Grid) ClassMix
Building
Traffic
Traffic Sign
Vegetation
Terrain
Sky
Person
Ruder
Car
Truck
Bus
Motorcycla
Bicycle
Figure 14: Visual results on Cityscapes validation set with 50 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
32

---

Ground
Reco+CassMix
Grs (150 Labels)
ARCO-SAG (9 Grid) + ClassMix
ARCO-SAG (16 Grid) ClassMix
ARCO SAG (25 Grid)
ClassMix
ARCO-SG Grij
(16 Grijj
ARCO-SG Gri))
Ground Truch
Supervised
Rics--ClassIx
Case 2 (150 Labelg)
(9 Grid)
ClassMix
ARCO-SAG Grid) ClassMix
ARCO-SG Gri) ClasMix
ARCO-SG (16 Gri) CbssMix
ARCO-SG (25 Grid) ChasMix
Road
Sidewalk
Building
Wall
Fence
Pale
Light
Sign
Vegecazion
Person
Rider
Truck
Bus
Train
Mctorcycle
Bicycle
Undefined
Figure 15: Visual results on Cityscapes validation set with 150 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared all other SSL methods.
33

---

Ground Truth
Supervised
ClassMii
Labels)
ARCO-SAG (9 Grid)
ClassMix ARCO-SAG (16 Grid)
ChasMix ARCO-SAG (25 Grid)
ClassMix
ARCO-SG (9 Grid)
ClassMix
ARCO-SG (16 Grid)
Classix
ARCO-SG (25 Grid) + ClassMix
Ground Truth
Supervised
Reco+ClassMi
2 (50 Labels)
ARCO-SAG (9 Grid)
ClassMix ARCO-SAG Grid) ClassMix ARCO-SAG (25 Grid) ClassMix
ARCO-SG (9 Grid) ClassMix
ARCO-SG (16 Grid) ClassMix
ARCO-SG (25 Grid) + ClassMix
Wall
Floor
Cabinet
Bed
Chair
Sofa
Table
Door
Window
Counter
Desk
Shelves
Curtain
Dresser
Pillow
Mirror
Ceiling
Books
Fridge
Paper
Towel
Curtain
Nightstand
Toilet
Sink
Lamp
Bathtub
Bag
Underdefined
Bookshelf
Picture
Floor Mac
Clothes
Whiteboard
Person
Figure 16: Visual results on SUN RGB-D validation set with labels. As is shown, ARCO-SG and AAG consistently yield more accurate and sharper boundaries compared to all other SSL. methods.
34

---

Ground Truth
Supervised
Reco + ClasMix
Case (150 Labels)
ARCO-SAG Grid)
ClassMix ARCO-SAG (16 Grid)
ClassMix ARCO-SAG (25 Grid)
ClassMix
ARCO-SG (9 Grid)
ClassMix
ARCO-SG (16 Grid)
ClassMix
ARCO-SG (25 Grid) + ClassMix
Ground Truth
Supervised
Reco+ClassMx
Case 2 (150 Labels)
ARCO-SAG Grid) ClassMix ARCO-SAG (16 Grid)
ClassMix ARCO-SAG (25 Grid)
ClassMix
ARCO-SG (9 Grid) ClassMix
ARCO-SG (16 Grid) ClassMix
ARCO-SG (25 Grid)
ClassMix
Wall
Floor
Cabinet
Bed
Chair
Sofa
Table
Door
Window
Counter
Blinds
Desk
Shelves
Curtain
Dresser
Pillow
Mirror
Ceiling
Books
Fridge
TV
Paper
Towel
Bath Curtain
Box
Nighsstand
Toilet
Sink
Lamp
Bathoub
Bag
Underdefined
Bookshelf
Picture
Floor Mac
Clothes
Whiteboard
Person
Figure 17: Visual results on SUN RGB-D validation set with 150 labels. is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
35

---

Ground Truth
Supervised
Reco ClassMix
(500 Labels)
ARCO-SAG (9 Grid) + ClassMix ARCO-SAG (16 Grid)
ClassMix ARCO-SAG (25 Grid)
ClassMix
ARCO-SG Grid)
ClassMix
ARCO-SG (16 Grid)
ClassMix
ARCO-SG Gri))
ClassMix
Ground Truth
Supervised
Reco+ Rlas+Classsix
2 (500 Labels)
ARCO-SAG (9 Grid)
ChasMix ARCO-SAG (16 Grid)
ClasMix ARCO-SAG (25 Grid)
ClassMix
ARCO-SG (9 Grid) ClassMix
ARCO-SG (16 Grid) ChasMix
ARCO-SG (25 Grid)
ClassMix
Floor
Cabinet
Bed
Chair
Sofa
Table
Door
Window
Counter
Blinds
Desk
Shelves
Curtain
Dresser
Pillow
Mirror
Ceiling
Books
Fridge
TV
Paper
Towel
Bath Curtain
Box
Nightstand
Toilec
Sink
Lamp
Bathtub
Bag
Underdefined
Bookshelf
Picture
Floor Mac
Cloches
Whiteboard
Person
Figure 18: Visual results on SUN RGB-D validation set with 500 labels. is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL. methods.
36

---

Table 9: Quantitative comparisons (DSC[%] f ASD[ [vDel) 4) across the three labeled ratio settings (1%, 5%. 10%) on the 3D MP-MRI and 3D LA benchmarks. All 3D experiments are conducted [78, 90, 91, 92, 93, 8, 94, 9, 57, 95, 62, 50, 59, 96, 53, 16, 17] in the identical setting for fair comparisons. Best and second-best results are coloured blue and red, respectively. VNet-F (fully- supervided) and VNet-L (semi-supervided) are considered as the upper bound and the lower bound for the performance comparison. Please refer to the text for discussion.
MP-MRI
LA
1% Labeled
Labeled
Labeled
Labeled (1%)
Labeled (5%)
Labeled (10%)
Method
Liver
Liver
Atrium (I.A)
Left (LA)
Atrium (I.A)
[78]
93.1/5.73
93.1/5.73
93.1/5.73
91.5/1151
91.5/1.51
91.5/1.51
Vlet-L.
68.6/33.4
81.6/12.5
87.9/7.55
40.0/21.2
52.6/9.87
82.7/3.26
EM [90]
73.2/30.1
86,0/15.8
91.9/6.99
48.3/21.3
81.1/4.68
82.7/4.7
CCT [91]
74.2/24.5
86.0/15.8
90.9/9.54
40.3/13.8
70.8/8.22 31
82.0/5.25
DAN [92]
69,4/31.4
38.3/10.2
91.1/8.76
5/22/2
78.8/6.53
80,2/5.37
URPC [57]
72.7/29.9
89.8/9.00
91.7/7.41
65.0%%99
80.2/5.48
83.1/4.57
DTC [9]
88.
89.6/10.1
90.5/11.2
36.2/11.7
83.6/2.81
87.1/2.23
DCT [93]
74.1/31.1
88.2/12.4
90.1/11.1
42.9/19.1
80.1/9.06
80.4/9.18
ICT
72.3/30.5
88.6/11.2
91.178.46
47.7/16.0
98.4/6.96
85,4/4.14
MT [59]
73.8/29.4
87,7/12.8
92.0/7.15
58 1/178
77.0/8.15
82,8/5.90 90
VANT [8]
71.6/31.2
87.1/12.8
91.3/9.71
60.3/11.
82.3/3.82
87.8/2.12
(94]
78.8/19.6
88.4/13.1
88.7/13.1
51.5/14.6
81.6/3.58
87.5/2.59
CPS [62]
80.0/17.1
89.2/10.8
91.0/916
45.1/22.0
79.7/9.26
80.7/5.16
GCL [50]
78.9/14.2
87.9/8.29
90.4/5.68
52.6/12.8
75.5/7 60
84.8/4.22
KC-Net [96]
79,7/20.2
90.1/8.27
92,4/4.4
83.6/2.70
87.6/1.88
SS Net [53]
88.0/8.93
90.9/9.94
92.0/5.67
43.4/14.8
86.3/2.31
88.6/1.90 .9
ACTION [16]
86,5/13.6
90.3/12.3
90.9/10.0
71.1/6.23
86.6/2.24
88.7/2.10
MONA [17]
91.3/5.31
92.2/9.46
92.3/8.16
72.8/10.7
$7.0/2.81
89 39.5/22.40
ARCO-SAB SAG (ours)
91.5/6.82
92.5/6.95
92.6/7.54
73.2/6.47
86.9/2.73
89. 1/2.30
ARCO-SG (ours)
91,6/6.60
92.5/6.31
92.8/664
75.0/4.06
87.8/1.66
89,9/1.47
(4)
151
LiSs
REDOE
(4)
191
Figure 19: Loss landscape visualization of pixel-wise contrastive loss Leentuss with ARCO-SG. Loss plots are generated with same original images randomly chosen from ACDC [81], LiTS [82], MMWHS [83]. LA [84], and MP-MRI, respectively. z-axis denotes the loss value at each pixel.
37

---

1M
Figure 20: Effects of hyperparameters d1, A2, Ag. We report Dice and ASD of ARCO-SG on ACDC with 1% labeled ratio.
38


Reviews:
[{"review_id": "QLH1AMSfoe", "reviewer": "Reviewer_J4c5", "summary": "The authors propose a sampling strategy to improve contrastive-learning-based medical image segmentation performance with limited labeled data and training stability. The sampling strategy is used in conjunction with a previously-published contrastive semi-supervised training strategy. The main contributions include this sampling strategy, some theoretical support that the strategy should improve training stability, and experiments evaluating the method.", "questions": "Questions and suggestions discussed above", "limitations": "Limitations and impacts adequately addressed.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The baseline comparison experiments are thorough.\n- The authors address a real applied problem (that most medical image segmentation datasets have limited data) with a new method and theoretical support, resulting in a well-rounded paper. \n", "weaknesses": "Major weaknesses\n- The paper’s writing is a major limitation. The prose is difficult to understand, which limits the entire paper—it is hard to clearly understand the motivation, the proposed method, the contributions, or the benefits. The work would benefit from more rounds of grammatical revision; it is hard to parse what the author is trying to communicate a lot of the time.\n- Partly due to the writing issues, it is unclear how the proposed sampling strategy differs from previous dense contrastive learning-based image segmentation strategies; it seems that the methodological contributions here are minor, if existent.\n- The method is complex, consisting of two backbones (connected via EMA), global/local instance discrimination losses, augmentations, supervised losses, nearest neighbor loss, global contrastive loss, unsupervised loss… with all of these components, a very thorough ablation section is needed to understand how much the novel component (a pixel-level sampling strategy) is contributing to performance. The existing ablation section is not so thorough. As a result, the paper does not contribute much understanding about the strengths/weaknesses of different components of this complex pipeline.\n\nMinor weaknesses \n- MONA is not a well-known training strategy; it would be useful to provide a longer overview on what MONA does and how the proposed approach differs. This discussion could go in an appendix.\n- The same concept is often referred to using different words: model “convergence,” “robustness,” and “stability.” I’m not sure if you’re always talking about the same concept, or if you are using different words to refer to the same idea. If the latter, it helps the reader to always use the same word.\n- This is a minor stylistic note, but the use of bold and italics is often distracting.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging our contribution, appreciating the strong performance improvement on our medical image segmentation tasks, and providing constructive suggestions for the presentation of our work! We’ve made a substantial revision to the paper, which addresses all the issues, with emphasis on clarity of the explanation of our work and appropriate ablation experiments on different components. If you have further concerns, please feel free to contact us.\n\n> **Q1**: Writing revision?\n\n**A1**: Thank you for the great suggestion! We will thoroughly assess the wording in our current manuscript and polish them accordingly. Here we highlight our motivation and contribution.\nOur motivation lies in the observation that the sampling procedure in contrastive learning introduces an additional source of variance, which could result in model collapse and undermine the overall performance of the model.\nTo this end: we have devised two sampling techniques that (1) are easy to implement, functioning in a plug-and-play manner; (2) theoretically demonstrate variance reduction properties; and (3) empirically deliver improved segmentation quality in 8 benchmarks, i.e., 5 2D/3D medical and 3 semantic segmentation datasets, with different label settings.\n\n> **Q2**: Difference between our sampling strategy and dense contrastive learning (CL) segmentation.\n\n**A2**: Existing dense CL segmentation methods [46,20] use patch-level strategy, e.g., [46] first partitions the image into fixed-size patches (3x3), and then randomly selects 13 patches per image. For each of these chosen patches, they further select 5 negative samples from patches located in different regions of the same image. Thus, they implement dense CL with these local regions, as in Eqn. 3 [46]. In contrast, our SG/SAG first partition the image with respect to different classes into grids with the same size, and then sample, within the same grid, pixels semantically close to each other with high probability. Thus, we implement CL with these sampled different class pixels, as in Eqn. E2 (Appendix Line 766 - 767). This essentially allows us to obtain better segmentation quality and label efficiency by improving feature variance reduction in training, as shown in Sections 4.\n\n> **Q3**: Ablation on different components.\n\n**A3**: Thank you for your constructive feedback! We agree that the importance of providing a comprehensive ablation study to discern the contribution of each individual component, especially the novel pixel-level sampling strategy (Please see **“global” response PDF file** for requested ablations). \n\nTo clarify, our work focuses on semi-supervised medical image segmentation, where we adhere to the widely accepted training standards, such as supervised loss, data augmentation, and EMA training, as referenced in [27,52,5,81,49]. Furthermore, to substantiate the necessity of the various components, we refer to the one-page **“global” response PDF file**. Tables 1,2,3 show the comparative results of various components, including pixel-level sampling variants (i.e., naive sampling (NS), Stratified Group (SG), and Stratified-Antithetic Group (SAG)), contrastive loss, nearest neighbor loss, unsupervised loss, and the global/local instance discrimination losses on the ACDC dataset with a 1% label ratio. These findings demonstrate:\n\n1. The positive contribution of each component to performance gains.\n2. The benefits of employing SG/SAG sampling over the NS setting, resulting in marked improvements in Dice.\n3.  Figure 5 (Line 323 - 347) illustrates how SG/SAG enhance convergence with reduced standard deviations, indicating enhanced robustness.\n\nOur results underscore the efficacy of our proposed method, especially in the realm of medical image segmentation. We are grateful for your valuable suggestions and will ensure these findings and clarifications are prominently featured in our final revision. We remain receptive to any further insights to enhance the quality of our paper.\n\n> **Q4**: Overview MONA.\n\n**A4**: Thank you for pointing this out! Indeed we have conducted a comprehensive review of the MONA framework. Due to limited rebuttal space, please refer to Appendix C (Lines 691 - 716), E (Lines 732 - 777), and F (Lines 778 - 789) for further details. In our final revision, we will follow your constructive advice to transfer the key details from the Appendix to the main paper for better clarity and visibility.\n\n> **Q5**: model convergence, robustness, and stability are unclear.\n\n**A5**: Many thanks! Here we further clarify the terms “convergence,” “robustness,” and “stability” and the difference between them.\nSpecifically, “convergence” is a common concept which simply refers to the training speed, which is the number of epochs required to learn an accurate model.\n“Stability” is a desired property of a sampling method. It refers to the standard deviation of the sampling method. A stronger stability means the sampling method has a small standard deviation, which is good in the pixel sampling procedure of contrastive learning.\n“Robustness” is a comprehensive concept used to describe the performance of a segmentation model. A model is said to be robust if (1) it has a high segmentation quality with only using extremely limited labels in long-tailed medical data (please see Line 141); (2) and fast convergence speed (please see Line 207).\n\n> **Q6**: This is a minor stylistic note, but the use of bold and italics is often distracting.\n\n**A6**:  Thanks for pointing this out. We will modify it accordingly.\n\nWe have extensively revised our manuscript, focusing on enhancing the clarity of our explanations and conducting detailed ablation studies on various components. We trust these adjustments improve the paper's readiness for publication. Please do not hesitate to reach out should you have any additional feedback or questions.\n"}, {"review_id": "eCuWMQim0P", "reviewer": "Reviewer_QWqe", "summary": "This paper proposes two new sampling strategies, SG and SAG, for contrastive learning in semi-supervised frameworks. Compared with randomly sampling pixels for contrastive learning, pixels are grouped into several subsets, and then pixels are sampled from each subset. The proposed method is proven to reduce the variance of sampled pixels. Solid experiments are conducted to support the above claim.", "questions": "See weaknesses.", "limitations": "The authors have included a discussion on the potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Solid experiments and good performance.\n2. Simple yet effective method that benefits contrastive learning for semi-supervised frameworks.\n2. Sound theoretical analysis.", "weaknesses": "1. It would be better to also compare NS with the proposed SG/SAG in natural image datasets.\n2. What is the application scenario of SAG? SG seems to have better performance and stability than SAG.\n3. Line 222-223, what is the meaning of \"p is orthogonal to p'\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging our contribution to the medical image analysis field, appreciating the good performance improvement on our multi-class medical segmentation task, and providing constructive suggestions for the presentation of our work! We’ve made a substantial revision to the paper, which addresses all the issues, with emphasis on experiments in natural image datasets and clarity of the explanation of our work. If you have further concerns, please feel free to contact us.\n\n> **Q1**: It would be better to also compare NS with the proposed SG/SAG in natural image datasets?\n\n**A1**: Thank you for the great suggestion! We agree with your point that it is better to compare NS with the proposed SG/SAG on natural image datasets. Indeed we have compared NS with the proposed SG/SAG on three natural image datasets in Appendix (i.e, Cityscapes [84], Pascal VOC 2012 [85], indoor scene segmentation dataset – SUN RGB-D [86]). All the experiments are conducted under the same experimental setting [87]. For your convenience, the following Table shows the comparison results on Cityscapes, Pascal VOC, and SUN RGB-D (Please see Appendix K line 806 - 824 for qualitative/visual results).\n| | | | Pascal VOC | | | CityScapes | | | | SUN RGB-D  | | |\n| :----------- | -----------: | :-----------: | :-----------: | :-----------: | -----------: | :-----------: | :-----------: | :-----------: | -----------: | :-----------: | :-----------: | :-----------: |\n| Method | 60 labels |120 labels | 600 labels | all labels |20 labels| 50 labels |150 labels | all labels | 50 labels |150 labels |500 labels|all labels|\n| Supervised | 39.4 | 55.5 | 64.6 | 77.8 | 38.2 | 45.9 | 55.4 | 70.9 | 20.0 | 29.2 | 38.9 | 51.8 |\n| ReCo [87] + ClassMix | 57.1 | 69.4 | 73.2 | - | 49.9 | 57.9 | 65.0 | - | 30.5 | 40.4 | 44.6 | - |\n| ARCO-SAG (9 Grid) + ClassMix | 58.3 | 70.5 |75.4 | - | 50.2 | 60.2 | 66.5 | - | 31.5 | 40.9 | 45.7 | - |\n| ARCO-SAG (16 Grid) + ClassMix | 58.7 | 70.9 | 75.1 | - | 50.1 | 60.6 | 66.3 | - | 37.8 | 40.2 |45.7 | - |\n| ARCO-SAG (25 Grid) + ClassMix | 59.1 | 70.9 | 74.9 | - | 49.8 |60.6 | 66.7 | - |**38.5**| 40.5 | 45.5 | - |\n| ARCO-SG (9 Grid) + ClassMix | 59.2|**71.8**| 75.3 | - | 52.5 | 60.9 | **66.8**| - | 32.4 | 41.4 | 46.6 | - |\n| ARCO-SG (16 Grid) + ClassMix |**59.6**| 71.7 |**75.5**| - |**53.7**|61.2| 66.2 | - | 37.7 |41.0 | 46.4 | - |\n| ARCO-SG (25 Grid) + ClassMix | 59.5|71.7| 75.2 | - | 51.5 |**61.8**|66.4| - |38.3|**41.5**|**47.3**| - |\n\nAs we can see, we can see that for all cases, SG and SAG consistently improve performance, compared to NS, in all the semi-supervised settings. The results, quantitatively (Appendix Table 5 Page 25) and qualitatively (Appendix Page 26 - 34) clearly prove the effectiveness of our proposed SG/SAG. Due to the limited response space, we will surely provide detailed discussions on them in our revision.\n\n> **Q2**: What is the application scenario of SAG? SG seems to have better performance and stability than SAG.\n\n**A2**: We greatly appreciate your attention to this matter. SAG, compared with SG, only samples half the number of pixels, and the other half are chosen as the opposite with respect to the group center, making it theoretically more efficient. This is described by the equation $c_m -p = p’ - c_m$ following Line 219. In other words, these two pixels, $p$ and $p’$, are opposite of each other w.r.t. $c_m$. Consequently, this theoretical efficiency suggests a wider scope for SAG in various real-world applications, encompassing fields like 3D rendering, augmented reality (AR), and virtual reality (VR).\n\nTo be more specific, SAG will outperform SG if the value of the component function $h(\\cdot)$ are negatively correlated for two pixels $p$ and $p’$ that are opposite of each other w.r.t. $c_m$. To demonstrate this more rigorously, we consider the variance of the aggregate function restricted to two pixels, which can be written as $Var [h(p)+h(p’)]$.\n\nFor SG sampling, $p$ and $p’$ are independently sampled in the pixel group $m$, and therefore we have $Var [h(p)+h(p’)] = Var[h(p)] + Var[h(p’)] = 2\\sigma_m^2$. In stark contrast, for SAG, since the choice of $p’$ depends on $p$, we have $Var [h(p)+h(p’)] = Var[h(p)] + Var[h(p’)] + 2 Cov[h(p), h(p’)] = 2 \\sigma_m^2 + 2 Cov[h(p), h(p’)]$. This indicates that, when $Cov[h(p), h(p’)] < 0$, i.e., $h(p)$ and $h(p’)$ are negatively correlated, it holds that $Var [h(p)+h(p’)] < 2\\sigma_m^2$. In such cases SAG would have smaller variance than SG. Due to the limited response space, we will ensure a more detailed discussion on these methods and their potential applications in our final revision.\n\n> **Q3**: Line 222-223, what is the meaning of \"p is orthogonal to p' \"?\n\n**A2**: The “orthogonal” sign means ‘independent’. In line 222, \"p is orthogonal to p' \" means that the pixel p and pixel p’  are sampled independently. Thank you for the suggestion, and we will further clarify in the revision.\n\nWe deeply appreciate your patience and engagement throughout this review process. We have conducted an extensive revision of our paper, which addresses all the issues. This revision focuses particularly on enhancing the clarity of our method's explanation and has conducted appropriate comparisons through experiments on three natural image datasets. We believe that these improvements have significantly shaped our work for the better and brought it closer to being ready for publication. Please do not hesitate to reach out to us should you have any further concerns. Your ongoing interest and insightful feedback on our work are greatly valued.\n"}, {"review_id": "xk5nGSvS9e", "reviewer": "Reviewer_LvuQ", "summary": "The authors present ARCO, a novel semi-supervised contrastive learning framework that employs a stratified group sampling strategy (i.e. **SG** and **SAG**) to compute gradient estimators with reduced variance, thereby enhancing representation learning in dense contrastive learning. By improving dense contrastive learning, ARCO addresses issues related to class imbalancedness and enhances the performance of semi-supervised segmentation, particularly in scenarios with long-tail distributed anatomical classes. The authors provide theoretical evidence demonstrating the effectiveness of the proposed sampling techniques in reducing the variance of the aggregation function, specifically the contrastive loss. The efficacy of the proposed sampling technique is validated on eight 2D/3D benchmark datasets with different label settings, further reinforcing its effectiveness and practical applicability.", "questions": "* The proposed two sampling techniques are intriguing. I wonder if you could kindly provide insights on the applicability of these sampling techniques to different tasks. Such information would prove beneficial not only to the current domain but also to other related domains and scenarios. Understanding the potential applications of these techniques would undoubtedly contribute to the advancement of various fields.\n \n* Lemma 3.1 demonstrates that the variance of SAG is at most twice of that of SG, which implies that the variance reduction magnitude of SAG and SG are roughly at the same level. Could the authors further elaborate on when SAG be preferred to SG?\n ", "limitations": "The authors have addressed the potential broader impact in clinical scenarios.\n", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "\nThe motivations are clear, and the method is reasonable. This study shows me a new insight/direction to consider medical image segmentation. The following are my detailed comments.\n\n---\n\n**Empirical contribution**\n\nPixel/voxel-wise sampling constitutes a critical facet of contrastive learning at the pixel/voxel level. With the aid of variance-reduction estimation, the authors proffer two pragmatic approaches - Stratified Group (SG) and Stratified-Antithetic Group (SAG), tailored for pixel/voxel-level segmentation tasks with exceedingly scarce labels.  \n\n* The authors introduce a novel framework termed ARCO (strAtifed gRoup COntrastive learning) devised for multi-class segmentation tasks. This framework appears to be both intriguing and efficacious. The authors undertake a rigorous validation of the proposed methodologies across eight benchmark datasets, encompassing three 2D medical image segmentation, two 3D medical image segmentation, and three semantic segmentation benchmarks.  \n\n* The empirical results, both quantitative and qualitative, attest to the efficacy of the proposed model across all label ratios and datasets. For instance, the model demonstrates a marked enhancement in segmentation accuracy (up to 4.1% absolute improvements in Dice coefficient) on the challenging multi-class MMWHS dataset under a 1% label setting.  \n\n* The authors conduct comprehensive ablation studies to substantiate that the proposed mechanisms merit consideration. These studies encompass eight benchmark datasets, diverse network architectures, and varying label ratios to validate the efficacy, model-agnostic nature, and label efficiency of the proposed methodology.  \n\n* Lastly, the proposed methodologies are not only facile to implement but also boast of universal applicability. For instance, they can be seamlessly integrated into any scenario necessitating pixel/voxel sampling.  The paper presents a robust and versatile framework for pixel/voxel-level contrastive learning, which is empirically validated through extensive experiments and ablation studies. The methodologies are characterized by ease of implementation and broad applicability, making them a valuable contribution to the field of image segmentation.\n\n---\n**Theoretical contribution**\n\n* The proposed methodologies, SG (Stratified Group) and SAG (Stratified-Antithetic Group), have exhibited remarkable efficacy in the experimental study. Consequently, it is intriguing to ascertain whether theoretical insights can elucidate this enhancement in performance. To this end, the paper furnishes a cogent theoretical analysis of the methodologies, revealing that the variance-reduction attribute of the two sampling methods is instrumental to their performance.\n\n* First and foremost, in Section 3.3, the paper meticulously delineates the SG and SAG sampling methodologies through lucid mathematical equations. SG is executed by segregating pixels into mutually exclusive groups, followed by uniform sampling of a specified number of pixels from each group. SAG, which is predicated on SG, imposes an additional constraint of symmetry among the sampled pixels within each group.\n\n* Subsequently, the sampled pixels are amalgamated through an aggregation function, which acts as an estimator for the target quantity. In the realm of image segmentation, this quantity could be, for instance, the contrastive loss function. It is posited that an optimal balance must be struck in the sample size; an overly diminutive sample size may fail to encapsulate the salient information from the underlying image, whereas an excessively large sample size would entail high computational complexity. Thus, the ideal scenario would be for SG to capture the crux of the image information through a relatively modest sample size. The paper demonstrates that SG possesses this attribute by establishing that it achieves reduced variance in comparison to the naïve sampling method (i.e., uniform random sampling from all pixels). Specifically, Theorem 3.2 establishes that SG is an unbiased sampling methodology, with a variance that does not exceed that of naïve sampling. More precisely, the variance of SG can be decomposed into the variance of naïve sampling minus a non-negative term. This non-negative term is conjectured to be almost certainly greater than zero, as it would be zero only if the expectation of the aggregation function within each group is identical to the expectation over the entire image, which is highly improbable. Thus, Theorem 3.2 suggests that SG is likely to consistently outperform naïve sampling.\n\n* Figure 5 reveals that SG/SAG exhibits marginally expedited training convergence compared to naïve sampling. This observation is theoretically substantiated in the concluding paragraph of Section 3, which is commendable.\n\n* In my assessment, the theoretical analysis presented in the paper is inextricably linked to the empirical component and provides a persuasive rationale for the empirical performance augmentation of SG vis-à-vis naïve sampling, thereby bringing the empirical narrative full circle.\n---\n\n**To sum up**\n\n**1. Clarity** \n\nThe manuscript is eloquently composed, proffering a lucid and cogent progression of information. The authors adeptly elucidate the procedural framework, facilitating the readers' comprehension of the proposed methodologies, namely, the two instance sampling methods - Stratified Group (SG) and Stratified-Antithetic Group (SAG). In Section 3.3, the SG and SAG sampling methods are meticulously delineated through precise mathematical formulations. The discourse within the Methodology section (Section 3) furnishes an exhaustive exposition of the innovations introduced by the study, adeptly accentuating the distinct contributions of the research to the scholarly domain.\n\n**2. Novelty** \n\nThis work is the first work, empirically and theoritically, to validate the variance-reduction approach within the context of pixel/voxel-level contrastive learning for semi-supervised medical image segmentation, particularly in scenarios characterized by a paucity of labels.\n\n**3. Experimental Comprehensiveness** \n\nThe authors undertake a comprehensive suite of experiments encompassing eight medical datasets, which include both 2D and 3D modalities, as well as semantic segmentation benchmarks. Additionally, a diverse array of contrastive learning frameworks and varying label ratios are employed to rigorously assess the efficacy, model-agnostic properties, and label efficiency of the proposed methodology. This extensive experimental evaluation substantiates the robustness and versatility of the technique in the domain of medical image segmentation.\n\n**4. Theoretical Implication** \n\nThe authors furnish a cogent and meticulously articulated theoretical analysis of the proposed approach, elucidating the underlying principles with clarity and precision. This analytical exposition contributes to a deeper understanding of the methodology's foundations and its implications.\n\n\n\n\n", "weaknesses": "The proposed method is very interesting. There is no obvious weakness in the proposed ARCO. However, I do have a few questions. See the following section.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging our contribution and providing suggestions for the presentation of our work! In particular, we agree that discussing the potential application and more detailed classification are necessary to position this work properly. If you have further concerns, please feel free to contact us.\n\n> **Q1**: The proposed two sampling techniques are intriguing. I wonder if you could kindly provide insights on the applicability of these sampling techniques to different tasks. Such information would prove beneficial not only to the current domain but also to other related domains and scenarios. Understanding the potential applications of these techniques would undoubtedly contribute to the advancement of various fields?\n\n**A1**: Thank you for the great suggestion! Our sampling techniques can provide pragmatic solutions for enhancing variance reduction, thereby fostering their application in a wide array of real-world applications and sectors. These include but are not limited to 3D rendering, augmented reality (AR), virtual reality (VR), trajectory prediction, and autonomous driving. We agree with the reviewer and appreciate them for the constructive suggestion. We will add the potential applications in our final revision.\n\n> **Q2**: Lemma 3.1 demonstrates that the variance of SAG is at most twice that of SG, which implies that the variance reduction magnitude of SAG and SG are roughly at the same level. Could the authors further elaborate on when SAG be preferred to SG?\n\n**A2**: Thank you for the great suggestion! SAG, compared with SG, only samples half the number of pixels, and the other half are chosen as the opposite with respect to the group center. This is described by the equation $c_m -p = p’ - c_m$ following Line 219. In other words, these two pixels, $p$ and $p’$, are opposite of each other w.r.t. $c_m$.\n\nIn general, SAG will outperform SG if the value of the component function $h(\\cdot)$ are negatively correlated for two pixels $p$ and $p’$ that are opposite of each other w.r.t. $c_m$. To demonstrate this more rigorously, we consider the variance of the aggregate function restricted to two pixels, which can be written as $Var [h(p)+h(p’)]$.\n\nFor SG sampling, $p$ and $p’$ are independently sampled in the pixel group $m$, and therefore we have $Var [h(p)+h(p’)] = Var[h(p)] + Var[h(p’)] = 2\\sigma_m^2$. In stark contrast, for SAG, since the choice of $p’$ depends on $p$, we have $Var [h(p)+h(p’)] = Var[h(p)] + Var[h(p’)] + 2 Cov[h(p), h(p’)] = 2 \\sigma_m^2 + 2 Cov[h(p), h(p’)]$. This indicates that, when $Cov[h(p), h(p’)] < 0$, i.e., $h(p)$ and $h(p’)$ are negatively correlated, it holds that $Var [h(p)+h(p’)] < 2\\sigma_m^2$. In such cases SAG would have smaller variance than SG. We will make corresponding revisions and provide detailed elaborations of them.\n\nOverall, thank you again for your suggestions and review! We believe that discussing the potential application and more detailed classification will greatly improve the paper. We will include all the modifications in our final revision. We hope that the revision puts our work in better shape for publication. Please feel free to contact us for further concerns.\n"}, {"review_id": "hTQbWtdT8X", "reviewer": "Reviewer_SMeb", "summary": "Edit: Updating score from 6 t o7 based on the discussions.\n\nExtracting useful representations from unlabelled data for label efficient training of medical image segmentation task is a widely studied problem. This work approaches learning useful representations using contrastive learning (CL) within a semi-supervised setting. Strategies for obtaining variance reducing pixel partitions for CL are presented, along with theoretical analysis that show their variance reduction properties. The variance reduction estimation is also used to improve training stability and convergence. Comprehensive experiments on multiple medical imaging and computer vision datasets are performed showing strong performance improvements compared to other SSL methods. The authors show their method is label efficient across all the datasets.", "questions": "See Weaknesses.", "limitations": "Authors do not discuss any limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Focusing on variance reduction guarantees to extract contrastive samples for pixel level training is a strong contribution of this work. \n\n* The theoretical analyses showing their unbiasedness and use of variance reduction techniques to improve training stability can have important applications in related domains like self-supervised learning.\n\n* The  experimental evaluation is extensive, with strong performance improvements on multiple datasets.\n\n* Contrastive loss landscape visualisation in Fig 3 is insightful; that the contrastive learning holds across datasets is quite convincing.", "weaknesses": "* **Robustness**: This work makes two main claims about the usefulness of their CL framework. Firstly, and convincingly so, about label efficiency. There are several places in the paper where model robustness is alluded to, or strong claims made, without any evidence. This could be because the authors view robustness simply to be good performance across multiple datasets? If so, this should be clarified. Currently, the claims about model robustness are misleading. See [1,2,3] for different robustness analyses of deep neural networks.\n\n* **Assumption in Th. 3.2**: The guarantees of $Var[H_{SG}] < Var[H_{NS}]$ only holds if different $P_m$ do not have the same expected value over the aggregation function h(x;p). How is this ensured? In medical images, there are scenarios where the differences between classes are small in both intensity and feature spaces? How does the variance reduction guarantees hold in such situations? \n\n* **Aggregation functions**: Aren't the aggregation functions some type of a distance measure? And why is it expensive to compute these on dense pixel grids (L213)? \n\n* **Main method in Appendix**: While I appreciate all the details presented in this paper, moving the main method to the Appendix is not a good idea. Several of the important details are in the Appendix and does not serve the purpose of what Appendices are supposed to be.\n\n* **Literature overview**: I was curious as to why the authors refrained from discussing self-supervised representation learning both when motivating the work, and also in their general discourse. \n\n[1] Bastani, Osbert, et al. \"Measuring neural net robustness with constraints.\" Advances in neural information processing systems 29 (2016).\n\n[2] Carlini, Nicholas, and David Wagner. \"Towards evaluating the robustness of neural networks.\" 2017 ieee symposium on security and privacy (sp). Ieee, 2017.\n\n[3] Singh, Gagandeep, et al. \"Fast and effective robustness certification.\" Advances in neural information processing systems 31 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging our contribution to the self-supervised field, appreciating the strong performance improvement on multiple datasets, and providing constructive suggestions for the presentation of our work! We’ve made a substantial revision to the paper, which addresses all the issues, with emphasis on improving clarity of our work. If you have further concerns, please feel free to contact us.\n\n> **Q1**: clarifying robustness in our paper.\n\n**A1**: Thank you for the great suggestion! “Robustness” is a comprehensive concept used to describe the performance of a segmentation model. A model is said to be robust if (1) it has a high segmentation quality with only using extremely limited labels in long-tailed medical data (please see Line 141); (2) and fast convergence speed (please see Line 207). We agree with the reviewer and appreciate them for raising the concern about the wording. We will surely polish them in our final revision.\n\n> **Q2**: Assumption in Th. 3.2: The guarantees of variance reduction only holds if $P_m$  do not have the same expected value over the aggregation function h(x;p). How is this ensured? In medical images, there are scenarios where the differences between classes are small in both intensity and feature spaces? How does the variance reduction guarantees hold in such situations?\n\n**A2**: Thm 3.2 simply indicates that, provided that all the pixel groups $\\{P_m\\}_m$ do not have exactly same distributions, it is guaranteed $Var[\\hat{H}(SG)] < Var[\\hat{H}(NS)]$.\n\nConsidering the unlikely condition where all pixel groups share an identical distribution within the feature space, it can be confidently asserted that SG will almost certainly realize a reduction in variance.\n\nExpanding on the previous point, the degree of variance reduction achievable, as anticipated, is contingent on the differentiation among pixel groups. We totally agree with you that there exist scenarios where the differences between classes are small in both intensity and feature spaces. Under these circumstances, the effect of variance reduction attributed to stratified group sampling would be comparatively insignificant. This condition aligns with the mathematical principle that variance is inherently linked to the extent of diversity in the dataset.\n\n> **Q3**: Aren't the aggregation functions some type of a distance measure? And why is it expensive to compute these on dense pixel grids (L213)?\n\n**A3**: Aggregation function refers to a function that is additive in pixels. That is, it can be expressed as a sum of functions of pixels, as is defined in Eq (3.1). Therefore, an aggregation function should be viewed as a general concept, which is not limited to a distance measure.\nIn our paper, a loss function (e.g. $L_{contrast}$) is an example of an aggregation function. The computation would be overwhelming on dense pixel grids because in this case, we will have to aggregate information from all available pixel points across all images. Note that a typical 2D image can have 256x256 pixels, and there can be tens of thousands of training medical images (e.g., LiTS: 16684 slices). Thank you for the feedback, and we will further clarify in the revision.\n\n\n> **Q4**: While I appreciate all the details presented in this paper, moving the main method to the Appendix is not a good idea. Several of the important details are in the Appendix and does not serve the purpose of what Appendices are supposed to be?\n\n**A4**: Thank you for the great suggestion! We agree with your point that the relevant main method details should be in the main paper. In our final revision, we will follow your constructive advice to move the key details from the appendix to the main paper.\n\n> **Q5**: I was curious as to why the authors refrained from discussing self-supervised representation learning both when motivating the work, and also in their general discourse?\n\n**A5**: Thank you for your insightful suggestion! We agree with your view that an in-depth discussion on self-supervised representation learning would enhance the comprehension and context of our work. We truly appreciate your constructive advice and will be incorporating it to improve our manuscript. Given the limited space for this rebuttal, we aim to expand one subsection within the 'Related Work' section and offer a comprehensive analysis of self-supervised representation learning within the 'Introduction' section in our final revision. In line with your valuable advice, we're sharing a few snippets of our intended discussion below.\n ```\nSelf-supervised representation learning is a subclass of unsupervised learning, but with the critical distinction that it incorporates “inherent” supervision from the input data itself. The primary aim of self-supervised representation learning is to enable the model to learn the most useful representations from the large amount of unlabelled data for various downstream tasks. Self-supervised learning typically relies on pretext tasks, including predictive, contextual, and generative or reconstructive tasks. Among them, contrastive learning is considered as a popular approach for self-supervised representation learning by pulling the representations of similar instances closer and representations of dissimilar instances further apart in the learned feature space.\n```\n\nYour advice is greatly appreciated. We will make corresponding revisions to clarify and provide details on them.\n\nThank you once more for your valuable feedback! We have performed a substantial revision to the paper, placing particular emphasis on enhancing the clarity of our methodology. These changes will be incorporated into our final submission. We trust that these modifications better position our work for publication. Please do not hesitate to reach out if you have further comments or queries.\n"}]

Author rebuttal:
In response to the query from Reviewer-J4c5 (Question 3), we appreciate the emphasis on the depth of our analysis. Given the constraints of this rebuttal format, we wish to assure our commitment to augmenting the ablation section for a more comprehensive understanding.  

To fully understand the contribution of each component, especially our novel pixel-level sampling strategy, we present comparative outcomes of components. These include pixel-level sampling variants (i.e., naive sampling (NS), Stratified Group (SG), and Stratified-Antithetic Group (SAG)), global contrastive loss, nearest neighbor loss, unsupervised loss, the global/local instance discrimination losses, and data augmentations assessed on the ACDC dataset with a 1% label ratio. For a detailed overview, please consult the “global” response PDF file.


Meta-review: The paper introduces a couple of sampling strategies to improve contrastive learning in semi-supervised learning for medical image segmentation. The proposed method is theoretically shown to reduce the variance of sampled pixels, which improves training stability. Experiments show empirical evidence for claims. The paper received scores 7,7,5,4, where the main strengths are: 1) thorough experiments, 2) realistic problem, and 3) sound theoretical analysis. The main weaknesses are: 1) unclear writing; 2) method complexity; 3) missing comparison in natural image datasets; and 4) unclear claims. The rebuttal seems to have addressed many of the issues. There are some good suggestions to improve paper clarity, so I highly encourage the authors to take the suggestions into consideration for the preparation of the final version of the paper.
Decision: Accept (poster)