SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chenyu You<sup>1</sup> Weicheng Dai<sup>1</sup> Yifei Min<sup>1</sup> Fenglin Liu<sup>2</sup> David A. Clifton<sup>2</sup>
S. Kevin Zhou<sup>3</sup>
Lawrence Staib<sup>1</sup>
James S. Duncan<sup>1</sup>
$^{1}$ Yale University  $^{2}$ University of Oxford  $^{3}$ University of Science and Technology of China
For medical image segmentation, contrastive learning is the dominant practice to improve the quality of visual representations by contrasting semantically similar and dissimilar pairs of samples. This is enabled by the observation that without accessing ground truth labels, negative examples with truly dissimilar anatomical features, if sampled, can significantly improve the performance. In reality, however, these samples may come from similar anatomical regions and the models may struggle to distinguish the minority tail-class samples, making the tail classes more prone to misclassification, both of which typically lead to model collapse. In this paper, we propose ARCO, a semi-supervised contrastive learning (CL) framework with stratified group theory for medical image segmentation. In particular, we first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks with extremely limited labels. Furthermore, we theoretically prove these sampling techniques are universal in variance reduction. Finally, we experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings, and our methods consistently outperform state-of-the-art semi-supervised methods. Additionally, we augment the CL frameworks with these sampling techniques and demonstrate significant gains over previous methods. We believe our work is an important step towards semi-supervised medical image segmentation by quantifying the limitation of current self-supervision objectives for accomplishing such challenging safety-critical tasks.
Model robustness and label efficiency are two highly desirable perspectives when it comes to building reliable medical segmentation models. In the context of medical image analysis, a model is said to be robust if (1) it has a high segmentation quality with only using extremely limited labels in long-tailed medical data; (2) and fast convergence speed [1, 2, 3]. The success of traditional supervised learning depends on training deep networks on a large amount of labeled data, but this improved segmentation/model robustness often comes at the cost of annotations and clinical expertise [4, 5, 6]. Therefore, it is difficult to adopt these models in real-world clinical applications.
Recently, a significant amount of research efforts [7, 8, 9, 10] have resorted to unsupervised or semi-supervised learning techniques for improving the segmentation robustness. One of the most effective methods is contrastive learning (CL) [11, 12, 13, 14]. It aims to learn useful representations
Codes are available on here.
by contrasting semantically similar (positive) and dissimilar (negative) pairs of data points sampled from the massive unlabeled data. These methods fit particularly well with real-world clinical scenarios as we assume only access to a large amount of unlabelled data coupled with extremely limited labels. However, pixel-level contrastive learning with medical image segmentation is quite impractical since sampling all pixels can be extremely time-consuming and computationally expensive [15]. Fortunately, recent studies [16, 17] provide a remedy by leveraging the popular strategy of bootstrapping, which first actively samples a sparse set of pixel-level representation (queries), and then optimize the contrastive objective by pulling them to be close to the class mean averaged across all representations in this class (positive keys), and simultaneously pushing apart those representations from other class (negative keys). The demonstrated imbalancedness and diversity across various medical image datasets, as echoed in [18], show the positive sign of utilizing the massive unlabeled data with extremely limited annotations while maintaining the impressive segmentation performance compared to supervised counterparts. Meanwhile, it can lead to substantial memory/computation reduction when using pixel-level contrastive learning framework for medical image segmentation.
Nevertheless, in practical clinical settings, the deployed machine learning models often ask for strong robustness, which is far beyond the scope of segmentation quality for such challenging safety-critical scenarios. This leads to a more challenging requirement, which demands the models to be more robust to the collapse problems whereby all representations collapse into constant features [14, 13, 19] or only span a lower-dimensional subspace [20, 21, 22, 23], as one main cause of such fragility could be attributed to the non-smooth feature space near samples [24, 25] (i.e., random sampling can result in large feature variations and even annotation information alter). Thus, it is a new perspective: how to sample most informative pixels/voxels towards improving variance reduction in training semi-supervised contrastive learning models. This inspires us to propose a new hypothesis of semi-supervised CL. Specifically, when directly baking in variance-reduction sampling into semi-supervised CL frameworks for medical image segmentation, the models can further push toward state-of-the-art segmentation robustness and label efficiency.
In this paper, we present ARCO, a semi-supervised strAtified gRoup Contrastive learning framework with two perspectives (i.e., segmentation/model robustness and label efficiency), and with the aid of variance-reduction estimation, realize two practical solutions - Stratified Group (SG) and Stratified-Antithetic Group (SAG) - for selecting the most semantically informative pixels. ARCO is a group-based sampling method that builds a set of pixel groups and then proportionally samples from each group with respect to the class distribution. The main idea of our approach is via first partitioning the image with respect to different classes into grids with the same size, and then sampling, within the same grid, pixels semantically close to each other with high probability, with minimal additional memory footprint.
Subsequently, we show that baking ARCO into contrastive pre-training (i.e., MONA [17]) provides an efficient pixel-wise contrastive learning paradigm to train deep networks that perform well in long-tailed medical data. ARCO is easy to implement, being built on top of off-the-shelf pixel-level contrastive learning framework [13, 14, 26, 27, 28], and consistently improve overall segmentation quality across all label ratios and datasets (i.e., five 2D/3D medical and three semantic datasets).
Our theoretical analysis shows that, ARCO is more label efficient, providing practical means for computing the gradient estimator with improved variance reduction. Empirically, our approach achieves competitive results across eight 2D/3D medical and semantic segmentation benchmarks. Our proposed framework has several theoretical and practical contributions:
- We propose ARCO, a new CL framework based on stratified group theory to improve the label efficiency and model robustness trade-off in CL for medical image segmentation. We show that incorporating ARCO coupled with two special sampling methods, Stratified Group and Stratified-Antithetic Group, into the models provides an efficient learning paradigm to train deep networks that perform well in those long-tail clinical scenarios.
- To our best knowledge, we are the first work to show the benefit of certain variance-reduction techniques in CL for medical image segmentation. We demonstrate the unexplored advantage of the refined gradient estimator in handling long-tailed medical image data.
- We conduct extensive experiments to validate the effectiveness of our proposed method using a variety of datasets, network architectures, and different label ratios. For segmentation robustness/accuracy, we show that our proposed method by demonstrating superior segmentation accuracy (up to  $11.08\%$  absolute improvements in Dice). For label efficiency, our
method trained with different labeled ratios – consistently achieves competitive performance improvements across all eight 2D/3D medical and semantic segmentation benchmarks.
- Theoretical analysis of ARCO shows improved variance reduction with optimization guarantee. We further demonstrate the intriguing property of ARCO across the different pixel-level contrastive learning frameworks.
Medical Image Segmentation. Contemporary medical image segmentation approaches typically build upon fully convolutional networks (FCN) [29] or UNet [30], which formulates the task as a dense classification problem. In general, current medical image segmentation methods can be cast into two sets: network design and optimization strategy. One is to optimize segmentation network design for improving feature representations through dilated/atrous/deformable convolutions [31, 32, 33], pyramid pooling [34, 35, 36], and attention mechanisms [37, 38, 39]. Most recent works [40, 41, 6] reformulates the task as a sequence-to-sequence prediction task by using the vision transformer (ViT) architecture [42, 43]. The other is to improve optimization strategies, by designing loss function to better address class imbalance [44] or refining uncertain pixels from high-frequency regions improving the segmentation quality [45, 46, 47, 48, 49]. In contrast, we take a leap further to a more practical clinical scenario by leveraging the massive unlabeled data with extremely limited labels in the learning stage. Moreover, we focus on building model-agnostic, label-efficiency framework to improve segmentation quality by providing additional supervision on the most confusing pixels for each class. In this work, we question how medical segmentation models behave under such imbalanced class distributions and whether they can perform well in those challenging scenarios through sampling methods.
Semi-Supervised Learning (SSL). SSL aims to train models with a combination of labeled, weakly-labeled and unlabelled data. In recent years, there has been a surge of work on semi-supervised medical segmentation [8, 9, 50, 48, 16, 51, 52, 17, 10, 53, 54], which makes it hard to present a complete overview here. We therefore only outline some key milestones related to this study. In general, it can be roughly categorized into two groups: (1) Consistency regularization was first proposed by [55], which aims to impose consistency corresponding to different perturbations into the training, such as consistency regularization [56, 57], pi-model [58], and mean-teacher [59, 60]. (2) Self-training was initially proposed in [61], which aims at using a model's predictions to obtain noisy pseudo-labels for performance boosts with minimal human labor, such as pseudo-labeling [7, 62], model uncertainty [8, 63], confidence estimation [64, 65, 66], and noisy student [67]. These methods usually lead to competitive performance but fail to prevent collapse due to class imbalanceness. In this work, we focus on semi-supervised medical segmentation with extremely limited labels since the medical image data is extremely diverse and often long-tail distributed over anatomical classes. We speculate that a good medical segmentation model is expected to distinguish the minority tail-class samples and hence achieve better performance under additional supervision on hard pixels.
Contrastive Self-Supervised Learning. Self-supervised representation learning is a subclass of unsupervised learning, but with the critical distinction that it incorporates "inherent" supervision from the input data itself [68]. The primary aim of self-supervised representation learning is to enable the model to learn the most useful representations from the large amount of unlabelled data for various downstream tasks. Self-supervised learning typically relies on pretext tasks, including predictive [69, 70, 71], contextual [72, 73], and generative [74] or reconstructive [75] tasks.
Among them, contrastive learning is considered as a popular approach for self-supervised representation learning by pulling the representations of similar instances closer and representations of dissimilar instances further apart in the learned feature space [11, 12, 13, 14]. The past five years have seen tremendous progress related to CL in medical image segmentation [50, 23, 76, 48, 16, 17, 77], and it becomes increasingly important to improve representation in label-scarcity scenarios. The key idea in CL [11, 12, 13, 14] is to learn representations from unlabeled data that obey similarity constraints by pulling augmented views of the same samples closer in a representation space, and pushing apart augmented views of different samples. This is typically achieved by encoding a view of a data into a single global feature vector. However, the global representation is sufficient for simple tasks like image classification, but does not necessarily achieve decent performance, especially for more challenging dense prediction tasks. On the other hand, several works on dense contrastive
Figure 1: Pipeline overview. Our semi-supervised segmentation model  $F$  takes a 2D/3D medical image  $x$  as input and outputs the segmentation map and the representation map. We leverage a simplification of MONA pipeline [17] which is composed of two stages: (1) relational semi-supervised pre-training: on labeled data, the student network is trained by the ground-truth labels with the supervised loss  $\mathcal{L}_{\mathrm{sup}}$ ; while on unlabeled data, the student network takes the augmened and mined embeddings from the EMA teacher for instance discrimination  $\mathcal{L}_{\mathrm{inst}}$  in the global and local manner, (2) anatomical contrastive reconstruction fine-tuning: on labeled data, the student network is trained by the ground-truth labels with the supervised loss  $\mathcal{L}_{\mathrm{sup}}$ ; while on unlabeled data, the student network takes the representation maps and pseudo labels from the EMA teacher to give more importance to tail class  $\mathcal{L}_{\mathrm{contrast}}$ , exploit the inter-instance relationship  $\mathcal{L}_{\mathrm{nn}}$ , and compute unsupervised loss  $\mathcal{L}_{\mathrm{unsup}}$ . See Appendix M for details of the visualization loss landscapes.
learning [50, 23], aim at providing additional supervision to capturing intrinsic spatial structure and fine-grained anatomical correspondence, while these methods may suffer from class imbalance issues. Particularly, very recent work [16, 17] for the first time demonstrates the imbalancedness phenomenon can be mitigated by performing contrastive learning yet lacking stability. By contrast, a key motivation of our work is to bridge the connection between model robustness and label efficiency, which we believe is an important and under-explored area. We hence focus on variance-reduced estimation in medical image segmentation, and show that certain variance-reduction techniques can help provide more efficient approaches or alternative solutions for handling collapse issues, and improving model robustness in terms of accuracy and stability. To the best of our knowledge, we are the first to provide a theoretical guarantee of robustness by using certain variance-reduction techniques.
In this section we set-up our semi-supervised medical segmentation problem, introduce key definitions and notations and formulate an approach to incorporate stratified group theory. Then, we discuss how our proposed ARCO can directly bake in two perspectives into deep neural networks: (1) model robustness, and (2) label efficiency.
3.1 Preliminaries and setup
Problem Definition. In this paper, we consider the multi-class medical image segmentation problem. Specifically, given a medical image dataset  $(\mathcal{X},\mathcal{Y})$ , we wish to automatically learn a segmentator, which assigns each pixel to their corresponding  $K$ -class segmentation labels. Let us denote  $\mathbf{x}$  as the input sample of the student and teacher networks  $F(\cdot)^2$ , consisting of an encoder  $E$  and a decoder  $D$ , and  $F$  is parameterized by weights  $\theta_s$  and  $\theta_t$ .
Background. Contrastive learning aims to learn effective representations by pulling semantically close neighbors together and pushing apart other non-neighbors [11]. Among various popular contrastive learning frameworks, MONA [17] is easy-to-implement while yielding the state-of-the-art performance for semi-supervised medical image segmentation so far. The main idea of MONA is to discover diverse views (i.e., augmented/mined views) whose anatomical feature responses are homogeneous within the same or different occurrences of the same class type, while at the same time being distinctive for different class types.
2The student and teacher networks both adopt the 2D UNet [30] or 3D VNet [78] architectures.
Hereinafter, we are interested in showing that certain variance-reduction techniques coupled with CL frameworks are particularly beneficial in long-tail pixel/voxel-level segmentation tasks with extremely limited labels. We hence build our ARCO as a simplification of the MONA pipeline [17], without additional complex augmentation strategies, for deriving the model robustness and label efficiency properties of our medical segmentation model. Figure 1 overviews the high-level workflow of the proposed ARCO framework. Training ARCO involves a two-phase training procedure: (1) relational semi-supervised pre-training, and (2) anatomical contrastive fine-tuning. To make the discussion self-contained, we defer the full details of ARCO to the appendix E.
3.2 Motivation and Challenges
Intuitively, the contrastive loss will learn generalizable, balanced and diverse representations for downstream medical segmentation tasks if the positive and negative pairs correspond to the desired latent anatomical classes [50, 16, 17]. Yet, one critical constraint in real-world clinical scenarios is severe memory bottlenecks [15, 16]. To address this issue, current pixel-level CL approaches [16, 17] for high-resolution medical images devise their aggregation rules by unitary simulators, i.e., Naïve Sampling (NS), that determines the empirical estimate from all available pixels. Despite the blessing of large learning capacity, such aggregation rules are unreliable "black boxes". It is never well understood which rule existing CL models should use for improved model robustness and label efficiency; nor is it easy to compare different models and assess the model performance. Moreover, unitary simulators, especially naïve sampling, often incur high variances and fail to identify semantically similar pixels [24], limiting CL stability. As demonstrated in Figure 3, regions of similar anatomical features should be grouped together in the original medical images, resulting in corresponding plateau regions in the visualization of the loss landscape. This is consistent with the observations uncovered by the recent empirical findings [79, 80].
Figure 2: Overview of three sampling methods. (1) Naïve Sampling, (2) Stratified Group Sampling, and (3) Stratified-Antithetic Group Sampling.
If we take a unified mathematical perspective, the execution of simulation can be represented either through an adaptive rule, or by a unitary simulation. To tackle the two critical issues, we look back at adaptive rules. We hence propose two straightforward yet effective techniques - Stratified Group (SG) and Stratified-Antithetic Group (SAG) - to mitigate the undesirable high-variance limitation, and turn to the following idea of sampling the most representative pixels from groups of semantically similar pixels. In particu
lar, our proposed solution is based on stratified group simulation to adaptively characterize anatomical regions found on different medical images. This characterization is succinct, and regions with the same anatomical properties within different medical images are identifiable. In practice, we first partition the image with respect to different classes into grids with the same size, and then sampling, within the same grid, the pixels semantically close to each other with high probability, with minimal additional memory footprint (Figure 2).
In what follows, we will theoretically demonstrate the important properties of such techniques (i.e., SG and SAG), especially in reduced variance and unbiasedness. Here the reduced variance implies more robust gradient estimates in the backpropagation, and leads to faster and stabler training in theory, as corroborated by our experiments (Section 4). Empirically, we will demonstrate many practical benefits of reduced variances including improved model robustness, i.e., faster convergence and better segmentation quality, through mitigating the collapse issue.
3.3 Stratified Group Sampling
To be consistent with the previous notation, we denote an arbitrary image from the given medical image dataset as  $\mathbf{x} \in \mathcal{X}$ , and  $\mathcal{P}$  as the set of pixels. For arbitrary function  $h: \mathcal{X} \times \mathcal{P} \to \mathbb{R}$ , we define
Figure 3: Loss landscape visualization of pixel-wise contrastive loss  $\mathcal{L}_{\mathrm{contrast}}$  with ARCD-SG. Loss plots are generated with same original images randomly chosen from ACDC [81], LiTS [82], MMWHS [83], LA [84], and MP-MRI, respectively.  $z$ -axis denotes the loss value at each pixel. For each example of the five benchmarks, the left subplot indicates that similar anatomical features are grouped together in the original medical images, as shown by different anatomical regions in different colors.
the aggregation function  $H^3$  as:
$$
H (\mathbf {x}) = \frac {1}{| \mathcal {P} |} \sum_ {p \in \mathcal {P}} h (\mathbf {x}; p). \tag {3.1}
$$
As a large cardinality of  $\mathcal{P}$  prevents efficient direct computation of  $H$ , an immediate approach is to compute  $H(\mathbf{x})$  by first sampling a subset of pixels  $\mathcal{D} \subseteq \mathcal{P}$  according to certain sampling strategy, and then computing  $\widehat{H}(\mathbf{x}; \mathcal{D}) = \sum_{p \in \mathcal{D}} h(\mathbf{x}; p) / |\mathcal{D}|$ . SG sampling achieves this by first decomposing the pixels into  $M$  disjoint groups  $\mathcal{P}_m$  satisfying  $\cup_{m=1}^{M} \mathcal{P}_m = \mathcal{P}$ , and then sampling  $\mathcal{D}_m \subseteq \mathcal{P}_m$  so that  $\mathcal{D} = \cup_{m=1}^{M} \mathcal{D}_m$ . The SG sampling can then be written as:
$$
\widehat {H} _ {\mathrm {S G}} (\mathbf {x}; \mathcal {D}) = \frac {1}{M} \sum_ {m = 1} ^ {M} \frac {1}{| \mathcal {D} _ {m} |} \sum_ {p \in \mathcal {D} _ {m}} h (\mathbf {x}; p).
$$
SAG, built upon SG, adopts a similar form, except for an additionally enforced symmetry on  $\mathcal{D}_m$ :  $\forall m, \exists c_m \in \mathcal{P}_m$ , such that for any  $p \in \mathcal{D}_m$ ,
$$
c _ {m} - p = p ^ {\prime} - c _ {m}, \text {f o r s o m e} p ^ {\prime} \in \mathcal {D} _ {m}.
$$
Here  $c_{m}$  denotes the center of the group  $\mathcal{P}_m$ . The implementation of SG and SAG involves two steps: (1) to create groups  $\{\mathcal{P}_m\}_{m=1}^M$ , and (2) to generate each  $\mathcal{D}_m \subseteq \mathcal{P}_m$ . For the latter, we consider independent sampling within and between groups, i.e.,  $\mathcal{D}_m \perp \mathcal{D}_{m'}$  for  $m \neq m'$ , and  $p \perp p' \forall p$ ,  $p' \in \mathcal{D}_m$ , where the variance of SG sampling is as follows.
Lemma 3.1. Suppose in SG sampling, for each  $m$ ,  $\mathcal{D}_m$  is sampled from  $\mathcal{P}_m$  with sampling variance  $\sigma_m^2$  and sample size  $|\mathcal{D}_m| = n_m$ . Then the variance satisfies  $\mathrm{Var}[\widehat{H}_{\mathrm{SG}}] = \sum_{m=1}^{M} \sigma_m^2 n_m / n$ , and SAG with the same sample size satisfies  $\mathrm{Var}[\widehat{H}_{\mathrm{SAG}}] \leq 2 \mathrm{Var}[\widehat{H}_{\mathrm{SG}}]$ .
To ensure the unbiasedness property, we adopt the setting of proportional group sizes [85, 86], i.e.,  $|\mathcal{D}_m| \propto |\mathcal{P}_m|$  for all  $m$ . It turns out that such setting also enjoys the variance-reduction property.