SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that  $\sigma_m^2$  is the variance of the  $h(\mathbf{x};p)$  where  $p\sim \mathcal{P}_m$  uniformly.
Analysis of SAG. We now consider SAG. Recall that, compared to SG, SAG first samples  $n_m / 2$  pixels from  $\mathcal{P}_m^6$  from each  $m = 1, \dots, M$  in the same way as SG. SAG then deterministically picks the rest  $n_m / 2$  pixels to be the reflection pixels of the first half (see Figure 2(3)). Let's choose arbitrary group  $m$  and denote by  $p$  and  $p'$  the sampled pixel and its reflection from that group. Note that the variance of  $p$  satisfies  $\mathrm{Var}_{\mathrm{SAG}}[h(\mathbf{x};p)] = \sigma_m^2$  since  $p$  is sampled in a same way as in the case of SG. Observe that, by symmetry, the variance of  $p$  and  $p'$  should be the same  $\mathrm{Var}_{\mathrm{SAG}}[h(\mathbf{x};p)] = \mathrm{Var}_{\mathrm{SAG}}[h(\mathbf{x};p)] = \sigma_m^2$ . It follows that:
$$
\begin{array}{l} \operatorname {V a r} _ {\mathrm {S A G}} \left[ h (\mathbf {x}; p) + h (\mathbf {x}; p ^ {\prime}) \right] \\ = \operatorname {V a r} _ {\mathrm {S A G}} [ h (\mathbf {x}; p) ] + \operatorname {V a r} _ {\mathrm {S A G}} [ h (\mathbf {x}; p ^ {\prime}) ] + 2 \operatorname {C o v} _ {\mathrm {S A G}} [ h (\mathbf {x}; p), h (\mathbf {x}; p ^ {\prime}) ] \\ \leq 4 \sigma_ {m} ^ {2}, \\ \end{array}
$$
For simplicity, we ignore the rounding issue.
where the second step holds because the correlation between  $h(\mathbf{x};p)$  and  $h(\mathbf{x};p^{\prime})$  is at most 1. It follows that:
$$
\begin{array}{l} \mathrm {V a r} [ \widehat {H} _ {\mathrm {S A G}} ] = \sum_ {m = 1} ^ {M} \frac {| \mathcal {P} _ {m} | ^ {2}}{| \mathcal {P} | ^ {2}} \frac {1}{| \mathcal {D} _ {m} | ^ {2}} \mathrm {V a r} \left[ \sum_ {p \in \mathcal {D} _ {m}} h (\mathbf {x}; p) \right] \\ = \sum_ {m = 1} ^ {M} \frac {n _ {m} ^ {2}}{n ^ {2}} \frac {1}{n _ {m} ^ {2}} \sum_ {p, p ^ {\prime}} \operatorname {V a r} [ h (\mathbf {x}; p) + h (\mathbf {x}; p ^ {\prime}) ] \\ \leq \sum_ {m = 1} ^ {M} \frac {n _ {m} ^ {2}}{n ^ {2}} \frac {1}{n _ {m} ^ {2}} \frac {n _ {m}}{2} 4 \sigma_ {m} ^ {2} \\ = \sum_ {m = 1} ^ {M} \frac {2 n _ {m}}{n ^ {2}} \sigma_ {m} ^ {2} \\ = 2 \operatorname {V a r} [ \hat {H} _ {\mathrm {S G}} ]. \\ \end{array}
$$
Proof of Theorem 3.2. Denote  $w_{m} = n_{m} / n$ , where  $n_m = |\mathcal{D}_m|$ . The unbiasedness is straightforward: by proportional sampling, the probability that an arbitrary pixel in group  $\mathcal{P}_m$  being chosen is equal to  $|\mathcal{D}_m| / |\mathcal{P}_m|$ . Since  $|\mathcal{D}_m| = |\mathcal{P}_m|$ , the probability of being chosen is equal across all pixels, and hence an arbitrary pixel  $p \in \mathcal{D} = \cup_{m} \mathcal{D}_{m}$  is equally likely to be any pixel in the population  $\mathcal{P}$ . As a result, we have
$$
\begin{array}{l} \mathbb {E} [ \widehat {H} _ {\mathrm {S G}} (\mathbf {x}; \mathcal {D}) ] = \mathbb {E} \left[ \sum_ {p \in \mathcal {D}} h (\mathbf {x}; p) / | \mathcal {D} | \right] = \mathbb {E} _ {p} [ h (\mathbf {x}; p) ] \\ = H (\mathbf {x}), \\ \end{array}
$$
where the first equality is by the construction of  $\widehat{H}_{\mathrm{SG}}(\mathbf{x};\mathcal{D})$ , the second equality is by symmetry of  $p\in \mathcal{D}$ , and the last inequality is by the above conclusion that  $p$  is equally possible to be any pixel in  $\mathcal{P}$ . This finishes the proof that  $\widehat{H}_{\mathrm{SG}}$  is unbiased.
The variance of  $\hat{H}_{\mathrm{SG}}$  is equal to
$$
\operatorname {V a r} \left[ \widehat {H} _ {\mathrm {S G}} \right] = \sum_ {m = 1} ^ {M} \frac {n _ {m} ^ {2}}{n ^ {2}} \frac {\sigma_ {m} ^ {2}}{n _ {m}},
$$
where  $\sigma_m^2$  is the variance of the  $f(\mathbf{x};p)$  when  $p\sim \mathcal{D}_m$  uniformly. Under the case of proportional sampling, i.e.,  $n_m\propto |\mathcal{D}_m|$  for all  $m = 1,\dots ,M$  (for simplicity we assume such  $n_m$  's are all integers), the variance becomes
$$
\mathrm {V a r} [ \widehat {H} _ {\mathrm {S G}} ] = \sum_ {m = 1} ^ {M} \frac {(w _ {m} \cdot n) ^ {2}}{n ^ {2}} \frac {\sigma_ {m} ^ {2}}{w _ {m} \cdot n} = \sum_ {m = 1} ^ {M} w _ {m} \frac {\sigma_ {m} ^ {2}}{n}.
$$
We now consider NS (i.e. naive sampling). By definition, the random sampling samples pixels uniformly from the set  $\mathcal{P}$  of all pixels. Since  $n$  is the total number of pixels sampled, the variance of the  $\widehat{H}_{\mathrm{NS}}$  is given as  $\operatorname{Var}[\widehat{H}_{NS}] = \sigma^2 / n$  where  $\sigma^2$  is the sampling variance of  $h(\mathbf{x}; p)$  given  $p \sim \mathcal{P}$  uniformly. To determine  $\sigma^2$ , we apply a variance decomposition trick via conditioning. Specifically, the sampling from NS is a two-step process: (1) sample the group index  $m$  from  $[M]$ , (2) sample the pixel uniformly from  $\mathcal{P}_m$ . Applying the law of total variance, we have
$$
\begin{array}{l} \operatorname {V a r} _ {p \sim \mathcal {P}} ^ {\text {u n i f .}} [ h (\mathbf {x}; p) ] \\ = \mathbb {E} [ \operatorname {V a r} [ h (\mathbf {x}; p) ] | p \in \mathcal {P} _ {m} ] + \operatorname {V a r} [ \mathbb {E} [ h (\mathbf {x}; p) | p \in \mathcal {P} _ {m} ] ] \\ = \sum_ {m = 1} ^ {M} w _ {m} \sigma_ {m} ^ {2} + \sum_ {m = 1} ^ {M} \left(\underset {p \sim \mathcal {P} _ {m}} {\mathbb {E}} [ h (\mathbf {x}; p) ] - \underset {p \sim \mathcal {P}} {\mathbb {E}} [ h (\mathbf {x}; p) ]\right) ^ {2}. \\ \end{array}
$$
As a result, we conclude that, for any image  $\mathbf{x}$ , the sampling variance of SG estimate and NS estimate satisfies
$$
\begin{array}{l} \operatorname {V a r} \left[ \widehat {H} _ {\mathrm {N S}} \right] = \operatorname {V a r} \left[ \widehat {H} _ {\mathrm {S G}} \right] \\ + \frac {1}{n} \sum_ {m = 1} ^ {M} \left(\mathbb {E} _ {p \sim \mathcal {P} _ {m}} [ h (\mathbf {x}; p) ] - \mathbb {E} _ {p \sim \mathcal {P}} [ h (\mathbf {x}; p) ]\right) ^ {2}, \\ \end{array}
$$
A.3 Further Details on Convergence
We first make the following assumptions about the loss function and the gradient estimate in our learning problem.
Assumption A.1 (Smoothness). The objective function  $\mathcal{L}(\cdot)$  is  $L$ -smooth, i.e.,  $\mathcal{L}$  is differentiable and  $\| \nabla \mathcal{L}(\theta) - \nabla \mathcal{L}(\theta') \|_2 \leq L (\|\theta - \theta'\|_2)$  for all  $\theta, \theta'$  in the domain of  $\nabla \mathcal{L}$ .
Assumption A.2 (Bounded variance). There exists some  $\sigma_g^2 > 0$  such that for all  $\theta$  in the domain,
$$
\mathbb {E} \left[ \| g (\theta) - \nabla \mathcal {L} (\theta) \| ^ {2} \right] \leq \sigma_ {g} ^ {2}.
$$
These assumptions are standard in optimization theory and in various settings [87, 88, 89]. The first assumption allows us to characterize the loss landscape of our learning problem, and the second assumption ensures the gradient estimate does not deviate largely from the truth, which is essential for the training to converge. With these two assumptions, it is guaranteed that
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \| \nabla \mathcal {L} (\theta) \| _ {2} ^ {2} \right] \leq C \left(\frac {1}{T} + \frac {\sigma_ {g}}{\sqrt {T}}\right).
$$
Proposition 1. Suppose the step sizes in SGD are taken to be  $\min \left\{1 / L, \alpha / (\sigma_g \sqrt{T})\right\}$  where  $T$  is the number of steps. Then under Assumption A.1 and A.2, there exists some constant  $C$  depending on  $\mathcal{L}, \alpha, \text{and } L$ , such that
Proposition 1 shows that, in expectation, the convergence rate of the average gradient norm consists of a fast rate  $1 / T$  and a slow rate  $\sigma_g / \sqrt{T}$ , with the slow rate being the dominant term. Importantly, the slow rate depends on the variance of the gradient estimate. This suggests that a variance reduced gradient estimate allows SGD to reach an approximate local minimum with less iterations, leading to faster convergence of the training loss. This intuition is corroborated by Fig. 5, where we directly visualize the training trajectory of contrastive loss versus epoch for three sampling methods. We observe that SG features a faster loss decay and a narrower error bar, showing that it outperforms other methods in both the convergence speed and the stability. Furthermore, Proposition 1 indicates that our proposed sampling techniques are universal, since it applies to all scenarios as long as the mild assumptions A.1 and A.2 are satisfied. In other words, when it comes to scenarios that involve pixel/voxel-level sampling on 2D/3D images, utilizing SG/SAG instead of naive sampling can lead to enhanced stability and decreased variance.
Our experiments are conducted on five 2D/3D representative datasets in semi-supervised medical image segmentation literature, including 2D benchmarks (i.e., ACDC [81], LiTS [82], and MMWHS [83]) and 3D benchmarks (i.e., LA [84] and in-house MP-MRI).
- The ACDC dataset was collected from MICCAI 2017 ACDC challenge [81], consisting of 200 3D cardiac cine MRI scans with 3 classes - left ventricle (LV), myocardium (Myo), and right ventricle (RV). Following [53, 9, 10], we utilize 120, 40, and 40 scans for training,
validation, and testing  $^{7}$ . Note that  $1\%$ ,  $5\%$ , and  $10\%$  label ratios are the ratio of patients. The splitting details are in the supplementary material. For pre-processing, we normalize the intensity of each 3D scan into  $[0,1]$  by using min-max normalization, and re-sample images and segmentation maps to  $256 \times 256$  pixels.
- The LiTS dataset was collected from MICCAI 2017 Liver Tumor Segmentation Challenge [82], consisting of 131 contrast-enhanced 3D abdominal CT volumes with 2 classes – liver and tumor. Note that  $1\%$ ,  $5\%$ , and  $10\%$  label ratios are the ratio of patients. We utilize 100 and 31 scans for training and testing, with random order. The splitting details are in the supplementary material. For pre-processing, we follow the setting in [6] by truncating the intensity of each 3D scan into  $[-200, 250]$  HU for removing irrelevant and redundant features, normalizing each 3D scan into  $[0, 1]$ , and re-sample images and segmentation maps to  $256 \times 256$  pixels.
- The MMWHS dataset was collected from MICCAI 2017 challenge [83], consisting of 20 3D cardiac MRI scans with 7 classes – left ventricle (LV), left atrium (LA), right ventricle (RV), right atrium (RA), myocardium (Myo), ascending aorta (AAo), and pulmonary artery (PA). Note that  $1\%$ ,  $5\%$ , and  $10\%$  label ratios are based on the ratio of patients. Following [17], we utilize 15 and 5 scans for training and testing. The splitting details are in the supplementary material. For pre-processing, we normalize the intensity of each 3D scan into  $[0,1]$  by using min-max normalization, and re-sample images and segmentation maps to  $256 \times 256$  pixels.
- The LA dataset [84] was a representative 3D benchmark, consisting of 100 gadolinium-enhanced MRI scans with one class - left atrium (LA), with an isotropic resolution of  $0.625 \times 0.625 \times 0.625\mathrm{mm}^3$ . Note that  $1\%$ ,  $5\%$ , and  $10\%$  label ratios are the ratio of patients. The fixed split (i.e.,  $5\%$ , and  $10\%$ ) [8, 53] uses 80 and 20 scans for training and testing  $^8$ , and  $1\%$  label ratio is randomly split. The splitting details are in the supplementary material. For pre-processing, we crop all the scans at the heart region, normalize the intensities of each 3D scan into  $[0, 1]$ , and randomly crop all the training sub-volumes into  $112 \times 112 \times 80\mathrm{mm}^3$ .
- The Multi-phasic MRI (MP-MRI) dataset was an in-house 3D dataset, consisting of 160 multi-phasic MRI scans with one class - liver, each of which includes T1 weighted DCE-MRI images at three-time points (i.e., pre-contrast, arterial phase, and venous phases). Three images are mutually registered to the arterial phase images, with an isotropic voxel size of  $1.00 \times 1.00 \times 1.00\mathrm{mm}^3$ . The dataset is randomly divided into 100 scans for training, 40 for validation, and 20 for testing. Note that  $1\%$ ,  $5\%$ , and  $10\%$  label ratios are the ratio of patients. The splitting details are in the supplementary material. For pre-processing, we normalize the intensity of each 3D scan into [0, 1] by using min-max normalization, and re-sample images and segmentation maps to  $256 \times 256$  pixels.
In our experiments, all of our evaluated methods have been trained using similar settings for simplicity in reproducing our results. All experiments are conducted with PyTorch [101] on an NVIDIA RTX 3090 Ti. We adopt an SGD optimizer with momentum 0.9 and weight decay  $10^{-4}$ . The initial learning rate is set to 0.01. We use the 2D UNet [30] or 3D VNet [78] backbones as the segmentation network under different labeled ratio settings (i.e.,  $1\%$ ,  $5\%$ ,  $10\%$  labeled ratios). Following [17], we adopt the feature pyramid network (FPN) [102] architecture as the representation head  $\psi_r$  with a separate 512-dimension output layer. The momentum hyperparameter is set to 0.99.
For pre-training, the networks are trained for 100 epochs with a batch size of 6. We apply the mined views with  $d = 5$ , following [17]. As for fine-tuning, the networks are trained for 200 epochs with a batch size of 8. The learning rate decays by a factor of 10 every 2500 iterations during the training. We apply the temperature with  $\tau_{t} = 0.01$ ,  $\tau_{s} = 0.1$ , and  $\tau = 0.5$ , respectively. The size of the memory bank is set to 36. For the CL training, we use the implementation from [17] and leave all parameters on their default settings, e.g., we apply the hyperparameters with  $\lambda_1 = 0.01$ ,  $\lambda_2 = 1.0$ , and  $\lambda_3 = 1.0$ . For other hyper-parameters in all the evaluated methods, we adopt the suggested settings in the original papers because they are not of direct interest to us.
<sup>7</sup>https://github.com/HiLab-git/SSL4MIS/tree/master/data/ACDC
<sup>8</sup>https://github.com/ycwu1997/SS-Net/tree/main/data/LA
Table 4: Ablation on data augmentation strategies: (1) random rotation; (2) random cropping; and (3) horizontal flipping, compared to our methods with two settings (i.e., w/ no augmentation and w/ all three augmentation). Note that we use the identical data augmentation strategies (i.e., random rotation, random cropping, and horizontal flipping), as [30, 90, 91, 92, 93, 8, 94, 9, 57, 95, 62, 50, 59, 96, 53, 16, 17] for fair comparisons.
Method	DSC[%]↑	ASD[voxel]↓
• ARCO-SAG (ours)	84.9	1.47
w/o random rotation	84.1	2.87
w/o random cropping	84.4	3.47
w/o horizontal flipping	73.7	6.70
w/ no augmentation	70.8	9.83
○ ARCO-SG (ours)	85.5	0.947
w/o random rotation	83.8	3.27
w/o random cropping	84.6	1.62
w/o horizontal flipping	78.8	4.15
w/ no augmentation	76.2	7.74
For 2D medical segmentation, we follow the same data augmentations [9, 53] to the teacher's input and the student's input, respectively. The augmentations include random rotation, random cropping, and horizontal flipping. For 3D medical segmentation, we follow the same data augmentations [8, 9, 96, 53] to the teacher's input and the student's input, respectively. The augmentations include random rotation and random flipping. We evaluate our methods on 3D segmentation results with two classical metrics: (1) Dice coefficient (DSC) and (2) Average Symmetric Surface Distance (ASD). Note that, for all the evaluated methods, we make no additional modifications during the training process for fair evaluations. We run all our experiments in the same environments with fixed random seeds (Hardware: Single NVIDIA GeForce RTX 3090 GPU; Software: PyTorch 1.10.2+cu113, and Python 3.8.11).
D 2D/3D Methods in Comparison
2D Medical Segmentation: For experiments, we use 2D UNet [30] as backbone, and compare ARCO with previous state-of-the-art medical segmentation methods: (1) Baseline (i.e., UNet-F/UNet-L [30]) using both fully-supervised and limited-supervised settings; and (2) SSL-based: EM [90], CCT [91], DAN [92], URPC [57], DTC [9], DCT [93], ICT [95], MT [59], UAMT [8], SASSNet [94], CPS [62], GCL [50], MC-Net [96], SS-Net [53], ACTION [16], and MONA [17]. Note that among all the above evaluated methods, several methods use a contrastive objective, including GCL [50], SS-Net [53], ACTION [16], and MONA [17].
3D Medical Segmentation: For experiments, we use 3D VNet [78] as backbone, and compare ARCO with previous state-of-the-art medical segmentation methods: (1) Baseline (i.e., VNet-F/VNet-L [78]) using both fully-supervised and limited-supervised settings; and (2) SSL-based: EM [90], CCT [91], DAN [92], URPC [57], DTC [9], DCT [93], ICT [95], MT [59], UAMT [8], SASSNet [94], CPS [62], GCL [50], MC-Net [96], SS-Net [53], ACTION [16], and MONA [17]. Note that among all the above evaluated methods, several methods use a contrastive objective, including GCL [50], SS-Net [53], ACTION [16], and MONA [17].
E Framework Overview
In the following, we provide a concise overview of ARCO, consisting of two training phases: (1) relational semi-supervised pre-training, and (2) anatomical contrastive fine-tuning. Note that, in this paper, sampling strategies in pixel-level CL framework are of direct interest, so we use a simplification of MONA [17] without using any additional complicated augmentation strategies.
(1) Relational Semi-Supervised Pretraining. Given an unlabeled sample, ARCO has two ways to define augmented and mined views: (1) ARCO augments the samples to be  $\mathbf{x}^1$  and  $\mathbf{x}^2$  as augmented views, with two separate data augmentation operators; and (2) ARCO randomly samples  $d$  mined views (i.e.,  $\mathbf{x}^3$ ) from the unlabeled dataset with additional augmentation. The pairs  $[\mathbf{x}^1, \mathbf{x}^2]$  are then processed by  $[F_s, F_t]$ , and in a similar way  $\mathbf{x}^3$  is processed by  $F_t$  (See Figure 1(a) in main paper), outputting three global features  $[\mathbf{h}^1, \mathbf{h}^2, \mathbf{h}^3]$  after  $E$  and their local features  $[\mathbf{f}^1, \mathbf{f}^2, \mathbf{f}^3]$  after
Table 5: Quantitative comparisons (DSC[%] ↑ / ASD[voxel] ↓) across the three labeled ratio settings (1%, 5%, 10%) on the LiTS benchmark. All experiments are conducted as [30, 90, 91, 92, 93, 8, 94, 9, 57, 95, 62, 50, 59, 96, 53, 16, 17] in the identical setting for fair comparisons. Best and second-best results are coloured blue and red, respectively. UNet-F (fully-supervised) and UNet-L (semi-supervised) are considered as the upper bound and the lower bound for the performance comparison. Please refer to the text for discussion. We adopt the identical data augmentation (i.e., random rotation, random cropping, and horizontal flipping) for fair comparisons.
Method	LiTS
Average	1% Labeled	Average	5% Labeled	Average	10% Labeled
Liver	Lesion	Liver	Lesion	Liver	Lesion
UNet-F [30]	68.2/ 16.9	90.6/8.14	45.8/25.6	68.2/16.9	90.6/8.14	45.8/25.6	68.2/16.9	90.6/8.14	45.8/25.6
UNet-L	50.8/30.4	76.5/15.2	25.0/45.6	63.4/30.4	90.5/9.84	36.4/50.9	64.6/28.3	88.4/18.6	40.9/37.9
EM [90]	56.6/38.4	86.4/26.3	26.9/50.5	61.2/33.3	87.6/9.47	34.7/57.1	62.9/38.5	87.4/21.3	38.3/55.7
CCT [91]	52.4/52.3	79.1/42.0	25.7/62.6	61.4/26.1	84.6/12.3	38.2/39.9	63.4/23.0	88.8/7.64	38.0/38.5
DAN [92]	57.1/28.3	84.5/19.2	29.6/37.4	62.3/25.8	88.6/9.64	36.1/42.1	63.2/30.7	87.3/15.4	39.1/46.1
URPC [57]	55.5/34.6	83.0/27.3	28.0/42.0	60.9/21.4	85.5/8.11	36.3/34.6	62.5/24.8	84.9/12.9	40.1/36.7
DTC [9]	39.3/37.5	68.1/10.7	10.5/64.3	59.2/18.6	85.0/7.54	33.3/29.7	62.5/24.8	84.9/12.9	40.1/36.7
DCT [93]	57.7/38.5	87.0/22.4	28.3/54.6	60.8/34.4	89.2/12.6	32.5/56.2	61.9/31.7	86.2/19.3	37.5/44.1
ICT [95]	58.3/32.2	86.5/22.8	30.1/41.5	60.1/39.1	86.8/12.6	33.3/65.6	62.5/32.4	88.1/16.7	36.9/48.2
MT [59]	54.7/24.7	83.1/10.2	26.3/39.1	60.9/23.7	87.5/6.34	34.4/41.1	62.3/23.7	88.5/9.32	36.1/38.1
UAMT [8]	55.5/34.6	83.0/27.3	28.0/42.0	61.5/24.7	84.5/10.6	38.6/38.8	62.9/23.6	87.4/7.78	38.4/39.6
SASSNet [94]	39.6/42.7	69.0/14.7	10.3/7.06	60.4/25.3	86.1/11.6	34.7/39.0	62.4/21.1	86.4/8.31	38.3/33.9
CPS [62]	57.7/39.6	87.0/22.4	28.3/54.6	59.5/26.3	84.0/9.01	34.9/43.5	62.1/30.8	88.6/18.3	35.6/43.3
GCL [50]	59.3/29.5	88.6/14.2	30.0/44.9	63.3/20.1	90.7/9.46	35.9/30.8	65.0/37.2	91.3/10.0	38.7/64.3
MC-Net [96]	60.9/32.1	87.1/17.8	34.8/46.5	61.6/19.8	86.3/8.21	36.9/31.4	63.4/29.9	89.0/13.1	38.0/46.8
SS-Net [53]	55.0/35.9	89.6/19.8	20.5/51.9	59.1/24.6	87.5/11.2	30.6/38.1	63.4/19.8	91.1/7.33	35.8/32.2
ACTION [16]	61.0/24.5	89.8/16.9	32.2/32.3	66.3/23.6	93.0/6.41	39.5/40.8	67.2/20.4	92.8/5.08	41.6/35.8
MONA [17]	62.2/24.9	91.3/13.9	34.0/35.9	66.6/16.6	93.1/7.74	40.1/25.4	68.3/18.0	93.4/8.88	43.3/27.0
ARCO-SAG (ours)	64.1/20.6	91.5/7.63	36.8/33.5	67.3/13.2	93.0/7.10	41.7/19.3	69.4/14.9	93.3/7.60	45.5/22.2
○ ARCO-SG (ours)	65.5/20.2	92.5/5.59	38.4/34.7	68.4/11.3	93.7/6.63	43.0/16.0	70.1/13.5	94.1/5.01	46.1/22.1
Liver Lesion
Figure 6: Visual results on LiTS with  $1\%$ ,  $5\%$ ,  $10\%$  label ratios. As is shown, ARCO-SG and ARCO-SAG consistently produce more accurate predictions on anatomical regions and boundaries compared to all other SSL methods.
$D$ . These features are fed to the two-layer non-linear projectors for outputting global and local embeddings  $\mathbf{v}_g$  and  $\mathbf{v}_l$ .
To alleviate the collapse issues [26, 27, 16], we make the architecture asymmetric between the student and teacher pipeline by further feeding both the global and local embeddings  $\mathbf{v}$  with respect to the student branch into the non-linear predictor, producing  $\mathbf{w}$  in both global and local manners. After passing through the nonlinear projectors and predictor, the relational similarities between augmented and mined embeddings are computed using the softmax transform, which can be formulated as:  $\mathbf{u}_s = \log \frac{\exp\left(\sin\left(\mathbf{w}^1,\mathbf{v}^3\right) / \tau_s\right)}{\sum_{n = 1}^{N}\exp\left(\sin\left(\mathbf{w}^1,\mathbf{v}_n^3\right) / \tau_s\right)}$ ,  $\mathbf{u}_t = \log \frac{\exp\left(\sin\left(\mathbf{w}^2,\mathbf{v}^3\right) / \tau_t\right)}{\sum_{n = 1}^{N}\exp\left(\sin\left(\mathbf{w}^2,\mathbf{v}_n^3\right) / \tau_t\right)}$ , where  $\tau_s$  and  $\tau_t$  are different temperature parameters. The unsupervised instance discrimination loss (i.e., Kullback-Leibler divergence  $\mathcal{KL}$ ) can be defined as:
$$
\mathcal {L} _ {\text {i n s t}} = \mathcal {K L} \left(\mathbf {u} _ {s} \| \mathbf {u} _ {t}\right). \tag {E.1}
$$
<sup>9</sup>We omit details of local instance discrimination setting for simplicity in following contexts.
The parameters of the teacher model are the exponential moving average (EMA) of the student model parameters that are updated by the stochastic gradient descent. For pretraining, the entire loss consists of the global and local instance discrimination loss, and supervised segmentation loss  $\mathcal{L}_{\mathrm{sup}}$  (i.e., equal combination of Dice loss and cross-entropy loss), i.e.,  $\mathcal{L} = \mathcal{L}_{\mathrm{inst}}^{\mathrm{global}} + \mathcal{L}_{\mathrm{inst}}^{\mathrm{local}} + \mathcal{L}_{\mathrm{sup}}$ .
(2) Anatomical Contrastive Finetuning (ACF). We use the pre-trained network weights as the initialization for subsequent fine-tuning (See Figure 1(b) in Main Paper). For semi-supervised training, we follow two principles described in [17]  $^{10}$ : (1) tailness: giving more importance to tail class hard pixels; and (2) diversity: ensuring anatomical diversity in the set of different sampled images.
Following the abovementioned principles, we employ a two-step routine: (1) Tailness: we first perform anatomical contrastive formulation. Specifically, we additionally attach the representation head  $\psi_r^{11}$ , and generate a higher  $n$ -dimensional dense representation with the same spatial resolution as the input image. A pixel-level contrastive loss is designed to pull queries  $\mathbf{r}_q \in \mathcal{R}$  to be similar to the positive keys  $\mathbf{r}_k^+ \in \mathcal{R}$ , and push apart the negative keys  $\mathbf{r}_k^- \in \mathcal{R}$ . The semi-supervised contrastive loss  $\mathcal{L}_{\mathrm{contrast}}$  is defined as: