SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {\text {c o n t r a s t}} = \sum_ {c \in \mathcal {C}} \sum_ {\mathbf {r} _ {q} \sim \mathcal {R} _ {q} ^ {c}} - \log \frac {\exp \left(\mathbf {r} _ {q} \cdot \mathbf {r} _ {k} ^ {c , +} / \tau\right)}{\exp \left(\mathbf {r} _ {q} \cdot \mathbf {r} _ {k} ^ {c , +} / \tau\right) + \sum_ {\mathbf {r} _ {k} ^ {-} \sim \mathcal {R} _ {k} ^ {c}} \exp \left(\mathbf {r} _ {q} \cdot \mathbf {r} _ {k} ^ {-} / \tau\right)}, \tag {E.2}
$$
where  $\mathcal{C}$  is a set including all available classes in the current mini-batch, and  $\tau$  is a temperature hyperparameter. We refer to  $\mathcal{R}_q^c,\mathcal{R}_k^c,\mathbf{r}_k^{c, + }$  as a query set including all representations within this class  $c$ , a negative key set including all representations whose labels is not in class  $c$ , and the positive key which is the  $c$ -class mean representation, respectively. Consider  $\mathcal{P}$  is a set including all pixel coordinates with the same resolution as  $R$ , these queries and keys are then defined as:
$$
\mathcal {R} _ {q} ^ {c} = \bigcup_ {[ i, j ] \in \mathcal {A}} \mathbb {1} (\mathbf {y} _ {[ i, j ]} = c) \mathbf {r} _ {[ i, j ]}, \mathcal {R} _ {k} ^ {c} = \bigcup_ {[ i, j ] \in \mathcal {A}} \mathbb {1} (\mathbf {y} _ {[ i, j ]} \neq c) \mathbf {r} _ {[ i, j ]}, \mathbf {r} _ {k} ^ {c, +} = \frac {1}{| \mathcal {R} _ {q} ^ {c} |} \sum_ {\mathbf {r} _ {q} \in \mathcal {R} _ {q} ^ {c}} \mathbf {r} _ {q}. \tag {E.3}
$$
(2) Diversity: we leverage the first-in-first-out (FIFO) memory bank [13] to search for  $K$ -nearest neighbors, and use the semi-supervised nearest neighbor loss  $\mathcal{L}_{\mathrm{nn}}$  in a way that maximizing cosine similarity, to exploit the inter-instance relationship.
For fine-tuning, the total loss includes contrastive loss  $\mathcal{L}_{\mathrm{contrast}}$ , nearest neighbors loss  $\mathcal{L}_{\mathrm{nn}}$ , unsupervised cross-entropy loss  $\mathcal{L}_{\mathrm{unsup}}$  and supervised segmentation loss  $\mathcal{L}_{\mathrm{sup}}$ :  $\mathcal{L}_{\mathrm{sup}} + \lambda_1\mathcal{L}_{\mathrm{contrast}} + \lambda_2\mathcal{L}_{\mathrm{unsup}} + \lambda_3\mathcal{L}_{\mathrm{nn}}$ . See Appendix N for the ablation of hyperparameters.
F Model Architecture
Figure 1 provides an overview over our approach. Our semi-supervised segmentation model  $F$  takes an 2D/3D medical image  $x$  as input and outputs the segmentation map and the representation map. We leverage MONA pipeline [17] including two stages: (1) relational semi-supervised pre-training: on labeled data, the student network is trained by the ground-truth labels with the supervised loss  $\mathcal{L}_{\mathrm{sup}}$ ; while on unlabeled data, the student network takes the augmened and mined embeddings from the EMA teacher for instance discrimination  $\mathcal{L}_{\mathrm{inst}}$  in the global and local manner, (2) our proposed anatomical contrastive reconstruction fine-tuning: on labeled data, the student network is trained by the ground-truth labels with the supervised loss  $\mathcal{L}_{\mathrm{sup}}$ ; while on unlabeled data, the student model takes the representation maps and pseudo labels from the EMA teacher to give more importance to tail class  $\mathcal{L}_{\mathrm{contrast}}$ , exploit the inter-instance relationship  $\mathcal{L}_{\mathrm{nn}}$ , and compute unsupervised loss  $\mathcal{L}_{\mathrm{unsup}}$ . See Appendix M for details of the visualization loss landscapes.
G More Experiment Results on LiTS
Figure 6 and Table 5 show qualitative and quantitative results, where our ARCO-SG and ARCO-SAG provide better segmentation results than all other methods. This clearly demonstrates the superiority of our models.
In this paper, sampling strategies in pixel-level contrastive learning frameworks are of direct interest, so we use a simplification of MONA [17] – without using any additional complicated augmentation strategies.
11The representation head is only applied during training, and is removed during inference
Figure 7: Visual results on MMWHS with  $1\%$ ,  $5\%$ ,  $10\%$  label ratios. As is shown, ARCO-SG and ARCO-SAG consistently produce more accurate predictions on anatomical regions and boundaries compared to all other SSL methods.
Figure 8: Visual results on LA with  $1\%$ ,  $5\%$ ,  $10\%$  label ratios. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
H More Experiment Results on MMWHS
We run the baselines and our methods on the third medical image segmentation dataset (i.e., MMWHS [83]) under various label ratios (i.e.,  $1\%$ ,  $5\%$ ,  $10\%$ ), reporting results in Table 6 and Figure 7. This clearly demonstrates the effectiveness of our models.
I More 3D Experiment Results on LA [84]
We run the baselines and our methods on the fourth 3D medical image segmentation dataset (i.e., LA [84]) under various label ratios (i.e.,  $1\%$ ,  $5\%$ ,  $10\%$ ), reporting results in Table 9 and Figure 8. This clearly demonstrates the robustness of our models.
J More 3D Experiment Results on MP-MRI
We run the baselines and our methods on the five 3D medical image segmentation dataset (i.e., MP-MRI) under various label ratios (i.e.,  $1\%$ ,  $5\%$ ,  $10\%$ ), reporting results in Table 9 and Figure 9. This clearly shows the superiority of our models.
Figure 9: Visual results on MP-MRI with  $1\%$ ,  $5\%$ ,  $10\%$  label ratios. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
K More Experiment Results on Semantic Segmentation
To further validate the effectiveness, we experiment on three popular segmentation benchmarks (i.e., Cityscapes [97], Pascal VOC 2012 [98], indoor scene segmentation dataset - SUN RGB-D [99]) in the semi-supervised full-label settings. We follow the identical setting [100] to sample labelled images to ensure that every class appears sufficiently in our three datasets, i.e., CityScapes, Pascal VOC, and SUN RGB-D. Specifically, we aim to have the least frequent class appear in at least 5, 15, and 50 images in each dataset, respectively. In CityScapes, we have at least 12 semantic classes represented in our labeled images, while in Pascal VOC and SUN RGB-D we have at least 3 and 1 semantic classes, respectively. We compare our ARCO-SG and ARCO-SAG under various grid settings (i.e., 9, 16, 25) to baselines (supervised and semi-supervised ReCo). Table 7 shows the mean IoU validation performance. We can see that for all cases, ARCO-SG and ARCO-SAG consistently improve performance, compared to ReCo, in all the semi-supervised settings. For example, under the fewest label setting in each dataset, compared to ReCo, applying stratified group sampling (SG) can improve results by an especially significant margin, with up to  $2.4 - 7.8\%$  relative gains. As shown in Pascal VOC 2012 [98] - 60 labels (Figure 10), 200 labels (Figure 11), 600 labels (Figure 12), Cityscapes [97] - 20 labels (Figure 13), 50 labels (Figure 14), 150 labels (Figure 15), SUN RGB-D [99] - 50 labels (Figure 16), 150 labels (Figure 17), 500 labels (Figure 18), ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to ReCo. All those clearly demonstrate the superiority of our models.
L Generalization across Label Ratios and Frameworks
Besides generalizing well across different datasets and diverse labeled settings, we additionally evaluate the performance of SG and SAG sampling coupled with different CL frameworks (i.e., MoCov2 [103], kNN-MoCo [104], SimCLR [14], BYOL [26], ISD [27], VICReg [28]) on ACDC under
various label ratios (i.e.,  $1\%$ ,  $5\%$ ,  $10\%$ ). In this work, we mainly study the state-of-the-art CL frameworks from the computer vision domain hereinafter for ablations, considering their superior performance in the task of image classification. Note that, for each fair comparison, we strictly follow the default setting in these CL frameworks [103, 104, 14, 26, 27, 28] for pretraining, and fine-tune each networks using the same settings in Appendix C. Full experimental details are in Appendix C.
Since they work orthogonally to our pretraining strategy, we conduct a comprehensive comparison of these CL-based frameworks in Table 8. Clearly, our proposed SG and SAG sampling help significantly in improving the segmentation performance across all the CL-based frameworks and is capable of being integrated with previous frameworks for further enhanced model robustness. Moreover, it is important to observe that the performance benefits of our methods increase significantly with a lower label setting. This observation augments the necessity of our proposed methods while training networks with high label efficiency.
M Final Checkpoint Loss Landscapes
From visualizations in Figure 19, ARCO-SG converges to much flatter loss valleys, which evidences their robustness in learning anatomical features.
N Ablation on Different Training Settings
Hyperparameter Selection. For grid search, we detail the tuning steps here. The tuning is done in sequential order.  $\lambda_{1}$  is chosen from  $\{0.001, 0.005, 0.01, 0.05, 0.1, 1.0\}$ , and  $\lambda_{2}, \lambda_{3}$  are chosen from  $\{0.1, 0.2, 0.5, 1.0, 2.0, 10.0\}$ . We use the validation set to search over hyperparameters and find the best hyperparameter on ACDC with  $1\%$  labeled ratio. As shown in Figure 20, with a carefully tuned hyperparameters  $\lambda_{1} = 0.01$ ,  $\lambda_{2} = 1.0$ , and  $\lambda_{3} = 1.0$ , such setting achieves superior performance compared to others.
Table 6: Quantitative comparisons (DSC[%] ↑ / ASD[voxel] ↓) across the three labeled ratio settings (1%, 5%, 10%) on the MMWHS benchmark. All experiments are conducted as [30, 90, 91, 92, 93, 8, 94, 9, 57, 95, 62, 50, 59, 96, 53, 16, 17] in the identical setting for fair comparisons. Best and second-best results are coloured blue and red, respectively. UNet-F (fully-supervised) and UNet-L (semi-supervised) are considered as the upper bound and the lower bound for the performance comparison. Please refer to the text for discussion. Note that, Left Ventricle → LV, Right Ventricle → RV, Left Atrium → LA, Right Atrium → RA, Myocardium → Myo, Ascending Aorta → AA, Pulmonary Artery → PA. We adopt the identical data augmentation (i.e., random rotation, random cropping, and horizontal flipping) for fair comparisons.
Table 7: Quantitative comparisons (Intersection Over Union (IoU) ↑) for Pascal VOC, CityScapes and SUN RGB-D datasets. All experiments are conducted as [100] in the identical setting for fair comparisons. Best and second-best results are colored blue and red, respectively.
Method	Pascal VOC	CityScapes	SUN RGB-D
60 labels	200 labels	600 labels	all labels	20 labels	50 labels	150 labels	all labels	50 labels	150 labels	500 labels	all labels
Supervised	39.4	55.5	64.6	77.8	38.2	45.9	55.4	70.9	20.0	29.2	38.9	51.8
ReCo [100] + ClassMix	57.1	69.4	73.2	-	49.9	57.9	65.0	-	30.5	40.4	44.6	-
ARCO-SAG (9 Grid) + ClassMix	58.3	70.5	75.4	-	50.2	60.2	66.5	-	31.5	40.9	45.7	-
ARCO-SAG (16 Grid) + ClassMix	58.7	70.9	75.1	-	50.1	60.6	66.3	-	37.8	40.2	45.7	-
ARCO-SAG (25 Grid) + ClassMix	59.1	70.9	74.9	-	49.8	60.6	66.7	-	38.5	40.5	45.5	-
ARCO-SG (9 Grid) + ClassMix	59.2	71.8	75.3	-	52.5	60.9	66.8	-	32.4	41.4	46.6	-
ARCO-SG (16 Grid) + ClassMix	59.6	71.7	75.5	-	53.7	61.2	66.2	-	37.7	41.0	46.4	-
ARCO-SG (25 Grid) + ClassMix	59.5	71.7	75.2	-	51.5	61.8	66.4	-	38.3	41.5	47.3	-
Table 8: Ablation on different contrastive learning frameworks with respect to various labeled ratio settings (1%, 5%, 10%). Experiments are conducted on ACDC using UNet [30] as the backbone. Here we report the segmentation performance in terms of DSC[%] and ASD[mm]. On all three labeled settings, incorporating our methods (i.e., SG and SAG) consistently achieve superior model robustness gains compared to naive sampling across different state-of-the-art CL frameworks.
Category	Method	1% Labeled	5% Labeled	10% Labeled
DSC ↑	ASD ↓	DSC ↑	ASD ↓	DSC ↑	ASD ↓
MoCov2 [103]	NS	76.4	5.64	83.3	3.78	83.8	3.17
•SAG	80.9	4.16	84.7	3.53	84.6	3.84
○SG	81.4	4.01	85.1	1.48	84.9	3.08
kNN-MoCo [104]	NS	78.3	4.54	83.8	3.74	84.1	2.97
•SAG	81.8	4.08	84.6	3.41	84.7	2.58
○SG	83.9	3.17	85.4	3.17	85.0	2.47
SimCLR [14]	NS	74.9	4.89	80.9	3.19	84.1	2.78
•SAG	78.5	4.01	83.8	2.68	85.8	2.01
○SG	79.1	3.49	84.3	2.31	86.0	1.76
BYOL [26]	NS	77.3	5.01	82.9	3.14	84.8	1.67
•SAG	79.8	4.12	85.0	2.81	85.7	1.36
○SG	80.2	3.79	85.8	1.67	85.9	1.13
ISD [27]	NS	79.3	3.54	81.2	2.86	85.6	1.90
•SAG	81.4	3.50	83.1	2.53	86.3	1.61
○SG	82.2	2.04	83.7	1.97	86.7	1.34
VICReg [28]	NS	64.0	10.6	79.1	4.18	82.9	3.89
•SAG	81.1	3.49	83.8	3.02	86.3	2.14
○SG	81.6	3.12	84.1	2.78	86.8	2.01
ARCO (ours)	NS	82.6	1.43	86.9	1.07	87.7	1.33
•SAG	84.9	1.47	87.1	0.848	88.5	1.40
○SG	85.5	0.947	88.7	0.841	89.4	0.776
Case I (60 Labels)
Case 2 (60 Labels)
Bicycle Bird Boat Bottle Bus Car Cat
Bottle Bus Car Cat
Background Aeroplane
Table Dog Horse
Motorbike Person Plant Sheep Sofa Train
Chair Cow Monitor
Invalid
Figure 10: Visual results on Pascal validation set with 60 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
Reco+ClassMix
Case I (200 Labels)
ARCO-SAG (16 Grid) + ClassMix
ARCO-SAG (25 Grid) + ClassMix
ARCO-SAG (9 Grid) + ClassMix
ARCO-SG (16 Grid) + ClassMix
ARCO-SG (25 Grid) + ClassMix
ARCO-SG (9 Grid) + ClassMix
Ground Truth
ARCO-SAG (9 Grid) + ClassMix
Case 2 (200 Labels)
ARCO-SG (9 Grid) + ClassMix
Aeroplan
Boat
Bottle
Motor
#
Invalid
：
D
Figure 11: Visual results on Pascal validation set with 200 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
Chair
Cow
Case I (600 Labels)
Case 2 (600 Labels)
ARCO-SG (9 Grid) + ClassMix
#
bike
Persor
Chair
Figure 12: Visual results on Pascal validation set with 600 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
Case I (20 Labels)
ARCO-SAG (9 Grid) + ClassMix
ARCO-SG (9 Grid) + ClassMix
Ground Truth
ARCO-SG (25 Grid) + ClassMix
Reco+ClassMix
Case 2 (20 Labels)
ARCO-SAG (9 Grid) + ClassMix
dewalk
c Light
Traf
etation
Terrain
Sky
Figure 13: Visual results on Cityscapes validation set with 20 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
Person
Rider
Mg
cycle
Bicyc
$\therefore m = \frac{3}{11}$  ;
e Pole
ign.
Terr
Sky
Figure 14: Visual results on Cityscapes validation set with 50 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
Case I (150 Labels)
Case 2 (150 Labels)
ARCO-SAG (9 Grid) + ClassMix
Supervised
ARCO-SAG (16 Grid) + ClassMix
Reco+ClassMix
ARCO-SAG (25 Grid) + ClassMix
Road
Figure 15: Visual results on Cityscapes validation set with 150 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
gn
getation
Figure 16: Visual results on SUN RGB-D validation set with 50 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
Figure 17: Visual results on SUN RGB-D validation set with 150 labels. As is shown, ARCO-SG and ARCO-SAG consistently yield more accurate and sharper boundaries compared to all other SSL methods.
Case I (500 Labels)
Case 2 (500 Labels)
ARCO-SAG (16 Grid) + ClassMix ARCO-SAG (25 Grid) + ClassMix
visual results on SUN RGB-D validation set with 500 labels. As is shown, ARCO-SG-G consistently yield more accurate and sharper boundaries compared to all other SSL
Table 9: Quantitative comparisons (DSC[%] ↑ / ASD[voxel] ↓) across the three labeled ratio settings (1%, 5%, 10%) on the 3D MP-MRI and 3D LA benchmarks. All 3D experiments are conducted as [78, 90, 91, 92, 93, 8, 94, 9, 57, 95, 62, 50, 59, 96, 53, 16, 17] in the identical setting for fair comparisons. Best and second-best results are coloured blue and red, respectively. VNet-F (fully-supervised) and VNet-L (semi-supervised) are considered as the upper bound and the lower bound for the performance comparison. Please refer to the text for discussion.
3D Method	MP-MRI	LA
1% Labeled Liver	5% Labeled Liver	10% Labeled Liver	1 Labeled (1%) Left Atrium (LA)	4 Labeled (5%) Left Atrium (LA)	8 Labeled (10%) Left Atrium (LA)
VNet-F [78]	93.1/5.73	93.1/5.73	93.1/5.73	91.5/1.51	91.5/1.51	91.5/1.51
VNet-L	68.6/33.4	81.6/12.5	87.9/7.55	40.0/21.2	52.6/9.87	82.7/3.26
EM [90]	73.2/30.1	86.0/15.8	91.9/6.89	48.3/21.3	81.1/4.68	82.7/4.77
CCT [91]	74.2/24.5	86.0/15.8	90.9/9.54	40.3/13.8	70.8/8.31	82.0/5.25
DAN [92]	69.4/31.4	88.3/10.2	91.1/8.76	38.5/22.0	78.8/6.53	80.2/5.37
URPC [57]	72.7/29.9	89.8/9.00	91.7/7.41	65.0/8.97	80.2/5.48	83.1/4.57
DTC [9]	78.6/18.8	89.6/10.1	90.5/11.2	36.2/11.7	83.6/2.81	87.1/2.23
DCT [93]	74.1/31.1	88.2/12.4	90.1/11.1	42.9/19.1	80.1/9.06	80.4/9.18
ICT [95]	72.3/30.5	88.6/11.2	91.1/8.46	47.7/16.0	78.4/6.96	85.4/4.14
MT [59]	73.8/29.4	87.7/12.8	92.0/7.15	58.1/17.8	77.0/8.15	82.8/5.90
UAMT [8]	71.6/31.2	87.1/12.8	91.3/9.71	60.3/11.3	82.3/3.82	87.8/2.12
SASSNet [94]	78.8/19.6	88.4/13.1	88.7/13.1	51.5/14.6	81.6/3.58	87.5/2.59
CPS [62]	80.0/17.1	89.2/10.8	91.0/9.16	45.1/22.0	79.7/9.28	80.7/5.16
GCL [50]	78.9/14.2	87.9/8.29	90.4/5.68	52.6/12.8	75.5/7.60	84.8/4.22
MC-Net [96]	79.7/20.2	90.1/8.27	92.4/4.34	44.3/14.1	83.6/2.70	87.6/1.82
SS-Net [53]	88.0/8.93	90.9/9.94	92.0/5.67	43.4/14.8	86.3/2.31	88.6/1.90
ACTION [16]	86.5/13.6	90.3/12.3	90.9/10.0	71.1/6.23	86.6/2.24	88.7/2.10
MONA [17]	91.3/5.31	92.2/9.46	92.3/8.16	72.8/10.7	87.0/2.81	89.5/2.40
ARCO-SAG (ours)	91.5/6.82	92.5/6.95	92.6/7.54	73.2/6.47	86.9/2.73	89.1/2.30
ARCO-SG (ours)	91.6/6.60	92.5/6.31	92.8/6.64	75.0/4.06	87.8/1.66	89.9/1.47
Figure 19: Loss landscape visualization of pixel-wise contrastive loss  $\mathcal{L}_{\mathrm{contrast}}$  with ARCO-SG. Loss plots are generated with same original images randomly chosen from ACDC [81], LiTS [82], MMWHS [83], LA [84], and MP-MRI, respectively.  $z$ -axis denotes the loss value at each pixel.
Figure 20: Effects of hyperparameters  $\lambda_{1},\lambda_{2},\lambda_{3}$ . We report Dice and ASD of ARCO-SG on ACDC with  $1\%$  labeled ratio.