SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.2 (Unbiasedness and Variance of SG).  $SG$  with proportional group sizes is unbiased, and has a variance no larger than that of NS. That is:  $\mathbb{E}[\hat{H}_{\mathrm{SG}}(\mathbf{x})] = H(\mathbf{x})$ , and
$$
\mathrm {V a r} [ \widehat {H} _ {\mathrm {S G}} ] = \mathrm {V a r} [ \widehat {H} _ {\mathrm {N S}} ] - \frac {1}{n} \sum_ {m = 1} ^ {M} \left(\mathbb {E} _ {p \sim \mathcal {P} _ {m}} [ h (\mathbf {x}; p) ] - \mathbb {E} _ {p \sim \mathcal {P}} [ h (\mathbf {x}; p) ]\right) ^ {2}.
$$
The last term is the intra-group variance, which captures the discrepancy between the pixel groups  $\{\mathcal{P}_m\}_{m=1}^M$ . Theorem 3.2 guarantees that the variance of SG is no larger than that of NS, and SG
3The pixel-level contrastive loss  $\mathcal{L}_{\mathrm{contrast}}$  is an example of an aggregation function (up to normalizing constant) according to Eqn. (E.2).
The choice of  $c_{m}$  is flexible. For example, if the convex hull of the pixels in  $\mathcal{P}_m$  form a circle, then  $c_{m}$  can be taken as the geometric center.
has strictly less variance than NS as long as all the pixel groups do not share an equal mean over  $h(\mathbf{x};p)$ , which is almost-sure in medical images (See Figure 3). For SAG, Lemma 3.1 guarantees its variance is of the same magnitude as that of SG, and at worst differs by a factor of 2. Since the pixel/voxel-level contrastive loss  $\mathcal{L}_{\mathrm{contrast}}$  is an aggregation function over pixels by definition (E.2), it benefits from the variance-deduction property of SG/SAG. In Section 4.1, we will see that such variance reduction allows ARCO to achieve better segmentation accuracy, especially along the boundary of the anatomical regions (Figure 4).
Training Convergence. We further demonstrate the benefit of variance reduction estimation in terms of training stability. Specifically, leveraging techniques from standard optimization theory [87, 88, 89], we can show that variance-reduced gradient estimator through SG sampling leads to faster training convergence. Suppose we have a loss function  $\mathcal{L}(\theta)$  with the model parameter  $\theta$ , and use stochastic gradient descent (SGD) as the optimizer. A gradient estimate  $g(\theta) \approx \nabla \mathcal{L}(\theta)$  is computed at each iteration. It is well-known that the convergence of SGD depends on the quality of the estimate  $g(\theta)$  [87]. Specifically, we make the common assumptions that the loss function is smooth and the gradient estimate has bounded variance (More details in Appendix A.3), which can be formulated as below:
$$
\left\| \nabla \mathcal {L} (\theta) - \nabla \mathcal {L} \left(\theta^ {\prime}\right) \right\| _ {2} \leq L \left(\left\| \theta - \theta^ {\prime} \right\| _ {2}\right), \quad \mathbb {E} \left[ \left\| g (\theta) - \nabla \mathcal {L} (\theta) \right\| ^ {2} \right] \leq \sigma_ {g} ^ {2}.
$$
Under these two assumptions, the average expected gradient norm of the learned parameter satisfies the following:
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \| \nabla \mathcal {L} (\theta_ {t}) \| _ {2} ^ {2} \right] \leq C \left(\frac {1}{T} + \frac {\sigma_ {g}}{\sqrt {T}}\right).
$$
For general non-convex loss function, the above implies convergence to some local minimum. Importantly, the slow rate  $(\sigma_g / \sqrt{T})$  depends on standard deviation  $\sigma_{g}$ , indicating a faster convergence can indeed be achieved with a more accurate gradient estimate. This indicates our proposed sampling techniques demonstrate universality in variance reduction, as they can be applied to a wide range of scenarios that involve pixel/voxel-level sampling (See Appendix A.3). In Figure 5, we observe that using SG enables faster loss decay with smaller error bar, showing that it outperforms other methods in both convergence speed and stability. See Section 4.2 and Appendix A.3 for more details.
In this section, we present experimental results to validate our proposed methods across various datasets and different label ratios in Appendix B. We use 2D UNet [30] or 3D VNet [78] as our backbones. Further implementation details are discussed in Appendix C.
In this subsection, we first examine whether our proposed ARCO can generalize well across various datasets and label ratios. Then, we investigate to what extent ARCO coupled with two samplers can realize two essential properties: (1) model robustness; and (2) label efficiency. The quantitative results for all the compared methods on eight popular datasets: (1) Medical image segmentation tasks: three 2D benchmarks (i.e., ACDC [81], LiTS [82], MMWHS [83]), two 3D benchmarks (i.e., LA [84], in-house MP-MRI) under various label ratios (i.e.,  $1\%$ ,  $5\%$ ,  $10\%$ ) are collected in Table 1, Table 5 (Appendix G), Table 6 (Appendix H), and Table 9 (Appendix I and J), respectively; (2) General computer vision tasks: To further validate the effectiveness, we experiment on three popular segmentation benchmarks (i.e., Cityscapes [97], Pascal VOC 2012 [98], indoor scene segmentation dataset - SUN RGB-D [99]) in the semi-supervised full-label settings. We follow the identical setting [100] to sample labelled images to ensure that every class appears sufficiently in our three datasets, (i.e., CityScapes, Pascal VOC, and SUN RGB-D). The results are collected in Appendix Section K. Several consistent observations can be drawn from these extensive evaluations with eighteen segmentation networks.
$\bullet$  Superior Performance Across Datasets. We demonstrate that ARCO achieves superior performance across all datasets and label ratios. In specific, our experiments consider three 2D benchmarks
5Codes are available on here.
Table 1: Quantitative comparisons (DSC[%]  $\uparrow$  / ASD[voxel]  $\downarrow$ ) across the three labeled ratio settings (1%, 5%, 10%) on the ACDC benchmark. All experiments are conducted as [30, 90, 91, 92, 93, 8, 94, 9, 57, 95, 62, 50, 59, 96, 53, 16, 17] in the identical setting for fair comparisons. Best and second-best results are coloured blue and red, respectively. UNet-F (fully-supervised) and UNet-L (semi-supervised) are considered as the upper bound and the lower bound for the performance comparison. Note that, Right Ventricle  $\rightarrow$  RV, Myocardium  $\rightarrow$  Myo, Left Ventricle  $\rightarrow$  LV. We adopt the identical data augmentation (i.e., random rotation, random cropping, and horizontal flipping) for fair comparisons.
Method	ACDC
Average	1 Labeled (1%)	Average	3 Labeled (5%)	Average	7 Labeled (10%)
RV	Myo	LV	RV	Myo	LV	RV	Myo	LV
UNet-F [30]	91.5/0.996	90.5/0.606	88.8/0.941	94.4/1.44	91.5/0.996	90.5/0.606	88.8/0.941	94.4/1.44	91.5/0.996	90.5/0.606	88.8/0.941	94.4/1.44
UNet-L	40.3/22.7	29.0/25.4	43.6/15.3	48.2/27.5	51.7/13.1	36.9/30.1	54.9/4.27	63.4/5.11	79.5/2.73	65.9/0.892	82.9/2.70	89.6/4.60
EM [90]	43.1/18.1	38.7/23.1	42.0/12.0	48.7/19.4	59.8/5.64	44.2/11.1	63.2/3.23	71.9/2.57	75.7/2.73	68.0/0.892	76.5/2.70	82.7/4.60
CCT [91]	48.6/19.2	38.7/28.0	49.2/14.8	57.9/17.0	59.1/10.1	44.6/19.8	63.2/6.04	69.4/4.32	75.9/3.60	67.2/2.90	77.5/3.32	82.9/4.59
DAN [92]	48.9/17.5	45.4/19.7	41.0/8.88	60.4/23.8	56.4/15.1	47.1/21.7	58.1/11.6	63.9/11.9	76.5/3.01	75.7/2.61	73.3/3.11	80.5/3.31
URPC [57]	43.0/21.1	38.6/22.8	41.7/14.4	48.6/26.0	58.9/8.14	50.1/12.6	60.8/4.10	65.8/7.71	83.1/1.68	77.0/0.742	82.2/0.505	90.1/3.79
DTC [9]	51.7/17.5	39.3/23.3	54.6/9.12	61.3/20.2	56.9/7.59	35.1/19.7	62.9/6.01	72.7/7.59	84.3/4.04	83.8/3.72	83.5/4.63	85.6/3.77
DCT [93]	49.7/16.4	42.4/20.4	48.8/10.6	57.9/18.2	58.5/10.8	41.2/21.4	63.9/5.01	70.5/6.05	78.1/2.64	70.7/1.75	77.7/2.90	85.8/3.26
ICT [95]	42.1/21.0	36.5/18.5	43.4/11.1	46.3/33.5	59.0/6.59	48.8/11.4	61.4/4.59	66.6/3.83	80.6/1.64	75.1/0.898	80.2/1.53	86.6/2.48
MT [59]	42.9/15.1	32.5/21.9	46.2/8.99	50.1/14.7	58.3/11.2	39.0/21.5	58.7/7.47	77.3/4.72	80.1/2.33	75.2/1.22	79.2/2.32	86.0/3.45
UAMT [8]	36.9/15.2	32.5/21.9	46.2/8.99	50.1/14.7	48.3/9.14	37.6/18.9	50.1/4.27	57.3/4.17	81.8/4.04	79.9/2.73	80.1/3.32	85.4/6.07
SASNet [94]	42.6/24.8	29.8/34.7	45.4/13.3	52.5/26.6	57.8/6.36	47.9/11.7	59.7/4.51	65.8/2.87	84.7/1.83	81.8/0.769	82.9/1.73	89.4/2.99
CPS [62]	51.5/15.3	41.1/17.7	52.0/7.27	61.4/21.0	61.0/2.92	43.8/2.95	64.5/2.84	74.8/2.95	78.8/3.41	74.0/1.95	78.1/3.11	84.5/5.18
GCL [50]	59.7/14.3	49.5/25.3	60.9/6.28	68.8/11.5	70.6/2.24	56.5/1.99	70.7/1.67	84.8/3.05	87.0/0.751	86.9/0.584	81.8/0.820	92.5/0.849
MC-Net [96]	53.4/17.17	43.0/25.3	51.2/7.41	60.8/15.11	62.8/2.59	52.7/5.14	62.6/0.807	73.1/1.81	86.5/1.89	85.1/0.745	84.0/2.12	90.3/2.81
SS-Net [53]	63.4/2.94	64.7/3.32	57.0/1.81	68.4/3.70	65.8/2.28	57.5/3.91	65.7/2.02	74.2/0.896	86.8/1.40	85.4/1.19	84.3/1.44	90.6/1.57
ACTION [16]	81.0/3.45	76.9/3.09	78.4/2.07	87.5/5.17	86.6/1.20	85.2/0.734	84.7/0.909	89.8/1.97	87.2/1.47	86.1/0.976	85.7/1.11	89.7/2.33
MONA [17]	82.6/1.43	80.2/1.57	79.9/1.10	87.8/1.43	86.9/1.07	84.7/1.01	85.4/0.731	90.6/1.48	87.7/1.33	86.9/0.687	85.7/1.70	90.5/2.14
ARCO-SAG (ours)	84.9/1.47	81.7/1.98	81.9/0.903	90.9/1.52	87.1/0.848	85.6/0.414	85.1/0.930	90.6/1.20	88.5/1.40	87.1/0.635	86.2/1.04	92.2/2.53
ARCO-SG (ours)	85.5/0.947	81.8/1.19	83.8/0.801	90.9/0.853	88.7/0.841	88.2/0.618	85.9/0.673	91.9/1.23	89.4/0.776	90.2/0.701	86.5/0.787	91.6/0.839
Figure 4: Visual results on ACDC with  $1\%$ ,  $5\%$ ,  $10\%$  label ratios. ARCO consistently produce more accurate predictions on anatomical regions and boundaries compared to all other SSL methods.
(i.e., ACDC [81], LiTS [82], MMWHS [83]), two 3D benchmarks (i.e., LA [84], in-house MP-MRI), and different label ratios (i.e.,  $1\%$ ,  $5\%$ ,  $10\%$ ). As shown in Table 1, Table 5 (Appendix G), Table 6 (Appendix H), and Table 9 (Appendix I and J), we observe that our methods consistently outperform all the compared SSL-based methods by a considerable margin across all datasets and label ratios, which validates the superior performance of our proposed methods in both segmentation accuracy and label efficiency. For example, compared to the second-best MONA, our ARCO-SG under  $\{1\%, 5\%, 10\%\}$  label ratios achieves  $\{2.9\% \uparrow, 1.8\% \uparrow, 1.7\% \uparrow\}$ ,  $\{3.3\% \uparrow, 1.8\% \uparrow, 1.8\% \uparrow\}$ ,  $\{4.1\% \uparrow, 2.0\% \uparrow, 1.8\% \uparrow\}$ ,  $\{0.3\% \uparrow, 0.3\% \uparrow, 0.5\% \uparrow\}$ ,  $\{2.2\% \uparrow, 0.8\% \uparrow, 0.4\% \uparrow\}$  in average Dice across ACDC, LiTS, and MMWHS, MP-MRI, and LA, respectively. Our ARCO-SAG achieves  $\{84.9\%, 87.1\%, 88.5\%\}$ ,  $\{64.1\%, 67.3\%, 69.4\%\}$ ,  $\{86.1\%, 88.6\%, 89.3\%\}$ ,  $\{91.5\%, 92.5\%, 92.6\%\}$ ,  $\{73.2\%, 86.9\%, 89.1\%\}$  in averaged Dice across ACDC, LiTS, MMWHS, MP-MRI, and LA. These results indicate that our methods can generalize to different clinical scenarios and label ratios.
$\Theta$  Across Label Ratios and Robustified Methods. To further validate the label efficiency property of our ARCO, we evaluate our ARCO-SG and ARCO-SAG with limited labeled training data available (e.g.,  $1\%$  and  $5\%$ ). As demonstrated in Table 1, Table 5 (Appendix G), Table 6 (Appendix H), and Table 9 (Appendix I and J), our models under  $5\%$  label ratios surpass all the compared SSL methods by a significant performance margin. For example, compared to MONA, we observe our methods to
Table 2: Ablation on component aspect: (1) tailness/  $\mathcal{L}_{\mathrm{contrast}}$  ; (2) diversity/  $\mathcal{L}_{\mathrm{nn}}$
Method	DSC[%]↑	ASD[voxel]↓
Vanilla	49.3	7.11
ARCO-SAG (ours)	84.9	1.47
w/o tailness-SAG	60.9	4.11
w/o diversity	78.6	1.68
○ARCO-SG (ours)	85.5	0.947
w/o tailness-SG	60.9	4.11
w/o diversity	79.3	1.26
ARCO-NS	82.6	1.43
w/o tailness-NS	60.9	4.11
w/o diversity	75.2	2.07
Table 3: Ablation on loss function: (1) unsupervised loss/  $\mathcal{L}_{\mathrm{unsup}}$  ; (2) global instance discrimination loss/  $\mathcal{L}_{\mathrm{inst}}^{\mathrm{global}}$  ; and (3) local instance discrimination loss/  $\mathcal{L}_{\mathrm{inst}}^{\mathrm{local}}$
Method	DSC[%]↑	ASD[voxel]↓
• ARCO-SAG (ours)	84.9	1.47
w/o Lunsup	81.2	1.87
w/o Lglobalinst	84.0	2.64
w/o Llocalinst	83.3	2.63
○ ARCO-SG (ours)	85.5	0.947
w/o Lunsup	81.9	1.04
w/o Lglobalinst	84.1	2.10
w/o Llocalinst	83.8	2.11
push the best segmentation accuracy higher by  $0.3\% \sim 2.0\%$  in Dice on ACDC, LiTS, and MMWHS, MP-MRI, and LA, respectively. For example, the best segmentation accuracy on MMWHS rises from  $87.3\%$  to  $89.3\%$ . This suggests that our SSL-based approaches – without compromising the best achievable segmentation results – robustly improve performance using very limited labels, and further lead to a much-improved trade-off between SSL schemes and supervised learning schemes by avoiding a large amount of labeled data.
Similar to our results under  $5\%$  label ratio, our ARCO-SG and ARCO-SAG trained with  $1\%$  label ratio demonstrate sufficient performance boost compared to MONA by an especially significant margin, with up to  $0.3\% \sim 4.1\%$  relative improvement in Dice. Taking the extremely limited label ratio (i.e.,  $1\%$ ) as an indicator: (1) on 3D LA, ARCO-SG achieves  $2.2\%$  higher average Dice, and 6.64 lower average ASD than the second best MONA; (2) on LiTS, ARCO-SG achieves  $3.3\%$  higher average Dice, and 4.7 lower average ASD than the second best MONA; and (3) considering the more challenging clinical scenarios (i.e., 7 anatomical classes), ARCO-SG achieves  $5.1\%$  higher average Dice, and 2.26 lower average ASD than the second-best MONA on MMWHS. It highlights the superior performance of ARCO is not only from improved label efficiency but also credits to the superior model robustness.
Qualitative Results. We provide qualitative illustrations of ACDC, LiTS, MMWHS, LA, MP-MRI in Figure 4, Figure 6 (Appendix G), Figure 7 (Appendix H), Figure 8 (Appendix I), and Figure 9 (Appendix J), respectively. As shown in Figure 4, we observe that ARCO appears a significant advantage, where the edges and the boundaries of different anatomical regions are clearly more pronounced, such as RV and Myo regions. More interestingly, we found that in Figure 6, though all methods may confuse ambiguous tail-class samples such as small lesions, ARCO-SG and ARCO-SAG still produces consistently sharp and accurate object boundaries compared to the current approaches. We also observe similar results for ARCO on MMWHS in Figure 7 (Appendix H), where our approaches can regularize the segmentation results to be smooth and shape-consistent. Our findings suggest that ARCO improves model robustness mainly through distinguishing the minority tail-class samples.
In this subsection, we conduct various ablations to better understand our design choices. For all the ablation experiments the models are trained on ACDC with  $1\%$  labeled ratio.
Importance of Loss Components. We analyse several critical components of our method in the final performance and conduct comprehensive ablation studies on the ACDC dataset with a  $1\%$  label ratio to validate their necessity. First, at the heart of our method is the combination of three losses:  $\mathcal{L}_{\mathrm{contrast}}$  for tailness, and  $\mathcal{L}_{\mathrm{nn}}$  for diversity (See Section 3 for more details). We deactivate each component and then evaluate the resulting models, as shown in Table 2. As is shown, global contrastive loss  $\mathcal{L}_{\mathrm{contrast}}$  and nearest neighbor loss  $\mathcal{L}_{\mathrm{nn}}$  can boost performance by a large margin. Moreover, incorporating our methods (i.e., SG and SAG) consistently achieve superior model robustness gains compared to naive sampling (i.e., NS), both of which suggests the importance of these components. Second, we compare the impact of different loss function (e.g., unsupervised loss/ $\mathcal{L}_{\mathrm{unsup}}$ , global instance discrimination loss/ $\mathcal{L}_{\mathrm{inst}}^{\mathrm{global}}$ , and local instance discrimination loss/ $\mathcal{L}_{\mathrm{inst}}^{\mathrm{local}}$ ). As shown in Table 3, using each loss function consistently achieves significant performance gains, which demonstrates positive contribution of each component to performance gains.
Figure 5: Visualization of training trajectories given by  $\mathcal{L}_{\mathrm{contrast}}$  vs. epochs on ACDC under  $10\%$  label ratio. The proposed ARCO is compared in terms of different sampling methods: Naive Sampling (NS), Stratified Group (SG) Sampling, and Stratified-Antithetic Group (SAG) Sampling. The solid line and shaded area of each sampling method denote the mean and variance of test accuracies over 3 independent trials. Clearly, we observe SG sampling consistently outperforms the other sampling methods in convergence speed and training stability. SAG slightly outperforms NS.
Importance of Augmentation Components. We further investigate the impact of data augmentation on the ACDC dataset with a  $1\%$  label ratio. As is shown in Table 4 (in Appendix), employing each data augmentation strategy consistently results in notable performance improvements, underscoring the efficacy of these data augmentations. Of note, ARCO-SAG/ARCO-SG using all three augmentations and ARCO-SAG/ARCO-SG using no augmentation are considered as the upper bound and the lower bound for the performance comparison. These results show that each augmentation strategy systematically boosts performance by a large margin, which suggests improved robustness.
Stability Analyses. In Figure 5, we show the stability analysis results on ARCO over different sampling methods. As we can see, our SG and SAG sampling facilitates convergence during the training. More importantly, SG sampling has stable performance with small standard derivations, which aligns with our hypothesis that our proposed sampling method can be viewed as the form of variance regularization. Moreover, loss landscape visualization of different loss functions (Figure 3) reveals similar conclusions.
Extra Study. More investigations about (1) generalization across label ratios and frameworks in Appendix L; (2) final checkpoint loss landscapes in Appendix M; (3) ablation on different training settings are in Appendix N.
5 Conclusion and Discussion of Broader Impact
In this paper, we propose ARCO, a new semi-supervised contrastive learning framework for improved model robustness and label efficiency in medical image segmentation. Specifically, we propose two practical solutions via stratified group theory that correct for the variance introduced by the common sampling practice, and achieve significant performance benefits. Our theoretical findings indicate Stratified Group and Stratified-Antithetic Group Sampling provide practical means for improving variance reduction. It presents a curated and easily adaptable training toolkit for training deep networks that generalize well beyond training data in those long-tail clinical scenarios. Moreover, our sampling techniques can provide pragmatic solutions for enhancing variance reduction, thereby fostering their application in a wide array of real-world applications and sectors. These include but are not limited to 3D rendering, augmented reality, virtual reality, trajectory prediction, and autonomous driving. We hope this study could be a stepping stone towards quantifying the limitation of current self-supervision objectives for accomplishing such challenging safety-critical tasks.
Broader Impact. Defending machine learning models against inevitable variance will have the great potential to build more reliable and trustworthy clinical AI. Our findings show that the stratified group theory can provide practical means for improving variance reduction, leading to realistic deployments in a large variety of real-world clinical applications. Besides, we should address the challenges of fairness or privacy in the medical image analysis domain as our future research direction.
[87] Dimitri P Bertsekas and John N Tsitsiklis. Gradient convergence in gradient methods with errors. SIAM Journal on Optimization, 10(3):627-642, 2000.
[88] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-311, 2018.
[89] Zeyuan Allen-Zhu. How to make the gradients small stochastically: Even faster convex and nonconvex sgd. In NeurIPS, 2018.
[90] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, 2019.
[91] Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross-consistency training. In CVPR, 2020.
[92] Yizhe Zhang, Lin Yang, Jianxu Chen, Maridel Fredericksen, David P Hughes, and Danny Z Chen. Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In MICCAI, 2017.
[93] Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan Yuille. Deep co-training for semi-supervised image recognition. In ECCV, 2018.
[94] Shuailin Li, Chuyu Zhang, and Xuming He. Shape-aware semi-supervised 3d semantic segmentation for medical images. In MICCAI, 2020.
[95] Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. In IJCAI, 2019.
[96] Yicheng Wu, Minfeng Xu, Zongyuan Ge, Jianfei Cai, and Lei Zhang. Semi-supervised left atrium segmentation with mutual consistency training. In MICCAI, 2021.
[97] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[98] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 2015.
[99] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, 2015.
[100] Shikun Liu, Shuaifeng Zhi, Edward Johns, and Andrew J Davison. Bootstrapping semantic segmentation with regional contrast. arXiv preprint arXiv:2104.04465, 2021.
[101] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
[102] Tsung-Yi Lin, Piotr Dólar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[103] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
[104] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc V Gool. Revisiting contrastive methods for unsupervised learning of visual representations. In NeurIPS, 2021.
Appendix to
Rethinking Semi-Supervised Medical Image Segmentation:
A Variance-Reduction Perspective
Sec. A provides additional theoretical analysis.
Sec. B provides additional details on the training datasets.
Sec. C provides additional details on the implementation.
Sec. D provides details on 2D/3D methods in comparison.
Sec. E provides additional framework details.
Sec. F provides model architecture details.
Sec. G provides more experimental results on the 2D LiTS [82] dataset.
Sec. H provides more experimental results on the 2D MMWHS [83] dataset.
Sec. I provides more experimental results on the 3D LA [84] dataset.
Sec. J provides more experimental results on the 3D MP-MRI dataset.
Sec. K provides more experimental results on three semantic segmentation benchmarks (Cityscapes [97], Pascal VOC 2012 [98], SUN RGB-D [99]).
Sec. L forms a better understanding of generalization across label ratios and frameworks.
Sec. M provides final checkpoint loss landscapes.
Sec. N provides additional details on the hyperparameter selection.
A Theoretical Results Details
Proof. By definition of the estimate  $\widetilde{H}$ , since the sampling from each group is independent, the variance can be written as
$$
\operatorname {V a r} \left[ \widehat {H} _ {\mathrm {S G}} \right] = \frac {1}{n ^ {2}} \sum_ {m = 1} ^ {M} \sum_ {p \in \mathcal {D} _ {m}} \sigma_ {m} ^ {2}.
$$
Since  $|\mathcal{D}_m| = n_m$ , it follows that
$$
\operatorname {V a r} \left[ \widehat {H} _ {\mathrm {S G}} \right] = \sum_ {m = 1} ^ {M} \frac {n _ {m} ^ {2}}{n ^ {2}} \frac {\sigma_ {m} ^ {2}}{n _ {m}}.
$$