SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zifu Wang*
Xuefei Ning2
Matthew B. Blaschko
<sup>1</sup> ESAT-PSI, KU Leuven, Leuven, Belgium
$^{2}$  Department of Electronic Engineering, Tsinghua University, Beijing, China
Intersection over Union (IoU) losses are surrogates that directly optimize the Jaccard index. Leveraging IoU losses as part of the loss function have demonstrated superior performance in semantic segmentation tasks compared to optimizing pixelwise losses such as the cross-entropy loss alone. However, we identify a lack of flexibility in these losses to support vital training techniques like label smoothing, knowledge distillation, and semi-supervised learning, mainly due to their inability to process soft labels. To address this, we introduce Jaccard Metric Losses (JMLs), which are identical to the soft Jaccard loss in standard settings with hard labels but are fully compatible with soft labels. We apply JMLs to three prominent use cases of soft labels: label smoothing, knowledge distillation and semi-supervised learning, and demonstrate their potential to enhance model accuracy and calibration. Our experiments show consistent improvements over the cross-entropy loss across 4 semantic segmentation datasets (Cityscapes, PASCAL VOC, ADE20K, DeepGlobe Land) and 13 architectures, including classic CNNs and recent vision transformers. Remarkably, our straightforward approach significantly outperforms state-of-the-art knowledge distillation and semi-supervised learning methods. The code is available at https://github.com/zifuwangggg/JDTLosses.
The Jaccard index, also known as the intersection over union (IoU), is a widely used metric in the evaluation of semantic segmentation models. Its appeal lies in its scale invariance and its superior ability to reflect the perceptual quality of a model compared to pixel-wise accuracy [17, 40]. In line with the principles of empirical risk minimization in statistical learning theory, the metric used for evaluation should also be optimized during training [60]. Consequently, directly optimizing IoU via differentiable surrogates has found considerable attention in the literature [46, 49, 2, 72, 17, 73].
Notably, IoU is only defined when both predictions and ground-truth labels are discrete binary values, i.e., they reside on the vertices of a  $p$ -dimensional hypercube  $\{0,1\}^p$ . However, the output of neural networks are soft probabilities that are in the interior of the hypercube  $[0,1]^p$ . To extend the value of IoU from vertices to the entire hypercube, two popular strategies are currently in use. The first strategy relaxes set counting with norm functions, exemplified by the soft Jaccard loss (SJL) [46, 49] and the soft Dice loss (SDL) [57, 17]. The second strategy computes the Lovasz extension of the IoU, such as the Lovasz hinge loss [72] and the Lovasz-Softmax loss (LSL) [2]. These losses facilitate a plug-and-play use and have significantly enhanced the performance of segmentation models, outperforming pixel-wise losses such as the cross-entropy loss (CE) and the focal loss [35]. For instance, Rakhlin et al. [50] won the land cover segmentation task of the DeepGlobe Challenge [12] utilizing LSL. Additionally, SDL is becoming increasingly prominent in recent segmentation works [5, 9, 8, 29], such as Segment Anything [29]. Moreover, SJL and SDL are now the standard for training medical imaging models [17, 27].
*Correspondence to: zifu.wang@kuleuven.be
Despite their widespread application, current IoU losses lack the flexibility needed to accommodate key training techniques. Specifically, while these losses relax predictions from  $\{0,1\}^p$  to  $[0,1]^p$ , they neglect the possibility of labels residing in  $[0,1]^p$ , making them incompatible with soft labels. Soft labels have been employed in numerous state-of-the-art models and have proven effective in enhancing network accuracy [37, 68, 62] and calibration [20, 44, 42, 82]. For example, label smoothing (LS) [58] generates soft labels by taking a weighted average of one-hot hard labels and a uniform distribution over labels. In knowledge distillation (KD) [22], class probabilities generated by a teacher network serve as soft labels to guide a student model. In semi-supervised learning (SSL) such as MixMatch [4],  $k$  views of an unlabeled image are fed to a classifier, and the predictions are averaged to create soft labels.
Motivated by the limitation of existing IoU losses, particularly their inability to accommodate soft labels and, consequently, techniques like LS, KD, and SSL, we propose two variants of SJL, termed Jaccard metric losses (JMLs) since they are metrics on  $[0,1]^p$ . JMLs yield the same value as SJL on hard labels but are fully compatible with soft labels. Therefore, we can safely replace the existing implementation of SJL with JMLs without affecting the performance on hard labels. To embed our losses with use cases (LS, KD, SSL), we first introduce boundary label smoothing (BLS) which facilitates the integration of label smoothing into the segmentation task. BLS can be used independently and also synergizes with KD and SSL. Subsequently, we present a confidence-based scheme for selecting classes to contribute to the loss computation, thereby enhancing performance in KD. We conduct extensive experiments on 4 datasets (Cityscapes [11], PASCAL VOC [18], ADE20K [81], DeepGlobe Land [12]) across 13 architectures, including classic CNNs [21, 54] and recent vision transformers [69]. Our results demonstrate significant improvements over the cross-entropy loss. Moreover, our straightforward approach outperforms state-of-the-art segmentation KD and SSL methods by a substantial margin.
Given a segmentation output  $\dot{x} \in \{1, \dots, C\}^p$  and a ground-truth  $\dot{y} \in \{1, \dots, C\}^p$  where  $C$  is the number of classes, for each class  $c$ , we define the set of predictions as  $x^c = \{\dot{x} = c\}$ , the set of ground-truth as  $y^c = \{\dot{y} = c\}$ , the union as  $u^c = x^c \cup y^c$ , the intersection as  $v^c = x^c \cap y^c$ , the symmetric difference (the set of mispredictions) as  $m^c = (x^c \setminus y^c) \cup (y^c \setminus x^c)$ , and the Jaccard index as  $\mathrm{IoU}^c = |v^c| / |u^c|$ . For multi-class segmentation,  $\mathrm{IoU}^c$  are averaged across classes, yielding the mean IoU (mIoU). In the sequel, we will encode sets as binary vectors  $x^c, y^c, u^c, v^c, m^c \in \{0, 1\}^p$  where  $p$  is the number of pixels, and denote  $|x^c| = \sum_{i=1}^{p} x_i^c$  the cardinality of the corresponding set. For simplicity, we will drop the superscript  $c$  in the following.
In order to optimize IoU in a continuous setting, we need (almost everywhere) differentiable interpolations of this discrete score. In particular, we want to extend the IoU loss
$$
\Delta_ {\mathrm {I o U}}: x \in \{0, 1 \} ^ {p}, y \in \{0, 1 \} ^ {p} \mapsto 1 - \frac {| v |}{| u |} = \frac {| m |}{| y \cup m |} \tag {1}
$$
with  $\overline{\Delta}_{\mathrm{IoU}}$  so that it attains a value with any vector of predictions  $\tilde{x} \in [0,1]^p$ . In what follows, when the context is clear, we will use  $x$  and  $\tilde{x}$  interchangeably.
The soft Jaccard loss (SJL) [46, 49] generalizes IoU by realizing that when  $x, y \in \{0,1\}^p$ ,  $|v| = \langle x,y\rangle$  and  $|u| = |x| + |y| - |v| = \| x\|_1 + \| y\|_1 - \langle x,y\rangle$ . Therefore, SJL replaces the set notation with vector functions:
$$
\bar {\Delta} _ {\mathrm {S J L}, L ^ {1}}: x \in [ 0, 1 ] ^ {p}, y \in \{0, 1 \} ^ {p} \mapsto 1 - \frac {\langle x , y \rangle}{\| x \| _ {1} + \| y \| _ {1} - \langle x , y \rangle}. \tag {2}
$$
The  $L^1$  norm can be replaced with the squared  $L^2$  norm [17]:
$$
\bar {\Delta} _ {\mathrm {S J L}, L ^ {2}}: x \in [ 0, 1 ] ^ {p}, y \in \{0, 1 \} ^ {p} \mapsto 1 - \frac {\langle x , y \rangle}{\| x \| _ {2} ^ {2} + \| y \| _ {2} ^ {2} - \langle x , y \rangle}. \tag {3}
$$
2.2 The Limitation of Existing IoU Losses
The primary shortcoming of current IoU losses is that they do not necessarily have desired properties when presented with soft labels, i.e., when  $y \in [0,1]^p$ . This limitation impedes their application in crucial training techniques like LS, KD, and SSL.
Consider the case of  $\overline{\Delta}_{\mathrm{SJL},L^1}$ , and for simplicity, a single pixel scenario:  $\overline{\Delta}_{\mathrm{SJL},L^1} = 1 - \frac{xy}{x + y - xy}$ . It is easy to confirm that for any  $y > 0$ ,  $\overline{\Delta}_{\mathrm{SJL},L^1}$  is minimized at  $x = 1$  since it monotonically decreases as a function of  $x$ . Hence,  $\overline{\Delta}_{\mathrm{SJL},L^1}$  is in general not minimized when  $x = y$ , a basic property anticipated from a loss function. Further analysis for high-dimensional cases and additional experiments on real datasets are provided in Appendix C and E, respectively.
$\overline{\Delta}_{\mathrm{SJL},L^2}$  does not exhibit this issue, since  $\overline{\Delta}_{\mathrm{SJL},L^2} = 0 \Leftrightarrow |x|_2^2 + |y|_2^2 - 2\langle x,y\rangle = 0 \Leftrightarrow x = y$ . However, it is known to yield inferior results compared to its  $L^1$  counterpart, possibly due to its flatter nature around the minimum [17]. In practice, it is rarely used. For instance, in SMP [25], a popular open-source semantic segmentation project, only the  $L^1$  version is implemented. Our evaluations in Appendix E confirm its inferior performance relative to the  $L^1$  version. Additionally, the approach of substituting set notation with the  $L^1$  norm is widely utilized in numerous other works, including the soft Dice loss [57, 17], the soft Tversky loss [53], the focal Tversky loss [1], and others. The soft Dice loss is also included in the formulation of the PQ loss [61] which is used in panoptic segmentation [28]. Consequently, all of them struggle with soft labels.
Losses based on the Lovasz extension, such as the Lovasz-Softmax loss [2], the Lovasz hinge loss [72], and the PixIoU loss [73], cannot handle soft labels as the Lovasz extension is not well-defined for  $y \in (0,1)^p$ . More details and comparisons with the Lovasz-Softmax loss can be found in Appendix D and E, respectively. Automatically searched loss functions, such as Auto Seg-Loss [33] and AutoLoss-Zero [32], also fail to accommodate soft labels as their search space is confined to integral labels.
In summary, despite their widespread adoption in recent works on semantic segmentation [17, 27, 9, 8, 29] and panoptic segmentation [5, 61, 74, 75], these losses all exhibit a common shortcoming: an inability to handle soft labels. In this paper, we specifically concentrate on re-designing SJL. Other losses, including the soft Dice loss, the soft Tversky loss, and the focal Tversky loss, are addressed in our subsequent work [66].
2.3 Jaccard Metric Losses
We can rewrite the intersection  $|v|$  and the union  $|u|$  as a function of the set difference  $|m|$ :
$$
| v | = \frac {1}{2} (| x | + | y | - | m |) \text {a n d} | u | = | v | + | m |. \tag {4}
$$
Note that  $|m| = \| x - y\| _1$ . Combining these yields:
$$
| v | = \langle x, y \rangle = \frac {1}{2} \left(\| x + y \| _ {1} - \| x - y \| _ {1}\right), \tag {5}
$$
$$
| u | = \langle x, y \rangle + \| x - y \| _ {1} = \frac {1}{2} (\| x + y \| _ {1} + \| x - y \| _ {1}), \tag {6}
$$
where the equalities hold when  $x, y \in \{0, 1\}^p$ .
After eliminating erroneous combinations that have the same issue as SJL, we are left with two candidates  $\overline{\Delta}_{\mathrm{JML},1}$ ,  $\overline{\Delta}_{\mathrm{JML},2}:[0,1]^p\times [0,1]^p\to [0,1]$  that are defined as:
$$
\bar {\Delta} _ {\mathrm {J M L}, 1} = 1 - \frac {\| x + y \| _ {1} - \| x - y \| _ {1}}{\| x + y \| _ {1} + \| x - y \| _ {1}}, \tag {7}
$$
$$
\bar {\Delta} _ {\mathrm {J M L}, 2} = 1 - \frac {\langle x , y \rangle}{\langle x , y \rangle + \| x - y \| _ {1}}. \tag {8}
$$
It is a well-known result that  $\Delta_{\mathrm{IoU}}$  is a metric on  $\{0,1\}^p$  [30]. In Theorem 2.1 (see Appendix F for the proof), we show that both  $\overline{\Delta}_{\mathrm{JML},1}$  and  $\overline{\Delta}_{\mathrm{JML},2}$  are also metrics on  $[0,1]^p$ . Therefore, we call them Jaccard Metric Losses (JMLs).
Figure 1: Loss value vs. prediction with  $y = 0.1$  (left) and  $y = 0.9$  (right).
Theorem 2.1. Both  $\overline{\Delta}_{JML,1}$  and  $\overline{\Delta}_{JML,2}$  are metrics on  $[0,1]^p$ . Neither  $\overline{\Delta}_{SJL,L^1}$  nor  $\overline{\Delta}_{SJL,L^2}$  is a metric on  $[0,1]^p$ .
Recall the definition of a metric:
Definition 2.2 (Metric [14]). A mapping  $f: M \times M \to \mathbb{R}$  is called a metric on  $M$  if for all  $a, b, c \in M$ , it satisfies the following conditions: (i) (reflexivity).  $f(a, a) = 0$ . (ii) (positivity).  $a \neq b \Rightarrow f(a, b) > 0$ . (iii) (symmetry).  $f(a, b) = f(b, a)$ . (iv) (triangle inequality).  $f(a, c) \leq f(a, b) + f(b, c)$ .
Having a loss function  $\overline{\Delta}$  that is a metric carries numerous benefits. Reflexivity and positivity collectively imply that  $\forall x, y \in [0,1]^p$ ,  $x = y \Leftrightarrow \overline{\Delta} = 0$ , meaning that  $\overline{\Delta}$  would be compatible with soft labels. The triangle inequality also provides insightful guidance. Applied to KD, it yields:
$$
\overline {{\Delta}} (S, L) \leq \overline {{\Delta}} (S, T) + \overline {{\Delta}} (T, L). \tag {9}
$$
Here,  $S$  represents the student model,  $T$  the teacher model, and  $L$  the ground-truth labels. In the KD, we initially train the teacher model using the ground-truth labels, and then minimize the loss between the teacher and the student model. Equation (9) suggests that if we adhere to this process—equivalent to minimizing the right-hand side of the equation—we also minimize the upper bound on the student model's loss with respect to the ground-truth labels.
Furthermore, as Theorem 2.3 shows (see Appendix G for the proof), when only hard labels are involved,  $\overline{\Delta}_{\mathrm{JML},1}$ ,  $\overline{\Delta}_{\mathrm{JML},2}$  and  $\overline{\Delta}_{\mathrm{SJL},L^1}$  are identical. Hence, we can safely replace the existing implementation of SJL with JMLs. When soft labels are introduced,  $\overline{\Delta}_{\mathrm{JML},1}$  and  $\overline{\Delta}_{\mathrm{JML},2}$  might yield different values. We discuss their distinctions in Appendix H. From both theoretical and empirical perspectives, we find that  $\overline{\Delta}_{\mathrm{JML},1}$  is slightly more favorable than  $\overline{\Delta}_{\mathrm{JML},2}$ . As a default in our experiments, we use  $\overline{\Delta}_{\mathrm{JML},1}$ .
Theorem 2.3.  $\forall x\in [0,1]^p$ $y\in \{0,1\} ^p$  and  $x\in \{0,1\} ^p$ $y\in [0,1]^p$ $\overline{\Delta}_{JML,1} = \overline{\Delta}_{JML,2} = \overline{\Delta}_{SJL,L^1}$ $\exists x,y\in [0,1]^p,\overline{\Delta}_{JML,1}\neq \overline{\Delta}_{JML,2}\neq \overline{\Delta}_{SJL,L^1}$
Figure 1 plots the loss value of  $\overline{\Delta}_{\mathrm{JML,1}}$  and the cross-entropy loss (CE) for soft labels  $y = 0.1$  (left) and  $y = 0.9$  (right). Typically, the model is randomly initialized, resulting in a low initial confidence. Although the gradient of CE is substantial at the beginning, it plateaus as training progresses and the prediction is close to the target. Conversely,  $\overline{\Delta}_{\mathrm{JML,1}}$  offers effective supervision later in the process. However, when  $y$  is extremely small (e.g.,  $y = 0.1$ ),  $\overline{\Delta}_{\mathrm{JML,1}}$  can become overly steep, potentially causing problems in KD. This issue will be explored in greater detail in Section 2.4.2.
2.4 Use Cases
Soft labels find wide-ranging applications. In this paper, we investigate three of the most common use cases: label smoothing (LS), knowledge distillation (KD) and semi-supervised learning (SSL). We leave the discussion on SSL in Appendix I.
2.4.1 Label Smoothing
In LS [58], the one-hot encoding is combined with a uniform distribution under a smoothing coefficient  $\epsilon$ , resulting in soft labels  $SL^{\epsilon}$ . In JML-LS, a network  $H$  is trained with the following loss:
$$
\mathcal {L} _ {\mathrm {J M L - L S}} = \lambda_ {\mathrm {C E}} \mathcal {L} _ {\mathrm {C E}, \mathrm {L S}} + \lambda_ {\mathrm {J M L}} \mathcal {L} _ {\mathrm {J M L}, \mathrm {L S}} \tag {10}
$$
such that  $\mathcal{L}_{\mathrm{CE,LS}} = \mathrm{CE}(H,SL^{\epsilon})$  and  $\mathcal{L}_{\mathrm{JML,LS}} = \mathrm{JML}(H,SL^{\epsilon})$
LS provides a regularization effect during training. However, when we regard semantic segmentation as a pixel-wise classification task and apply LS to every pixel in the image without distinction, we fail to consider spatial differences. This is particularly evident with SegFormer-B3 on ADE20K, which achieves an overall pixel-wise accuracy of  $81.91 \pm 0.02\%$ , but drops to  $47.98 \pm 0.01\%$  in the boundary region. These areas, due to their inherent ambiguity, require stronger regularization. Moreover, our empirical findings suggest that smoothing non-boundary regions only yields marginal improvements. Therefore, we introduce boundary label smoothing (BLS), which only applies smoothing to labels near the boundary. We refer to the resulting loss as JML-BLS. Specifically, for every pixel  $i$ , we examine its  $k \times k$  neighborhood  $N_{i}$ , and the pixel  $i$  is regarded as a boundary pixel if there exists a pixel  $j \in N_{i}$  such that their ground-truth labels are different:  $y_{i} \neq y_{j}$ . This can be efficiently computed by applying a max pooling layer to the one-hot encoding.
2.4.2 Knowledge Distillation
In KD [22], besides the ground-truth label  $L$ , a student model  $S$  is trained to minimize the discrepancy to a teacher network  $T$  simultaneously:
$$
\mathcal {L} _ {\mathrm {C E}, \mathrm {K D}} = \mu_ {\mathrm {L}} \mathcal {L} _ {\mathrm {C E}, \mathrm {L}} + \mu_ {\mathrm {T}} \mathcal {L} _ {\mathrm {C E}, \mathrm {T}} \tag {11}
$$
where  $\mathcal{L}_{\mathrm{CE,L}} = \mathrm{CE}(S,L)$  and  $\mathcal{L}_{\mathrm{CE,T}} = \mathrm{CE}(S,T)$ .
In JML-KD, we add additional supervision from JML:
$$
\mathcal {L} _ {\mathrm {J M L}, \mathrm {K D}} = \nu_ {\mathrm {L}} \mathcal {L} _ {\mathrm {J M L}, \mathrm {L}} + \nu_ {\mathrm {T}} \mathcal {L} _ {\mathrm {J M L}, \mathrm {T}} \tag {12}
$$
where  $\mathcal{L}_{\mathrm{JML,L}} = \mathrm{JML}(S,L)$  and  $\mathcal{L}_{\mathrm{JML,T}} = \mathrm{JML}(S,T)$ .
Combining these, we have
$$
\mathcal {L} _ {\mathrm {J M L - K D}} = \lambda_ {\mathrm {C E}} \mathcal {L} _ {\mathrm {C E}, \mathrm {K D}} + \lambda_ {\mathrm {J M L}} \mathcal {L} _ {\mathrm {J M L}, \mathrm {K D}}. \tag {13}
$$
Existing segmentation KD methods rely on pixel-wise supervision either in the output [36, 70] or in the feature space [64, 55]. Our  $\mathcal{L}_{\mathrm{JML,T}}$  however, enables direct distillation of the teacher's IoU information to the student which aligns with the final evaluation metric.
Moreover, recall that JMLs are averaged for each class, making it crucial to determine which classes contribute to the loss value at each iteration. We refer to these classes as active classes. In KD, the student uses information from non-target classes to learn the class relationship from the teacher [59, 77]. However, a teacher network may also include noisy and unhelpful predictions for unimportant classes. Therefore, we propose to filter out unimportant classes based on the confidence level of the teacher network. Specifically, JMLs are computed only over classes where the teacher's confidence exceeds a predefined threshold. This approach has two benefits: first, the student will not be distracted by irrelevant classes, aiding in generalization. Second, as shown in Figure 1, we find that the loss value of JMLs can become excessively steep around the target when the ground-truth is low (e.g.  $y = 0.1$ ). Therefore, ignoring these classes can stabilize the training process.
We have also observed that the student benefits from a teacher trained with JML-BLS. In contrast, in classification tasks, it has been found that a teacher trained with LS can detrimentally affect the student's performance [44]. This is often cited as a counterexample to the notion that a more accurate teacher will necessarily distill a better student. We believe that the teacher, trained through JML-BLS, acquires intricate boundary information, and this "dark knowledge" is implicitly passed to the student, resulting in a more accurate student.
Experimental setups. We adopt Pytorch Image Models (timm) [67], which provides implementations and ImageNet [13] pre-trained weights for various backbones. In our experiments, CNN backbones include ResNet-101/50/18 [21] and MobileNetV2 [54]; transformer backbones contain MiT-B5/B3/B1/B0 [69]. Segmentation methods consist of DeepLabV3+ [7], DeepLabV3 [6], PSPNet
[78], UNet [51] and SegFormer [69]. We evaluate models on Cityscapes [11], PASCAL VOC [18], ADE20K [81] and DeepGlobe Land [12]. In summary, 13 models and 4 datasets are studied in the sequel. More details of each model are in Appendix A and training recipes are in Appendix B.
Evaluation metrics. To provide a comprehensive comparison, we report both overall pixel-wise accuracy (Acc) and mean intersection over union (mIoU). To evaluate the effects of our methods on calibration, we present the expected calibration error (ECE) [20] and the ECE computed only over the boundary region, denoted as BECE. For Cityscapes, PASCAL VOC, and ADE20K, we repeat the experiments 3 times (except for SSL experiments that are single runs) and report performance on the validation set. For DeepGlobe Land, we conduct 5-fold cross-validation. All results are presented in the format of mean  $\pm$  standard deviation. We do not apply any test-time augmentation.
3.1 Results on Accuracy
We report results on Cityscapes (Table 1), PASCAL VOC (Table 2), ADE20K (Table 3), and DeepGlobe Land (Table 13, Appendix J). Key takeaways from our results are as follows:
JML significantly improves accuracy (Acc and mIoU). Compared to training with CE alone, incorporating JML as part of the loss function can significantly enhance a model's mIoU. The improvement is typically more than  $2\%$  on Cityscapes, PASCAL VOC, and ADE20K. Additionally, it can also increase a model's Acc. This suggests that the benefits of JML are not solely due to its alignment with the evaluation metric (mIoU), but also because it aids the overall optimization process.
JML benefits more from soft labels. For instance, the improvements of mIoU for DL3-R101 on PASCAL VOC are  $0.48\%$  (CE vs. CE-BLS) and  $1.25\%$  (JML vs. JML-BLS), respectively. We adopted training procedures that are heavily optimized for CE, which might explain why JML requires stronger regularization. More qualitative results can be found in Figure 7 and Figure 8 (Appendix L).
KD is more effective than BLS, while BLS performs well on datasets with simple boundary condition. Both BLS and KD consistently improve the model's accuracy. Despite its simplicity, BLS can yield significant improvements, especially on PASCAL VOC where boundary condition is less complex. As boundary condition becomes more intricate, as in Cityscapes and ADE20K, KD, seen as learned label smoothing [76], typically outperforms BLS.
SOTA results on segmentation KD. Without bells and whistles, our simple approach that only uses soft labels greatly exceeds SOTA segmentation KD methods, as shown in Table 4. Indeed, the model trained only with hard labels in JML already achieves a comparable result as some of these methods. For example, on PASCAL VOC, DL3-R18 achieves  $74.42 \pm 0.52\%$  mIoU while CIRKD [70], a complex distillation method only attains  $74.50\%$  mIoU. Note that our baseline is not stronger than theirs: our DL3-R18 trained with CE has  $72.47 \pm 0.33\%$  mIoU while their number is  $73.21\%$  (because we use a smaller batch size, see Appendix B).
SOTA results on segmentation SSL. By incorporating JML-BLS with AugSeg [80] (refer to Appendix I for further details), we achieve SOTA segmentation SSL results as illustrated in Table 11 (Appendix I). Notably, improvements are particularly significant on smaller splits - exceeding a  $4\%$  enhancement over AugSeg on the 92 split.
3.2 Results on Calibration