SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} f (x) = f \left(\sum_ {i = 1} ^ {p} \alpha_ {i} x _ {i}\right) (52) \\ = f \left(\left(1 - \alpha_ {p}\right) \frac {\sum_ {i = 1} ^ {p - 1} \alpha_ {i} x _ {i}}{1 - \alpha_ {p}} + \alpha_ {p} x _ {p}\right) (53) \\ \geq \left(1 - \alpha_ {p}\right) f \left(\frac {\sum_ {i = 1} ^ {p - 1} \alpha_ {i} x _ {i}}{1 - \alpha_ {p}}\right) + \alpha_ {p} f \left(x _ {p}\right) (54) \\ = \left(1 - \alpha_ {p}\right) f \left(\frac {1 - \alpha_ {p} - \alpha_ {p - 1}}{1 - \alpha_ {p}} \frac {\sum_ {i = 1} ^ {p - 2} \alpha_ {i} x _ {i}}{1 - \alpha_ {p} - \alpha_ {p - 1}} + \frac {\alpha_ {p - 1}}{1 - \alpha_ {p}} x _ {p - 1}\right) + \alpha_ {p} f (x _ {p}) (55) \\ \geq \left(1 - \alpha_ {p} - \alpha_ {p - 1}\right) f \left(\frac {\sum_ {i = 1} ^ {p - 2} \alpha_ {i} x _ {i}}{1 - \alpha_ {p} - \alpha_ {p - 1}}\right) + \alpha_ {p - 1} f \left(x _ {p - 1}\right) + \alpha_ {p} f \left(x _ {p}\right) (56) \\ \geq \dots \geq \sum_ {i = 1} ^ {p} \alpha_ {i} f \left(x _ {i}\right) (57) \\ = \sum_ {i = 1} ^ {p} \alpha_ {i} l \left(x _ {i}\right) (58) \\ \geq \mathbb {C} l (x). (59) \\ \end{array}
$$
Figure 3: A counterexample that both  $\overline{\Delta}_{\mathrm{JML},1}$  and  $\overline{\Delta}_{\mathrm{JML},2}$  are not piece-wise concave in 2D.
Although both  $\overline{\Delta}_{\mathrm{JML,1}}$  and  $\overline{\Delta}_{\mathrm{JML,2}}$  are not concave as Figure 2 shows, if we can divide the hypercube into several sub-spaces such that  $\overline{\Delta}_{\mathrm{JML,1}}$  and  $\overline{\Delta}_{\mathrm{JML,2}}$  are equal at the vertices and concave at each sub-space, then we can still apply Theorem H.3. However, the fact that both of them are piece-wise concave in 1D does not hold in higher dimension. Indeed, we can find a counterexample in 2D. Let  $y = [0.5,0.5], x = [0.4087,0.7855], x' = [0.6285,0.7551]$ . Both  $x$  and  $x'$  are in the sub-space  $yab$  and
$$
0. 5 \bar {\Delta} _ {\mathrm {J M L}, 1} (x, y) + 0. 5 \bar {\Delta} _ {\mathrm {J M L}, 1} \left(x ^ {\prime}, y\right) > \bar {\Delta} _ {\mathrm {J M L}, 1} \left(0. 5 x + 0. 5 x ^ {\prime}, y\right), \tag {60}
$$
$$
0. 5 \bar {\Delta} _ {\mathrm {J M L}, 2} (x, y) + 0. 5 \bar {\Delta} _ {\mathrm {J M L}, 2} \left(x ^ {\prime}, y\right) > \bar {\Delta} _ {\mathrm {J M L}, 2} \left(0. 5 x + 0. 5 x ^ {\prime}, y\right). \tag {61}
$$
How do they perform empirically? In Table 10, we compare  $\overline{\Delta}_{\mathrm{JML},1}$  and  $\overline{\Delta}_{\mathrm{JML},2}$  on Cityscapes and PASCAL VOC using DL3-R18. We find  $\overline{\Delta}_{\mathrm{JML},1}$  is slightly superior to  $\overline{\Delta}_{\mathrm{JML},2}$ .
I Semi-supervised Learning
We focus on augmentation-based semi-supervised learning approaches [4, 80]. Broadly speaking, in their approaches, the data consists of both supervised and unsupervised images, and a model  $H$  is trained to minimize the following loss:
$$
\mathcal {L} _ {\mathrm {C E}, \mathrm {S S L}} = \eta_ {\mathrm {S}} \mathcal {L} _ {\mathrm {C E}, \mathrm {S}} + \eta_ {\mathrm {U}} \mathcal {L} _ {\mathrm {C E}, \mathrm {U}} \tag {62}
$$
such that  $\mathcal{L}_{\mathrm{CE,S}} = \mathrm{CE}(H(\mathcal{A}_s(x_s)),L(\mathcal{A}_s(x_s)))$  and  $\mathcal{L}_{\mathrm{CE,U}} = \mathrm{CE}(H(\mathcal{A}_u(x_u)),T(\mathcal{A}_u(x_u)))$
Here,  $\mathcal{A}_s$  and  $\mathcal{A}_u$  are data augmentations applied to labeled images  $x_{s}$  and unlabeled images  $x_{u}$ , respectively.  $L$  is the ground-truth label and  $T$  is usually the exponential moving averaging of the model weights.
In JML-SSL, we introduce additional supervision from JML:
$$
\mathcal {L} _ {\mathrm {J M L}, \mathrm {S S L}} = \theta_ {\mathrm {S}} \mathcal {L} _ {\mathrm {J M L}, \mathrm {S}} + \theta_ {\mathrm {U}} \mathcal {L} _ {\mathrm {J M L}, \mathrm {U}} \tag {63}
$$
where  $\mathcal{L}_{\mathrm{JML,S}} = \mathrm{JML}(H(\mathcal{A}_s(x_s)),L(\mathcal{A}_s(x_s)))$  and  $\mathcal{L}_{\mathrm{JML,U}} = \mathrm{JML}(H(\mathcal{A}_u(x_u)),T(\mathcal{A}_u(x_u)))$
Combining these yields
$$
\mathcal {L} _ {\mathrm {J M L - S S L}} = \lambda_ {\mathrm {C E}} \mathcal {L} _ {\mathrm {C E}, \mathrm {S S L}} + \lambda_ {\mathrm {J M L}} \mathcal {L} _ {\mathrm {J M L}, \mathrm {S S L}}. \tag {64}
$$
To further harness the power of soft labels, we follow the pipelines of AugSeg [80] and stack BLS on top of their augmentations:  $\mathcal{A}_s(x_s) = \mathrm{BLS}(\mathcal{A}_{s,\mathrm{AugSeg}}(x_s))$  and  $\mathcal{A}_u(x_u) = \mathrm{BLS}(\mathcal{A}_{u,\mathrm{AugSeg}}(x_u))$ .
Here,  $\mathcal{A}_{s,\mathrm{AugSeg}}$  and  $\mathcal{A}_{u,\mathrm{AugSeg}}$  denote augmentations applied to labeled and unlabeled images in AugSeg, respectively. We refer readers to their paper for more details.
As shown in Table 11, we achieve SOTA results on semi-supervised learning. In Table 12, we ablate the role of each loss term on PASCAL VOC using DL3+-R101.
Table 11: Comparing with SOTA segmentation SSL methods on PASCAL VOC of various splits using DL3+-R101. All results are mIoU (\%). Red: the best in a column.
Method	92	183	366	732	1464
Supervise	43.92	59.10	65.88	70.87	74.97
U2PL [63]	67.98	69.15	73.66	76.16	79.49
iMAS [79]	70.0	75.3	79.1	80.2	82.0
UniMatch [71]	75.2	77.2	78.8	79.9	81.2
AugSeg [80]	71.09	75.45	78.80	80.37	81.36
JML-SSL	76.43	78.47	79.16	81.21	82.27
Table 12: Ablating each loss term on PASCAL VOC using DL3+-R101. All results are mIoU (%). Red: the best in a column.
Loss	92	183
\( \mathcal{L}_{\text{CE,S}} \)	43.92	59.10
\( +\mathcal{L}_{\text{CE,U}} \)	71.09	75.45
\( +\mathcal{L}_{\text{JML,S}} \)	72.89	76.20
\( +\mathcal{L}_{\text{JML,U}} \)	74.74	77.74
+BLS	76.43	78.47
J More Results on DeepGlobe Land
We provide additional results on DeepGlobe Land in Table 13. Furthermore, we present the outcomes of BLS using DL3-R50 and KD using DL3-R18 across 5 folds of DeepGlobe Land in Table 14. Given that DeepGlobe Land is a relatively small dataset, comprising only 803 images, results for each fold can vary significantly. Nevertheless, when observing the differences between CE and JML, the mean improvement exceeds twice the standard deviation.
Table 13: Results on DeepGlobe Land (\%). Best results within CE and JML groups are highlighted in red and green, respectively. Best results across CE and JML groups are underscored.
Model	Metric	CE	CE-BLS	CE-KD	JML	JML-BLS	JML-KD
UNet-R50	Acc ↑	86.89 ± 0.87	86.95 ± 0.99	-	86.59 ± 0.84	86.87 ± 0.79	-
mIoU ↑	68.80 ± 1.07	68.86 ± 0.81	-	69.29 ± 0.87	69.73 ± 0.89	-
ECE ↓	1.77 ± 0.52	1.68 ± 0.46	-	12.34 ± 1.18	11.68 ± 1.55	-
BECE ↓	21.93 ± 0.83	21.90 ± 1.21	-	40.97 ± 0.90	39.22 ± 1.06	-
UNet-R18	Acc ↑	84.99 ± 1.15	85.28 ± 0.97	85.51 ± 0.76	85.46 ± 0.56	85.53 ± 0.59	85.78 ± 0.66
mIoU ↑	64.38 ± 1.50	64.93 ± 0.79	66.40 ± 0.86	66.95 ± 0.25	66.87 ± 0.30	67.62 ± 0.27
ECE ↓	1.05 ± 0.48	1.47 ± 0.60	1.38 ± 0.61	10.32 ± 0.862	9.15 ± 0.79	9.83 ± 1.53
BECE ↓	23.23 ± 0.54	22.32 ± 0.75	25.23 ± 0.75	40.17 ± 0.48	38.13 ± 0.63	39.24 ± 0.94
UNet-MB2	Acc ↑	84.93 ± 1.11	84.92 ± 1.22	85.64 ± 1.02	85.54 ± 0.88	85.82 ± 0.66	85.90 ± 0.73
mIoU ↑	64.38 ± 1.21	64.03 ± 1.73	66.29 ± 1.39	66.63 ± 0.73	67.04 ± 0.77	67.46 ± 0.99
ECE ↓	1.64 ± 0.75	2.04 ± 0.74	1.52 ± 0.34	10.22 ± 1.37	10.01 ± 0.92	9.67 ± 0.86
BECE ↓	21.15 ± 0.89	20.26 ± 0.74	24.76 ± 1.17	40.40 ± 0.89	39.28 ± 0.27	38.82 ± 1.12
Table 14: Results of BLS using DL3-R50 and KD using DL3-R18 on 5 folds of DeepGlobe Land. All results are mIoU (\%).
Method	Loss	0	1	2	3	4	μ ± σ
BLS	CE	67.56	69.82	69.59	67.88	69.46	68.86 ± 0.81
JML	68.13	70.48	70.60	69.49	69.93	69.73 ± 0.89
Diff	0.57	0.66	1.01	0.61	1.47	0.86 ± 0.34
KD	CE	64.93	67.47	66.22	66.37	67.00	66.40 ± 0.86
JML	67.20	67.96	67.50	67.62	67.84	67.62 ± 0.27
Diff	2.27	0.49	1.28	1.25	0.84	1.22 ± 0.59
K More Results on Calibration
CE is a proper scoring rule [20], while the soft Dice loss is not [3]. Although the soft Dice loss can significantly increase a model's segmentation performance, it can hurt model calibration [41, 3]. We observe the same phenomenon with JML and demonstrate a potential solution by training the model with soft labels as delineated in Section 3.2. Calibration of the model can be further improved using post-hoc [20, 15, 39, 47, 52] and trainable calibration methods [31, 48]. However, these methods extend beyond the scope of this paper.
Besides, we find that models trained with JML, despite having poorer top-class calibration as quantified by ECE, actually attain superior multi-class calibration as evaluated by the static calibration error (SCE) [45]. In Table 15, we report both ECE and SCE on Cityscapes and PASCAL VOC using DL3-R101 $^2$ . It is crucial to highlight that models trained with soft labels may exhibit worse SCE. On one side, soft labels provide an additional regularization during training, which is advantageous for calibration [44]. However, on the flip side, soft labels can represent unrealistic distributions, potentially detrimental for multi-class calibration.
Table 15: Calibration results on Cityscapes and PASCAL VOC using DL3-R101. Best results within CE and JML groups are highlighted in red and green, respectively. Best results across CE and JML groups are underscored.
Dataset	Metric	CE	CE-BLS	JML	JML-BLS
CS	ECE (%)	0.76 ± 0.05	0.97 ± 0.04	2.74 ± 0.03	2.09 ± 0.02
SCE (%)	2.18 ± 0.02	2.46 ± 0.04	2.08 ± 0.01	2.02 ± 0.01
BECE (%)	16.14 ± 0.22	11.59 ± 0.12	30.78 ± 0.19	21.20 ± 0.07
BSCE (%)	23.81 ± 0.03	23.89 ± 0.04	23.40 ± 0.01	23.23 ± 0.01
VOC	ECE (%)	2.33 ± 0.03	2.01 ± 0.04	3.97 ± 0.10	3.20 ± 0.04
SCE (%)	2.57 ± 0.02	2.59 ± 0.04	2.40 ± 0.06	2.35 ± 0.03
BECE (%)	20.54 ± 0.12	17.25 ± 0.14	32.70 ± 0.09	22.30 ± 0.02
BSCE (%)	20.94 ± 0.04	20.89 ± 0.01	20.58 ± 0.01	24.81 ± 0.10
L Figures
Figure 4: The best  $\epsilon$  and mIoU (\%) for different  $k$  on PASCAL VOC using DL3-R18.
Figure 5: Effects of  $\epsilon$  on PASCAL VOC using DL3-R101/50/18.
Figure 6: Code to compute active classes.
Figure 7: Qualitative results on Cityscapes.
Figure 8: Qualitative results on PASCAL VOC.
In our code, there was an error in the computation of SCE. As a result, values reported here differ from those in our earlier arXiv version.
(a) DL3-R101 mIoU (%)
(b) DL3-R101 ECE  $(\%)$
(c) DL3-R50 mIoU (%)
(d) DL3-R50 ECE  $(\%)$
(e) DL3-R18 mIoU (%)
(f) DL3-R18 ECE  $(\%)$
Figure 5: Effects of  $\epsilon$  on PASCAL VOC using DL3-R101/50/18.  $\epsilon = 0$  is the baseline (no smoothing). The highest and the lowest mean values are highlighted in red and green horizontal lines, respectively.
import torch   
from torch.nn/modules.loss import _Loss   
class JDTLoss(_Loss):   
...   
#prob.shape  $=$  label.shape  $=$  (batch_size，num_classes,  $\mathrm{H}*\mathrm{W})$  def compute.active_classes(  $|$  self, prob, label, active_classes_mode, num_classes): if active_classes_mode  $= =$  "ALL": mask  $=$  torch.ones(num_classes, dtype  $\equiv$  torch bool) elif active_classes_mode  $= =$  "PRESENT": mask  $=$  torch.argmax(layer, dim=1).unique() elif active_classes_mode  $= =$  "PROB": mask  $=$  torch.amax(prob, dim=(0,2)) > selfthreshold elif active_classes_mode  $= =$  "LABEL": mask  $=$  torch.amax(layer, dim=(0,2)) > selfthreshold elif active_classes_mode  $= =$  "BOTH": mask  $=$  torch.amax(prob + label, dim=(0,2)) > self. threshold   
active_classes  $=$  torch.zeros(num_classes, dtype  $\equiv$  torch(bool device  $\equiv$  prob_device)   
active_classes[mask]  $= 1$    
return active_classes
Figure 6: Code to compute active classes. Please refer to our codebase for more details.
Image and GT
Teacher
Student with KD
Student without KD
Figure 7: Qualitative results on Cityscapes. Confidence maps are in black and white. Predictions are in RGB.
Image and GT
Student with KD
Student without KD
Figure 8: Qualitative results on PASCAL VOC. Confidence maps are in black and white. Predictions are in RGB.