SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CE-BLS and CE-KD sometimes hurt calibration. Contrary to the common belief in classification [44, 42, 82], we find that both CE-BLS and CE-KD can sometimes deteriorate model calibration. For instance, in Cityscapes experiments, the lowest ECE is usually obtained with CE only. Nevertheless, CE-BLS can still improve model calibration near the boundary.
JML compromises calibration. Although the soft Dice loss can significantly improve a model's segmentation performance, it is well known to yield poorly calibrated models [41, 3]. We confirm that this is also the case for JML. Interestingly, while models trained with JML exhibit inferior top-class calibration as measured by ECE, they actually achieve better multi-class calibration as indicated by the static calibration error (SCE). We provide more detailed results on calibration in Appendix K.
JML-BLS and JML-KD consistently improve calibration. Although JML presents challenges in model calibration, these can be greatly mitigated by training models with soft labels. In JML experiments, both JML-BLS and JML-KD reliably enhance model calibration.
Table 1: Results on Cityscapes (\%). Best results within CE and JML groups are highlighted in red and green, respectively. Best results across CE and JML groups are underscored.
Model	Metric	CE	CE-BLS	CE-KD	JML	JML-BLS	JML-KD
DL3-R101	Acc ↑	96.10 ± 0.02	96.11 ± 0.03	-	96.10 ± 0.04	96.25 ± 0.03	-
mIoU ↑	78.67 ± 0.32	78.70 ± 0.26	-	80.29 ± 0.17	80.66 ± 0.24	-
ECE ↓	0.76 ± 0.05	0.97 ± 0.04	-	2.74 ± 0.03	2.09 ± 0.02	-
BECE ↓	16.14 ± 0.22	11.59 ± 0.12	-	30.78 ± 0.19	21.20 ± 0.07	-
DL3-R50	Acc ↑	95.77 ± 0.08	95.83 ± 0.08	-	95.93 ± 0.04	96.05 ± 0.02	-
mIoU ↑	76.45 ± 0.68	77.02 ± 0.82	-	78.68 ± 0.42	79.10 ± 0.35	-
ECE ↓	0.62 ± 0.02	1.00 ± 0.03	-	2.84 ± 0.02	2.18 ± 0.01	-
BECE ↓	16.23 ± 0.16	11.67 ± 0.18	-	31.12 ± 0.10	20.51 ± 0.12	-
DL3-R18	Acc ↑	95.28 ± 0.02	95.31 ± 0.05	95.40 ± 0.04	95.46 ± 0.04	95.49 ± 0.05	95.59 ± 0.02
mIoU ↑	72.88 ± 0.44	73.14 ± 0.12	74.09 ± 0.28	75.55 ± 0.13	76.26 ± 0.17	76.68 ± 0.33
ECE ↓	0.68 ± 0.06	0.88 ± 0.01	0.98 ± 0.03	3.07 ± 0.01	2.49 ± 0.05	2.47 ± 0.01
BECE ↓	17.28 ± 0.19	13.25 ± 0.16	16.26 ± 0.18	32.10 ± 0.08	22.66 ± 0.10	22.57 ± 0.17
DL3-MB2	Acc ↑	95.24 ± 0.01	95.28 ± 0.03	95.30 ± 0.04	95.45 ± 0.02	95.51 ± 0.04	95.56 ± 0.03
mIoU ↑	72.19 ± 0.12	72.54 ± 0.26	72.88 ± 0.15	75.32 ± 0.42	75.81 ± 0.14	75.94 ± 0.24
ECE ↓	0.69 ± 0.05	0.96 ± 0.03	0.99 ± 0.04	3.09 ± 0.02	2.43 ± 0.04	2.50 ± 0.03
BECE ↓	17.64 ± 0.15	13.14 ± 0.20	16.55 ± 0.10	32.08 ± 0.29	21.61 ± 0.09	22.57 ± 0.04
PSP-R18	Acc ↑	95.13 ± 0.05	95.07 ± 0.03	95.14 ± 0.02	95.25 ± 0.02	95.30 ± 0.02	95.39 ± 0.01
mIoU ↑	72.61 ± 0.34	72.43 ± 0.05	72.79 ± 0.15	74.96 ± 0.16	75.31 ± 0.13	75.75 ± 0.31
ECE ↓	0.68 ± 0.07	0.97 ± 0.01	1.08 ± 0.05	3.17 ± 0.01	2.44 ± 0.02	2.52 ± 0.01
BECE ↓	17.54 ± 0.24	13.61 ± 0.20	16.96 ± 0.21	32.65 ± 0.07	22.30 ± 0.15	23.17 ± 0.08
Table 2: Results on PASCAL VOC (\%). Best results within CE and JML groups are highlighted in red and green, respectively. Best results across CE and JML groups are underscored.
Model	Metric	CE	CE-BLS	CE-KD	JML	JML-BLS	JML-KD
DL3-R101	Acc↑	94.68 ± 0.02	94.75 ± 0.05	-	94.97 ± 0.13	95.34 ± 0.08	-
mIoU↑	78.39 ± 0.09	78.87 ± 0.44	-	80.26 ± 0.45	81.52 ± 0.41	-
ECE↓	2.33 ± 0.03	2.01 ± 0.04	-	3.97 ± 0.10	3.20 ± 0.04	-
BECE↓	20.54 ± 0.12	17.25 ± 0.14	-	32.70 ± 0.09	22.30 ± 0.02	-
DL3-R50	Acc↑	94.23 ± 0.05	94.28 ± 0.06	-	94.60 ± 0.02	94.87 ± 0.02	-
mIoU↑	76.93 ± 0.32	77.23 ± 0.18	-	78.97 ± 0.12	79.76 ± 0.15	-
ECE↓	2.05 ± 0.03	1.77 ± 0.08	-	4.19 ± 0.02	3.25 ± 0.02	-
BECE↓	20.95 ± 0.13	17.30 ± 0.15	-	33.07 ± 0.10	20.41 ± 0.01	-
DL3-R18	Acc↑	92.96 ± 0.05	93.17 ± 0.09	93.13 ± 0.07	93.28 ± 0.14	93.72 ± 0.05	93.75 ± 0.05
mIoU↑	72.47 ± 0.33	72.99 ± 0.42	72.96 ± 0.30	74.42 ± 0.52	75.60 ± 0.24	75.89 ± 0.29
ECE↓	1.83 ± 0.12	1.26 ± 0.08	1.68 ± 0.05	4.92 ± 0.14	3.79 ± 0.03	3.94 ± 0.07
BECE↓	21.83 ± 0.29	17.87 ± 0.08	19.42 ± 0.04	34.35 ± 0.09	22.45 ± 0.08	22.15 ± 0.19
DL3-MB2	Acc↑	92.47 ± 0.12	92.45 ± 0.02	92.54 ± 0.04	92.74 ± 0.07	93.13 ± 0.11	93.21 ± 0.01
mIoU↑	70.14 ± 0.54	70.36 ± 0.27	70.54 ± 0.10	72.35 ± 0.32	73.25 ± 0.18	73.55 ± 0.27
ECE↓	1.92 ± 0.14	1.46 ± 0.09	1.78 ± 0.03	5.19 ± 0.05	4.14 ± 0.08	4.24 ± 0.01
BECE↓	22.65 ± 0.34	19.62 ± 0.45	20.62 ± 0.34	35.10 ± 0.17	23.38 ± 0.19	23.09 ± 0.11
PSP-R18	Acc↑	92.93 ± 0.12	93.09 ± 0.09	93.13 ± 0.18	93.20 ± 0.08	93.65 ± 0.04	93.60 ± 0.10
mIoU↑	72.10 ± 0.55	72.76 ± 0.52	72.84 ± 0.57	74.20 ± 0.24	75.04 ± 0.22	74.93 ± 0.36
ECE↓	1.92 ± 0.22	1.40 ± 0.16	1.81 ± 0.16	4.76 ± 0.04	3.70 ± 0.01	3.99 ± 0.07
BECE↓	22.39 ± 0.39	18.95 ± 0.12	20.55 ± 0.47	34.72 ± 0.06	23.45 ± 0.20	23.53 ± 0.37
Table 3: Results on ADE20K (\%). Best results within CE and JML groups are highlighted in red and green, respectively. Best results across CE and JML groups are underscored.
Model	Metric	CE	CE-BLS	CE-KD	JML	JML-BLS	JML-KD
SegFormer-B5	Acc ↑	82.48 ± 0.09	82.53 ± 0.06	-	82.72 ± 0.11	82.80 ± 0.10	-
mIoU ↑	48.17 ± 0.35	48.50 ± 0.26	-	49.95 ± 0.28	50.22 ± 0.32	-
ECE ↓	5.94 ± 0.05	5.14 ± 0.15	-	11.84 ± 0.06	9.12 ± 0.15	-
BECE ↓	19.98 ± 0.11	14.59 ± 0.17	-	34.78 ± 0.19	19.88 ± 0.28	-
SegFormer-B3	Acc ↑	81.91 ± 0.02	82.04 ± 0.11	-	82.31 ± 0.13	82.54 ± 0.12	-
mIoU ↑	46.51 ± 0.29	46.88 ± 0.24	-	48.68 ± 0.27	49.24 ± 0.24	-
ECE ↓	5.56 ± 0.12	4.56 ± 0.13	-	11.57 ± 0.13	10.63 ± 0.22	-
BECE ↓	19.98 ± 0.23	4.56 ± 0.13	-	34.47 ± 0.07	28.17 ± 0.36	-
SegFormer-B1	Acc ↑	78.75 ± 0.05	78.92 ± 0.11	78.96 ± 0.03	79.41 ± 0.10	79.53 ± 0.12	79.62 ± 0.04
mIoU ↑	38.79 ± 0.20	38.95 ± 0.25	39.29 ± 0.16	41.70 ± 0.17	42.03 ± 0.29	42.52 ± 0.17
ECE ↓	4.18 ± 0.04	3.18 ± 0.16	4.90 ± 0.05	11.33 ± 0.08	10.46 ± 0.18	10.50 ± 0.12
BECE ↓	18.70 ± 0.09	13.89 ± 0.20	19.23 ± 0.09	33.25 ± 0.05	27.38 ± 0.25	26.29 ± 0.33
SegFormer-B0	Acc ↑	75.49 ± 0.06	76.37 ± 0.02	76.48 ± 0.02	76.85 ± 0.04	76.98 ± 0.11	76.99 ± 0.06
mIoU ↑	30.49 ± 0.15	33.48 ± 0.07	34.54 ± 0.11	36.65 ± 0.13	36.78 ± 0.08	37.05 ± 0.11
ECE ↓	2.61 ± 0.12	2.10 ± 0.20	3.56 ± 0.04	10.77 ± 0.12	9.88 ± 0.17	10.22 ± 0.04
BECE ↓	17.61 ± 0.17	13.33 ± 0.12	17.53 ± 0.13	31.84 ± 0.28	26.09 ± 0.29	24.99 ± 0.14
Table 4: Comparing with SOTA segmentation KD methods on Cityscapes and PASCAL VOC. All results are mIoU (%). JML-KD: we increase the batch size to 16 for Cityscapes experiments to match training details in [70, 23, 24].
Dataset	Model	CE	SKD [36]	IFVD [64]	CD [55]	CIRKD [70]	MasKD [24]	DIST [23]	JML-KD
CS	DL3-R18	72.88	75.42	75.59	75.55	76.38	77.00	77.10	77.91 ± 0.16
DL3-MB2	72.19	73.82	73.50	74.66	75.42	75.26	-	77.53 ± 0.20
PSP-R18	72.61	73.29	73.71	74.36	74.73	75.34	76.31	77.33 ± 0.38
VOC	DL3-R18	72.47	73.51	73.85	74.02	74.50	-	-	75.89 ± 0.29
PSP-R18	72.10	74.07	73.54	73.99	74.78	-	-	74.93 ± 0.36
4 Ablation Studies
4.1 JML Weights
It is common in recent works [5, 9, 8, 29] to balance CE and the Dice loss with equal weights. For JML, we adopt  $0.25 / 0.75$  in all our experiments and find it slightly superior than  $0.5 / 0.5$ .
Table 5: Ablating different values of  $\lambda_{\mathrm{CE}} / \lambda_{\mathrm{JML}}$  on PASCAL VOC using DL3-R101 and DL3-R18. All results are mIoU (\%). Red: the best in a row. Green: the worst in a row.
Model	0.10/0.90	0.25/0.75	0.50/0.50	0.90/0.10	1.00/0.00
DL3-R101	79.80 ± 0.40	80.26 ± 0.45	79.92 ± 0.12	78.73 ± 0.30	78.39 ± 0.09
DL3-R18	73.67 ± 0.39	74.42 ± 0.52	74.30 ± 0.29	73.21 ± 0.23	72.47 ± 0.33
4.2 JML-BLS
Although BLS is sensitive to the choice of  $\epsilon$ , it effectively increases model accuracy and calibration. The effect of the smoothing coefficient  $\epsilon$  on PASCAL VOC with DL3-R101/50/18 is shown in Figure 5 (Appendix L). Interestingly, for DL3-R50 and DL3-R18, the optimal  $\epsilon$  that achieves the highest mIoU also yields the lowest ECE.
We need a strong smoothing coefficient near the boundary. The optimal  $\epsilon$  for different kernel size  $k$  on PASCAL VOC with DL3-R18 is shown in Figure 4 (Appendix L). Note that  $k = \infty$  implies LS is applied to every pixel, i.e. vanilla LS. Generally, as  $k$  increases, we need to decrease the strength of smoothing. The best result is obtained when we only smooth a small region near the boundary with a strong smoothing coefficient ( $k = 3$  and  $\epsilon = 0.50$ ).
4.3 JML-KD
Loss terms. We examine the contribution of each loss term in Table 6 using a DL3-R18 student. Adding JML terms significantly improves the student's performance.
Table 6: Evaluating different losses terms on Cityscapes and PASCAL VOC using a DL3-R18 student.  ${\mathcal{L}}_{\mathrm{{JML}} - \mathrm{{BLS}}}$  means we train the teacher with JML-BLS. All results are mIoU (%). Red: the best in a column.
\( {\mathcal{L}}_{\text{CE,L }} \)	\( {\mathcal{L}}_{\text{JML,L }} \)	\( {\mathcal{L}}_{\text{CE,KD }} \)	\( {\mathcal{L}}_{\text{JML,KD }} \)	\( {\mathcal{L}}_{\text{JML-BLS }} \)	CS	VOC
✓	-	-	-	-	72.88 ± 0.44	72.47 ± 0.33
✓	✓	-	-	-	75.55 ± 0.13	74.42 ± 0.52
✓	✓	✓	-	-	75.74 ± 0.25	74.65 ± 0.42
✓	✓	✓	✓	-	76.16 ± 0.37	75.05 ± 0.31
✓	✓	✓	✓	✓	76.68 ± 0.33	75.89 ± 0.29
Filtering out unimportant classes based on teacher's confidence is useful. We examine the impact of active classes on PASCAL VOC with DL3-R18 in Table 7. In particular, we propose to ignore classes where the soft label, i.e. teacher's confidence, is low (marked as LABEL). And in ALL, we include all classes. In PRESENT, we select the class with the maximum confidence from the teacher's prediction in the class dimension. In PROB, we skip classes where the student itself is not confident. In BOTH, we take both the teacher's and student's confidences into account. The code to compute active classes is in Figure 6 (Appendix L).
We observe that using ALL classes in JML-KD can misguide the student. In most of the teacher's output classes, the confidence is usually very low. It can be challenging and potentially detrimental for the student to precisely mimic these numbers. With PRESENT, the performance remains similar to that without soft labels. This is because the effectiveness of using soft labels come from the non-argmax classes. PROB and BOTH achieve similar performance, but both are worse than LABEL.
Table 7: Comparing active classes on PASCAL VOC using DL3-R18. All results are mIoU (%).
Active Classes	ALL	PRESENT	PROB	BOTH	LABEL (Ours)
JML-KD	75.21 ± 0.33	74.50 ± 0.25	75.51 ± 0.53	75.39 ± 0.31	75.89 ± 0.29
Teacher's calibration is beneficial. In Sec. 2.4.2, we suggest that a teacher trained with JML-BLS offers more boundary information to the student. But what exactly is this boundary information? We
believe it relates to the teacher's calibration. The student, with less capacity, often struggles to mimic a more powerful teacher, especially near ambiguous boundary regions. However, if the teacher is more calibrated and outputs a less peaked distribution, the student will learn this uncertainty rather than attempting to match an unrealistic distribution that it lacks the capacity to replicate.
In Table 8, we compare three teachers with various accuracy and calibration on PASCAL VOC. In particular, both T2 and T3 are trained with JML-BLS but with different smoothing parameter  $\epsilon$ , while T1 is not. T2, although having a lower mIoU, is more calibrated than T1. Consequently, T2's student is more accurate and better calibrated than T1's.
Table 8: Comparing 3 teachers of different accuracy and calibration on PASCAL VOC (%). Prefix T stands for the teacher and prefix S the student. Red: the best in a row.
Metrics	T1	T2	T3
T ECE ↓	4.20	3.47	3.24
T BECE ↓	33.17	20.73	20.42
T mIoU ↑	79.09	78.75	79.82
S ECE ↓	4.76 ± 0.09	4.01 ± 0.07	3.94 ± 0.07
S BECE ↓	33.86 ± 0.10	22.21 ± 0.06	22.15 ± 0.05
S mIoU ↑	75.05 ± 0.31	75.43 ± 0.35	75.89 ± 0.29
IoU is a commonly employed metric in semantic segmentation, and IoU losses strive to optimize this metric directly. IoU only obtains values when both predictions and ground-truth labels are discrete binary vectors  $\{0,1\}^p$ , but the neural network often predicts soft probabilities (after the softmax or sigmoid layer) in  $[0,1]^p$ . Interpolating IoU values from  $\{0,1\}^p$  to  $[0,1]^p$  has primarily followed two routes. One involves relaxing set counting as norm functions, as seen in the soft Jaccard loss [46, 49], the soft Dice loss [57], the soft Tversky loss [53] and the focal Tversky loss [1]. The other capitalizes on the fact that IoU is submodular [72], allowing for the application of the convex Lovasz extension of submodular functions. For instance, the Lovasz-Softmax loss [2], the Lovasz hinge loss [72] and the PixIoU loss [73]. Nevertheless, as IoU values are extended for predictions from  $\{0,1\}^p$  to  $[0,1]^p$ , the fact that labels can also fall within  $[0,1]^p$  is often overlooked. As a result, these IoU losses are incompatible with soft labels.
Besides semantic segmentation, IoU is adopted across a wide spectrum of fields. Due to its discrete nature, its probabilistic extensions [56] have found use in object detection [19], medical imaging [16] and information retrieval [26, 43].
6.1 How to tune the hyper-parameters of JML?
In this section, we delve into some critical hyper-parameters of JML. For a comprehensive list, please refer to the accompanying code.
$\mathbf{mIoU}^{\mathrm{D}} / \mathbf{mIoU}^{\mathrm{I}} / \mathbf{mIoU}^{\mathrm{C}}$  (default:  $1.0 / 0.0 / 0.0$ ): the weight of the loss to optimize  $\mathbf{mIoU}^{\mathrm{D}} / \mathbf{mIoU}^{\mathrm{I}} / \mathbf{mIoU}^{\mathrm{C}}$  [65]. The appropriate choice is mainly dependent on the targeted evaluation metrics. Given that the prevailing metric is  $\mathbf{mIoU}^{\mathrm{D}}$  (per-dataset mIoU), we recommend to set  $\mathbf{mIoU}^{\mathrm{D}}$  to 1.0 and  $\mathbf{mIoU}^{\mathrm{I}} / \mathbf{mIoU}^{\mathrm{C}}$  to 0.0. However, it is imperative to acknowledge the inherent trade-offs when using JML to optimize different metrics [65].
alpha/beta (default: 1.0/1.0): the coefficient of false positives/negatives in the Tversky loss. For instance, when alpha and beta are both set to 1.0, the configuration corresponds to JML. Conversely, an alpha and beta value of 0.5 each leads to DML [66]. When the evaluation metric is IoU, JML is advised, while it is the Dice score, DML is more appropriate. The general Tversky loss is useful when separate weights are required for false positives/negatives.
active_classes_mode (default: PRESENT for hard labels and ALL for soft labels): the mode to compute active classes. With hard labels, it is suggested to use PRESENT, since the loss aligns more effectively with the evaluation metric, especially when (i) the dataset contains a large number of classes (e.g. ADE20K), and/or (ii) evaluated with fine-grained mIoUs [65]. With soft labels, the optimal choice varies based on specific applications (see Table 7).
6.2 How to use JML?
Combine JML with CE. Our findings suggest that incorporating CE, particularly during the initial stage of training, can expedite convergence (as illustrated in Figure 1). Consequently, relying solely on JML is not recommended. For a discussion on the balancing weights, please consult Section 4.1.
Tune the hyper-parameters of JML. We have endeavored to minimize the necessity of extensive tuning. In the majority of scenarios, default hyper-parameter values (as illustrated above) should suffice. However, some situations might benefit from further refinement. In particular, pay attention to active_classes_mode when dealing with soft labels.
Tune the hyper-parameters of training settings. Current training recipes have been heavily optimized for CE. Although we find JML generally aligns with these hyper-parameters, due diligence is required. Notably, one of the advantages of JML is its ability to speed up convergence in the later training phase (as depicted in Figure 1). As a consequence, training with a combination of CE and JML often converges in considerably fewer epochs compared to training with CE alone. Please refer to [65] for a comparison of the number of epochs with those in MMSegmentation.
Be careful with distributed training. In our concurrent work [65], we observe that the loss presents trade-offs when optimizing different mIoU variants. If the per-GPU batch size is small (which is often the case) and the loss is computed on each GPU independently, the loss may inadvertently optimize for mIoU $^{\mathrm{I}}$  (per-image mIoU). This can lead to suboptimal outcomes when evaluated with mIoU $^{\mathrm{D}}$  (per-dataset mIoU).
Be careful with extremely class-imbalance. Aligned with mIoU, JML is designed to compute the loss on a per-class basis and subsequently average these individual losses. In preliminary experiments on some medical datasets, we identified instances of severe class imbalances. In such scenarios, the class-wise loss may inadvertently amplify the significance of underrepresented classes, potentially disturb the training process. To address this, one might consider (i) oversampling the minority classes, and/or (ii) adopting a class-agnostic loss computation, where the intersection and union are calculated over all pixels.
In both this study and our subsequent work [66], the focus is to extend the losses in the field of image segmentation. It would be intriguing to apply these losses to other tasks, such as long-tailed classification [34]. Moreover, although we adopt these losses in the label space, they present potential in quantifying the similarity between two feature vectors [24], potentially serving as an alternative to the  $L^p$  norm or cosine similarity.
This paper is driven by the observation that current IoU losses fall short when dealing with soft labels, which substantially limits their adaptability to crucial training techniques. To address this limitation, we introduce the Jaccard metric losses (JMLs). While these losses are identical to the soft Jaccard loss in a conventional hard-label setting, they offer full compatibility with soft labels.
Our results demonstrate that integrating JMLs with label smoothing, knowledge distillation, and semi-supervised learning leads to notable improvements in both accuracy and calibration. This is consistent across a spectrum of datasets and network architectures, encompassing classic CNNs as well as recent vision transformers. Remarkably, the proposed methods, which are simple and solely rely on soft labels, surpass state-of-the-art segmentation knowledge distillation and semi-supervised learning techniques by a significant margin.
In our follow-up study [66], we delve into the extensions of various other losses, including the soft Dice loss, soft Tversky loss, and focal Tversky loss. Given their equivalence to their original counterparts in a standard hard-label context and their enhanced compatibility with soft labels, we recommend to replace the existing implementations with ours.
We acknowledge support from the Research Foundation - Flanders (FWO) through project numbers G0A1319N and S001421N, and funding from the Flemish Government under the Onderzoeksprogramma Artificièle Intelligentie (AI) Vlaanderen programme. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government.
We thank Ilya Bogdanov for his help on the proof.
[1] Nabila Abraham and Naimul Mefraz Khan. A Novel Focal Tversky Loss Function with Improved Attention U-Net for Lesion Segmentation. ISBI, 2019.
[2] Maxim Berman, Amal Rannen Triki, and Matthew B. Blaschko. The Lovasz-softmax loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks. CVPR, 2018.
[3] Jeroen Bertels, David Robben, Dirk Vandermeulen, and Paul Suetens. Theoretical analysis and experimental validation of volume bias of soft Dice optimized segmentation maps in the context of inherent uncertainty. MIA, 2021.
[4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. MixMatch: A Holistic Approach to Semi-Supervised Learning. NeurIPS, 2019.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Detection with Transformers. ECCV, 2020.
[6] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking Atrous convolution for semantic image segmentation. arXiv, 2017.
[7] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with Atrous separable convolution for semantic image segmentation. ECCV, 2018.
[8] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention Mask Transformer for Universal Image Segmentation. CVPR, 2022.
[9] Bowen Cheng, Alexander G Schwing, and Alexander Kirillov. Per-Pixel Classification is Not All You Need for Semantic Segmentation. NeurIPS, 2021.
[10] MMSegmentation Contributors. MMSegmentation: OpenMMLab Semantic Segmentation Toolbox and Benchmark, 2020.
[11] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR, 2016.
[12] Ilke Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar. DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images. CVPR Workshop, 2018.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.
[14] Michel Marie Deza and Elena Deza. Encyclopedia of Distances. Springer, 2009.
[15] Zhipeng Ding, Xu Han, Peirong Liu, and Marc Niethammer. Local Temperature Scaling for Probability Calibration. ICCV, 2021.
[16] Reuben Dorent, Thomas Booth, Wenqi Li, Carole H. Sudre, Sina Kafiabadi, Jorge Cardoso, Sebastien Ourselin, and Tom Vercauteren. Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasets. MIA, 2020.