Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels
Authors: Zifu Wang
Xuefei Ning
Matthew B. Blaschko
Venue: NeurIPS 2023 poster (2023)
Keywords: Semantic Segmentation

Abstract: Intersection over Union (IoU) losses are surrogates that directly optimize the Jaccard index. Leveraging IoU losses as part of the loss function have demonstrated superior performance in semantic segmentation tasks compared to optimizing pixel-wise losses such as the cross-entropy loss alone. However, we identify a lack of flexibility in these losses to support vital training techniques like label smoothing, knowledge distillation, and semi-supervised learning, mainly due to their inability to process soft labels. To address this, we introduce Jaccard Metric Losses (JMLs), which are identical to the soft Jaccard loss in standard settings with hard labels but are fully compatible with soft labels. We apply JMLs to three prominent use cases of soft labels: label smoothing, knowledge distillation and semi-supervised learning, and demonstrate their potential to enhance model accuracy and calibration. Our experiments show consistent improvements over the cross-entropy loss across 4 semantic segmentation datasets (Cityscapes, PASCAL VOC, ADE20K, DeepGlobe Land) and 13 architectures, including classic CNNs and recent vision transformers. Remarkably, our straightforward approach significantly outperforms state-of-the-art knowledge distillation and semi-supervised learning methods. The code is available at \href{https://github.com/zifuwanggg/JDTLosses}{https://github.com/zifuwanggg/JDTLosses}.

Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels
Zifu Wang'*
Xuefei Ning?
Matthew B. Blaschko
ESAT-PSI, KU Leuven, Leuven, Belgium
Department of Electronic Engineering, Tsinghua University, Beijing, China
Abstract
Intersection over Union (IoU) losses are surrogates that directly optimize the Jaccard index. Leveraging IoU losses as part of the loss function have demonstrated superior performance in semantic segmentation tasks compared to optimizing pixel- wise losses such as the cross-entropy loss alone. However, we identify a lack of flexibility in these losses to support vital training techniques like label smoothing, knowledge distillation, and -uperviseee learning, mainly due to their inability to process soft labels. To address this, introduce Jaccard Metric Losses (JMLs), which are identical to the soft Jaccard loss in standard settings with hard labels but are fully compatible with soft labels. We apply JMLs to three prominent use cases of soft labels: label smoothing, knowledge distillation and semi-supervised learning, and demonstrate their potential enhance model accuracy and calibration. Our experiments show consistent improvements over the cross-entropy loss across 4 semantic segmentation datasets (Cityscapes, PASCAL VOC, ADE20K, DeepGlobe Land) and 13 architectures, including classic CNNs and recent vision transformers. Remarkably, straightforward approach significantly outperforms state-of-the- art knowledge distillation and semi-supervised learning methods. The code is available httpss/gghabbboom///uwwg///////o
Introduction
The Jaccard index, also known as the intersection over union (IoU), is widely used metric in the evaluation of semantic segmentation models. Its appeal lies in its scale invariance and its superior ability to reflect the perceptual quality of model compared to pixel-wise accuracy [17 40 In line with the principles of empirical risk minimization in statistical learning theory, the metric used for evaluation should also be optimized during training (60). Consequently, directly optimizing IoU via 73).
differentiable surrogates has found considerable attention in the literature [46] 49 2
Notably, IoU is only defined when both predictions and ground-truth labels discrete binary values, i.e., they reside on the vertices of a p-dimensional hypercube (0, 1)" However, the output of neural networks are soft probabilities that the value of IoU from vertices to the entire hypercube, two popular strategies are currently in use. The first strategy relaxes set counting with norm functions, exemplified by the soft Jaccard loss (SJI.) [6 49] and the soft Dice loss (SDL) [57 17]. The second strategy computes the Lovasz extension of the IoU, such as the Lovasz hinge loss and the Lovasz-Softmax loss (LSL)) 2 These losses facilitate plug-and-play use and have significantly enhanced the performance of segmentation models, outperforming pixel-wise losses such as the cross-entropy loss (CE) and the focal loss (35). For instance, Rakhlin et al. (50] won the land cover segmentation task of the DeepGlobe Challenge 12 utilizing LSL, Additionally, SDL is becoming increasingly prominent in recent segmentation works [5 9 29 such as Segment Anything [29]. Moreover, SJL and SDL now the standard
the interior of the hypercube [o, 1]2. To extend
for training medical imaging models 27).
"Correspondence to: zifu.wang bul.uven.be
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Despite their widespread application, current IoU losses lack the flexibility needed to accommodate key training techniques. Specifically, while these losses relax predictions from {0. 1 to [o. 1 they neglect the possibility of labels residing in [0, 1JP, making them incompatible with soft labels. Soft. labels have been employed in numerous state-of-the-art models and have proven effective in enhancing network accuracy [37 68, 62] and calibration [20, 44 42
[2]. For example, label smoothing (LS) [58] generates soft labels by taking a weighted average of one-hot hard labels and a uniform distribution over labels. In knowledge distillation (KD) [22), class probabilities generated by a teacher network serve as soft labels to guide a student model. In semi-supervised learning (SSL) such MixMatch [4) & views of unlabeled image are fed to classifier, and the predictions averaged to create soft labels.
Motivated by the limitation of existing IoU losses, particularly their inability to accommodate soft labels and, consequently, techniques like L.,, KD, and SSI., we propose two variants of SJI,, termed Jaceard metric losses (JMLs) since they are metrics on [0, JMLs yield the same value as SJL on hard labels but are fully compatible with soft labels. Therefore, we can safely replace the existing implementation of SJL with JMLs without affecting the performance on hard labels. To embed our losses with use cases (LS, KD, SSL), we first introduce boundary label smoothing (BLS) which facilitates the integration of label smoothing into the segmentation task. BLS can be used independently and also synergizes with KD and SSL. Subsequently, we present a confidence- based scheme for selecting classes to contribute to the loss computation, thereby enhancing performance in KD. We conduct extensive experiments on 4 datasets (Cityscapes [T]) PASCAL VOC [8]. ADE20K 88, DeepGlobe Land [22 across 13 architectures, including classic CNNs 21
54 and recent
vision transformers [69] Our results demonstrate significant improvements over the cross-entropy
loss. Moreover, our straightforward approach outperforms stateoof-the-ar segmentation KD and SSL methods by a substantial margin.
2 Methods
2.1 Preliminaries
Given a segmentation output E (I. and a ground-truth ÿ E {1..... C}2 where C' is the number of classes, for each class c, we define the set of predictions as x { = c). the set of ground-truth as y {ÿ c), the union as uf x" U y the intersection as of x" n y", the symmetric difference (the set of mispredictions) as m (z (f) U (v/) (t), and the Jaccard index as loU = J0e ///u]. For multi-class segmentation, IoU are averaged across classes, yielding the mean loU (mloU). In the sequel, we will encode sets binary vectors xº, yf, u², m E {0. } where p is the number of pixels, and denote |x*|| = Σ. x the cardinality of the corresponding set. For simplicity, we will drop the superscript in the following.
In order to optimize IoU in continuous setting, we need (almost everywhere) differentiable interpo- lations of this discrete score. In particular, we want to extend the IoU loss
m
AIOU : E 10, 10,11 +1
(I)
Ju]
y
Mml
with Ars so that it attains a value with any vector of predictions ã E [0, 1P. In what follows, when the context is clear, we will use and ã interchangeably.
The soft Jaccard loss (SJL.)
46
49
generalizes IoU by realizing that when x, y E (0,1), |o|
(x, y) and |u| = |æ|| + lyl
|l|
| 1 + ||vll (x, v). Therefore, SJL replaces the set notation
with vector functions:
(a, v)
ASIL :2
[0, 1] E {0, 10.1 +1 | + [lvll (xt, v²
(2)
The I norm can be replaced with the squared 7,2 norm 17
{x, v)
SSIL x 10. 10.1P E (0. 1p
(3)
x 2 +
|
(z. y)

---

2.2 The Limitation of Existing IoU Losses
The primary shortcoming of current IoU losses is that they do not necessarily have desired properties when presented with soft labels. i.e., when E [0, 1|.. This limitation impedes their application in crucial training techniques like LS, KD, and SSL.
z-リーzø is easy to confirm that for any > O, ASIL, is minimized at x = since monotonically decreases as function of Hence, Asn, is in general not minimized when r: = basic property anticipated from a loss function. Further analysis for high-dimensional cases and additional experiments on real datasets are provided in Appendix CrvennixCCandd and respectively.
Consider the case of ASH,L', , and for simplicity, single pixel scenario: ASH.
ASH, does not cxhibit this issue, since ASH, = + |x|3 + |u13 - 2(xx y) = + x
However, it is known to yield inferior results compared to its L counterpart, possibly due to its flatter nature around the minimum [17] In practice, it is rarely used. For instance, in SMP (25], popular open-source semantic segmentation project, only the L version is implemented. Our evaluations in Appendix confirm its inferior performance relative to the L' version. Additionally, the approach of substituting set notation with the L norm is widely utilized in numerous other works, including the soft Dice loss [57 17). the soft Tversky loss [53] the focal Tversky loss [1], and others. The soft Dice loss is also included in the formulation of the PQ loss (61 which is used in panoptic segmentation 28
Consequently, all of them struggle with soft labels.
Losses based on the Lovasz extension, such as the Lovasz-Softmaa loss [2], the Lovasz hinge loss (72. and the PixIoU loss 73,, cannot handle soft labels the Lovasz extension is not well-defined for y E (0, 1)P. More details and comparisons with the Lovasz-Softmax loss can be found in Appendix D and El respectively. Automatically searched loss functions, such as Auto Seg-Loss [33] and AutoLoss Zero [32] also fail to accommodate soft labels as their search space confined to integral labels.
In summary, despite their widespread adoption in recent works on semantic segmentation 17
8 (9] and panoptic segmentation [5] 61 77, 75), these losses all exhibit a common shortcoming: an inability to handle soft labels. In this paper, we specifically concentrate on re-designing SJI. Other losses, including the soft Dice loss, the soft Tversky loss, and the focal Tversky loss, are addressed in our subsequent work [66].
2.3 Jaccard Metric Losses
We can rewrite the intersection |o| and the union |u] as a function of the set difference |m]:
(|>] + lvv (mm)) and |u| = |u] + |m/.
(4)
Note that |m|= = || yll.. Combining these yields:
|o| (x, v) _ (1|= + 1|l1 - || vlli),
(5)
|u| (I, 3) + || 8lli = マ (1|= + vll1 + || vlli),
(6)
where the equalities hold when v, E {0, 1}p..
After eliminating erroneous combinations that have the same issue as SJL, we are left with two candidates AIML.; AIML2 1], [0, 11"d + [0, that are defined as:
|| 2/11
| v/11
AMIL = 1
[ 1/11 + 2/ v/1
(ar y)
A = 1
(8)
(x, 3) + ||x -
a well-known result that Aps a metric on {0, 1}2 (30) In Theorem 2.1 (see Appendix F for
the proof), we show that both AMMI, and AMML, are also metrics on [0, 1j2 Therefore, we call them Jaccard Metrie Losses (JMLs).
3

---

10
10
M.
04-
0.-
value
06-
94.
0.
0.2
04
06
0.
1.0
0.
02
04
06
0.8
1.0
Prodiction
Prodiction
Figure 1: Loss value vs. prediction with y = (left) and y = 0.9 (right).
Theorem 2.1. Both AMLLI and AIML2 are metrics on [0, 1P.. Neither ASH, nor Asn L2 is a metric on [0, 1|P.
Recall the definition of metric:
Definition 2.2 (Metric 14).. A mapping f M x M + is called a metric on M if for all a, b.b.e c E M, it satisfies the following conditions: () (reflexivity). f(a. (ii) (positivity). # b = f(a.b) > O. (iii) (symmetry). f(a, b) = f(b, a). (iv) (triangle inequality). f(a, e) f(a,b) c(b
Having a loss function A that is metric carries numerous benefits. Reflexivity and positivity collectively imply that Vr,y y E [0, 1]2, + A = O, meaning that A would be compatible with soft labels. The triangle inequality also provides insightful guidance. Applied KD, yields:
A L) S A(S,T) + A(T,
Here, SS represents the student model, T the teacher model, and L the ground-truth labels. the KD, we initially train the teacher model using the ground-truth labels, and then minimize the loss between the teacher and the student model. Equation (9) suggests that if we adhere to this process-equivalent minimizing the right-hand side of the equation- also minimize the upper bound on the student model's 's loss with respect to the ground-trun labels.
Furthermore, Theorem 2.3 shows (see Appendix G for the proof), when only hard labels involved, AIML, AMLL2 and ASHL, are identical. Hence, we can safely replace the existing implementation of SJL with JMLs. When soft labels are introduced, AIML, and XJML2 might yield different values. We discuss their distinctions Appendix H From both theoretical and empirical perspectives, we find that 1MML, is slightly more favorable than AML As a default in our experiments, we use AIMI,
Theorem 2.3. Vr E [0, 1]", E 10. and (0,1)", y 10, A AIMLZ AsHI Br, E [0, 1P" AMLI # A & ASH.
Figure plots the loss value of AMML and the cross-entropy loss (CE) for soft labels y = 0.1 (left) and y 0.9 (right). Typically, the model is randomly initialized, resulting in low initial confidence. Although the gradient CE substantial at the beginning, it plateaus as training progresses and the prediction close to the target. Conversely, AIML, offers effective supervision later the process. However, when y extremely small (c.g.. y 0.1), AIMLI can become overly steep, potentially causing problems in KD. This issue will be explored in greater detail in Section 2.4.2
2.4
Use Cases
Soft labels find wide-ranging applications. In this paper, investigate three of the most common use cases: label smoothing (LS), knowledge distillation (KD) and muni-sspervieed learning (SSL). We leave the discussion SSL. in Appendix
2.4.1
Label Smoothing
In LS 58 the one-hot encoding is combined with a uniform distribution under a smoothing coef- ficient resulting in soft labels SL* In JML-LS, network H is trained with the following loss:

---

LIMLIS ACELCE, + AIML IIMI such that LC = SE(H and LIML, JML(H,SL')
(10)
LS provides a regularization effect during training. However, when we regard semantic segmentation as pixel-wise classification task and apply LS to every pixel in the image without distinction, we fail to consider spatial differences. This particularly evident with SegFormer-B3 on ADE20K, which achieves overall pixel-wise accuracy of region. These areas, due to their inherent ambiguity, require stronger regularization. Moreover, our empirical findings suggest that smoothing non-boundary regions only yields marginal improvements. Therefore, we introduce boundary label smoothing (BLS), which only applies smoothing to labels near the boundary. We refer to the resulting loss JML-BLS. Specifically, for every pixel examine its le x k neighborhood N, and the pixel is regarded as boundary pixel if there exists pixel j E N, such that their ground ground-truth labels are different: Yi * Yリ- This can be efficiently computed by applying a max pooling layer to the one-hot encoding.
but drops to 47.98 in the boundary
2.4.2 Knowledge Distillation
In KD [22] besides the groumd-truth label L, student model S is trained to minimize the discrepancy to teacher network simultaneously:
LCE, KD = MLLCEE PTLCE,
(11)
where L = CE(S, L) and LCET CE(S,T). In JML.KDD, add additional supervision from JMI.:
LIML, KD = CILL, + VILMM
(12)
where LIMI, JML(S, L) and LIMI, = JML(S,T).
Combining these, we have
LIML-DD ACELCE, + AIMLCIMI KD-
(13)
Existing segmentation KD methods rely on pixel-wise supervision either in the output [36 70 or in the feature space 664 (55). Our LJML, however, enables direct distillation of the teacher's IoU information to the student which aligns with the final evaluation metric.
Moreover, recall that JMLs are averaged for each class, making it crucial determine which classes contribute to the loss value at each iteration. We refer to these classes as active classes. In KD, the student uses information from non-target classes learn the class relationship from the teacher [59 777. However, a teacher network may also include noisy and unhelpful predictions for unimportant classes. Therefore, propose filter out unimportant classes based the confidence level of the teacher network. Specifically, JMLs are computed only over classes where the teacher's 's confidence exceeds predefined threshold. This approach has two benefits: first, the student will not be distracted by irrelevant classes, aiding in generalization. Second, as shown in Figure [ we find that the loss value of JMLs can become excessively steep around the target when the ground-trutte low (e.g. 0.1). Therefore, ignoring these classes can stabilize the training process.
We have also observed that the student benefits from teacher trained with JML-BLS. In contrast, in classification tasks, it has been found that teacher trained with LS can detrimentally affect the student's performance [44). This is often cited as counterexample the notion that a more accurate teacher will necessarily distill better student. We believe that the teacher, trained through JML-BLS, acquires intricate boundary information, and this "dark knowledge" is implicitly passed the student, resulting in more accurate student.
3 Experiments
Experimental setups. We adopt Pytorch Image Models (timm) [7], which provides implemen- tations and ImageNet [13 pre-trained weights for various backbones. In our experiments, CNN backbones include 1es/et-101/50/18 [2I] and MobileNerV2 (54; transformer backbones contain
MiT-B5/B3/B 1/B0 66]. Segmentation methods consist of DeepLabV3+ [7]. DeepLab (6), PSPNet
5

---

78 UNet [5T and SegFormer [69. We evaluate models on Cityscapes [TI. PASCAL VOC [18] ADE20K 8T and DeepGlobe Land [2] In summary, 13 models and 4 datasets are studied in the sequel. More details of each model are in Appendix A and training recipes are in Appendix B
Evaluation metrics. To provide comprehensive comparison, accuracy (Acc) and mean intersection over union (mloU). To evaluate the effects of our methods on calibration, we present the expected calibration error (ECE) [20] and the ECE computed only over the boundary region, denoted as BECE. For Cityscapes, PASCAL VOC, and ADE20K, we repeat the experiments times (except for SSL experiments that are single runs) and report performance on the validation set. For DeepGlobe Land, we conduct f-fold cross-validation. All results are presented in the format of standard deviation. We do not apply any test-time augmentation.
report both overall pixel-wise
3.1 Results on Accuracy
We report results on Cityscapes (Table D. PASCAL VOC (Table 2 DeepGlobe Land (Table 13 Appendix D.. Key takeaways from our results are as follows:
ADE20K (Table
3). and
JML significantly improves accuracy (Acc and mloU). Compared to training with CE alone, incorporating JML as part of the loss function can significantly enhance a model's mloU. The improvement is typically more than 2% Cityscapes, PASCAL VOC, and ADE20K. Additionally, it can also increase a model's Ace. This suggests that the benefits of JML are not solely due to its alignment with the evaluation metric (mloU), but also because it aids the overall optimization process.
JML benefits more from soft labels. For instance, the improvements of mloU for DL3-RI01 on PASCAL VOC are 0.48% (CE vs. CE-BLS) and 1.25% (JML vs. JML-BLS), respectively. We adopted training procedures that are heavily optimized for CE, which might explain why JML requires stronger regularization. More qualitative results can be found in Figure 7 and Figure 8 (Appendix D
KD is more effective than BLS, while BLS performs well on datasets with simple boundary condition. Both BLS and KD consistently improve the model's 's accuracy. Despite its simplicity, BLS can yield significant improvements, especially on PASCAL VOC where boundary condition less complex. As boundary condition becomes more intricate, as in Cityscapes and ADE20K, KD, seen as learned label smoothing 76), typically outperforms BLS.
SOTA results on segmentation KD Without bells and whistles, our simple approach that only uses soft labels greatly exceeds SOTA segmentation KD methods, as shown in Table Indeed, the model trained only with hard labels in JML already achieves a comparable result as some of these methods. For example, on PASCAL VOC, DL3-R18 18 achieves 74.42 = 0.52% mloU while CIRKD [70). a complex distillation method only attains 74.50% mloU. Note that our baseline is not stronger than theirs: our DI.3-R88 trained with CE has 72.47 + 0.33% mIoU while their number is 73.21% (because we use a smaller batch size, see Appendix B)
SOTA results segmentation SSL. By incorporating JML-BLS with AugSeg
80
(refer to
Appendix
[ for further details), we achieve SOTA segmentation SSL results as illustrated in Table []] (Appendix J. Notably, improvements are particularly significant on smaller splits- exceeding a 4% enhancement over AugSeg on the 92 split.
3.2 Results on Calibration
CE-BLS and CE-KD sometimes hurt calibration. Contrary to the common belief in classification we find that both CE-BLS and CE-KD can sometimes deteriorate model calibration. For instance, in Cityscapes experiments, the lowest ECE is usually obtained with CE only. Nevertheless, CE-BLS can still improve model calibration near the boundary.
44
42 32
JML compromises calibration. Although the soft Dice loss can significantly improve a model's segmentation performance, it is well known to yield poorly calibrated models 3] We confirm that this is also the case for JML. Interestingly, while models trained with JML exhibit inferior top-class calibration measured by ECE, they actually achieve better multi-class calibration indicated by the static calibration error (SCE). We provide more detailed results on calibration in Appendix K
JML-BLS and JML-KD consistently improve calibration. Although JML presents challenges in model calibration, these can be greatly mitigated by training models with soft labels. In JMI. experiments, both JML-BLA BLS and JML-KD reliably enhance model calibration.

---

Table Results on Cityscapes (%). Best results within CE and JMI. groups are highlighted in red and green, respectively. Best results across CE and JML groups are underscored.
Model
Metri
CL:BLS
CL
JML
JML
KD
Ace
96.10 0.02
96.11 0.03
96. 10 0.04
95.25
0.03
m +
78.67 + 0.32
78.70 + 0.26
89.29 0.17
50.68 + 0.24
ECE
0.00
-
2.09 0.00
BECE
16.14:20.22
11.59 11.39:00.12
30.78 0.19
21.20 0.07
Ace
85.77 = 0.08
96.53 4: 0.08
0.00
95.93 1:33::000 0.04
96.05 = 0.02
0.82
ECE
0.62
0.02
1.00 0.03
2.84 0.02
2.18 0.01
BECE
16.23
1: 0.16
11.67 4 0.18
31.12 0.10
20.51 4: 0.12
9,046
DL3-KIS
mlol
72.88
+: 0.44
73.14
0.12
74.09 + 0.28
75.53 0.13
76.26
0.17
76.08
0.33
ECE
0.68
0.06
0.88 = 0.01
0.98 0.03
3.07 = 0.01
2.49 + 0.05
2.47 + 0.01
0.19
0.00
Ace
95.24
0.01
95.28 + 6.03
96.30 0.04
96.45 0.00
95.51 0.04
95.56 0.03
DL3-MB2
mloU1 T
72.19
1..9::..1 0.12
72:54 4: 0.26
72.38 0.15
75.32
75.61 0.14
75.94 0.21
0.09
0.99
2.43
BECE
17.64 = 0.36
13.14 0.20
16.53 0.10
32.08 0.29
21.61 = 0.09
22.57 = 0.04
Acc
0.03
4
15..5.002
95.90
mlaU
72.61
0.34
72.43
0.05
72.79 0.15
74.96
75.31 0.13
75.76 0.31
BCE L
0.68
0.07
0.97 + 0.01
1.05
0.05
3.17
0.01
2.44
0.02
2.52 0.01
BECE.
17.54 + 0.24
13.61 0.20
16.96
+ 0.21
32.65 + 0.07
22.30 0.15
23.17 0.08
Table 2: Results PASCAL VOC (%). Best results within CE and JML groups highlighted in red and green, respectively. Best results across CE and JML groups are underscored.
CE-BLS
JML
JLC
RD
Ace
94.68
= 0.02
94.76
2 0.05
94.97 2 0.13
95.34 = 0.08
DL3 D13-
mloU
ECE
78.39 1: 0.09
0.00
78.87 0.44
80.28
0.10
0.45
81.52 1: 0.
3.20
BECE
20.54 4: 0.12
17.25
0.14
32.70 0.09
22.30 0.02
Ace
04.23 1 0.05
91.28
0.00
04.80:00.00
94.81 0.002
76.93
77.23
0.18
78.97
79.78
BCE L
2.05 0.03
1.77 0.08
4.19 0.02
3.25 2255210 0.02
BECE.
20.95 - 0.13
17:30 0
33.07 0.10
20.41 0.01
Ace
92.06
03.17 0.00
93.13 0.07
93.28 0.14
93.72 0.05
DL3-IS
mlaU1
72.47 0:33
72.99 00099:1000 0.42
72.96 1 0.30
74. 0.52
75.60 - 0.24
73.89
0.29
ECE
+
1.68 0.05
+
3.79 0.03
BECE
21.83
0.29
0.08
19.42 0.04
34.35 0.09
22.45 0.08
22.15
0.19
Apc
92.47
=: 0.32
92.45 0.02
92.34 0.04
92.74
93.13 =: 0.11
94.21 = 0.01
DI3-MR2
0.27
ECE
1.02 0.114
1.46 0.00
1.76 = 0.03
5.19 0.05
4.14 0.08
4.24 0.01
BECE.]
22.66
0.34
19.62 0122:::000 0.45
20.62 2 0.34
3..10
23.38 - 0.19
23.00 = 0.11
6.09
08.20
PSP-RIB
mlaU1
72.10
0.55
72.76 0.52
72.84 0.57
74.20
0.24
75.04 0.22
74.93
0.36
ECB
1.92
0.22
1.40 +: 0.18
1.81 0.16
4.76 2: 0.04
3.70 0.01
3.99 + 0.07
BECK
Table 3: Results ADE20K (%). Best results within CE and JML groups are highlighted in red and green, respectively. Best results across CE and JML groups are underscored.
CF-KD
JML-KD
Acc
62.48
0.00
62.33
0.06
82.12
0.11
52 50 1 10
SegPormer-85
mleU
0.35
48.50
0.26
49.95
0.28
50.22
0.32
5.94
5.14 = 0.15
11.84
0.06
9.12 0.15
BECE L
19.98 2: 0.11
14.59 4 0.17
34.78 1 0.19
19.88 1: 0.28
Segforrer D3
mloU
46.51
0.29
46.88 1 0.24
48.68 0.27
49.24
0.24
BCE
5.54 +: 0.12
4.56
0.13
11.57 + 0.13
10.63
+: 0.22
4.06
0.13
Acc
78.76 2: 0.05
78.92 4: 0.11
78.96 0.03
79.41
0.10
79.53
4: 0.12
79.62 =: 0.0000 0.04
SegFormer-B1
minU T
38.79 -:
38.95 + 0.25
39.29 + 0.16
41.70 + 0.17
42.03
- 0.29
42.62 = 0.17
0.004
BECE 1
18.70 0.00 0.09
13.89 +: 0.20
19.23
0.09
33,25 4 0.05
27.38
0.25
26.29 0.33
Acc
75.49
0.06
76.37 1 0.02
76.48 0.02
76.85 1 0.04
76.98
0.11
76.99
0.06
0.11
SegPormer-H0
ECE
2.61
0.12
2.10
0.20
3.56
0.04
10.77
0.12
9.88
0.17
10.22
0.01
BECE
17.01 0.17
13.33 + 0.12
17.53
0.13
31,84
0.28
26.09 + 0.29
24.99
0.14
Table 4: Comparing with SOTA segmentation KD methods on Cityscapes and PASCAL VOC. All
results are mloU (%). JML-KD: we increase the batch size 16 for Cityscapes experiments to
match training details in 70 23
24
Datneet
Model
CF
SKD 56
IFVD (54
CD) 83
CIRKD 708
MisKD) 24
DIST 23
IMI-KD
DLS-RI
72.85
15.59
73.55
70.38
77.00
77.10
77.91
0.1G
73.50
75.26
77.53
0.20
PSP-R18
72.61
73.29
75.71
74.36
74.73
75.34
76.31
77.33
+ 0.38
DE3-R18
72.47
73.51
73.85
74.00
74.50
75.59
1 0.29
PS
73.54
74.78

---

Ablation Studies
4.1 JML Weights
It is common in recent works [ 8 29 to balance CE and the Dice loss with equal weights. For JML, we adopt 0.25/0.75 in all our experiments and find slightly superior than 0.5/0.5.
Table 5: Ablating different values of ACE AMM. on PASCAL VOC using DL3-R101 and DL3-R18. All results are mloU (%). Red: the best in a row. Green: the worst in row.
Model
0250075
0,500050
0,000100
10000000
113--101
79.80 = 0.40
0.45
79.92 + 0.12
78.73 0.30
78.38 0.09
DL3-R18
73.67 + 0.39
74.42 1 0.52
74.30 4 0.29
73.21 0.2.
12.47 1 0.33
4.2 JML-BLS
Although BLS is sensitive to the choice of c, it effectively increases model accuracy and calibra- tion. The effect of the smoothing coefficient on PASCAL VOC with 113-R101/50/18 is shown in Figure (5 (Appendix D. Interestingly, for DL3-R50 and DL3-R18, the optimal that achieves the highest mloU also yields the lowest ECE.
We need strong smoothing coefficient near the boundary. The optimal for different kernel size on PASCAL VOC with DL3-R18 shown Figure (Appendix Note that do implies LS is applied to every pixel, i.e. vanilla LSS. Generally, as l increases, we need decrease the strength of smoothing. The best result is obtained when we only smooth a small region near the boundary with a strong smoothing coefficient (k = 3 and 0.50).
4.3 JML-KD
Loss terms. We examine the contribution of each loss in Table @using DL3-R18 student. Adding JML terms significantly improves the student's performance.
Table 6: Evaluating different losses terms Cityscapes and PASCAL VOC using DL3-RI8 student. LIML-BLS means we train the teacher with JML-BLS. All results are mloU (%). Red: the best in a column.
DILL
LEE.DD
SD
72.88 2: 0.44
72.47
0.33
75.55 = 0.13
74.42
0.52
76.16
70.05
0.31
76.68 = 0.33
75.89
0.29
Filtering out unimportant classes based on teacher's confidence is useful. We examine the impact of active classes on PASCAL VOC with DL3-R 18 in Table In particular, we propose to ignore classes where the soft label, i.e. teacher's confidence, is low (marked as LABEL). And in ALL, we include all classes. PRESENT, we select the class with the maximum confidence from the teacher's prediction in the class dimension. In PROB, skip classes where the student itself is not confident. In BOTH, we take both the teacher's and student's 's confidences into account. The code to compute active classes is in Figure ( (Appendix D
We observe that using ALL classes JML-KD can misguide the student. most of the teacher's output classes, the confidence is usually very low. It can be challenging and potentially detrimental for the student to precisely mimic these numbers. With PRESENT, the performance remains similar to that without soft labels. This is because the effectiveness of using soft labels come from the non-argmax classes. PROB and BOTH achieve similar performance, but both are worse than LABEL.
Table 7: Comparing active classes on PASCAL VOC using DL3-R18. All results are mloU (%).
Active Classes
ALL
FRBSETT
FSUB
BOTH
LABEL (Oun)
0.33
D.53
76.30
0.299
Teacher's calibration is beneficial. Sec. 2.4.2 we suggest that teacher trained with JMI-BISS offers more boundary information to the student. But what exactly is this boundary information? We

---

believe it relates to the teacher's calibration. The student, with less capacity, often struggles to mimic more powerful teacher, especially near ambiguous boundary regions. However, if the teacher more calibrated and outputs a less peaked distribution, the student will learn this uncertainty rather than attempting to match an unrealistic distribution that lacks the capacity to replicate.
In T we compare three teachers with various accuracy and calibration on PASCAL VOC. In particular, both T2 and T3 are trained with JML-BLS but with different smoothing parameter e, while TI is not. T2, although having lower mloU, is more calibrated than TI. Consequently, T2's student more accurate and better calibrated than Tl's.
Table 8: Comparing teachers of different accuracy and calibration on PASCAL VOC (%). Prefix stands for the teacher and prefix S the student. Red: the best in row.
TECE
4.20
3.47
3:24
T BBCFF 1
33.17
20.73
20.2 0.07
79.09
79.80
SECE
4.76
4.01
+ 0.07
8 BBCE J
33.86 2: 0.10
22.21 - 0.06
22.15 4 0.05
S mloU1
75.05
0.005:00..3 0.31
75,43 4 0.35
75.89 4: 0.29
Related Works
IoU is a commonly employed metric in semantic segmentation, and IoU losses strive to optimize this metric directly. IoU only obtains values when both predictions and ground-truth labels are discrete binary vectors (0. 1)2, but the neural network often predicts soft probabilities (after the softmax or sigmoid layer) in [0, 1j.. Interpolating IoU values from (0, 1}2 to [0, 1]" has primarily followed two
routes. One involves relaxing set counting as norm functions, as seen in the soft Jaccard loss [46 49)
the soft Dice loss [57], the soft Tversky loss (53) and the focal Tversky loss []. The other capitalizes on the fact that IoU is submodular 72). allowing for the application of the convex Lovasz extension of submodular functions. For instance, the -offza- loss 22, the Lovasz hinge loss [72] and the PixIoU loss 773) Nevertheless, as IoU values extended for predictions from {0, 1)" to [0, 1]", the fact that labels can also fall within [0, 1|. is often overlooked. As a result, these IoU losses are incompatible with soft labels.
Besides semantic segmentation, IoU is adopted across wide spectrum of fields. Due its discrete nature, its probabilistic extensions [56] have found use in object detection [19], medical imaging [16]
and information retrieval [26 43
Discussion
6.1 How to tune the hyper-parameters of JML?
In this section, we delve into some critical hyper-parameters JML. For a comprehensive list, please refer to the accompanying code.
moouu//mo' (default: 1.0/0.0/0.0): the weight of the loss to optimize
[65]. The appropriate choice is mainly dependent on the targeted evaluation metrics. Given that the prevailing metric is mloU" (per-dataset mloU), we recommend set mloU" 1.0 and mloU' / to 0.0. However, it is imperative to acknowledge the inherent trade-offs when using JML to optimize different metrics (65
alpha/beta (default: 1.0/1.0): the coefficient of false positives/neeaatves in the Tversky loss. For instance, when alpha and beta are both set to 1.0, the configuration corresponds to JML. Conversely, an alpha and beta value of 0.5 each leads to DML (66). When the evaluation metric is IoU, JML is advised, while it is the Dice score, DML is more appropriate. The general Tversky loss is useful when separate weights are required for false positivess/eegaties.
active classes_ mode (default: PRESENT for hard labels and ALL for soft labels): the mode to compute active classes. With hard labels, is suggested use PRESENT, since the loss aligns more effectively with the evaluation metric, especially when (i) the dataset contains large number of classes (e.g. ADE20K), and/or (ii) evaluated with fnn-grained mloUs (65) With soft labels, the optimal choice varies based specific applications (see Table 7).

---

6.2 How to use JML?
Combine JML with CE. Our findings suggest that incorporating CE, particularly during the initial stage of training, can expedite convergence (as illustrated in Figure I Consequently, relying solely on JML is not recommended. For a discussion on the balancing weights, please consult Section 4.1
Tune the papar-parameter of JML. We have endeavored to minimize the necessity of extensive tuning. the majority of scenarios, default nyper-parameter values (as illustrated above) should suffice. However, some situations might benefit from further refinement. In particular, pay attention _edee mode when dealing with soft labels.
to
of training settings. Current training recipes have been heavily optimized for CE. Although we find JML generally aligns with these hyper-parameters, due diligence is required. Notably, one of the advantages of JML is its ability to speed up convergence in the later training phase (as depicted in Figure m. As consequence, training with combination of CE and JML often converges in considerably fewer epochs compared to training with CE alone. Please refer to [65 for a comparison of the number of epochs with those in MMSegmentation.
Tune the
Be careful with distributed training. In our concurrent work 65 we observe that the loss presents trade-offss when optimizing different mloU variants. If the per-GPU batch size is small (which is often the case) and the loss is computed on each GPU independently, the loss may inadvertently optimize for mloU (per-image mloU). This can lead to suboptimal outcomes when evaluated with mloUD (per-dataset mloU).
Be careful with extremely class-imbalance. Aligned with mloU, JML is designed compute the loss on a per-class basis and subsequently average these individual losses. In preliminary experiments on some medical datasets, identified instances of severe class imbalances. In such scenarios, the class-wise loss may inadvertently amplify the significance of underrepresented classes, potentially disturb the training process. To address this, one might consider (i) oversampling the minority classes, and/or (ii) adopting a class-agnostic loss computation, where the intersection and union are calculated over all pixels.
Limitation
In both this study and our subsequent work [66]. the focus is to extend the losses in the field of image segmentation. would be intriguing to apply these losses to other tasks, such as long-tailed classification [34]. Moreover, although we adopt these losses in the label space, they present potential quantifying the similarity between two feature vectors [24], potentially serving as an alternative to the L norm or cosine similarity.
8 Conclusion
This paper is driven by the observation that current IoU losses fall short when dealing with soft labels, which substantially limits their adaptability to crucial training techniques. To address this limitation, we introduce the Jaccard metric losses (JMLs). While these losses are identical to the soft Jaccard loss in conventional hard-label setting, they offer full compatibility with soft labels.
Our results demonstrate that integrating JMLs with label smoothing, knowledge distillation, and semi-supervised learning leads to notable improvements in both accuracy and calibration. This is consistent across a spectrum of datasets and network architectures, encompassing classic CNNs as well as recent vision transformers, Remarkably, the proposed methods, which are simple and solely rely on soft labels, surpass state-of-the art segmentation knowledge distillation and semi-supervised learning techniques by a significant margin.
In our follow-up study (66), we delve into the extensions of various other losses, including the soft Dice loss, soft Tversky loss, and focal Tversky loss. Given their equivalence to their original counterparts in standard hard-label context and their enhanced compatibility with soft labels, we recommend to replace the existing implementations with ours.
10

---

Acknowledgements
We acknowledge support from the Research Foundation Flanders (FWO) through project numbers G0A1319N and S00142IN, and funding from the Flemish Government under the Onderzoekspro- gramma Artificiële Intelligentie (AI) Vlaanderen programme. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation Flanders (FWO) and the Flemish Government.
We thank Iya Bogdanov for his help on the proof.
References
[1] Nabila Abraham and Naimul Mefraz Khan. A Novel Focal Tversky Loss Function with Improved Attention U-Net for Lesion Segmentation. ISBI, 2019.
[2] Maxim Berman, Amal Rannen Triki, and Matthew B. Blaschko. The Lovasz-softmax loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Uinn Union Measure in Neural Networks. CVPR, 2018.
[3] Jeroen Bertels, David Robben, Dirk Vandermeulen, and Paul Suetens. Theoretical analysis and experimental validation of volume bias of soft Dice optimized segmentation maps in the context of inherent uncertainty. MIA, 2021.
[4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. MixMatch: A Holistic Approach to cemi-Supervised Learning. NeurIPS, 2019.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Detection with Transformers. ECCV, 2020.
[6] Liang-Chiee Chigh Chen, George Papandreou, Florian Schroff, and Hartwig Adam, Rethinking Atrous convolution for semantic image segmentation. arXiv, 2017.
[7] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with Atrous separable convolution for semantic image segmentation. ECCV, 2018.
[8] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention Mask Transformer for Universal Image Segmentation. CVPR, 2022.
[9] Bowen Cheng, Alexander G Schwing, and Alexander Kirillov. Per-Pixel Classification Not All You Need for Semantic Segmentation. NeurlPS, 2021.
[10] MMSegmentation Contributors. MMSegmentation: OpenMMLab Semantic Segmentation
Toolbox and Benchmark, 2020.
[11] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR, 2016.
[12] like Demir, Krzysztof Koperski, David I Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar. DeepGlobe 2018: A Challenge Parse the Earth through Satellite Images. CVPR Workshop, 2018.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Larle-Scale Hierarchical Image Database. CVPR, 2009.
[14] Michel Marie Deza and Elena Deza. Encyclopedia of Distances. Springer, 2009.
[15] Zhipeng Ding, Xu Han, Peirong Liu, and Mare Niethammer. Local Temperature Scaling for Probability Calibration. ICCV, 2021.
[16] Reuben Dorent, Thomas Booth, Wenqi Li, Carole H. Sudre, Sina Kafiabadi, Jorge Cardoso, Sebastien Ourselin, and Tom Vercauteren. Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasets. MIA, 2020.

---

[17] Tom Eelbode, Jeroen Bertels, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Biss- chops, and Matthew B. Blaschko. Optimization for medical image segmentation: Theory and Practice When Evaluating With Dice Score or Jaccard Index. TMI, 2020.
[18] Mark Everingham, Lue Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. IJCV, 2009.
[19] Di Feng. Zining Wang. Yiyang Zhou, Lars Rosenbaum, Fabian Timm, Klaus Dietmayer, Masayoshi Tomizuka, and Wei Zhan. Labels are Not Perfect: Inferring Spatial Uncertainty in Object Detection. TITS, 2022.
[20] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML, 2017.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[22] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NeurIPS Workshop, 2015.
[23] Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge Distillation from A Stronger Teacher. NeurlPS, 2022.
[24] Tao Huang, Yuan Zhang, Shan You, Fei Wang, Chen Qian, Jian Cao, and Chang Xu. Masked Distillation with Receptive Tokens. ICLR, 2023.
[25] Pavel lakubovskii. Segmentation models pytorch, 2019.
[26] Sergey Ioffe. Improved Consistent Sampling, Weighted Minhash and L1 Sketching. ICDM, 2010.
[27] Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Hein. nnU-Net: a self-configuring method for deep learning- based biomedical image segmentation. Nature Methods, 2021.
[28] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar. Panoptic Segmentation. CVPR, 2019.
[29] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment Anything. ICCV, 2023.
[30] Sven Kosub. A note on the triangle inequality for the Jaccard distance. PRL, 2019.
[31] Aviral Kumar. Sunita Sarawagi, and Ujjwal Jain. Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings. ICML, 2018
[32] Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang, and Xizhou Zhu. AutoLoss Searching Loss Functions from Scratch for Generic Tasks. CVPR, 2022.
[33] Hao Li, Chenxin Tao, Xizhou Zhu, Xiaogang Wang. Gao Huang, and Jifeng Dai. Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentation. ICLR, 2021.
[34] Xiaoya Li, Xiaofei Sun, Yuxian Meng. Junjun Liang, Fei Wu, and Jiwei Li. Dice Loss for -ata-imbalanced NLP Tasks. ACL, 2020.
[35] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection. TPAMI, 2018
[36] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured Knowledge Distillation for Semantic Segmentation. CVPR, 2019.
[37] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. CVPR, 2022.
12

---

[38] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. ICLR, 2019.
[39] Xingchen Ma and Matthew B Blaschko. Meta-Cal: Well-controlled Post-hoc Calibration by Ranking. ICML, 2021.
[40] Lena Maier-Hein et al. Metrics Reloaded: Recommendations for image analysis validation. arXiv, 2023.
[41] Alireza Mehrtash, William M. Wells, Clare M. Tempany, Purang Abolmaesumi, and Tina Kapur. Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation. TMI, 2020.
[42] Hossein Mobahi, Mehrdad Farajtahar, and Peter I. Bartlett. Self-Distillation Amplifies Regular- ization in Hilbert Space. NeurIPS, 2020.
[43] Ryan Moulton and Yunjiang Jiang. Maximally Consistent Sampling and the Jaccard Index of Probability Distributions. ICDM Workshop, 2018.
[44] Rafael Müller, Simon Korbllihh, and Geoffrey Hinton. When Does Label Smoothing Help? NeurlPS, 2019.
[45] Jeremy Nixon, Mike Dusenberry. Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. Mcasuring Calibration in Deep Learning. CVPR Workshop. 2019.
[46] Sebastian Nowozin. Optimal Decisions from Probabilistic Models: the Intersection-over-UU Union Case. CVPR, 2014.
[47] Teodora Popordanoska, Jeroen Bertels, Dirk Vandermeulen, Frederik Maes, and Matthew B. Blaschko. On the relationship between calibrated predictors and unbiased volume estimation. MICCAI, 2021.
[48] Teodora Popordanoska, Raphael Sayer, and Matthew B Blaschko. A Consistent and Differen- tiable Lp Canonical Calibration Error Estimator. NeurIPS, 2022.
[49] Md Atiqur Rahman and Yang Wang. Optimizing intersection-ovvr-miion in deep neural networks for image segmentation. ISVC, 2016.
[50] Alexander Rakhlin, Alex Davydow, and Sergey Nikolenko. Land Cover Classification from Satellite Imagery With U-Net and Lovász-Softmax Loss. CVPR Workshop, 2018.
[51] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI, 2015.
[52] Axel-Jan Rousseau, Thijs Becker, Jeroen Bertels, Matthew B. Blaschko, and Dirk Valkenborg. Post training uncertainty calibration of deep networks for medical image segmentation. ISBI, 2021.
[53] Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and Ali Gholipour. Tversky loss function for image segmentation using 3D fully convolutional deep networks. MICCAI Workshop. 2017.
[54] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chieh Chen. MobileNerV2: Inverted Residuals and Linear Bottlenecks. CVPR, 2018.
[55] Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise Knowledge Distillation for Dense Prediction. ICCV, 2021.
[56] H. Späth. The minisum location problem for the Jaccard metric. OR Spektrum, 1981.
[57] Carole H Sudre, Wenqi Li, Tom Vercauteren, Séhastien Ourselin, and M Jorge Cardoso. Gen- eralised Dice overlap as a deep learning loss function for highly unbalanced segmentations. MICCA/ Workshop. 2017.
[58] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zhigniew Wojna. Rethinking the Inception Architecture for Computer Vision. CVPR, 2016.
13

---

[59] Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin. Anima Singh, Ed H Chi, and Sagar Jain. Understanding and Improving Knowledge Distillation. arXiv, 2020.
[60] Vladimir N Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
[61] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers. CVPR, 2021.
[62] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, Xiaogang Wang. and Yu Qiao. Internlmage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions. CVPR, 2023.
[63] Yuchao Wang, Haochen Wang. Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi CVPR, 2022.
Le. Le.
Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels.
[64] Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class Feature Variation Distillation for Semantic Segmentation. ECCV, 2020.
[65] Zifu Wang, Maxim Berman, Amal Rannen- Triki, Philip H.S. Torr, Devis Tuia. Tinne Tuytelaars, Luc Van Gool, Jiaqian Yu, and Matthew B. Blaschko. Revisiting Evaluation Metries for Semantic Segmentation: Optimization and Evaluation of Fine-grained Intersection over Union. NeurlPS, 2023.
[66] Zifu Wang. Teodora Popordanoska, Jeroen Bertels, Robin Lemmens, and Matthew B. Blaschko. Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels. MICCAI, 2023.
[67] Ross Wightman. Pytorch Image Models, 2019.
[68] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders. CVPR, 2023.
[69] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- Former: Simple and Efficient Design for Semantic Segmentation with Transformers. NeurIPS, 2021.
[70] Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Cross- Image Relational Knowledge Distillation for Semantic Segmentation. CVPR, 2022.
[71] Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, and Yinghuan Shi. Revisiting Weak-to-Stronn Consistency in Semi-Supervised Semantic Segmentation. CVPR, 2023.
[72] Jiaqian Yu and Matthew B. Blaschko. The Lovász Hinge: Novel Convex Surrogate for Submodular Losses. TPAMI, 2018.
[73] Jiaqian Yu, Jingtao Xu, Yiwei Chen, Weiming Li, Qiang Wang, Byung In Yoo, and Jae-Joon Han. Learning generalized intersection over union for dense pixelwise prediction. ICML, 2021.
[74] Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation. CVPR, 2022.
[75] Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hatwig Adam, Alan Yuille, and Liang-Chich Chen. k-means Mask Transformer. ECCV, 2022.
[76] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting Knowledge Distillation via Label Smoothing Regularization. CVPR, 2020.
[77] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled Knowledge Distillation. CVPR, 2022.
[78] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid Scene Parsing Network. CVPR, 2017.
14

---

[79] Zhen Zhao, Sifan Long, Jimin Pi, Jingdong Wang. and Luping Zhou. Instance-specific and Model-adaptive Supervision for Semi-supervised Semantic Segmentation. CVPR, 2023.
[80] Zhen Zhao, Lihe Yang, Sifan Long, Jimin Pi, Luping Zhou, and Jingdong Wang. Augmentation Matters: A Simple-yet-Effecttiv Approach to Semi-supervised Semantic Segmentation. CVPR, 2023.
[81] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene Parsing Through ADE20K Dataset. CVPR, 2017.
[82] Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking Soft Labels for Knowledge Distillation: A Bias- Variinar Tradeoff Perspec- tive. ICLR, 2021.
15

---

A Architectures
Our study includes 13 models, comprising both classic CNNs and recent vision transformers. Details of each model are presented in Table FLOPs computations are based on an input size of 512 x 1024. Inference latency measurements are conducted with the same input size on NVIDIA A100. We estimate the training memory requirements using a ground-truth size of 8 x 19 x 512 x 1024 (batch_sizz) num_classes, H, W), also on NVIDIA A100.
Table 9; Details of each model.
Method
Backbome
4Parrams (M)
FL.PPs
Latency (mK)
Memory (GB)
DeepLabV3+
RosNectO1
62.55
528.52
20.57
42.05
DeepLab
ResNer101
87.10
26.20
DeepLabv3
RosNect0
17.48
Decplabb3
ResNerl
13.98
121.20
4.93
6.43
DeepLabVS
MobileNetV2
S3
3.79
31,68
5.89
16.74
PSPNet
ResNett8
21
109.50
4.82
UNer
18.09
UNer 51
ResNert
14.49
46.42
6.13
7.76
UNet 51
MobileNetV2
S4
4,05
22,63
8.15
11.16
20
43.32
SegFomer
26.15
SegFomer
40
McTRI
69
15.48
63.85
11.07
15.99
Remeer
M:TBO
6
5,01
45.45
8.98
13.45
B
Training Details
By default, we adopted the training details outlined in 70 23 24], except for the reduction of the batch size to 8. In particular, we utilize SGD with weight decay of 0.0005 and a momentum of 0.9. The initial learning rate 0.01, and decayed according to iterations is 40K for Cityscapes [1] and PASCAL VOC [8]. 10K for DeepGlobe Land [12]. The crop size 512 x 1024 for Cityscapes [T] 512 512 for PASCAL VOC [18] and DeepGlobe Land [12]. In Table 4 for fair comparison with the baselines, we increase the batch size of JML-KD to 16 Cityscapes experiments, and all other parameters retain at their default values.
total iters 00.9 The number of
For experiments with vision transformers, we adhered to the training recipes in [10]. Specifically, we use Adam W [8] with a weight decay of 0.01. We begin with an initial learning rate 0.00006, iter
The number of iterations is 40K and the crop size is 512 x 512 for ADE20K [8I. The batch size is 8.
and is decayed by (1 total
For semi-supervised learning, we follow training details in 80) In particular, we use SGD with a weight decay of 0.0001 and momentum of 0.9. The initial learning rate is 0.001, and is decayed according to (1 - lotal Reys
iter ,0.9 The number of epochs is 80 and the crop size is 512 x 512 for PASCAL
VOC [18]. The batch size 16.
For JML experiments, we set /L/PT/ http://////////////// hts/s///////// to 0.5 and ACE/ AMML to 0.25/0.75.
C More analysis of ASHL.
For v, y E 1)", let us rewrite Asn. for a particular pixel é:
J33li b
ASHLL
(14)
2i + V/
IzAle + a
such that
a
Ij +
Vj
ZzUli
(15)
j
b Σ マーリlー
(16)
Assume ab + and predictions independent from each other. Take the derivative with respect to Z;;:
d
5² (a + b)ys
(17)
dæ
(x;; + U/ 3. + a)
16

---

The numerator has two roots:
r1 -(a
(a + 6)2 + 4b
(18)
-(a+ b) + (a + 6)2 + 48
2
(19)
It is easy to see that S0S r2 S 1. Therefore we have
dA dæ;
y 2 12. otherwise.
(20)
That is, A will push 2; towards either if ys 72, or if y < r2, rather than making it close to ge- In practice, predictions are not independent from each other, hence the exact behavior of Asn, is a complex process, However, the above analysis still provides an insight of how ASIL will react soft labels.
D
The Lovasz-Softmax Loss
Based on the fact that IoU submodular [72), the Lovasz-Softmax loss (LSL) [2] computes the convex Lovasz extension of IoU. Specifically, for each prediction z; [0, mispredictions are computed as
1
T2
=
m,
(21)
otherwise.
The Lovasz extension be applied such that
LLsL :m E 10, (0, 10 Σ mmg.(m)
where (mm) = ( - ( n-11 and π is permutation ordering of m such that my; ... m
Loss functions that rely on the Lovasz extension, such as LSL 22, the Lovasz hinge loss 72 and
the PixIoU loss 73 cannot handle soft labels because they need to compute gi(m) and it requires
y E {0, 13P.
E JMLs vs. other IoU Losses
Table 10 we evaluate the performance AIMI, AIM.2 ASHLLI, A and TLst on Cityscapes and PASCAL VOC using DL3-R18.
As elaborated Appendix ASILL is equivalent to AIML. in the context of hard labels. Nevertheless, with soft labels, ASH., has a tendency to push predictions towards vertices, rather than optimizing for the ideal z y scenario. Our experiments on PASCAL VOC reveal that models trained with ASH.L' using soft labels outperformed by their hard-label counterparts. In fact, predictions for multiple classes might be simultaneously pushed towards 1, creating conflict and potentially destabilizing the training process.
Conversely, Asn,L is free from the aforementioned issue, as highlighted in Section 2.2 Nevertheless, it is found to be less effective than Agn. L.1 when employed with hard labels, possibly due to its flattened behavior near the minimum [17]. Our empirical results further substantiate that it is surpassed by AIML on Cityscapes and PASCAL VOC, for both hard and soft labels.
Lastly, ALSL cannot take soft labels input, given its reliance on the Lovasz extension. Our evaluations indicate that its performance is similar to that of AJML with hard labels.
17

---

Table 10: Comparing AMMII 1M.. AIIL ASIL. and Aistt. on Cityscapes and PASCAL VOC using DL3-R18. All results are mloU (%). Red: the best in a column.
Duuaset
Hard
BLS
APaLI
75.35 = 0.13
76.26 1 0.17
76.68 1: 0.33
75.35 = 0.13
76.31 + 0.23
76.45 4: 0.26
Ssn,
75.55
= 0.13
75.78
+ 0.34
76.83 0.43
A s.
75.28 0.26
75.51
1 0.38
78.87 + 0.52
ALsL
75.53
4.20 0.26
74.42
0.52
75.60 + 0.24
76.89 + 0.29
AjMLa
74.42
0.02
0.52
75.31 + 0.31
75.49 + 0.35
VOC
Ssn,
74.42
0.52
0.20 74.13
74.24 0.61
0.30 73.72
74.83 0.32
A1st.
74.45
F Proof of Theorem 2.1
Proof.
ii IIMI, is metric on [0, 1P. The proof was given in [56]. Here we provide sketch for the proof of the triangle inequality. Recall the definition of AIMLI
1 ||>||| + |llll || -lla
2| ull1
AILL
(22)
||>|| + |llll1 ||= vlla
| + |/yl| + || 8ll1
Given fixed point E [0, 1jP, the key is to define new function d':
0
=-==y=a
d' (Ix, v)
d(z,g)
otherwise;
(23)
and show that if is metric on [0, 1J", then d' is also metric [0, 1|2.. The proof can be found in [56].
Substitute the definition of the L' norm as d and let 0. Since the L' norm is metric on [0, 1]", we can conclude that AMMI, is also a metric on [0, 1|"
(ii) AMM.2 is a metric on [0, 1|" Conditions (i) (iii) are obvious. To show the triangle inequality, note that
(b, c) Σ bio;
(24)
Σ(bb at)a afcs))
(25)
$ "Eb 1b, -ail + Σ acc
(26)
= |a bll + (a, c).
(27)
Similarly,
(a, b) s bb cllı (a,
(28)
Hence
AIML, b) (b,
(29)
11a
b1l1
||b clla
+ 1b cl12 + (b,c)
(30)
|a
6|/2 + (a, b)
||a
b1l1
||b - c|l1
+
(31)
aa
b//1 + bbb
c|l2 + (a,c) c)
I|a - b11
|b c|11
(a, c)
||a - b111 + ||b - cll
(32)
|a b|l
+ ||b cll1 (a, c)
||a
c|l1
(33)
aa c|| + (a,c)
AM, c).
(34)
Note that if p 2 q > 0,r 2 0, we have p 2 d. The last inequality follows from
p4r
||a b/|1 + |b cll1 2 ||a cll
18

---

(iii) AsH,, is not metric on [0, 1|" For instance, if a = 0.5, Amn., (a. = 2/3 # 0.
(iv) Asn. r.2 is not a metric on [0. 1]". For instance, if a 0.8, 0.4, 0.2, does not satisfy the triangle inequality.
G Proof of Theorem 2.3
Proof.
Vo E [0,1]², y E {0, 10 and æ {0,1}", y [0, ASHL
AIMI
IM Due to symmetry, we only need to prove the first part. Note that
||=:|| = Σ
(m==0) + Σ (m==1)
(35)
I1lll1 = Σ V1 = Σ (w==1)
(36)
(o, y) = Σ silli = Σ (v:=1)0;
(37)
| v|l Σ 10 Vel Σ 11 (u==))af
Σ l (m=1)a + Σ
I (w==1)-
(38)
Combine these, we have
Σ, (v:=))
ASILL' AIML AML22 E. (0== E, (w=1)
(39)
(ii) Rx y E 1J, ASH & AMILI + AM,L For instance, if 0.8, y = 0.5, Asn. #
JIMLI + AIML22
H AMLII vs. AJML2
Definition H.1 (Convex Closure). The convex closure of set function f x {0, 13 [0, is
Cʃ: (0, 11" + |0, 1]
min
(40)
s.t.
0 1,
(41)
(42)
0 0.
(43)
The convex closure extends set function that is only defined at the vertices {0, 1)P the whole hypercube [0, 1 by linearly interpolating the values at these vertices. We plot the loss value of AIML,1, A and the convex closure of ASHL in Figure 2 when y 0.5. Note that I overlaps with the convex closure at [0, 0.5]. We can see that AIML.1 I AIML,2 and AIMLI is closer to the convex closure. Generally, we have
Theorem H.2. Vx, y [0, 1]", AMLI & A
Proof.
AIMLI & AIML2
(44)
| + ||y|] || - vll
(a, v)
2 0
(45)
|| + [Iall + || vll
(z, v)
2 v11
(1|=)) + I|>l|))[ı - alli
||x - ul? 2(xr y) || vll 2 0.
(46)
19

---

1,0
A
08-
Coowa Cloore
Los value
0.6
0.4
02-
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Prediction
Figure 2: Comparing 1I AIML,2 and the convex closure with y = 0.5.
We can switch the element of x and whenever x; S for some i. Doing this, each term in the last inequality remains the same. Let us denote the new variable as x', y/ such that x/ 2 yf for all i. Hence
(1|=|1= + || Ull - || ull 2(x,, u) 1|= - vll1
(47)
=(1x='| |12 + [|//|l)) - - - -||=' - - 2(x',(2222222 v/>'''/'''' - y''ll2
(48)
-(1l=' [| + [1=" -
(I|='11 11e/111)2 2/ -2(e', (1le') - -lb2/l1))
(49)
-2(1k=/11 -
(50)
-2 -2> (1) - -y() E (1 =D>x 20
(51)
Theorem H.3. Given a set function : {0. + [0.1] and two concave functions f.g
10, 112 + [0. 1J. If Va E {0, ((x) f(z) g(x) and Va E [0, f(x) S g(x), then
Vo E [0, 1. [f(x) - CI(x)) S [g(x) Clr))..
Proof. It suffices to show Vo [0, 1|",, f(x) 2 Clxx). Ye E 1J, we have Σ 0i2; such
that 2 0. Di = I and x4 E (0. 1). Therefore
f(x) = f(} f(_ a,a;))
(52)
1=1
E 1 aiT;
= =f((1- ap)
+ apIp)
(53)
Op
2 (1 dp)
Ogiti)
+ +.app (1p)
(54)
app
Up Ap-1 2p-2 ajii
(1 mp)
ap-1 xp-1) + (ppp
(55)
Opp
1 Op 0p-1
1 1
pp-
i= ait's aitis
2 (1 - ap -ap- -)
) + ap-15(0p-1) + a (x)ppp)
(56)
ap ap-1
2 2... 2 Σ (f1())
(57)
= Σ adl(xs)
(58)
=
2 Cl(x).
(59)
20

---

Figure 3: counterexample that both AIMLI and AIML are not piece-wise concave in 2D.
Although both AMLLI and JMLL2 are not concave as Figure 2 shows, we can divide the hypercube into several sub-spaces such that 1MML and IMLL2 are equal at the vertices and concave at each sub-space, then we can still apply Theorem H. However, the fact that both of them are piece- wise concave in 1D does not hold in higher dimension. Indeed, we can find a counterexample 2D. Let [0.6285, 0.7551.. Both x and x' are in the sub-space yab
y [0.5, 0.5], [0.4087, 0.7855), and
0.5 0.5 MLL.I (z, v) + MML (x', 3) > IMML + 0.5z', ,y),
(60)
0.5 J.5A v) + M.54
A1LL2 0.5z',
(61)
How do they perform empirically? In Table 10 compare AIMI, and A Cityscapes and PASCAL VOC using DL3-R18. We find I is slightly superior to AIML2.
I Semi-supervised Learning
We focus augmentation beaee semi-super rumi-supervieed learning approaches 80) Broadly speaking, in their approaches, the data consists of both supervised and unsupervised images, and a model H is trained to minimize the following loss:
LCE, SSL TSLCEE LcE.
(62)
such that LCE, CE(H(AA,I.) L(A(())) and LCE (z.)) T(A,,))
Here, A, and A., are data augmentations applied labeled images v, and unlabeled images Ta. respectively. L is the ground-truth label and 7' is usually the exponential moving averaging of the model weights.
In JML-SSL, we introduce additional supervision from JML:
LIML SSL = 05..MM. + CULMM
(63)
where LIMI, = (ML(H(A.(E.)) ((A((x.)) and LIMI. (H.),
Combining these yields
LIML-SSL = ACE SSL + AIMLLIML.SE
(64)
To further harness the power of soft labels, we follow the pipelines of AugSeg 80 and stack BL. on top of their augmentations: A.(x,) BLS(A, AugSog(Z2) and An (xw) BLS Aug
Aug (zw)).
21

---

Here, Ae. AugSog and Augaug denote augmentations applied to labeled and unlabeled images in AugSeg, respectively. We refer readers their paper for more details.
As shown in Table T1 we achieve SOTA results on semi-supervised learning. Table 12 we ablate the role of each loss term PASCAL VOC using DL3+-R101.
Table 11: Comparing with SOTA segmentation SSL methods on PASCAL VOC of various splits using DL3+-R101, All results are mloU (%). Red: the best column.
Matiod
92
183
365
752
1464
Supervise
43.92
39.10
65.88
U2FL
IMAS
77
70.0
75.3
79.1
80.2
82.0
UntMatch 7
75.2
77.2
78.8
79.9
81.2
AugSeg 80
71,09
75.45
78.80
80.37
81.36
JML-SSL
78.47
79.16
Table 12: Ablating each loss term on PASCAL VOC using DL3+-R101, All results are mloU (%). Red: the best in column.
Loos
92
183
CCE.S
43.92
59.10
U
72.80
+ CINL
74.74
77.74
+B1S
76.43
78.47
More Results on DeepGlobe Land
We provide additional results on DeepGlobe Land in Table 13 Furthermore, we present the outcomes of BLS using DL3-R50 and KD using DL3-R18 across 5 folds of DeepGlobe Land in Table [ Given that DeepGlobe Land is relatively small dataset, comprising only 803 images, results for each fold can vary significantly. Nevertheless, when observing the differences between CE and JML, the mean improvement exceeds twice the standard deviation.
Table 13: Results on DeepGlobe Land (%). Best results within CE and JML groups are highlighted in red and green, respectively. Best results across CE and JML groups are underscored.
Model
CE-KD
IML-KD
Asc
86.59
0.87
36,95 1 0.99
86.59 0.84
66.87 1: 0.78
UNe:-550
mloU
68.80
1.07
68.86
4
+ 0.89
1.37 = 0.52
1.68 +
12.34 +
BECE
21.03 1: 0.83
21.00
1.21
40.97 1 0.90
39.22 - 1.06
1.15
0.97
0.76
0.06
UNe: R18
mfoU
64.38
1.50
64.93
1: 0.79
66.40
0.86
66. 1: 0.25
66.87 2: 0.30
67.62
0.27
BCE
1.05 +: 0.48
1.47
0.80
1,38 + 0.81
10.32 4: 0.862
9.15 0.79
9.84 +: 1.53
BECE 1
23.23 1 0.54
22.32
1 0.75
25.23
0.75
40.17 1 0.48
38.13 1: 0.63
39
0.94
Acc
84.03
1.11
84,92
1.22
85.64
1.02
85.54 1 0.88
85.82 2 0.66
85,90 0.73
UNa-MB2
mloU 1
64.38
1.21
64.03
+: 1.73
86.29
1.39
66.63 4: 0.73
67.04 4: 0.77
67.46 0.99
ECE
2.04
0.74
RRCF
21.15 0.89
20.26 0.74
24.76 1.17
40.40 + 0.89
39.28 - 0.27
38.82 =: 1.12
Table 14: Results of BLS using DL3-R50 and KD using DL3-R18 on 5 folds of DeepGlobe Land. All results mloU (%).
Mathod
Luss
2
CE
67.56
67.88
69.46
86
+:
BLS
70.60
60.93
69.73
INE
0.57
0.66
1.01
0.61
1.47
0.36 4: 0.34
CL
61.3
67.47
66.22
06.37
67:00
86. 40
1: 0.88
DiI
2.27
0.49
1.28
1.25
0.84
1.22 1 0.59
K More Results on Calibration
CE a proper scoring rule 20), while the soft Dice loss is not [3] Although the soft Dice loss can significantly increase a model's segmentation performance, it can hurt model calibration [1] We observe the same phenomenon with JML and demonstrate potential solution by training the model with soft labels as delineated in Section 3.2 Calibration of the model can be further improved using
post-hoe 20 15 39
47 (52) and trainable calibration methods
However, these methods
extend beyond the scope of this paper.
22

---

Besides, we find that models trained with JML, despite having poorer top-class calibration as quantified by BCE, actually attain superior multi-class calibration evaluated by the static calibration emor (SCE) 45. In Table 15 we report both ECE and SCE on Cityscapes and PASCAL. VOC using D103-R11 It is crucial to highlight that models trained with soft labels may exhibit worse SCE. On one side, soft labels provide an additional regularization during training, which is advantageous for calibration [44]. However, on the flip side, soft labels can represent unrealistic distributions, potentially detrimental for multi-class calibration.
Table 15: Calibration results on Cityscapes and PASCAL VOC using DL3-R101. Best results within CE and JML groups are highlighted in red and green, respectively. Best results across CE and JML groups are underscored.
DiEase
IMI-BES
BCE (5)
0.76 0.0G
0.97
+ 0.04
2.74 0.03
2.09
0.02
0.00
BECE (15)
16.14 0.22
11.59
0.12
30.78 2 0.19
21.20
0.07
DSCE (3c)
23.81 0.00
23.89
= 0.01
23.40
23.23 1 0.01
VOC
SCE (9)
2.57
0.02
2.39 1 0.04
2.40
2.35
0.03
BECK (18)
20.04 0.12
17:26
0.14
82.70
0.09
22.30 + 0.02
0.07
0.10
L
Figures
Figure
The best and mloU (%) for different &: on PASCAL VOC using DL3-R18.
Figure
Effects of on PASCAL VOC using DL3-R101/50/18.
Figure
Code to compute active classes
Figure
Qualitative results on Cityscapes.
Figure
Qualitative results PASCAL VOC.
mloU
0.8
75.5
mioU (86
0.6
75.0
74.5
0.2
74.0
0.0
3
5
Figure The best and mloU (%) for different on PASCAL. VOC using DL33-18.
"In our code, there was an error in the computation of SCE, As a result, values reported here differ from those in earlier arXiv version.
23

---

$2.5
4.
$2.0
81.5
mlat (8)
50-
79.5
2.5
0.35
0.50
0.55
0.50
0.35
0.40
0.45
0.50
0.60
0.65
(a) DL3-R101 mloU (%)
(b) DL3-RI01 ECE (%)
80.0
4.0
mld
79.5
3.5
0
0.35
0.40
0.45
0.50
0.55
0.50
0.65
0.35
0.40
0.45
0.50
0.55
0.69
0.5
(c) DL3-R50 mloU (%)
(d) DL3-R50 ECE (%)
76.5
5.5
76.0
74.5
74.
73.5
3.5
(c) DL3-R18 mloU (%)
(f) DL3-R18 ECE (%)
Figure 5: Effects of e on PASCAL VOC using DL3-RI01/50// e = 0 is the baseline (no smoothing). The highest and the lowest mean values are highlighted in red and green horizontal lines, respectively.
24

---

import torch
from torch. module.. loss import Loss
class JDTLoss (_Los))
prob. shape label shape (batch_size, num_classes, W)
def conpute_active_cles (self, prob,
10
label,
12
active.clases_mode,
13
num_classes:
14
if
mode == "ALL":
mask torch ones (num_classes. dtype=torch. bool)
elif aetive_classes_ng
"PRESENT";
17
mask torch torch.argmax (label, dim-1) unique () elif active_classes_mode mask = torch aprob dim=(0, 2)) > self threshold elif active.classes_node
18
"PROB":
19
"LABEL":
mask torch (label,,,, dim=(0, 2)) self threshold
22
elif active_classe__odee
"BOTH":
23
mask torch amax (prob label, dim=(0, 2)) self
threshold
25
actsve_classes torch zeros classes,
26
dtype-torch bool, device=prob device)
27
active_classed [mask]
30
return active_classes
Figure 6: Code to compute active classes. Please refer to our codebase for more details.
25

---

Image and GT
Teacher
Student with KD
Student without KD
Figure 7: Qualitative results on Cityscapes. Confidence maps are in black and white. Predictions are in RGB.
26

---

Image and GT
Teacher
Student with KD
Student without KD
Figure 8: Qualitative results on PASCAL VOC. Confidence maps are in black and white. Predictions in RGB.
27


Reviews:
[{"review_id": "BGhFFpvqKT", "reviewer": "Reviewer_jE79", "summary": "In this paper, the authors propose Jaccard Metric Losses (JMLs) for the training with soft labels in semantic segmentation. The authors analyze the limitations of previous IoU losses for soft labels, and propose JMLs to solve such limitations. Experiments for label smoothing, knowledge distillation and semi-supervised learning demonstrate the effectiveness of JMLs.", "questions": "Please solve my concerns in Weaknesses.", "limitations": "No", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is interesting that using Jaccard-based losses to solve the soft label training problem.\n2. This paper is well written and easy to read.\n3. Experiments show large improvements for multiple architectures.", "weaknesses": "1. The models in experiments seem a bit old. It would be great to show the performance with more recent architectures, such as Mask2Former.\n2. Only small models are compared in experiments. It is important to show the performance for larger models to demonstrate the scalability.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the evaluation that our paper is interesting, well-written, and has large improvements over multiple architectures. We respond to your concerns in terms of architecture selections as follows.\n\n* **The models in experiments seem a bit old.** In the main paper, we mainly experiment with DeepLabV3, DeepLabV3+ and PSPNet. We select these models to be in line with SOTA segmentation KD papers (SKD, IFVD, CWD, CIRKD, MasKD, DIST) and SOTA segmentation SSL papers (U2PL, iMAS, UniMatch, AugSeg). Moreover, the reviewer could kindly refer to Appendix J, where we present results on ADE20K using SegFormer (NeurIPS2021, ~1500 citations), which is one of the most popular and recent segmentation models.\n\n* **Mask2Former (CVPR2022, ~500 citations).** Thanks for this specific suggestion. We'd like to share our considerations here. This model represents a significant deviation from other mainstream segmentation models such as DeepLabV3, DeepLabV3+, PSPNet, SegFormer, UPerNet, UNet, etc. Its innovation does not only lie in its architecture design (masked attention), but also in its training settings (the Hungarian algorithm and mask loss). These distinctive features aim to unite semantic/panoptic/instance segmentation, but unavoidably make training on each individual task more complex. To evaluate JMLs' potential in a more clean setting and avoid additional compounding factors, our evaluation does not include this specific method design. We'll add the discussion to our paper.\n\n* **Only small models are compared in experiments.** Thanks for this comment and suggestion. We have experimented with DeepLabV3-ResNet101 and DeepLabV3+-ResNet101. These models are fairly large. Also, just for some reference, numerous excellent segmentation literature, including the baselines in our paper (SKD, IFVD, CWD, CIRKD, MasKD, DIST, U2PL, iMAS, UniMatch, AugSeg) have not experimented with larger models. To position the ability of JML in this vaster literature, we select these models for comparison. Moreover, to echo your valuable concern and further underscore the effectiveness of JML on larger models, in **Table 15 (global response)**, we deliver additional results on ADE20K using SegFormer-B5, which is the largest model in the SegFormer family. \n\nWe hope our responses can solve your concerns. Please let us know if you have any follow-up comments."}, {"review_id": "Ah6k7LnwSo", "reviewer": "Reviewer_L5GW", "summary": "The paper focuses on making IoU losses flexible to process soft labels. Jaccard Metric Losses are presented that works the same as soft Jaccard loss with hard labels but it is also compatible with soft labels. Experiments are presented in several tasks: label smoothing, knowledge distillation, and semi-supervised learning. The results show improvement over baselines.", "questions": "Please address the concerns in the weaknesses section.\n\n-- The rebuttal addressed most of my concerns. Hence, I increase the score.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper focuses on addressing an important issue with IoU losses that they are not flexible to process soft labels\n+ Experiments on several tasks (i.e., label smoothing, knowledge distillation, and semi-supervised learning) shows promising results.", "weaknesses": "There are several confusions regarding experimental results.\n\n--There are discrepancies between the results presented in Table 5 and Table 3 for Cityscapes with DL3-R18. Table 3 reports 77.91 ± 0.16 whereas Table 5 reports 76.68 ± 0.33 as the best performance. For VOC, the best results match in the tables (75.89 ± 0.29). However, there is confusion between JML-KD and JML-BLS. For VOC, the JML-KD results in Table 3 match JML-BLS results in Table 5.\n\n-- As shown in Table 3, in knowledge distillation proposed student achieves 78.14. This is a bit surprising as the Teacher DeepLabV3-R101 has a lower mIoU of 78.07 (as reported in DIST[21]). \n\n-- The reported performance in Table 1 and Table 2 do not compare with SOTA performance as shown in the papers. For example, the deeplabv3  paper with DL3-R101 model reports 81.3 mIoU in CityScapes and 85.7 mIoU in Pascal VOC. The presented proposed method results are lower compared to that and baselines are lower too. We see this for all the cases in Tables 1 and 2. I would suggest the authors use the proposed loss with the best-performing model and losses from the prior works to avoid any confusion.  To better convince the readers,  I think improving on the best results reported in prior works would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the recognition of our motivation and promising results over several tasks. We answer your questions with respect to experimental settings as follows. Please also refer to our response to **reviewer FApN** for a more discussion of experimental setups.\n\n* **There are discrepancies between the results presented in Table 5 and Table 3 for Cityscapes with DL3-R18.** In all tables except for Table 3, we follow the training setting as mentioned in Appendix B. In particular, we use a batch size of 8 in Table 1 and 5. Training on Cityscapes is very expensive due to the large crop size. However, in order to match the training setting in SOTA segmentation KD papers (CIRKD, MasKD, DIST), as mentioned in the caption of Table 3, we increase the batch size from 8 to 16 solely for Cityscapes experiments. Therefore, the performance of DeepLabV3-ResNet18 on Cityscapes in Table 3 is higher than those in Table 1 and 5. We will ensure this distinction is highlighted in the revised version of the paper.\n\n* **Table 5.** This table is designated for an ablation study of JML-KD. In Table 5, we isolate the contribution of each proposed term, which are defined in section 2.4 and explained in the caption. In particular, $\\mathcal{L}_{\\text{JML-BLS}}$ means we train the teacher with JML-BLS. Hence, in Table 2, 3, 5, the results of JML-KD on PASCAL VOC using DeepLabV3-ResNet18 should match, i.e. $75.89 \\pm 0.29$.\n\n* **Teacher of DIST vs. ours.** Their teacher is DeepLabV3-ResNet101 trained with CE (78.07), while ours is DeepLabV3-ResNet50 trained with JML-BLS ($79.10\\pm0.35$). We choose ResNet50 instead of ResNet101 as the teacher to save the computational cost, and the reason for training the teacher with JML-BLS is described in line 187-192 and ablated in Table 5, 7.\n\n* **Results of DeepLabV3-ResNet101 in [1] vs. ours.** In [1], 81.3 on Cityscapes is on the test set, with flipping inference and multi-scale inference. Their results on the val set, without flipping inference and multi-scale inference, is 77.82. 85.7 on PASCAL VOC is also on the test set, with flipping inference, multi-scale inference, and COCO pretraining. Their results on the val set, without flipping inference, multi-scale inference, and COCO pretraining, is 78.05. We do not include these extra ingredients, i.e. inference strategies (flipping, multi-scale) and COCO pretraining, and report the results on val set, following and for a fair comparison with other segmentation papers considered in our study (SKD, IFVD, CWD, CIRKD, MasKD, DIST, U2PL, iMAS, UniMatch, AugSeg). For example, the performance of DeepLabV3-ResNet101 on Cityscapes reported in DIST is 78.07. Ours trained with CE on Cityscapes val set and PASCAL VOC val set are $78.67 \\pm 0.32$ and $78.39 \\pm 0.09$, respectively.\n\n* **Comparing with SOTA results.** In **Table 16 (global response)**, we compare with MMSegmentation, which provides a large-scale benchmark of various segmentation models, on Cityscapes, PASCAL VOC and ADE20K. We obtain comparable results on Cityscapes and PASCAL VOC using DeepLabV3-ResNet101. Note that on ADE20K, SegFormer-B3 is trained with 160K iterations in MMSegmentation, while ours is only 40K.\n\nWe hope that our responses can address your concerns. Please let us know if you have any follow-up comments.\n\n[1] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking Atrous convolution for semantic image segmentation. arXiv, 2017."}, {"review_id": "09wt7Rf3WN", "reviewer": "Reviewer_FApN", "summary": "The paper proposes JML (Jaccard Metric Losses) for segmentation. This builds up on previous IoU-based losses but with the added ability to process soft labels or probability distributions. JML losses can be further applied as follows:\nJML-BLS (Label smoothening Jaccard-based IoU loss, which is only computed at boundary pixels based on empirical analysis).\nJML-KD (Applying the soft JML loss to between student and teacher, as well as applying the soft JML loss between student and GT).", "questions": "1. In tables 1-13, Did each experiment (row) start with the same set of pre-trained weights?\n2. How did you tune the hyper-parameters for JML losses (specifically loss weights)? How easy(or difficult) is it to obtain the results provided in the paper.\n3. Does adding JML ever reduce performance (with a higher weight, etc)\n4. Could you address limitations of this work", "limitations": "No - The authors are suggested to address the limitations of this method (if any).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Presentation:\nThe reviewer appreciates authors' clarity of presentation of their concept. The paper is quite clear and tables are easy to understand. Mathematical notations have been thoroughly reviewed and hence the paper reads well.\n\nOriginality:\nAlthough the work takes inspiration from IoU loss, and is quite a simple update to the original IoU loss concept, the technical contribution is quite clear and holds up. There has been work on IoU losses before but they are not used widely in literature, hence, there is a scope for improvement in this area. This work is a beneficial building block to further research in IoU losses.\n\nTransparency:\nThe reviewer appreciates the code being made available.\n\nResults:\nThe JML loss seems to work on a wide range of datasets, and more importantly, backbones. There are 12 different architectures in the paper as well as supp, and the loss works for both CNNs as well as transformer based backbones. This is very interesting as it shows the capability of this loss to provide a high impact in our field.\n\n", "weaknesses": "There is a lot of information in the appendices, which might be more beneficial than some of the things in the main paper. One such example is the experiment with transformer-based backbones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of the originality, transparency, and potentially high impact of our work. We are encouraged. We answer your questions about experimental setups and hyperparameter tuning as follows.\n\n* **Experimental setups.** We assure that each experiment starts with the same set of pre-trained weights, with the exception of those in Table 3, 4 where we adopt numbers reported in those SOTA papers. Since different SOTA methods usually adopt diverse codebases, pretrained weights and training hyper-parameters, these tables cannot offer a completely fair comparison. However, as noted in line 194-200, our checkpoints are from the standard timm library pretrained on vanilla ImageNet (not ImageNet21K). Furthermore, as outlined in line 224-229, our baseline student model does not unfairly (e.g. longer training epochs, larger crop size, pre-training on larger datasets, etc) outperform those in other KD methods.\n \n* **How did you tune the hyper-parameters for JML losses?** We use a 0.25/0.75 ratio to mix CE and JML, following the suggestion in MMSegmentation. The ratio of 0.5/0.5 is also widely used in many segmentation papers, e.g. DETR, Mask2Former, SAM, etc.\n\n* **Does adding JML ever reduce performance?** According to your question, in **Table 17 (global response)**, we add ablation studies for JML weights on PASCAL VOC using both DeepLabV3-ResNet101 and DeepLabV3-ResNet18. Compared with using CE alone (the rightmost column), adding JML will not reduce performance across various loss weights.\n\n* **Could you address limitations of this work?** Thank you for drawing attention to this important aspect. We recognize several limitations and future research directions, and we will add this discussion into our revised manuscript. To elucidate:\n  * Currently, we only demonstrate JMLs' effectiveness as a metric in the label space. Nevertheless, in many applications or techniques, feature-space metrics are beneficial, e.g., the feature distillation technique. Whether JML can be successfully extended to the feature space is worth studying.\n  * Our application of JMLs is predominantly in semantic segmentation. However, given the exceptional efficacy of JMLs in optimizing Acc (as detailed in lines 210-214), we anticipate their applicability even in standard classification tasks. This is especially relevant in scenarios with data imbalance, as recently delved into by the NLP community [4].\n  * While our current study concentrates on three soft label applications (LS, KD, SSL), there are a plenty of scenarios incorporating soft labels that worth deeper exploration. This includes areas such as neural architecture search [1], dynamic inference [2], and adversarial machine learning [3], among others.\n\n* **Suggestions for writing.** Thank you for your suggestion. We will add the results on ADE20K into the main text of the revised paper.\n\nWe hope that our responses can address your concerns. Please let us know if you have any follow-up comments.\n\n[1] Li, Changlin, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xiaojun Chang. Block-wisely supervised neural architecture search with knowledge distillation. CVPR, 2020.\n\n[2] Yu, Jiahui, and Thomas S. Huang. Universally slimmable networks and improved training techniques. ICCV, 2019.\n\n[3] Papernot, Nicolas, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. S&P, 2016.\n\n[4] Li, Xiaoya, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. Dice loss for data-imbalanced NLP tasks. ACL, 2020."}, {"review_id": "bjsDgq4FaF", "reviewer": "Reviewer_83cg", "summary": "The paper introduces Jaccard Metric Losses (JMLs) as a flexible alternative to Intersection over Union (IoU) losses for optimizing the Jaccard index in semantic segmentation tasks. Unlike IoU losses, JMLs can effectively process soft labels, enabling the use of important training techniques like label smoothing, knowledge distillation, and semi-supervised learning.", "questions": "Why CE group is much better than JML group in terms of calibration?", "limitations": "Needs to determine hyperparameters like $\\epsilon$", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Jaccard Metric Losses (JMLs) introduce a novel approach by mirroring the soft Jaccard loss in standard settings with hard labels while remaining compatible with soft labels. This flexibility is a significant contribution as it enables the incorporation of vital training techniques that rely on soft labels, such as label smoothing, knowledge distillation, and semi-supervised learning. This novel aspect makes JMLs an interesting and valuable addition to the field.\n2. The paper demonstrates good results by showcasing consistent improvements over the cross-entropy loss across multiple semantic segmentation datasets (Cityscapes, PASCAL VOC, ADE20K, DeepGlobe Land) and a variety of architectures, including both classic CNNs and recent vision transformers. \n3. The paper provides theoretical analysis of JMLs, which enhances the understanding of their properties and behavior.", "weaknesses": "1. The paper highlights IoU losses as the baseline and the main motivation for introducing JMLs. However, the lack of direct comparisons between IoU losses and JMLs in the experiments is a weakness. While certain comparisons may not be possible, such as in knowledge distillation (KD) where IoU loss baselines require teachers, it would have been beneficial to include comparative analyses that demonstrate the importance of adapting IoU losses to soft labels. This would have provided a stronger justification for the need and effectiveness of JMLs.\n2. Cross-entropy (CE) achieves better calibration performance in Tables 1 and 2. However, more discussion and explanation of this observation is needed. Understanding the reasons behind this disparity and addressing any potential limitations or trade-offs associated with JMLs' calibration performance would have added depth to the paper's analysis and interpretation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the recognition of our experiments results and theoretical analysis. We answer your questions about comparisons with other IoU losses and calibration below.\n\n* **Comparisons with other IoU losses.** Please refer to Appendix C-H. In particular, in Table 9, various losses are compared on Cityscapes and PASCAL VOC using DeepLabV3-ResNet18. We provide additional results on PASCAL VOC using DeepLabV3-ResNet101 in **Table 18 (globe response)**. In summary, JMLs are comparable with other IoU losses using hard labels, but can additionally benefit from soft labels.\n\n* **Why CE group is much better than JML group in terms of calibration?** We have a more detailed discussion of calibration in Appendix L. We appreciate your suggestion and will include additional discussions in the revised version. In particular, the reason why CE obtains better calibration than JML is because CE is a proper scoring rule [1], while SDL/SJL/JML will push predictions towards 0/1 distributions. This phenomenon is widely observed and discussed in the literature [2,3,4,5]. However, soft labels, as well as post-training techniques can greatly improve the calibration of models trained with SDL, bringing them closer to the calibration levels achieved by CE [2]. Furthermore, as highlighted in Appendix L, our findings indicate that JML attains superior calibration as measured by SCE.\n\n* **Choosing hyper-parameters (BLS $\\epsilon$, JML weights).** We admit that BLS is sensitive to the choice of $\\epsilon$. Nevertheless, Figure 6 (Appendix M) suggests that JML-BLS maintains solid performance across a broad range of $\\epsilon$ values. Besides, please refer to our response to **Reviewer FApN** for a discussion on the selection of JML weights. \n\nWe hope that our responses can address your concerns. Please let us know if you have any follow-up comments.\n\n[1] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML, 2017.\n\n[2] Axel-Jan Rousseau, Thijs Becker, Jeroen Bertels, Matthew B. Blaschko, and Dirk Valkenborg. Post training uncertainty calibration of deep networks for medical image segmentation. ISBI, 2021.\n\n[3] Jeroen Bertels, David Robben, Dirk Vandermeulen, and Paul Suetens. Theoretical analysis and experimental validation of volume bias of soft Dice optimized segmentation maps in the context of inherent uncertainty. MIA, 2021.\n\n[4] Jorg Sander, Bob D. de Vos, Jelmer M. Wolterink, andIvana Isgum. Towards increased trustworthiness of deep learning segmentation methods on cardiac MRI. SPIE, 2019.\n\n[5] Alireza Mehrtash, William M. Wells, Clare M. Tempany, Purang Abolmaesumi, and Tina Kapur. Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation. TMI, 2020."}, {"review_id": "ZXoxqmlHuD", "reviewer": "Reviewer_KUTw", "summary": "This paper presents a new loss function for semantic segmentation based on the IoU loss. The authors found that the vanilla IoU loss is incompatible with the soft labels when label smoothing and knowledge distillation training techniques are used. Experiments show that the proposed approach works well on four widely-used semantic segmentation datasets. Ablation experiments also show some insightful take-home message for the readers.", "questions": "- Some results using Transformers should be moved to the main paper though the improvement compared to the CNNs is not that significant.\n\n- It is also recommended to describe some works on semantic segmentation in the related work section.\n\n- Failure cases should be provided and discussed.", "limitations": "Not provided.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of this paper is interesting. The authors observe an interesting problem of the vanilla IoU loss.\n\n- The novelty of this paper is clear. Improving the vanilla IoU loss is a meaningful topic for semantic segmentation.\n\n- The results are good. On four segmentation benchmarks, the proposed approach receives improvements.", "weaknesses": "- In the main paper, the authors only show results based on ConvNets. The results based on Transformers are provided in the supplementary materials. From the results, it can be seen that the improvement of the proposed approach on ConvNets are higher than that on Transformers. I think the authors should explain why this happens.\n\n- Some visualizations should be moved to the main paper. I think it is interesting to see how the segmentation results change when the proposed method is used.\n\n- I also would like to recommend the authors to add some failure case analysis to see what readers can further do for semantic segmentation.\n\n- I am also interested in the results when large-sized models are used, for instance, SegFormer-B5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our motivation, novelty and experimental results. We address your questions about transformer experiments below. \n\n* **SegFormer-B5 experiments.** Please refer to **Table 15 (global response)**.\n \n* **Improvements on transformers are relatively lower.** We believe it is more of a limitation of BLS and KD, as reflected by relatively lower improvements in mIoU when comparing CE vs. CE-KD, and JML vs. JML-KD. Nonetheless, JML groups continue to outperform CE groups by a large margin on ADE20K using transformers (around 2\\% in mIoU, even with SegFormer-B5). Moreover, we still observe a pronounced enhancement in calibration upon training with soft labels. Although our study primarily focuses on BLS, KD, and SSL, it is worth mentioning that soft labels have diverse applications beyond these, all of which could potentially gain from JML. As for the explanation, several studies [1,2,3,4] have highlighted the intricate challenges KD grapples with when integrated with transformers. For instance, to effectively utilize KD with transformers, the student model should learn from the teacher during both the pretraining and fine-tuning phases [1,2,3]. In contrast, we directly adopt pretrained weights from the timm library to avoid expensive pretraining. We will incorporate this discussion into the revised version.\n\n* **Suggestions of writing.** We value your constructive feedback on our writing and will make the necessary revisions accordingly. In particular, we will move the results on ADE20K into the main text in the revised paper.\n\nWe hope that our responses can address your concerns. Please let us know if you have any follow-up comments.\n\n[1] Sanh, Victor, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. NeurIPS Workshop, 2019.\n\n[2] Jiao, Xiaoqi, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. EMNLP Findings, 2020.\n\n[3] Wu, Kan, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. ECCV, 2022.\n\n[4] Lin, Sihao, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and Gang Wang. Knowledge distillation via the target-aware transformer. CVPR, 2022."}]

Author rebuttal:
We thank all reviewers for your time and efforts. The questions and suggestions are very valuable for us to improve our work. In this global response, we attach a one-page pdf that contains additional experiments. 

Meta-review: The paper receives four positive ratings and one borderline rejection. The AC took a close look at the paper, reviews, and rebuttals. Specifically, for reviewer L5GW who provided the negative review, the AC finds that the raised questions are addressed well.

The paper demonstrates an effective loss function for semantic segmentation, which is applicable for various techniques, e.g., label smoothing, distillation, and semi-supervised learning. The authors also show extensive experiments with various model architectures and benchmark datasets. Therefore, the AC recommends to accept this paper.
Decision: Accept (poster)