Title: Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels
Venue: NeurIPS 2023
Accepted: yes
Keywords: Semantic Segmentation
Abstract: Intersection over Union (IoU) losses are surrogates that directly optimize the Jaccard index. Leveraging IoU losses as part of the loss function have demonstrated superior performance in semantic segmentation tasks compared to optimizing pixel-wise losses such as the cross-entropy loss alone. However, we identify a lack of flexibility in these losses to support vital training techniques like label smoothing, knowledge distillation, and semi-supervised learning, mainly due to their inability to process soft labels. To address this, we introduce Jaccard Metric Losses (JMLs), which are identical to the soft Jaccard loss in standard settings with hard labels but are fully compatible with soft labels. We apply JMLs to three prominent use cases of soft labels: label smoothing, knowledge distillation and semi-supervised learning, and demonstrate their potential to enhance model accuracy and calibration. Our experiments show consistent improvements over the cross-entropy loss across 4 semantic segmentation datasets (Cityscapes, PASCAL VOC, ADE20K, DeepGlobe Land) and 13 architectures, including classic CNNs and recent vision transformers. Remarkably, our straightforward approach significantly outperforms state-of-the-art knowledge distillation and semi-supervised learning methods. The code is available at \href{https://github.com/zifuwanggg/JDTLosses}{https://github.com/zifuwanggg/JDTLosses}.

Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels

Abstract

Intersection over Union (IoU) losses are surrogates that directly optimize the Jaccard index. Leveraging IoU losses as part of the loss function have demonstrated superior performance in semantic segmentation tasks compared to optimizing pixelwise losses such as the cross-entropy loss alone. However, we identify a lack of flexibility in these losses to support vital training techniques like label smoothing, knowledge distillation, and semi-supervised learning, mainly due to their inability to process soft labels. To address this, we introduce Jaccard Metric Losses (JMLs), which are identical to the soft Jaccard loss in standard settings with hard labels but are fully compatible with soft labels. We apply JMLs to three prominent use cases of soft labels: label smoothing, knowledge distillation and semi-supervised learning, and demonstrate their potential to enhance model accuracy and calibration. Our experiments show consistent improvements over the cross-entropy loss across 4 semantic segmentation datasets (Cityscapes, PASCAL VOC, ADE20K, DeepGlobe Land) and 13 architectures, including classic CNNs and recent vision transformers. Remarkably, our straightforward approach significantly outperforms state-of-theart knowledge distillation and semi-supervised learning methods. The code is available at https://github.com/zifuwanggg/JDTLosses.

1 Introduction

The Jaccard index, also known as the intersection over union (IoU), is a widely used metric in the evaluation of semantic segmentation models. Its appeal lies in its scale invariance and its superior ability to reflect the perceptual quality of a model compared to pixel-wise accuracy [17, 40]. In line with the principles of empirical risk minimization in statistical learning theory, the metric used for evaluation should also be optimized during training [60]. Consequently, directly optimizing IoU via differentiable surrogates has found considerable attention in the literature [46, 49, 2, 72, 17, 73].

Notably, IoU is only defined when both predictions and ground-truth labels are discrete binary values, i.e., they reside on the vertices of a p -dimensional hypercube { 0 , 1 } p . However, the output of neural networks are soft probabilities that are in the interior of the hypercube [0 , 1] p . To extend the value of IoU from vertices to the entire hypercube, two popular strategies are currently in use. The first strategy relaxes set counting with norm functions, exemplified by the soft Jaccard loss (SJL) [46, 49] and the soft Dice loss (SDL) [57, 17]. The second strategy computes the Lovasz extension of the IoU, such as the Lovasz hinge loss [72] and the Lovasz-Softmax loss (LSL) [2]. These losses facilitate a plug-and-play use and have significantly enhanced the performance of segmentation models, outperforming pixel-wise losses such as the cross-entropy loss (CE) and the focal loss [35]. For instance, Rakhlin et al. [50] won the land cover segmentation task of the DeepGlobe Challenge [12] utilizing LSL. Additionally, SDL is becoming increasingly prominent in recent segmentation works [5, 9, 8, 29], such as Segment Anything [29]. Moreover, SJL and SDL are now the standard for training medical imaging models [17, 27].

1

Despite their widespread application, current IoU losses lack the flexibility needed to accommodate key training techniques. Specifically, while these losses relax predictions from { 0 , 1 } p to [0 , 1] p , they neglect the possibility of labels residing in [0 , 1] p , making them incompatible with soft labels. Soft labels have been employed in numerous state-of-the-art models and have proven effective in enhancing network accuracy [37, 68, 62] and calibration [20, 44, 42, 82]. For example, label smoothing (LS) [58] generates soft labels by taking a weighted average of one-hot hard labels and a uniform distribution over labels. In knowledge distillation (KD) [22], class probabilities generated by a teacher network serve as soft labels to guide a student model. In semi-supervised learning (SSL) such as MixMatch [4], k views of an unlabeled image are fed to a classifier, and the predictions are averaged to create soft labels.

Motivated by the limitation of existing IoU losses, particularly their inability to accommodate soft labels and, consequently, techniques like LS, KD, and SSL, we propose two variants of SJL, termed Jaccard metric losses (JMLs) since they are metrics on [0 , 1] p . JMLs yield the same value as SJL on hard labels but are fully compatible with soft labels. Therefore, we can safely replace the existing implementation of SJL with JMLs without affecting the performance on hard labels. To embed our losses with use cases (LS, KD, SSL), we first introduce boundary label smoothing (BLS) which facilitates the integration of label smoothing into the segmentation task. BLS can be used independently and also synergizes with KD and SSL. Subsequently, we present a confidence-based scheme for selecting classes to contribute to the loss computation, thereby enhancing performance in KD. We conduct extensive experiments on 4 datasets (Cityscapes [11], PASCAL VOC [18], ADE20K [81], DeepGlobe Land [12]) across 13 architectures, including classic CNNs [21, 54] and recent vision transformers [69]. Our results demonstrate significant improvements over the cross-entropy loss. Moreover, our straightforward approach outperforms state-of-the-art segmentation KD and SSL methods by a substantial margin.

2 Methods

2.1 Preliminaries

Given a segmentation output ˙ x ∈ { 1 , ..., C } p and a ground-truth ˙ y ∈ { 1 , ..., C } p where C is the number of classes, for each class c , we define the set of predictions as x c = { ˙ x = c } , the set of ground-truth as y c = { ˙ y = c } , the union as u c = x c ∪ y c , the intersection as v c = x c ∩ y c , the symmetric difference (the set of mispredictions) as m c = ( x c \ y c ) ∪ ( y c \ x c ) , and the Jaccard index as IoU c = | v c | / | u c | . For multi-class segmentation, IoU c are averaged across classes, yielding the mean IoU (mIoU). In the sequel, we will encode sets as binary vectors x c , y c , u c , v c , m c ∈ { 0 , 1 } p where p is the number of pixels, and denote | x c | = ∑ p i =1 x c i the cardinality of the corresponding set. For simplicity, we will drop the superscript c in the following.

In order to optimize IoU in a continuous setting, we need (almost everywhere) differentiable interpolations of this discrete score. In particular, we want to extend the IoU loss

with ∆ IoU so that it attains a value with any vector of predictions ˜ x ∈ [0 , 1] p . In what follows, when the context is clear, we will use x and ˜ x interchangeably.

The soft Jaccard loss (SJL) [46, 49] generalizes IoU by realizing that when x, y ∈ { 0 , 1 } p , | v | = ⟨ x, y ⟩ and | u | = | x | + | y | - | v | = ∥ x ∥ 1 + ∥ y ∥ 1 -⟨ x, y ⟩ . Therefore, SJL replaces the set notation with vector functions:

The L 1 norm can be replaced with the squared L 2 norm [17]:

2.2 The Limitation of Existing IoU Losses

The primary shortcoming of current IoU losses is that they do not necessarily have desired properties when presented with soft labels, i.e., when y ∈ [0 , 1] p . This limitation impedes their application in crucial training techniques like LS, KD, and SSL.

Consider the case of ∆ SJL ,L 1 , and for simplicity, a single pixel scenario: ∆ SJL ,L 1 = 1 -xy x + y -xy . It is easy to confirm that for any y > 0 , ∆ SJL ,L 1 is minimized at x = 1 since it monotonically decreases as a function of x . Hence, ∆ SJL ,L 1 is in general not minimized when x = y , a basic property anticipated from a loss function. Further analysis for high-dimensional cases and additional experiments on real datasets are provided in Appendix C and E, respectively.

∆ SJL ,L 2 does not exhibit this issue, since ∆ SJL ,L 2 = 0 ⇔ | x | 2 2 + | y | 2 2 -2 ⟨ x, y ⟩ = 0 ⇔ x = y . However, it is known to yield inferior results compared to its L 1 counterpart, possibly due to its flatter nature around the minimum [17]. In practice, it is rarely used. For instance, in SMP [25], a popular open-source semantic segmentation project, only the L 1 version is implemented. Our evaluations in Appendix E confirm its inferior performance relative to the L 1 version. Additionally, the approach of substituting set notation with the L 1 norm is widely utilized in numerous other works, including the soft Dice loss [57, 17], the soft Tversky loss [53], the focal Tversky loss [1], and others. The soft Dice loss is also included in the formulation of the PQ loss [61] which is used in panoptic segmentation [28]. Consequently, all of them struggle with soft labels.

Losses based on the Lovasz extension, such as the Lovasz-Softmax loss [2], the Lovasz hinge loss [72], and the PixIoU loss [73], cannot handle soft labels as the Lovasz extension is not well-defined for y ∈ (0 , 1) p . More details and comparisons with the Lovasz-Softmax loss can be found in Appendix D and E, respectively. Automatically searched loss functions, such as Auto Seg-Loss [33] and AutoLoss-Zero [32], also fail to accommodate soft labels as their search space is confined to integral labels.

In summary, despite their widespread adoption in recent works on semantic segmentation [17, 27, 9, 8, 29] and panoptic segmentation [5, 61, 74, 75], these losses all exhibit a common shortcoming: an inability to handle soft labels. In this paper, we specifically concentrate on re-designing SJL. Other losses, including the soft Dice loss, the soft Tversky loss, and the focal Tversky loss, are addressed in our subsequent work [66].

2.3 Jaccard Metric Losses

We can rewrite the intersection | v | and the union | u | as a function of the set difference | m | :

Note that | m | = ∥ x -y ∥ 1 . Combining these yields:

where the equalities hold when x, y ∈ { 0 , 1 } p .

After eliminating erroneous combinations that have the same issue as SJL, we are left with two candidates ∆ JML,1 , ∆ JML,2 : [0 , 1] p × [0 , 1] p → [0 , 1] that are defined as:

It is a well-known result that ∆ IoU is a metric on { 0 , 1 } p [30]. In Theorem 2.1 (see Appendix F for the proof), we show that both ∆ JML,1 and ∆ JML,2 are also metrics on [0 , 1] p . Therefore, we call them Jaccard Metric Losses (JMLs).

0.0

0.2

0.4

0.6

0.8

1.0

Prediction

0.0

0.2

0.4

0.6

0.8

1.0

Loss value

CE

JML

0.0

0.2

0.4

0.6

0.8

1.0

Prediction

0.0

0.2

0.4

0.6

0.8

1.0

CE

JML

Theorem 2.1. Both ∆ JML,1 and ∆ JML,2 are metrics on [0 , 1] p . Neither ∆ SJL ,L 1 nor ∆ SJL ,L 2 is a metric on [0 , 1] p .

Recall the definition of a metric:

̸

Definition 2.2 (Metric [14]) . A mapping f : M × M → R is called a metric on M if for all a, b, c ∈ M , it satisfies the following conditions: (i) (reflexivity). f ( a, a ) = 0 . (ii) (positivity). a = b = ⇒ f ( a, b ) > 0 . (iii) (symmetry). f ( a, b ) = f ( b, a ) . (iv) (triangle inequality). f ( a, c ) ≤ f ( a, b ) + f ( b, c ) .

Having a loss function ∆ that is a metric carries numerous benefits. Reflexivity and positivity collectively imply that ∀ x, y ∈ [0 , 1] p , x = y ⇔ ∆ = 0 , meaning that ∆ would be compatible with soft labels. The triangle inequality also provides insightful guidance. Applied to KD, it yields:

Here, S represents the student model, T the teacher model, and L the ground-truth labels. In the KD, we initially train the teacher model using the ground-truth labels, and then minimize the loss between the teacher and the student model. Equation (9) suggests that if we adhere to this process-equivalent to minimizing the right-hand side of the equation-we also minimize the upper bound on the student model's loss with respect to the ground-truth labels.

Furthermore, as Theorem 2.3 shows (see Appendix G for the proof), when only hard labels are involved, ∆ JML,1 , ∆ JML,2 and ∆ SJL ,L 1 are identical. Hence, we can safely replace the existing implementation of SJL with JMLs. When soft labels are introduced, ∆ JML,1 and ∆ JML,2 might yield different values. We discuss their distinctions in Appendix H. From both theoretical and empirical perspectives, we find that ∆ JML,1 is slightly more favorable than ∆ JML,2. As a default in our experiments, we use ∆ JML,1.

̸

̸

Figure 1 plots the loss value of ∆ JML,1 and the cross-entropy loss (CE) for soft labels y = 0 . 1 (left) and y = 0 . 9 (right). Typically, the model is randomly initialized, resulting in a low initial confidence. Although the gradient of CE is substantial at the beginning, it plateaus as training progresses and the prediction is close to the target. Conversely, ∆ JML,1 offers effective supervision later in the process. However, when y is extremely small (e.g., y = 0 . 1 ), ∆ JML,1 can become overly steep, potentially causing problems in KD. This issue will be explored in greater detail in Section 2.4.2.

2.4 Use Cases

Soft labels find wide-ranging applications. In this paper, we investigate three of the most common use cases: label smoothing (LS), knowledge distillation (KD) and semi-supervised learning (SSL). We leave the discussion on SSL in Appendix I.

2.4.1 Label Smoothing

In LS [58], the one-hot encoding is combined with a uniform distribution under a smoothing coefficient ϵ , resulting in soft labels SL ϵ . In JML-LS, a network H is trained with the following loss:

̸

LS provides a regularization effect during training. However, when we regard semantic segmentation as a pixel-wise classification task and apply LS to every pixel in the image without distinction, we fail to consider spatial differences. This is particularly evident with SegFormer-B3 on ADE20K, which achieves an overall pixel-wise accuracy of 81 . 91 ± 0 . 02% , but drops to 47 . 98 ± 0 . 01% in the boundary region. These areas, due to their inherent ambiguity, require stronger regularization. Moreover, our empirical findings suggest that smoothing non-boundary regions only yields marginal improvements. Therefore, we introduce boundary label smoothing (BLS), which only applies smoothing to labels near the boundary. We refer to the resulting loss as JML-BLS. Specifically, for every pixel i , we examine its k × k neighborhood N i , and the pixel i is regarded as a boundary pixel if there exists a pixel j ∈ N i such that their ground-truth labels are different: y i = y j . This can be efficiently computed by applying a max pooling layer to the one-hot encoding.

2.4.2 Knowledge Distillation

In KD [22], besides the ground-truth label L , a student model S is trained to minimize the discrepancy to a teacher network T simultaneously:

where L CE, L = CE ( S, L ) and L CE, T = CE ( S, T ) .

In JML-KD, we add additional supervision from JML:

where L JML, L = JML ( S, L ) and L JML, T = JML ( S, T ) .

Combining these, we have

Existing segmentation KD methods rely on pixel-wise supervision either in the output [36, 70] or in the feature space [64, 55]. Our L JML, T, however, enables direct distillation of the teacher's IoU information to the student which aligns with the final evaluation metric.

Moreover, recall that JMLs are averaged for each class, making it crucial to determine which classes contribute to the loss value at each iteration. We refer to these classes as active classes . In KD, the student uses information from non-target classes to learn the class relationship from the teacher [59, 77]. However, a teacher network may also include noisy and unhelpful predictions for unimportant classes. Therefore, we propose to filter out unimportant classes based on the confidence level of the teacher network. Specifically, JMLs are computed only over classes where the teacher's confidence exceeds a predefined threshold. This approach has two benefits: first, the student will not be distracted by irrelevant classes, aiding in generalization. Second, as shown in Figure 1, we find that the loss value of JMLs can become excessively steep around the target when the ground-truth is low (e.g. y = 0 . 1 ). Therefore, ignoring these classes can stabilize the training process.

We have also observed that the student benefits from a teacher trained with JML-BLS. In contrast, in classification tasks, it has been found that a teacher trained with LS can detrimentally affect the student's performance [44]. This is often cited as a counterexample to the notion that a more accurate teacher will necessarily distill a better student. We believe that the teacher, trained through JML-BLS, acquires intricate boundary information, and this "dark knowledge" is implicitly passed to the student, resulting in a more accurate student.

3 Experiments

Experimental setups. We adopt Pytorch Image Models (timm) [67], which provides implementations and ImageNet [13] pre-trained weights for various backbones. In our experiments, CNN backbones include ResNet-101/50/18 [21] and MobileNetV2 [54]; transformer backbones contain MiT-B5/B3/B1/B0 [69]. Segmentation methods consist of DeepLabV3+ [7], DeepLabV3 [6], PSPNet

[78], UNet [51] and SegFormer [69]. We evaluate models on Cityscapes [11], PASCAL VOC [18], ADE20K [81] and DeepGlobe Land [12]. In summary, 13 models and 4 datasets are studied in the sequel. More details of each model are in Appendix A and training recipes are in Appendix B.

Evaluation metrics. To provide a comprehensive comparison, we report both overall pixel-wise accuracy (Acc) and mean intersection over union (mIoU). To evaluate the effects of our methods on calibration, we present the expected calibration error (ECE) [20] and the ECE computed only over the boundary region, denoted as BECE. For Cityscapes, PASCAL VOC, and ADE20K, we repeat the experiments 3 times (except for SSL experiments that are single runs) and report performance on the validation set. For DeepGlobe Land, we conduct 5-fold cross-validation. All results are presented in the format of mean ± standard deviation. We do not apply any test-time augmentation.

3.1 Results on Accuracy

We report results on Cityscapes (Table 1), PASCAL VOC (Table 2), ADE20K (Table 3), and DeepGlobe Land (Table 13, Appendix J). Key takeaways from our results are as follows:

JML significantly improves accuracy (Acc and mIoU). Compared to training with CE alone, incorporating JML as part of the loss function can significantly enhance a model's mIoU. The improvement is typically more than 2% on Cityscapes, PASCAL VOC, and ADE20K. Additionally, it can also increase a model's Acc. This suggests that the benefits of JML are not solely due to its alignment with the evaluation metric (mIoU), but also because it aids the overall optimization process.

JML benefits more from soft labels. For instance, the improvements of mIoU for DL3-R101 on PASCAL VOC are 0.48% (CE vs. CE-BLS) and 1.25% (JML vs. JML-BLS), respectively. We adopted training procedures that are heavily optimized for CE, which might explain why JML requires stronger regularization. More qualitative results can be found in Figure 7 and Figure 8 (Appendix L).

KD is more effective than BLS, while BLS performs well on datasets with simple boundary condition. Both BLS and KD consistently improve the model's accuracy. Despite its simplicity, BLS can yield significant improvements, especially on PASCAL VOC where boundary condition is less complex. As boundary condition becomes more intricate, as in Cityscapes and ADE20K, KD, seen as learned label smoothing [76], typically outperforms BLS.

SOTA results on segmentation KD. Without bells and whistles, our simple approach that only uses soft labels greatly exceeds SOTA segmentation KD methods, as shown in Table 4. Indeed, the model trained only with hard labels in JML already achieves a comparable result as some of these methods. For example, on PASCAL VOC, DL3-R18 achieves 74 . 42 ± 0 . 52% mIoU while CIRKD [70], a complex distillation method only attains 74 . 50% mIoU. Note that our baseline is not stronger than theirs: our DL3-R18 trained with CE has 72 . 47 ± 0 . 33% mIoU while their number is 73 . 21% (because we use a smaller batch size, see Appendix B).

SOTA results on segmentation SSL. By incorporating JML-BLS with AugSeg [80] (refer to Appendix I for further details), we achieve SOTA segmentation SSL results as illustrated in Table 11 (Appendix I). Notably, improvements are particularly significant on smaller splits - exceeding a 4% enhancement over AugSeg on the 92 split.

3.2 Results on Calibration

CE-BLS and CE-KD sometimes hurt calibration. Contrary to the common belief in classification [44, 42, 82], we find that both CE-BLS and CE-KD can sometimes deteriorate model calibration. For instance, in Cityscapes experiments, the lowest ECE is usually obtained with CE only. Nevertheless, CE-BLS can still improve model calibration near the boundary.

JML compromises calibration. Although the soft Dice loss can significantly improve a model's segmentation performance, it is well known to yield poorly calibrated models [41, 3]. We confirm that this is also the case for JML. Interestingly, while models trained with JML exhibit inferior top-class calibration as measured by ECE, they actually achieve better multi-class calibration as indicated by the static calibration error (SCE). We provide more detailed results on calibration in Appendix K.

JML-BLS and JML-KD consistently improve calibration. Although JML presents challenges in model calibration, these can be greatly mitigated by training models with soft labels. In JML experiments, both JML-BLS and JML-KD reliably enhance model calibration.

4 Ablation Studies

4.1 JML Weights

It is common in recent works [5, 9, 8, 29] to balance CE and the Dice loss with equal weights. For JML, we adopt 0.25/0.75 in all our experiments and find it slightly superior than 0.5/0.5.

4.2 JML-BLS

Although BLS is sensitive to the choice of ϵ , it effectively increases model accuracy and calibration. The effect of the smoothing coefficient ϵ on PASCAL VOC with DL3-R101/50/18 is shown in Figure 5 (Appendix L). Interestingly, for DL3-R50 and DL3-R18, the optimal ϵ that achieves the highest mIoU also yields the lowest ECE.

We need a strong smoothing coefficient near the boundary. The optimal ϵ for different kernel size k on PASCAL VOC with DL3-R18 is shown in Figure 4 (Appendix L). Note that k = ∞ implies LS is applied to every pixel, i.e. vanilla LS. Generally, as k increases, we need to decrease the strength of smoothing. The best result is obtained when we only smooth a small region near the boundary with a strong smoothing coefficient ( k = 3 and ϵ = 0 . 50 ).

4.3 JML-KD

Loss terms. We examine the contribution of each loss term in Table 6 using a DL3-R18 student. Adding JML terms significantly improves the student's performance.

Filtering out unimportant classes based on teacher's confidence is useful. We examine the impact of active classes on PASCAL VOC with DL3-R18 in Table 7. In particular, we propose to ignore classes where the soft label, i.e. teacher's confidence, is low (marked as LABEL ). And in ALL , we include all classes. In PRESENT , we select the class with the maximum confidence from the teacher's prediction in the class dimension. In PROB , we skip classes where the student itself is not confident. In BOTH , we take both the teacher's and student's confidences into account. The code to compute active classes is in Figure 6 (Appendix L).

We observe that using ALL classes in JML-KD can misguide the student. In most of the teacher's output classes, the confidence is usually very low. It can be challenging and potentially detrimental for the student to precisely mimic these numbers. With PRESENT , the performance remains similar to that without soft labels. This is because the effectiveness of using soft labels come from the non-argmax classes. PROB and BOTH achieve similar performance, but both are worse than LABEL .

Teacher's calibration is beneficial. In Sec. 2.4.2, we suggest that a teacher trained with JML-BLS offers more boundary information to the student. But what exactly is this boundary information? We

believe it relates to the teacher's calibration. The student, with less capacity, often struggles to mimic a more powerful teacher, especially near ambiguous boundary regions. However, if the teacher is more calibrated and outputs a less peaked distribution, the student will learn this uncertainty rather than attempting to match an unrealistic distribution that it lacks the capacity to replicate.

In Table 8, we compare three teachers with various accuracy and calibration on PASCAL VOC. In particular, both T2 and T3 are trained with JML-BLS but with different smoothing parameter ϵ , while T1 is not. T2, although having a lower mIoU, is more calibrated than T1. Consequently, T2's student is more accurate and better calibrated than T1's.

5 Related Works

IoU is a commonly employed metric in semantic segmentation, and IoU losses strive to optimize this metric directly. IoU only obtains values when both predictions and ground-truth labels are discrete binary vectors { 0 , 1 } p , but the neural network often predicts soft probabilities (after the softmax or sigmoid layer) in [0 , 1] p . Interpolating IoU values from { 0 , 1 } p to [0 , 1] p has primarily followed two routes. One involves relaxing set counting as norm functions, as seen in the soft Jaccard loss [46, 49], the soft Dice loss [57], the soft Tversky loss [53] and the focal Tversky loss [1]. The other capitalizes on the fact that IoU is submodular [72], allowing for the application of the convex Lovasz extension of submodular functions. For instance, the Lovasz-Softmax loss [2], the Lovasz hinge loss [72] and the PixIoU loss [73]. Nevertheless, as IoU values are extended for predictions from { 0 , 1 } p to [0 , 1] p , the fact that labels can also fall within [0 , 1] p is often overlooked. As a result, these IoU losses are incompatible with soft labels.

Besides semantic segmentation, IoU is adopted across a wide spectrum of fields. Due to its discrete nature, its probabilistic extensions [56] have found use in object detection [19], medical imaging [16] and information retrieval [26, 43].

6 Discussion

6.1 How to tune the hyper-parameters of JML?

In this section, we delve into some critical hyper-parameters of JML. For a comprehensive list, please refer to the accompanying code.

mIoU D /mIoU I /mIoU C (default: 1.0/0.0/0.0): the weight of the loss to optimize mIoU D /mIoU I /mIoU C [65]. The appropriate choice is mainly dependent on the targeted evaluation metrics. Given that the prevailing metric is mIoU D (per-dataset mIoU), we recommend to set mIoU D to 1.0 and mIoU I /mIoU C to 0.0. However, it is imperative to acknowledge the inherent trade-offs when using JML to optimize different metrics [65].

alpha/beta (default: 1.0/1.0): the coefficient of false positives/negatives in the Tversky loss. For instance, when alpha and beta are both set to 1.0, the configuration corresponds to JML. Conversely, an alpha and beta value of 0.5 each leads to DML [66]. When the evaluation metric is IoU, JML is advised, while it is the Dice score, DML is more appropriate. The general Tversky loss is useful when separate weights are required for false positives/negatives.

active_classes_mode (default: PRESENT for hard labels and ALL for soft labels): the mode to compute active classes. With hard labels, it is suggested to use PRESENT , since the loss aligns more effectively with the evaluation metric, especially when (i) the dataset contains a large number of classes (e.g. ADE20K), and/or (ii) evaluated with fine-grained mIoUs [65]. With soft labels, the optimal choice varies based on specific applications (see Table 7).

6.2 How to use JML?

Combine JML with CE. Our findings suggest that incorporating CE, particularly during the initial stage of training, can expedite convergence (as illustrated in Figure 1). Consequently, relying solely on JML is not recommended. For a discussion on the balancing weights, please consult Section 4.1.

Tune the hyper-parameters of JML. We have endeavored to minimize the necessity of extensive tuning. In the majority of scenarios, default hyper-parameter values (as illustrated above) should suffice. However, some situations might benefit from further refinement. In particular, pay attention to active_classes_mode when dealing with soft labels.

Tune the hyper-parameters of training settings. Current training recipes have been heavily optimized for CE. Although we find JML generally aligns with these hyper-parameters, due diligence is required. Notably, one of the advantages of JML is its ability to speed up convergence in the later training phase (as depicted in Figure 1). As a consequence, training with a combination of CE and JML often converges in considerably fewer epochs compared to training with CE alone. Please refer to [65] for a comparison of the number of epochs with those in MMSegmentation.

Be careful with distributed training. In our concurrent work [65], we observe that the loss presents trade-offs when optimizing different mIoU variants. If the per-GPU batch size is small (which is often the case) and the loss is computed on each GPU independently, the loss may inadvertently optimize for mIoU I (per-image mIoU). This can lead to suboptimal outcomes when evaluated with mIoU D (per-dataset mIoU).

Be careful with extremely class-imbalance. Aligned with mIoU, JML is designed to compute the loss on a per-class basis and subsequently average these individual losses. In preliminary experiments on some medical datasets, we identified instances of severe class imbalances. In such scenarios, the class-wise loss may inadvertently amplify the significance of underrepresented classes, potentially disturb the training process. To address this, one might consider (i) oversampling the minority classes, and/or (ii) adopting a class-agnostic loss computation, where the intersection and union are calculated over all pixels.

7 Limitation

In both this study and our subsequent work [66], the focus is to extend the losses in the field of image segmentation. It would be intriguing to apply these losses to other tasks, such as long-tailed classification [34]. Moreover, although we adopt these losses in the label space, they present potential in quantifying the similarity between two feature vectors [24], potentially serving as an alternative to the L p norm or cosine similarity.

8 Conclusion

This paper is driven by the observation that current IoU losses fall short when dealing with soft labels, which substantially limits their adaptability to crucial training techniques. To address this limitation, we introduce the Jaccard metric losses (JMLs). While these losses are identical to the soft Jaccard loss in a conventional hard-label setting, they offer full compatibility with soft labels.

Our results demonstrate that integrating JMLs with label smoothing, knowledge distillation, and semi-supervised learning leads to notable improvements in both accuracy and calibration. This is consistent across a spectrum of datasets and network architectures, encompassing classic CNNs as well as recent vision transformers. Remarkably, the proposed methods, which are simple and solely rely on soft labels, surpass state-of-the-art segmentation knowledge distillation and semi-supervised learning techniques by a significant margin.

In our follow-up study [66], we delve into the extensions of various other losses, including the soft Dice loss, soft Tversky loss, and focal Tversky loss. Given their equivalence to their original counterparts in a standard hard-label context and their enhanced compatibility with soft labels, we recommend to replace the existing implementations with ours.

References

Nabila Abraham and Naimul Mefraz Khan. A Novel Focal Tversky Loss Function with Improved Attention U-Net for Lesion Segmentation. ISBI , 2019.

Maxim Berman, Amal Rannen Triki, and Matthew B. Blaschko. The Lovasz-softmax loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks. CVPR , 2018.

Jeroen Bertels, David Robben, Dirk Vandermeulen, and Paul Suetens. Theoretical analysis and experimental validation of volume bias of soft Dice optimized segmentation maps in the context of inherent uncertainty. MIA , 2021.

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. MixMatch: A Holistic Approach to Semi-Supervised Learning. NeurIPS , 2019.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Detection with Transformers. ECCV , 2020.

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking Atrous convolution for semantic image segmentation. arXiv , 2017.

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with Atrous separable convolution for semantic image segmentation. ECCV , 2018.

Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention Mask Transformer for Universal Image Segmentation. CVPR , 2022.

Bowen Cheng, Alexander G Schwing, and Alexander Kirillov. Per-Pixel Classification is Not All You Need for Semantic Segmentation. NeurIPS , 2021.

MMSegmentation Contributors. MMSegmentation: OpenMMLab Semantic Segmentation Toolbox and Benchmark, 2020.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR , 2016.

Ilke Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar. DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images. CVPR Workshop , 2018.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR , 2009.

Michel Marie Deza and Elena Deza. Encyclopedia of Distances . Springer, 2009.

Zhipeng Ding, Xu Han, Peirong Liu, and Marc Niethammer. Local Temperature Scaling for Probability Calibration. ICCV , 2021.

Reuben Dorent, Thomas Booth, Wenqi Li, Carole H. Sudre, Sina Kafiabadi, Jorge Cardoso, Sebastien Ourselin, and Tom Vercauteren. Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasets. MIA , 2020.

Tom Eelbode, Jeroen Bertels, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, and Matthew B. Blaschko. Optimization for medical image segmentation: Theory and Practice When Evaluating With Dice Score or Jaccard Index. TMI , 2020.

Mark Everingham, Luc Van Gool, Christopher K I Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. IJCV , 2009.

Di Feng, Zining Wang, Yiyang Zhou, Lars Rosenbaum, Fabian Timm, Klaus Dietmayer, Masayoshi Tomizuka, and Wei Zhan. Labels are Not Perfect: Inferring Spatial Uncertainty in Object Detection. TITS , 2022.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML , 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR , 2016.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NeurIPS Workshop , 2015.

Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge Distillation from A Stronger Teacher. NeurIPS , 2022.

Tao Huang, Yuan Zhang, Shan You, Fei Wang, Chen Qian, Jian Cao, and Chang Xu. Masked Distillation with Receptive Tokens. ICLR , 2023.

Pavel Iakubovskii. Segmentation models pytorch, 2019.

Sergey Ioffe. Improved Consistent Sampling, Weighted Minhash and L1 Sketching. ICDM , 2010.

Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Hein. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods , 2021.

Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar. Panoptic Segmentation. CVPR , 2019.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment Anything. ICCV , 2023.

Sven Kosub. A note on the triangle inequality for the Jaccard distance. PRL , 2019.

Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings. ICML , 2018.

Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang, and Xizhou Zhu. AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks. CVPR , 2022.

Hao Li, Chenxin Tao, Xizhou Zhu, Xiaogang Wang, Gao Huang, and Jifeng Dai. Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentation. ICLR , 2021.

Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. Dice Loss for Data-imbalanced NLP Tasks. ACL , 2020.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection. TPAMI , 2018.

Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured Knowledge Distillation for Semantic Segmentation. CVPR , 2019.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. CVPR , 2022.

Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. ICLR , 2019.

Xingchen Ma and Matthew B Blaschko. Meta-Cal: Well-controlled Post-hoc Calibration by Ranking. ICML , 2021.

Lena Maier-Hein et al. Metrics Reloaded: Recommendations for image analysis validation. arXiv , 2023.

Alireza Mehrtash, William M. Wells, Clare M. Tempany, Purang Abolmaesumi, and Tina Kapur. Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation. TMI , 2020.

Hossein Mobahi, Mehrdad Farajtabar, and Peter L Bartlett. Self-Distillation Amplifies Regularization in Hilbert Space. NeurIPS , 2020.

Ryan Moulton and Yunjiang Jiang. Maximally Consistent Sampling and the Jaccard Index of Probability Distributions. ICDM Workshop , 2018.

Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When Does Label Smoothing Help? NeurIPS , 2019.

Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. Measuring Calibration in Deep Learning. CVPR Workshop , 2019.

Sebastian Nowozin. Optimal Decisions from Probabilistic Models: the Intersection-over-Union Case. CVPR , 2014.

Teodora Popordanoska, Jeroen Bertels, Dirk Vandermeulen, Frederik Maes, and Matthew B. Blaschko. On the relationship between calibrated predictors and unbiased volume estimation. MICCAI , 2021.

Teodora Popordanoska, Raphael Sayer, and Matthew B Blaschko. A Consistent and Differentiable Lp Canonical Calibration Error Estimator. NeurIPS , 2022.

MdAtiqur Rahman and Yang Wang. Optimizing intersection-over-union in deep neural networks for image segmentation. ISVC , 2016.

Alexander Rakhlin, Alex Davydow, and Sergey Nikolenko. Land Cover Classification from Satellite Imagery With U-Net and Lovász-Softmax Loss. CVPR Workshop , 2018.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI , 2015.

Axel-Jan Rousseau, Thijs Becker, Jeroen Bertels, Matthew B. Blaschko, and Dirk Valkenborg. Post training uncertainty calibration of deep networks for medical image segmentation. ISBI , 2021.

Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and Ali Gholipour. Tversky loss function for image segmentation using 3D fully convolutional deep networks. MICCAI Workshop , 2017.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR , 2018.

Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise Knowledge Distillation for Dense Prediction. ICCV , 2021.

H. Späth. The minisum location problem for the Jaccard metric. OR Spektrum , 1981.

Carole H Sudre, Wenqi Li, Tom Vercauteren, Sébastien Ourselin, and M Jorge Cardoso. Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations. MICCAI Workshop , 2017.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. CVPR , 2016.

Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and Improving Knowledge Distillation. arXiv , 2020.

Vladimir N Vapnik. The Nature of Statistical Learning Theory . Springer, 1995.

Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers. CVPR , 2021.

Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, Xiaogang Wang, and Yu Qiao. InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions. CVPR , 2023.

Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels. CVPR , 2022.

Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class Feature Variation Distillation for Semantic Segmentation. ECCV , 2020.

Zifu Wang, Maxim Berman, Amal Rannen-Triki, Philip H.S. Torr, Devis Tuia, Tinne Tuytelaars, Luc Van Gool, Jiaqian Yu, and Matthew B. Blaschko. Revisiting Evaluation Metrics for Semantic Segmentation: Optimization and Evaluation of Fine-grained Intersection over Union. NeurIPS , 2023.

Zifu Wang, Teodora Popordanoska, Jeroen Bertels, Robin Lemmens, and Matthew B. Blaschko. Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels. MICCAI , 2023.

Ross Wightman. Pytorch Image Models, 2019.

Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders. CVPR , 2023.

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. NeurIPS , 2021.

Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. CrossImage Relational Knowledge Distillation for Semantic Segmentation. CVPR , 2022.

Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, and Yinghuan Shi. Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation. CVPR , 2023.

Jiaqian Yu and Matthew B. Blaschko. The Lovász Hinge: A Novel Convex Surrogate for Submodular Losses. TPAMI , 2018.

Jiaqian Yu, Jingtao Xu, Yiwei Chen, Weiming Li, Qiang Wang, Byung In Yoo, and Jae-Joon Han. Learning generalized intersection over union for dense pixelwise prediction. ICML , 2021.

Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation. CVPR , 2022.

Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hatwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means Mask Transformer. ECCV , 2022.

Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting Knowledge Distillation via Label Smoothing Regularization. CVPR , 2020.

Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled Knowledge Distillation. CVPR , 2022.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid Scene Parsing Network. CVPR , 2017.

Zhen Zhao, Sifan Long, Jimin Pi, Jingdong Wang, and Luping Zhou. Instance-specific and Model-adaptive Supervision for Semi-supervised Semantic Segmentation. CVPR , 2023.

Zhen Zhao, Lihe Yang, Sifan Long, Jimin Pi, Luping Zhou, and Jingdong Wang. Augmentation Matters: A Simple-yet-Effective Approach to Semi-supervised Semantic Segmentation. CVPR , 2023.

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene Parsing Through ADE20K Dataset. CVPR , 2017.

Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective. ICLR , 2021.

A Architectures

Our study includes 13 models, comprising both classic CNNs and recent vision transformers. Details of each model are presented in Table 9. FLOPs computations are based on an input size of 512 × 1024 . Inference latency measurements are conducted with the same input size on a NVIDIA A100. We estimate the training memory requirements using a ground-truth size of 8 × 19 × 512 × 1024 ( batch_size , num_classes , H , W ), also on a NVIDIA A100.

B Training Details

By default, we adopted the training details outlined in [70, 23, 24], except for the reduction of the batch size to 8. In particular, we utilize SGD with a weight decay of 0.0005 and a momentum of 0.9. The initial learning rate is 0.01, and is decayed according to (1 -iter total iters ) 0 . 9 . The number of iterations is 40K for Cityscapes [11] and PASCAL VOC [18], 10K for DeepGlobe Land [12]. The crop size is 512 × 1024 for Cityscapes [11], 512 × 512 for PASCAL VOC [18] and DeepGlobe Land [12]. In Table 4, for a fair comparison with the baselines, we increase the batch size of JML-KD to 16 in Cityscapes experiments, and all other parameters retain at their default values.

For experiments with vision transformers, we adhered to the training recipes in [10]. Specifically, we use AdamW [38] with a weight decay of 0.01. We begin with an initial learning rate of 0.00006, and is decayed by (1 -iter total iters ) 1 . The number of iterations is 40K and the crop size is 512 × 512 for ADE20K [81]. The batch size is 8.

For semi-supervised learning, we follow training details in [80]. In particular, we use SGD with a weight decay of 0.0001 and a momentum of 0.9. The initial learning rate is 0.001, and is decayed according to (1 -iter total iters ) 0 . 9 . The number of epochs is 80 and the crop size is 512 × 512 for PASCAL VOC [18]. The batch size is 16.

For JML experiments, we set µ L /µ T /ν L /ν T /η S /η U /θ S /θ U to 0.5 and λ CE /λ JML to 0.25/0.75.

C More analysis of ∆ SJL ,L 1

For x, y ∈ [0 , 1] p , let us rewrite ∆ SJL ,L 1 for a particular pixel i :

̸

̸

̸

̸

Assume ab = 0 and predictions are independent from each other. Take the derivative with respect to x i :

such that

̸

The numerator has two roots:

It is easy to see that r 1 ≤ 0 ≤ r 2 ≤ 1 . Therefore we have

That is, ∆ SJL ,L 1 will push x i towards either 1 if y i > r 2 , or 0 if y i < r 2 , rather than making it close to y i . In practice, predictions are not independent from each other, hence the exact behavior of ∆ SJL ,L 1 is a complex process. However, the above analysis still provides an insight of how ∆ SJL ,L 1 will react to soft labels.

D The Lovasz-Softmax Loss

Based on the fact that IoU is submodular [72], the Lovasz-Softmax loss (LSL) [2] computes the convex Lovasz extension of IoU. Specifically, for each prediction x i ∈ [0 , 1] , mispredictions are computed as

The Lovasz extension can be applied such that

where g i ( m ) = ∆ IoU ( { π 1 , ..., π i } ) -∆ IoU ( { π 1 , ..., π i -1 } ) and π is a permutation ordering of m such that m π 1 ≥ ... ≥ m π p .

Loss functions that rely on the Lovasz extension, such as LSL [2], the Lovasz hinge loss [72] and the PixIoU loss [73] cannot handle soft labels because they need to compute g i ( m ) and it requires y ∈ { 0 , 1 } p .

E JMLs vs. other IoU Losses

In Table 10, we evaluate the performance of ∆ JML,1, ∆ JML,2, ∆ SJL ,L 1 , ∆ SJL ,L 2 and ∆ LSL on Cityscapes and PASCAL VOC using DL3-R18.

As elaborated in Appendix C, ∆ SJL ,L 1 is equivalent to ∆ JML in the context of hard labels. Nevertheless, with soft labels, ∆ SJL ,L 1 has a tendency to push predictions towards vertices, rather than optimizing for the ideal x = y scenario. Our experiments on PASCAL VOC reveal that models trained with ∆ SJL ,L 1 using soft labels are outperformed by their hard-label counterparts. In fact, predictions for multiple classes might be simultaneously pushed towards 1, creating a conflict and potentially destabilizing the training process.

Conversely, ∆ SJL ,L 2 is free from the aforementioned issue, as highlighted in Section 2.2. Nevertheless, it is found to be less effective than ∆ SJL ,L 1 when employed with hard labels, possibly due to its flattened behavior near the minimum [17]. Our empirical results further substantiate that it is surpassed by ∆ JML on Cityscapes and PASCAL VOC, for both hard and soft labels.

Lastly, ∆ LSL cannot take soft labels as input, given its reliance on the Lovasz extension. Our evaluations indicate that its performance is similar to that of ∆ JML with hard labels.

F Proof of Theorem 2.1

Proof. (i) ∆ JML,1 is a metric on [0 , 1] p . The proof was given in [56]. Here we provide a sketch for the proof of the triangle inequality. Recall the definition of ∆ JML,1:

Given a fixed point a ∈ [0 , 1] p , the key is to define a new function d ′ :

and show that if d is a metric on [0 , 1] p , then d ′ is also a metric on [0 , 1] p . The proof can be found in [56].

Substitute the definition of the L 1 norm as d and let a = 0 . Since the L 1 norm is a metric on [0 , 1] p , we can conclude that ∆ JML,1 is also a metric on [0 , 1] p .

(ii) ∆ JML,2 is a metric on [0 , 1] p . Conditions (i) - (iii) are obvious. To show the triangle inequality, note that

Note that if p ≥ q > 0 , r ≥ 0 , we have p p + r ≥ q q + r . The last inequality follows from ∥ a -b ∥ 1 + ∥ b -c ∥ 1 ≥ ∥ a -c ∥ 1 .

Similarly,

Hence

̸

(iii) ∆ SJL ,L 1 is not a metric on [0 , 1] p . For instance, if a = 0 . 5 , ∆ SJL ,L 1 ( a, a ) = 2 / 3 = 0 .

̸

(iv) ∆ SJL ,L 2 is not a metric on [0 , 1] p . For instance, if a = 0 . 8 , b = 0 . 4 , c = 0 . 2 , it does not satisfy the triangle inequality.

G Proof of Theorem 2.3

Proof. (i) ∀ x ∈ [0 , 1] p , y ∈ { 0 , 1 } p and x ∈ { 0 , 1 } p , y ∈ [0 , 1] p , ∆ SJL ,L 1 = ∆ JML,1 = ∆ JML,2. Due to symmetry, we only need to prove the first part. Note that

Combine these, we have

̸

̸

̸

(ii) ∃ x, y ∈ [0 , 1] p , ∆ SJL ,L 1 = ∆ JML,1 = ∆ JML,2. For instance, if x = 0 . 8 , y = 0 . 5 , ∆ SJL ,L 1 = ∆ JML,1 = ∆ JML,2.

H ∆ JML,1 vs. ∆ JML,2

Definition H.1 (Convex Closure) . The convex closure of a set function f : x ∈ { 0 , 1 } p → [0 , 1] is

The convex closure extends a set function that is only defined at the vertices { 0 , 1 } p to the whole hypercube [0 , 1] p by linearly interpolating the values at these vertices. We plot the loss value of ∆ JML,1 , ∆ JML,2 and the convex closure of ∆ SJL ,L 1 in Figure 2 when y = 0 . 5 . Note that ∆ JML,1 overlaps with the convex closure at [0 , 0 . 5] . We can see that ∆ JML,1 ≤ ∆ JML,2 and ∆ JML,1 is closer to the convex closure. Generally, we have

Theorem H.2. ∀ x, y ∈ [0 , 1] p , ∆ JML,1 ≤ ∆ JML,2.

Proof.

0.0

0.2

0.4

0.6

0.8

1.0

Prediction

0.0

0.2

0.4

0.6

0.8

1.0

Loss value

JML

, 1

JML

, 2

We can switch the element of x and y whenever x i ≤ y i for some i . Doing this, each term in the last inequality remains the same. Let us denote the new variable as x ′ , y ′ such that x ′ i ≥ y ′ i for all i . Hence

Theorem H.3. Given a set function l : { 0 , 1 } p → [0 , 1] and two concave functions f, g : [0 , 1] p → [0 , 1] . If ∀ x ∈ { 0 , 1 } p , l ( x ) = f ( x ) = g ( x ) and ∀ x ∈ [0 , 1] p , f ( x ) ≤ g ( x ) , then ∀ x ∈ [0 , 1] p , | f ( x ) -C l ( x ) | ≤ | g ( x ) -C l ( x ) | .

Proof. It suffices to show ∀ x ∈ [0 , 1] p , f ( x ) ≥ C l ( x ) . ∀ x ∈ [0 , 1] p , we have x = ∑ p i =1 α i x i such that α i ≥ 0 , ∑ p i =1 α i = 1 and x i ∈ { 0 , 1 } . Therefore

a

b

c

d

y

x

x

′

Although both ∆ JML,1 and ∆ JML,2 are not concave as Figure 2 shows, if we can divide the hypercube into several sub-spaces such that ∆ JML,1 and ∆ JML,2 are equal at the vertices and concave at each sub-space, then we can still apply Theorem H.3. However, the fact that both of them are piece-wise concave in 1D does not hold in higher dimension. Indeed, we can find a counterexample in 2D. Let y = [0 . 5 , 0 . 5] , x = [0 . 4087 , 0 . 7855] , x ′ = [0 . 6285 , 0 . 7551] . Both x and x ′ are in the sub-space yab and

How do they perform empirically? In Table 10, we compare ∆ JML,1 and ∆ JML,2 on Cityscapes and PASCAL VOC using DL3-R18. We find ∆ JML,1 is slightly superior to ∆ JML,2.

I Semi-supervised Learning

We focus on augmentation-based semi-supervised learning approaches [4, 80]. Broadly speaking, in their approaches, the data consists of both supervised and unsupervised images, and a model H is trained to minimize the following loss:

Here, A s and A u are data augmentations applied to labeled images x s and unlabeled images x u , respectively. L is the ground-truth label and T is usually the exponential moving averaging of the model weights.

In JML-SSL, we introduce additional supervision from JML:

Combining these yields

To further harness the power of soft labels, we follow the pipelines of AugSeg [80] and stack BLS on top of their augmentations: A s ( x s ) = BLS ( A s, AugSeg ( x s )) and A u ( x u ) = BLS ( A u, AugSeg ( x u )) .

Here, A s, AugSeg and A u, AugSeg denote augmentations applied to labeled and unlabeled images in AugSeg, respectively. We refer readers to their paper for more details.

As shown in Table 11, we achieve SOTA results on semi-supervised learning. In Table 12, we ablate the role of each loss term on PASCAL VOC using DL3+-R101.

J More Results on DeepGlobe Land

We provide additional results on DeepGlobe Land in Table 13. Furthermore, we present the outcomes of BLS using DL3-R50 and KD using DL3-R18 across 5 folds of DeepGlobe Land in Table 14. Given that DeepGlobe Land is a relatively small dataset, comprising only 803 images, results for each fold can vary significantly. Nevertheless, when observing the differences between CE and JML, the mean improvement exceeds twice the standard deviation.

K More Results on Calibration

CE is a proper scoring rule [20], while the soft Dice loss is not [3]. Although the soft Dice loss can significantly increase a model's segmentation performance, it can hurt model calibration [41, 3]. We observe the same phenomenon with JML and demonstrate a potential solution by training the model with soft labels as delineated in Section 3.2. Calibration of the model can be further improved using post-hoc [20, 15, 39, 47, 52] and trainable calibration methods [31, 48]. However, these methods extend beyond the scope of this paper.

Besides, we find that models trained with JML, despite having poorer top-class calibration as quantified by ECE, actually attain superior multi-class calibration as evaluated by the static calibration error (SCE) [45]. In Table 15, we report both ECE and SCE on Cityscapes and PASCAL VOC using DL3-R101 2 . It is crucial to highlight that models trained with soft labels may exhibit worse SCE. On one side, soft labels provide an additional regularization during training, which is advantageous for calibration [44]. However, on the flip side, soft labels can represent unrealistic distributions, potentially detrimental for multi-class calibration.

L Figures

Figure 4: The best ϵ and mIoU (%) for different k on PASCAL VOC using DL3-R18.

Figure 5: Effects of ϵ on PASCAL VOC using DL3-R101/50/18.

Figure 6: Code to compute active classes.

Figure 7: Qualitative results on Cityscapes.

Figure 8: Qualitative results on PASCAL VOC.

3

5

9

74.0

74.5

75.0

75.5

76.0

mIoU (%)

0.0

0.2

0.4

0.6

0.8

1.0

mIoU

0

0.35

0.40

0.45

0.50

0.55

0.60

0.65

79.5

80.0

80.5

81.0

81.5

82.0

82.5

mIoU (%)

(a) DL3-R101 mIoU (%)

0

0.35

0.40

0.45

0.50

0.55

0.60

0.65

2.5

3.0

3.5

4.0

4.5

ECE (%)

(b) DL3-R101 ECE (%)

0

0.35

0.40

0.45

0.50

0.55

0.60

0.65

78.5

79.0

79.5

80.0

80.5

mIoU (%)

(c) DL3-R50 mIoU (%)

0

0.35

0.40

0.45

0.50

0.55

0.60

0.65

3.0

3.5

4.0

4.5

ECE (%)

(d) DL3-R50 ECE (%)

0

0.35

0.40

0.45

0.50

0.55

0.60

0.65

73.5

74.0

74.5

75.0

75.5

76.0

76.5

mIoU (%)

(e) DL3-R18 mIoU (%)

0

0.35

0.40

0.45

0.50

0.55

0.60

0.65

3.5

4.0

4.5

5.0

5.5

ECE (%)

(f) DL3-R18 ECE (%)

Image and GT

Teacher

Student with KD

Student without KD

I

Image and GT

Teacher

Student with KD

Student without KD

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new loss function for semantic segmentation based on the IoU loss. The authors found that the vanilla IoU loss is incompatible with the soft labels when label smoothing and knowledge distillation training techniques are used. Experiments show that the proposed approach works well on four widely-used semantic segmentation datasets. Ablation experiments also show some insightful take-home message for the readers.\n\nSTRENGTHS:\n- The motivation of this paper is interesting. The authors observe an interesting problem of the vanilla IoU loss.\n\n- The novelty of this paper is clear. Improving the vanilla IoU loss is a meaningful topic for semantic segmentation.\n\n- The results are good. On four segmentation benchmarks, the proposed approach receives improvements.\n\nWEAKNESSES:\n- In the main paper, the authors only show results based on ConvNets. The results based on Transformers are provided in the supplementary materials. From the results, it can be seen that the improvement of the proposed approach on ConvNets are higher than that on Transformers. I think the authors should explain why this happens.\n\n- Some visualizations should be moved to the main paper. I think it is interesting to see how the segmentation results change when the proposed method is used.\n\n- I also would like to recommend the authors to add some failure case analysis to see what readers can further do for semantic segmentation.\n\n- I am also interested in the results when large-sized models are used, for instance, SegFormer-B5.\n\nQUESTIONS:\n- Some results using Transformers should be moved to the main paper though the improvement compared to the CNNs is not that significant.\n\n- It is also recommended to describe some works on semantic segmentation in the related work section.\n\n- Failure cases should be provided and discussed.\n\nLIMITATIONS:\nNot provided.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Jaccard Metric Losses (JMLs) as a flexible alternative to Intersection over Union (IoU) losses for optimizing the Jaccard index in semantic segmentation tasks. Unlike IoU losses, JMLs can effectively process soft labels, enabling the use of important training techniques like label smoothing, knowledge distillation, and semi-supervised learning.\n\nSTRENGTHS:\n1. Jaccard Metric Losses (JMLs) introduce a novel approach by mirroring the soft Jaccard loss in standard settings with hard labels while remaining compatible with soft labels. This flexibility is a significant contribution as it enables the incorporation of vital training techniques that rely on soft labels, such as label smoothing, knowledge distillation, and semi-supervised learning. This novel aspect makes JMLs an interesting and valuable addition to the field.\n2. The paper demonstrates good results by showcasing consistent improvements over the cross-entropy loss across multiple semantic segmentation datasets (Cityscapes, PASCAL VOC, ADE20K, DeepGlobe Land) and a variety of architectures, including both classic CNNs and recent vision transformers. \n3. The paper provides theoretical analysis of JMLs, which enhances the understanding of their properties and behavior.\n\nWEAKNESSES:\n1. The paper highlights IoU losses as the baseline and the main motivation for introducing JMLs. However, the lack of direct comparisons between IoU losses and JMLs in the experiments is a weakness. While certain comparisons may not be possible, such as in knowledge distillation (KD) where IoU loss baselines require teachers, it would have been beneficial to include comparative analyses that demonstrate the importance of adapting IoU losses to soft labels. This would have provided a stronger justification for the need and effectiveness of JMLs.\n2. Cross-entropy (CE) achieves better calibration performance in Tables 1 and 2. However, more discussion and explanation of this observation is needed. Understanding the reasons behind this disparity and addressing any potential limitations or trade-offs associated with JMLs' calibration performance would have added depth to the paper's analysis and interpretation.\n\nQUESTIONS:\nWhy CE group is much better than JML group in terms of calibration?\n\nLIMITATIONS:\nNeeds to determine hyperparameters like $\\epsilon$", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes JML (Jaccard Metric Losses) for segmentation. This builds up on previous IoU-based losses but with the added ability to process soft labels or probability distributions. JML losses can be further applied as follows:\nJML-BLS (Label smoothening Jaccard-based IoU loss, which is only computed at boundary pixels based on empirical analysis).\nJML-KD (Applying the soft JML loss to between student and teacher, as well as applying the soft JML loss between student and GT).\n\nSTRENGTHS:\nPresentation:\nThe reviewer appreciates authors' clarity of presentation of their concept. The paper is quite clear and tables are easy to understand. Mathematical notations have been thoroughly reviewed and hence the paper reads well.\n\nOriginality:\nAlthough the work takes inspiration from IoU loss, and is quite a simple update to the original IoU loss concept, the technical contribution is quite clear and holds up. There has been work on IoU losses before but they are not used widely in literature, hence, there is a scope for improvement in this area. This work is a beneficial building block to further research in IoU losses.\n\nTransparency:\nThe reviewer appreciates the code being made available.\n\nResults:\nThe JML loss seems to work on a wide range of datasets, and more importantly, backbones. There are 12 different architectures in the paper as well as supp, and the loss works for both CNNs as well as transformer based backbones. This is very interesting as it shows the capability of this loss to provide a high impact in our field.\n\nWEAKNESSES:\nThere is a lot of information in the appendices, which might be more beneficial than some of the things in the main paper. One such example is the experiment with transformer-based backbones.\n\nQUESTIONS:\n1. In tables 1-13, Did each experiment (row) start with the same set of pre-trained weights?\n2. How did you tune the hyper-parameters for JML losses (specifically loss weights)? How easy(or difficult) is it to obtain the results provided in the paper.\n3. Does adding JML ever reduce performance (with a higher weight, etc)\n4. Could you address limitations of this work\n\nLIMITATIONS:\nNo - The authors are suggested to address the limitations of this method (if any).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on making IoU losses flexible to process soft labels. Jaccard Metric Losses are presented that works the same as soft Jaccard loss with hard labels but it is also compatible with soft labels. Experiments are presented in several tasks: label smoothing, knowledge distillation, and semi-supervised learning. The results show improvement over baselines.\n\nSTRENGTHS:\n+ The paper focuses on addressing an important issue with IoU losses that they are not flexible to process soft labels\n+ Experiments on several tasks (i.e., label smoothing, knowledge distillation, and semi-supervised learning) shows promising results.\n\nWEAKNESSES:\nThere are several confusions regarding experimental results.\n\n--There are discrepancies between the results presented in Table 5 and Table 3 for Cityscapes with DL3-R18. Table 3 reports 77.91 ± 0.16 whereas Table 5 reports 76.68 ± 0.33 as the best performance. For VOC, the best results match in the tables (75.89 ± 0.29). However, there is confusion between JML-KD and JML-BLS. For VOC, the JML-KD results in Table 3 match JML-BLS results in Table 5.\n\n-- As shown in Table 3, in knowledge distillation proposed student achieves 78.14. This is a bit surprising as the Teacher DeepLabV3-R101 has a lower mIoU of 78.07 (as reported in DIST[21]). \n\n-- The reported performance in Table 1 and Table 2 do not compare with SOTA performance as shown in the papers. For example, the deeplabv3  paper with DL3-R101 model reports 81.3 mIoU in CityScapes and 85.7 mIoU in Pascal VOC. The presented proposed method results are lower compared to that and baselines are lower too. We see this for all the cases in Tables 1 and 2. I would suggest the authors use the proposed loss with the best-performing model and losses from the prior works to avoid any confusion.  To better convince the readers,  I think improving on the best results reported in prior works would be helpful.\n\nQUESTIONS:\nPlease address the concerns in the weaknesses section.\n\n-- The rebuttal addressed most of my concerns. Hence, I increase the score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
In this paper, the authors propose Jaccard Metric Losses (JMLs) for the training with soft labels in semantic segmentation. The authors analyze the limitations of previous IoU losses for soft labels, and propose JMLs to solve such limitations. Experiments for label smoothing, knowledge distillation and semi-supervised learning demonstrate the effectiveness of JMLs.

STRENGTHS:
1. It is interesting that using Jaccard-based losses to solve the soft label training problem.
2. This paper is well written and easy to read.
3. Experiments show large improvements for multiple architectures.

WEAKNESSES:
1. The models in experiments seem a bit old. It would be great to show the performance with more recent architectures, such as Mask2Former.
2. Only small models are compared in experiments. It is important to show the performance for larger models to demonstrate the scalability.

QUESTIONS:
Please solve my concerns in Weaknesses.

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07