SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[17] Tom Eelbode, Jeroen Bertels, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, and Matthew B. Blaschko. Optimization for medical image segmentation: Theory and Practice When Evaluating With Dice Score or Jaccard Index. TMI, 2020.
[18] Mark Everingham, Luc Van Gool, Christopher K I Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. *IJCV*, 2009.
[19] Di Feng, Zining Wang, Yiyang Zhou, Lars Rosenbaum, Fabian Timm, Klaus Dietmayer, Masayoshi Tomizuka, and Wei Zhan. Labels are Not Perfect: Inferring Spatial Uncertainty in Object Detection. TITS, 2022.
[20] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML, 2017.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[22] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NeurIPS Workshop, 2015.
[23] Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge Distillation from A Stronger Teacher. NeurIPS, 2022.
[24] Tao Huang, Yuan Zhang, Shan You, Fei Wang, Chen Qian, Jian Cao, and Chang Xu. Masked Distillation with Receptive Tokens. ICLR, 2023.
[25] Pavel Iakubovskii. Segmentation models pytorch, 2019.
[26] Sergey Ioffe. Improved Consistent Sampling, Weighted Minhash and L1 Sketching. ICDM, 2010.
[27] Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Hein. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 2021.
[28] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar. Panoptic Segmentation. CVPR, 2019.
[29] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment Anything. ICCV, 2023.
[30] Sven Kosub. A note on the triangle inequality for the Jaccard distance. PRL, 2019.
[31] Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings. ICML, 2018.
[32] Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang, and Xizhou Zhu. AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks. CVPR, 2022.
[33] Hao Li, Chenxin Tao, Xizhou Zhu, Xiaogang Wang, Gao Huang, and Jifeng Dai. Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentation. ICLR, 2021.
[34] Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. Dice Loss for Data-imbalanced NLP Tasks. ACL, 2020.
[35] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dólar. Focal Loss for Dense Object Detection. TPAMI, 2018.
[36] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured Knowledge Distillation for Semantic Segmentation. CVPR, 2019.
[37] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. CVPR, 2022.
[38] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. *ICLR*, 2019.
[39] Xingchen Ma and Matthew B Blaschko. Meta-Cal: Well-controlled Post-hoc Calibration by Ranking. ICML, 2021.
[40] Lena Maier-Hein et al. Metrics Reloaded: Recommendations for image analysis validation. arXiv, 2023.
[41] Alireza Mehrtash, William M. Wells, Clare M. Tempany, Purang Abolmaesumi, and Tina Kapur. Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation. TMI, 2020.
[42] Hossein Mobahi, Mehrdad Farajtabar, and Peter L Bartlett. Self-Distillation Amplifies Regularization in Hilbert Space. NeurIPS, 2020.
[43] Ryan Moulton and Yunjiang Jiang. Maximally Consistent Sampling and the Jaccard Index of Probability Distributions. ICDM Workshop, 2018.
[44] Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When Does Label Smoothing Help? NeurIPS, 2019.
[45] Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. Measuring Calibration in Deep Learning. CVPR Workshop, 2019.
[46] Sebastian Nowozin. Optimal Decisions from Probabilistic Models: the Intersection-over-Union Case. CVPR, 2014.
[47] Teodora Popordanoska, Jeroen Bertels, Dirk Vandermeulen, Frederik Maes, and Matthew B. Blaschko. On the relationship between calibrated predictors and unbiased volume estimation. MICCAI, 2021.
[48] Teodora Popordanoska, Raphael Sayer, and Matthew B Blaschko. A Consistent and Differentiable Lp Canonical Calibration Error Estimator. NeurIPS, 2022.
[49] Md Atiqur Rahman and Yang Wang. Optimizing intersection-over-union in deep neural networks for image segmentation. ISVC, 2016.
[50] Alexander Rakhlin, Alex Davydow, and Sergey Nikolenko. Land Cover Classification from Satellite Imagery With U-Net and Lovász-Softmax Loss. CVPR Workshop, 2018.
[51] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI, 2015.
[52] Axel-Jan Rousseau, Thijs Becker, Jeroen Bertels, Matthew B. Blaschko, and Dirk Valkenberg. Post training uncertainty calibration of deep networks for medical image segmentation. ISBI, 2021.
[53] Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and Ali Gholipour. Tversky loss function for image segmentation using 3D fully convolutional deep networks. MICCAI Workshop, 2017.
[54] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR, 2018.
[55] Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise Knowledge Distillation for Dense Prediction. ICCV, 2021.
[56] H. Späth. The minisum location problem for the Jaccard metric. OR Spektrum, 1981.
[57] Carole H Sudre, Wenqi Li, Tom Vercauteren, Sébastien Ourselin, and M Jorge Cardoso. Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations. MICCAI Workshop, 2017.
[58] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. CVPR, 2016.
[79] Zhen Zhao, Sifan Long, Jimin Pi, Jingdong Wang, and Luping Zhou. Instance-specific and Model-adaptive Supervision for Semi-supervised Semantic Segmentation. CVPR, 2023.
[80] Zhen Zhao, Lihe Yang, Sifan Long, Jimin Pi, Luping Zhou, and Jingdong Wang. Augmentation Matters: A Simple-yet-Effective Approach to Semi-supervised Semantic Segmentation. CVPR, 2023.
[81] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene Parsing Through ADE20K Dataset. CVPR, 2017.
[82] Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective. ICLR, 2021.
A Architectures
Our study includes 13 models, comprising both classic CNNs and recent vision transformers. Details of each model are presented in Table 9. FLOPs computations are based on an input size of  $512 \times 1024$ . Inference latency measurements are conducted with the same input size on a NVIDIA A100. We estimate the training memory requirements using a ground-truth size of  $8 \times 19 \times 512 \times 1024$  (batch_size, num_classeses, H, W), also on a NVIDIA A100.
Table 9: Details of each model.
Method	Backbone	#Params (M)	FLOPs (G)	Latency (ms)	Memory (GB)
DeepLabV3+ [7]	ResNet101 [21]	62.58	528.32	26.47	42.68
DeepLabV3 [6]	ResNet101 [21]	87.10	714.73	26.20	33.99
DeepLabV3 [6]	ResNet50 [21]	68.11	559.14	17.48	21.02
DeepLabV3 [6]	ResNet18 [21]	13.98	121.29	4.93	6.43
DeepLabV3 [6]	MobileNetV2 [54]	3.79	31.68	5.89	16.74
PSPNet [6]	ResNet18 [21]	12.79	109.90	4.82	6.26
UNet [51]	ResNet50 [21]	76.07	409.35	18.09	24.76
UNet [51]	ResNet18 [21]	14.49	46.42	6.15	7.76
UNet [51]	MobileNetV2 [54]	4.06	22.63	8.15	11.16
SegFormer [69]	MiTB5 [69]	83.77	198.37	43.32	41.99
SegFormer [69]	MiTB3 [69]	46.40	124.52	26.15	28.29
SegFormer [69]	MiTB1 [69]	15.48	63.85	11.07	15.99
SegFormer [69]	MiTB0 [69]	5.01	45.46	8.98	13.45
By default, we adopted the training details outlined in [70, 23, 24], except for the reduction of the batch size to 8. In particular, we utilize SGD with a weight decay of 0.0005 and a momentum of 0.9. The initial learning rate is 0.01, and is decayed according to  $(1 - \frac{\text{iter}}{\text{total iters}})^{0.9}$ . The number of iterations is  $40\mathrm{K}$  for Cityscapes [11] and PASCAL VOC [18],  $10\mathrm{K}$  for DeepGlobe Land [12]. The crop size is  $512 \times 1024$  for Cityscapes [11],  $512 \times 512$  for PASCAL VOC [18] and DeepGlobe Land [12]. In Table 4, for a fair comparison with the baselines, we increase the batch size of JML-KD to 16 in Cityscapes experiments, and all other parameters retain at their default values.
For experiments with vision transformers, we adhered to the training recipes in [10]. Specifically, we use AdamW [38] with a weight decay of 0.01. We begin with an initial learning rate of 0.00006, and is decayed by  $(1 - \frac{\mathrm{iter}}{\mathrm{total~iters}})^{1}$ . The number of iterations is 40K and the crop size is  $512 \times 512$  for ADE20K [81]. The batch size is 8.
For semi-supervised learning, we follow training details in [80]. In particular, we use SGD with a weight decay of 0.0001 and a momentum of 0.9. The initial learning rate is 0.001, and is decayed according to  $(1 - \frac{\mathrm{iter}}{\mathrm{total~iters}})^{0.9}$ . The number of epochs is 80 and the crop size is  $512 \times 512$  for PASCAL VOC [18]. The batch size is 16.
For JML experiments, we set  $\mu_{\mathrm{L}} / \mu_{\mathrm{T}} / \nu_{\mathrm{L}} / \nu_{\mathrm{T}} / \eta_{\mathrm{S}} / \eta_{\mathrm{U}} / \theta_{\mathrm{S}} / \theta_{\mathrm{U}}$  to 0.5 and  $\lambda_{\mathrm{CE}} / \lambda_{\mathrm{JML}}$  to 0.25/0.75.
C More analysis of  $\overline{\Delta}_{\mathrm{SJL},L^1}$
For  $x, y \in [0,1]^p$ , let us rewrite  $\overline{\Delta}_{\mathrm{SJL}, L^1}$  for a particular pixel  $i$ :
$$
\overline {{\Delta}} _ {\mathrm {S J L}, L ^ {1}} = 1 - \frac {x _ {i} y _ {i} + b}{x _ {i} + y _ {i} - x _ {i} y _ {i} + a} \tag {14}
$$
$$
a = \sum_ {j \neq i} x _ {j} + \sum_ {j \neq i} y _ {j} - \sum_ {j \neq i} x _ {j} y _ {j}, \tag {15}
$$
$$
b = \sum_ {j \neq i} x _ {j} y _ {j}. \tag {16}
$$
Assume  $ab \neq 0$  and predictions are independent from each other. Take the derivative with respect to  $x_{i}$ :
$$
\frac {\mathrm {d} \overline {{\Delta}} _ {\mathrm {S J L} , L ^ {1}}}{\mathrm {d} x _ {i}} = - \frac {y _ {i} ^ {2} + (a + b) y _ {i} - b}{(x _ {i} + y _ {i} - x _ {i} y _ {i} + a) ^ {2}}. \tag {17}
$$
The numerator has two roots:
$$
r _ {1} = \frac {- (a + b) - \sqrt {(a + b) ^ {2} + 4 b}}{2}, \tag {18}
$$
$$
r _ {2} = \frac {- (a + b) + \sqrt {(a + b) ^ {2} + 4 b}}{2}. \tag {19}
$$
It is easy to see that  $r_1 \leq 0 \leq r_2 \leq 1$ . Therefore we have
$$
\frac {\mathrm {d} \overline {{\Delta}} _ {\mathrm {S J L} , L ^ {1}}}{\mathrm {d} x _ {i}} \left\{ \begin{array}{l l} \leq 0 & \text {i f} y _ {i} \geq r _ {2}, \\ > 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {20}
$$
That is,  $\overline{\Delta}_{\mathrm{SJL},L^1}$  will push  $x_{i}$  towards either 1 if  $y_{i} > r_{2}$ , or 0 if  $y_{i} < r_{2}$ , rather than making it close to  $y_{i}$ . In practice, predictions are not independent from each other, hence the exact behavior of  $\overline{\Delta}_{\mathrm{SJL},L^1}$  is a complex process. However, the above analysis still provides an insight of how  $\overline{\Delta}_{\mathrm{SJL},L^1}$  will react to soft labels.
D The Lovasz-Softmax Loss
Based on the fact that IoU is submodular [72], the Lovasz-Softmax loss (LSL) [2] computes the convex Lovasz extension of IoU. Specifically, for each prediction  $x_{i} \in [0,1]$ , mispredictions are computed as
$$
m _ {i} = \left\{ \begin{array}{l l} 1 - x _ {i} & \text {i f} y _ {i} = 1, \\ x _ {i} & \text {o t h e r w i s e .} \end{array} \right. \tag {21}
$$
The Lovasz extension can be applied such that
$$
\overline {{\Delta}} _ {\mathrm {L S L}}: m \in [ 0, 1 ] ^ {p}, y \in \{0, 1 \} ^ {p} \mapsto \sum_ {i = 1} ^ {p} m _ {i} g _ {i} (m)
$$
where  $g_{i}(m) = \Delta_{\mathrm{IoU}}(\{\pi_{1},\dots,\pi_{i}\}) - \Delta_{\mathrm{IoU}}(\{\pi_{1},\dots,\pi_{i-1}\})$  and  $\pi$  is a permutation ordering of  $m$  such that  $m_{\pi_1} \geq \ldots \geq m_{\pi_p}$ .
Loss functions that rely on the Lovasz extension, such as LSL [2], the Lovasz hinge loss [72] and the PixIoU loss [73] cannot handle soft labels because they need to compute  $g_{i}(m)$  and it requires  $y \in \{0,1\}^{p}$ .
E JMLs vs. other IoU Losses
In Table 10, we evaluate the performance of  $\overline{\Delta}_{\mathrm{JML},1}$ ,  $\overline{\Delta}_{\mathrm{JML},2}$ ,  $\overline{\Delta}_{\mathrm{SJL},L^1}$ ,  $\overline{\Delta}_{\mathrm{SJL},L^2}$  and  $\overline{\Delta}_{\mathrm{LSL}}$  on Cityscapes and PASCAL VOC using DL3-R18.
As elaborated in Appendix C,  $\overline{\Delta}_{\mathrm{SJL},L^1}$  is equivalent to  $\overline{\Delta}_{\mathrm{JML}}$  in the context of hard labels. Nevertheless, with soft labels,  $\overline{\Delta}_{\mathrm{SJL},L^1}$  has a tendency to push predictions towards vertices, rather than optimizing for the ideal  $x = y$  scenario. Our experiments on PASCAL VOC reveal that models trained with  $\overline{\Delta}_{\mathrm{SJL},L^1}$  using soft labels are outperformed by their hard-label counterparts. In fact, predictions for multiple classes might be simultaneously pushed towards 1, creating a conflict and potentially destabilizing the training process.
Conversely,  $\overline{\Delta}_{\mathrm{SJL},L^2}$  is free from the aforementioned issue, as highlighted in Section 2.2. Nevertheless, it is found to be less effective than  $\overline{\Delta}_{\mathrm{SJL},L^1}$  when employed with hard labels, possibly due to its flattened behavior near the minimum [17]. Our empirical results further substantiate that it is surpassed by  $\overline{\Delta}_{\mathrm{JML}}$  on Cityscapes and PASCAL VOC, for both hard and soft labels.
Lastly,  $\overline{\Delta}_{\mathrm{LSL}}$  cannot take soft labels as input, given its reliance on the Lovasz extension. Our evaluations indicate that its performance is similar to that of  $\overline{\Delta}_{\mathrm{JML}}$  with hard labels.
Table 10: Comparing  ${\bar{\Delta }}_{\mathrm{{JML}},1}$  ,  ${\bar{\Delta }}_{\mathrm{{JML}},2}$  ,  ${\bar{\Delta }}_{\mathrm{{SIL}},L1}$  ,  ${\bar{\Delta }}_{\mathrm{{SIL}},L2}$  and  ${\bar{\Delta }}_{\mathrm{{LSL}}}$  on Cityscapes and PASCAL VOC using DL3-R18. All results are mIoU (%). Red: the best in a column.
Dataset	Loss	Hard	BLS	KD
CS	ΔJML,1	75.55 ± 0.13	76.26 ± 0.17	76.68 ± 0.33
ΔJML,2	75.55 ± 0.13	76.31 ± 0.23	76.45 ± 0.26
ΔSJL,L1	75.55 ± 0.13	75.78 ± 0.34	75.83 ± 0.43
ΔSJL,L2	75.28 ± 0.26	75.51 ± 0.38	75.87 ± 0.52
ΔLSL	75.53 ± 0.26	-	-
VOC	ΔJML,1	74.42 ± 0.52	75.60 ± 0.24	75.89 ± 0.29
ΔJML,2	74.42 ± 0.52	75.31 ± 0.31	75.49 ± 0.35
ΔSJL,L1	74.42 ± 0.52	74.13 ± 0.20	74.24 ± 0.61
ΔSJL,L2	73.72 ± 0.30	74.87 ± 0.38	74.83 ± 0.32
ΔLSL	74.45 ± 0.36	-	-
F Proof of Theorem 2.1
Proof. (i)  $\overline{\Delta}_{\mathrm{JML,1}}$  is a metric on  $[0,1]^p$ . The proof was given in [56]. Here we provide a sketch for the proof of the triangle inequality. Recall the definition of  $\overline{\Delta}_{\mathrm{JML,1}}$ :
$$
\bar {\Delta} _ {\mathrm {J M L}, 1} = 1 - \frac {\| x \| _ {1} + \| y \| _ {1} - \| x - y \| _ {1}}{\| x \| _ {1} + \| y \| _ {1} + \| x - y \| _ {1}} = \frac {2 \| x - y \| _ {1}}{\| x \| _ {1} + \| y \| _ {1} + \| x - y \| _ {1}}. \tag {22}
$$
Given a fixed point  $a \in [0,1]^p$ , the key is to define a new function  $d'$ :
$$
d ^ {\prime} (x, y) = \left\{ \begin{array}{l l} 0 & \text {i f} x = y = a, \\ \frac {d (x , y)}{d (x , a) + d (y , a) + d (x , y)} & \text {o t h e r w i s e}; \end{array} \right. \tag {23}
$$
and show that if  $d$  is a metric on  $[0,1]^p$ , then  $d'$  is also a metric on  $[0,1]^p$ . The proof can be found in [56].
Substitute the definition of the  $L^1$  norm as  $d$  and let  $a = 0$ . Since the  $L^1$  norm is a metric on  $[0,1]^p$ , we can conclude that  $\overline{\Delta}_{\mathrm{JML},1}$  is also a metric on  $[0,1]^p$ .
(ii)  $\overline{\Delta}_{\mathrm{JML},2}$  is a metric on  $[0,1]^p$ . Conditions (i) - (iii) are obvious. To show the triangle inequality, note that
$$
\begin{array}{l} \langle b, c \rangle = \sum_ {i} b _ {i} c _ {i} (24) \\ = \sum_ {i} \left(\left(b _ {i} - a _ {i}\right) c _ {i} + a _ {i} c _ {i}\right) (25) \\ \leq \sum_ {i} \left| b _ {i} - a _ {i} \right| + \sum_ {i} a _ {i} c _ {i} (26) \\ = \| a - b \| _ {1} + \langle a, c \rangle . (27) \\ \end{array}
$$
$$
\langle a, b \rangle \leq \| b - c \| _ {1} + \langle a, c \rangle . \tag {28}
$$
$$
\begin{array}{l} \bar {\Delta} _ {\mathrm {J M L}, 2} (a, b) + \bar {\Delta} _ {\mathrm {J M L}, 2} (b, c) (29) \\ = \frac {\| a - b \| _ {1}}{\| a - b \| _ {1} + \langle a , b \rangle} + \frac {\| b - c \| _ {1}}{\| b - c \| _ {1} + \langle b , c \rangle} (30) \\ \geq \frac {\left\| a - b \right\| _ {1}}{\left\| a - b \right\| _ {1} + \left\| b - c \right\| _ {1} + \langle a , c \rangle} + \frac {\left\| b - c \right\| _ {1}}{\left\| a - b \right\| _ {1} + \left\| b - c \right\| _ {1} + \langle a , c \rangle} (31) \\ = \frac {\| a - b \| _ {1} + \| b - c \| _ {1}}{\| a - b \| _ {1} + \| b - c \| _ {1} + \langle a , c \rangle} (32) \\ \geq \frac {\left\| a - c \right\| _ {1}}{\left\| a - c \right\| _ {1} + \langle a , c \rangle} (33) \\ = \bar {\Delta} _ {\mathrm {J M L}, 2} (a, c). (34) \\ \end{array}
$$
Note that if  $p \geq q > 0, r \geq 0$ , we have  $\frac{p}{p + r} \geq \frac{q}{q + r}$ . The last inequality follows from  $\| a - b \|_1 + \| b - c \|_1 \geq \| a - c \|_1$ .
(iii)  $\overline{\Delta}_{\mathrm{SJ},L^1}$  is not a metric on  $[0,1]^p$ . For instance, if  $a = 0.5$ ,  $\overline{\Delta}_{\mathrm{SJ},L^1}(a,a) = 2/3 \neq 0$ .
(iv)  $\overline{\Delta}_{\mathrm{SJL},L^2}$  is not a metric on  $[0,1]^p$ . For instance, if  $a = 0.8$ ,  $b = 0.4$ ,  $c = 0.2$ , it does not satisfy the triangle inequality.
G Proof of Theorem 2.3
Proof. (i)  $\forall x\in [0,1]^p$ $y\in \{0,1\} ^p$  and  $x\in \{0,1\} ^p$ $y\in [0,1]^p$ $\overline{\Delta}_{\mathrm{JSL},L^1} = \overline{\Delta}_{\mathrm{JML},1} =$ $\overline{\Delta}_{\mathrm{JML},2}$ . Due to symmetry, we only need to prove the first part. Note that
$$
\| x \| _ {1} = \sum_ {i} x _ {i} = \sum_ {i} \mathbb {1} _ {\left(y _ {i} = 0\right)} x _ {i} + \sum_ {i} \mathbb {1} _ {\left(y _ {i} = 1\right)} x _ {i} \tag {35}
$$
$$
\| y \| _ {1} = \sum_ {i} y _ {i} = \sum_ {i} \mathbb {1} _ {\left(y _ {i} = 1\right)} \tag {36}
$$
$$
\langle x, y \rangle = \sum_ {i} x _ {i} y _ {i} = \sum_ {i} \mathbb {1} _ {\left(y _ {i} = 1\right)} x _ {i} \tag {37}
$$
$$
\left\| x - y \right\| _ {1} = \sum_ {i} \left| x _ {i} - y _ {i} \right| = \sum_ {i} \mathbb {1} _ {\left(y _ {i} = 0\right)} x _ {i} - \sum_ {i} \mathbb {1} _ {\left(y _ {i} = 1\right)} x _ {i} + \sum_ {i} \mathbb {1} _ {\left(y _ {i} = 1\right)}. \tag {38}
$$
Combine these, we have
$$
\bar {\Delta} _ {\mathrm {S J L}, L ^ {1}} = \bar {\Delta} _ {\mathrm {J M L}, 1} = \bar {\Delta} _ {\mathrm {J M L}, 2} = \frac {\sum_ {i} \mathbb {1} _ {(y _ {i} = 1)} x _ {i}}{\sum_ {i} \mathbb {1} _ {(y _ {i} = 0)} x _ {i} + \sum_ {i} \mathbb {1} _ {(y _ {i} = 1)}}. \tag {39}
$$
(ii)  $\exists x, y \in [0,1]^p, \overline{\Delta}_{\mathrm{SJL},L^1} \neq \overline{\Delta}_{\mathrm{JML},1} \neq \overline{\Delta}_{\mathrm{JML},2}$ . For instance, if  $x = 0.8, y = 0.5, \overline{\Delta}_{\mathrm{SJL},L^1} \neq \overline{\Delta}_{\mathrm{JML},1} \neq \overline{\Delta}_{\mathrm{JML},2}$ .
H  $\overline{\Delta}_{\mathrm{JML,1}}$  vs.  $\overline{\Delta}_{\mathrm{JML,2}}$
Definition H.1 (Convex Closure). The convex closure of a set function  $f: x \in \{0, 1\}^p \to [0, 1]$  is
$$
\mathbb {C} f: [ 0, 1 ] ^ {p} \rightarrow [ 0, 1 ] = \min  _ {\alpha} \sum_ {i = 1} ^ {p} \alpha_ {i} f \left(x _ {i}\right), \tag {40}
$$
$$
\text {s . t .} \sum_ {i = 1} ^ {p} \alpha_ {i} = 1, \tag {41}
$$
$$
\sum_ {i = 1} ^ {p} \alpha_ {i} x _ {i} = x, \tag {42}
$$
$$
\alpha_ {i} \geq 0. \tag {43}
$$
The convex closure extends a set function that is only defined at the vertices  $\{0,1\}^p$  to the whole hypercube  $[0,1]^p$  by linearly interpolating the values at these vertices. We plot the loss value of  $\overline{\Delta}_{\mathrm{JML},1}$ ,  $\overline{\Delta}_{\mathrm{JML},2}$  and the convex closure of  $\overline{\Delta}_{\mathrm{S JL},L^1}$  in Figure 2 when  $y = 0.5$ . Note that  $\overline{\Delta}_{\mathrm{JML},1}$  overlaps with the convex closure at  $[0,0.5]$ . We can see that  $\overline{\Delta}_{\mathrm{JML},1} \leq \overline{\Delta}_{\mathrm{JML},2}$  and  $\overline{\Delta}_{\mathrm{JML},1}$  is closer to the convex closure. Generally, we have
Theorem H.2.  $\forall x,y\in [0,1]^p,\overline{\Delta}_{JML,I}\leq \overline{\Delta}_{JML,2}.$
$$
\begin{array}{l} \bar {\Delta} _ {\mathrm {J M L}, 1} \leq \bar {\Delta} _ {\mathrm {J M L}, 2} (44) \\ \Rightarrow \frac {\| x \| _ {1} + \| y \| _ {1} - \| x - y \| _ {1}}{\| x \| _ {1} + \| y \| _ {1} + \| x - y \| _ {1}} - \frac {\langle x , y \rangle}{\langle x , y \rangle + \| x - y \| _ {1}} \geq 0 (45) \\ \Rightarrow \left(\| x \| _ {1} + \| y \| _ {1}\right) \| x - y \| _ {1} - \| x - y \| _ {1} ^ {2} - 2 \langle x, y \rangle \| x - y \| _ {1} \geq 0. (46) \\ \end{array}
$$
Figure 2: Comparing  $\overline{\Delta}_{\mathrm{JML,1}}$ ,  $\overline{\Delta}_{\mathrm{JML,2}}$  and the convex closure with  $y = 0.5$ .
We can switch the element of  $x$  and  $y$  whenever  $x_{i} \leq y_{i}$  for some  $i$ . Doing this, each term in the last inequality remains the same. Let us denote the new variable as  $x^{\prime}, y^{\prime}$  such that  $x_{i}^{\prime} \geq y_{i}^{\prime}$  for all  $i$ . Hence
$$
\begin{array}{l} \left(\| x \| _ {1} + \| y \| _ {1}\right) \| x - y \| _ {1} - \| x - y \| _ {1} ^ {2} - 2 \langle x, y \rangle \| x - y \| _ {1} (47) \\ = \left(\| x ^ {\prime} \| _ {1} + \| y ^ {\prime} \| _ {1}\right) \| x ^ {\prime} - y ^ {\prime} \| _ {1} - \| x ^ {\prime} - y ^ {\prime} \| _ {1} ^ {2} - 2 \langle x ^ {\prime}, y ^ {\prime} \rangle \| x ^ {\prime} - y ^ {\prime} \| _ {1} (48) \\ = \left(\| x ^ {\prime} \| _ {1} + \| y ^ {\prime} \| _ {1}\right) \left(\| x ^ {\prime} \| _ {1} - \| y ^ {\prime} \| _ {1}\right) - \left(\| x ^ {\prime} \| _ {1} - \| y ^ {\prime} \| _ {1}\right) ^ {2} - 2 \langle x ^ {\prime}, y ^ {\prime} \rangle \left(\| x ^ {\prime} \| _ {1} - \| y ^ {\prime} \| _ {1}\right) (49) \\ = 2 \left(\| x ^ {\prime} \| _ {1} - \| y ^ {\prime} \| _ {1}\right) \left(\| y ^ {\prime} \| _ {1} - \langle x ^ {\prime}, y ^ {\prime} \rangle\right) (50) \\ = 2 \sum_ {i} \left(x _ {i} ^ {\prime} - y _ {i} ^ {\prime}\right) \sum_ {i} \left(1 - x _ {i} ^ {\prime}\right) y _ {i} ^ {\prime} \geq 0. (51) \\ \end{array}
$$
Theorem H.3. Given a set function  $l: \{0,1\}^p \to [0,1]$  and two concave functions  $f, g: [0,1]^p \to [0,1]$ . If  $\forall x \in \{0,1\}^p$ ,  $l(x) = f(x) = g(x)$  and  $\forall x \in [0,1]^p$ ,  $f(x) \leq g(x)$ , then  $\forall x \in [0,1]^p$ ,  $|f(x) - \mathbb{C}l(x)| \leq |g(x) - \mathbb{C}l(x)|$ .
Proof. It suffices to show  $\forall x\in [0,1]^p,f(x)\geq \mathbb{C}l(x). \forall x\in [0,1]^p$  , we have  $x = \sum_{i = 1}^{p}\alpha_{i}x_{i}$  such that  $\alpha_{i}\geq 0,\sum_{i = 1}^{p}\alpha_{i} = 1$  and  $x_{i}\in \{0,1\}$  . Therefore