SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To study the correctness of estimated class distribution, we provide the following theorem, which gives an upper bound on the difference between the estimated class proportion  $\hat{\gamma}_k$  and the true class proportion  $\gamma_k^*$  (its proof is given in Appendix A). A similar result can be derived for  $\hat{\rho}_k$ .
Theorem 1. Assume the estimated class proportion  $\hat{\gamma}_k = \frac{1}{n}\sum_{i=1}^n \mathbb{I}(y_{ik} = 1)$ , and the true class proportion  $\gamma_k^* = \frac{1}{m}\sum_{j=1}^m \mathbb{I}(y_{jk} = 1)$  for any  $k \in [q]$ , where  $n$  and  $m$  are the numbers of labeled and unlabeled examples that satisfy  $m >> n$ . Then, with the probability larger than  $1 - 2n^{-1} - 2m^{-1}$ , we have,  $\forall k \in [q], |\hat{\gamma}_k - \gamma_k^*| \leq \frac{\sqrt{\log n}}{\sqrt{2n}} + \frac{\sqrt{\log m}}{\sqrt{2m}}$ .
Theorem 1 tells us that the correctness of the estimated class distribution mainly depends on the number of labeled and unlabeled data. In general, the bound is dominated by the first term, since it always satisfies  $m >> n$ . By neglecting the second term, we can see that  $\forall k \in [q], \gamma_k^* \to \hat{\gamma}_k$  in the parametric rate  $\mathcal{O}_p(1 / \sqrt{n})$ , where  $\mathcal{O}_p$  denotes the order in probability. Obviously, as the number of training examples increase, the estimated class distribution would quickly converge to the true one.
4.2 Generalization Bound
Moreover, we study the generalization performance of CAP. Before providing the main results, we first define the true risk with respect to the classification model  $f(\pmb{x};\theta)$ :
$$
R (f) = \mathbb {E} _ {(\boldsymbol {x}, \boldsymbol {y})} [ \mathcal {L} (f (\boldsymbol {x}), \boldsymbol {y}) ].
$$
Our goal is to learn a good classification model by minimizing the empirical risk  $\widehat{R}(f) = \widehat{R}_l(f) + \widehat{R}_u(f)$ , where  $\widehat{R}_l(f)$  and  $\widehat{R}_u(f)$  are respectively the empirical risk of the labeled loss  $\mathcal{L}_l(f(\boldsymbol{x}), \boldsymbol{y})$  and unlabeled loss  $\mathcal{L}_u(f(\boldsymbol{x}), \boldsymbol{y})$ :
$$
\widehat {R} _ {l} (f) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} (f (\boldsymbol {x} _ {i}), \boldsymbol {y} _ {i}), \quad \widehat {R} _ {u} (f) = \frac {1}{m} \sum_ {j = 1} ^ {m} \mathcal {L} _ {u} (f (\boldsymbol {x} _ {j}), \boldsymbol {y} _ {j}).
$$
Note that during the training, we cannot train a model directly by optimizing  $\widehat{R}_u(f)$ , since the labels of unlabeled data are inaccessible. Instead, we train the model with  $\widehat{R}_u'(f) = \frac{1}{m}\sum_{j=1}^{m}\mathcal{L}_u(f(\boldsymbol{x}_j),\hat{\boldsymbol{y}}_j)$ , where  $\hat{\boldsymbol{y}}_j$  represents the pseudo-label vector of the instance  $\boldsymbol{x}_j$ .
Let  $\ell(f_k(\boldsymbol{x})) = y_k\ell_1(f_k(\boldsymbol{x})) + (1 - y_k)\ell_0(f_k(\boldsymbol{x}))$  be the loss for the class  $k$ , and  $L_{\ell}$  be any (not necessarily the best) Lipschitz constant of  $\ell$ . Let  $\mathcal{R}_N(\mathcal{F})$  be the expected Rademacher complexity [31] of  $\mathcal{F}$  with  $N = m + n$  training points. Let  $\hat{f} = \arg \min_{f\in \mathcal{F}}\widehat{R}(f)$  be the empirical risk minimizer, where  $\mathcal{F}$  is a function class, and  $f^{\star} = \arg \min_{f\in \mathcal{F}}R(f)$  be the true minimizer. We derive the following theorem, which provides a generalization error bound for the proposed method (its proof is given in Appendix B).
Theorem 2. Suppose that  $\ell(\cdot)$  is bounded by  $B$ . For some  $\epsilon > 0$ , if  $\sum_{j=1}^{m} |\mathbb{I}(f_k(x_j) \geq \tau(\alpha_k)) - \mathbb{I}(y_{jk} = 1)| / m \leq \epsilon$  for any  $k \in [q]$ , for any  $\delta > 0$ , with probability at least  $1 - \delta$ , we have
$$
R (\hat {f}) - R (f ^ {\star}) \leq 2 q B \epsilon + 4 q L _ {\ell} R _ {N} (\mathcal {F}) + 2 q B \sqrt {\frac {\log \frac {2}{\delta}}{2 N}}.
$$
From Theorem 4, it can be observed that the generalization performance of  $\hat{f}$  mainly depends on two factors, i.e., the pseudo-labeling error  $\epsilon$  and the number of training examples  $N$ . Apparently, a smaller pseudo-labeling error  $\epsilon$  often leads to better generalization performance. Thanks to its ability to capture the true class distribution, CAP can achieve a much smaller pseudo-labeling error  $\epsilon$  than existing instance-aware pseudo-labeling methods, which is beneficial for obtaining better classification performance. This can be further validated by our empirical results in Section 5.3. The second factor is the number of training examples. As  $N \to \infty$  and  $\epsilon \to 0$ , Theorem 4 shows that the empirical risk minimizer  $\hat{f}$  converges to the true risk minimizer  $f^{\star}$ .
In this section, we first perform experiments to validate the effectiveness of the proposed method; then, we perform ablation studies to analyze the mechanism behind CAP.
Datasets To evaluate the performance of the propose method, we conduct experiments on three benchmark image datasets, including Pascal VOC-2012 (VOC for short)  $^{2}$  [12], MS-COCO-2014 (MS-COCO for short)  $^{3}$  [25], and NUS-WIDE (NUS for short)  $^{4}$  [7]. The detailed information of these datasets can be found in the appendix. For each dataset, we randomly sample a proportion  $p \in \{0.05, 0.1, 0.15, 0.2\}$  of examples with full labels while the others without any supervised information. Following the previous works [8, 53], we report the mean average precision (mAP) on the test set for each method.
Comparing methods To validate the effectiveness of the proposed method, we compare it with five groups of methods: 1) three instance-aware pseudo-labeling methods: Top-1 (Eq.(2)), Top- $k$  (Eq.(3)), IAT (Eq.(4)); 2) two state-of-the-art MLML methods: LL [19] (includes three variants LL-R, LL-Ct, and LL-Cp), PLC [46]; 3) Two state-of-the-art SSL methods: Adsh [14], FreeMatch [44]; 4) One state-of-the-art SSMLL method: DRML [43]; 5) Two baseline methods, BCE, ASL [34]. DRML is the only deep SSMLL method whose source code could be found on the Internet.
<sup>2</sup>http://hostrobots.ox.ac.uk/pascal/VOC/
<sup>3</sup>https://cocodataset.org
4https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/NUS-WIDE.html
Table 1: Comparison results on VOC and COCO in terms of mAP (\%). The best performance is highlighted in bold.
Method	VOC	COCO
p = 0.05	p = 0.1	p = 0.15	p = 0.2	p = 0.05	p = 0.1	p = 0.15	p = 0.2
BCE	67.95	75.35	78.19	79.38	58.90	63.75	65.91	67.33
ASL	71.46	78.00	79.69	80.77	59.12	63.82	66.10	67.51
LL-R	75.69	80.96	82.31	83.55	59.31	64.25	66.61	68.01
LL-Ct	75.77	81.04	82.31	83.50	59.33	64.23	66.69	68.11
LL-Cp	75.79	81.03	82.36	83.68	59.27	64.19	66.68	68.12
PLC	74.49	80.35	82.35	83.39	59.85	65.03	67.62	69.14
Top-1	75.77	80.78	82.65	83.72	57.62	62.84	65.50	66.96
Top-k	75.07	80.20	81.99	83.16	58.25	63.52	66.11	67.49
IAT	73.24	80.27	82.39	83.55	60.34	65.54	67.88	69.25
ADSH	75.37	80.34	82.80	83.93	60.75	65.37	67.70	69.01
FreeMatch	75.11	80.66	82.63	83.60	59.94	64.46	66.79	68.04
DRML	61.77	71.01	72.98	74.49	53.60	57.06	58.53	59.24
Ours	76.16	82.16	83.48	84.41	62.43	67.36	69.11	70.41
Table 2: Comparison results on NUS in terms of mAP (\%). The best performance is highlighted in bold.
Method	ASL	LL-R	PLC	Top-1	Top-k	IAT	ADSH	FreeMatch	DRML	Ours
p = 0.05	42.87	40.20	43.55	40.99	40.89	42.58	43.94	43.12	30.61	44.82
p = 0.10	46.50	44.95	47.51	45.07	45.04	46.60	47.28	46.65	35.09	48.24
p = 0.15	48.42	47.32	49.75	47.43	47.22	48.76	49.22	48.74	37.91	49.90
p = 0.20	49.65	48.31	50.71	48.49	48.37	49.62	49.93	49.59	39.98	51.06
Furthermore, most MLML methods cannot be applied to the SSMLL scenario, since they assume that a subset of labels have been annotated for each training instance. The detailed information of these methods can be found in the appendix.
Implementation We employ ResNet-50 [16] pre-trained on ImageNet [36] for training the classification model. We adopt RandAugment [9] and Cutout [10] for data augmentation. We employ AdamW [29] optimizer and one-cycle policy scheduler [10] to train the model with maximal learning rate of 0.0001. The number of warm-up epochs is set as 12 for all datasets. The batch size is set as 32, 64, and 64 for VOC, MS-COCO, and NUS. Furthermore, we perform exponential moving average (EMA) for the model parameter  $\theta$  with a decay of 0.9997. For all methods, we use the ASL loss as the base loss function, since it shows superiority to BCE loss [34]. We perform all experiments on GeForce RTX 3090 GPUs. The random seed is set to 1 for all experiments.
5.2 Comparison Results
Table 1 and Table 2 report the comparison results between CAP and the comparing methods in terms of mAP on VOC, COCO, and NUS. From the tables, we can see that: 1) DRML obtains unfavorable performance, even worse than baselines BCE and ASL, since it performs two-stage training, which may destroy its representation learning. The original paper did not report the results on these three datasets. Therefore, it is rather important to design an effective SSMLL method in deep learning paradigm. 2) CAP outperforms three instance-aware pseudo-labeling methods, which demonstrates that by utilizing CAT strategy, CAP can precisely estimate the class distribution of unlabeled data and thus obtain desirable pseudo-labeling performance. 3) The performance of CAP is better than that of two state-of-the-art SSL methods. To achieve better performance, we have made several modifications for these two methods not limited to the following: a) use the ASL loss; b) adopt stronger data augmentations; c) change the training scheme to make them more suitable for the multi-label scenario. 4) CAP achieves the best performance in all cases and significantly outperforms
(a)  $p = 0.05$
(b)  $p = 0.1$
(c)  $p = 0.15$
(d)  $p = 0.2$
Figure 2: Pseudo-labeling performance in terms of CF1 score on VOC, COCO. Each row corresponds one dataset.
(a) VOC,  $\eta_0 = 1$
Figure 3: Performance of CAP on VOC and COCO in terms of mAP (\%) with the increase of  $\eta_{1}$  and  $\eta_{0}$ .
(b) VOC,  $\eta_{1} = 1$
(c) COCO,  $\eta_0 = 1$
(d) COCO,  $\eta_{1} = 1$
the comparing methods, especially when the number of labeled examples is small. These results convincingly validate the effectiveness of the proposed method.
5.3 Study on the Performance of Pseudo-Labeling
In this section, we explain why CAP is better than the conventional instance-aware pseudo-labeling methods. Figure 2 illustrates the performance of different pseudo-labeling methods in terms of CF1 score on VOC and COCO. From the figures, we can see that CAP achieves the best performance in all cases. As discussed above, the pseudo-labeling performance mainly depends on two factors, i.e., the quality of model predictions and the correctness of estimated class proportions. CAP improves the pseudo-labeling performance by precisely estimating the class distribution. An interesting observation is that at the first epoch, when the model predictions are the same for four methods, our method significantly outperforms the comparing methods, since it is able to capture the true class proportions. These results validate that CAP can achieve better pseudo-labeling performance.
5.4 Study on the Influence of Reliable Interval
As mentioned above, to improve the performance, instead of using all pseudo-labels, we can train the model with only reliable pseudo-labels within the reliable intervals that are controlled by  $\eta_1$ ,  $\eta_0$ . Figure 3 illustrates the performance of CAP as  $\eta_1$  and  $\eta_0$  change in the ranges of [0.8, 0.85, 0.9, 0.95, 1] and [0.95, 0.96, 0.97, 0.98, 0.99, 1]. From the figures, it can be observed that discarding the unreliable positive pseudo-labels would improve the performance, but discarding the unreliable negative pseudo-labels would degrade the performance. One possible reason behind the phenomenon is due to the significant positive-negative imbalance in MLL data, i.e., the number of negative labels is often much greater than that of positive labels. This leads the model to be sensitive to false positive labels, while being robust to false negative labels. In our main experiments (Table 1 and Table 2), we set
$\eta_{1} = 1, \eta_{0} = 1$ . In practice, we are expected to achieve better performance by tuning the parameter  $\eta_{1}$ .
The paper studies the problem of semi-supervised multi-label learning, which aims to train a multi-label classifier by leveraging the information of unlabeled data. Different from the conventional instance-aware pseudo-labeling methods, we propose to assign pseudo-labels to unlabeled instances in a class-aware manner, with the aim of capturing the true class distribution of unlabeled data. Towards this goal, we propose the CAT strategy to obtain an estimated class distribution, which has been proven to be a desirable estimation of the true class distribution based on our observations. Theoretically, we first perform an analysis on the correctness of estimated class distribution; then, we provide the generalization error bound for CAP and show its dependence to the pseudo-labeling performance. Extensive experimental results on multiple benchmark datasets validate that CAP can achieve state-of-the-art performance.
In general, the performance of pseudo-labeling depends mainly on two factors, i.e., the quality of the model predictions and the correctness of the estimated class distribution. This work focuses on the latter. In future, we plan to boost the performance of SSMLL by improving the quality of model predictions.
Sheng-Jun Huang was supported by Natural Science Foundation of Jiangsu Province of China (BK20222012, BK20211517), the National Key R&D Program of China (2020AAA0107000), and NSFC (62222605). Masashi Sugiyama was supported by JST CREST Grant Number JPMJCR18A2.
[1] Mokhtar S Bazaraa, Hanif D Sherali, and Chitharanjan M Shetty. Nonlinear programming: theory and algorithms. John Wiley & Sons, 2013.
[2] Emanuel Ben-Baruch, Tal Ridnik, Itamar Friedman, Avi Ben-Cohen, Nadav Zamir, Asaf Noy, and Lihi Zelnik-Manor. Multi-label classification with partial annotations using class-aware selective loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4764-4772, 2022.
[3] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785, 2019.
[4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems, 32, 2019.
[5] Gang Chen, Yangqiu Song, Fei Wang, and Changshui Zhang. Semi-supervised multi-label learning by solving a sylvester equation. In Proceedings of the 2008 SIAM international conference on data mining, pages 410-419. SIAM, 2008.
[6] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5177-5186, 2019.
[7] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus-wide: a real-world web image database from national university ofSingapore. In Proceedings of the ACM international conference on image and video retrieval, pages 1-9, 2009.
[8] Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic. Multi-label learning from single positive labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 933-942, 2021.
[42] Bo Wang, Zhuowen Tu, and John K Tsotsos. Dynamic label propagation for semi-supervised multi-class multi-label classification. In Proceedings of the IEEE international conference on computer vision, pages 425-432, 2013.
[43] Lichen Wang, Yunyu Liu, Can Qin, Gan Sun, and Yun Fu. Dual relation semi-supervised multi-label learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6227-6234, 2020.
[44] Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. Freematch: Self-adaptive thresholding for semi-supervised learning. In The Eleventh International Conference on Learning Representations, 2023.
[45] Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A classrebalancing self-training framework for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10857-10866, 2021.
[46] Ming-Kun Xie, Jia-Hao Xiao, and Sheng-Jun Huang. Label-aware global consistency for multi-label learning with single positive labels. In Advances in Neural Information Processing Systems, 2022.
[47] Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference on Machine Learning, pages 11525-11536. PMLR, 2021.
[48] Yang Yang, Zhao-Yang Fu, De-Chuan Zhan, Zhi-Bin Liu, and Yuan Jiang. Semi-supervised multi-modal multi-instance multi-label deep network with optimal transport. IEEE Transactions on Knowledge and Data Engineering, 33(2):696-709, 2019.
[49] Yang Yang, Yi-Feng Wu, De-Chuan Zhan, Zhi-Bin Liu, and Yuan Jiang. Complex object classification: A multi-modal multi-instance multi-label deep network with optimal transport. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2594-2603, 2018.
[50] Wang Zhan and Min-Ling Zhang. Inductive semi-supervised multi-label learning with constraining. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1305-1314, 2017.
[51] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34:18408-18419, 2021.
[52] Feipeng Zhao and Yuhong Guo. Semi-supervised multi-label learning with incomplete labels. In Twenty-fourth international joint conference on artificial intelligence, 2015.
[53] Donghao Zhou, Pengfei Chen, Qiong Wang, Guangyong Chen, and Pheng-Ann Heng. Acknowledging the unknown for multi-label learning with single positive labels, 2022.
[54] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5982-5991, 2019.
Theorem 3. Assume the estimated class proportion  $\hat{\gamma}_k = \frac{1}{n}\sum_{i=1}^n \mathbb{I}(y_{ik} = 1)$ , and the true class proportion  $\gamma_k^* = \frac{1}{m}\sum_{j=1}^m \mathbb{I}(y_{jk} = 1)$  for any  $k \in [q]$ , where  $n$  and  $m$  are the numbers of labeled and unlabeled examples that satisfy  $m >> n$ . Then, with the probability larger than  $1 - 2n^{-1} - 2m^{-1}$ , we have,  $\forall k \in [q], |\hat{\gamma}_k - \gamma_k^*| \leq \frac{\sqrt{\log n}}{\sqrt{2n}} + \frac{\sqrt{\log m}}{\sqrt{2m}}$ .
Proof. The proof is mainly based on Hoeffding's inequality that can be defined as follows.
Lemma 1. (Hoeffding's inequality). Let  $z_{1}, \ldots, z_{N}$  be independent random variables bounded by  $[a_{i}, b_{i}]$ . Then  $\hat{z} = \frac{1}{N} \sum_{i=1}^{N} z_{i}$  obeys for any  $\nu > 0$
$$
\operatorname * {P r} (| \hat {z} - \mathbb {E} [ \hat {z} ] \geq \nu |) \leq 2 \exp (- \frac {2 N ^ {2} \nu^ {2}}{\sum_ {i = 1} ^ {N} (b _ {i} - a _ {i}) ^ {2}}).
$$
Let  $\bar{\gamma}_k = p(y_k = 1)$  represents the expected class proportion. According to Hoeffding's inequality, for any  $k\in [q]$ , we have
$$
\operatorname * {P r} (| \hat {\gamma} _ {k} - \bar {\gamma} _ {k} | \leq \frac {\sqrt {\log n}}{\sqrt {2 n}}) \geq 1 - 2 n ^ {- 1}
$$
or equivalently, with the probability at least  $1 - 2n^{-1}$ , we have  $|\hat{\gamma}_k - \bar{\gamma}_k| \leq \sqrt{\log n} / \sqrt{2n}$ . Similarly, with the probability at least  $1 - 2m^{-1}$ , for any  $k \in [q]$ , we have  $|\gamma_k^* - \bar{\gamma}_k| \leq \sqrt{\log m} / \sqrt{2m}$ . By applying the triangle inequality, with the probability at least  $1 - 2n^{-1} - 2m^{-1}$ , for any  $k \in [q]$ , we have
$$
\left| \gamma_ {k} ^ {*} - \hat {\gamma} _ {k} \right| \leq \frac {\sqrt {\log n}}{\sqrt {2 n}} + \frac {\sqrt {\log m}}{\sqrt {2 m}}.
$$
Theorem 4. Suppose that  $\ell(\cdot)$  is bounded by  $B$ . For some  $\epsilon > 0$ , if  $\sum_{j=1}^{m} |\mathbb{I}(f_k(x_j) \geq \tau(\alpha_k)) - \mathbb{I}(y_{jk} = 1)| / m \leq \epsilon$  for any  $k \in [q]$ , for any  $\delta > 0$ , with probability at least  $1 - \delta$ , we have
$$
R (\hat {f}) - R (f ^ {*}) \leq 2 q B \epsilon + 4 q L _ {\ell} R _ {N} (\mathcal {F}) + 2 q B \sqrt {\frac {\log \frac {2}{\delta}}{2 N}}.
$$
Proof. Before proving the theorem, we first provide two useful lemmas as follows.
We primarily derive the uniform deviation bound between  $R(f)$  and  $\widetilde{R}(f)$ , which is a simple extension of the result in the binary setting [31].
Lemma 2. Suppose that the loss function  $\ell$  is  $L_{\ell}$ -Lipschitz continuous w.r.t.  $\theta$ . For any  $\delta > 0$ , with probability at least  $1 - \delta$ , we have
$$
\left| R (f) - \widehat {R} (f) \right| \leq 2 q L _ {\ell} \mathcal {R} _ {n + m} (\mathcal {F}) + q B \sqrt {\frac {\log \frac {2}{\delta}}{2 (n + m)}} \tag {1}
$$
Proof. In order to prove this lemma, we define the Rademacher complexity of  $\mathcal{L}$  and  $\mathcal{F}$  with  $m + n$  training examples as follows:
$$
\begin{array}{l} \mathcal {R} _ {n + m} (\mathcal {L} \circ \mathcal {F}) \\ = \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}, \boldsymbol {\sigma}} \left[ \sup  _ {f \in \mathcal {F}} \sum_ {i = 1} ^ {n} \sigma_ {i} \mathcal {L} (f (\boldsymbol {x} _ {i}), \boldsymbol {y} _ {i}) + \sum_ {j = 1} ^ {m} \sigma_ {j} \mathcal {L} (f (\boldsymbol {x} _ {j}), \boldsymbol {y} _ {j}) \right] \\ \end{array}
$$
Considering that  $\mathcal{L}(f(\pmb{x}),\pmb{y}) = \sum_{k = 1}^{q}\ell (f_k,y_k)$ , we have
$$
\begin{array}{l} \mathcal {R} _ {n + m} (\mathcal {L} \circ \mathcal {F}) \leq q \mathcal {R} _ {n + m} (\ell \circ \mathcal {F}) \tag {10} \\ \leq q L _ {\ell} \mathcal {R} _ {n + m} (\mathcal {F}) \\ \end{array}
$$
where the second line is due to the Lipschitz continuity of the loss function  $\ell$ .
Then, we proceed the proof by showing that the one direction  $\sup_{f\in \mathcal{F}}R(f) - \widehat{R} (f)$  is bounded with probability at least  $1 - \delta /2$  , and the other direction can be proved similarly. Note that replacing an example  $(x_{j},y_{j})$  with another  $(x_j',y_j')$  leads to a change of  $\sup_{f\in \mathcal{F}}R(f) - \widehat{R} (f)$  at most  $\frac{qB}{n + m}$  due to the fact that  $\ell$  is bounded by  $B$  . According to McDiarmid's inequality [31], for any  $\delta >0$  , with probability at least  $1 - \delta /2$  , we have
$$
\sup  _ {f \in \mathcal {F}} R (f) - \widehat {R} (f) \leq \mathbb {E} \left[ \sup  _ {f \in \mathcal {F}} R (f) - \widehat {R} (f) \right] + q B \sqrt {\frac {\log \frac {2}{\delta}}{2 (n + m)}} \tag {11}
$$
According to the result in [31] (Theorem 3.3) that shows  $\mathbb{E}[\sup_{f\in \mathcal{F}}R(f) - \widehat{R} (f)]\leq 2\mathcal{R}_m(\mathcal{F})$ , by further considering the other direction  $\sup_{f\in \mathcal{F}}\widehat{R} (f) - R(f)$ , with probability at least  $1 - \delta$ , we have
$$
\sup  _ {f \in \mathcal {F}} \left| R (f) - \widehat {R} (f) \right| \leq 2 q L _ {\ell} \mathcal {R} _ {m} (\mathcal {F}) + q B \sqrt {\frac {\log \frac {2}{\delta}}{2 n + m}} \tag {12}
$$
Then, we can bound the difference between  $\widehat{R}(f)$  and  $\widehat{R}'(f)$  as follows
Lemma 3. Suppose that  $\ell(\cdot)$  is bounded by  $B$ . For some  $\epsilon > 0$ , if  $\sum_{j=1}^{m} |\mathbb{I}(f_k(x_j) \geq \tau(\alpha_k)) - \mathbb{I}(y_{jk} = 1)| / m \leq \epsilon$  for any  $k \in [q]$ , for any  $f \in \mathcal{F}$ , we have:
$$
\left| \widehat {R} _ {u} ^ {\prime} (f) - \widehat {R} _ {u} (f) \right| \leq q B \epsilon \tag {13}
$$