SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The inference model  $q_{\phi}(\mathbf{y}|\mathbf{x},\mathbf{s})$  and the generative model  $p_{\theta}(\mathbf{s}|\mathbf{x},\mathbf{y})$  are instantiated by fully-connected neural networks with ReLU activations, where the hidden dimensionalities are set to [256; 512; 256] and [256; 512] respectively. For fair comparison with existing PML approaches, the prediction model is implemented as a linear model. To compute the objective function in Eq. (3), a trade-off parameter  $\alpha$  is introduced for the KL-divergence term and Monte Carlo sampling with sampling number  $L = 1$  is conducted to estimate the first expectation term, where the temperature parameter  $\tau = 2/3$  as suggested by [27]. In the following experiments, we set  $\alpha \geq 1$  so that the objective function is still a valid lower bound of the data log-likelihood. For network optimization, Adam with a batch size of 128, weight decay of  $10^{-4}$ , momentum of 0.999 and 0.9 is employed. In this paper, all experiments are conducted on one V100 GPU.
4.2 Comparative Studies
$\mathrm{PARD}^4$  is compared against six well-established PML approaches with parameter configurations suggested in respective literatures:
- FPML [47]: FPML employs the low-rank approximation of the instance-label association matrix to estimate the labeling confidence and then trains multi-label predictor.  $[\lambda_1 = 0.1, \lambda_2 = 1, \lambda_3 = 10]$
${}^{4}$  Code package of PARD is publicly available at http://palm.seu.edu.cn/zhangml/files/PARD.rar.
Table 2: Predictive performance of each comparing approach (mean±std. deviation) in terms of Average precision, where  $\bullet/\circ$  indicates whether PARD is significantly superior/inferior to one comparing approach via paired  $t$ -test at 0.05 significance level.  $\uparrow (\downarrow)$  indicates the larger (smaller) the value, the better the performance. Best results are shown in boldface.
Data sets	γ%	Average precision↑
FPML	PARVLS	PML-NI	PML-MD	UPML-HL	UPML-RL	PARD
YeastBP	0.202±0.013●	0.059±0.003●	0.430±0.018○	0.310±0.016●	0.392±0.020●	0.170±0.009●	0.417±0.015
YeastCC	0.375±0.019●	0.145±0.009●	0.610±0.022●	0.527±0.027●	0.596±0.022●	0.459±0.021●	0.622±0.020
YeastMF	0.279±0.020●	0.114±0.007●	0.471±0.022●	0.422±0.027●	0.451±0.016●	0.329±0.023●	0.491±0.028
Musicemotion	0.571±0.013●	0.603±0.011●	0.600±0.012●	0.646±0.011	0.656±0.012○	0.639±0.012●	0.650±0.007
Music_style	0.702±0.014●	0.720±0.012●	0.733±0.012●	0.716±0.012●	0.734±0.010●	0.703±0.012●	0.742±0.009
corel5k	100	0.485±0.014●	0.391±0.017●	0.473±0.016●	0.491±0.019●	0.516±0.015●	0.514±0.015●	0.533±0.013
150	0.478±0.013●	0.408±0.011●	0.453±0.013●	0.479±0.015●	0.514±0.015●	0.512±0.015●	0.523±0.013
200	0.474±0.013●	0.395±0.021●	0.438±0.012●	0.472±0.011●	0.503±0.009●	0.511±0.013	0.516±0.011
250	0.470±0.012●	0.391±0.022●	0.426±0.011●	0.470±0.019●	0.502±0.013●	0.507±0.014	0.513±0.012
rcv1-s1	100	0.682±0.010●	0.520±0.021●	0.693±0.014●	0.678±0.010●	0.712±0.013●	0.689±0.007●	0.720±0.008
150	0.678±0.011●	0.513±0.022●	0.664±0.013●	0.656±0.011●	0.705±0.013●	0.690±0.008●	0.716±0.010
200	0.673±0.010●	0.495±0.021●	0.649±0.010●	0.648±0.010●	0.704±0.012●	0.685±0.005●	0.710±0.010
250	0.666±0.011●	0.488±0.016●	0.630±0.011●	0.640±0.011●	0.693±0.015●	0.672±0.009●	0.706±0.009
Corel16k-s1	100	0.535±0.010●	0.430±0.010●	0.518±0.011●	0.522±0.011●	0.539±0.012●	0.525±0.007●	0.551±0.012
150	0.529±0.008●	0.423±0.012●	0.499±0.011●	0.512±0.011●	0.532±0.011●	0.522±0.009●	0.545±0.012
200	0.523±0.010●	0.404±0.012●	0.487±0.013●	0.507±0.012●	0.504±0.011●	0.519±0.009●	0.536±0.014
250	0.514±0.011●	0.389±0.011●	0.476±0.013●	0.501±0.011●	0.507±0.014●	0.516±0.008●	0.531±0.014
iaprtc12	100	0.603±0.006●	0.591±0.008●	0.599±0.009●	0.600±0.008●	0.601±0.009●	0.563±0.008●	0.621±0.011
150	0.600±0.006●	0.585±0.008●	0.580±0.010●	0.583±0.010●	0.597±0.007●	0.559±0.007●	0.615±0.009
200	0.597±0.008●	0.574±0.008●	0.563±0.008●	0.569±0.009●	0.600±0.008●	0.554±0.008●	0.610±0.008
250	0.588±0.009●	0.561±0.008●	0.543±0.009●	0.560±0.009●	0.582±0.008●	0.547±0.006●	0.601±0.009
espgame	100	0.498±0.007●	0.472±0.007●	0.478±0.008●	0.491±0.009●	0.501±0.008●	0.466±0.009●	0.515±0.008
150	0.496±0.007●	0.456±0.007●	0.459±0.007●	0.481±0.007●	0.509±0.008	0.471±0.006●	0.511±0.009
200	0.495±0.007●	0.445±0.007●	0.446±0.005●	0.473±0.007●	0.475±0.012●	0.466±0.006●	0.508±0.008
250	0.489±0.009●	0.427±0.009●	0.431±0.007●	0.467±0.007●	0.490±0.008●	0.460±0.006●	0.501±0.009
- PARVLS [48]: A two-step approach which elicits credible labels via label propagation and then induces multi-label predictor by virtual label splitting.  $[k = 10, \alpha = 0.95$ , and  $thr = 0.9]$
- PML-NI [35]: PML-NI performs disambiguation by learning a noisy label identifier with sparsity regularization.  $[\beta = 0.5, \gamma = 0.5$ , and search for  $\lambda \in \{1, 10, 100\}]$
- PML-MD [37]: PML-MD disambiguates between ground-truth and noisy labels in a meta-learning fashion and learns multi-label predictor with a confidence-weighted ranking loss.
- UPML-HL [36]: A disambiguation-free approach which directly induces the prediction model by optimizing an unbiased estimator of the Hamming loss.
- UPML-RL [36]: A disambiguation-free approach which directly induces the prediction model by optimizing an unbiased estimator of the Ranking loss.
Following [37], we take out  $10\%$  examples in each data set as hold-out validation set, so that PML-MD can use this clean validation set to perform meta-disambiguation. The remaining  $90\%$  examples are randomly splitted into training set and test set with a ratio of 9:1 for training and evaluation respectively. During training, PML-MD performs meta-learning on the noisy training set and the clean validation set. While other approaches including ours only employ this validation set to search hyperparameters. In PARD, the trade-off parameter  $\alpha$  is searched in  $\{1,2,5,10,20,50,100\}$ . For fair comparison, all deep approaches adopt a linear model to instantiate the predictor and share the same optimizer with the learning rate searched in  $\{1e-3,3e-3,1e-2,3e-2\}$ . For each data set, we repeat the random splitting process for ten times and record the average predictive performance across ten training/test trials.
Due to page limit, Table 2 and Table 3 report detailed experimental results in terms of Average precision and Ranking loss. Results on other metrics can be found in Appendix A. Furthermore, the Wilcoxon signed-ranks test [33] at 0.05 significance level is conducted to analyze whether PARD achieves statistically superior performance to other comparing approaches. Table 4 summarizes the  $p$ -value statistics on each evaluation metric. Based on these results, it is impressive to observe that:
Table 3: Predictive performance of each comparing approach (mean±std. deviation) in terms of Ranking loss, where  $\bullet/\circ$  indicates whether PARD is significantly superior/inferior to one comparing approach via paired  $t$ -test at 0.05 significance level.  $\uparrow (\downarrow)$  indicates the larger (smaller) the value, the better the performance. Best results are shown in boldface.
Data sets	γ%	Ranking loss ↓
FPML	PARVLS	PML-NI	PML-MD	UPML-HL	UPML-RL	PARD
YeastBP	0.338±0.007●	0.482±0.005●	0.197±0.012●	0.218±0.008●	0.222±0.015●	0.309±0.011●	0.175±0.010
YeastCC	0.305±0.016●	0.480±0.010●	0.156±0.017●	0.190±0.017●	0.167±0.017●	0.212±0.015●	0.135±0.017
YeastMF	0.373±0.019●	0.533±0.010●	0.226±0.018●	0.235±0.014●	0.239±0.023●	0.286±0.017●	0.192±0.015
Musicemotion	0.274±0.012●	0.252±0.009●	0.249±0.010●	0.231±0.010	0.223±0.009○	0.236±0.011●	0.228±0.007
Music_style	0.160±0.010●	0.150±0.007●	0.139±0.008	0.143±0.007●	0.138±0.007	0.149±0.007●	0.137±0.009
corel5k	100	0.255±0.013●	0.357±0.020●	0.280±0.012●	0.251±0.011●	0.236±0.014●	0.229±0.013●	0.223±0.012
150	0.260±0.013●	0.327±0.011●	0.296±0.015●	0.260±0.014●	0.235±0.014	0.235±0.013	0.233±0.010
200	0.263±0.011●	0.338±0.021●	0.312±0.013●	0.264±0.012●	0.254±0.010●	0.240±0.012	0.242±0.010
250	0.268±0.012●	0.347±0.024●	0.322±0.013●	0.267±0.015●	0.257±0.013●	0.242±0.013	0.245±0.010
rcv1-s1	100	0.101±0.004●	0.222±0.013●	0.106±0.007●	0.093±0.005●	0.086±0.005●	0.087±0.003●	0.076±0.003
150	0.102±0.005●	0.240±0.020●	0.126±0.006●	0.105±0.005●	0.089±0.004●	0.089±0.004●	0.079±0.003
200	0.107±0.006●	0.262±0.022●	0.138±0.005●	0.111±0.004●	0.092±0.007●	0.093±0.003●	0.084±0.004
250	0.114±0.006●	0.277±0.017●	0.153±0.005●	0.115±0.005●	0.099±0.006●	0.100±0.004●	0.090±0.003
Corel16k-s1	100	0.219±0.006●	0.289±0.007●	0.247±0.009●	0.221±0.008●	0.212±0.006●	0.216±0.006●	0.205±0.006
150	0.226±0.006●	0.300±0.011●	0.262±0.009●	0.228±0.006●	0.224±0.007●	0.220±0.007●	0.212±0.007
200	0.235±0.007●	0.329±0.011●	0.274±0.011●	0.232±0.008●	0.250±0.009●	0.223±0.006	0.221±0.009
250	0.245±0.009●	0.369±0.010●	0.286±0.012●	0.238±0.007●	0.249±0.011●	0.228±0.006●	0.223±0.008
iaprtc12	100	0.189±0.004●	0.207±0.005●	0.206±0.006●	0.192±0.005●	0.199±0.007●	0.210±0.004●	0.180±0.006
150	0.191±0.004●	0.212±0.005●	0.222±0.007●	0.203±0.005●	0.203±0.006●	0.211±0.005●	0.186±0.005
200	0.196±0.004●	0.227±0.006●	0.239±0.006●	0.214±0.005●	0.203±0.006●	0.216±0.005●	0.191±0.005
250	0.204±0.006●	0.241±0.005●	0.257±0.006●	0.222±0.006●	0.218±0.007●	0.221±0.006●	0.199±0.005
espgame	100	0.252±0.005●	0.281±0.006●	0.285±0.004●	0.258±0.006●	0.256±0.006●	0.277±0.008●	0.241±0.005
150	0.256±0.006●	0.300±0.005●	0.302±0.005●	0.266±0.006●	0.248±0.006●	0.265±0.006●	0.245±0.006
200	0.258±0.006●	0.313±0.006●	0.314±0.005●	0.272±0.006●	0.277±0.008●	0.269±0.005●	0.248±0.006
250	0.266±0.006●	0.330±0.007●	0.329±0.007●	0.274±0.006●	0.269±0.007●	0.275±0.006●	0.255±0.005
Table 4: Summary of the Wilcoxon signed-ranks test for PARD against other comparing approaches at 0.05 significance level.  $p$ -values are shown in the brackets.
PARD against	FPML	PARVLS	PML-NI	PML-MD	UPML-HL	UPML-RL
Average precision	win [1.2e-5]	win [1.2e-5]	win [1.8e-5]	win [1.2e-5]	win [1.7e-5]	win [1.2e-5]
Hamming loss	win [8.2e-3]	win [4.6e-5]	win [2.1e-5]	win [8.5e-5]	tie [7.0e-1]	win [3.5e-5]
One-error	win [1.2e-5]	win [1.2e-5]	win [4.6e-5]	win [1.2e-5]	win [1.8e-4]	win [1.2e-5]
Coverage	win [1.2e-5]	win [1.2e-5]	win [1.4e-5]	win [1.2e-5]	win [2.4e-5]	win [4.0e-5]
Ranking loss	win [1.2e-5]	win [1.2e-5]	win [1.2e-5]	win [1.2e-5]	win [2.0e-5]	win [2.5e-5]
- Across all evaluation metrics, PARD achieves the best performance in  $82.4\%$  cases over all the 25 data sets.
- As shown in Table 4, PARD achieves statistically better performance against other approaches which rely on heuristics or ad-hoc rules for disambiguation. The superior performance of PARD is consistent across almost all real-world data sets and synthetic data sets under varied noise levels, which provides a strong evidence for the effectiveness of probabilistic graphical disambiguation to facilitate partial multi-label learning.
- Meantime, PARD significantly outperforms PML-MD which also disambiguates PML data in a principled manner. Note that PML-MD requires an accurately annotated validation set to perform disambiguation during the learning process. The superior performance of PARD against PML-MD indicates that PARD is an effective and more flexible approach for principled disambiguation.
4.3 Further Analyses
4.3.1 Ablation Study
To validate that the disambiguation process in PARD is effective for facilitating partial multi-label learning, we implement a baseline which induces prediction model with cross-entropy loss by directly treating all the candidate labels as valid ones. As shown in Figure 2, a clear performance degradation
Figure 2: Predictive performance of PARD and the baseline in terms of Average precision and Ranking loss on real-world data sets.
(a) YeastCC
(b) Music_style
Figure 3: Validation performance of PARD with varying trade-off parameter  $\alpha$ .
is witnessed in the baseline model, which demonstrates the disambiguation process induced from the variational lower bound can facilitate partial multi-label learning.
4.3.2 Parameter Sensitivity
Figure 3 gives some illustrative examples on how the performance of PARD changes when the value of the trade-off parameter  $\alpha$  changes. The performance of PARD is relatively stable as the value of  $\alpha$  changes within a reasonable range, which is a desirable property when deploying PARD in real-world applications. Similar results can be observed on other data sets.
4.3.3 Running Time Comparison
For PARD, the training phase only involves a pair of forward and backward computations among the inference, generative and prediction models, which is efficient thanks to the training techniques presented in section 3.3. Figure 4 illustrates the empirical training and test time of each comparing approach, which shows that PARD is competitive with existing approaches in time overhead.
(a) Training time of comparing approaches
Figure 4: Running time (training/test) of each comparing approach on real-world data sets. For histogram illustration, the  $y$ -axis corresponds to the logarithm of running time.
(b) Test time of comparing approaches
Our primary focus is on partial multi-label learning in this paper. However, there are other weakly-supervised multi-label learning problems that receive increasing attention from related community. The probabilistic graphical disambiguation framework of PARD currently has not been tested on these problems, such as learning with partial labels which deals with incompletely annotated data [7, 19, 1, 4], learning with single positive label which aims to induce an accurate multi-label predictor from multi-label data annotated with only one positive labels [5, 28, 15, 38, 16, 22], and learning with general noisy labels where noise may exist in every label of inaccurately annotated data [18, 21, 36]. In addition, we do not implement PARD and investigate its behavior in large model environment, which are also important to practitioners. We will explore these points in the future.
In this paper, an attempt towards principled disambiguation for partial multi-label learning is presented, where a directed graphical model is tailored to describe the generative process of partial multi-label data. By maximizing the data log-likelihood on given partial multi-label data set with a unified surrogate objective derived by variational inference, PARD achieves to disambiguate the candidate label set and induce the prediction model simultaneously. Comprehensive experiments show the superiority of our approach. Since disambiguation lies in the heart of partial multi-label learning, we hope that PARD will encourage more future researches to explore alternative implementations for principled disambiguation.
The authors wish to thank the anonymous reviewers for their helpful comments and suggestions. This work was supported by the National Science Foundation of China (62225602), and the Big Data Computing Center of Southeast University.
[1] Emanuel Ben Baruch, Tal Ridnik, Itamar Friedman, Avi Ben-Cohen, Nadav Zamir, Asaf Noy, and Lihi Zelnik-Manor. Multi-label classification with partial annotations using class-aware selective loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4754-4762, New Orleans, LA, 2022.
[2] Matthew R. Boutell, Jiebo Luo, Xipeng Shen, and Christopher M. Brown. Learning multi-label scene classification. Pattern Recognition, 37(9):1757-1771, 2004.
[3] Nan Cao, Teng Zhang, and Hai Jin. Partial multi-label optimal margin distribution machine. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, pages 2198-2204, Montreal, Canada, 2021.
[4] Tianshui Chen, Tao Pu, Hefeng Wu, Yuan Xie, and Liang Lin. Structured semantic transfer for multi-label recognition with partial labels. In Proceedings of the 36th AAAI Conference on Artificial Intelligence, pages 339-346, Virtual Event, 2022.
[5] Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic. Multi-label learning from single positive labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 933-942, virtual, 2021.
[6] Timothee Cour, Benjamin Sapp, and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research, 12:1501-1536, 2011.
[7] Thibaut Durand, Nazanin Mehrasa, and Greg Mori. Learning a deep convnet for multi-label classification with partial labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 647-657, Long Beach, CA, 2019.
[8] A. Elisseeff and J. Weston. A kernel method for multi-labelled classification. In Advances in Neural Information Processing Systems 14, pages 681-687, Vancouver, Canada, 2001.
[43] Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. In Advances in Neural Information Processing Systems 34, pages 27119-27130, virtual, 2021.
[44] Ning Xu, Jun Shu, Yun-Peng Liu, and Xin Geng. Variational label enhancement. In Proceedings of the 37th International Conference on Machine Learning, pages 10597-10606, Virtual Event, 2020.
[45] Yan Yan and Yuhong Guo. Partial label learning with batch label correction. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, pages 6575-6582, New York, NY, 2020.
[46] Yan Yan and Yuhong Guo. Adversarial partial multi-label learning with label disambiguation. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, pages 10568-10576, Virtual Event, 2021.
[47] Guoxian Yu, Xia Chen, Carlotta Domeniconi, Jun Wang, Zhao Li, Zili Zhang, and Xindong Wu. Feature-induced partial multi-label learning. In Proceedings of the IEEE International Conference on Data Mining, pages 1398-1403, Singapore, 2018.
[48] Min-Ling Zhang and Jun-Peng Fang. Partial multi-label learning via credible label elicitation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10):3587-3599, 2020.
[49] Min-Ling Zhang, Fei Yu, and Cai-Zhi Tang. Disambiguation-free partial label learning. IEEE Transactions on Knowledge and Data Engineering, 29(10):2155-2167, 2017.
[50] Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26(8):1819-1837, 2014.
[51] Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1):44-53, 2018.
[52] Zhi-Hua Zhou. Open-environment machine learning. National Science Review, 9(8):nwac123, 2022.
[53] Yue Zhu, James T. Kwok, and Zhi-Hua Zhou. Multi-label learning with global and local label correlation. IEEE Transactions on Knowledge and Data Engineering, 30(6):1081-1094, 2018.