Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the Single-Positive Multi-Label Learning problem in which the learner only observes one label in the label set of each instance. Based on the estimated unobserved labels for each instance, the authors rewrite the expected risk to obtain an unbiased estimator for the SPMLL problem. The authors propose a label enhancement process to estimate the unobserved labels by enforcing that the recovered label should not be completely different from the confidence estimated by the current prediction. Experiments show the effectiveness of the proposed approach.
strengths_and_weaknesses: Strengths 
+ This paper studies an interesting but critical problem, which appears in many real-world applications.
+ The proposed label enhancement process seems to give a nice approximation for the unobserved labels in the SPMLL problem.
+ The empirical studies seem convincing and show the effectiveness of their proposed method.

Weaknesses
- The proposed theoretical analysis is established on the assumption that the proposed label enhancement process accurately recovers the unobserved labels. Based on this premise, the theoretical analysis seems standard and lacks new insights.
questions: Is there any analysis that how the correctness of the recovered unobserved labels affects the estimation error bound? It is also suggested to provide some experimental results for the accuracy of recovered unobserved labels.
limitations: The authors have adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Nowadays, the weakly-supervised problem has become more and more popular since the difficulty in annotation acquiring process. Single-positive multi-label learning, where each example is annotated with only one relevant label, can draw the attention of the AI audience. This paper proposes a novel single-positive multi-label learning method via adopting an unbiased risk estimator enhancing and recovering the latent soft label as a label enhancement process. In addition, this paper establishes an estimation error bound that guarantees the learning consistency of the proposed method and proves that the risk minimizer could converge to the Bayes risk minimizer of fully supervised learning. Extensive experiments are conducted and show significant results. 
strengths_and_weaknesses: This paper for the first time proposes an unbiased risk estimator for single-positive multi-label learning and shows that one can successfully learn a theoretically grounded multi-label classifier for the single-positive multi-label learning problem, which is novel and solid. I think this point motivates the novel solution of solving the single-positive multi-label learning problem by recovering the latent soft label as a label enhancement process. The latent soft label is recovered by an inference model and the topological structure in the data, where the posterior density of the latent soft label is inferred via taking on the approximate Beta density and an evidence lower bound for the optimization is deduced. This is novel and interesting. The established estimation error bound guarantees the learning consistency of the proposed method and demonstrates that the obtained risk minimizer would converge to the optimal risk minimizer of fully supervised learning as the number of training data tends to infinity. According to the experiment results, the proposed method achieves superior performance to other comparing methods. The paper is very well organized and easy to follow. The paper can be improved by refining some problems:
1.	The authors should give more details about how to analytically calculate the second part of Eq. (9). 
2.	The paper should add more details about label enhancement in related work.
3.	The paper could show the recovered soft label matrix D over all training examples could converge on other datasets. 
4.	The charts and figures in the paper take up too much space (like Table 5, Figure 2 and Figure 3), the authors should try to adjust them.

questions: 1.How to calculate the second part of Eq. (9) analytically?
limitations: The paper has addressed the limitations that the employment of data annotators might be decreased which could lead to a negative societal impact.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a novel method for single-positive multi-label learning (SPMLL). The method derives an unbiased risk estimator and establishes a corresponding empirical risk estimator by generating latent soft labels with label enhancement. The authors prove that the proposed method could approximately converge to the optimal risk minimizer of fully supervised multi-label learning, which shows that a single-positive label of each instance is sufficient to train the predictive multi-label model in SPMLL. Experiments on benchmark datasets validate the effectiveness of the proposed method. 
strengths_and_weaknesses: This paper proposes a novel single-positive multi-label learning method, which derives an unbiased risk estimator and generates the soft labels in the corresponding empirical risk estimator. The authors prove that the proposed method adopted with the unbiased risk estimator is risk-consistent for SPMLL. To the best of my knowledge, this is the first attempt to consider the learning consistency in SPMLL problem. The theoretical results show that training a fine predictive multi-label model on single-positive labeled data is possible, which is vital for the SPMLL setting. I really appreciate the motivation and the theoretical results of the proposed method. This paper generates soft labels via approximating the Beta density parameterized by GCN, which could be optimized by the deduced evidence lower bound. This is novel and effective to estimate the soft label in the empirical risk estimator. The authors analyzed the experiments in detail and the Wilcoxon signed-ranks show that the proposed method achieves significant performance against other methods. Besides, the authors conduct the experiments to validate the helpfulness and the convergence of label enhancement. The paper is clearly written and well organized. 

This paper could be further improved. The authors should add more details about learning consistency in section 3 since it is vital for SPMLL. In addition, it would be good to add some more discussion about why the proposed method could achieve superior performance against the vanilla variant by approximating d_ij with the confidence of the predictive model.

questions: why the proposed method could achieve superior performance against the vanilla variant by approximating d_ij with the confidence of the predictive model?
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers a cost-effective way on multi-label learning, i.e., single-positive multi-label learning (SPMLL). Theoretically, an unbiased risk estimator is derived and an estimation error bound based on it is established. Practically, a method called SMILE is proposed, which leverages the latent soft labels recovered in a label enhancement process. Experiments on the benchmark datasets validate the effectiveness of the proposed method.
strengths_and_weaknesses: This paper studies single-positive multi-label learning (SPMLL), which is a cost-effective way to solve the problem of high annotation cost of multi-label learning. The authors propose an unbiased risk estimator of SPMLL, which is shown to be guaranteed to approximately converge to the optimal risk minimizer of fully supervised learning. The authors give in-depth theoretical analysis to derive the error bound to demonstrate its learning consistency. 
Extensive experimental results on multiple datasets and evaluation metrics demonstrate the effectiveness of the proposed method.
There are some questions:
1. In the real world, if data annotators are required to annotate only one relevant label for multi-label instances, a common phenomenon is that people tend to choose the most common and easy to see label for annotation, while the probability of those uncommon and hidden labels being annotated may be greatly reduced. This fact is somewhat different from the experimental setup in which a positive label of randomly selected instances is used to construct a dataset. How to deal with this difference in practical application? The authors should consider and add reasonable discussion.
2. Noting that the performance of proposed method SMILE on one-error metric is not ideal on image and scene datasets, the authors should add some explanation. As far as I am known, the label cardinality of these two datasets is close to 1. Is this related to this phenomenon?

questions: 1. In the real world, if data annotators are required to annotate only one relevant label for multi-label instances, a common phenomenon is that people tend to choose the most common and easy to see label for annotation, while the probability of those uncommon and hidden labels being annotated may be greatly reduced. This fact is somewhat different from the experimental setup in which a positive label of randomly selected instances is used to construct a dataset. How to deal with this difference in practical application? The authors should consider and add reasonable discussion.
2. Noting that the performance of proposed method SMILE on one-error metric is not ideal on image and scene datasets, the authors should add some explanation. As far as I am known, the label cardinality of these two datasets is close to 1. Is this related to this phenomenon?

limitations: Yes
review_title: null

Meta-review: metareview: This paper studies the single-positive multi-label learning problem, in which each example is annotated with only one relevant label. The problem is practical and challenging. To address this problem, this paper proposes a new unbiased estimator with a theoretical guarantee. The idea is novel and technically sound. The experimental results also demonstrate the effectiveness of the proposal. All reviewers agree that this study is novel and solid. So I recommend acceptance.
confidence: Certain
Decision: Accept