Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies a new Multi-Label Learning (MLL) setting, i.e., the Single Positive MLL (SPML), where only one positive label is observed. The authors use AN to learn SPML. Meanwhile, they put forward Pseudo-Labeling Consistency (PLC) regularization and Label-Aware Global Consistency (LAC) regularization to address the false negative problem raising by AN. The experiments show that the proposed method (AN + PLC + LAC) achieves the best performance. 
strengths_and_weaknesses: The main strengths are as follows:
+ Although AN is not stable for SPML, the authors put forward PLC and LAC to address the false negative problem raising by AN. 
+ The proposed method AN+PLC+LAC is shown to have the best performance compared with exiting SPML methods. The ablation study and case study are clear. 

Besides, I also have several questions:
- PLC, i.e., Eq. (4) has the indicator function, which is not easily to be trained. I expect to see the convergence of the proposed method. 
- The intra-class connection matrix I has a size of $B^2$, which is such an enormous matrix. What exactly is the size of the matrix in your experimental datasets? Is it possible to train with such an enormous matrix?
- LAC, i.e, Eq. (7) only involves the matrix I without the matrix $\bar{I}$. Where does $\bar{I}$ function?
questions: Please refer to my comments above in the Strengths And Weaknesses. 
limitations: The authors haven't claimed any limitations.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers the multi-label learning with single positive labels. In such scenario, each image is only associated with a single positive label. To recover the potential positive labels, the paper first pseudo-labels the unobserved labels based on the model predictions. To boost the labeling performance, the method considers the global consistency regularization for label-wise embeddings to learn more distinctive feature representations. The experiments are conducted in multiple benchmark multi-label datasets. Experimental results show the superiority of the proposed method.
strengths_and_weaknesses: Strength
1.	The paper proposes an effective solution to SPML problems. The main contribution of this paper is to utilize the label-aware global consistency to encourage the feature representations to maintain a more compact structure. The proposed method is well motivated and easy to follow.
2.	The experiments are performed on multiple benchmark datasets to compare with state-of-art methods. The proposed method achieves impressive performances and shows significant superiority to the comparing methods. Furthermore, the ablation studies and sensitivity analyses of hyper-parameters are reported. 

Weakness

1.	The experimental settings are less clear. For example, it is not introduced how to assign a single label for each image. The paper is suggested to provide some details.
2.	The label correlations are not considered in the proposed method. The label correlations have been regarded as a fundamental element for performing multi-label classification. Is it beneficial for improving the performance of SPML?
3.	In experiments, the proposed method shows large superiority on COCO than that on other datasets. Authors are suggested to provide detailed discussions about this observation.
4.	There are some language mistakes. The paper is suggested to be carefully proofread.

questions: Questions
1.	In many real-world scenarios, annotators may assign each image with a subset of all ground-truth labels instead of a single positive label. Can the proposed method solve this problem?
2.	In Section 4.4, Figure 3 illustrates the performance curves as the value of threshold increases. It seems that the method only achieves better performance with the increase of threshold on COCO dataset. Authors are suggested to give some detailed discussion about this phenomenon.
3.	Besides similarities among embeddings from a class label, there exists similarities among embedding from different labels. Can such similarity information help learn a good classification model? Why?

limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper deals with the single positive multi-label learning (SPML) task. Different from vanilla multi-label learning, in SPML, only a single (e.g., the most obvious) positive label is annotated. The authors propose a label-aware global consistency loss for the task. With a two-stage optimization, the method achieves good performance on various benchmarks.

strengths_and_weaknesses: The paper is easy to read, and the method is simple yet effective. The authors utilizes the manifold instead of the clustering assumptions in SPML, and show the advantage of the asumption based on the empirical experiments. 

Here are some suggestions for the paper:
1. Figure 1 is not clear enough. By comparing the single-label and multi-label cases, the authors may first show the difference between vanilla multi-label learning and SPML. While there are some elements (such as the embeddings and regularizations) in the figure, which make the figure a bit confusing.
2. The label-wise embedding before Eq. 1 is not clear. The authors use z to denote the label embedding, but z does not exist in Eq. 1. Since the label embedding is one of the most important elements in the global consistency loss, the authors need to make it clear. Does the label embedding useful in vanilla multi-label learning (e.g., in Eq. 1-2 in the paper).
3. Since the true number of positive labels cannot be accessible in many real-world scenarios, the authors need to discuss how the proposed method can deal with this case before the experiments.

questions: The paper is good in general, and here are some questions on the implementation details.
1. Does the label embedding in Eq. 7 instance-specific? It is a bit confusing to use $K$ to index them since it is hard to infer where the label embedding comes from.
2. How about the performance without a pre-trained backbone? Does the performance of the method rely on the pre-trained backbone? The authors may consider showing the performnace without a pre-trained backbone on a certain dataset.
3. The authors may consider some more criteria to evaluate SPML. For example, to evaluate how many latent labels it could infer from the data.
4. Will the performance of the model relates to the (maximum) number of positive labels in the dataset. 


-------------------------post rebuttal----------------------
I have read the response, and the authors have addressed my concerns. 
limitations: The authors have adequately addressed the limitations and potential negative social impact of the work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a Label-Aware global Consistency(LAC) regularization to solve the Single Positive Multi-label Learning (SPML) problem, where each instance is only annotated with one of the positive labels related. The proposed Label-Aware global Consistency(LAC) regularization leverages the manifold structure information via encouraging the global consistency among label-wise embedding, i.e., making intra-class embedding closer while keeping away inter-class embedding.  Finally, they conduct extensive experimental results on multiple benchmark datasets, demonstrating that the proposed method can achieve superior performance.
strengths_and_weaknesses: Strength:
1. This paper proposes a novel Label-Aware global Consistency(LAC) regularization to solve the Single Positive Multi-label Learning (SPML) problem, where each instance is only annotated with one of the positive labels related. 
2. This paper successfully applies ‘Clustering assumption’ in SPL and achieves good performance.
3. The visualization of the problem background, the proposed method, and the experiment results are good, which makes the reader easy to understand this work.

Weakness: 
1.  The basis for saying the sentences in Page 5 Line 150-153 needs to be introduced.   
2. In Page 3 Line 104, it mentions that this paper adopts the attention-based method to construct the label-wise embedding model, but in the following introduction about the proposed method, this point seems to be neglected in the proposed method, which is only related to Section 4.5 Case Study.   



questions: Data Report: In Table 1, the mAP of $L_{AN}$ on the dataset COCO is reported as 64.1. But in Table 2 about the ablation study on COCO, the method ‘assuming negative’ is reported as 68.5. From Page 8 Line 267-268, we can know that they are in the same setting using the same method. I wonder why there exists a 4.4% difference between them.
  

limitations: None
review_title: null

Meta-review: metareview: This paper studies the single-positive multi-label learning problem. To address this problem, the authors adopt the pseudo-labels to recover the potential positive labels and adopt the global consistency regularization for label-wise embeddings to learn more distinctive feature representations. Experimental results demonstrate the superiority of the proposal. All reviewers agree to accept this paper, so I recommend acceptance. Moreover, I still have some more suggestions: 1) The font size in Figure 3 could be larger to make the plot clear. 2) The reference format is not unified. I suggest the authors revise the reference format carefully.
confidence: Certain
Decision: Accept