Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a loss function to learn from multi-label data with only one-positive annotation. Specially, the proposed loss function considers three factors: i) the existing asymmetric loss; ii) it uses a threshold to differentiate true-positive and true-negative in the unlabeled candidate labels. The self-labeled data is further used in the asymmetric loss; iii) a regularizer which ensures the data augmentation will not change the model too much. These items are weighted and added together. The paper analyzes some theoretical properties of the problem and the proposed loss function, and finally, shows sufficient experimental results to demonstrate the effectiveness of the proposed method. 
strength_and_weaknesses: Strength
1. It proposes an effective loss function for learning from multi-labeled data with only one positive annotation.
2. It has shown sufficient experimental results and overall, the performance of the proposed method is superior
3. The paper is clearly written and easy to follow

Weakness
1. The proposed technique is a combination of existing techniques, may lack technical novelty
2. The theoretical results may not be convincing (will explain in detail later)  
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. The quality of the paper is high, especially considering the sufficient experimental results provided. One concern could be the theory part. Specially, Theorem 5.2 is correct, however, it does add value to the study, since this is a too trivial result. Theorem 5.3 and Theorem 5.4, should have added value to the paper, especially considering the equivalence of the proposed loss function to the weighted hamming loss. However, the condition only enforces the positive label (f_k(x)) to be correctly predicted but did not imply anything about the negative labels (f_k(x)<threshold). Without conditions on the correct prediction of negative labels, I cannot be convinced that the proposed loss functions depending on negative labels could have the provided theoretical property. I would like to see more evidence on this part. 

Some minor comments
1. f_k(x) may not be defined in the paper before using it.
2. Due to the randomness in generating the data, it is better to generate multiple datasets instead of one to reduce the impact of randomness
3. The paper has provided theoretical results showing the equivalence to weighted hamming loss but the experiments test only MAP. It is better to provide results on more metrics, especially those closely related to the theoretical properties of the proposed method. 
summary_of_the_review: The paper has shown a simple, straightforward, and effective solution to a kind of weakly supervised multi-label learning problem. The studied problem is important and the solution is empirically effective, although the proposed method is not novel. The theoretical results may also need to be strengthened.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the single positive labels problem in multi-label learning. A new loss function called leveraged asymmetric loss with disambiguation (LASD) is proposed, which is an improved version of the existing leveraged asymmetric loss (ASL) with an explicit pseudo label disambiguation. The proposed algorithm further adds a consistency regularization to the LASD loss. Theoretic analysis provides a link between the proposed LASD and Hamming loss. Empirical evaluations are reported using four benchmark datasets against the state-of-the-art literature.
strength_and_weaknesses: Strengths:
-	the presentation is clear
-	the proposed solution appears to be correct
-	the theoretic analysis appears to make sense
-	the evaluations as well as the ablation studies appear to be extensive
Weaknesses:

I have two major concerns with this work. First, the novelty. The proposed loss function is an extension from the existing LAS loss. While there is arguably something new with the extension, novelty-wise it is incremental at best.

Second, the proposed consistency regularization, while conceptually it makes sense, lacks justification in terms of the current form. Why do we have to consider two augmented samples, not one, or not three? It appears to be rather arbitrary and ad hoc.

clarity,_quality,_novelty_and_reproducibility: As I stated earlier, the presentation is clear and the work appears to be reproducible. Also as I stated earlier, the novelty is incremental. So overall the quality of this work needs further justification.
summary_of_the_review: This work addresses the single positive labels problem in multi-label learning. The paper proposes a new loss function explicitly addressing the asymmetric imbalance between positive and negative labels. The theoretical analysis makes sense and the proposed method is supported by evaluations. I have concern on the novelty and the method.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper seeks to study a weakly supervised problem called Single Positive Labels (SPL) in multi-label learning where only one positive label is observed for each examples. It proposes a loss function named leveraged asymmetric loss with disambiguation (LASD) following multi-label version of focal loss for SPL problem, and tries to prove that learning with the proposed loss is consistent.
strength_and_weaknesses: Strengths:
1. The topic is interesting.
2. The method is easy to understand.
3. Although unsuccessful, the theoretical analysis of the proposed method is to be encouraged.

Weaknesses:
1. The theoretical analysis is deeply flawed. This paper proves the risk consistency of LASD by proving the risk of the classifier learned with LASD approaches to the risk of the classifier **learned with a specifically defined and  (most likely, at least, unproven) non-classification-calibrated multi-label loss function**. Such "consistency" is meaningless because classification-calibrated is the most basic requirement for a loss function.
2. The assumption of Theorem 5.3 is unrealistic. It assumes that if the classifier f can already successfully classify (i.e., the optimal classifier), then LASD is spurious risk-consistent. One wants the learning to be consistent so that the learned classifier is guaranteed to be (asymptotically) equivalent to the optimal classifier, rather than assuming that the classifier is the optimal one first and then proving something else. It's a complete reversal of cause and effect.
3. Assumption 5.1 and Theorem 5.2 are disconnected from the main content. They are neither an original contributions nor relevant to the approach presented in the paper.
clarity,_quality,_novelty_and_reproducibility: As far as I know, this paper is not the first work to solve SPL problem from the perspective of risk consistency [One Positive Label is Sufficient: Single-Positive Multi-Label Learning with Label Enhancement. Xu, et al.]
summary_of_the_review: The theoretical analysis of the paper needs further revision and improvement.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a novel loss function called leveraged asymmetric loss with disambiguation (LASD) for the problem of multi-label learning from single positive labels (SPL) where only one positive annotation is available. A pair of leverage parameters are employed to address the severe negative-positive imbalance. This paper links the SPL loss with losses for ordinary multi-label classification from the perspective of risk consistency and proves the consistency of the proposed LASD loss to Hamming loss. Finally, experiments including comparing it with other state-of-art methods and ablation are conducted to validate the effectiveness of their method.
strength_and_weaknesses: Strengths:
1. A pair of leverage parameters are employed to address the severe negative-positive imbalance of the SPL problem and the usage of consistency regularization can further improve the performance.
2. The proposed method is easy to understand and reproduce.

Weakness:
1. The most significant contribution of this work is analyzing an SPL loss function from the perspective of risk consistency. However, the assumption of Theorem 5.3 is unreasonable, in the inequation $ \left|1_{\left[f_k(x) \geq \tau\right]}-1_{[k \in \vec{y}]}\right| \leq \varepsilon $, the left part only has the value of 0 and 1, letting $\varepsilon \rightarrow 0$ is equivalent to assuming that the pseudo positive labels selected by the threshold are just real positive labels. In this case, the proof of the consistency between LASD loss and Hamming loss makes no sense.
2. Theorem 5.4 guarantees the effectiveness of LASD for the SPL problem in solving the severe positive-negative label imbalance by properly selecting the leverage parameters $\lambda_{+}, \lambda_{-}$. But another import hyper-parameter $\tau$ which is used to select pseudo-positive labels in LASD loss is neglected. It would be better if there is theoretical guidance for the selection of the threshold rather than assuming that the pseudo-positive labels selected by the threshold are just real positive labels.
3. Leveraged asymmetric loss (LAS) has been proposed to alleviate the extreme imbalance of negative-positive labels in the SPL problem. LASD loss just adds a threshold to select the pseudo-positive labels which also has been widely used in previous work. There is not enough innovation in this work.
4. there is a lack of experiments to verify the assumption of Theorem 5.3 and Theorem 5.4.
5. In Table 2, the improvement of the performance on datasets VOC12 and CUB is not obvious enough.

clarity,_quality,_novelty_and_reproducibility: This paper links the SPL loss with losses for ordinary multi-label classification from the perspective of risk consistency and proves the consistency of the proposed LASD loss to Hamming loss. Extensive experiments are conducted to validate the effectiveness of their method. However, the assumption of Theorem 5.3 that the pseudo-positive labels selected by the threshold are just real positive labels is unreasonable, which makes the proof of the risk consistency unconvincing.

Leveraged asymmetric loss (LAS) has been proposed to alleviate the extreme imbalance of negative-positive labels in the SPL problem. LASD loss just adds a threshold to select the pseudo-positive labels which also has been widely used in previous work, the same is true for consistency regularization. There is not enough innovation in this work.

summary_of_the_review: A pair of leverage parameters are employed to address the severe negative-positive imbalance of the SPL problem and the usage of consistency regularization can further improve the performance. However, the assumption that the pseudo-positive labels selected by the threshold are just real positive labels is unreasonable and there is a lack of experiments to verify the assumption.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers the problem of multi-label classification from single positive labels. The authors assume that the single label is uniformly drawn from fully observed labels. For this assumption, they study the unbiasedness of introduced loss functions. The experimental studies illustrate the advantages of the proposed approach.

Unfortunately, the theoretical contribution has several drawbacks. The term risk consistency is somehow mixed with the statistical consistency usually considered in the context of learning reductions and surrogate losses. Here, the risk consistency is related to noisy labels and unbiased losses. This is not clearly stated in the paper. The notation and theoretical results need thorough revision as several issues have been pointed out by the reviewers. The current results might be still recovered, but it is not clear whether they are of any real importance. The cost-sensitive Hamming loss is not really widely used. It is not clear why the authors claim the opposite.

The authors might be interested in the analysis of the pick-one label approach given in "Multilabel reductions: what is my loss optimising?" or "A no-regret generalization of hierarchical softmax to extreme multi-label classification".

justification_for_why_not_higher_score: There is only one score above the bar, but even this reviewer has pointed out several issues in the paper. The theoretical part is not in the mature state, contains several issues, and the results might not be of any importance as the authors show that the introduced loss is unbiased for a very specific loss, not really widely used. Interestingly, they claim the opposite, but even the cited papers do not consider this loss function (at least I did not find it in these papers).
Decision: Reject