Title: ComRank: Ranking Loss for Multi-Label Complementary Label Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-label learning

complementary label leaning

weakly supervised learning
Abstract: Multi-label complementary label learning (MLCLL) is a weakly supervised paradigm that addresses multi-label learning (MLL) tasks using complementary labels (i.e., irrelevant labels) instead of relevant labels. Existing methods typically adopt an unbiased risk estimator (URE) under the assumption that complementary labels follow a uniform distribution. However, this assumption fails in real-world scenarios due to instance-specific annotation biases, making URE-based methods ineffective under such conditions. Furthermore, existing methods underutilize label correlations inherent in MLL. To address these limitations, we propose ComRank, a ranking loss framework for MLCLL, which encourages complementary labels to be ranked lower than non-complementary ones, thereby modeling pairwise label relationships. Theoretically, our surrogate loss ensures Bayes consistency under both uniform and biased cases. Experiments demonstrate the effectiveness of our method in MLCLL tasks. The code is available at https://github.com/JellyJamZhu/ComRank.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a ranking loss framework for multi-label complementary label learning (MLCLL). Different from existing methods that are based on the assumption that complementary labels follow a uniform distribution, the proposed framework considers the instance-dependent assumption and introduces a complementary ranking loss for the MLCLL task.\n\nSTRENGTHS AND WEAKNESSES:\nThis work introduces a ranking loss framework for multi-label complementary label learning (MLCLL). Different from existing methods that are based on the assumption that complementary labels follow a uniform distribution, the proposed framework considers the instance-dependent assumption and introduces a complementary ranking loss for the MLCLL task.\n\nStrengths:\n\n1.\tThe writing of this work is clear and easy to follow.\n2.\tThe presented mathematical proofs are rigorous.\n3.\tThe motivation of this work is convictive and experimental results verify the effectiveness of the proposed complementary ranking loss.\n\nWeaknesses:\n\n1.\tMy main concern focuses on the section of \"Complementary Label Distribution\". Regarding the uniform distribution assumption, it’s limitations illustrated in this work are easy to follow and persuasive. However, the introduced biased distribution assumption in Eq. (5) is quite challenging to follow and interpret. From my understanding from Eq. (5), $p(k={\\overline y} | x)$ is inversely proportional to $p(k \\in Y | x)$. However, it is quite different from the biased distribution assumption. I wonder if there is a misunderstanding on my part. It would be helpful if the authors could offer a more detailed and intuitive explanation for Eq. (5).\n2.\tAnother concern of mine is about \"Assumption 4.4\". I think that Assumption 4.4 is not suitable for a multi-label learning setting, it is more suitable from the single-label learning setting. As the example provided on Page 4, Line 162, if there are more than on animals in an image, a complementary label provided by annotators cannot be solely based on the image’s features. However, Assumption 4.4 plays an important role in this work, I wonder if there is a misunderstanding on my part. It would be helpful if the authors could offer a more detailed and intuitive explanation for Assumption 4.4.\n\nI would be willing to give a higher score if the authors can address the above-mentioned concerns in the response.\n\nQUESTIONS:\nPlease refer to \"Strengths And Weaknesses\"\n\nLIMITATIONS:\nPlease refer to \"Strengths And Weaknesses\"", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel algorithm for multi-label complementary label learning (MLCLL), supported by solid theoretical analysis and strong empirical results.\n\nHowever, it suffers from certain conceptual inconsistencies (see Issue 1–2), incomplete consideration of related work (see Issue 3), and potential flaws in the experimental design (see Issues 4–5).\n\nOverall, although I raised a lot of issues, the merits outweigh the flaws. I suggest a weak accept.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Clear motivation. Due to practical concerns such as privacy protection, MLCLL is an important problem to study. This paper is among the first to point out that in real-world scenarios, complementary labels are not sampled uniformly at random, but tend to be semantically opposed to true labels.\n\n2. Strong theoretical foundation. The proposed method is proven to be Bayes-consistent, which is a desirable theoretical property.\n\n3. Extensive experiments. The paper provides a comprehensive empirical evaluation across a wide range of baselines, datasets, metrics, and ablation analyses.\n\nWeaknesses\n1. Potential contradiction in modeling assumptions. Although Assumption 4.4 (instance-dependence of complementary labels) facilitates tractable loss construction and avoids unobservable dependencies on the true label set Y, it potentially conflicts with the authors' earlier intuition that complementary labels are selected with semantic exclusion from the true labels—e.g., “desert” being chosen over “lake” when the true label is \"water\". This highlights a fundamental tension between model identifiability and the realism of annotator behavior modeling.\n\n2. Theoretical comparison with URE is insufficient. While the proposed ComRank loss is theoretically proven to be Bayes-consistent in preserving the ranking order of label posterior probabilities, it does not offer an unbiased estimation of the original classification risk E[L(g(x),Y)] as URE does under the uniform assumption. Therefore, the theoretical advantage of ComRank is conditional and context-specific, and it does not universally dominate URE in terms of consistency with classification-oriented loss functions. To summarize, the paper criticizes URE for its bias but praises ComRank for its consistency. However, consistency alone does not imply superiority. Could the authors provide analysis or evidence regarding the (lack of) unbiasedness of ComRank?\n\n3. Missing discussion on related ranking-based multi-label learning (MLL) methods.\nThe paper lacks a discussion of prior work on ranking-based approaches in MLL, which may lead to an overstatement of novelty. For example:[1] shows that in probabilistic MLL (PML), ranking between positive and negative labels can help disambiguate false positives. [2] shows that in MLML settings, missing labels can be assumed to rank between positive and negative labels. I suggest the authors add a paragraph in the related work section discussing ranking-based MLL methods.\n\nAdditionally, Section 3 (Preliminaries) is somewhat redundant. It uses two equations to explain what is essentially standard background. I recommend simplifying this section by presenting only one general MLL risk minimization formula and using a comparative table to illustrate the differences between MLL and MLCLL in terms of labels, learners, and loss definitions—along with symbol definitions.\n\n4. Unrealistic experimental assumption about complementary labels.\nThe experiments assume that each instance has exactly one complementary label. This may not reflect realistic scenarios and could bias the evaluation in favor of the proposed method.\nSpecifically, the proposed method relies on pairwise ranking between positive and complementary labels. With only one complementary label, the computational cost is O(K−1), but if the number of complementary labels grows to K/2, the complexity increases to O(K^2/4). Please clarify the implications of this.\n\n5. Evaluation metrics are incomplete.\nThe experiments rely solely on ranking-based metrics, while classification-based metrics are not reported. Some baselines are designed for MLL or PML, which may not be directly comparable with the proposed method. If including these baselines is necessary, more recent and ranking-aware baselines should be included to ensure fairness and meaningful comparison.\n\n6. Minor issues in Appendix A. The #Domain refers to data source rather than the data type, all datasets used are tabular data. The #Domain of BibTeX should be Text rather than Images. \n\n[1] Partial multi-label learning, AAAI, Vol. 32, 2018.\n\n[2] A ranking-based problem transformation method for weakly supervised multi-label learning, PR, 2024\n\nQUESTIONS:\nSee above weaknesses (especially 1–5) for necessary revisions or clarifications. The overall direction is promising, but the paper would benefit from addressing the above concerns to improve its clarity, fairness, and theoretical rigor.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a ranking loss framework for Multi-label complementary label learning under both uniform and biased cases. The theoretical analysis is satisfying.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper presents a well-organized structure with clear logical flow and professional presentation, meeting the high standards of academic writing in this field.\n2. The theoretical analysis demonstrates particular rigor, offering valuable insights into the limitations of Unbiased Risk Estimator (URE) within the MLCLL context. This contribution significantly enhances our theoretical understanding of the problem.\n3. The work introduces a novel weakly supervised multi-label learning paradigm that addresses an important gap in current research, with demonstrated practical relevance to real-world applications.\n\n**Weaknesses:**\n1. The assertion that the proposed framework can directly capture label correlation information from rankings (Lines 45-46) requires stronger empirical validation. Including results or analyses that explicitly demonstrate this capability would strengthen the paper’s validity.\n2. The relationship between Theorem 4.5 (URE under biased distribution) and the proposed ComRank requires deeper explanation. The author should explicitly state the theoretical connections.\n3. The formulation of Assumption 4.4 (instance-dependent assumption) requires further clarification to ensure rigorous theoretical grounding.\n\nQUESTIONS:\nPlease see the weaknesses which need for more evidences and explanations.\n\nAdditionally, please answer the following questions:\n\n1. The author should provide more empiricial analysis to prove \"In MLCLL, based on the fact that complementary labels are known to be irrelevant, while non-complementary labels may include relevant ones, it is generally desirable for the predicted probabilities of complementary labels to rank lower than those of non-complementary labels.\"\n\n2. I hope to see more experiment results on more challenging MLL benchmark datasets, including VOC and COCO, to strengthen the empirical evaluation.\n\n3. Since the author used the exponential function as the surrogate loss, I expect to see more ablation studies about the impact of selecting different forms of surrogate loss, including log loss, sigmoid loss, and softmax loss.\n\nLIMITATIONS:\nThe relationship between Theorem 4.5 (URE under biased distribution) and the proposed ComRank requires deeper explanation. The author should explicitly state the theoretical connections.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses multi-label complementary label learning. It identifies limitations in existing methods relying on unbiased risk estimators, which assume uniform distribution of complementary labels—an assumption often violated in real-world scenarios with annotation biases, and these methods also underutilize label correlations.To tackle this, the paper proposes ComRank, a ranking loss framework that encourages complementary labels to be ranked lower than non-complementary ones, capturing pairwise label relationships.\nExperiments show ComRank outperforms existing methods on various datasets with both uniform and biased complementary labels. Its contributions include: analyzing why URE-based methods fail under biased distributions; introducing ranking loss to MLCLL via ComRank, which has theoretical guarantees under both uniform and biased cases and shows strong experimental performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n(1) Strong theoretical foundation: Comprehensively analyzes URE limitations and proves Bayes consistency of ComRank under both uniform and biased complementary label distributions.\n(2) Advances MLCLL: Introduces ranking loss to model label correlations, filling gaps in URE-based methods.\n(3) Rigorous experiments: Uses diverse datasets with uniform/biased labels, outperforming baselines across metrics like Average Precision and Ranking Loss.\nWeaknesses\n(1) Lacks computational complexity analysis, critical for scaling to large datasets.\n(2) Neglect label sparsity effects: Multi-label data often has sparse labels, but the paper doesn't test ComRank under varying sparsity. With extreme sparsity, few non-complementary labels may undermine its ranking logic, leaving its stability in such common real-world scenarios unproven.\n\nQUESTIONS:\n(1)Is there an error in defining a sample of Cartesian product used in MLL in the Preliminaries of Part 3? According to the definition in the text, it is a single label sample. Is it more accurate to define Y={1,2,..., K} as Y ∈ {0,1} ^ K .\n(2)The paper claims ComRank “directly captures label correlation information from rankings”, but this mechanism is not explicitly validated. Maybe you clarify it by Comparing ComRank’s performance on datasets with strong vs. weak label correlations. If correlation strength correlates with ComRank’s improvement over baselines, it would validate the claim.\n\nLIMITATIONS:\nThe paper only mentions that ComRank is restricted to the single complementary label setting, with extending to multiple complementary labels noted as future work. It fails to discuss other critical limitations, such as the sensitivity to the choice of surrogate loss (the paper uses an exponential function without exploring alternatives like logistic loss) and scalability issues with large label spaces, where the pairwise comparisons in the loss function could lead to high computational costs.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}