Title: RankMatch: A Novel Approach to Semi-Supervised Label Distribution Learning Leveraging Rank Correlation between Labels
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-label learning

label distribution learning
Abstract: Pseudo label based semi-supervised learning (SSL) for single-label and multi-label classification tasks has been extensively studied; however, semi-supervised label distribution learning (SSLDL) remains a largely unexplored area.   Existing SSL methods fail in SSLDL because the pseudo-labels they generate only ensure overall similarity to the ground truth but do not preserve the ranking relationships between true labels, as they rely solely on KL divergence as the loss function during training.  These skewed pseudo-labels lead the model to learn incorrect semantic relationships, resulting in reduced performance accuracy.   To address these issues, we propose a novel SSLDL  method called \textit{RankMatch}.  \textit{RankMatch} fully considers the ranking relationships between different labels during the training phase with labeled data to generate higher-quality pseudo-labels. Furthermore, our key observation is that a flexible utilization of pseudo-labels can enhance SSLDL performance. Specifically, focusing solely on the ranking relationships between labels while disregarding their margins helps prevent model overfitting. Theoretically, we prove that incorporating ranking correlations enhances SSLDL performance and establish generalization error bounds for \textit{RankMatch}. Finally, extensive real-world experiments validate its effectiveness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores an important yet insufficiently studied challenge in LDL—the preservation of ranking structures in the presence of biased or partial annotations. The authors propose a principled approach to tackle this issue by integrating ranking constraints into the learning process. The method is intuitive and supported by solid theoretical analysis. Empirical evaluations demonstrate its effectiveness across multiple datasets. While there is room for improving the clarity of the presentation and broader comparisons, the paper makes a meaningful contribution and opens up new directions for improving robustness in LDL frameworks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.The paper proposes a novel method for semi-supervised label distribution learning (SSLDL), named RankMatch, which innovatively incorporates inter-label ranking correlations. This is a significant contribution, as ranking-based information has been largely underexplored in previous SSLDL approaches. By leveraging label ranking structures, the method enhances the flexibility and robustness of pseudo-labeling in weakly supervised scenarios.\n2.The proposed method demonstrates broad applicability, particularly in domains where label ambiguity is inherent—such as facial emotion recognition or crowd counting. Its ability to effectively utilize both labeled and unlabeled data reduces reliance on fully labeled datasets, thus improving the practicality and scalability of SSLDL in real-world applications.\n3.The paper is well-organized and clearly written. The theoretical development is presented in a logical and gradual manner, and the experimental results are comprehensive, using well-established metrics to demonstrate the method’s superiority. The empirical design is thoughtful, and results are easy to interpret.\n\nWeaknesses:\nThe paper would benefit from providing the implementation code of RankMatch. Open-sourcing the code would promote reproducibility and enable other researchers to validate and build upon the proposed method, thereby increasing its impact within the community.\n\nQUESTIONS:\nQ1. Why did the authors not include commonly used multi-label evaluation metrics such as mean Average Precision (mAP) or Hamming Loss in the performance evaluation? These metrics might better reflect the model’s behavior in ranking or partially correct predictions.\n\nQ2. In Equation (3), a ranking margin \\delta is introduced. How is this hyperparameter chosen? Is the same value of \\delta used across all datasets? Is the model performance sensitive to this margin?\n\nQ3. The paper mentions filtering pseudo-labels based on ranking consistency (Eq. 5). Could the authors clarify the specific criterion used for filtering? For example, is a threshold applied, and if so, has it been tuned or validated experimentally?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a critical yet underexplored problem in label distribution learning (LDL)—namely, the challenge of ranking preservation under weak supervision. Although this issue inherently exists in LDL scenarios, it has been largely overlooked by prior work. The proposed method presents an elegant solution by introducing a novel ranking-based formulation, which is both theoretically grounded and empirically validated. The experimental results are promising, and the theoretical insights enhance the method’s credibility. Given the clarity of the motivation and the demonstrated effectiveness, this work is a meaningful step forward and has the potential to influence future research in the area.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n• The paper proposes a novel ranking-based framework, *RankMatch*, to address label distribution learning, which offers a fresh perspective compared to traditional regression-based objectives.\n\n• The use of pairwise label ranking allows the model to capture relative importance among labels, aligning well with the goal of label distribution modeling.\n\n• The method is simple yet effective, and achieves consistent improvements across multiple benchmark datasets.\n\n• The writing is clear and the motivation is well established.\n \n**Weaknesses**:\n\nThe method section is divided into too many small subsections, which slightly affects the overall readability. A more concise structure with fewer headings may improve clarity.\n\nQUESTIONS:\n1. Where is the Related Work section located? It would be helpful to clarify whether related work is integrated into the introduction or presented as a standalone section.\n\n2. Some notations (e.g., in Eq. 3 and Eq. 5) could benefit from brief clarification for readers unfamiliar with the setup.\n\n3. While the paper motivates the use of label distribution learning (LDL), it would be helpful to elaborate on specific real-world tasks where the distributional supervision provides a clear benefit.\n\n4. How sensitive is the method to the choice of hyperparameters? While exact tuning is not necessary, a short discussion would be useful.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RankMatch, a semi-supervised learning framework for label distribution learning (LDL) problem using pseduo-labels. As existing pseudo-labels fails to consider the ranking relationships between different true labels, RankMatch addresses this issue by including the ranking correlations between labels into the learning process. Specifically, the paper proposes a Pairwise Ranking Relation (PRR) Loss, which improves the pseudo-labels to capture the ranking correlations between labels. In addition, a flexible pseudo-label training stratygy is proposed to prevent the model from overfitting to the absolute label differences. Experiments are conducted on several LDL benchmarks and showed competitive results on the ranking-based metrics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThis paper first improves the pseudo-labeling strategy of existing semi-supervised learning algorithms in label-distribution learning scenario, by introducing the ranking correlations between labels. This approach seems to be original, and is effective to improve the pseudo-labeling mechanism from the empirical results. In addition, the ranking correlation is supported by the theoretical gaurantee, making the proposed method plausible. \n\n\nWeaknesses\n\n- All the considered benchmarks are small-scale and not standardized, and the proposed method needs to be evaluated on more large-scale and commonly used benchmarks in multi-label classification and semi-supervised learning literature, such as Pascal VOC, COCO, and OpenImages for multi-label classificaiton and CIFAR10/100 and STL10 for semi-supervised learning. \n\n- The considered evaluation metrics include only correlation-based ones, and accuracy related metrics such as top-1 accuracy, precision/recall, and mAP are not considered, which are commonly adopted in single/multi-label classification domains. \n\n- The proposed method which utilizes the relationships across different classes has underlying assumption that the class distribution of labeled and unlabeled data are identical. As it is more natural that unlabeled data often violate the condition, it is unclear whether the proposed method can be robust under such a condition.\n\nQUESTIONS:\n- Please see the weaknesses above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies semi-supervised label distribution learning (SSLDL), a generalization of semi-supervised multi-label classification in which a small set of examples has dense label distributions and a larger set is unlabeled. The goal is to learn to predict full label distributions for new samples. The authors argue that pseudo-labeling, when combined with similarity-based distribution matching (e.g. CE, KL-div), while very effective for SSL, falls short for SSLDL because it fails to learn ranking correlations between labels. To address this issue, the authors introduce RankMatch, a deep-learning framework based on consistency regularization and pairwise label ranking. The main methodological contribution is the Pairwise Relevance Ranking (PRR) loss, which has two components: (1) The supervised PRR term aligns predicted and ground-truth label rankings, preserving relative class margins.  (2) The unsupervised PRR term penalizes incorrect ranking relationships on pseudo-labeled data, without enforcing margin constraints. Pseudo-labeled distributions are generated via an ensemble of different augmented “views” of each sample. In addition, the authors theoretically show that PRR reduces the generalization error and provide a generalization bound for the proposed method. Extensive empirical evaluation on multiple datasets and supervision levels demonstrates that RankMatch outperforms adapted baselines in both distribution alignment and ranking quality.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n- Addresses the relevant SSLDL task, reducing annotation costs to train with dense distributions while leveraging modern deep architectures.\n\n- Introduces PRR losses that effectively tackle ranking alignment under supervised and unsupervised regimes.\n\n- Provides theoretical generalization bounds that connect PRR minimization to error reduction.\n\n- Demonstrates consistent empirical gains in diverse datasets and supervision regimes, showing improvements in the quality of both predicted distributions and label rankings.\n\n**Weaknesses**:\n\n- W1 - Hyperparameters (λ and t) exhibit sensitivity; tuning under very limited labeled data may be unstable or unfeasible, with unclear requirements for validation data.\n\n- W2 - Most baselines are adapted from related but distinct tasks (e.g. supervised distribution learning, semi-supervised multi-label learning); comparison to the latest SSLDL-specific methods (e.g., [21, 31]) is missing.\n\n- W3 - The ablation study does not disentangle the effect of the supervised and unsupervised components of the PRR loss.\n\nQUESTIONS:\n## Questions for the Authors\n\n- **Q1: Relation to sharpening methods:** Is the absence of entropy minimization the key difference between the unsupervised consistency loss (Sec. 2.2.1) and methods like FixMatch or MixMatch (the latter also uses ensemble-based soft pseudo-labels)?\n\n- **Q2: Definition of δ (L102)**: The paper currently defines\n\n  ```math\n  δ = d_{x_i}^{y_k} − d_{x_k}^{y_j}.\n  ```\n\n  Would it be more consistent to set\n\n  ```math\n  δ = d_{x_i}^{y_j} − d_{x_i}^{y_k}\n  ```\n  to compare distances to two labels for the same input?\n\n* **Q3: Hyperparameter tuning.**\n  Can you provide practical guidelines or heuristics for selecting λ and t under very limited labeled data? What minimum validation-set size ensures stable tuning?\n\n* **Q4: Additional baselines.**\n  Could you evaluate against recent SSLDL-specific approaches [21, 31]? Including such comparisons could clarify any performance gap.\n\n* **Q5: Complexity analysis.**\n  Can you comment on the computational overhead of ensemble pseudo-label generation, especially for large-scale datasets?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}