Title: Entropy-Calibrated Label Distribution Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: label distribution learning

learning with ambiguity

entropy calibration
Abstract: Label Distribution Learning (LDL) has emerged as a powerful framework for estimating complete conditional label distributions, providing crucial reliability for risk-sensitive decision-making tasks. While existing LDL algorithms exhibit competent performance under the conventional LDL performance evaluation methods, two key limitations remain: (1) current algorithms systematically underperform on the samples with low-entropy label distributions, which can be particularly valuable for decision making, and (2) the conventional performance evaluation methods are inherently biased due to the numerical imbalance of samples. In this paper, through empirical and theoretical analyses, we find that excessive cohesion between anchor vectors contributes significantly to the observed entropy bias phenomenon in LDL algorithms. Accordingly, we propose an inter-anchor angular regularization term that mitigates cohesion among anchor vectors by penalizing over-small angles. Besides, to alleviate the numerical imbalance of high-entropy samples in test set, we propose an entropy-calibrated aggregation strategy that obtains the overall model performance by evaluating performance on the low-entropy and high-entropy subsets of the overall test set separately. Finally, we conduct extensive experiments on various real-world datasets to demonstrate the effectiveness of our proposal.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper points out that in the current field of label distribution learning, the existing algorithms underperform on low-entropy samples, and the existing evaluation metrics of label distribution learning cannot adequately capture the prediction performance of samples with different levels of entropy. Aiming at these two problems, this paper theoretically analyzes the underlying reasons for the underperformance on low-entropy samples, designs a regularization term according to the theoretical results, and proposes a novel evaluation method for label distribution learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: 1) This paper presents the problem of underperformance on low-entropy samples and the bias of evaluation metrics in existing LDL studies, which are widespread and have serious impacts on the downstream decision-making tasks of LDL because low-entropy samples usually harbor expensive misjudgement costs. 2) This paper analyzes the causes of the underperformance problem of existing LDL algorithms on low-entropy samples from the theoretical and empirical viewpoints. The theoretical and visualization results reliably reveal the relationship between the hypothesis space and the low-entropy underperformance of the widely-adopted softmax model, which is instructive for the design of subsequent LDL algorithms. 3) This paper proposes a regularization term based on the results of the theoretical analysis, gives several mathematical properties of the regularization term, and verifies the validity of the regularization term through extensive experiments. 4) This paper proposes a new evaluation method that can effectively capture the prediction performance of samples with different levels of entropy.\n\nWeaknesses: 1) The experimental configurations of this paper are not adequately shown, such as the specific configuration of the optimizer used and the iteration stopping conditions. 2) The writing of the paper could be further improved, such as the mismatch of left and right parentheses in line 196, the lack of period at the end of the sentence in line 197, and in the captions of the subfigures in Fig. 5, Fig. (b) and Fig. (c) show the entropy mean and the entropy variance of the dataset but Fig. (a) shows only the mean value. The paragraph subheadings before line 267 end with a period, while the paragraph subheadings after line 267 end without a period, which is not uniform in format. The dataset name in the image caption in Figure 1 does not match the dataset name in the caption.\n\nQUESTIONS:\n1. In order to make the anchors as dispersed as possible, why not just minimize the inner product of the anchors instead of minimizing the cosine similarity of the two anchors? After all, cosine similarity imposes a greater computational burden compared to inner product?\n2. Are there some empirical suggestions for hyperparameter selection?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

comments: SUMMARY:
This paper addresses the problem of entropy bias in Label Distribution Learning (LDL), where models tend to perform poorly on low-entropy samples due to overly cohesive anchor vectors. To mitigate this, the authors propose an Inter-anchor Angular Regularization (IAR) term that penalizes anchor vectors with small angular separation, thereby enhancing the model’s ability to represent low-entropy samples. Additionally, they introduce an Entropy-Calibrated Aggregation (ECA) strategy for fairer model evaluation by separately assessing performance on low- and high-entropy subsets. Experimental results demonstrate the effectiveness of the proposed methods.

STRENGTHS AND WEAKNESSES:
Strengths

1.	The paper tackles the entropy bias in LDL, which is interesting.

2.	The proposed method is built upon both empirical and theoretical analyses, and is well-motivated.

3.	This work also proposes a new evaluation mechanism termed ECA for fairer model evaluation, which sounds reasonable.

Weaknesses

1.	In line 42, “the samples with high-entropy” should be “the samples with low-entropy”.

2.	For better clarity, it would be beneficial to mention the term “ECA” in Section 4.

3.	In the main text, it would be beneficial to reference the Appendices explicitly. For example, after presenting Theorem 3.1, a note should be added indicating that its proof is provided in Appendix A.

QUESTIONS:
1.	What is the form of the function $f (\cdot)$? Does it use only anchor vectors as classifier weights?

2.	Are the anchor vectors treated as learnable parameters? It would be helpful to clarify the background of LDL.

LIMITATIONS:
The limitations are included in Section 6.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-22

comments: SUMMARY:
The authors demonstrate that many Label Distribution Learning (LDL) methods tend to make significantly larger errors on low-entropy instances compared to high-entropy ones, a disparity that standard aggregate metrics often obscure. To address this, they present Theorem 3.1, which links the cohesion—measured by small pairwise angles—of anchor vectors in softmax models to a lower bound on the entropy of the predicted label distributions. This theoretical insight helps explain why low-entropy samples are particularly challenging to model accurately. To mitigate this issue, the authors introduce an Inter-Anchor Angular Regularizer (IAR) that penalizes overly small angles between anchor vectors, encouraging more dispersed and expressive representations. Additionally, they propose a new evaluation metric, Entropy-Calibrated Aggregation (ECA), which separately averages performance on low- and high-entropy test subsets to provide a more nuanced assessment. Experimental results across eight benchmark datasets using four recent LDL baselines plus ridge regression, along with ablation studies (“ER”, “EW”) and a hyperparameter analysis, show that their approach yields substantial improvements on low-entropy samples without degrading performance on high-entropy ones. Overall, their method achieves the best ECA scores on six out of eight datasets.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Connecting anchor geometry to entropy bias is simple yet convincing; the angle-entropy bound is intuitive and mathematically clean.
2. Clear assumptions and and full proofs

Weaknesses:
1. All baselines are non-deep or shallow deep models. Demonstrating IAR on a modern backbone (e.g., ViT-LDL) would strengthen the claim of universality.

QUESTIONS:
See W1

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-28

comments: SUMMARY:
This paper identifies and addresses the "entropy bias" in Label Distribution Learning (LDL), where existing algorithms perform poorly on low-entropy samples, which are crucial for decision-making. The authors propose two key innovations:

- Inter-anchor Angular Regularization (IAR): A regularization term that penalizes overly cohesive anchor vectors to better capture low-entropy distributions.

- Entropy-Calibrated Aggregation (ECA): An evaluation strategy that balances performance assessment across low- and high-entropy test samples.

The paper offers both theoretical analysis (including an entropy lower bound theorem) and extensive experiments on eight datasets, demonstrating consistent performance gains, especially on low-entropy samples.

STRENGTHS AND WEAKNESSES:
Strengths:

1. Novel identification of entropy bias in LDL, a previously overlooked but practically important issue.
2. Strong theoretical analysis supporting the IAR formulation, with well-structured theorems and proofs.
3. Comprehensive experiments, including multiple baselines, evaluation metrics, ablations, and significance testing.

Weaknesses:

1. The ECA evaluation section is mathematically overloaded and may benefit from simplification or partial relocation to the appendix.
2. There is limited discussion on applicability to non-anchor-based architectures such as Transformer-based LDL models.

QUESTIONS:
1. Can IAR be extended to architectures without explicit anchor vectors, such as attention-based or Transformer models?
2. How sensitive is the performance to the choice of entropy threshold in ECA? Would a learned threshold work better?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper finds that excessive cohesion between anchor vectors contributes significantly to the observed entropy bias phenomenon in LDL algorithms. It accordingly proposes an inter-anchor angular regularization term that mitigates cohesion among anchor vectors by penalizing over-small angles. To alleviate the numerical imbalance of high-entropy samples in the test set, it proposes an entropy-calibrated aggregation strategy that obtains the overall model performance by evaluating performance on the low-entropy and high-entropy subsets of the overall test set separately. Experiments demonstrate the effectiveness of the proposed method.  The main contributions of this paper are:\n\n- It analyzes the generation mechanism of entropy bias from both empirical and theoretical perspectives, and consequently proposes an assumption that the underperformance of LDL models on low-entropy samples is significantly driven by the cohesion of anchor vectors.\n\n- It proposes IAR (i.e., an Inter-anchor Angular Regularization term) to penalize the anchor vectors with over-small angles.\n\n- It proposes ECA (i.e., an Entropy-Calibrated Aggregation strategy) to calculate the overall model performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The problem studied in this paper is interesting and valuable.\n- The paper is well-organized, which is easy to follow.\n- The theoretical work improves the value of the paper.\n\n\n\n**Weakness**\n\n- The paper primarily uses KL divergence and cosine similarity as evaluation metrics. Although these two metrics can measure the differences between the model outputs and the true label distribution from different perspectives, they may not fully reflect the model's performance in real-world applications. For instance, in tasks that require accurate prediction confidence or uncertainty estimation, relying solely on KL divergence and cosine similarity may be insufficient to thoroughly assess the model’s reliability and effectiveness.\n\n- Although the paper demonstrates the stability of the results through multiple randomized experiments and reports the mean and standard deviation, the analysis of the statistical significance of performance differences across different datasets and algorithms is not sufficiently thorough. For example, on some high-entropy datasets, the performance gap between IAR and other algorithms is not significant, while on low-entropy datasets, IAR shows a noticeable performance gap compared to some SOTA algorithms. These issues are not adequately explored or explained. The authors should further analyze the underlying reasons for these phenomena.\n- The impact of the hyperparameter $\\alpha$ on model performance varies across different datasets. The authors should further analyze the underlying reasons for this variation and provide practical recommendations for its application.\n\nQUESTIONS:\n1. How about the performances of different algorithms on other LDL metrics (e.g., Chebyshev distance, Intersection similarity, etc.)?\n\n2. On some high-entropy datasets, the performance gap between IAR and other algorithms is not significant, while on low-entropy datasets, IAR shows a noticeable performance gap compared to some SOTA algorithms. The authors should further analyze the underlying reasons for these phenomena.\n\n3. The impact of the hyperparameter $\\alpha$ on model performance varies across different datasets. The authors should further analyze the underlying reasons for the hyperparameter $\\alpha$ on model performance varies across different datasets, and provide practical recommendations for its application.\n\nLIMITATIONS:\nThe authors could further discuss the potential societal impact of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}