Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies Label Enhancement (LE). Instead of enhancing label distribution from logical labels, it proposes to recovery label distribution from multi-label ranking annotations. To achieve that, the authors establish several theories and put forward a new method. Experimental results validate the advantages of recovering label distribution from ranking over logical labels. 
strengths_and_weaknesses: On one hand, the strengths include:
+ LE with ranking annotation is original, which hasn't been noticed in the field of LE. Besides, the authors theoretically prove that LE from ranking is better than LE from logical labels, as shown in corollary 4.
+ The method DRAM is novel. DRAM uses the Dirichlet mixtures and EM algorithm to recover label distribution from rankings. 
+ Most importantly, the experiments show that DRAM has remarkable improvements over existing LE (from logical labels). Moreover, the DRAM+LN combination even outperforms SA and DM with the ground-truth label distributions, which is impressive. 

On the other hand, for me, the main weakness mainly lies in that the paper is not easy to follow. I understand that the authors have put necessary proofs and details to the supplementary material, but still the Sections 2 and 3 are not easily understandable. Besides, I expect to see the results in terms of KL divergence because it is such an important measure metric for LDL. 
questions:  I expect to see the results in terms of KL divergence in the rebuttal. 
limitations: I believe the authors have clarified the limitations of their paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This article propose a generic framework named DRAM(label Distribution predicting from
multi-label RAnking via conditional Dirichlet Mixtures). It allows to flexibly encode the prior knowledge about the tasks by a scoring function, and it integrates the processes of recovering and learning label distributions end-to-end. Besides, the author theoretically investigate the relation between multi-label ranking and label distribution and define the notion of EAE to quantify the quality of an annotation, and give the bounds of EAE for multi-label ranking.

strengths_and_weaknesses: Generally speaking, the article has a certain degree of innovation and the theoretical results are plentiful. The structure of the article is complete and the thinking is clear. Therefore, we hold the opinion of weak reception.
1. In Definition 1, the author states that $z$ and $\hat z$ are independent, but from Corollary 1 it can be seen that ${\delta }$ and ${\hat \delta }$ are closely related, so the independence of  $z$ and $\hat z$ needs further explanation.


questions: 2. It can be seen from Corollary 1 that the optimal solution of  ${\hat \delta ^*} = ((2m + 1)\delta  + m){(m + 1)^{ - 2}}$ does not approach  ${\delta }$, whether this shows the capability boundary of the method? In addition, the process of theoretically guiding the algorithm is mainly reflected in the update of ${{\hat \delta } _n} = \left| {{\sigma _n}} \right|{(\left| {{\sigma _n}} \right| + 1)^{ - 2}}$. From the Corollary 3, ${\hat \delta }  \in [m{(m + 1)^{ - 2}},{m^{ - 1}}]$, why does  ${\delta }$ take the lower bound of the interval instead of a certain value in the middle ?

limitations: 3. When analyzing how the number of mixture components K affects the performance of the method, the authors point out that “t can be seen that the performance gets better first and then worse as K increases.”, it seems that such a conclusion cannot be simply drawn from the figure.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies how to exploit multi-label ranking, a compromise w.r.t. annotation cost, to learn a predictive model on label distribution. The authors theoretically investigate the relation between multi-label ranking and label distribution and demonstrate its superior over the logical labels. Then a general framework involving custom knowledge is proposed to recover and learn label distribution end-to-end. The extensive experiments show the effectiveness of the proposal.
strengths_and_weaknesses: This is the first paper that proposes to predict label distribution from the view of multi-label ranking and shows the good guarantees for performance. The originality is good and the main idea of this paper is clearly presented. They also show better results than other compared methods. Despite that, I still have some concerns:
1. What is the time cost of the EM algorithm used to train model? The authors should add more analysis on time complexity.
2. Note that the datasets used in this paper are relatively small, so what is the capability of the proposed framework on large datasets? In fact, label ranking is often associated with high time complexity.
3. Are there any selection criteria for scoring function? Note that only DRAM+LN is tested on all datasets and how do different scoring functions affect the results? 
4. Multi-label ranking may still require a high cost. The authors can add some discussion on cost of various methods (the proposal, LE and LDL methods), for example, the cost comparison when achieving the same performance.
5. Some spelling mistakes need to be corrected, e.g., “internel” in the third sentence in section 2.1 should be “internal”.


questions: 1. What is the time cost of the EM algorithm used to train model? The authors should add more analysis on time complexity.
2. Note that the datasets used in this paper are relatively small, so what is the capability of the proposed framework on large datasets?
3. Are there any selection criteria for scoring function? Note that only DRAM+LN is tested on all datasets and how do different scoring functions affect the results? 
4. Multi-label ranking may still require a high cost. The authors can add some discussion on cost of various methods (the proposal, LE and LDL methods), for example, the cost comparison when achieving the same performance.

limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper aims at predicting label distribution from multi-label ranking. It is different from the existing label distribution learning (LDL) and label enhancement (LE), where LDL learns a predictive mapping on training set with label distributions annotations and LE generates label distributions from instances with logical labels. This work is a compromise w.r.t. annotation cost but has good guarantees for performance.
strengths_and_weaknesses: Strengths:
1. This paper has a good motivation.
2. This paper proposed a framework of label distribution predicting from multi-label ranking.
3. Some experimental results validate the superiority of the proposed method.

Weaknesses:
1. It seems that the theoretical analysis has little relatedness with the proposed framework and there are not experiments to support the theoretical results. It is something that looks beautiful. 
2. The trade-off parameter $\lambda$ is selected from a very wide range by five-fold cross-validation. Is there a good default choice? Parameter sensitivity should be made.  Besides, five-fold cross-validation is conducted over the whole data set or only the 70% data for training?

questions: I have listed some weaknesses above. Here, my further concern is why the general LDL data sets are not used in experiments. It is very easy to construct data sets with label rankings and with binary labels from LDL data set. The 15 LDL data sets release by Geng can be found at: http://palm.seu.edu.cn/xgeng/LDL/index.htm
limitations: Authors points out two limitations in Section 6. For the 2nd one, i.e., "If several labels actually describe the instance to the same degree, then requiring experts to give the strict order of these labels may lead to errors and invalidate Theorem 1", authors should extend the proposed framework to deal with such kind of data sets, not treat it as a future work. After all, it is more realistic that some labels have the same description degree.
review_title: null

Meta-review: metareview: This paper studies the problem of predicting label distribution from multi-label ranking. First, the authors give a theoretical analysis to prove the superiority of the multi-label ranking over the logical labels. Then an end-to-end framework called DRAM is proposed for recovering and learning label distributions. The corresponding experiments validate the effectiveness of the proposed algorithms. Overall, this work is technically solid. The concerns raised by reviewers are not so serious and have been answered. I recommend accepting this paper.
confidence: Less certain
Decision: Accept