Predicting Label Distribution from Multi-label Ranking
Authors: Yunan Lu
Xiuyi Jia
Venue: NeurIPS 2022 Accept (2022)
Keywords: label distribution
label polysemy
multi-label ranking

Abstract: Label distribution can provide richer information about label polysemy than logical labels in multi-label learning. There are currently two strategies including LDL (label distribution learning) and LE (label enhancement) to predict label distributions. LDL requires experts to annotate instances with label distributions and learn a predictive mapping on such a training set. LE requires experts to annotate instances with logical labels and generates label distributions from them. However, LDL requires costly annotation, and the performance of the LE is unstable. In this paper, we study the problem of predicting label distribution from multi-label ranking which is a compromise w.r.t. annotation cost but has good guarantees for performance. On the one hand, we theoretically investigate the relation between multi-label ranking and label distribution. We define the notion of EAE (expected approximation error) to quantify the quality of an annotation, give the bounds of EAE for multi-label ranking, and derive the optimal range of label distribution corresponding to a particular multi-label ranking. On the other hand, we propose a framework of label distribution predicting from multi-label ranking via conditional Dirichlet mixtures. This framework integrates the processes of recovering and learning label distributions end-to-end and allows us to easily encode our knowledge about current tasks by a scoring function. Finally, we implement extensive experiments to validate our proposal.

Predicting Label Distribution from Multi-label Ranking
Yunan Lu, Xiuyi Jia"
School of Computer Science and Engineering Nanjing University of Science and Technology, Nanjing 210094, China {luyn, (aaxy)@nuust edu.
Abstract
Label distribution can provide richer information about label polysemy than logical labels in multi-label learning. There currently strategies including LDL (label distribution learning) and LE (label enhancement) to predict label distri- butions. LDL requires experts to annotate instances with label distributions and learn a predictive mapping on such a training set. LE requires experts to annotate instances with logical labels and generates label distributions from them. However, LDL requires costly annotation, and the performance of the LE is unstable. In this paper, we study the problem predicting label distribution from multi-label ranking which is a compromise w.r.t. annotation cost but has good guarantees for performance. On the one hand, we theoretically investigate the relation between multi-label ranking and label distribution. We define the notion of EAE (expected approximation error) to quantify the quality of an annotation, give the bounds of EAE for multi-label ranking, and derive the optimal range of label distribution corresponding to a particular multi-label ranking. On the other hand, we propose a framework of label distribution predicting from multi-label ranking via conditional Dirichlet mixtures. This framework integrates the processes of recovering and learning label distributions end-to-end and allows us to easily encode our knowl- edge about current tasks by scoring function. Finally, we implement extensive experiments to validate our proposal.
Introduction
The label polysemy problem has been a popular research topic in machine learning area, in which an instance is described by multiple labels simultaneously. MLL (multi-label learning) 23 deals with label polysemy by assigning vector with logical values to the instance, in which each logical value indicates whether the corresponding label associated with the instance. However, MIL only gives which labels can describe the instance, but cannot directly answer a question with more polysemy, ie.. how much does each label describe the instance. Hence, label distribution 4 a real-valued vector that explicitly gives the description degrees of labels to instance, is introduced to answer this question. Obviously, label distribution provides richer information about label polysemy than logical labels, and it has been applied in many practical application scenarios, such as sentiment 30.. and so on.
analysis 14 35| 39], facial age estimation
There are two main methods for obtaining label distribution to characterize label polysemy. The first type is LDL (label distribution learning) 4. that is, learning a predictive mapping from feature vector to a label distribution. LDL. requires experts to directly annotate the instances with label distributions as a training set. Such methods focus on how to design well performed L.D. algorithm. 38 that improve LDL performance by mining
Typical works include some algorithms [00 20 25 37
Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

label correlations, and some [2 31 that improve the ability to fit complex label distributions by introducing more flexible models. The second type is LE (label enhancement) [4 that is, generating label distributions from vector of logical labels. L.E requires experts to annotate the instances with logical labels first, and then recover the label distribution by analyzing the features and labels of the training instances, finally train predictive model by these recovered label distributions. In other words, LE can be regarded as the pre-processing of LDL to obtain the label distribution. Such methods focus on how to recover the true label distribution accurately from the given logical labels. For example, most LE algorithms [12 22 29 34, 40 consider recovering more accurate label distributions by mining sample and label correlations.
Since LDL methods are trained directly on instances with true label distributions, they usually produce better performance. However, annotating instances with label distributions is costly and even impractical in some cases (34] In contrast, LE methods only require experts to annotate the instance with logical labels, thus reducing the annotation cost. However, there is no reliable theory to guarantee that the label distribution recovered from logical labels converges to the true label distribution. In terms of the annotation form, two labels taking the same logical value are indistinguishable in the MLL case, while the label distribution usually characterizes them as labels with different description degrees. In terms of solution, logical labels provide a large solution space for the label distribution (e.g.. the label distribution component corresponding to a label with a logical value of 1 can take any real value from 0 to 1), making the solution unstable and inaccurate. Therefore, we propose hypothesis that the label distribution recovered from logical labels is not guaranteed to have the same label ranking the true label distribution, while accurate label ranking is the key to recovering and predicting an accurate label distribution
26 27
Fortunately, multi-label ranking 1 2 7 is good annotation form to address the above problems. Multi-label ranking requires experts to give which labels are relevant to the instance and further to give the ranking (strict order) of these relevant labels. Although the annotation cost of multi-label ranking is slightly higer than that of logical labels, it guarantees a ranking consistent with the true label distribution and constrains the approximated label distribution in narrow solution space. Hence, in this paper, we investigate the problem of predicting label distribution from multi-label ranking.
On the one hand, we theoretically investigate the relation between multi-label rankings and label description vector (unnormalized label distributions). We define the notion of EAE (expected approximation error) to quantify the quality of annotation w.r.t. recovering the true label description vector; we derive the EAE for multi-label ranking and logical labels to clarify the advantage of multi-label ranking; we give the bounds of EAE for multi-label ranking, and derive the optimal range of label description vector corresponding to a particular multi-label ranking.
On the other hand, we propose a generic framework named DRAM (label Distribution predicting from multi-label RAnking via conditional Dirichlet Mixtures). This framework first forms eemi-adaptive prior p" (d) for the label distribution via scoring function with a predefined functional form and adaptive parameters, then models the predictive distribution p(d[z) (x) by conditional Dirichlet mixtures, finally learns the model parameters by minimizing the cross-entropy of p(d]æ= relative to p (d). This framework has two merits: 1) allows us to flexibly encode our prior knowledge about the tasks by a scoring function, and 2) it integrates the processes of recovering and learning label distributions end-to-end. Besides, we design a comparison method whose main idea to transform the dataset with multi-label rankings into the dataset with logical labels such that any existing LE method can be borrowed. Finally, to validate our proposal, we conduct experiments on reduced LDL datasets and a new real-world dataset that we create directly according to the task. The experimental results show that our method significantly outperforms the comparison methods and also outperforms the LDL methods directly trained on the examples with true label distribution in most cases.
In some literature, "multi-label ranking" is learning task; in this paper. it only denotes an annotation form.
'In general, label distribution d satisfies that each element d; E [0, 1] and Σ d: 1. For simplicity. we sometimes do not consider Σ d; 1. We call such an unnormalized label distribution a label description vector.
2

---

2 Theoretical analysis
2.1 Preliminary
Let a: denote the feature vector of the instance and y = {ye Ji denote the label set. The label description vector is the expert's internal view of how much does each label describe the instance; 2, E [0, 1] indicates the description degree of yi to æ: If the expert is asked to annotate the instance with logical labels, the internal label description vector will be degenerately expressed as a logical label vector & E {0, 1 the element 4 = 1 (or 4 0) in I means that the label yi is the relevant (or irrelevant) the instance æ. Let denote the number of relevant labels. If the expert is asked to annotate the instance with a multi-label ranking, the internal label description vector will be degenerately expressed as permutation o (which represents total strict order) on the relevant labels; indicates that the label Vo. is at the i-th position in ascending order of the description degree; for j E [M] a, the label yj is an irrelevant label for æ, where [M] - {1, 2, , .M.
Since both logical labels and multi-label ranking are reduced versions of the internal label description vector, there is consistency between them, which can be described in the following two assumptions:
logical label vector I, and ā > 0 the minimum margin F of label description degrees, then we have z E St, where S = { (Vl; = 0, 24 = 0) A (7l = 1,6 S 23 S 1)}.
Assumption If the expert's 's internal label description vector is expressed
Assumption 2 If the expert's 's internal label description vector z is expressed as multi-label ranking 0, and 0 is the minimum margin of label description degrees, then we have 2 Sg. where S f= (Vi E o, 8 S 2, $ 1) A (Vi [m Zo: S 20.41 8) (Vi [M]] 0)).
Note that the margin 5 is an implicit variable in the annotation process, which is not explicitly indicated by the annotation results. Therefore, the range of internal label description vector (e.g. S and 5 is also implicit, so the range determined by the implicit interval 5 is called the implicit range. Although is implicit, we can predefine an explicit margin o; the range determined by the explicit margin (e.g. s and st) is called the explicit range. We are then able to generate a label description vector from the explicit range approximate the internal label description vector.
2.2 Theoretical results
We first define EAE to measure the quality of a certain annotation form w.r.t. approximating the internal label description vector.
Definition 1 Suppose that an instance is annotated with an annotation r; ō and å are implicit and explicit margins, respectively; St. and S are the implicit and explicit ranges, respectively. Then the expected approximation error of to the internal label description vector is
8,3 JEess EES VOVA
|| - #|[dzdd
V =
dz,
V
dē. 2ESS
(I)
SESS
Eq. is essentially derived from the expectation of the squared Euclidean distance between z and &, i... E |||= *1 which measures the average distance between the estimated label description vector and the internal one given the ranges St and SÇ. Since z and & are independent, p(z, 2) p()) Besides, we do not consider additional assumptions to reduce the uncertainty of z and é; we assume that and 2 follow the uniform distributions on St. and S². i.e., p(z)p(2) = (V;5V;)) and derive the Eq. (I. Next, we give the EAE of multi-label ranking.
Theorem If an instance is annotated by multi-label ranking labels, and are the implicit and explicit margins, respectively, then the EAE of a is
is the number of relevant
(m 1) + 82) 2m(6 + (4m + 2)0ō 2).
6(m
'Note that it is hard for an expert to rank a set of labels with close description degrees. We hence introduce the minimum margin& 8 to represent the smallest difference of description degrees required for the expert to distinguish and rank these labels.
3

---

0.15
upper bound mean value
0.10
0.05
0.00
10
15
10
15
(a) Upper bound and mean value of e /m.
(b) Lower bound of 9
5.5 5.8
ca
Figure 1: Visualization of corollaries. The horizontal coordinates of the figures above indicate the number of relevant labels m.
Theorem can be proved by mathematical induction, and the details can be found in the appendix. Before giving more corollaries, need specify the range the margins, i.c., and 8.
Lemma If an instance is annotated by mullii-labe ranking then the margins and satisfy that 0 < 8 v m and 0 < S m
Next we give some interesting corollaries to understand Theorem
Corollary If an instance is annotated by a multi-label ranking a, m is the number of relevant
labels, the explicit margin 8* minimizing the EAE of is 8- = ((2m 1)6 + m)(m + 1)-?
clear that the optimal explicit margin 8+ depends on the implicit margin o; hence we cannot obtain an exact optimum for 8*. Nevertheless, Corollary helps us to narrow down the range of the optimal explicit margin considerably, i.e., m(m + 1)- <m-1.
Corollary If an instance is annotated by a multi-label ranking a, m is the number of relevant labels, 0 s 8 S m- m(m + 1)-2 < 3 m-1, then the FAF of a is bounded by:
0 < c m(m² + 4m + 2)
(3)
6(m 1)3
5
Corollary 2 gives mn-dependent bounds on the EAE of the multi-label ranking. See the appendix for details of the proof.
Corollary If an instance is annotated by a multi-label ranking o,, m is the number of relevant labels, and are uniform over [0, 10 and [m(m + 1 m m- respectively, then we have:
2m + 8m3 + 8m2 4m +1
5.5
(4)
36m(m 1)3
6,8
Corollary 3 can be obtained by simple integral calculation, as detailed in the appendix. In Fig. ((a]
we visualize the expected value in Corollary 3 and the upper bound in Corollary 2 It is obvious that can obtain significant performance gain by ranking small number of labels.
Theorem If an instance is annotated logical label vector I is the number of relevant labels, ò and 8 are the implicit and explicit margins, respectively, then the EAE of is
8,5 m (28 + G
8 1).
(5)
6
Theorem Sannremmeen gives the EAE of the logical label vector, and the proof is detailed in the appendix.

---

Corollary Suppose that & 3, and hho are the EAE of the logical label vector and the EAE of the multi-label ranking respectively, we have the following inequality holds for m 2 3:
6.6L
a
5.b
2
7m
48
(82
26)
m mmm - 1)m7 2 + 20m + 9)
48 + 1)3
7m m 46m 30m² + 7m + > 0.
48m(m 1)a
Corollary shows the advantage of the multi-label ranking over the logical labels w.r.t. approximating the true label description vector. It is obvious that as the number of relevant labels m increases,
8.8 8.6 increases least the rate of O(m), which visualized in Fig. 1(b)
3 Algorithms
In this section, we consider how to train model on the dataset {(e,, )n for predicting label distributions. We propose a framework DRAM to deal with this problem. Besides, we also design comparison method.
3.1 DRAM framework
First, we describe how to enhance multi-label rankings into label distributions. Then, formally give the predictive model. Finally, we derive generic EM algorithm for our framework that works for any instantiation of the basic models and consider concrete instantiation.
3.1.1 Recovering label distributions from multi-label rankings
In order to learn a mapping from instance features to label distributions, we consider enhancing multi-label rankings to label distributions. The process of enhancing the simple label (e.g., multi-label ranking and logical label) into the label distribution can be viewed as selecting label distributions that satisfy predefined prior assumption from those consistent with simple labels. For example, some algorithms 99] (34 select those label distributions that satisfy the smoothness assumption [24]: In fact, according the no-free-lunch axiom, no prior assumption can work for all tasks; hence do not consider a concrete assumption in our framework. We provide a semi semi-adaptive scoring function o(d; 0) such that any assumption can be easily encoded. The scoring function allows us build prior distribution of label distribution p" (d))
Zp 2 (d: ( T(td E St
p² (d)
where S is the explicit range of Zp. is normalization constant, and II(-) denotes the indicator function. The integral term in Eq. (7) intuitively indicates how many label description vectors can be normalized to d. According to Corollary we set corresponding to the example (æ,, ox,) as Io,,,(( + 1)-2 The functional form of Φi d; 0) predefined and the parameters can learned adaptively. For example, we can predefine of d: 0) Gaussian likelihood function, and leave its mean and variance be learned.
3.1.2 Predictive model: conditional Dirichlet mixtures
Here we need to determine the distribution form of d conditioned on æ. We use Dirichlet distribution model dla. Since & is any non-negative real-valued function, the prior distribution of d usually multimodal. Therefore, we model p(d) xz) with the mixture of Dirichlet distributions:
K
p(d[z) Σ Dir(d
(8)
where a. 3 ......... 3 are learnable parameters; f(x; a) outputs K-dimensional positive real- valued vector with sum of and fa(æ; a)) is k-th value; f(x; @*) outputs a didimnssion positive real-valued vector; Dir(-) denotes Dirichlet distribution whose details are in the appendix.
5

---

Algorithm Generic DRAM
Require: training set (( o nn) ]== testing instance x", score function Ø number of mixture components K, number of Monte Carlo samples L;
1: at 0, B .... , < Initialize model parameters, converged do
2: while the likelihood
3: for - 1, 2, ... N 1,2, L do
4:
8n - In. +)
(1) - Generate sample uniformly from S
5:
6:
g(cn") Infer the posterior of latent variable cn in Eq. 9
7:
a, "..,,'"... 3K <- Update the model parameters as in Eq. (10):
8: +- Predict the label distribution for instance x" as in Eq.
9: return the label distribution d' for instance x²;
3.1.3 Learning algorithm
We consider minimizing the cross-entropy of p(d x)) relative to p=().. i.c. maximizing (d [In p(d[==)). Since p" (d) is usually complex distribution and (a) [In p(d]=)] involves the integration of p* (d), it is often intractable. Therefore, we approximate it by the importance sampling method (whose detailed derivation can found the appendix):
In [np(d/))] ~ Σ o(d") In Inp(d (x) (d)
d
20) = Z
2(0) Uni(z|()))
Ej o(
where ( Uni(z|St) denotes that (6) is sampled uniformly from S and 7 equals to the sum of all elements in (0) Since our model contains discrete latent variables, we use the EM algorithm to train the model. We introduce the variational distribution q(ch") and obtain
(6) In p( On Fa p((c (mn)
In (nd
KL (a(c)) ((cl)))((c Ml (d am))
((ch1)
e-posterior
ELBO (Evidence Lower Bound)
We then alternate between M-step (maximizing ELBO) and E-step (minimizing the c-posterior error). E-step:
Infer the posterior latent variables to minimize the e-posterior error:
fa( a Dir ( [f(x)i
1nn
(6) A g(cll
k)) Σ 2j fj (c,;; a Dir ( (f(xxi
h [K].
(9)
M-step: Update the model parameters to maximize the ELBO:
I
N
o 0)
K
(0) Tak In fa(xni a Dir d f(xni 3*))
arg max 0.0.91 .9
ΣΣ
1= :
Σ
(10)
7nk
Once the model parameters are learned, we can predict the label distribution for the test instance xx according Eq. (8) To evaluate DRAM, we take the expectation of the label distribution based on p(d[x=") as a deterministic output:
K d'
M
Σ
- Zx
(a) Bh), Za = E fil B²)
(1I)
The overall learning process shown in the Algorithm
3.1.4 Instantiation: DRAM with linear learner and noninformative scoring function
Here we consider concrete instantiation for our framework. For simplicity, use the noninforma- tive scoring function, i.c., o(d; 0) 1. f(æ; a) and f(x; 3²) modelled linear models with

---

the softmax and softplus activation functions, respectively. To avoid over-fitting, we regularize the parameters (3*) k== by L2 norm. Then Eq. (00) can be rewritten as:
arg min a.8", ,"..h
A.
1012
Yik
(i) In (fa( Dir(d (f ())):
km
(12)
exp(a a',)) Σラ= exp(a) xh))
(x)
(2n = In(1 + exp(3) df))), xy (m 1]:
where or is the k-th parameter vector in a. G. is the j-th parameter vector in B² and X is a trade-off parameter. We use L-BFGS 17 to optimize Eq. [22 We denote this instantiation as DRAM-LN.
3.2 Comparison method: data transformation
Since there is existing method that can directly predict label distribution from multi-label rank- ing, we propose a comparison method called DT (dataset transformation). The main idea of DT is to transform multi-label rankings into logical labels. Then, we can recover label distributions by any existing LE algorithm and learn a label distribution predictor by any existing LDL al- gorithm. Specifically, the multi-label ranking dataset { W will be transformed into U (< Eyeo.... Uy "o,lo=l1")))))))))) (r) |o where n, is the set of the 1st, 2nd, and i-th elements in n vector, vy is an o -dimensional one-hot vector with the j-th element being 1, and the instance weight ,o,, |- is to avoid that the model learning favours instances with more relevant labels.
4 Related work
Our research is mainly related to LE 34 and LDL 4 LE is a process of recovering label distributions from logical labels in the dataset. Most of the existing LE algorithms follow some basic assumptions. For example, some LE algorithms 99| 15 22 34] assume that instances with similar features have similar label distributions; some LE algorithms
assume that semantically related labels also have close values in the label distribution; some LE algorithms
12 32
33 16] assume that the label distribution is the oiwdeimesio representation of feature and logical label vectors.
LDL is the learning process on the instances annotated by label distributions. The LDL problem can be addressed in two ways. The first is to directly design algorithms that match the prerequisites of the LDL problem. Some prominent examples include the LDL algorithms 11
19
20
25
371 that
mine label correlation and the LDL algorithms that maintain label ranking [ classification racy [26]
[27] [28]. Another way is to extend existing learning algorithms. For example, LDSVR 5 fits each component of the label distribution by support vector machine; LDLLogitBoost [31
extends the boosting method by additive weighted regressors; LDLF (21) designs a normalization
layer to model the multi-modal distribution by extending differentiable decision trees.
5 Experiments
5.1 Datasets and evaluation measures
We adopt several widely used label distribution datasets, including Movie Emotionó (8). Twitter- LDL, and Flickr-LDL 36 We manually reduce the label distributions in these datasets to multi-label rankings, train the model on these multi-label rankings, and then evaluate the model using the original label distributions in these datasets In addition, we create a dataset called NSRD (Natural Scene with multi-labe Rankings and label Distributions). The instances and label sets in NSRD are the same as in the Ratural-Scent 77. Three experts are requested to annotate the instances with multi- label rankings and label distributions. Then we can directly train and evaluate the model using the labti-label rankings and label distributions, respectively. Details of these datasets can be found in the
Although there many label distribution datasets, only adopt those whose label distributions generated by expert annotation rather than algorithms experimental instruments.
'These datasets contain some label distributions with identical values, e.g., [0.3, 0.3, 0.4, 0J which does not satisfy the prerequisites of this paper, remove them and their corresponding feature vectors beforehand.
7

---

Table 1: Experimental results ((ran) std f-test) evaluated by four measures.
Dataset
Method
Cheh (4)
Canher (4)
Cosine (1)
Rho (1)
DRAM-LN
(2) + 0.001
(2) 1.058 + 0.008
(2) + 0.001
(1) = 0.006
DT+VI+SA
(6) = 0.001
(6) (6)1.337 0.007 *
(6)0.888 0.002
(5)0.685
0.020
DT+VI-DM
07)6. +: 0.003
17)5.355 0.017
(7)0.884 +: 0.004
(4)0.712
0.006
Movie
DT+GL+SA
(4) 1: 0.001
(5) 1.207 1: 0.003
(4)0.912 1: 0.0001
(6)
0.010
DM
(3) + 0.001
1.204 0.004
(3)0.915 0.001
02)0.717
0.006
GT+SA
(5) = 0.003
(3) 1.200 + 0.021
(5)0.903 + 0.003
(7)0.625 = 0.0100
(1)0.114
(1)0.9
(1)0.936
(3)0.715
DRAM-LN
(1)0.282 1 0.004
3.953 1 0.050
(1)0.785 0.006
(1) 0.009
DT+VI+SA
0.316 + 0.007
3.) 0.073
(3)0.743 0.015
(3)0.506 0.035
DT-VI-DM
03)0 + 0.006
3,) + 0.005
(,)0. = 0.012
(2)0.564 = 0.051
Emotion6
DT+GLASA
0.36
4.)22121
(5)0.685 0.011
(70.311
DT-GL+DM
0.)0. + 0.007
4.)4. + 0.037
(4)0.708 0.010
(6) 0.36. 0.035
GT+SA
0.)6 = 0.011
(7) 0.045
(70.1911
05)
0.0341
GT+DM
0.380 + 0.009
4.769
0.043
(6)0.643 + 0.014
(4)0.473
0.029
DRAM-LN
(1)3
6.526 0.018
())0.828 0.009
(.)
DT+VI+SA
(6) 0.546 + 0.003
6.)64 + 0.020
(6)..22 0.005
(7) = 0.019
DT+VI-DM
(7) 0.500 + 0.010
66)6668 0.023
(7)0. 0.007
(4)0.559
0.011
Twitter-LDL
DT+GLASA
0.502 + 0.003
(2)6.545 0.018
(3)0,682 + 0.001
(3)0,578 0.005
DT-GL+DM
(5) 0.533 1 0.003
63)6. 1 0.018
(4)0.656 1 0.001
(2)0.592 0.006
GT+SA
0.436 0.017
(7)37 0.026
(5)0.653 0.000
0.516 0.009
GT+DM
0.)7 = 0.001
6.)6 # 0.017
(2) 0.000
(5) = 0.006
DRAM-LN
(1)0.324 + 0.005
6.013 0.017
(1) 0.001
(1) 0.006
DT+VI+SA
0.456 1 0.005
6,166 1 0.021
(5)0.657 1: 0.006
(6)0.542 0.021
DT+VI-DM
(7) 0.4.77 1 0.011
6.146 0.040
(7) 1.002
02)5. 0.010
Flickr-LDL
DT+GL+SA
(4) 0.400 + 0.005
6,076 + 0.021
(3)0.690 + 0.000
(4)0.573 = 0.005
0.) DT-GL+DM
6.
(4)0.671
(3)0.592
GT+SA
0.44. + 0.00S
(7) + 0.020
(6.0.344
(7) 0.007
GT4DM
(2) 0.363 = 0.001
(6) 6.360 0.014
(2)0.720 0.005
05) = 0.005
DRAM-LN
(3) 0.509 + 0.006
(.) + 0.017
(3) 0.009
(2) = 0.013
DI+VI+SA
0.576
7.835 0.028
06).162 0.013
0)).118 =
DT
0.555 + 0.007
(6)7.81 + 0.029
0.)4. 0.034
NSRD-el
DT
04)0.574 = 0.006
(4)..741 0.028
(2)0.523
(4 = 0.012
DT-GL+DM
0.579 + 0.006
7.755
0.027
(5)0.509
4.0007
(00.462 0.013
GT+SA
(1) 1 0.008
(3) 7.700 0.021
())0.610
(3)0.447 0.008
GT+DM
0.488 + 0.010
7.659 0.035
0.004
05)4.3 0.013
DRAM-LN
(3) + 0. 0.006
(.) 0.017
(3)0.599 +.0000
(2).459 = 0.013
DI+VI+SA
05)0 = 0.006
(7) 0.021
160.169900001
(70.119 0.028
DT+VIDM
(7) 0.588 1 0.009
(6) 7.7799 0.030
(7)0.465
0.001
(610.251 0.025
NSRD.22
DT+GLASA
(4) 0.568 1 0.005
(4) 0.025
(4)0.525
0.004
(4)0.444 0.016
DT-GL+DM
(6) 0.574 = 0.006
(5) +: 0.028
(0.0011
(1) = 0.014
GT+SA
(1)
7.680
0.006
0)0.150 0.012
GT+DM
(2) 0.485 + 0.012
(2) 7.664 11.664:00 0.035
(20.600 00.014
(5) 0.014
DN
0.554 1 0.011
7.699 1 0.023
(3)
0.003
(2)0.455 0.012
DT.VI+SA
(4) 0.615 - 0.009
7.7.845 + 0.031
(0).456 40.018
(7) = 0.023
DT-VI-DM
(7) 0.638 0.005
(67.817 0.013
00.016
0.)
NSRD-e3
&TAGLASA
0.)1. 1 0.004
(4)7.760
0.021
(5) 0.015
DT-GL+DM
(6) 0.622 = 0.005
(5)
0.019
(..0.9
(1) = 0.016
GT+SA
(1) 090 + 0.012
7.77. # 0.028
(1)0,601 0.019
03)0.113 0.011
GT+DM
(2) 0.520 1 0.009
(2) 7.701 1 0.030
(2)0.599 0.016
0.440 0.012
appendix. We used the six distance-based measures suggested in the paper and ranking- based measure suggested in the paper [13) to evaluate the performance of the model, which are Cheb (Chebyshev distance), Clark (Clark distance), Canber (Canberra distance), KL (Kullback-Leibler divergence), Cosine (cosine coefficient), Intersec (intersection similarity), and Rho (Spearman's 's rho coefficient). Due to page limitations, we only show the results on Cheb, Canber, Cosine, and Rho. Results on other measures are similar.
5.2 Comparison methods
On the one hand, we compare DRAM with the baseline method DT proposed in Section (Graph Laplacian LE) 34 and SA (specialized LDL algorithm with BFGS optimizer) [4 are
3.2 GL
the classical LE and LDL algorithms respectively. VI (LE with variational inference)
33 and
DM (L.DL with label distribution manifold) 25 are the state-of-the-art LE and LDL algorithms, respectively. We combine them in pairs to construct four comparison methods, i.e., DS+GL+SA, DT+GL+DM, VI+V and DM+ The hyperparameter configuration of GL, VI and DM

---

Table 2: Average ranks of methods.
7.48
Method
Cbeb
Canber
Cosine
Rho
1.69
DRAM-LN
2,00
1.14
2.00
1.43
DT-GL-DDM
5.00
4.14
4.29
2.29
DT+GLASA
4.29
3.71
3.86
4,71
DT+VI+DM
6.43
5.43
6.29
4.29
0:41
DT+VI+SA
4.86
5.43
5.43
6.00
GT+DM
2.43
3.43
(cussier of Ixiskorn mangroandd))
3.00
Figure 2: Performance with varying K on NSRD.
follows their respective literature. For our method, we set 3 and L 20, and is selected from {10-5,5 5, x 10 10 x 10-4 -0-4.... 10' x 10') by five-fold cross- -ross-validation. For the above comparison methods, since the label distributions are unavailable during training, the hyperparameter configuration that gives the highest Rho on the validation set will be used. On the other hand, we directly train DM and SA on the ground-truth label distributions for comparison. We refer to these two as GT+DM and GT+SA for short, respectively. For these two comparison methods, the hyperparameter configuration that gives the best Cheb, Canber, Cosine, and Rho on the validation set will be used. Each method is run for ten times random dataset partitions (70% for training and 30% for test); the average values and standard derivations are recorded.
5.3 Results and discussions
Table I shows the four performance measures of each method on four reduced datasets and NSRD dataset. Since the NSRD is annotated by three experts, can obtain three corresponding datasets, denoted by the "-el". "-e2" and "-e3" suffixes, respectively; these three datasets have the same feature vectors and different annotation values. Each experimental result is formatted as "(rank) -td I-test"; "(rank)" denotes the rank of each method among the seven comparison methods; /o indicates whether DRAM-LN is statistically superior/inferior to the corresponding methods (pairwise two-tailed 1-test at 0.05 significance level): if neither nor o is shown, means that there no significant difference between the corresponding method and DRAM-LN: "L" denotes "the lower the better", and "f" denotes "the higher the better". Table shows the average rank of each comparison method on each measure. Overall, our method achieves significant advantages. On the Emotion6, (witter-LDL, and Flickr-LDL datasets, our method significantly outperforms almost any comparison methods. On the NSRD and Movie datasets, our method only inferior the GTT-based methods on Cheb and Cosine measures. It is worth noting that DRAM-LN and D'T-based methods outperform GT-based methods in many cases, such as the performance on Emotion6, Twitter-LDL. and Flickr-L.D) datasets. We believe this is because some datasets are difficult to annotate; thus, the label distributions given by experts are noisy; then, fitting such label distributions exactly may lead to overfitting. This argument can be further supported by the fact that GT hased methods outperform our method on the NSRD dataset (where the label distributions are carefully annotated and less noisy).
Figure 2 shows how the number of mixture components K affects the performance of our method on NSRD dataset. To save space, we show the average performance on NSRD-e1, NSRD-e2 and NSRD-e3 rather than showing them separately. It is obvious that the mixture model (K >> 1) always outperform the single model (K 1). addition, it can be seen that appropriately increasing the Dirichlet components in the mixture can improve the model capacity and thus improve the predictive performance, too many Dirichlet components may lead to overfitting and thus degrade the predictive performance.
Limitations and conclusion
Limitations. EAE defined for the label description vector and does not directly reflect the approximation error to the label distribution; this limitation arises because normalizing the label description vector to label distribution will lead to an extremely complex closed form of EAE. We do not believe that this limitation have a significant impact on our main results since the approximation error to the label distribution does not exceed that to the label description vector. For example, the true and estimated label description vectors are and tz (where is scaler), respectively, they

---

will produce non-zero errors w.r.t. the label description vector; but and tz are the same after normalization, i.e., they do not produce enrors w.r.t. the label distribution. If several labels actually describe the instance to the same degree, then requiring experts to give the strict order of these labels may lead to errors and invalidate Theorem we plan to extend the theoretical results to this case in the future. Fortunately, DRAM framework can suit this case by a minor modification on the explicit range, i.c., allowing the description degree of these labels to be identical in line 5 of Algorithm []
Conclusion. We derive some theorems and corollaries to reveal the relation between multi-label ranking and label distribution, and propose a generic framework, DRAM, for predicting label distribution from multi-label ranking. DRAM is cost-effective: It is trained on the examples with multi-label rankings and achieves performance comparable to that of LDL methods which require expensive label distribution annotations; DRAM is flexible: It allows users to easily encode their prior knowledge by scoring function; DRAM is end-to-end: integrates the processes of recovering and learning label distributions into one learning criterion, rather than performing them separately. Experimental results show the superiority of our proposal.
Acknowledgments
This work was partially supported by the National Key Research and Development Program of China under Grant 2019YFB1 106900, the National Natural Science Foundation of China (62176123, U20B2064), and the Fundamental Research Funds for the Central Universities (3092002113)).
References
[1] Klaus Brinker, Johannes Fürnkranz, and Eyke Hüllermeier. A unified model for multilabel classification and ranking. In European Conference on Artificial Intelligence, pages 489-493, 2006.
[2] Klaus Brinker and Eyke Hüllermeier. Case-based multilabel ranking. In International Joint Conference on Artificial Intelligence, pages 702-707, 2007.
[3] Binbin Gao, Hongyu Zhou, Jianxin Wu, and Xin Geng. Age estimation using expectation of label distribution learning. In International Joint Conference on Artificial Intelligence, pages 712-718, 2018.
[4] Xin Geng. Label distribution learning. IEEE Transactions on Knowledge and Data Engineering, 28(7):1734-17) 2016.
[5] Xin Geng and Peng Hou. Pre-release prediction of crowd opinion on movies by label distribution learning. In International Joint Conference Artificial Intelligence, pages 3511-3511 2015.
[6] Xin Geng, Kate Smith-Miles, and Zhi-Hua Zhou. Facial age estimation by learning from label distributions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35:2401-2412, 2013.
[7] Xin Geng, Renyi Zheng, Jiaqi Lv, and Yu Zhang. Multilabel ranking with inconsistent rankers. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2021.
[8] Maya R. Gupta and Yihua Chen. Theory and use of the em algorithm. Foundations and Trends Signal Processing, 4(3):223-296, 2011.
[9] Peng Hou, Xin Geng, and Minling Zhang. Multi-label manifold learning. In AAAI Conference on Artificial Intelligence, pages 1680-1686, 2016.
[10] Xiuyi Jia, Weiwei Li, Junyu Liu, and Yu Zhang. Label distribution learning by exploiting label correlations. In AAAI Conference on Artificial Intelligence, pages 3310-3317, 2018.
[11] Xiuyi Jia, Weiwei Li, Junyu Liu, and Yu Zhang. Label distribution learning by exploiting label correlations. In AAAI Conference on Artificial Intelligence, pages 3310-3317, 2018
[12] Xiuyi Jia, Yunan Lu, and Fangwen Zhang. Label enhancement by maintaining positive and negative label relation. IEEE Transactions Knowledge and Data Engineering, 2021.
10

---

[13] Xiuyi Jia, Xiaoxia Shen, Weiwei Li, Yunan Lu, and Jihua Zhu. Label distribution learning by maintaining label ranking relation. IEEE Transactions on Knowledge and Data Engineering. pages 1-1, 2021.
[14] Xiuyi Jia, Xiang Zheng, Weiwei Li, Changqing Zhang, and Zechao Li. Facial emotion dis- tribution learning by exploiting low-rank label correlations locally. In IEEE Conference on Computer Vision and Patlern Recognition, pages 9833-9842, 2019.
[15] Yukun Li, Minling Zhang, and Xin Geng. Leveraging implicit relative information for effective multi-label learning. In IEEE International Conference on Data Mining, pages 251-260 2015.
[16) Xinyuan Liu, Jihua Zhu, Qinghai Zheng, Zhongyu Li, Ruixin Liu, and Jun Wang. Bidirectional loss function for label enhancement and distribution learning. Knowledge- Based System, 213:106690, 2021.
[17] Jorge Nocedal and Stephen Wright. Numerical Optimization. Springer, 2006.
[18] Kuan-Chuar Peng, Tsuhan Chen, Amir Sadovnik, and Andrew Gallagher. A mixed bag of emotions:
Model, predict, and transfer emotion distributions. In IEEE Conference on Computer Vision and Puttern Recognition, pages 860-868, 2015.
[19] Tingting Ren, Xiuyi Jia, Weiwei Li, Lei Chen, and Zechao Li. Label distribution learning with label-specific features. In International Joint Conference on Artificial Intelligence, pages 3318-3324, 2019.
[20] Tingting Ren, Xiuyi Jia, Weiwei Li, and Shu Zhao. Label distribution learning with label corre- lations via low-rank approximation. In International Joint Conference on Artificial Intelligence, pages 2019.
[21] Wei Shen, Kai Zhao, Yilu Guo, and Alan Yuille. Label distribution learning forests. In Advances Neural Information Processing Systems, 2017.
[22] Haoyu Tang, Jihua Zhu, Qinghai Zheng, Jun Wang, Shanmin Pang, and Zhongyu Li. Label enhancement with sample correlations via low-rank representation. In AAAI Conference Artificial Intelligence, pages 5932-5939, 2020.
[23] Grigorios Tsoumakas and loannis Katakis. Multi-label classification: An overview. Interna- tional Journal of Data Warehousing and Mining, 3(3):1-13, 2006.
[24] Jesper F. van Engelen and Holger H. Hoos. A survey semi-supervised learning. Machine Learning, 109:373 440, 2020.
[25] Jing Wang and Xin Geng. Label distribution learning by exploiting label distribution manifold. IEEE Transactions on Neural Networks and Learning Systems, pages 1-14, 2021.
[26] Jing Wang and Xin Geng. Label distribution learning machine. In International Conference on Machine Learning, pages 10749-10759, 2021.
[27] Jing Wang and Xin Geng. Learn the highest label and rest label description degrees. In International Joint Conference on Artificial Intelligence, pages 3097-3103, 2021.
[28] Jing Wang, Xin Geng, and Hui Xue. Re-weighting large margin label distribution learning for classification. IEEE Transactions on Puttern Analysis and Machine Intelligence, pages 1-1, 2021.
[29] Ke Wang, Ning Xu, Miaogen Ling, and Xin Geng. Fast label enhancement for label distribution learning. IEEE Transactions on Knowledge and Data Engineering, pages 1-1, 2021.
[30] Xin Wen, Biying Li, Haiyun Guo, Zhiwei Liu, Guosheng Hu, Ming Tang, and Jinqiao Wang. Adaptive variance based label distribution learning for facial age estimation. In European Conference on Computer Vision, pages 379-395, 2020.
[31] Chao Xing, Xin Geng, and Hui Xue. Logistic boosting regression for label distribution learning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4489-4499 2016.

---

[32] Ning Xu, Yun-Peng Liu, and Xin Geng. Label enhancement for label distribution learning.
IEEE Transactions on Knowledge and Data Engineering, 33(4):1632-1643,, 2021.
[33] Ning Xu, Jun Shu, Yun-Peng Liu, and Xin Geng. Variational label enhancement. In International Conference Machine Learning, pages 10597-10606, 2020.
[34] Ning Xu, An Tao, and Xin Geng. Label enhancement for label distribution learning. In International Joint Conference Artificial Intelligence, pages 2926-2932, 2018.
[35] Jufeng Yang, Dongyu She, and Ming Sun. Joint image emotion classification and distribution learning via deep convolutional neural network. In International Joint Conference on Artificial Intelligence, pages 3266-3272. 2017.
[36] Jufeng Yang, Ming Sun, and Xiaoxiao Sun. Learning visual sentiment distributions via aug- mented conditional probability neural network. In AAAI Conference on Artificial Intelligence, 2017.
pages
[37] Peng Zhao and Zhi-Hua Zhou. Label distribution learning by optimal transport. In AAAI Conference on Artificial Intelligence, pages 4506-4513, 2018.
[38] Xiang Zheng, Xiuyi Jia, and Weiwei Li. Label distribution learning by exploiting sample correlations locally. In AAAI Conference on Artificial Intelligence, pages 4556-4563, 2018.
[39] Deyu Zhou, Xuan Zhang, Yin Zhou, Quan Zhao, and Xin Geng. Emotion distribution learning from texts. In Conference Empirical Methods in Natural Language Processing, pages 638-648 2016.
[40] Wenfang Zhu, Xiuyi Jia, and Weiwei Li. Privileged label enhancement with multi-label learning. In International Joint Conference on Artificial Intelligence, pages 2376-2382, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] All proofs are detailed
in the appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes] The supplemen- tal material provides a detailed instruction for reproducing the main results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
12

---

(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We include the new dataset we created in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No]
(e) Did you discuss whether the data you using/curating contains personally identifi- able information or offensive content? [Yes] The data we are using doesn't 't contain personally identifiable information or offensive content
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "SNu_osq8WJ", "reviewer": "Reviewer_YEUF", "summary": "This paper studies how to exploit multi-label ranking, a compromise w.r.t. annotation cost, to learn a predictive model on label distribution. The authors theoretically investigate the relation between multi-label ranking and label distribution and demonstrate its superior over the logical labels. Then a general framework involving custom knowledge is proposed to recover and learn label distribution end-to-end. The extensive experiments show the effectiveness of the proposal.", "questions": "1. What is the time cost of the EM algorithm used to train model? The authors should add more analysis on time complexity.\n2. Note that the datasets used in this paper are relatively small, so what is the capability of the proposed framework on large datasets?\n3. Are there any selection criteria for scoring function? Note that only DRAM+LN is tested on all datasets and how do different scoring functions affect the results? \n4. Multi-label ranking may still require a high cost. The authors can add some discussion on cost of various methods (the proposal, LE and LDL methods), for example, the cost comparison when achieving the same performance.\n", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is the first paper that proposes to predict label distribution from the view of multi-label ranking and shows the good guarantees for performance. The originality is good and the main idea of this paper is clearly presented. They also show better results than other compared methods. Despite that, I still have some concerns:\n1. What is the time cost of the EM algorithm used to train model? The authors should add more analysis on time complexity.\n2. Note that the datasets used in this paper are relatively small, so what is the capability of the proposed framework on large datasets? In fact, label ranking is often associated with high time complexity.\n3. Are there any selection criteria for scoring function? Note that only DRAM+LN is tested on all datasets and how do different scoring functions affect the results? \n4. Multi-label ranking may still require a high cost. The authors can add some discussion on cost of various methods (the proposal, LE and LDL methods), for example, the cost comparison when achieving the same performance.\n5. Some spelling mistakes need to be corrected, e.g., “internel” in the third sentence in section 2.1 should be “internal”.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SG9wdAsr77q", "reviewer": "Reviewer_rKfn", "summary": "This paper studies Label Enhancement (LE). Instead of enhancing label distribution from logical labels, it proposes to recovery label distribution from multi-label ranking annotations. To achieve that, the authors establish several theories and put forward a new method. Experimental results validate the advantages of recovering label distribution from ranking over logical labels. ", "questions": " I expect to see the results in terms of KL divergence in the rebuttal. ", "limitations": "I believe the authors have clarified the limitations of their paper. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "On one hand, the strengths include:\n+ LE with ranking annotation is original, which hasn't been noticed in the field of LE. Besides, the authors theoretically prove that LE from ranking is better than LE from logical labels, as shown in corollary 4.\n+ The method DRAM is novel. DRAM uses the Dirichlet mixtures and EM algorithm to recover label distribution from rankings. \n+ Most importantly, the experiments show that DRAM has remarkable improvements over existing LE (from logical labels). Moreover, the DRAM+LN combination even outperforms SA and DM with the ground-truth label distributions, which is impressive. \n\nOn the other hand, for me, the main weakness mainly lies in that the paper is not easy to follow. I understand that the authors have put necessary proofs and details to the supplementary material, but still the Sections 2 and 3 are not easily understandable. Besides, I expect to see the results in terms of KL divergence because it is such an important measure metric for LDL. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IaIn2W2yIVI", "reviewer": "Reviewer_KFoB", "summary": "This article propose a generic framework named DRAM(label Distribution predicting from\nmulti-label RAnking via conditional Dirichlet Mixtures). It allows to flexibly encode the prior knowledge about the tasks by a scoring function, and it integrates the processes of recovering and learning label distributions end-to-end. Besides, the author theoretically investigate the relation between multi-label ranking and label distribution and define the notion of EAE to quantify the quality of an annotation, and give the bounds of EAE for multi-label ranking.\n", "questions": "2. It can be seen from Corollary 1 that the optimal solution of  ${\\hat \\delta ^*} = ((2m + 1)\\delta  + m){(m + 1)^{ - 2}}$ does not approach  ${\\delta }$, whether this shows the capability boundary of the method? In addition, the process of theoretically guiding the algorithm is mainly reflected in the update of ${{\\hat \\delta } _n} = \\left| {{\\sigma _n}} \\right|{(\\left| {{\\sigma _n}} \\right| + 1)^{ - 2}}$. From the Corollary 3, ${\\hat \\delta }  \\in [m{(m + 1)^{ - 2}},{m^{ - 1}}]$, why does  ${\\delta }$ take the lower bound of the interval instead of a certain value in the middle ?\n", "limitations": "3. When analyzing how the number of mixture components K affects the performance of the method, the authors point out that “t can be seen that the performance gets better first and then worse as K increases.”, it seems that such a conclusion cannot be simply drawn from the figure.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Generally speaking, the article has a certain degree of innovation and the theoretical results are plentiful. The structure of the article is complete and the thinking is clear. Therefore, we hold the opinion of weak reception.\n1. In Definition 1, the author states that $z$ and $\\hat z$ are independent, but from Corollary 1 it can be seen that ${\\delta }$ and ${\\hat \\delta }$ are closely related, so the independence of  $z$ and $\\hat z$ needs further explanation.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9-_wBdkaN6d", "reviewer": "Reviewer_b3Gr", "summary": "This paper aims at predicting label distribution from multi-label ranking. It is different from the existing label distribution learning (LDL) and label enhancement (LE), where LDL learns a predictive mapping on training set with label distributions annotations and LE generates label distributions from instances with logical labels. This work is a compromise w.r.t. annotation cost but has good guarantees for performance.", "questions": "I have listed some weaknesses above. Here, my further concern is why the general LDL data sets are not used in experiments. It is very easy to construct data sets with label rankings and with binary labels from LDL data set. The 15 LDL data sets release by Geng can be found at: http://palm.seu.edu.cn/xgeng/LDL/index.htm", "limitations": "Authors points out two limitations in Section 6. For the 2nd one, i.e., \"If several labels actually describe the instance to the same degree, then requiring experts to give the strict order of these labels may lead to errors and invalidate Theorem 1\", authors should extend the proposed framework to deal with such kind of data sets, not treat it as a future work. After all, it is more realistic that some labels have the same description degree.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper has a good motivation.\n2. This paper proposed a framework of label distribution predicting from multi-label ranking.\n3. Some experimental results validate the superiority of the proposed method.\n\nWeaknesses:\n1. It seems that the theoretical analysis has little relatedness with the proposed framework and there are not experiments to support the theoretical results. It is something that looks beautiful. \n2. The trade-off parameter $\\lambda$ is selected from a very wide range by five-fold cross-validation. Is there a good default choice? Parameter sensitivity should be made.  Besides, five-fold cross-validation is conducted over the whole data set or only the 70% data for training?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept