Predicting Label Distribution from Ternary Labels
Authors: Yunan Lu
Xiuyi Jia
Venue: NeurIPS 2024 poster (2024)
Keywords: label distribution
label polysemy
multi-label
ternary label
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: Our paper proposes to predict label distribution from ternary labels and demonstrates the effectiveness theoretically and methodologically.

Abstract: Label distribution learning is a powerful learning paradigm to deal with label polysemy and has been widely applied in many practical tasks. A significant obstacle to the effective utilization of label distribution is the substantial expenses of accurate quantifying the label distributions. To tackle this challenge, label enhancement methods automatically infer label distributions from more easily accessible multi-label data based on binary annotations. However, the binary annotation of multi-label data requires experts to accurately assess whether each label can describe the instance, which may diminish the annotating efficiency and heighten the risk of erroneous annotation since the relationship between the label and the instance is unclear in many practical scenarios. Therefore, we propose to predict label distribution from ternary labels, allowing experts to annotate labels in a three-way annotation scheme. They can annotate the label as "$0$" indicating "uncertain relevant" if it is difficult to definitively determine whether the label can describe the instance, in addition to the binary annotation of "$1$" indicating "definitely relevant" and "$-1$" indicating "definitely irrelevant". Both the theoretical and methodological studies are conducted for the proposed learning paradigm. In the theoretical part, we conduct a quantitative comparison of approximation error between ternary and binary labels to elucidate the superiority of ternary labels over binary labels. In the methodological part, we propose a Categorical distribution with monotonicity and orderliness to model the mapping from label description degrees to ternary labels, which can serve as a loss function or as a probability distribution, allowing most existing label enhancement methods to be adapted to our task. Finally, we experimentally demonstrate the effectiveness of our proposal.

Predicting Label Distribution from Ternary Labels
Yunan Lu, Xiuyi Jia"
School of Computer Science and Engineering Nanjing University of Science and Technology. Nanjing 210094, China [luyn jiaxy)enjust. eew.puutttuuu....o.o.e...eeee
Abstract
Label distribution learning is a powerful learning paradigm to deal with label polysemy and has been widely applied in many practical tasks. A significant obstacle to the effective utilization of label distribution is the substantial expenses of accurate quantifying the label distributions. To tackle this challenge, label enhancement methods automatically infer label distributions from more easily accessible multi-label data based on binary annotations. However, the binary annotation of multi-label data requires experts to accurately assess whether each label can describe the instance, which may diminish the annotating efficiency and heighten the risk of erroneous annotation since the relationship between the label and the instance is unclear in many practical scenarios. Therefore, we propose predict label distribution from ternary labels, allowing experts to annotate labels a three-way annotation scheme. They can annotate the label as "0" indicating "uncertain relevant" if it is difficult to definitively determine whether the label can describe the instance, in addition the binary annotation of "1" indicating "definitely relevant" and "" indicating "definitely irrelevant". Both the theoretical and methodological studies are conducted for the proposed learning paradigm. the theoretical part, conduct quantitative comparison of approximation error between temary and binary labels to elucidate the superiority of ternary labels over binary labels. In the methodological part, propose Categorical distribution with monotonicity and orderliness to model the mapping from label description degrees to temarr labels, which distribution, allowing most existing label enhancement methods to be adapted to task. Finally, experimentally demonstrate the effectiveness of proposal.
a loss function
probability
Introduction
LDL (Label Distribution Learning) (2 is effective learning paradigm for addressing label polysemy (i.e., the cases where an instance can be described by multiple labels). Distinct from multi-label learning [22]. which assign a binary-valued vector to each instance, LDL assigns each instance real-valued vector, akin to descrete probability distributions, to represent the description degree of each label to the instance. Label distributions provide fine-grat ime-grrined information about label polysemy,
and thus have been applied to many practical tasks, such as sentiment analysis g 30 35
facial age
estimation 4 24 and movie rating prediction
fundamental bottleneck hindering LDI. the difficulty in acquiring ground-truth label distributions, the accurate quantification of these distributions can be prohibitively expensive. Therefore, LE (Label Enhancement) [29] is proposed to automatically infer label distributions from the more easily accessible multi-label data. Multi-label data is based on binary annotations (i.c., utilizing binary values +1 to annotate each label) which demand that experts accurately identify whether each label
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Label
Annotation
Happy
Feur
Happy
irrelevant (-1)
Sad
irrelevant (-1)
Surprise
Surprise
relevant (+1)
Sad
Fear
uncertain (?)
Figure I: annotation from JAFFE dataset
can describe the instance. However, accurate identification is challenging in real-world tasks due to the prevalence of ambiguous instances and labels which provoke uncertain binary associations between the instances and the labels. For example, the facial image in Figure can be definitely (negatively or positively) associated to "happy". "sad", and "surprise", whereas the association with "fear" is uncertain, since is not clear whether the emotion comes from being stimulated by something frightening. experts are coerced into providing a definitive annotation in uncertain cases, not only does it diminish the efficiency of annotating, but it also heightens the risk of erroneous annotation.
Therefore, we propose to predict label distribution from ternary labels. Inspired by the philosophies of hhy decision or three-world thinking [11, where decision-makers are provided with the flexibility to delay judgment when the available information inadequate to support a determined decision, ternary labels take values from {-1,0, 0, 1). where "=11" indicates whether the label can describe the instance, and "0" denotes that the relationship between the label and the instance conduct theoretical and methodological
is uncertain. For the proposed learning paradigm, studies. In the theoretical part, we first quantify the errors of approximating the ground trrut label description vector from ternary labels and binary labels, respectively. Further, quantitative comparison of approximation eπror between ternary and binary labels to elucidate the superiority of ternary labels. In the methodological part, we propose CateMO distribution (Categorical distribution with Monotonicity and Orderliness) to model the mapping from label description degrees to ternary labels, which serve the probability distribution for generating ternary labels as the loss function for measuring the inconsistency between the ternary labels and label distributions, allowing most existing LE methods to be adapted to our task. Specifically, governing the generation of ternary labels from label description degrees, and formalize them as the assumptions about the probabilistic monotonicity and orderliness of ternary labels. Further, we derive the probability mass function of CateMO to ensure probabilistic monotonicity and orderliness. Finally, we create two comparison algorithms and evaluate the prediction performance on three real-world datasets. Experimental results unequivocally affirm the superiority of ternary labels. Our contributions can be summarized as follows:
conduct
first analyze the rules
We propose predict label distribution from ternary labels, which not only enhances the annotation accuracy but also significantly reduces the annotating cost when contrasted with the traditional binary annotating methods.
We rigorously analyze the error of approximating the ground-trutt -tuut label description degrees by ternary and binary labels, respectively, which provides a quantitative elucidation of the superior performance of the ternary label.
We propose the CateMO distribution specifically designed capture the mapping from label description degrees to ternary labels, which is theoretically constructed to maintain the monotonicity and ordinality of the probabilities associated with ternary labels.
2
Related Work
To address the challenge of obtaining accurate ground-truth label distributions, LE (Label Enhance- ment) 29 is proposed as a method to automatically infer label distributions from more accessible
In general, the elements of label distribution are assumed to be non non-negative and sum to 1. For simplicity, we disregard the restriction that the sum is 1 in this paper, and refer to the unnormalized label distribution as label description vector. The element of label description vector is called label description degree.

---

Existing works on principled manner and uncovering the underlying patterns of the observed data.
multi-label data, including binary labels (29 and multi-label rankings [3 15 LE can be broadly categorized into discriminative LE methods and generative LE methods. Discrimi- native LE directly treats the label distribution as conditional probabilities of the labels given feature observations, and subsequently mine additional information to estimate the conditional probabilities. Generative LE, on the other hand, focus on describing the generation process of observations in
Current discriminative LE methods generally strive to optimize both Dist(z, y) and 02(z), where Dist(z; z. y) measures the inconsistency between the label description degrees z and the more accessible labels (including binary labels and multi- rankings), and (2( is the regularization terms based on various sources of information. Typically, Dist( z. y) is modeled using MSE (Mean Squared Error), and the information for 9 z)) includes instance relationships and label correlations. For instance, several algorithms
assume that the instance manifolds based on features are similar to the instance manifolds based on label distributions. Additionally, several algorithms [1] 26 29 32 operate under the assumption that instances with similar feature vectors exhibit similarity in their label distributions. To incorporate label correlations, some algorithms [1 25 regularize the label distributions using the Graph Laplacian operators of label correlation graphs. LEPNLR (6], on the other hand, enhances the labels by preserving the ranking relation of labels. Existing generative LE methods generally decompose the joint distribution of complete data as p(z)|=), p(... |z), and p(z), where p(y) (z) models the relationship between y and x, and p(-.. - [z) captures the relationships between & and other observed variables. Current works primarily focus on modeling p(y|() and
S 10
20
21
23
p(....(2) |z). For instance, LEVI [77 28 and GLEMR [12] model p(y) (z) as Gaussian distribution and a weakkking-preseeeee distribution, respectively. GLERB [16] and LEIC [14 model p(... (z) by incorporating instance relationships and label correlations into generation processes.
Furthermore, similar to ternary labels, the binary labels with missing values [17 are formally represented by {0, =1). However, it should be emphasised that they inherently differ in the following aspects. First, they differ in the origin of the labels with value of 0, i.e., the missing labels in the literature [17] and the uncertain label in ternary labels. The missing labels stem from the "absence", ie., the association between the label and the corresponding instance undocumented or unannotated rather than undeterminable. By contrast, the uncertain labels stem from the "uncertainty", i.e., it is difficult for experts to definitively determine whether the label is relevant to the corresponding instance. Second, they differ in the range of the underlying label description degree. The description degree of the missing labels to the corresponding instance can take any value in the interval [0, 1]. since the missing labels may be relevant labels, irrelevant labels, or uncertain labels. By contrast, the description degree of the uncertain labels to the corresponding instance take values in a small sub-interval of [0, 1]. which is much tigher than the range of the description degrees of missing labels.
3
Quantitative analysis
3.1 Preliminary
Given instance with feature vector of æ:, the label description vector of the instance is denoted by whose m-th element 2m E [0. 1] indicates the description degree of the m-th label to the instance æ. an instance is annotated with binary labels, the label deseription vector will be degenerately expressed as a vector of binary values E (-1, 1 whose m-th element b,. denotes whether the m-th label can describe the instance not. If an instance is annotated with ternary labels, the label description vector will be degenerately expressed as a vector of ternary values & E {-1, 0, 1 1 where 8m = 1 denotes that the m-th label can describe and cannot describe the instance, respectively. and Sm = 0 denotes that the association between the m-th label and the instance æ is uncertain.
and sm
3.2 Approximation error analysis
To quantify the advantages of the ternary label over the binary label, we leverage EAE (Expected Approximation Error) [3] [5] which measures the error of approximating the true label description degree by a more accessible label such as binary label, multi-labe ranking, ternary label. The definition of EAE is formalized as follows:
Definition 1 Given a label with the true label description degree E I, if the label is annotated by reduced label with the approximate label description degree â I, then the expected approximation
3

---

error of the reduced label, i.e., (I, is quantified as:
v(2,I)=
1 ZEI ZEE VV dEde
V= ZEI
dz,
V
då.
(I)
According to Definition | it can be seen that EAE depends on I and I, so below we propose the following assumptions to determine I, İ and their relationships.
Assumption As stated the introduction, a label in practical cases can be positive (i.e., the label can describe the instance), negative (ie.. the label cannot describe the instance). and uncertain (i.e., the association between the label and the instance is uncertain). If label is positive, then the tree label description degree 2 E (K,, 1). If the label is negative, then the true label description degree E [0, r), where T E [0, k). 1he label is uncertain, then the true label description degree z E [T, x]] That is, I,.
V (s= I. = { (s= 1A= (K, (s= -11=
(2)
a label is annotated by ternary label then have the s-based label description degree 2 E Is.
Is = (3= (R,
(0, v 0 (+)
where 0 S
R S are the predefined parameters as approximations to and k. respectively, since T and K are unavailable from the annotation results.
If a label is annotated by a binary label b, then we have the b-based label description degree E I
Is {2: (b=-1 -1 (0) v (6 = 1b= (
where& & E [+ must hold, otherwise there exists some label description degree corresponding to "b A -1" or "b A which is semantically contradictory.
Assumption Since label be positive, negative uncertain in practical cases, we discuss the relationships between the true label and the annotation results in each of the three cases below.
If a positive or negative label (i.e., s = +1) is annotated by ternary value or a binary value b, then we have å = s. If an uncertain label (i.e., 0) is annotated by a ternary value or binary value b, then we have 3 = p0b = - 0) P. and p(: 1|s 0) = where 0 S p 1.
Next, we give the analytical form of EAE of the binary label and the ternary label.
Theorem Given a label equiprobably being positive, negative, and uncertain, if the label is annotated by a ternary value, then we have the expected approximation error:
2 (T (x) 2 (++
2
1 (fx RT) 3
1 (R)(
1 (1-k- R). (5)
6
18
If the label is annotated by binary value, then we have the expected approximation enror:
t +
1+
2-_ ÇT EN TK 3T + E E-3
E2 (I, I.]] =
6
9
3
18
(6)
where p p(: -1ss 0). Suppose that & ~ Uni(E & < & < R),
Uni(p
0 0 p $ 1).
[T, k| ~ Uni([T,<] 10 T < 1), we have
B. (2) (Ig, I,)] 36 (882 + 4ff - 12k + 882 - 8f + 7):
(7)
EB [Ig, In) 12x2 + 27A - 2F2 -3 3).
(B < E E. (Ig, if and only if - 2 + (f 1)² < 3/2 and + < R.

---

ur
0:18
0. 01
006
0.577
0s
-.15
150
-04
0.12
100
-0ors
00
00 01
02
(a)
Pg. (1 2.12 (
(h) E &. (T, I.)]
(c) E B (Tg, [I I.)]
Figure Distributions of the values of FB, L 128,, B12 [v(2,,211 E [v In]], and E, &. (I, I.) over [f, A]. where f. e.r. (T,, I.) and E, g. &. p.r, (I, I.)] (which are defined in Equation (7) measures the average EAE binary labels and ternary labels, respectively; PB,.-Iv (Is ,,, (V is defined in Equation
8 which measures the approximate proportion
of cases where the ternary label is inferior to the binary label for different [T, Kr P.
The detailed proof of Theorem 1 can be found in Appendix. Figure visualizes the results of Theorem Specifically, Figure ( shows the relationship between the predefined parameters [f, which defined as:
and P E, I.
FEL.INCIL.L..2B IVC
Σ (T I(E.
Iv I.]]),
(8)
G= ==[ KE r E r(k, 10 r(1, 10- r (1,
where I((-) is an indicator function that outputs if the internal condition is true, and 0 otherwise; r(u, 2) outputs an increasing sequence [0, v. 2u,.... u]. Equation (8) measures the approximate
proportion of cases where the ternary label is inferior to the binary label for all possible [T, &, pl in g.
Figure 2(b) and Figure 2(c) show the distributions of the average EAE of the binary label and ternary label, respec- tively. Obviously, the ternary label outperforms the bi- nary label in most cases. Specifically, the binary label shows superiority only the extreme cases, i.e., (f, R) {(0, 0), (0, 0.1), (0.9, 1), (1, 1)), but f and & basically never take these values practical applications since both the bi- nary and ternary labels exhibit very high approximation er- ror in these cases. In addition, Theorem [ also derives that (M),,, I E... (v. I.)] if and only if + (+- f)2 < and <. R, which is visualized as the overlapping area of the red and blue regions in Figure 3 can be seen that the overlapping area is essentially consistent to the blue area in Figure 2(a) Therefore, the ternary label is su- perior to the binary label w.r.t. approximating the ground-trutt -ruuh label description degrees.
11a 4
R
P.k
Figure 3: The visualization of
and (R 3)² + +
4 CateMO: Categorical distribution with monotonicity and orderliness
In order to learn label distributions from ternary labels, existing LE methods can be borrowed, fundamental frameworks of which have been visualized in Figure 4 and illustrated in Section 2 can observed that both discriminative and generative LE methods necessitate modeling the relationship between the label description degrees z and the more accessible labels y. Therefore,
5

---

Loss- Dist(z.8) + S2(z)
p 2, ...) p) 1z)p(z)
Dist(y, z)
(()
p(y)=)
p
y
z
y
Discriminative label enhancement
(b) Generative label enhancement
Figure 4: Fundamental frameworks of existing LE methods. y can be either binary labels or multi-label rankings. The learning target of discriminative LE methods can be decomposed as the inconsistency between y and the label description degrees z, i... Dist(z, y), and the regularization term of z based on other data, i.c., (2))). The joint distribution of the generation process in generative LE methods can be decomposed generative distributions of other observed variables, i.e., p(- p(....2.. (=)).
the conditional probability of y given 2, i.e., p(y]z), and the
we propose CateMO (Categorical distribution with Monotonicity and Orderliness) to model the conditional probability of ternary label given the label description degree. CateMO can serve as p(y|z) in generative LE methods, and the negative likelihood function of CateMO can be used as Dist y) in discriminative LE methods, address our task by replacing Dist(z, 8) and p(s|z) with CateMO. In the following subsections, we first provide intuitive discussion of the rules governing the generation of ternary labels and then formalize these rules as the assumptions about the probability monotonicity and orderliness of ternary labels. Furthermore, derive the parametric mass function for CateMO, which is theoretically guaranteed to maintain the probability monotonicity and orderliness of ternary labels.
that most existing LE methods can be employed to
4.1 Generation rules of ternary labels
On the one hand, we explore how the label description degree affects the probabilities of the label being positive, negative and uncertain. Obviously, label more likely to be positive if the description degree of the label is larger, label is more likely to be negative if the description degree of the label smaller. Besides, the uncertain label used to encode situations where the expert unsure whether the label can describe the instance, which arises from the fact that the probability of the label being positive close to the probability of the label being negative. Therefore, we believe that label more likely to be uncertain if the probabilities of the label being positive and negative are closer. The above intuitions can be formalized as follows:
Assumption (Probability monotonicity of ternary labels) Given any two real values en and 02
between 0 and 1, if 01 <, U2, then p(s = =1/2 v1) p(s 1|2 v2) and p(s =-1/2 vn) >
p(s = - = 12). 1f0 < p(s = 1/2 =|==0]] 01) - -p(s = -1/2 = 01) < p(s = 1/2 = (22) - -p(s
-1|2 02) or 0 < p(s= -1|= 11===01 U1) -p(s =1/== v1) < p(s -112 u2) pps 1/2 02).
then p(s 0,2 = 01) p(s = 00= U2).
On the other hand, we explore how the label description degree affects the orderliness among the probabilitics of the label being positive, negative and uncertain. Obviously, a label is most likely to be negative and least likely to be positive if the label description degree is sufficiently small; a label is most likely to be positive and least likely to be negative if the label description degree is sufficiently large; label is most likely to be uncertain if the label description degree is moderate. We formalize the intuition as follows:
Assumption 4 (Probability orderliness of ternary labels) There exists two real values 0 < 01 < 0/2 < 1, p(s 1/2) < p(s = 0|2) < p(s -11=) holds for any z < U1, p(s -1/2) < p(s 0|2) < p(s = 1|2) holds for any 2 > U2, max = 112).p(s = - -1/=)} < p(s = 0/2) holds for any 01 < 2 < 02-
4.2 Probability mass function of CateMO distribution
Since a ternary label can take three possible values, i.c., s E =0, we model ternary label by a categorical distribution, which can be formalized as follows:
p(s|2) Categorical(s [1((2), (p()), (())),
(9)

---

p(s
1
p(s 0
p(s 1
1.0
1.0
0.6
0.4
0.3
0.5
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
(a) A A, X, 2] = 12,
(b)
A A, X, = [10, 10, 20,
1
(c) ( X, A,
= [30, 5, 500,
10
Figure 5: p(s|2) on different parameters. The horizontal and vertical axes denote the label description degree and the probability p(s|z) defined by Equation (9) and Equation [T0, respectively.
where (0(=) = p(s -1[=), ((2) = p(s = 0f=), and s==) p(s 1|=) represent the generation principles from the label description degree to the negative, uncertain, and positive labels, respectively.
We name (0=), p02), and (22) as ternary generation functions. According to Assumption 3 preliminarily assume the parametric form of ternary generation functions as follows:
we
((2)= = 7
1
A=
(0() =
1 M(= 7
F(z))
M(=11)2
(10)
7
where Z = e A=
e A(z-2)² te - ,> 0. x> 0, A 0 and 0 < 1 are parameters. Intuitively, the parameters (A, X, A) largely governs the precision of CateMO distribution (despite the fact that they are not exactly equal), which is similar to the reciprocal of the temperature coefficient in the softmax layer deep neural network. Hence, we refer to the parameters X, X, and À as the precision of negative labels, positive labels, and uncertain labels, respectively. Figure 5 visualizes the shape of p[)[z) on different parameters, which can be found that some parameter configurations violate the above three assumptions. For example, Figure 5(a) violates Assumption 3 Figure 5(c) violates Assumption 3 and Assumption 4 at the same time. Therefore, we propose Theorem 2 to ensure that the ternary generation functions defined by Equation (IO) satisfy the proposed assumptions about probability monotonicity and orderliness of ternary labels.
Theorem Given À > 0, >>>>>> O, and X >>>00 0, the ternary functtono satisfy Assumption and Assumption 4 if the following conditions hold:
A - + max (2 + (1--))-))) <A A(1 - 2) -)-1,,211}
(11)
The details of the proof can be found in Appendix. Therefore, the probability mass function of CateMO distribution can be formalized as follows:
CateMO(s = Z A(2- 1)3 CateMO = 0 Z A(2-2))
CateMO(s -1 7-1 2 20 +e- A)=- + -T(--1)
(12)
s.t. 2 (2AV 22 -1
A - - 1 -
A.A. 0, max (3 + 207) - (11+ -1) -)) } < min -
Finally, to apply CateMO, simply replace Dist(y, y, in Figure 4(a) with the negative log- likellhooood of CateMO and replace p(y(2) in Figure 4(b) with CateMO, which can be formalized as follows:
M
Dist( 2) = Σ
log CateMO 8m 1mm) p(s|2) II m=1
CateMO im). (13)

---

5 Experiments
5.1 Datasets and evaluation measure
Although there are many LDL datasets, they all lack ground-truth ternary label data. Therefore, we select three real-world LDL datasets (i.e., JAFFE [T]. Painting [9], and Music [.], and manually re-annotate them with both binary labels and the ternary labels. The details of these datasets can be found in Appendix. We use five common LDL metries to evaluate the algorithm performance, which are Cheb (Chebyshev distance), KL "Kullback-Leible divergence), Cosine (cosine coefficient) (2), and Rho (Spearman's rho coefficient) 77. The lower values of Cheb and KL indicate the better performance. The higher values of Cosine, Intersec, and Rho indicate the better performance. We use ues and "4" to denote that the better performance represented by the higher and lower values of a metric, respectively.
5.2 Comparison methods and experimental procedures
Comparison methods Since there no LE method specifically for ternary labels, we design two approaches to construct the comparison algorithms. On the one hand, we design data transformation method (which is abbreviated as DT method) to transform the dataset with ternary labels into an extended dataset with binary labels so that any existing LE algorithm can be applied to ternary labels. Specifically, for any instance with uncertain labels, we transform the instance into two instances which set all uncertain labels to positive and negative labels, respectively. For instance, the example [x. [1, 0, 0]] will be transformed into x [1, 1, 1] and [x, [1, -, -1], respectively. On the other hand, we replace the loss term Dist( b) with MSE (s + 1)/2 z =12.. so that most existing LE methods can be used to enhance ternary labels. We abbreviate this method as MSE method. Besides, we select three recently proposed binary label enhancement algorithms: GL [29] LR [6], and MR [12]. The hyperparameter settings follow their respective literature. We combine GL, LR and MR algorithms with DT and MSE methods in pairs to construct six comparison algorithms: GL-DT, GL-MSE, LR-DT, LR-MSE, MR-DT, and MR-MSE. In terms of our proposal, we replace the conditional distribution p(blz) in MR with CateMO and replace Dist b) in GL. and LR with the negative log-likelihood function of CateMO, which constructs three algorithms: Gl.CateMM, LR-CateMO, and MR.CateMO. We set the parameters [ X, A] in CateMO as [49, 48, 12].
Experimental procedures We aim to test the performance of label distribution prediction based on different LE algorithms. Specifically, we use different LE methods to recover training label distributions and use these recovered label distributions to train an LDL model, whose performance on test instances will be reported. LDL-LRR 77) is used as the LDL model in this paper, whose hyperparameters À and / are selected from (10-6, 10-5
10-1) and {10-3 10-2 suggested [7]. We randomly partition the whole dataset (70% for training and 30% for testing), and repeat the above process ten times and report the average and standard deviation of the results.
102) as
5.3 Results and discussions
Table shows the prediction performance of the comparison algorithms on three datasets. Each result is formatted as "mcanistd". In the first column of Table I "MR", "LR" and "GL" denote the existing binary LE algorithms, the suffix "-LL." denotes that these algorithms run on binary labels directly, and the suffixes "-DT", "-MSE" and "-CatMMO" denote that these algorithms ternary labels by the DT method, MSE method and our proposed CateMO distribution, respectively. "Ground-Te -Truhh denotes that LDL-LRR is trained directly the ground-truth label distributions. In each area separated by dashed lines, bold and italics denote the 1st and 2nd, respectively. The results of statistical significance test are shown in Appendix. It can be seen that the performance of our proposed CateMO is better than other three comparison algorithms in all cases, and is close to "Ground-Trutee To visualize the advantages of ternary labels over binary labels, we show the relationship between binary labels, ternary labels, and label distributions in terms of annotation time and prediction performance in Figure The horizontal axis denotes the average time (in seconds) spent by an expert in annotating a label for an instance. The vertical axis denotes the average prediction performance calculated from Ta It can be seen that temary labels is superior binary labels in terms of both prediction performance and annotating cost.
8

---

Table 1: Results shown "sda where bold and italics denote the 1st and 2nd, respectively.
Method
Cheb ())
KL (1)
Cosine (t)
Intersec (1)
Rho (1)
JAFFE
Ground Tround-Truth
0.165 0.010
+115
0.897 = 0.009
0.793 0,011
0.463 = 0.051
MR-CateMO
0.169 +.011
0.154 0.015
0.892 + 0.009
0.791 0.011
0.477 = 0.051
MR-MSE
0.113
0.166 1 0.017
0.884 + 0.010
0.774 1 0.014
0.432 = 0.036
MR-DT
0.177 0.011
0.163
0.012
0.886 + 0.006
0.781 + 0.010
0.407 = 0.033
MR-LL
0.180
0.171 0.014
0.883 + 0.009
0.774 + 0.008
0.383 = 0.028
LR-CateMO
0.170 0.011
0.156 0.015
0.889 - 0.010
0.785 1 0.010
0.443 = 0.040
LR-MSE
0.178 1 0.012
0.161 1 0.017
0.884 1 0.011
0.778 1 0.011
0.422 0.045
LR-DT
0.178 0.012
0.161 + 0.017
0.884 = 0.011
0.778 + 0.011
0.424 = 0. 0.044
LR-LL
0,201
0.145
0.847
0.738
0.320 0.072
GL-CateMO
0.174
0.158 0.016
0.887 = 0.011
0.782 0.011
0.432 0.046
GL-MSE
0.178
0.012
0.162 0.016
0.884 = 0.011
0.777 + 0.011
0.482 = 0.044
GL-DT
0.206 + 0.017
0.210 + 0.023
0.857 + 0.013
0.745 + 0.016
0.381 = 0.058
GL-LL
0.199 0.014
0.311 + 0.156
0.852 + 0.024
0.742 + 0.027
0.323 = 0.078
Painting
Ground-Truth
0.252
0.737 = 0.007
1605:000000
0.316 = 0.038
MR-CateMO
0.262 0.010
0.561 0.022
0.723
0.008
0.593 0.010
0.275 0.033
MR-MSE
0.262 1 0.011
0.564 + 0.029
0.718 + 0.013
0.592 1 0.019
0.255 = 0.051
MR-DT
0.263 1 0.008
0.562 1 0.019
0.718 1 0.009
0.592 1 0.010
0.273 0.023
MR-LL
0.263 + 0.008
0.566 + 0.019
0.715 = 0.008
0.591 + 0.010
0.265 = 0.017
LR-CateMO
0.545 1.050
0.731 = 0.008
0.602 0.010
0.313 = 0.044
LR-MSE
0.266 + 0.011
0.575 + 0.025
0.712 = 0.011
0.587 0.012
0.229 = 0.033
LR-DT
0.255 0.009
0.549 = 0.025
0.730 = 0.009
0.601 + 0.011
0.310 = 0.042
LR-LL
0,268
0.617 0.044
0.685 = 0.018
0.576
0.195 = 0.069
GL-CateMO
0.254 0.010
0.542 0.022
0.734 - 0.008
0.604 0.011
0.316 0.046
GL-MSE
0.265 0.010
0.571 + 0.027
0.715 0.012
0.588 0.012
0.240 0.044
GL-DT
0.256
0.011
0.590 + 0.026
0.719 - 0.011
0.591 + 0.012
0.287 = 0.039
GLALI
0.260 + 0.010
0.610 0.055
0.710 + 0.008
0.588 + 0.011
0.224 0.068
Music
Ground-Truth
0.072
0.103 0.000
0.925 = 0.004
0.821 0.007
0.510 = 0.025
MR-CateMO
0.076 0.002
0.107 0.007
0.921 + 0.005
0.820 0.007
0.502 0.027
MR-MSE
0.076 ::6::00 0.002
0.108 0.006
0.920 = 0.004
0.817 1 0.007
0.469 = 0.029
MR-DT
0.088 1 0.006
0.118 1 0.009
0.910 1 0.008
0.808 1 0.009
0.496 - 0.034
MR-LL
0.083 0.004
0.115
0.912 + 0.008
0.813 0.009
0.499 = 0.022
LR-
0.103 0.002
0.924 0.005
0.820
0.506 0.024
LR-MSE
+.002
0.104 + 0.006
0.924 - 0.005
0.819 + 0.007
0.501 = 0.027
LR-DT
0.073 0.002
0.104 + 0.007
0.924 0.005
0.819 + 0.007
0.504 0.027
LR-LL
0.004
0.150 = 0.033
0.083
0.909 = 0.008
0.806 + 0.010
0.503 = 0.028
GL-CateMO
0.073
0.103 0.007
0.924
0.005
0.821 + 0.007
0.505
0.026
GL-MSE
0.073 1 0.002
0. 0.104 1 0.007
0.924 1 0.005
0.819 1 0.007
0.501 - 0.028
GL-DT
0.084 + 0.002
0.126 + 0.007
0.004
0.809 1 0.007
0.499 = 0.028
GL.LI.I
0.078 + 0.003
0.137 + 0.009
0.906 + 0.006
0.796 + 0.008
0.481 = 0.033

---

Binary label
Ternary label
Label distribution
Cheb (4)
KL (4)
Cosine (t)
Intersec (1)
Rho (1)
0.180
0.32
0.85
0.42
0.175
0.30
0.84
0.73
0.10
0.190
0.28
0.83
0.72
0.38
0.165
0.36
0.70
0.75
0.80
0.70
0.75
0.80
0.70
0.75
0.80
0.70
0.73
0.80
0.70
0.75
0.80
Annotating time
Annotating
Anzotating
Annotating time
Annotating
Figure 6: Cost-benefit analysis different forms of labels. The horizontal and vertical axes denote the average annotating time (in seconds) and performance, respectively.
Precision of negative labels
Precision of uncertain labels
Precision of positive labels
KI. (J) JAFFE
KI. (I.()) Painting
KT. (4) on Music
0.50
0.09
0.49
0.09
0.10
0.08
10
20
50
10
20
30
40
30
10
20
30
40
50
Values precision parameters
Values pracision parameters
Values precision parameters
Figure Recovery performance GI.-CateMO with varying precision parameters.
5.4 Effect of precision parameters
Figure shows how the precision parameters affect the recovery performance of GL-CateMO on JAFFE, Painting, and Music datasets. The recovery performance is computed by the following two steps. First, we run GL-CateMO algorithm on the dataset with ternary labels to recover the label distributions for instances. Second, we calculate the KI. divergence between the recovered label distributions and the ground-truth by the LDL metries. For certain precision parameter, the other two precision parameters are set to the values that give rise to the best recovery performance.
Limitations and conclusion
I Limitations In this paper, the parameters (i.e., A. X, and λ) of CateMO are pre-fixed. In fact, more adaptive approach is collaboratively learn these parameters and other model parameters. However, since the parameters of CateMO satisfy the conditions shown in Theorem in which the parameters are interdependent, may lead to difficulties in directly exploiting gradient descent optimization methods. Therefore, in future works, will explore how to appropriately reduce the parameter space of CateMO so that the parameters A, X, and À can be decoupled from each others.
Conclusion In this paper, we propose to predict label distribution from ternary labels, which reduces both the annotation inaccuracy and cost when contrasted with the traditional binary annotating methods. In the theoretical part, we analyze the approximation error for both ternary and binary labels, which provides quantitative elucidation of the superior performance of the ternary label. In the methodological part, we propose the CateMO distribution to model the mapping from label description degrees to ternary labels, which is theoretically constructed to maintain the monotonicity and ordinality of the probabilities associated with ternary labels. the experimental part, extensive experiments demonstrate the effecttveness of our proposal.
7 Acknowledgments
This work was partially supported by the National Natural Science Foundation China (62176123, 62476130), and the Natural Science Foundation of Jiangsu Province (BK20242045).
10

---

References
[1] Binbin Gao, Hongyu Zhou, Jianxin Wu, and Xin Geng. Age estimation using expectation of label distribution learning. In International Joint Conference on Artificial Intelligence, pages 712-718, 2018.
[2] Xin Geng. Label distribution learning. IEEE Transactions on Knowledge and Data Engineering. 28(7):1734-1777 2016.
[3] Xin Geng and Peng Hou. Pre-release prediction of crowd opinion on movies by label distribution learning. In International Joint Conference Artificial Intelligence, pages 351 1-3517, 2015.
[4] Xin Geng, Kate Smith-Miles, and Zhi-Hua Zhou. Facial age estimation by learning from label distributions. IEEE Transactions Pattern Analysis and Machine Intelligence, 35:2401-244 2-112, 2013.
[5] Peng Hou, Xin Geng, and Min-Ling Zhang. Multi-label manifold learning. In AAAI Conference on Artificial Intelligence, pages 1680-1686, 2016.
[6] Xiuyi Jia, Yunan Lu, and Fangwen Zhang. Label enhancement by maintaining positive and negative label relation. JEEE Transactions on Knowledge and Data Engineering, 35(2): 1008- 1720, 2023.
[7] Xiuyi Jia, Xiaoxia Shen, Weiwei Li, Yunan Lu, and Jihua Zhu. Label distribution learning by maintaining label ranking relation. IEEE Transactions on Knowledge and Data Engineering. 35(2):1695-11) 2023.
[8] Xiuyi Jia, Xiang Zheng, Weiwei Li, Changqing Zhang, and Zechao Li. Facial emotion dis- tribution learning by exploiting low-rank label correlations locally. In JEEE Conference Computer Vision and Pattern Recognition, pages 9841-9850, 2019.
[9] Harin Lee, Frank Hoeger, Marc Schoenwiesner, Minsu Park, and Nori Jacoby. Cross-cultural mood perception pop songs and its alignment with mood detection algorithms. In Interna- tional Society for Music Information Retrieval Conference, 2021.
[10] Xinyuan Liu, Jihua Zhu, Zhongyu Li, Zhiqiang Tian, Xiuyi Jia, and Lei Chen. Unified framework for learning with label distribution. Information Fusion, 75:116-130, 2021.
[11] Xinyuan Liu, Jihua Zhu, Qinghai Zheng, Zhongyu Li, Ruixin Liu, and Jun Wang. Bidirectional loss function for label enhancement and distribution learning. Knowledge-Basd Based System, 213:106690, 2021.
[12] Yunan Lu, Liang He, Fan Min, Weiwei Li, and Xiuyi Jia. Generative label enhancement with gaussian mixture and partial ranking. In AAAI Conference on Artificial Intelligence, pages 2023.
[13] Yunan Lu and Xiuyi Jia. Predicting label distribution from multi-label ranking. In Advances in Neural Information Processing Systems, pages 36931-36943, 2022.
[14] Yunan Lu, Weiwei Li, and Xiuyi Jia. Label enhancement via joint implicit representation clustering. In International Joint Conference on Artificial Intelligence, pages 4019-4027, 2023.
[15] Yunan Lu, Weiwei Li, Huaxiong Li, and Xiuyi Jia. Predicting label distribution from tie- allowed multi-label ranking. IEEE Transactions on Pattern Analysis and Machine Intelligence, (5(12):15364-1533 2023.
[16] Yunan Lu, Weiwei Li, Huaxiong Li. and Xiuyi Jia. Ranking- preserved generative label enhance- ment. Machine Learning, 112:4693-4722 2023.
[17] Jiaqi Lv, Ning Xu, Renyi Zheng, and Xin Geng. Weakly supervised multi-label learning via label enhancement, In International Joint Conference on Artificial Intelligence, pages 3101-3107, 2019.
11

---

[18] Michael Lyons, Shigeru Akamatsu, Miyuki Kamachi, and Jiro Gyoba. Coding facial expressions with gabor wavelets. In IEEE Interattonaal Conference on Automatic Face and Gesture Recognition, pages 200-205, 1998.
[19] Jana Machajdik and Allan Hanbury. Affective image classification using features inspired by psychology and art theory. In ACM International Conference on Multimedia, page 83-92, 2010.
[20] Ruifeng Shao, Xin Geng, and Ning Xu, Multi-label learning with label enhancement. In IEEE International Conference on Data Mining, pages 437-446, 2018
[21] Haoyu Tang, Jihua Zhu, Qinghai Zheng, Jun Wang, Shanmin Pang, and Zhongyu Li. Label enhancement with sample correlations via low-rank representation. In AAAI Conference Artificial Intelligence, pages 5932-5939, 2020.
[22] Grigorios Tsoumakas and loannis Katakis. Multi-label classification: An overview. Interna- tional Journal of Data Warehousing and Mining, 3(3):1-13, 2006.
[23] Tao Wen, Weiwei Li, Lei Chen, and Xiuyi Jia. Semi-supervised lahel enhancement via structured semantic extraction. International Journal of Machine Learning and Cybernetics, 13:1131-
1 1144, 2021.
[24] Xin Wen, Biying Li, Haiyun Guo, Zhiwei Liu, Guosheng Hu, Ming Tang, and Jinqiao Wang. Adaptive variance based label distribution learning for facial age estimation. In European Conference Computer Vision, pages 379-395, 2020.
[25] Ning Xu, Yun-Peng Liu, and Xin Geng. Label enhancement for label distribution learning. IEEE Transactions on Knowledge and Data Engineering, 33: 13:1-311643, 2021.
[26] Ning Xu, Jiaqi Lv, and Xin Geng. Partial label learning via label enhancement. In AAAI Conference on Artificial Intelligence, pages 5557-5564, 2019.
[27] Ning Xu, Jun Shu, Yun-Peng Liu, and Xin Geng. Variational label enhancement. In International Conference Machine Learning, volume 119, pages 10597-10606, 2020.
[28] Ning Xu, Jun Shu, Renyi Zheng, Xin Geng, Deyu Meng, and Min-Ling Zhang. Variational label enhancement. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(5):6537- 6551, 2023.
[29] Ning Xu, An Tao, and Xin Geng. Label enhancement for label distribution learning. In International Joint Conference on Artificial Intelligence, pages 1632-1643, 2018
[30] Jufeng Yang, Dongyu She, and Ming Sun. Joint image emotion classification and distribution learning via deep convolutional neural network. In International Joint Conference Artificial Intelligence, pages 3266-3272, 2017.
[31] Yiyu Yao. The dao of three-way decision and three-world thinking. International Journal of Approximate Reasoning, 162: 2023.
[32] Min-Ling Zhang, Qian-Wen Zhang, Jun-Peng Fang, Yukun Li, and Xin Geng. Leveraging implicit relative Transactions on Knowledge and Data Engineering, 33:2057-2070, 2021.
information for effective multi-label learning. 1EEE
[33] Qian Zhang, Yun Zhong, and Min-Ling Zhang. Feature-induced labeling information enrichment for multi-label learning. In AAAI Conference on Artificial Intelligence, pages 4446-4453, 2018.
[34] Qinghai Zheng, Jihua Zhu, Haoyu Tang, Xinyuan Liu, Zhongyu Li, and Huimin Lu. Generalized label enhancement with sample correlations. IEEE Transactions on Knowledge and Data Engineering, 35(1):482 2023.
[35] Deyu Zhou, Xuan Zhang, Yin Zhou, Quan Zhao, and Xin Geng. Emotion distribution learning from texts. In Conference on Empirical Methods in Natural Language Processing, pages 638-647, 2016.
12

---

A Appendix supplemental material
A.1 Proof of Theorem 1
Given a label equiprobably being positive, negative, and uncertain, if the label is annotated by a ternary value, then we have the expected approximation error:
Ea [Zs, I.)]
Emc(o.t)) (66 u, 8 (T Iamu)
S So (2 ddde
St (2-2)
Sx SN - d22d
+
3 fr
(R 35-Fi (N -T)
3 (1 R)(1 k)
2 ( + x)² + 2 ( R)2
1 (fx + RT) 3/2 (f + k) + + (1 - k R).
1
1
9
6
3
18
(14)
If the label is annotated by a binary value, then we have the expected approximation error:
1 (T. TT]] =
Σ
w==1 p(b
u, s
(I Ie=u) + p(6 u, s = Ix=0)
T T+ R
1 9
x) 62 fT Ex- TK
35 + 4x
--3
=
6
3
18
(15)
where p === =pp( -1 1/8 = 0). Suppose that & ~ Uni(é | i & S R), p Uni(p 10s
[T ~ Uni((T,A) x²]
0 5 T S K S 1), we have
-1 ( J drdk
L/'
1 Ba, (e,, [I)]dddddddk
36 '8822 + 44R 12
872 88 77)
(16)
So S So J²¹ dÉdpdrdk [- J Jo J
NN
1
E (T =
Es J.ldEddrddd
16 (2R² + 2FR 38 + 24² - 3F 3).
Furthermore,
Eg [I I.)] S E p.
8h + 47R
12A
8F2
88
+ 7 48² + 47k 6Å + 472
67 +
3
1
1
+ 42
7
+
50
(17)
2
4
3 2
1
k
+
f
-
=
<
4
4
8
A.2 Proof of Theorem
To ensure that Equation I0 satisfies the probability monotonicity of negative and positive labels, just need to ensure that 0z(z)/0= < 0 and 0p(z)/Oz > O. The partial derivative of (o(2) w.r.t. 2 can be formalized as:
ax(2)
(o2 2 =d))e + (A= - x )=)e">) c3.
(18)
where U1, U2 03 > 0. Since Àe - 7- de < 0, (Nz - d=)e"= < 0. Therefore, 0g(z)/02 < 0 if
A= 2 Az < 0 holds for any E (0, 1). This inequality can be equivalently transformed into:
2 X
1
(19)
z
Similarly, the partial derivative of (2) w.r.t. can be formalized as:
0<(2z)
((=) 21 Ae De + (X-
A + A2)e²²) vo.
(20)
Oz
13

---

where U1 Us. > 0. Since π - πy x 0, + > 0. Therefore, 0z(z)/0z > 0 if 2\ - 2 X + X > 0, i.e., X(= - -}> > -=-1) holds for any 2 E (0.
1). On the one hand, if
< = < 1, then À > X(= () -) On the other hand, if 2 < 2, then < I X(=-1)
To ensure that Equation 1T) satisfies the probability orderliness, we just need to ensure that equations
(=)) - (=() (i.e., X(= 1) X(= 2) and :0(=) *=()) (i.e. A=² >(= =) have only one
solution on E (0, 1). We denote g (2) = X(2 - 1)2 >(2 - 2) Since 91 (1) = - 1)2 < 0
and (0) = x - =² > λ - =²\=-1 = A(1 - 8) > 0. the quadratic equation gi (2) = 0 has
only one solution in
10. Similarly,
denote 92(z)
A=² X(= 2)² Since go (1(1
2Eλ 1-x-220 2
- X + À- À (1 (1-2) (1-2)>>>)) 1 -1
2) 2) -1 = = > 0 and 22 (2 = - _2 X < 0.
the quadratic equation g2(2) 0 has only one solution in Z E (0, 1).
To ensure that Equation (T0) satisfies the probability monotonicity of uncertain label, we just need to ensure that d'(=)//2z > 0 holds for any 0 < 2u and 0e(=) /0z 0 holds for any 20 < 1, where (20) = ((z)). i.c., 20 = -V vA) That is, make sure that 2g is the only point of maximum value of (=z). The partial derivative of (==) w.r.t. & is:
0p((2)
01 ((E) - du Az)e + (2) A ()
(21)
Oz
where 21 > We aim make sure that f(z) 0xp(z)/0z = 0 has only one solution in = E (0, 1)
22 A 22) A2 Az - X) -)- exp(2Xx + d=² X=² -
(22)
f(z)
s(=)
We first need examine whether the denominator of f()) is zero. Suppose that it is zero, we have (A - 2A - A)-1. Considering that X _λ À- 2\
if À > X then
2 2 > 1, which contradicts 2 (0, 1). If < X, then (1=- which also contradicts z E (0, 1). Besides, it is obvious that the denominator of f(z) is non-zero when X = X. Therefore, the denominator of f2) is non-zero. Since whether the partial derivative of f()) w.r.t z (i.c, 0f(z)/0= = (....) ) ?aYX 2AA - ΛĀ + AX)) is positive or negative has nothing to do with z, we ensure the monotonicity of fz) by the following inequality:
+ -X 2 22 X)
(23)
Besides, since 0g(z)/0z = (2X(1 2) + 2X2) exp(-) > 0, g(2) is increasing. Therefore, to make sure that ppz) has only one maximal point, we just need to ensure that f(0) > g(0) and f(1) < g(1), i.e.
A max { (a + cx) X, ((1 + exp()))(1 2)) 2)) 12
(24)
Finally, let the partial derivative of (=)) at 20 to be zero, we have
(Azo \zo) exp (=) + 1)) + (Azo - dzo + exp (0 (2T + )zo))
(25)
cexp A [=] + 1)) + Acxp (zo (2X + )20))
Considering 20
we have 2 = (2Xv + 22v (2AV -2
A.3 Datasets
JAFFE dataset The "JAFFE" dataset 18] comprises 213 facial emotion images posed by ten Japanese female models. Concerning the feature data, we employ the feature extraction method suggested in [2] to compress each image into 243-dimensional feature vector. As for the label data, each image is annotated with scores from {1,2, 3, 4. 5) by 60 individuals, indicating the relevance of the corresponding emotion to the facial image. The average score for each emotion utilized to represent the intensity of the emotion. Subsequently, we apply min-max normalization to scale the average scores to the interval (0. 1) and normalize the average scores of all emotions into the form of probability distributions to obtain the label distribution data. Besides, three experts annotate the instances in JAFFE with ternary labels and binary labels, and the final ternary labels and binary labels are determined by majority voting.
14

---

Painting dataset The "Painting" dataset [9 is created for exploring emotions within abstract art- works. Each image represented by d2-eemnnsinnll feature vector comprising three components: histogram features of the RGB attributes of the image, histogram features of the HSV attributes of the image, and GI.CM (Grey-Levvel Co-occurrence Matrix) features of the image. Regarding the label data, approximately 230 individuals annotate emotional scores across eight categories: amusement, anger, awe, content, disgust, excitement, fear, sadness. Each image received about 14 annotations. Subsequently, we apply min-max normalization to scale the average emotional scores to the interval (0, 1) and normalize the average scores all emotions into the form probability distributions to obtain label distribution data. Besides, three experts annotate the instances in Painting with ternary and binary labels, and the final ternary and binary labels are determined by majority voting.
Music dataset The "Music" dataset is extension of music dataset [9] and encompasses 360 popular songs from major music charts across different countries. We employ MFCC (Mel Frequency Cepstral Coefficients) to extracte
feature vector from each song and utilize PCA (Principal Component Analysis) to compress this vector into a 128-dimensional feature vector. Regarding the label data, participants from the UK, South Korea and Portugal rate their perceived moods (i.e., calm, tense, cheerful, sad, danceable, love, dreamy, electronic, and energy) for a given song on 4-level scale. Subsequently, interval (0, 1), and normalize the average scores of all moods into the form of probability distributions obtain label distribution data. Besides, three experts annotate the instances in Music with ternary and binary labels, and the final ternary and binary labels are determined by majority voting.
apply min-max normalization scale the scores the
A.4 Results of statistical significance test
Table 2: The counts of win/tie/loss after comparing CateMO with other comparison algorithms under pairwise two-tailed t-test with 0.05 significance level. Each entry is formatted as "win/tie/loss".
Method
Cheb
KL
Cosine
Intersec
Rho
MSE
5/4/0
5/4/0
6/3/0
5/4/0
7/2/0
DT
5/4/0
6/3/0
7/2/0
5/4/0
6/3/0
LL
8/1/0
9/0/0
9/0/0
8/1/0
9/0/0
Ground-Truth
0/5/4
0/4/5
0/4/5
0/5/4
0/4/5
We use pairwise two-tailed t-test with 0.05 significance level test whether CateMO is statistically superior or inferior to MSE, DT, LL, and ground-truth on three real-world datasets when combined with MR, LR, and GL. The counts of win/tie/loss after comparing CateMO with other comparison algorithms are shown in Table
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims made in the abstract and introduction clarifies both the theoretical and methodological contributions the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
The assumptions of the theoretical result are formally described in Assump-
tion Assumption 2 Assumption and Assumption in Section 3.2 and Section 4.1 The proof of Theorem is shown in Appendix A.T. and the proof of Theorem is shown in Appendix A.2
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
we give the details of the comparison algorithms, including the hyperparameter configurations and the method of dataset partitioning. For our proposed CateMO, we also give the values of X, X, X.
Justification: In Section 5.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: In terms of the code, the implementation of proposed CateMO is simple that no additional code files are needed to describe it. In terms the data, open access to the datasets involved in this paper requires license from the corresponding creator, and this paper is not authorized to distribute them.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are shown in Section 5.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The mean value and the standard derivation of the prediction performance is shown in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Computer resources have a negligible effect on both the experimental results and the main claims of this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in this paper does not violate the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all original papers that produced the datasets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
The expert's annotating task is as follows. expert should annotate 1 if the label can describe the instance, -1 if the label cannot describe the instance, and 0 if it is uncertain whether the label can describe the instance. This process is described in Section
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: There is no potential risk incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "1begRLwCkA", "reviewer": "Reviewer_uVMp", "summary": "The authors of this paper propose to predict label distribution from ternary labels, i.e., “0” indicating “uncertain relevant”, “1” indicating “definitely relevant” and “-1” indicating “definitely irrelevant”. Besides, they also provide theoretical analysis to show that the ternary label outperforms the binary label in most cases.", "questions": "1. Why do you use the Eq.(10) as the ternary generation functions?\n2. Are there any guidelines for selecting the parameters in Eq.(10)?\n3. Is the proposed CateMO method effective in generating accurate label distribution?", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is intuitive and straightforward.\n2. This paper proves that the ternary label is superior to the binary label in terms of expected approximation error in most cases.", "weaknesses": "1. The proposed Categorical distribution with Monotonicity and Orderliness (CateMO) contains four parameters, and parameter-sensitive analyses are absent.\n2. It is not explicitly shown whether the label distribution degrees generated by the CateMO are close to the real label distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Weakness (1) and Question (2)\n\nWe are very grateful for your precious suggestions. We additionally perform experiments to analyze the parameter sensitivity. The results are shown in Figure 1 in the submitted PDF in global responses, which will be added to the revised version if space permits.\n\nActually, there are only three parameters in CateMO, since the paremeter $\\\\hat z$ can be determined by the other parameters according to Eq. (12). In terms of how to determine the parameters $\\\\underline{\\\\lambda},\\\\lambda,\\\\overline{\\\\lambda}$ in Eq. (10), we offer the following two approaches.\n\n- End-to-end Learning. The most straightforward way to determine the parameters $\\\\underline{\\\\lambda},\\\\lambda,\\\\overline{\\\\lambda}$ is to co-optimize these parameters with other parameters in the learner in an end-to-end manner. This approach is capable of obtaining parameters adapted to the characteristics of the data distribution, but corresponds to a complex constrained optimization problem that requires a tailored optimization algorithm since the constrains of $\\\\underline{\\\\lambda},\\\\lambda,\\\\overline{\\\\lambda}$ are interdependent.\n\n- Pre-annotations Fitting. The parameters can also be estimated from a small number of pre-annotated data pairs $\\\\{(s\\_n,z\\_n)\\\\}\\_{n=1}^L$, since $p(s|z)$ possesses a relatively stable morphology across different datasets. Figure 2 in the submitted PDF in global responses shows the distribution of some of our annotation results on the Painting, Music and JAFFE datasets. Specifically, we randomly choose an instance and a label from the JAFFE, Painting, and Music datasets, and asked the experts to simultaneously annotate the instance-label relationship by a ternary value and a description degree value. The above process is repeated six hundred times (two hundred times for each dataset). All annotation results are collected and recorded as $\\\\mathcal A=\\\\{(s\\_n,z\\_n)\\\\}\\_{n=1}^L$. Figure 2 in the submitted PDF in global responses shows the empirical distribution of $p(s|z)$ according to $\\\\mathcal A$, which indicates that there are no significant differences in the empirical distributions of $p(s|z)$ across datasets. Therefore, we can estimate the parameters of CateMO reliably with a small amount of pre-annotated data pairs. Formally, we obtain the parameters by maximum likelihood estimation, i.e.,\n\n$$\n\\\\begin{aligned}\n&\\\\underline{\\\\lambda}^\\\\star, {\\\\lambda}^\\\\star, \\\\overline{\\\\lambda}^\\\\star \\\\leftarrow \\\\arg\\\\max\\_{\\\\underline{\\\\lambda}, {\\\\lambda}, \\\\overline{\\\\lambda}}\\\\quad \\\\sum\\_{(s\\_{n}, z\\_{n}) \\\\in \\\\mathcal A} \\\\log p(s\\_{n} \\\\vert z\\_{n}) \\\\\\\\\n\\\\text{s.t.}\\\\quad & {\\\\lambda} < \\\\min \\\\{{\\\\underline{\\\\lambda}}{(1-\\\\hat{z})^{-1}}, \\\\overline{\\\\lambda} \\\\hat{z}^{-1}\\\\},\\\\\\\\\n&{\\\\lambda} \\\\neq {-\\\\underline{\\\\lambda}\\\\overline{\\\\lambda}}{(\\\\hat{z}\\\\overline{\\\\lambda} - \\\\hat{z} \\\\underline{\\\\lambda} - \\\\overline{\\\\lambda})^{-1}}, \\\\\\\\\n&{\\\\lambda} > \\\\max\\\\left\\\\{\\\\left( \\\\hat{z} + \\\\hat{z} \\\\exp(\\\\overline{\\\\lambda}) \\\\right)^{-1}\\\\overline{\\\\lambda}, ((1+\\\\exp(\\\\underline{\\\\lambda})) (1-\\\\hat{z}))^{-1} \\\\underline{\\\\lambda} \\\\right\\\\} , \\\\\\\\\n& \\\\hat{z} = \\\\left( 2{\\\\lambda} \\\\sqrt{\\\\overline{\\\\lambda}} + 2{\\\\lambda} \\\\sqrt{\\\\underline{\\\\lambda}} \\\\right)^{-1} \\\\left(2{\\\\lambda} \\\\sqrt{\\\\overline{\\\\lambda}} - \\\\underline{\\\\lambda} \\\\sqrt{\\\\overline{\\\\lambda}} + \\\\overline{\\\\lambda} \\\\sqrt{\\\\underline{\\\\lambda}}\\\\right).\n\\\\end{aligned}\n$$\n\n\nBesides, in order to provide users with a more intuitive understanding of CateMO's parameters, we show the shape of CateMO with varying $\\\\underline{\\\\lambda},\\\\lambda,\\\\overline{\\\\lambda}$ in Figure 3 in the submitted PDF in global responses, which visualizes how the parameters $\\\\underline{\\\\lambda},\\\\lambda,\\\\overline{\\\\lambda}$ detemine CateMO.\n\n## Responses to Weakness (2) and Question (3)\n\nThank you for your questions. In Figure 3 in the submitted PDF in global responses, we show, across three datasets, the proximity of the label distributions recovered by CateMO to the ground-truth label distributions. From the experimental results, it can be seen that the label enhancement algorithm based on CateMO achieves significant advantages over others.\n\n## Responses to Question (1)\n\nThe proposed ternary generation function can be expressed as\n\n$$\n\\\\mathrm{softmax}(-\\\\underline{\\\\lambda} z^2,-\\\\lambda (z-\\\\hat z)^2, -\\\\overline\\\\lambda (z-1)^2),\n$$\n\nwhere $\\\\mathrm{softmax}(a,b,c)=[e^a/(e^a+e^b+e^c),e^b/(e^a+e^b+e^c),e^c/(e^a+e^b+e^c)]$. It can be seen that the proposed ternary generation function is essentially a softmax function. The reason why we use the softmax function as the basic form is that the softmax function is most commonly used for modelling probability mass functions of discrete variables and offers great simplicity in differential calculus.\n\nThe motivation for the components $-z^2,-(z-\\\\hat z)^2,-(z-1)^2$ is that the ternary generation functions are expected to maintain the following properties:\n\n- $s$​ is more likely to be $-1$​ when $z$​ is close to $0$​.\n- $s$ is more likely to be $1$ when $z$ is close to $1$.\n- $s$ is more likely to be $0$ when $z$ is far away from both $0$ and $1$​.\n\nFinally, the parameters $\\\\underline \\\\lambda,\\\\lambda,\\\\overline \\\\lambda$ are used to control the strength of each component in the softmax function, similar to the precision parameter in a Gaussian distribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kSGeY0ACYT", "reviewer": "Reviewer_3ktH", "summary": "In this manuscript, the authors explore how to learn the unknown label distribution from given ternary labels (0, 1, and -1). The main contributions of this work include: (1) a new label distribution prediction method is designed for handling ternary labels; (2) the theoretical analysis on the error of approximating the GT label description degrees by ternary and binary labels has been done.", "questions": "(1) I think that the task of “Prediction label distribution from ternary labels” is a branch of weakly supervised learning. However, there is no discussion between them in this manuscript. \n\n(2) Actually, prediction unknown label distribution or label enhancement has been employed to deal with some other machine learning tasks, e.g., multi-label learning, partial label learning, feature selection and so on. But, the authors do not mention the possible extension of the proposed model to these or other scenarios. \n\n(3) The advantages and disadvantages between the presented algorithm and existing discriminative/generative label enhancement ones should be discussed.\n\n(4) The objective function of the proposed method is not shown. I think it is important for readers to clearly understand that how the method learn the unknown label distributions from ternary labels. By the way, I do not think the unknown label description degree is equal to the probability of an instance belonging to the positive label or negative label, though they are usually positively correlated with each other. So, I wonder that the proposed model how to enrich the supervised information in the label space and generate the label description degree.\n\n(5) From Table 1, I find that the results of different approaches on some metrics are closed. Therefore, the statistical significance analysis should be done, which can show whether the accuracy differences are significant or not.\n\n(6) The parameter sensitivity analysis of the three hyper-parameters in the proposed model is missed.", "limitations": "I find that LDL-LRR is used as the LDL model in the testing phase. This two-stage strategy is a limitation of its application in real-world scenarios. I encourage the authors to talk about the possible solution on designing a one-stage extension model based on the proposed method.", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "(1)\tThe theoretical analysis on the error of approximating the GT label description degrees by ternary and binary labels is detailed.\n\n(2)\tThe proposed MR-CateMO can deliver better results than that of baseline methods on three datasets.", "weaknesses": "(1)\tActually, some similar work has been done on “predicting label distribution from ternary labels” by researchers. For example, the problem definition of this manuscript is same to [17]. Although [17] has been discussed in the related work part, the authors did not point out that this work has made effort to address the same problem. This is quite strange! From this perspective, the novelty of this task is limited. Moreover, some claims of the authors maybe wrong. For example, “Since there is no LE method for ternary labels, we design…”.\n\n(2)\tExperiments are insufficient, and some existing label enhancement algorithms should be compared, e.g., [a], [b], [17]. Moreover, according to the suggestion of [17], there are many datasets can be used for this task, but the authors of this manuscript only evaluate their approach on three datasets.\n\n[a] Fusion Label Enhancement for Multi-Label Learning. IJCAI, 2022.\n[b] Label enhancement via manifold approximation and projection with graph convolutional network. Pattern Recognition, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Weakness (1)\n\n(1) Responses to the concern about \"Weakly Supervised Multi-Label Learning via Label Enhancement\".\n\nWe greatly appreciate the constructive feedback you have provided on our paper. __*It should be highly emphasised that the ternary labels in our paper and the WSML (i.e., weakly-supervised multi-labels) are inherently different despite sharing the same representation ($-1$, $0$, $1$).*__ Specifically, they differ in the following aspects.\n\n| Summary of Difference | Weakly-Supervised Multi-Labels, WSML | Ternary Labels |\n| -- | -- | -- |\n| They differ in the origin of the label with value of $0$, i.e., the missing label in WSML and the uncertain label in ternary labels. | The missing label in WSML stems from its \"absence\", i.e., the association between the label and the corresponding instance is undocumented or unannotated rather than undeterminable. | The uncertain label in ternary labels stems from its \"uncertainty\", i.e., it is difficult for experts to definitively determine whether the label is relevant to the corresponding instance. |\n| They differ in the range of the underlying label description degree. | The description degree of the missing label to the corresponding instance take values in the interval $[0,1]$, i.e., $[0,\\\\tau)\\\\cup [\\\\tau,\\\\kappa]\\\\cup (\\\\kappa,1]$, since the missing label may be a relevant label, an irrelevant label or an uncertain label. | The description degree of the uncertain label to the corresponding instance take values in the interval $[\\\\tau,\\\\kappa]$. |\n| They differ in the informativeness of the supervision data. | WSML is less informative than binary label (a.k.a. multi-label) due to the loss of information pertaining to some of the relevant and irrelevant labels. | Ternary label is more informative than binary label since it additionally encodes the cases where the label is neither definitely relevant nor definitely irrelenvant to the instance. |\n\n\nAdmittedly, we acknowledge that our paper may not illustrate this aspect thoroughly. We pledge to improve this section in accordance with your valuable suggestions.\n\n(2) Responses to why there is no LE (i.e., Label Enhancement) method for ternary labels.\n\nAs illustrated in the above table, WSML and ternary labels differ in the origin of the label with value of $0$​. Therefore, we believe that it is unjustified to apply the algorithms of WSML directly for ternary labels. Nonetheless, we acknowledge your suggestions and we will further improve the claims.\n\n## Responses to Weakness (2)\n\n(1) Responses to the suggestion that some existing LE methods should be compared, e.g., [a], [b], [17].\n\nOur proposed CateMO distribution, serving as a plug-and-play component that enables most standard binary LE algorithms to be applied to ternary labels, rather than an algorithm that outperforms existing binary LE algorithms. Therefore, our proposed CateMO is not comparable to the papers [a], [b], [17]. Nevertheless, we will discuss the relationships between the papers [a], [b], [17] and our paper appropriately since these papers are related to our work.\n\n(2) Responses to the concern about datasets.\n\nThe datasets used for the experiments in the paper [17] are not applicable to the research context of this paper because those datasets lack ternary labels. However, ternary labels cannot be obtained by setting the binary labels to uncertain labels directly since the uncertain labels must be the labels that are really difficult to determine whether they are relevant to the instance.\n\n## Responses to Question (1)\n\nThe fundamental differences between WSML and ternary labels have already been illustrated in \"Responses to Weakness (1)\". We will comprehensively disscuss their relationships in the revision.\n\n## Responses to Question (2)\n\nThank you for your suggestions. Our paper primarily concentrates on the quantitative and qualitative relationships between ternary labels and label description degrees. Unfortunately, due to page limitation, possible extensions of our proposal were not elaborated upon. Under the space restriction, we will endeavor to incorporate a discussion of these extensions in the revised version.\n\n## Responses to Question (3)\n\nIn our paper, we have not proposed an \"algorithm\". As illustrated in Responses to Weakness (2), our proposed CateMO is a probability distribution, a plugin that is able to work with most discriminative/generative LE algorithms. Therefore, it is unnecessary to discuss the advantages/disadvantages of our proposed CateMO compared to the existing discriminative/generative LE algorithms.\n\n## Responses to Question (4)\n\nThere is no objective function to show in our paper since we have not proposed a “learning algorithm” in our paper. The mathematical details of our proposed CateMO are shown in Eq. (12), and Eq. (13) shows how CateMO works in conjunction with existing LE algorithms. Besides, in terms of how to enrich the supervision information, various LE algorithms have their own logic to enrich the supervision or to generate the label description degrees, e.g., GLLE uses the smoothness assumption, and LELR uses the potential ranking relationships within binary labels. Our proposed CateMO merely models the probabilistic relationship between the label description degree and the ternary label, which is theoretically guaranteed to satisfy the proposed basic assumptions.\n\n## Responses to Question (5)\n\nThe results of statistical significance test have been shown in the Table 2 of Appendix A.4 of our paper.\n\n## Responses to Question (6)\n\nThank you for your suggestion. We additionally perform experiments to analyze the parameter sensitivity. The results are shown in Figure 1 in the PDF of global response, which will be added to the revised version as space permits. More detailed discussion about the parameters $\\\\underline \\\\lambda,\\\\lambda,\\\\overline \\\\lambda$ can be found in the \"Responses to Weakness (1) and Question (2)\" section for Reviewer uVMp.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8nQ13f4kbs", "reviewer": "Reviewer_7vWN", "summary": "The authors propose a novel multi-label annotation scheme, transitioning from the traditional binary annotation to a ternary annotation. They have validated this new annotation scheme through both theoretical analysis and experimental verification.", "questions": "1. The proposed approach of quantifying the advantages of ternary labels over binary labels through approximation error analysis is commendable. However, I am curious about the impact of the proportion of \"uncertain relevant labels\" in practical scenarios. In other words, if the proportion of \"uncertain relevant labels\" is excessively high within the overall labeling, does the proposed ternary annotation scheme still offer a statistically significant advantage over the traditional binary annotation scheme?\n\n2. While the authors validate the superiority of the ternary annotation scheme through experiments, the fairness of the existing experimental setup in terms of method comparison needs further clarification. As the paper suggests, there is no established LE method for ternary labels, and the experimental model proposed by the authors is new. It is recommended that the authors discuss the fairness of the comparative experiments in greater detail.", "limitations": "the authors adequately addressed the limitations", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The authors clearly articulate the problem they aim to solve, and the illustrative diagrams effectively demonstrate the problem's setup.\n\n2. The proposed scheme is validated from both theoretical and experimental perspectives.", "weaknesses": "The field of multi-label learning has developed various learning models to address different limitations of the label space, such as weakly supervised learning, partial multi-label learning, multi-label learning with class-conditional noise, and weakly semi-supervised multi-label learning. Research in these areas is well-established. The relationship between the \"uncertain relevant\" concept mentioned in the paper and existing work needs further clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Weakness\n\nWe really appreciate your valuable comments. Essentially, partial multi-labels, weakly-supervised multi-labels, semi-supervised multi-labels, and multi-labels with noise are essentially weaker versions of binary labels (i.e., multi-labels) because they either contain noise (i.e., incorrectly supervision data) or miss the correct supervision data. In contrast, ternary labels represent an enhanced version of binary labels, as ternary labels not only provide definitely relevant labels and definitely irrelevant labels but also leverage the option of uncertain label to identify the labels that are more prone to generating noise. Besides, the detailed illustration about the relationship between this paper and weakly-supervised multi-label learning is shown in Responses to Weakness (1) for Reviewer 3ktH. Under the space restriction, we will incorporate a discussion about the relationships between our paper and other similar paradigms in the revised version.\n\n## Responses to Questions\n\nThank you for your insightful questions! In terms of the cases where the number of uncertain labels is excessively large, we give the following proposition:\n\n***Proposition***: *Given an uncertain label, i.e., $p(s=0)=1$, we denote the expected approximation errors produced by annotating the label with a ternary value and a binary value as $\\\\psi\\_s$ and $\\\\psi\\_b$, respectively. Suppose that* $\\\\hat\\\\xi\\\\sim \\\\rm{Uni}(\\\\hat \\\\xi\\\\mid \\\\hat\\\\tau \\\\le \\\\hat \\\\xi \\\\le \\\\hat\\\\kappa)$, $\\\\rho\\\\sim \\\\rm{Uni}(\\\\rho\\\\mid 0\\\\le\\\\rho\\\\le 1)$, $[\\\\tau,\\\\kappa]\\\\sim \\\\rm{Uni}([\\\\tau,\\\\kappa]\\\\mid 0\\\\le \\\\tau\\\\le \\\\kappa\\\\le 1)$*, we have*\n\n$$\n\\\\mathbb{E}\\_{\\\\hat\\\\xi,\\\\rho,\\\\tau,\\\\kappa}[\\\\psi\\_s - \\\\psi\\_b] = \\\\frac29(\\\\hat\\\\tau^2+\\\\hat\\\\kappa^2+\\\\hat\\\\tau\\\\hat\\\\kappa) - \\\\frac13(\\\\hat\\\\tau+\\\\hat\\\\kappa)+\\\\frac{1}{12}.\n$$\n\n*Furthermore, if $[\\\\hat\\\\tau,\\\\hat\\\\kappa]\\\\sim \\\\rm{Uni}([\\\\hat\\\\tau,\\\\hat\\\\kappa]\\\\mid 0\\\\le \\\\hat\\\\tau\\\\le \\\\hat\\\\kappa\\\\le 1)$, we have*\n\n$$\n\\\\begin{aligned}\n&\\\\mathbb P\\_{[\\\\hat\\\\tau,\\\\hat\\\\kappa]\\\\sim \\\\rm{Uni}([\\\\hat\\\\tau,\\\\hat\\\\kappa]\\\\mid 0\\\\le \\\\hat\\\\tau\\\\le \\\\hat\\\\kappa\\\\le 1)}\\\\left[\\\\mathbb{E}\\_{\\\\hat\\\\xi,\\\\rho,\\\\tau,\\\\kappa}[\\\\psi\\_s - \\\\psi\\_b]\\\\le 0\\\\right] \\\\\\\\\n=&  \\\\int\\_{\\\\frac29(\\\\hat\\\\tau^2+\\\\hat\\\\kappa^2+\\\\hat\\\\tau\\\\hat\\\\kappa) - \\\\frac13(\\\\hat\\\\tau+\\\\hat\\\\kappa)+\\\\frac{1}{12}\\\\le 0, 0\\\\le \\\\hat\\\\tau\\\\le \\\\hat\\\\kappa\\\\le 1} \\\\rm{d} \\\\hat\\\\tau \\\\rm{d}\\\\hat\\\\kappa  \\\\left(\\\\int\\_{0\\\\le \\\\hat\\\\tau\\\\le \\\\hat\\\\kappa\\\\le 1}\\\\rm{d} \\\\hat\\\\tau \\\\rm{d}\\\\hat\\\\kappa\\\\right)^{-1}\\\\\\\\\n=&\\\\frac{6+6\\\\sqrt 3 + \\\\sqrt 3\\\\pi}{24}\\\\approx 0.91.\n\\\\end{aligned}\n$$\n\nThe above proposition reveals the advantage of ternary labels over binary labels when all labels are uncertain labels, i.e., $p(s=0)=1$. According to the above proposition, the expected approximation error of binary annotating exceeds that of ternary annotating when $\\\\frac29(\\\\hat\\\\tau^2+\\\\hat\\\\kappa^2+\\\\hat\\\\tau\\\\hat\\\\kappa) - \\\\frac13(\\\\hat\\\\tau+\\\\hat\\\\kappa)+\\\\frac{1}{12}\\\\le 0$. Furthermore, if we equiprobably set $\\\\hat\\\\tau$ and $\\\\hat\\\\kappa$ to any two values in the interval $0\\\\le \\\\hat\\\\tau\\\\le \\\\hat\\\\kappa\\\\le 1$, then the probability that the ternary annotating possesses lower expected approximation error is 91% approximately. Besides, in terms of the comparative experiments, we will clarify the fairness of each comparative experiments in greater detail under the space restriction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "81IiPgBBA2", "reviewer": "Reviewer_Lfun", "summary": "The paper proposes a more cost-effective approach to label distribution inference, i.e., predicting label distributions from ternary labels. Theoretically, the paper elucidates the superiority of the ternary label by analyzing the error of approximating the ground-truth label description degrees by ternary and binary labels, respectively. Methodologically, the paper proposes a categorical distribution with dimensional monotonicity and orderliness, which is theoretically proven to preserve the monotonicity and ordinality of the probabilities associated with ternary labels, to capture the process of generating ternary labels from label description degrees.", "questions": "I have the following two questions. First, how does the proposed CateMO distribution work with existing label enhancement algorithms to infer label distributions from ternary labels? Could you please answer this question with some detailed examples? Second, to offer some intuition for setting the lambda parameters in practical applications, could you please provide a semantic interpretation of the lambda parameters of CateMO distribution?", "limitations": "The paper has adequately discussed the limitations and the potential solutions of the proposed method. Besides, I don't believe that the paper has a potential negative social impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tOriginality: The paper exhibits a high degree of originality. Specifically, by incorporating the philosophies of three-way decision and three-world thinking, the paper provides a novel learning paradigm, i.e., predicting label distribution from ternary labels, to address the trade-off between the accuracy and cost in quantifying label distributions. To the best of my knowledge, this is the first work to consider the philosophies of three-way decision and three-world thinking in the realm of label distribution inference. I am convinced that the combination is logical, as the label distribution encapsulates the polysemy within labels, and the three-way philosophies are established to preserve the uncertainty in decision-making, which means that their objectives are harmoniously aligned.\n2.\tQuality: The paper is of high quality, as the proposed paradigm and CateMO distribution have been rigorously established through both theoretical foundations and experimental validation. Moreover, the paper also comprises in-depth analysis and discussions of both theoretical and experimental results.\n3.\tClarity: The paper is well-articulated, with a coherent and logical structure. Furthermore, the inclusion of clear and relevant visual supports, such as diagrams and figures, effectively enhances the overall understanding of the findings.\n4.\tSignificance: The paper is an impactful contribution to the LDL domain, as it effectively reduces the barriers to entry for implementing LDL algorithms, thereby broadening the spectrum of their applicability.", "weaknesses": "1.\tThere is a lack of introduction to the three-way philosophies. Since most readers may not be familiar with the three-way philosophies, it is necessary for this paper to provide a proper introduction to the three-way philosophies, rather than just presenting their names and references.\n2.\tFigure 3 and Figure 2 of the paper appear in a wrong order. Figure 3 should be placed after Figure 2.\n3.\tIn Figure 3, the triangle area should be $0\\le \\hat{\\tau}\\le \\hat{\\kappa} \\le 1$, instead of $\\hat{\\tau}\\le \\hat{\\kappa}$, which should be corrected to make the figure more rigorous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Weaknesses\n\nThank you for your suggestions. Under the space restriction, we will incorporate some background knowledge about three-way philosophies in the revised version. Besides, we will rearrange the Figure 2 and Figure 3, and correct the mathematical formula in Figure 3.\n\n## Responses to Questions\n\nThank you for your questions! Take GLLE as an exmple, the objective function of GLLE is\n\n$$\n\\\\mathcal L=\\\\sum\\_{n=1}^N \\\\Vert f(\\\\boldsymbol x\\_n; \\\\boldsymbol W) - \\\\boldsymbol y\\_n \\\\Vert\\_2^2 + \\\\Omega(\\\\boldsymbol W),\n$$\n\nwhere$f(\\\\boldsymbol x\\_n; \\\\boldsymbol W)$ is a predictive model with learnable parameters of $\\\\boldsymbol W$, $\\\\boldsymbol y\\_n$ is a logical label vector, $\\\\Omega(\\\\boldsymbol W)$ is the regularization term. If we apply CateMO for GLLE, the objective function will be\n\n$$\n\\\\mathcal L=\\\\sum\\_{n=1}^N -\\\\log \\\\mathrm{CateMO}(\\\\boldsymbol s\\_n|f(\\\\boldsymbol x\\_n; \\\\boldsymbol W)) + \\\\Omega(\\\\boldsymbol W),\n$$\n\nwhere $\\\\boldsymbol s\\_n$​ is a ternary label vector.\n\nBesides, in terms of how to set the lambda parameters, we provide the following two methods:\n\n- End-to-end Learning. The most straightforward way to determine the parameters $\\\\underline{\\\\lambda},\\\\lambda,\\\\overline{\\\\lambda}$ is to co-optimize these parameters with other parameters in the learner in an end-to-end manner. This approach is capable of obtaining parameters adapted to the characteristics of the data distribution, but corresponds to a complex constrained optimization problem that requires a tailored optimization algorithm since the constrains of $\\\\underline{\\\\lambda},\\\\lambda,\\\\overline{\\\\lambda}$ are interdependent.\n\n- Pre-annotations Fitting. The parameters can also be estimated from a small number of pre-annotated data pairs $\\\\{(s\\_n,z\\_n)\\\\}\\_{n=1}^L$, since $p(s|z)$ possesses a relatively stable morphology across different datasets. Figure 2 in the submitted PDF in global responses shows the distribution of some of our annotation results on the Painting, Music and JAFFE datasets. Specifically, we randomly choose an instance and a label from the JAFFE, Painting, and Music datasets, and asked the experts to simultaneously annotate the instance-label relationship by a ternary value and a description degree value. The above process is repeated six hundred times (two hundred times for each dataset). All annotation results are collected and recorded as $\\\\mathcal A=\\\\{(s\\_n,z\\_n)\\\\}\\_{n=1}^L$. Figure 2 in the submitted PDF in global responses shows the empirical distribution of $p(s|z)$ according to $\\\\mathcal A$, which indicates that there are no significant differences in the empirical distributions of $p(s|z)$ across datasets. Therefore, we can estimate the parameters of CateMO reliably with a small amount of pre-annotated data pairs. Formally, we obtain the parameters by maximum likelihood estimation, i.e.,\n\n$$\n\\\\begin{aligned}\n&\\\\underline{\\\\lambda}^\\\\star, {\\\\lambda}^\\\\star, \\\\overline{\\\\lambda}^\\\\star \\\\leftarrow \\\\arg\\\\max\\_{\\\\underline{\\\\lambda}, {\\\\lambda}, \\\\overline{\\\\lambda}}\\\\quad \\\\sum\\_{(s\\_{n}, z\\_{n}) \\\\in \\\\mathcal A} \\\\log p(s\\_{n} \\\\vert z\\_{n}) \\\\\\\\\n\\\\text{s.t.}\\\\quad & {\\\\lambda} < \\\\min \\\\{{\\\\underline{\\\\lambda}}{(1-\\\\hat{z})^{-1}}, \\\\overline{\\\\lambda} \\\\hat{z}^{-1}\\\\},\\\\\\\\\n&{\\\\lambda} \\\\neq {-\\\\underline{\\\\lambda}\\\\overline{\\\\lambda}}{(\\\\hat{z}\\\\overline{\\\\lambda} - \\\\hat{z} \\\\underline{\\\\lambda} - \\\\overline{\\\\lambda})^{-1}}, \\\\\\\\\n&{\\\\lambda} > \\\\max\\\\left\\\\{\\\\left( \\\\hat{z} + \\\\hat{z} \\\\exp(\\\\overline{\\\\lambda}) \\\\right)^{-1}\\\\overline{\\\\lambda}, ((1+\\\\exp(\\\\underline{\\\\lambda})) (1-\\\\hat{z}))^{-1} \\\\underline{\\\\lambda} \\\\right\\\\} , \\\\\\\\\n& \\\\hat{z} = \\\\left( 2{\\\\lambda} \\\\sqrt{\\\\overline{\\\\lambda}} + 2{\\\\lambda} \\\\sqrt{\\\\underline{\\\\lambda}} \\\\right)^{-1} \\\\left(2{\\\\lambda} \\\\sqrt{\\\\overline{\\\\lambda}} - \\\\underline{\\\\lambda} \\\\sqrt{\\\\overline{\\\\lambda}} + \\\\overline{\\\\lambda} \\\\sqrt{\\\\underline{\\\\lambda}}\\\\right).\n\\\\end{aligned}\n$$\n\n\nBesides, in order to provide users with a more intuitive understanding of CateMO's parameters, we show the shape of CateMO with varying $\\\\underline{\\\\lambda},\\\\lambda,\\\\overline{\\\\lambda}$ in Figure 3 in the submitted PDF in global responses, which visualizes how the parameters $\\\\underline{\\\\lambda},\\\\lambda,\\\\overline{\\\\lambda}$ detemine CateMO.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely value the time and thoughtfulness each reviewer has dedicated to enhancing the quality of our paper. We have carefully considered each comment and ensured that each point is addressed. Attached please find a PDF file containing the relevant figures mentioned in the responses for each reviewer. Should any reviewer have further questions or require additional clarification, we are readily available and would be more than happy to assist.

Meta-review: The paper proposes a solution that generates label distributions from ternary labels -1/0/1, where “0” indicates “uncertain relevant”, “1” indicates “definitely relevant” and “-1” indicates “definitely irrelevant”.  Three reviewers provided positive reviews. The proposed method is intuitive and straightforward, and the paper is well-articulated, with a coherent and logical structure. This paper proves that the ternary label is superior to the binary label in terms of expected approximation error in most cases.

However, one reviewer raised the concern about the novelty of this paper. During the discussion period, concerns about the novelty of the issue, the limited methodology and the incomplete experiments were raised:

1. The formulation of "predicting label distribution from ternary labels " seems like a label enhancement (LE) process on -1/0/1 labels, i.e., a modified version of label enhancement. The introduction also shows that the issue arises from LE and the proposed methods just directly utilized existing LE methods with a minor change in the conditional probability.

2. This paper provided a solution for generating label distribution from 1/0/1 labeled data but failed to apply the solution to any learning tasks. Many top conference papers on LE applied the LE technique, i.e., generating label distribution to solve many learning tasks（e.g., label distribution learning, multi-label learning, partial label learning) in the experiments to illustrate the improvement of the LE technique for machine learning. 

3. The theoretical results cannot prove that the ternary label outperforms the binary label in all cases. The theoretical analysis in this paper just calculated the expected approximation error between generated label distributions and the ground-truth label distributions. However, the theoretical results about the enhancement of the classifier's generalization ability by generating label distribution from ternary labels might be more solid and meaningful than the current theoretical analysis. 

4. In the methodology, the proposed method merely adapts existing label enhancement techniques, introducing only a minor alteration to the conditional probability of the input labels. This modification contributes minimally to the field of label enhancement or label distribution learning.

After discussion with the reviewers and the SAC, we are inclined to accept this paper, however, the authors should revise all the aforementioned limitations in the camera-ready paper. Here is the suggestion on how they might address the limitations:

1. In the introduction section, it is necessary to clearly articulate the similarities and differences between this paper and the concept of label enhancement.

2.  In the experimental section, the authors can adopt the proposed method for some learning tasks（e.g., label distribution learning, multi-label learning, partial label learning) to illustrate the improvement of their method for machine learning. 

3. The theoretical result shows that the ternary label cannot outperform the binary label in some cases. The authors should fix them. The theoretical results about the enhancement of the classifier's generalization ability by generating label distribution from ternary labels would be appreciated. 

4. In the methodology, the proposed method merely adapts existing label enhancement techniques, introducing only a minor alteration to the conditional probability of the input labels. More technique contributions should be provided.
Decision: Accept (poster)