Predicting Label Distribution from Ternary Labels

Authors: yunan  lu
xiuyi  jia
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 02:32:14

Abstract: Label distribution learning is a powerful learning paradigm to deal with label polysemy and has been widely applied in many practical tasks. A significant obstacle to the effective utilization of label distribution is the substantial expenses of accurate quantifying the label distributions. To tackle this challenge, label enhancement methods automatically infer label distributions from more easily accessible multi-label data based on binary annotations. However, the binary annotation of multi-label data requires experts to accurately assess whether each label can describe the instance, which may diminish the annotating efficiency and heighten the risk of erroneous annotation since the relationship between the label and the instance is unclear in many practical scenarios. Therefore, we propose to predict label distribution from ternary labels, allowing experts to annotate labels in a three-way annotation scheme. They can annotate the label as "$0$" indicating "uncertain relevant" if it is difficult to definitively determine whether the label can describe the instance, in addition to the binary annotation of "$1$" indicating "definitely relevant" and "$-1$" indicating "definitely irrelevant". Both the theoretical and methodological studies are conducted for the proposed learning paradigm. In the theoretical part, we conduct a quantitative comparison of approximation error between ternary and binary labels to elucidate the superiority of ternary labels over binary labels. In the methodological part, we propose a Categorical distribution with monotonicity and orderliness to model the mapping from label description degrees to ternary labels, which can serve as a loss function or as a probability distribution, allowing most existing label enhancement methods to be adapted to our task. Finally, we experimentally demonstrate the effectiveness of our proposal.

Main paper: 
 introduction: 
 LDL (Label Distribution Learning) [2] is an effective learning paradigm for addressing label polysemy (i.e., the cases where an instance can be described by multiple labels). Distinct from multi-label learning [22] , which assign a binary-valued vector to each instance, LDL assigns each instance a real-valued vector, akin to descrete probability distributions, to represent the description degree of each label to the instance. Label distributions provide fine-grained information about label polysemy, and thus have been applied to many practical tasks, such as sentiment analysis [8, 30, 35] , facial age estimation [1, 4, 24] and movie rating prediction [3] .A fundamental bottleneck hindering LDL is the difficulty in acquiring ground-truth label distributions, as the accurate quantification of these distributions can be prohibitively expensive. Therefore, LE (Label Enhancement) [29] is proposed to automatically infer label distributions from the more easily accessible multi-label data. Multi-label data is based on binary annotations (i.e., utilizing binary values ±1 to annotate each label) which demand that experts accurately identify whether each label Figure 1 : An annotation from JAFFE dataset [18] . can describe the instance. However, accurate identification is challenging in real-world tasks due to the prevalence of ambiguous instances and labels which provoke uncertain binary associations between the instances and the labels. For example, the facial image in Figure 1 can be definitely (negatively or positively) associated to "happy", "sad", and "surprise", whereas the association with "fear" is uncertain, since it is not clear whether the emotion comes from being stimulated by something frightening. If experts are coerced into providing a definitive annotation in uncertain cases, not only does it diminish the efficiency of annotating, but it also heightens the risk of erroneous annotation.Therefore, we propose to predict label distribution from ternary labels. Inspired by the philosophies of three-way decision or three-world thinking [31] , where decision-makers are provided with the flexibility to delay judgment when the available information is inadequate to support a determined decision, ternary labels take values from {-1, 0, 1}, where "±1" indicates whether the label can describe the instance, and "0" denotes that the relationship between the label and the instance is uncertain. For the proposed learning paradigm, we conduct theoretical and methodological studies. In the theoretical part, we first quantify the errors of approximating the ground-truth label description vector2 from ternary labels and binary labels, respectively. Further, we conduct a quantitative comparison of approximation error between ternary and binary labels to elucidate the superiority of ternary labels. In the methodological part, we propose CateMO distribution (Categorical distribution with Monotonicity and Orderliness) to model the mapping from label description degrees to ternary labels, which can serve as the probability distribution for generating ternary labels or as the loss function for measuring the inconsistency between the ternary labels and label distributions, allowing most existing LE methods to be adapted to our task. Specifically, we first analyze the rules governing the generation of ternary labels from label description degrees, and formalize them as the assumptions about the probabilistic monotonicity and orderliness of ternary labels. Further, we derive the probability mass function of CateMO to ensure probabilistic monotonicity and orderliness. Finally, we create two comparison algorithms and evaluate the prediction performance on three real-world datasets. Experimental results unequivocally affirm the superiority of ternary labels. Our contributions can be summarized as follows:• We propose to predict label distribution from ternary labels, which not only enhances the annotation accuracy but also significantly reduces the annotating cost when contrasted with the traditional binary annotating methods.• We rigorously analyze the error of approximating the ground-truth label description degrees by ternary and binary labels, respectively, which provides a quantitative elucidation of the superior performance of the ternary label.• We propose the CateMO distribution specifically designed to capture the mapping from label description degrees to ternary labels, which is theoretically constructed to maintain the monotonicity and ordinality of the probabilities associated with ternary labels. 
related work: 
 To address the challenge of obtaining accurate ground-truth label distributions, LE (Label Enhancement) [29] is proposed as a method to automatically infer label distributions from more accessible multi-label data, including binary labels [29] and multi-label rankings [13, 15] . Existing works on LE can be broadly categorized into discriminative LE methods and generative LE methods. Discriminative LE directly treats the label distribution as conditional probabilities of the labels given feature observations, and subsequently mine additional information to estimate the conditional probabilities.Generative LE, on the other hand, focus on describing the generation process of observations in a principled manner and uncovering the underlying patterns of the observed data.Current discriminative LE methods generally strive to optimize both Dist(z, y) and Ω(z), where Dist(z, y) measures the inconsistency between the label description degrees z and the more accessible labels y (including binary labels and multi-label rankings), and Ω(z) is the regularization terms based on various sources of information. Typically, Dist(z, y) is modeled using MSE (Mean Squared Error), and the information for Ω(z) includes instance relationships and label correlations. For instance, several algorithms [5, 10, 20, 21, 23, 33, 34] assume that the instance manifolds based on features are similar to the instance manifolds based on label distributions. Additionally, several algorithms [11, 26, 29, 32] operate under the assumption that instances with similar feature vectors exhibit similarity in their label distributions. To incorporate label correlations, some algorithms [17, 25] regularize the label distributions using the Graph Laplacian operators of label correlation graphs. LEPNLR [6] , on the other hand, enhances the labels by preserving the ranking relation of labels. Existing generative LE methods generally decompose the joint distribution of complete data as p(y|z), p( [27, 28] and GLEMR [12] model p(y|z) as a Gaussian distribution and a ranking-preserved distribution, respectively. GLERB [16] and LEIC [14] model p(• • • |z) by incorporating instance relationships and label correlations into generation processes.Furthermore, similar to ternary labels, the binary labels with missing values [17] are formally represented by {0, ±1}. However, it should be emphasised that they inherently differ in the following aspects. First, they differ in the origin of the labels with value of 0, i.e., the missing labels in the literature [17] and the uncertain label in ternary labels. The missing labels stem from the "absence", i.e., the association between the label and the corresponding instance is undocumented or unannotated rather than undeterminable. By contrast, the uncertain labels stem from the "uncertainty", i.e., it is difficult for experts to definitively determine whether the label is relevant to the corresponding instance. Second, they differ in the range of the underlying label description degree. The description degree of the missing labels to the corresponding instance can take any value in the interval [0, 1], since the missing labels may be relevant labels, irrelevant labels, or uncertain labels. By contrast, the description degree of the uncertain labels to the corresponding instance take values in a small sub-interval of [0, 1], which is much tigher than the range of the description degrees of missing labels.3 Quantitative analysis 
preliminary: 
 Given an instance with feature vector of x, the label description vector of the instance is denoted by z whose m-th element z m ∈ [0, 1] indicates the description degree of the m-th label to the instance x. If an instance is annotated with binary labels, the label description vector will be degenerately expressed as a vector of binary values b ∈ {-1, 1} M whose m-th element b m denotes whether the m-th label can describe the instance or not. If an instance is annotated with ternary labels, the label description vector will be degenerately expressed as a vector of ternary values s ∈ {-1, 0, 1} M where s m = 1 and s m = -1 denotes that the m-th label can describe and cannot describe the instance, respectively, and s m = 0 denotes that the association between the m-th label and the instance x is uncertain.To quantify the advantages of the ternary label over the binary label, we leverage EAE (Expected Approximation Error) [13, 15] which measures the error of approximating the true label description degree by a more accessible label such as binary label, multi-label ranking, or ternary label. The definition of EAE is formalized as follows:Definition 1 Given a label with the true label description degree z ∈ I, if the label is annotated by a reduced label with the approximate label description degree ẑ ∈ Î, then the expected approximation error of the reduced label, i.e., ψ( Î, I), is quantified as:EQUATIONAccording to Definition 1, it can be seen that EAE depends on I and Î, so below we propose the following assumptions to determine I, Î and their relationships.Assumption 1 As stated in the introduction, a label in practical cases can be positive (i.e., the label can describe the instance), negative (i.e., the label cannot describe the instance), and uncertain (i.e., the association between the label and the instance is uncertain). If a label is positive, then the true label description degree z ∈ (κ, 1]. If the label is negative, then the true label description degree z ∈ [0, τ ), where τ ∈ [0, κ]. If the label is uncertain, then the true label description degree z ∈ [τ, κ].That is, z ∈ I s ,EQUATIONIf a label is annotated by a ternary label ŝ, then we have the ŝ-based label description degree ẑ ∈ I ŝ,EQUATIONwhere 0 ≤ τ ≤ κ ≤ 1 are the predefined parameters as approximations to τ and κ, respectively, since τ and κ are unavailable from the annotation results.If a label is annotated by a binary label b, then we have the b-based label description degree ẑ ∈ I b,EQUATIONwhere ξ ∈ [τ , κ] must hold, otherwise there exists some label description degree corresponding to " b = 1 ∧ ŝ = -1" or " b = -1 ∧ ŝ = 1", which is semantically contradictory.Assumption 2 Since a label can be positive, negative or uncertain in practical cases, we discuss the relationships between the true label and the annotation results in each of the three cases below.If a positive or negative label (i.e., s = ±1) is annotated by a ternary value ŝ or a binary value b, then we have ŝ = b = s.If an uncertain label (i.e., s = 0) is annotated by a ternary value ŝ or a binary value b, then we have ŝ = s, p( b = -1|s = 0) = ρ, and p( b = 1|s = 0) = 1 -ρ, where 0 ≤ ρ ≤ 1.Next, we give the analytical form of EAE of the binary label and the ternary label.Theorem 1 Given a label equiprobably being positive, negative, and uncertain, if the label is annotated by a ternary value, then we have the expected approximation error:EQUATION)If the label is annotated by a binary value, then we have the expected approximation error:EQUATIONwhere ρ = p( b = -1|s = 0). Suppose that ξ ∼ Uni( ξ | τ ≤ ξ ≤ κ), ρ ∼ Uni(ρ | 0 ≤ ρ ≤ 1), [τ, κ] ∼ Uni([τ, κ] | 0 ≤ τ ≤ κ ≤ 1), we have E ŝ,s,τ,κ [ψ(I ŝ, I s )] = 36 -1 (8κ 2 + 4τ κ -12κ + 8τ 2 -8τ + 7), E b,s, ξ,ρ,τ,κ [ψ(I b, I s )] = 16 -1 (2κ 2 + 2τ κ -3κ + 2τ 2 -3τ + 3). (7) E ŝ,s,τ,κ [ψ(I ŝ, I s )] ≤ E b,s, ξ,ρ,τ,κ [ψ(I b, I s )] if and only if κ -3 4 2 + τ -1 4 2 ≤ 38 and τ ≤ κ. E ŝ,s, ξ,ρ,τ,κ [ψ(I ŝ, I s )] over [τ , κ], where E b,s, ξ,ρ,τ,κ [ψ(I b, I s )] and E ŝ,s, ξ,ρ,τ,κ [ψ(I ŝ, I s )] (which are defined in Equation ( 7)) measures the average EAE of binary labels and ternary labels, respectively; PE ŝ,s [ψ(I ŝ,Is)]≥E b,s [ψ(I b ,Is)] is defined in Equation ( 8), which measures the approximate proportion of cases where the ternary label is inferior to the binary label for different [τ, κ, ξ, ρ].The detailed proof of Theorem 1 can be found in Appendix. Figure 2 visualizes the results of Theorem 1. Specifically, Figure 2 (a) shows the relationship between the predefined parameters [τ , κ] and PE ŝ,s [ψ(I ŝ ,Is)]≥E b,s [ψ(I b ,Is)] which is defined as:EQUATIONwhere I(•) is an indicator function that outputs 1 if the internal condition is true, and 0 otherwise; r(u, v) outputs an increasing sequence [0, v, 2v, . . . , u]. Equation ( 8) measures the approximate proportion of cases where the ternary label is inferior to the binary label for all possible [τ, κ, ξ, ρ] in G. Obviously, the ternary label outperforms the binary label in most cases. Specifically, the binary label shows superiority only in the extreme cases, i.e., (τ , κ) ∈ {(0, 0), (0, 0.1), (0.9, 1), (1, 1)}, but τ and κ basically never take these values in practical applications since both the binary and ternary labels exhibit very high approximation error in these cases. In addition, Theorem 1 also derives thatEQUATION2 ≤ 3 8 and τ ≤ κ, which is visualized as the overlapping area of the red and blue regions in Figure 3 . It can be seen that the overlapping area is essentially consistent to the blue area in Figure 2(a) . Therefore, the ternary label is superior to the binary label w.r.t. approximating the ground-truth label description degrees. 
catemo: categorical distribution with monotonicity and orderliness: 
 In order to learn label distributions from ternary labels, existing LE methods can be borrowed, the fundamental frameworks of which have been visualized in Figure 4 and illustrated in Section 2. It can be observed that both discriminative and generative LE methods necessitate modeling the relationship between the label description degrees z and the more accessible labels y. Therefore, we propose CateMO (Categorical distribution with Monotonicity and Orderliness) to model the conditional probability of ternary label given the label description degree. CateMO can serve as p(y|z) in generative LE methods, and the negative likelihood function of CateMO can be used as Dist(z, y) in discriminative LE methods, so that most existing LE methods can be employed to address our task by replacing Dist(z, s) and p(s|z) with CateMO. In the following subsections, we first provide an intuitive discussion of the rules governing the generation of ternary labels and then formalize these rules as the assumptions about the probability monotonicity and orderliness of ternary labels. Furthermore, we derive the parametric mass function for CateMO, which is theoretically guaranteed to maintain the probability monotonicity and orderliness of ternary labels.On the one hand, we explore how the label description degree affects the probabilities of the label being positive, negative and uncertain. Obviously, a label is more likely to be positive if the description degree of the label is larger; a label is more likely to be negative if the description degree of the label is smaller. Besides, the uncertain label is used to encode situations where the expert is unsure whether the label can describe the instance, which arises from the fact that the probability of the label being positive is close to the probability of the label being negative. Therefore, we believe that a label is more likely to be uncertain if the probabilities of the label being positive and negative are closer. The above intuitions can be formalized as follows:Assumption 3 (Probability monotonicity of ternary labels) Given any two real values v 1 and v 2 between 0 and 1, ifv 1 < v 2 , then p(s = 1|z = v 1 ) < p(s = 1|z = v 2 ) and p(s = -1|z = v 1 ) > p(s = -1|z = v 2 ). If 0 < p(s = 1|z = v 1 ) -p(s = -1|z = v 1 ) < p(s = 1|z = v 2 ) -p(s = -1|z = v 2 ) or 0 < p(s = -1|z = v 1 ) -p(s = 1|z = v 1 ) < p(s = -1|z = v 2 ) -p(s = 1|z = v 2 ), then p(s = 0|z = v 1 ) > p(s = 0|z = v 2 ).On the other hand, we explore how the label description degree affects the orderliness among the probabilities of the label being positive, negative and uncertain. Obviously, a label is most likely to be negative and least likely to be positive if the label description degree is sufficiently small; a label is most likely to be positive and least likely to be negative if the label description degree is sufficiently large; a label is most likely to be uncertain if the label description degree is moderate. We formalize the intuition as follows:Assumption 4 (Probability orderliness of ternary labels) There exists two real values 0≤ v 1 < v 2 ≤ 1, p(s = 1|z) < p(s = 0|z) < p(s = -1|z) holds for any z < v 1 , p(s = -1|z) < p(s = 0|z) < p(s = 1|z) holds for any z > v 2 , max{p(s = 1|z), p(s = -1|z)} < p(s = 0|z) holds for any v 1 < z < v 2 .Since a ternary label can take three possible values, i.e., s ∈ {0, ±1}, we model ternary label by a categorical distribution, which can be formalized as follows: where φ(z) = p(s = -1|z), φ(z) = p(s = 0|z), and φ(z) = p(s = 1|z) represent the generation principles from the label description degree to the negative, uncertain, and positive labels, respectively. We name φ(z), φ(z), and φ(z) as ternary generation functions. According to Assumption 3, we preliminarily assume the parametric form of ternary generation functions as follows:EQUATIONEQUATION)where Z = e -λz 2 + e -λ(z-ẑ) 2 + e -λ(z-1) 2 , λ > 0, λ > 0, λ > 0 and 0 < ẑ < 1 are parameters. Intuitively, the parameters (λ, λ, λ) largely governs the precision of CateMO distribution (despite the fact that they are not exactly equal), which is similar to the reciprocal of the temperature coefficient in the softmax layer of a deep neural network. Hence, we refer to the parameters λ, λ, and λ as the precision of negative labels, positive labels, and uncertain labels, respectively. Figure 5 visualizes the shape of p(s|z) on different parameters, which can be found that some parameter configurations violate the above three assumptions. For example, Figure 5 (a) violates Assumption 3, Figure 5 (c) violates Assumption 3 and Assumption 4 at the same time. Therefore, we propose Theorem 2 to ensure that the ternary generation functions defined by Equation (10) satisfy the proposed assumptions about probability monotonicity and orderliness of ternary labels.Theorem 2 Given λ > 0, λ > 0, and λ > 0, the ternary genertaion functions satisfy Assumption 3 and Assumption 4 if the following conditions hold:λ ̸ = -λλ(ẑλ -ẑλ -λ) -1 , ẑ = (2λ λ + 2λ λ) -1 (2λ λ -λ λ + λ λ), max{(ẑ + ẑe λ ) -1 λ, ((1 + e λ )(1 -ẑ)) -1 λ} < λ < min{λ(1 -ẑ) -1 , λẑ -1 }. (11)The details of the proof can be found in Appendix. Therefore, the probability mass function of CateMO distribution can be formalized as follows:EQUATIONFinally, to apply CateMO, we simply replace Dist(y, z) in Figure 4 (a) with the negative log-likelihood of CateMO and replace p(y|z) in Figure 4 (b) with CateMO, which can be formalized as follows:EQUATION5 Experiments 
datasets and evaluation measure: 
 Although there are many LDL datasets, they all lack ground-truth ternary label data. Therefore, we select three real-world LDL datasets (i.e., JAFFE [18] , Painting [19] , and Music [9] ), and manually re-annotate them with both binary labels and the ternary labels. The details of these datasets can be found in Appendix. We use five common LDL metrics to evaluate the algorithm performance, which are Cheb (Chebyshev distance), KL (Kullback-Leibler divergence), Cosine (cosine coefficient) [2] , and Rho (Spearman's rho coefficient) [7] . The lower values of Cheb and KL indicate the better performance. The higher values of Cosine, Intersec, and Rho indicate the better performance. We use "↑" and "↓" to denote that the better performance is represented by the higher and lower values of a metric, respectively.Comparison methods Since there is no LE method specifically for ternary labels, we design two approaches to construct the comparison algorithms. On the one hand, we design a data transformation method (which is abbreviated as DT method) to transform the dataset with ternary labels into an extended dataset with binary labels so that any existing LE algorithm can be applied to ternary labels. Specifically, for any instance with uncertain labels, we transform the instance into two instances which set all uncertain labels to positive and negative labels, respectively. For instance, the example[x, [1, 0, 0]] will be transformed into [x, [1, 1, 1]] and [x, [1, -1, -1]],respectively. On the other hand, we replace the loss term Dist(z, b) with MSE ∥(s + 1)/2 -z∥ 2 2 , so that most existing LE methods can be used to enhance ternary labels. We abbreviate this method as MSE method. Besides, we select three recently proposed binary label enhancement algorithms: GL [29] , LR [6] , and MR [12] . The hyperparameter settings follow their respective literature. We combine GL, LR and MR algorithms with DT and MSE methods in pairs to construct six comparison algorithms: GL-DT, GL-MSE, LR-DT, LR-MSE, MR-DT, and MR-MSE. In terms of our proposal, we replace the conditional distribution p(b|z) in MR with CateMO and replace Dist(z, b) in GL and LR with the negative log-likelihood function of CateMO, which constructs three algorithms: GL-CateMO, LR-CateMO, and MR-CateMO. We set the parameters [λ, λ, λ] in CateMO as [49, 48, 12] .We aim to test the performance of label distribution prediction based on different LE algorithms. Specifically, we use different LE methods to recover training label distributions and use these recovered label distributions to train an LDL model, whose performance on test instances will be reported. LDL-LRR [7] is used as the LDL model in this paper, whose hyperparameters λ and β are selected from {10 -6 , 10 -5 , . . . , 10 -1 } and {10 -3 , 10 -2 , . . . , 10 2 } as suggested [7] . We randomly partition the whole dataset (70% for training and 30% for testing), and repeat the above process ten times and report the average and standard deviation of the results.Table 1 shows the prediction performance of the comparison algorithms on three datasets. Each result is formatted as "mean±std". In the first column of Table 1 , "MR", "LR" and "GL" denote the existing binary LE algorithms, the suffix "-LL" denotes that these algorithms run on binary labels directly, and the suffixes "-DT", "-MSE" and "-CateMO" denote that these algorithms run on ternary labels by the DT method, MSE method and our proposed CateMO distribution, respectively. "Ground-Truth" denotes that LDL-LRR is trained directly on the ground-truth label distributions. In each area separated by dashed lines, bold and italics denote the 1st and 2nd, respectively. The results of statistical significance test are shown in Appendix. It can be seen that the performance of our proposed CateMO is better than other three comparison algorithms in all cases, and is close to "Ground-Truth". To visualize the advantages of ternary labels over binary labels, we show the relationship between binary labels, ternary labels, and label distributions in terms of annotation time and prediction performance in Figure 6 . The horizontal axis denotes the average time (in seconds) spent by an expert in annotating a label for an instance. The vertical axis denotes the average prediction performance calculated from Table 1 . It can be seen that ternary labels is superior to binary labels in terms of both prediction performance and annotating cost. Figure 7 shows how the precision parameters affect the recovery performance of GL-CateMO on JAFFE, Painting, and Music datasets. The recovery performance is computed by the following two steps. First, we run GL-CateMO algorithm on the dataset with ternary labels to recover the label distributions for instances. Second, we calculate the KL divergence between the recovered label distributions and the ground-truth by the LDL metrics. For a certain precision parameter, the other two precision parameters are set to the values that give rise to the best recovery performance. 
limitations and conclusion: 
 Limitations In this paper, the parameters (i.e., λ, λ, and λ) of CateMO are pre-fixed. In fact, a more adaptive approach is to collaboratively learn these parameters and other model parameters. However, since the parameters of CateMO satisfy the conditions shown in Theorem 2, in which the parameters are interdependent, it may lead to difficulties in directly exploiting gradient descent optimization methods. Therefore, in future works, we will explore how to appropriately reduce the parameter space of CateMO so that the parameters λ, λ, and λ can be decoupled from each others. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have cited all original papers that produced the datasets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: The expert's annotating task is as follows. An expert should annotate 1 if the label can describe the instance, -1 if the label cannot describe the instance, and 0 if it is uncertain whether the label can describe the instance. This process is described in Section 1.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 
 
 Appendix: In this paper, we propose to predict label distribution from ternary labels, which reduces both the annotation inaccuracy and cost when contrasted with the traditional binary annotating methods. In the theoretical part, we analyze the approximation error for both ternary and binary labels, which provides a quantitative elucidation of the superior performance of the ternary label.In the methodological part, we propose the CateMO distribution to model the mapping from label description degrees to ternary labels, which is theoretically constructed to maintain the monotonicity and ordinality of the probabilities associated with ternary labels. In the experimental part, extensive experiments demonstrate the effectiveness of our proposal.A.1 Proof of Theorem 1Given a label equiprobably being positive, negative, and uncertain, if the label is annotated by a ternary value, then we have the expected approximation error:EQUATION)If the label is annotated by a binary value, then we have the expected approximation error: EQUATION) where ρ = p( b = -1|s = 0). Suppose that ξ ∼ Uni( ξ | τ ≤ ξ ≤ κ), ρ ∼ Uni(ρ | 0 ≤ ρ ≤ 1), [τ, κ] ∼ Uni([τ, κ] | 0 ≤ τ ≤ κ ≤ 1), we have E ŝ,s,τ,κ [ψ(I ŝ, I s )] = 1 0 κ 0 dτ dκ -1 1 0 κ 0 E ŝ,s [ψ(I ŝ, I s )]dτ dκ = 36 -1 (8κ 2 + 4τ κ -12κ + 8τ 2 -8τ + 7) E b,EQUATION)To ensure that Equation ( 10) satisfies the probability monotonicity of negative and positive labels, we just need to ensure that ∂φ(z)/∂z < 0 and ∂φ(z)/∂z > 0. The partial derivative of φ(z) w.r.t. z can be formalized as:EQUATIONwhere v 1 , v 2 , v 3 > 0. Since λz -λ -λz < 0, (λz -λ -λz)e v2 < 0. Therefore, ∂φ(z)/∂z < 0 if λz -ẑλ -λz < 0 holds for any z ∈ (0, 1). This inequality can be equivalently transformed into:EQUATION)Similarly, the partial derivative of φ(z) w.r.t. z can be formalized as:EQUATIONwhere v 4 , v 5 , v 6 > 0. Since λ -λz + λz > 0, (λ -λz + λz)e v5 > 0. Therefore, ∂φ(z)/∂z > 0 if zλ -ẑλ -λz + λ > 0, i.e., λ(z -ẑ) > λ(z -1) holds for any z ∈ (0, 1). On the one hand, if ẑ < z < 1, then λ > λ(z -1)(z -ẑ) -1 . On the other hand, if z < ẑ, then λ < λ ẑ < λ(z-1) z-ẑ . To ensure that Equation (10) satisfies the probability orderliness, we just need to ensure that equations φ(z) = φ(z) (i.e., λ(z -1) 2 = λ(z -ẑ) 2 ) and φ(z) = φ(z) (i.e., λz 2 = λ(z -ẑ) 2 ) have only one solution on z ∈ (0, 1). We denote g 1 (z) = λ(z -1) 2 -λ(z -ẑ) 2 . Since g 1 (1) = -λ(ẑ -1) 2 < 0 and g 1 (0) = λ -ẑ2 λ > λ -ẑ2 λẑ -1 = λ(1 -ẑ) > 0, the quadratic equation g 1 (z) = 0 has only one solution in z ∈ (0, 1). Similarly, we denoteg 2 (z) = λz 2 -λ(z -ẑ) 2 Since g 2 (1) = 2ẑλ -ẑ2 λ -λ + λ = λ -(1 -ẑ) 2 λ > λ -(1 -ẑ) 2 λ(1 -ẑ) -1 =ẑλ > 0 and g 2 (0) = -ẑ 2 λ < 0, the quadratic equation g 2 (z) = 0 has only one solution in z ∈ (0, 1).To ensure that Equation (10) satisfies the probability monotonicity of uncertain label, we just need to ensure that ∂φ(z)/∂z > 0 holds for any 0 < z < z 0 and ∂φ(z)/∂z < 0 holds for any z 0 < z < 1, where φ(z 0 ) = φ(z 0 ), i.e., z 0 = √ λ(√ λ + √ λ) -1That is, make sure that z 0 is the only point of maximum value of φ(z). The partial derivative of φ(z) w.r.t. z is:EQUATIONwhere v 1 > 0. We aim to make sure that f (z) = ∂φ(z)/∂z = 0 has only one solution in z ∈ (0, 1)(-ẑλ + λz -λz)(ẑλ -λz + λz -λ) -1 f (z) = exp(2λz + λz 2 -λz 2 -λ) g(z).We first need to examine whether the denominator of f (z) is zero. Suppose that it is zero, wehave z = (λ -ẑλ)(λ -λ) -1 . Considering that λ -ẑλ > λ -ẑλẑ -1 = 0, if λ > λ, then z = (λ-ẑλ)(λ-λ) -1 > 1, which contradicts z ∈ (0, 1). If λ < λ, then z = (λ-ẑλ)(λ-λ) -1 < 0,which also contradicts z ∈ (0, 1). Besides, it is obvious that the denominator of f (z) is non-zero when λ = λ. Therefore, the denominator of f (z) is non-zero. Since whether the partial derivative of f (z) w.r.t. z (i.e., ∂f (z)/∂z = (• • • ) -2 (ẑλλ -ẑλλ -λλ + λλ)) is positive or negative has nothing to do with z, we ensure the monotonicity of f (z) by the following inequality:EQUATIONBesides, since ∂g(z)/∂z = (2λ(1 -z) + 2λz) exp(•) > 0, g(z) is increasing. Therefore, to make sure that φ(z) has only one maximal point, we just need to ensure that f (0) > g(0) and f (1) < g(1), i.e., λ > max ẑ + ẑ exp(λ)EQUATIONFinally, let the partial derivative of φ(z) at z = z 0 to be zero, we haveEQUATIONConsideringz 0 = √ λ √ λ+ √ λ , we have ẑ = (2λ √ λ + 2λ √ λ) -1 (2λ √ λ -λ √ λ + λ √ λ). □ A.3 DatasetsThe "JAFFE" dataset [18] comprises 213 facial emotion images posed by ten Japanese female models. Concerning the feature data, we employ the feature extraction method suggested in [2] to compress each image into a 243-dimensional feature vector. As for the label data, each image is annotated with scores from {1, 2, 3, 4, 5} by 60 individuals, indicating the relevance of the corresponding emotion to the facial image. The average score for each emotion is utilized to represent the intensity of the emotion. Subsequently, we apply min-max normalization to scale the average scores to the interval (0, 1) and normalize the average scores of all emotions into the form of probability distributions to obtain the label distribution data. Besides, three experts annotate the instances in JAFFE with ternary labels and binary labels, and the final ternary labels and binary labels are determined by majority voting.The "Painting" dataset [19] is created for exploring emotions within abstract artworks. Each image is represented by a 142-dimensional feature vector comprising three components: histogram features of the RGB attributes of the image, histogram features of the HSV attributes of the image, and GLCM (Grey-Level Co-occurrence Matrix) features of the image. Regarding the label data, approximately 230 individuals annotate emotional scores across eight categories: amusement, anger, awe, content, disgust, excitement, fear, sadness. Each image received about 14 annotations. Subsequently, we apply min-max normalization to scale the average emotional scores to the interval (0, 1) and normalize the average scores of all emotions into the form of probability distributions to obtain label distribution data. Besides, three experts annotate the instances in Painting with ternary and binary labels, and the final ternary and binary labels are determined by majority voting.The "Music" dataset is an extension of a music dataset [9] and encompasses 360 popular songs from major music charts across different countries. We employ MFCC (Mel Frequency Cepstral Coefficients) to extracte a 5992-dimensional feature vector from each song and utilize PCA (Principal Component Analysis) to compress this vector into a 128-dimensional feature vector.Regarding the label data, participants from the UK, South Korea and Portugal rate their perceived moods (i.e., calm, tense, cheerful, sad, danceable, love, dreamy, electronic, and energy) for a given song on a 4-level scale. Subsequently, we apply min-max normalization to scale the scores to the interval (0, 1), and normalize the average scores of all moods into the form of probability distributions to obtain label distribution data. Besides, three experts annotate the instances in Music with ternary and binary labels, and the final ternary and binary labels are determined by majority voting. Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Justification: There is no potential risk incurred by study participants.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.In general, the elements of a label distribution are assumed to be non-negative and sum to 1. For simplicity, we disregard the restriction that the sum is 1 in this paper, and refer to the unnormalized label distribution as label description vector. The element of a label description vector is called label description degree. 


Decision: The paper proposes a solution that generates label distributions from ternary labels -1/0/1, where “0” indicates “uncertain relevant”, “1” indicates “definitely relevant” and “-1” indicates “definitely irrelevant”.  Three reviewers provided positive reviews. The proposed method is intuitive and straightforward, and the paper is well-articulated, with a coherent and logical structure. This paper proves that the ternary label is superior to the binary label in terms of expected approximation error in most cases.

However, one reviewer raised the concern about the novelty of this paper. During the discussion period, concerns about the novelty of the issue, the limited methodology and the incomplete experiments were raised:

1. The formulation of "predicting label distribution from ternary labels " seems like a label enhancement (LE) process on -1/0/1 labels, i.e., a modified version of label enhancement. The introduction also shows that the issue arises from LE and the proposed methods just directly utilized existing LE methods with a minor change in the conditional probability.

2. This paper provided a solution for generating label distribution from 1/0/1 labeled data but failed to apply the solution to any learning tasks. Many top conference papers on LE applied the LE technique, i.e., generating label distribution to solve many learning tasks（e.g., label distribution learning, multi-label learning, partial label learning) in the experiments to illustrate the improvement of the LE technique for machine learning. 

3. The theoretical results cannot prove that the ternary label outperforms the binary label in all cases. The theoretical analysis in this paper just calculated the expected approximation error between generated label distributions and the ground-truth label distributions. However, the theoretical results about the enhancement of the classifier's generalization ability by generating label distribution from ternary labels might be more solid and meaningful than the current theoretical analysis. 

4. In the methodology, the proposed method merely adapts existing label enhancement techniques, introducing only a minor alteration to the conditional probability of the input labels. This modification contributes minimally to the field of label enhancement or label distribution learning.

After discussion with the reviewers and the SAC, we are inclined to accept this paper, however, the authors should revise all the aforementioned limitations in the camera-ready paper. Here is the suggestion on how they might address the limitations:

1. In the introduction section, it is necessary to clearly articulate the similarities and differences between this paper and the concept of label enhancement.

2.  In the experimental section, the authors can adopt the proposed method for some learning tasks（e.g., label distribution learning, multi-label learning, partial label learning) to illustrate the improvement of their method for machine learning. 

3. The theoretical result shows that the ternary label cannot outperform the binary label in some cases. The authors should fix them. The theoretical results about the enhancement of the classifier's generalization ability by generating label distribution from ternary labels would be appreciated. 

4. In the methodology, the proposed method merely adapts existing label enhancement techniques, introducing only a minor alteration to the conditional probability of the input labels. More technique contributions should be provided.

Reviews: review: paper_summary: paper_summary: The authors of this paper propose to predict label distribution from ternary labels, i.e., “0” indicating “uncertain relevant”, “1” indicating “definitely relevant” and “-1” indicating “definitely irrelevant”. Besides, they also provide theoretical analysis to show that the ternary label outperforms the binary label in most cases.
strength_weakness: strength_weakness: 1. The proposed Categorical distribution with Monotonicity and Orderliness (CateMO) contains four parameters, and parameter-sensitive analyses are absent.
2. It is not explicitly shown whether the label distribution degrees generated by the CateMO are close to the real label distribution.
questions: questions: 1. Why do you use the Eq.(10) as the ternary generation functions?
2. Are there any guidelines for selecting the parameters in Eq.(10)?
3. Is the proposed CateMO method effective in generating accurate label distribution?
limitations: limitations: Yes, the authors have adequately addressed the limitations.
paper_summary: paper_summary: In this manuscript, the authors explore how to learn the unknown label distribution from given ternary labels (0, 1, and -1). The main contributions of this work include: (1) a new label distribution prediction method is designed for handling ternary labels; (2) the theoretical analysis on the error of approximating the GT label description degrees by ternary and binary labels has been done.
strength_weakness: strength_weakness: (1)	Actually, some similar work has been done on “predicting label distribution from ternary labels” by researchers. For example, the problem definition of this manuscript is same to [17]. Although [17] has been discussed in the related work part, the authors did not point out that this work has made effort to address the same problem. This is quite strange! From this perspective, the novelty of this task is limited. Moreover, some claims of the authors maybe wrong. For example, “Since there is no LE method for ternary labels, we design…”.

(2)	Experiments are insufficient, and some existing label enhancement algorithms should be compared, e.g., [a], [b], [17]. Moreover, according to the suggestion of [17], there are many datasets can be used for this task, but the authors of this manuscript only evaluate their approach on three datasets.

[a] Fusion Label Enhancement for Multi-Label Learning. IJCAI, 2022.
[b] Label enhancement via manifold approximation and projection with graph convolutional network. Pattern Recognition, 2024.
questions: questions: (1) I think that the task of “Prediction label distribution from ternary labels” is a branch of weakly supervised learning. However, there is no discussion between them in this manuscript. 

(2) Actually, prediction unknown label distribution or label enhancement has been employed to deal with some other machine learning tasks, e.g., multi-label learning, partial label learning, feature selection and so on. But, the authors do not mention the possible extension of the proposed model to these or other scenarios. 

(3) The advantages and disadvantages between the presented algorithm and existing discriminative/generative label enhancement ones should be discussed.

(4) The objective function of the proposed method is not shown. I think it is important for readers to clearly understand that how the method learn the unknown label distributions from ternary labels. By the way, I do not think the unknown label description degree is equal to the probability of an instance belonging to the positive label or negative label, though they are usually positively correlated with each other. So, I wonder that the proposed model how to enrich the supervised information in the label space and generate the label description degree.

(5) From Table 1, I find that the results of different approaches on some metrics are closed. Therefore, the statistical significance analysis should be done, which can show whether the accuracy differences are significant or not.

(6) The parameter sensitivity analysis of the three hyper-parameters in the proposed model is missed.
limitations: limitations: I find that LDL-LRR is used as the LDL model in the testing phase. This two-stage strategy is a limitation of its application in real-world scenarios. I encourage the authors to talk about the possible solution on designing a one-stage extension model based on the proposed method.
paper_summary: paper_summary: The authors propose a novel multi-label annotation scheme, transitioning from the traditional binary annotation to a ternary annotation. They have validated this new annotation scheme through both theoretical analysis and experimental verification.
strength_weakness: strength_weakness: The field of multi-label learning has developed various learning models to address different limitations of the label space, such as weakly supervised learning, partial multi-label learning, multi-label learning with class-conditional noise, and weakly semi-supervised multi-label learning. Research in these areas is well-established. The relationship between the "uncertain relevant" concept mentioned in the paper and existing work needs further clarification.
questions: questions: 1. The proposed approach of quantifying the advantages of ternary labels over binary labels through approximation error analysis is commendable. However, I am curious about the impact of the proportion of "uncertain relevant labels" in practical scenarios. In other words, if the proportion of "uncertain relevant labels" is excessively high within the overall labeling, does the proposed ternary annotation scheme still offer a statistically significant advantage over the traditional binary annotation scheme?

2. While the authors validate the superiority of the ternary annotation scheme through experiments, the fairness of the existing experimental setup in terms of method comparison needs further clarification. As the paper suggests, there is no established LE method for ternary labels, and the experimental model proposed by the authors is new. It is recommended that the authors discuss the fairness of the comparative experiments in greater detail.
limitations: limitations: the authors adequately addressed the limitations
paper_summary: paper_summary: The paper proposes a more cost-effective approach to label distribution inference, i.e., predicting label distributions from ternary labels. Theoretically, the paper elucidates the superiority of the ternary label by analyzing the error of approximating the ground-truth label description degrees by ternary and binary labels, respectively. Methodologically, the paper proposes a categorical distribution with dimensional monotonicity and orderliness, which is theoretically proven to preserve the monotonicity and ordinality of the probabilities associated with ternary labels, to capture the process of generating ternary labels from label description degrees.
strength_weakness: strength_weakness: 1.	There is a lack of introduction to the three-way philosophies. Since most readers may not be familiar with the three-way philosophies, it is necessary for this paper to provide a proper introduction to the three-way philosophies, rather than just presenting their names and references.
2.	Figure 3 and Figure 2 of the paper appear in a wrong order. Figure 3 should be placed after Figure 2.
3.	In Figure 3, the triangle area should be $0\le \hat{\tau}\le \hat{\kappa} \le 1$, instead of $\hat{\tau}\le \hat{\kappa}$, which should be corrected to make the figure more rigorous.
questions: questions: I have the following two questions. First, how does the proposed CateMO distribution work with existing label enhancement algorithms to infer label distributions from ternary labels? Could you please answer this question with some detailed examples? Second, to offer some intuition for setting the lambda parameters in practical applications, could you please provide a semantic interpretation of the lambda parameters of CateMO distribution?
limitations: limitations: The paper has adequately discussed the limitations and the potential solutions of the proposed method. Besides, I don't believe that the paper has a potential negative social impact.

Comments: comment: Thanks for your responses. I have thoroughly reviewed the comments from other reviewers and the corresponding responses. I find that most of my concerns have been addressed by the authors. Currently, I have decided to retain my score.
Dear Reviewer:

We would like to kindly inquire whether our responses have adequately addressed the concerns you raised during the review process. Besides, we are also seeking to ascertain if there are any new question that you would like us to clarify.
Dear Reviewers:

The deadline imminent. Are there any new concerns about our paper or response? If so, please do not hesitate to communicate with us, and we will respond promptly.