Optimal Threshold Labeling for Ordinal Regression Methods
Authors: Ryoya Yamasaki
Venue: Accepted by TMLR ()

Abstract: For an ordinal regression task, a classification task for ordinal data, one-dimensional transformation (1DT)-based methods are often employed since they are considered to capture the ordinal relation of ordinal data well. They learn a 1DT of the observation of the explanatory variables so that an observation with a larger class label tends to have a larger value of the 1DT, and classify the observation by labeling that learned 1DT. In this paper, we study the labeling procedure for 1DT-based methods, which have not been sufficiently discussed in existing studies. While regression-based methods and classical threshold methods conventionally use threshold labelings, which label a learned 1DT according to the rank of the interval to which the 1DT belongs among intervals on the real line separated by threshold parameters, we prove that likelihood-based labeling used in popular statistical 1DT-based methods is also a threshold labeling in typical usages. Moreover, we show that these threshold labelings can be sub-optimal ones depending on the learning result of the 1DT and the task under consideration. On the basis of these findings, we propose to apply empirical optimal threshold labeling, which is a threshold labeling that uses threshold parameters minimizing the empirical task risk for a learned 1DT, to those methods. In experiments with real-world datasets, changing the labeling procedure of existing 1DT-based methods to the proposed one improved the classification performance in many tried cases.

Published in Transactions on Machine Learning Research (1/2023)
Optimal Threshold Labeling for Ordinal Regression Methods
Ryoya Yamasaki
Deportment of Systems Science
Graduate School of Informatics, Kyoto University 36-1 Yoshida- Honmachi, Sakyo-ku, Kyoto 606-8501 JAPAN
Reviewed on OpenReview: https: // openreview. net/ forum? id- milSAy1n652
Abstract
For an ordinal regression task, a classification task for ordinal data, one-dimensional trans- formation (IDT)-based methods often employed since they are considered to capture the ordinal relation of ordinal data well. They learn 1DT of the observation of the explanatory variables so that an observation with larger class label tends to have a larger value of the IDT, and classify the observation by labeling that. learned 1DT. In this paper, we study the labeling procedure for 1DT-based methods, which have not been sufficienly discussed in existing studics. While regression-bast based mothods and classical threshold methods con- ventionally use threshold labelings, which label learned 1DT according to the rank of the interval to which the 1DT belongs among intervals on the real line separated by threshold parameters, we prove that likelihood-based labeling used in popular statistical DDT-based methods is also threshold labeling in typical usages. Moreover, we show that these thresh- old labelings can be sub-optimal ones depending on the learning result of the 1DT and the task under consideration. On the basis of these findings, we propose to apply empirical optimal threshold labeling, which is threshold labeling that uses threshold parameters minimizing the empirical task risk for a learned 1DT, to those methods. In experiments with real-world datasets, changing the labeling procedure of existing 1DT-based methods to the proposed one improved the classification performance in many tried cases.
1 Introduction
Ordinal regression (OR) (or called ordinal classification) is the classification of ordinal data in which the underlying target variable categorical and labeled from a label set (ordinal scale) that is equipped with a natural ordinal relation for the explanatory variables; see Section 2.1 for a detailed formulation. The ordinal scale is typically formed as a graded (interval) summary of objective indicators like age groups {'0-9', '10- 19', .... '90-99', '100-'} or graded evaluation of subjectivity like human rating f'excellent', 'good', 'average', 'bad', 'terrible),, and ordinal data appear in various practical applications: age estimation (Niu et al., 2016; Cao et al., 2020), information retrieval (Tiu, 2011), movie rating (Yu et al., 2006), and questionnaire survey in social research (Bürkner & Vuorre, 2019).
One-dimensional transformation (1DT)-bused methods are often applied to the OR problems as a simple way to capture the ordinal relation of ordinal data: they learn a 1DT of the observation of the explanatory variables so that an observation with a larger class label tends to have a larger value of the 1DT, and classify the observation by labeling that learned 1DT, as will formalize in Section 2.2. For example, regressiion-bused methods and classical threshold methods (or called threshold models) conventionally use a threshold labeling, which labels a learned 1DT according to the rank of the interval to which the IDT belongs among intervals on the real line separated by threshold parameters. Regression- based hods (Kramer et al., 2001; Agarwal, 2008) learn a 1DT by solving a naive regression task that infers a class label by the 1DT a continuous scale, and often apply nearest-neighhor threshold (NNT) labeling that rounds the learned 1DT to its nearest label (see Section 3.1). Classical threshold methods (Shashua & Levin, 2003; Lin & Li, 2006; Chu & Keerthi, 2007; Lin & Li, 2012; Li & Lin, 2007; Pedregosa et al., 2017) learn

---

Published in Transactions on Machine Learning Research (1/2023)
1DT using an objective function different from the empirical task risk, and commonly use minimum threshold (MT) and summation threshold (ST) labelings that apply parameters obtained incidentally in that learning procedure as threshold parameters (see Sections 3.2). Also, statistical IDT-based methods (statistical methods) (McCullagh, 1980; Williams, 2006; Cao et al., 2020; Yamasaki, 2022), in which the learning procedure of the 1DT can be viewed as statistical modeling of conditional probabilities of the data, can apply läkelihood- (LB) labeling that is designed to minimize the task risk under the expectation that the assumed likelihood model is correctly specified to the data distribution (see Section 3.3).
We, however, have respective concerns on these existing labeling procedures. First, threshold parameters of NNT, MT, and ST labelings are generally not designed to minimize the task risk. So their threshold parameters can be sub-optimal for the minimization of the task risk, as we will demonstrate in Example 1. On the other hand, the LB labeling is designed to minimize the task risk if the assumed likelihood model is correctly specified to the distribution of the data (see Theorem 2). However, 1DT-based likelihood models have a strongly restricted representation ability and can be mis-specified to the data, and hence the LB labeling can degrade the classification performance depending on the data distribution.
Previous studies have done little theoretical work on the properties of these existing labelings. Therefore, we first study the relationship between these labelings. In particular, we show in Theorem 3 that not only the NNT, MT, and ST labelings but also the L.B labeling in typical usages is a threshold labeling. This finding motivates us to search for a better labeling function among the class of threshold labelings. Then, in Section 4, we propose to apply empirical optimal threshold (EOT) labeling, which is a threshold labeling that uses threshold parameters minimizing the empirical task risk for the learned 1DT, to IDT- based methods, under the expectation that the 1DT is learned successfully and the empirical (training) task risk becomes a good estimate of the (test) task risk. Here, the threshold parameters for the EOT labeling can be computed with a computational complexity of quasi-linear order ( (og n) regarding the training sample size n using a dynamic programming- (sce Section 5 for the relation of our proposal to several previous studies including this reference).
algorithm (Algorithm 1) mentioned in Lin & Li (2006)
In this study, we further performed numerical experiments of the OR task for real-world ordinal data to confirm the practical effectiveness of the EOT labeling (see Section 6 and Appendix D). The EOT labeling gave superior generalization performance (more exactly, smaller test task risk) than the NNT, MT, ST, and LB labelings, in many tried cases. Also, a modified 1DTTbasee method with the EOT labeling outperformed an existing based method using the ST labeling that has been declared by Cao et al. (2020) to be state-of-the-art in 2020 for the age estimation from the facial image.
Therefore, in this paper, we propose to change labeling procedures of (existing) 1DT-based OR met to the EOT labeling, on the ground of the fact (sce Example and Theorem 3) that the NNT, MT, and ST labelings and LB labeling in typical usages are possibly sub-optimal threshold labelings, and empirical effectiveness of the EOT labeling.
2 Preliminaries
2.1
Formulation of OR Problem
OR is the classification of ordinal data. The ordinal data have an underlying categorical target variable Y E [K] = K) that is equipped with an ordinal relation naturally interpretable in the relationship with explanatory variables X & Rd, where d and K are supposed to be integers larger than or equal to 1 and 3. respectively.' 1 We here suppose that the target class labels are encoded to 1, I,..., K in an order- onder-preserving manner, like from 'excellent'... ..... 'terrible' to 1.....5.
The task of the OR (OR task) is basically the same that of the standard (including cost-sensitive) classification, to obtain good classifter f Rd + [K]. For a user-specified task loss function { [K]2 + oo), it is formulated as minimization of the task risk B[((((X), where the expectation
Forr better modeling of the ordinal data, it would be important to provide mathematical characterization and further discussion of the natural ordinal relation. However, it would be related to learning procedure of the 1DT (defined in Scction 2.2) more closely, and necessity is not so great for the analysis and proposal of this study, so we will not mention in this paper. Refer to, for example, OR studies (da Costa al., 2008; Yamasaki, 2022) for the discussion on such characterizations.
2

---

Published in Transactions on Machine Learning Research (1/2023)
value [[-] is taken for all random variables in its argument (here X and Y). Popular task losses for OR tasks include not only the zero-one task loss kzo(j.k) == 1(j # k), where (c) takes 1 if a condition c is true and O otherwise, but also V-shaped losses (for cost-sensitive tasks) reflecting one's preference of smaller prediction errors over larger ones such as the absolute deviation task loss (ud(j.k) = -j - kl, squared task loss (i.) = (j- 10-kk 2)², and (2o.c kj.k) = [(j k| > c) with c c20 O.
For the evaluation in the OR task, one may use criteria that cannot decomposed into a sum of losses for each sample point: for example, quadratic weighted kappa (Cohen, 1960; 1968). Our discussion in this paper does not cover such criteria.
2.2 Formulation of One-Dimensional Transformation Bassed Methods and Threshold Methods
In this paper, we discuss only based OR methods. We here provide notations and terminologies common for all the 1DT-based OR methods.
Suppose that one has the sample D, = ((t)))))))))))) y=) each of which is drawn independently from an identical distribution of (X.Y). First, 1DT-based methods learn a 1DT a Bd - R of an observation of the explanatory variables from a user-specified class A S {a Rd - R} (1DT class) (c.g., a neural network with a fixed network architecture and learnable weight and bias parameters) possibly together with other objects that an observation with larger class label tends to have larger value of the IDT; we call this procedure the learning procedure (of the IDT). Next, IDT-based methods build up classifier f= ā with a learned 1DT ā and a labeling function h R - [K]; we call this procedure the labeling procedure. Most existing DDT-based methods can be seen as adopting one of the NNT, MT, ST, and LB labelings, depending on the properties of their learning procedure, as we review later in Section 3.
In particular, we call a labeling function that can be represented as
h(u) (uh 1) == 1+ Σ
1(u
(1)
with parameters = (11, (f1..... (x- E RR as the threshold labeling. Also, we call the paramcters t as the threshold parameters, and 1DT-based method using threshold labeling as the threshold method, in this paper. Note that this formulation of the threshold method is generalization of the one employed in most previous studies on conventional threshold methods that we will review latter in Section 3.2. The threshold labeling hehee (u; has the following properties:
Proposition 1. The threshold labeling (u; t) is non-decreasing and right-continuous in u E R and in- variand regarding the permutution of the threshold parameters Ix.-..,KK-1- Conversely, an arbilrary non- decreasing right-continuous function h certain threshold parometers t E R- (f.e., there crist t E RK-1 such that h(u) h (u:t) for any R) or their permulation. Also, if (1..... 'K-1 lake only L different values, then hehr t) has L change points " Uu, such that hur - e) # (u; with sufficiently small e > 0 for 1 == I...,,
[K] can be represented by a threshold labeling (int(;t) with
The last result implies that the threshold labeling is the simplest as the labeling function in the sense that the resulting classifier has only 1) decision boundaries for the learned 1DT at most.
Note that, in the learning procedure, since the empirical task risk 1, Σn (h((((:)))))) is discontinuous with respect to the IDT a and hence difficult to optimize numerically, many methods depend on another ob- jective function. In this paper, we call that objective function the empirical surrogate risk, its population version the surrogate risk, and its data-dependent minimal component the surrogale loss (function).
2.3 Formulation of Policy of This Study
The goal of this study is to improve the labeling procedure of DDT-b methods. Thus, regarding the learning procedure of the IDT, this paper adopts those by existing studies, and we do not discuss the goodness of the lcarning procedure. Assuming that a learned 1DT ā and task loss t are given, we will discuss the goodness of the labeling function h with the task risk (((()(((X))))) Y)] or the empirical task risk n
I Σn ((h(ā(x:)). y1) as criterion, since the aim of the OR task is to minimize the task risk.
3

---

Published in Transactions on Machine Learning Research (1/2023)
3 Review and Analysis of Existing 1DT-Based Methods and Labeling Functions
3.1 Regression-based Method with Nearest-Neighbor Threshold (NNT) Labeling
Regression-based methods (Kramer et al., 2001; Agarwal, 2008) learn a 1DT a by solving a naïve regression task that infers a class label by the 1DT: for regression loss function ø R x [K] - [0),0),
min aER 22
Ma((:)).)))
(2)
For example, Kramer et al. (2001) used the squared (SQ) loss (u, == (u y) and Agarwal (2008) used the odsiitie-deviation (AD) loss (u. == 1u yl, as regression loss function &.
As a labeling function h, regression-based methods often use the NNT labeling
hant (1) == (u; (1.5,
(3)
that is throshold labeling (: with the threshold parameters r k 0.5, k 1.....,,, K K-1. The NNT labeling rounds the learned 1DT to its nearest label.
The regression- based methods using the above two surrogate loss functions and NNT labeling have the following optimality guarantee for an OR task with a respoctive spocific task loss:
Theorem 1 (Pedregosa et al. (2017, Theorems 9 and 11)). Let (l, be (fad, Paa) or (laq. Qsa). Then, for the surrogale risk minimizer à E arg a. B[o(a(X),)) based on the surrogale loss $, the classifier fant = hant o a minimizes the task risk B(((f(X),Y) for the task loss (CC(fant)))) = miny B[C(F(X),)).
According to this theorem, the regression based methods (Kramer et al., 2001; Agarwal, 2008) would be ex- pectable to work well the sample size and 1DT class A are sufficiently large. On the other hand, if the 1DT class is strongly restricted, a classifier basod on the NNT labcling may be sub-optimal for the OR task (be- cause, generally, m[ (X), Y)] # B[((h(((X))) Y)] for funt hant à with ā arg min B[ X), Y) and h arg m X(((h(a(x))xx)))
3.2 Classical Threshold Method with Minimum and Summation Threshold (MT and ST) Labelings
Classical threshold methods have been studied actively the machine learning and statistical litera- ture (Shashua & Levin, 2003; Chu Koerthi, 2007: Li & Lin, 2007; Lin Li, 2012; Pedregosa et al., 2017: Cao et. al., 2020). Many of these methods formulated with learning procedure that simultaneously learns (K 1) parameters in addition to the 1DT
Σ b.
min BE
(4)
where call b (bi....,bk-)) bias parameters (BPs class), and R x R [K] - [0, oo) is a surrogate loss function. As the labeling function h, several early works (Shashua & Levin, 2003; Chu & Keerthi, 2007) use the MT labeling
bk-1) E k-1 the bias parameters, B RK-1 is user-specified class for the
h (u; b) min(k [K] ba} with bg
(5)
with learned bias parameters b = bx-1), and more recent works (Pedregosa et al., 2017; Cao et al., 2020) use the ST labeling
lus b) = hehr (u; b).
(6)
The MT and ST labelings threshold labelings and have the following relationship:
4

---

Published in Transactions on Machine Learning Research (1/2023)
Proposition 2. Given b RK-1 together with bo == -00, let rk be bi. with iz == min(j E (0....,k) ,} by b for each k = =1...., 1. Then, one has that hitt (1; b) = (uh t) with t = ((x....,(x-1))) Also, (u; b) (un b) if b1 S .... ......... bx-1-
We remark that the MT labeling (u; b) and ST labeling (u; b) are different when the learned bias parameters are not ordered.
A surrogate loss function Φ(u, b, k) for these threshold methods often built by replacing step functions
11 ) and 1 1 in the expression of the task loss ((.,k) = ((k,k) + Ei= l(I + 1, 1, 2 .) + Ei EK- ((((+1,K) - E(1, (1]+ + by convex surrogates widely-used in binary classification (Lin & Li, 2012; Pedregosa et al., 2017) such as the hinge, logistic, and exponential losses, so that b(.bb.) becomes continuous convex upper bound of (( k). For example, in the terminology of Pedregosa et al. (2017), the immediate threshold (IT) loss function
y 1, y ((a(x)) by-1) p(bb a(x)), otherwise
p(bi a(x)),
(((x)).by)) = f(a(x) bx-1),
(7)
is an upper bound of zero-one task loss Ez0, and the all threshold (AT) loss function
Σ pbbb - a(x)),
y if y
(a((x),
ΣK ((a(x)) - bk), Ex p(a(x)) - bbx) Ekey 4(bx - a(x)), otherwise
(8)
is an upper bound of absolute deviation task loss fad- For instance, SVOR-IMC (Chu & Keerthi, 2007)
uses the IT loss with (()) min(0, 1 u), fixed margin strategy (Shashua & Levin, 2003) and SVOR-EXC (Chu & Keerthi, 2007) use the AT loss with ((u) min(0, u), ORBoost-LR (Lin & Li, 2006) uses the IT loss with u(u) = e" ORBoost-ALL (Lin & Li, 2006) uses the AT loss with pp()) e e" and CORAL (Cao et al., 2020) uses the AT loss with p(u) = log( (1 "*)..
As the BPs class B, the non-restricted (non-ordered) class RK and ordered class (b E RK b are often applied. As a simple implementation of the ordered class, Franses & Paap (2001) mentioned to parametrize the bias parameters b
x-1 b1 .... S
bj = bj. and by = bk-1 bb,2 for k = 22....
(9)
with other parameters b'.....b'..
b/-11 E R. When the ordered BPs class is applied, the MT and ST labelings bring the same classification results. Also, for many AT loss functions, bias parameters b of the surrogate risk minimizer (ā, b) E arg aT B[ x),& b, Y)] are ensured to be ordered (b) ... s bx-1) (sce Chu & Keerthi (2003, Lemma 1) and Li & Lin (2007, Theorem 2)), and hence the use of the ordered BPs class will be justiffed.
The use of the surrogate loss y and the MT or ST labeling h, which make o(,, b. y) (with ordered b) to be an upper bound of the task loss ((h(-),)) y), is almost like a convention and may facilitate generalization analysis (Ti & Lin, 2007; Lin & Li, 2012), but the goodness of selecting the MT ST labeling is not supported by theoretical discussion. We demonstrate in the following example that the MT or ST labeling may be a negative factor that degrades the classification performance of threshold methods:
We here consider the IT loss (7) with p(u) min 1 u), which we denote Oninge-it it (u, b, k) and call Hinge-IT loss. The Hinge-IT loss is continuous convez upper bound of the zero-one task loss with the MT labeling (m: (), k) (and ST labeling lzo (= b).k)) when b is ordered (i.., b .... bx-1): So one may think that the Hinge-
Example 1.
&hinge-itl b, k) is a conver function and &hinge-itt-. b, k) 2 (:; IT loss and the task with the zero-one task loss ( -Task-Z) have friendly compatibility, from an analogy of a well-known result, classification calibration (Bartlett et al., 2006), in binary classéfication. However, the following demonstration shows that the MT labeling may be sub-optimal in minimizing the task risk as labeling function for the combination of the Hinge-TT IT loss and Task-Z.
5

---

Published in Transactions on Machine Learning Research (1/2023)
We consider 4-class OR problem (let K 4), and suppose that the data appear only on 4 different points _!!!!......! xr.11 in Rd and follow the probobility distribulion, Pr(z)(1) = 0.25 and (Pr(1[æ (r.,, (0.0,,.4.,00.1), (0.3, 0.5. (0.2. 0, 0.5, 0.3), (0.1, 0.0.00.5 for = 1...,,,,
Pr(K]
It can be shown that the surrogate risk minimizer (a, b) E arg mi fies bj b2 by 0, a(xl1)) = a d(xl2))
Bioninge-it (a (d)X),b b, Y) satis-
-1, and ((xl1)) = a(x14))= =1 (ignore the translation incariance) by several simple calculations. The MT labeling predicts a label of the data on x!i) as hmmt lmt for i = -.....4 ( ( (a(X); b). X)] = 0.6), despite that using different threshold parameters (say = (-),0,)) one can predict it as b (a(x:!));; = 2,3,33 for l....,, 4 and yield a smaller value of the task risk (hitzo(hmt (ā( (); 1), Y)] = 0.55).
= 1, 4, 4
O
3.3 Statistical Methods with Likelihood-Baste (LB) Labeling
In the OR research in statistics, several methods have been developed according to the statistical mod- eling of the conditional probabilities of the data through 1DT (McCullagh, 1980; Williams, 2006; Chu & Ghahramani, 2005; Yamasaki, 2022). For example, the cumulative link model (McCullagh, 1980), which is popular in the OR research, models the conditional probabilities Pr(y(æ), (x, E ER x [K] by
o(by-á(z)) a(x)),
if y= if k otherwise,
P(y;),ā(z)))) = (- - a(x));
(10)
((by-ā(z)) o(by-1--(a0
where o R [] is a link function that is non-decreasing and satisfies o(-oo) = 0 and (+co) = (i.e., o is a cumulative distribution (CPD) function), ā : Rd - R an assumed 1DT, and b = (b), .., bgb....)..b.... e R- are assumed bias parameters that satisfy bi .... bk-1- As a link function ir, ordinal logistic regression (OLR) (McCullagh, 1980) applies the CPD function of the logistic distribution (a.k.a. the sigmoid function) Togisttic (u) == 1) and cumulative probit model (Agresti, 2010, Section 5.2) and Gaussian process OR (GPOR) proposed by Chu & Ghahramani (2005) use the CPD function of the standard Gaussian distribution ka.k.a. the inverse function of the probit function) ggauss (4) == L (2x)- -122 e 12 dv.
In the learning procedure of the IDT, statistical methods apply surrogate loss functions associated with their statistical modeling. For the cumulative link model, a surrogate loss function & and the BPs class B for the learning procedure same as (4) should satisfy
(ū, arg min wER.bEB
Σ oly,or a, ()b),,,.
(11)
A popular instance of the surrogate loss function is the negative log likelihood (NLL) loss function (nn o)) == - log P(y; or, u, b)}, for which the learning procedure amounts to the maximum likelihood estimation for the model (10). Also, Cao et al. (2020) used the AT loss (8) with q(u) = - lo){ which we call the all negative log cumulative likelihoods (ANLCL) loss function and denote dautel (u, b, y; r),, for o Orogistic- The learning procedure for this loss function can be characterized as the minimization of the sum of the NLLs of the models of cumulative conditional probabilities Pr(Y I &|X x) for binary classification problems, 'k or less' vs. 'more than k' k = I,...,,-
K 1.
The above interpretation on using the surrogate losses S.ll and Pantel under the statistical model (10) can be mathematically understood as follows:
Theorem 2. Assume that the random variable () underlying the dala has condilional probabilities that can be represented as (10): Pr(y]æ) = (0z,,,,(()). b) for every y E [K] and any x E Rd in the support of the distribution of X with o that is non-decreasing and satisfies o (--oo) = 0 and (++00) =1 (and of (o-)) for o= = danlel) such as Nogistic and Tgauss, à Rd - R. and b E R- satisfying b .... ... S bx-1- Let A be {g: Rd +-RR R), and (b.B) be (dull, (b E RR1 1 b .... S bk-1l), (Dantel. IR (Rk-1), 1, or (fantel, (b E RK-1
by S .... $ bg-1)). Then, any surrogate risk minimizer (a, b) E arg min miiinennnee bee B[o(a(X)) b, Y)] satisfies (yy,o,a(x), = Pr(ylæ) for any x E Rd in the support of the distribution of X.
....
We abhreviate the marginal probability Pr(X Pr(æ) and the conditional probability Pr(Y = =y|X= = =)) Pr(y[m) hhhs abbreviation applies an estimate Pr well).
6

---

Published in Transactions on Machine Learning Research (1/2023)
For such methods, not only the threshold labelings but also the LB labeling that grounds the assumed statistical model is widely-used option for the labeling function. Considering Theorem 2 and the equality [() = م Pr(yl=)((f(xt)) y), and aiming to minimize the task risk, ming:gd-11 (()(f(X),)), these methods can predict a label with the LB labeling
Insu; or, b.0)= = min arg min
Σ (1s:o.u.D)((...) kE[K] y=1
(12)
with learned bias parameters b, under the expectation that the assumed statistical model (10) correctly represents the actual statistical behavior of the data and it is learned successfully. Note that there can be a tie situation where objective functions with different of (12) take the same value, and arg [[] operation outputs multiple labels. The overlaid min operation in (12) avoids such a tie situation.
These methods lend to perform better when their assumed statistical model represents the actual statistical behavior of the data well. One can, however, find that the condition in Theorem 2 is very restrictive. Therefore, in many practical situations, their statistical model would deviate from the actual statistical behavior of the data, and then their 1DT model may not be learned appropriately, and the LB labeling hb((;,, b, E) may be sub-optimal for the learned 1DT model ä.
One may still consider that the LB labeling is more flexible in that it is generally not restricted within the class of non-decreasing threshold labelings, and superior to threshold labelings. However, we found that the LB labeling takes the form of the threshold labeling, for typical statistical models such as ones in OLR and GPŌR (i.e., the link function o such Olugisticr gaaus)) and for typical task losses such e= lzo. Es lad, lzq.
Theorem 3. Suppose that or is non-decreasing and salisfies (--o)) =0 and (++00) and that by s.... s... S bk- Then, the LB labeling hyp or, b. e) is
(1) a certain threshold labeling hur t) for some t E RK-1, ((k.)) at each fixed k E [K] is non-
increasing in I for I I and non-decreasing in for for12 k, and (x,(()) = j) [(k, j j 1) - ((1,)) +
E(I, + 1) at each fired different k, E [K] is non-positive (resp. non-negative) for all j [K respectively when k <1 (resp. k 1), for example, b= (ad, bzg.
(ii) certain threshold labeling hubr(u; t) for some te RK-1, if l (zo. bzo, with E [K,()/2),, or is differentiable, o'(v) == (vv even and non-increasing in if v > O, and ((())-((((2)) is non- increasing in V1 with fixed v2 (and in V2 with fixed v1) if V1 < v2, for example, o = Oogistic. gaauss where [v] is the greatest integer less than or equal to v,
(r(n)-or(v))
(iii) the threshold labeling him (u; b) that is same as the MT labeling hma (u; b) and ST labeling h (uc b), if = Ead and (0)) 0.5.
Here, Theorem 3 (i) assumes that the task loss is V-shaped, and the condition on la in Theorem 3 (i) holds under the convexity of defined below:
Theorem 4. () at each fieed different k. E [K] is non-positive (resp. nen-negative) for all j E [K - 1] respectively when & < 1 (resp. k > 1), if the lask risk f is conver in the difference of the two arguments:
ks) S (j2 - k2) -j11 k1) k1) + (22-22) -2) (js- ks) (j2 - k2) - (j1- k1)
(j3-k3) - ((1-k))
0(2,, k2)
(13)
for all Ji.....k E [K] such that ji k1 # j2 - k2 and jn - k1 s js K3 s j2 - k2.
The condition on o in Theorem 3 (ii) comes from the consideration for non-convex task losses.
A result similar Theorem 3 also holds for 1DT-basod likelihood models other than the CL model (10); refer to Theorem 5 in Section B. For 1DT-based statistical methods, it may be common that their LB labeling is threshold labeling.
7

---

Published in Transactions on Machine Learning Research (1/2023)
4 Proposal of -DT-ba Methods with Empirical Optimal Threshold (EOT) Labeling
In typical usages, not only the NNT, MT, and ST labelings, but also the LB labeling is a threshold labeling, as we confirmed in Theorem 3. Thus, we consider that would be meaningful to aim for a better threshold labeling for improving the classification performance of existing 1DT-based methods. Recalling that the final goal is to make the task risk B[((f(X),Y) small, and expecting that the 1DT was learned successfully and the empirical (training) task risk becomes good estimate of the (test) task risk, we propose to apply the EOT labeling
h(u; beoc) with leot E arg min = Σ (a(a)) y)) taRK-1 n
1
(14)
that uses threshold parameters minimizing the empirical task risk for a given learned IDT model a.
The threshold parameters for the EOT labeling can be computed with a dynamic programming- al- gorithm (Algorithm 1) mentioned in Lin & Li (2006); see also a researchers site (https://homl woor....e....................... caltech.edu. wwtpa//htti///pgrraaaaooo orensemble/) / of Lin & Li (2006), and Soction C of our paper for its opti- malily guarantee. This algorithm first. sorts unique elements of (a(=;)); to {āj), and takes advantage of the recurrence relation (46) of the minimizer of the empirical task risk for sample points i's s.t. a(æ;) S aj along the ascending order of {ā,} to calculate threshold parameters minimizing the empirical task risk. It costs a computational complexity of quasi-linear order (n log n) regarding the training sample size n, which stems from the sorting operation in Line 2 while the rest operation in Lines 3-15 costs a computational complexity of (nK).
The NNT labeling for regression-basee methods (reviewed in Section 3.1) and LB labeling for statistical methods (reviewed in Section 3.3) have optimality guarantees for the task risk minimization under ideal situations; refer to Theorems and 2. Also, many threshold methods employ (K- - 1) bias parameters, and those bias parameters can be directly used in their labeling procedure like MT and ST labelings, as reviewed in Section 3.2. Presumably, for these reasons, a formulation that allows a threshold labeling with variable threshold parameters has not been considered for these methods. Our formulation in Section 2.2 introduces the threshold labeling with variable threshold parameters and clearly distinguishes the bias and threshold parameters. This is also an important contribution of this paper: due to this formulation, it becomes natural to consider the application of the threshold labeling to 1DT-based methods with different number of bias parameters than (K - 1) such PO-VS-SL (Yamasaki, 2022). Furthermore, this led to the EOT labeling that has potential to improve the classification performance.
We, however, have to provide a remark on the additional learning of the decision boundaries (here threshold parameters) after the learning of the learner model (here a 1DT): the additional learning generally has a risk of enlarging the generalization gap. One can adjust the labeling function so that h(g(x;)) = y; for every training example
L, l.....n n if allowing arbitrary formats and yi, = Ví2 for any i1, s.t. Zi = 3i2 but the resulting classifier h g would have quite low generalization performance. On the other hand, we here consider the additional learning of the labeling function among the class of threshold labelings. A threshold labeling has up to (K- 1)) decision boundaries, that is, it is strictly restricted, and we expect that the degree of the generalization gap will not differ much with any threshold labelings.
5 Relationship to Other Previous Studies
Most existing studies 1DT-based methods discuss the learning procedure. In Section 3, we reviewed point- wise 1DT-based methods in which the learning procedure is formulated with surrogate loss function defined for each point (a(æ;), yi), but some 1DT-based methods employ different formulations. For example, pair-wise 1DT-based methods including RankSVM (Herbrich et al., 1999) and RankBoost (Lin & Li, 2006) are formulated with a surrogate loss function that is defined for each pair of two points (a(2;).)) y;) and (((e)))))))))))) see also Lin & Li (2012); Gutierrez et al. (2015) for survey of such methods. The EOT labeling can be applied to those 1Da-b hoth well.
8

---

Published in Transactions on Machine Learning Research (1/2023)
Algorithm 1: Calculation of the threshold parameters for the EOT labeling
Input: Task loss E, learned 1DT a, and training data D,, /* Preparation of {ay) and y, for j =I, I....,N. jel be unique elements of (á(=;)) a', # d for all jı, [N] s.t. ji j2, and a(æ;) E { for all e [n].
1 Let (a)) N
2 Sort { in the ascending order, and represent the result as {āz) S $ ON.
3 Create sets y/ b a(x;) =
/* Calculate matrix L E RNNK
4 for k = L K do
5 L1.k = ===, Eyey, y(k.):).
6 for ==2..... N do
7
for = L.....,,K K do
/* An efficient implementation of LJA = minje[k] 45-1J Eyey, ((ky))
8
Limp j-111 (if k 1), LinLump:Lj-1..t (otherwise).
9
= + Eyey, ((k.y))
/* Calculate threshold parameters E.
10 I < minjarg LN,I).
11
# then
Let &x be value larger than aN (e.g., too) for k K,...,K K-
12
for - do
13
J - mang min Lj,)).
14
if I J then
| Tx (aj /2+1)/2 for k = J,...,I- and / - J.
15
/ # 1 then
Let & be value smaller than āi (e.g., -0o) for & l...., Output: Threshold parameters & == (T1....,Tke Tx-1).
On the other hand, there are several previous works that have studied the labeling procedures different from NNT, MT, ST, LB, and EOT labelings. We review the discussion of those previous works, and describe the relationship between their methods and the EOT labeling, in the following.
Herbrich et al. (1999) considered pairwise 1DT-based method based on hinge-type surrogate loss, for the task with the zero-one task loss. Their method (Herbrich et al., 1999, (12)) adopts a threshold labeling with threshold parameters determined by minimizing a unique criterion that emphasizes the shape the used hinge-lype loss and different. from the (empirical) surrogate risk, and has no optimality guarantee in the task risk minimization.
Lin & Li (2006) considered three methods; RankBoost (4), ORBoost-LR (7), and ORBoost-ALL (8) of this reference. The third method applies the AT loss with ς(u) = e (ORBoost-AL.L). and its labeling procedure is the same as the MT and ST labelings in this paper. Also, one can understand that the second method, ORBoost-LR (7), tries to minimize (an upper bound of) the empirical surrogate risk based on the IT loss function with (#) = em and ordered BPs class in its learning procedure, and its labeling procedure also can be seen follow the idea of the MT and ST labelings. Finally, the first method, RankBoost, wise 1DT-based method, and its objective function (4) for the labeling procedure is defined as the empirical task risk with the absolute deviation task loss. This labeling procedure is similar to the EOT labeling, but Lin & Li (2006) did not mention the relevance between that objective function and the task under considcration. Although Lin & Li (2006) presented a description on the threshold parameters determination based on the zero-one task loss at the part. following (4), they tried only the threshold parameters determined by minimizing the empirical task risk with the absolute deviation task loss even when they considered the task with the zero-one task loss their experiments. The main claim of our considcration in Section is that the threshold parameters should be determined via minimizing the (empirical) task risk for the task under consideration. Therefore, the FOT labeling in Section 4 be interpreted as variant of
pair-

---

Published in Transactions on Machine Learning Research (1/2023)
Table 1: Dataset properties, the total sample size ntot, the dimension d of the explanatory variables, and
the number K of classes of the target variable, of the RW datasets used for the experiments in Section 6.
COL
PAS
SQ1
SQ2
BON
TAE
AUT
NEW
TOY
ESL
Btot
24
36
52
52
57
151
205
215
300
488
25
51
52
37
54
71
2
K
3
3
3
5
3
6
5
9
BAS
EUQ
LEV
ERA
SWD
WQR
CAR
MORPH
CACD
AFAD
Atot
625
736
1000
1000
1000
1599
1728 21
55013
159402 128 x 3
164418 1282 x
91
10
128 x 3
K
5
9
4
4
55
49
26
those for Lin & Li (2006, (4)) with respect to the relevance between the objective function to determinc the threshold parameters and the task under consideration. Our contributions, namely, are to expand the threshold, and statistical methods reviewed
scope of applicability of the EOT labeling to regression- in Section 3, and to modify the EOT labeling to be performed based on the task loss function corresponding to the task under consideration, not the development of Algorithm 1 as method for the minimization of the empirical task risk with a given task loss function.
6 Numerical Experiments with Real-World Datasets
Purpose We performed numerical experiments using real-world (RW) datasets to answer the question, whether modified 1DT-based method with the EOT labeling can yield better classification performance (i.e., smaller test task risk) for the OR task than existing DDT-based methods using other labeling functions.
Datasets and Preprocessing In the experiments, we used the 17 various-domatt datasets, COL (contact-lenses), PAS (pasture), SQ1 (squash-stored), SQ2 (squash-unstored) BON (bondrate), TAE (tae), AUT (automobile),, NEW (newthyroid), TOY (da Costa et al., 2008). ESL (employee selection). BAS (balance-scale), EUQ (eucalyptus), LEV (lectures evaluation), ERA (employee (ateetiion/cceptancce). SWD (social workers decision), WQR (winequality-red), CAR (car evaluation) datasets, and the face-age datasets, MORPH (MORPH Album2), CACD, and AFAD datascts (Ricanck Tesafaye, 2006; Chen et al., 2014; Niu el al., 2016). The main reason why we used the various-domain and face-age datasets is respec- tively to experiment with many real-world datasets in various domains and to confirm whether the proposed method achieves the classification performance competitive to the state-of-the-art method in modern ap- plication. For most of the experimental settings, we referred to those of the previous studies (Cao et al., 2020; Gutierrez et al., 2015) with few modifications."
For the various-domain datasets, we used datasets, which Gutierrez et al. (2015) used as OR datasets, follow- ing the setting of the explanatory and target variables in Gutierrez et al. (2015). One can obtain the various- domain datasets from a researchers' site (http: //www uco wttp://////.///////ooreeeeeeeeee
of Gutierrez et al. (2015) our GitHub repository /tttps://gtthbb... http://www...c.o/moooscsrrrrrroo We purchased the MORPH dataset at httpp://ebbl the whole image with the nose tip, which was located by facial landmark detection (Sagonas et al., 2016), at the center by using EyepadAlign function by Raschka (2018). While this dataset contains 55,134 facial images with ages from 16 to 77, we used 55,013 images with ages from 16 to 70. The CACD dataset can be downloaded from the MORPH dataset. Since the CACD dataset collects images from the Internet using computer vision techniques, it includes some facial images inappropriate for our consideration. Excluding images, in which no face or more than two faces were detected in the preprocessing, from the original 163,446 images, we used 402 facial images in the age range of 14-62 years. For the AFAD dataset obtainable at https:// github. com af aadd-daatsttttrballl.. because faces in its images were already centered, we took no further
www.:uuccceeauue.... wdwe //220231_usttores//ebbbb and preprocessed it that the face spanned
gittubthhbbb cchub.io/CARc/// We preprocessed this dataset similarly to
FFor the face-age datasets, used a part of program codes published in https///gttabb..oo/aaccceeeeeee coral-cnn by Cao et al. (2020), but results of our reproduction of their method differ from theirs mainly because we changed the learning rate from 5 x 10-5 to 1052.5, See https://githh/: http:///tttubb.cccommmmaaaa com//yamasaiiryyyy//ttt for program codes that we used.
group/
10

---

Published in Transactions on Machine Learning Research (1/2023)
preprocessing, and used its 164,418 images with ages 15-40. For these face-age datasets, we treated the age rank as the target variable. Table summarizes basic dataset properties, the total sample size Boot the dimension d of the explanatory variables, and the number K of classes of the target variable, of all the used datascts.
For the various-domain datasets, we randomly divided each dataset into 72 % training, 8% validation, and 20% test sets. For the face-age datasets, we resized all images to 128 x 128 x 3 pixels (3 stems from RGB channels) and randomly divided each dataset into 72 % training, 8 % validation, and 20 % test sets, and the training phase used images randomly cropped with the size of 120 x 120 x pixels as input to improve the stability of the model against the difference of facial positions, and validation and test phases used images center-cropped to the same size, following procedures by Cao et al. (2020).
Tasks We considered the three popular OR tasks: minimization of the task risk for the zero-one task loss (j.k) = 1j * k) (Task- Z). that for the absolute deviation task loss k, Ij k| (Task-A), and that for the squared lask loss (j. (j - k)² (Task-S).
Methods For the various-domain datasets, we applied a 1DT class based on 4-layer fully-connected neural network, in which every hidden layer has 100 nodes activated with the sigmoid function in addition to bias nodes. Also, for the face-age datasets, we applied a IDT class based on ResNet-34 (He et al., 2016), a modorn CNN architecture, following (Cao ct al., 2020)'s 's implementation. It modifics a fully-connected (the number of classes)-output final layer of the conventional ResNet-34 to a fully-connected 1-output layer.
As the surrogate loss function, we tried the AD loss Padi the IT loss (7) with p(u)= = min -- (Hinge- IT), log(1 + e" ( (Logistic-IT); the AT loss (8) with şo()) min - u} (Hinge-AT), log(1 + e')) (oggssiccATT); the NLL loss 6.l for the statistical model (10) with O'ngistic (OLR-NLL).
As the BPs class, we tried the non-ordered class and the ordered class for the IT losses; the ordered class for the AT and OLR-NLL losses. Note that the AD loss has no bias parameters.
As the labeling procedure, we tried the NNT and EOT labeling for the AD loss; the MT, ST, and EOT labelings for the Hinge-IT, Hinge-AT and Logistic-IT losses; the MT, ST, LB, and EOT labelings for the Logistic-AT and OLR-NLL losses. When using the ordered BPs class, the MT and ST labeling yield the same result (see Proposition 2). Also, for the Logistic-AT AT and OLR-NLL losses with the ordered BPs class, MT, ST, and LB labelings yield the same result under the Task-A (sce Theorem 3).
Cao et al. (2020) declare that their method, which is a combination of the Logistic-AT loss and ST labeling, is the of method for the face-age datasets in 2020. Although they used the non-ordered BPs class, bias parameters of the surrogate risk minimizer are guarantced to be ordered, and hence using the ordered BPs class would have made little difference to the result. Thus, we treated results for the method with the Logistic-AT loss, ordered BPs class, and ST labeling as their results, in the comparison.
Training and Evaluation During the validation and test phases, models are evaluated based on the mean zero-one error (MZE), the mean absolute deviation error (MAE), and the root of the mean squared error (RMSE), which are defined for a classifier f and m used data points as 1 Σ (f m 1 Σ lan (f(aa))) and ( Σm lxq ((((=)))))))1 for the Task-Z, Task-A, and Task-S, respectively. Here, the root operation of the RMSE is only for adjusting the scale of the error and does not affect our discussion related to the optimality of the EOT labeling.
We ran 50 trials for the various-domain datasets and 10 trials for the face-age datasets, with randomly-set different divisions of training, validation, and test sets and initial parameters of the network. In each trial, we trained the network using Adam of the learning rate 1.5 and mini-batch size 256 (or 16 when the training sample size is less than 256) as optimization procedure for 500 epochs when Noot S 2000 (i.e., for the various-domain datasets) or 100 epochs otherwise (i.e., for the face-age datasets). Additionally, for methods using the EOT labeling, we calculated the threshold parameters according to Algorit 1 at the end of every training epoch. The above errors were evaluated on the validation set at the end of every
Forr numerical stability (to avoid log(O)), we used an approximation the NLL loss in which the logarithmic function log(-) of dull replaced log(-+ 10-8) in the learning procedure.
11

---

Published in Transactions on Machine Learning Research (1/2023)
70
(a) AD
(b) Hinge-IT, non-ordered
(c) Hingo-IT, ordored
(d) Hinge-AT, ordered
(c) Logistic-IT non-ordered
(f) Logistic-IT, ordered
NNT, training
NNT. test
MT, training
MT. test
ST, training
ST, test
LB, training LB,
EOT, training
EOT, test
(8) Logistic-AT, ordered
(h) OLR-NLL, ordered
Figure 1: The learning curves of the training and test errors over 100 training epochs in a certain trial. These are for a special case for the AFAD dataset and the RMSE for the Task-S. In Figures (c), (d), and (f)- the ST and MT labelings yield the same curves.
training epoch, and then we adopted the model at the timing with the smallest error among the obtained validation error sequences as the test model.
We judge the significance on the classification performance of the labeling function by the ono-sided Wilcoxon rank sum test with p-value 0.05 based on errors for all the trials of methods using different labeling functions, in each combination of the dataset, error, surrogate loss function, and BPs class.
Results Figure 1 shows the learning curves of the training and test errors. The EOT labeling results in smaller training errors as its design to do, which appears to in turn result in smaller test errors. Also, we can find that the FOT labeling may stabilize (or smooth) the learning curve from Figures (c) and (f). Tables 2, 3, and show the mean and standard deviation of the errors, for the test model, evaluated on the test set. In many tried cases, the EOT labeling outperformed the NNT, MT, ST, and LB labelings regarding the test task risk. In particular, for the face-age datasets, modified 1DT-based methods using the EOT labeling provided better performance than the method (Logistic-AT, Ordered, ST) in Cao et al. (2020) that was the state-of-the-art in 2020. These results suggest the success of the EOT labeling in the subject (aiming for a better labeling procedure) of our research.
7 Conclusion and Future Prospect
The NNT, MT, and ST labelings and the LB labeling in typical usages are threshold labelings that may be sub-optimal depending on the learning result of the 1DT, task under consideration, and data distribution. In this study we propose change the labeling procedure of the existing 1DT-bbaed methods to the EOT labeling that applies the threshold parameters minimizing the empirical task risk for given 1DT, in order to obtain higher classification performance. Experiments in this paper showed the usefulness of this proposal.
We are also interested in the design of the learning procedure, especially the selection of the surrogale loss function, for the threshold method. One may be able to undertake systematic discussion on the goodness of the loss function by fixing components of the threshold method other than the loss function to the optimal ones. In such discussion, the EOT labeling will serve as the optimal other component. This is future prospect.
12

---

Published in Transactions on Machine Learning Research (1/2023)
Table 2: Mean (M) and standard deviation (S) of the test MZF for Task-7 and RW datasets in the form 'Ms'. The smaller the error, the better that mothod is for that datasct and that task. In cach block specified with the dataset, error, and learning procedure, we highlighted in bold font the best results that were tied
with each other and superior to all other results with respect to the one-sided Wilcoxon rank sum test with
a significance level of 0.05, if they cxist. Also, we colored the best results in red for cach combination of
dataset and error.
Learning Labeling|
COL
PAS
SQI
SQ2
BON
TAE
AUT
NEW
TOY
ESL
NNT EOT
364 207
345 126
360 1022
.365
.497 .633
.427 ost
.280.s82
.035.024
.022
.311048
AD
.380
333,134
.376.154
.353,192
.503.159
.432 089
.290.077
.037.024
.085,042
.295 .038
MT
364 177
.35 148
.395
.356.13
.512 154
.461 096
.291 095
.039 024
.080 o3s
.320 .088
non-ordered
Hinge-IT
EOT
ST
380 180
364 177
.35, 138
385
148
.377.17 137
395 161
.477.12 345.127
.356 1356.13
.512 154
.461.090
.411 087
.280.074
.290
.039.02
.034.02
.80 ons
.069 n411
.320 .048
.285,04s
Hinge-IT
MT ST
356 185
.355 107
420 167
.389 135
.537 166
.451 098
.293 .82
.033.z2 028 .035
.055.o 033
.303 042
ordered
EOT
364 182
.333
.384 1, 129
.376.14
.520 .199
.414 0077
.287
.064.022
.289
Hinge-AT AT
MT ST EOT
388 206
.367 159
.396 155
333.133
.503. 1646
.448 084
.288 091
032.o25
074..36
.303 p37
ordered
332
320 150
.396 158
.351,132
.485 .22
.419 0s9
.293
.034.027
.072
.288
Logistic-IT non-ordered
MT
364.186
.38 171
.442 154
.43. 143
.560. 1183
.415.093
.338 079
.03S 029
.071.aa
.313.000 .033
ST
.34 186
.398
171
.442
154
.433
143
.560
.415
.309 072
.038
.071 .03
.312 053
EOT
.352 194
.355 128
.391 143
.356,18
.513,142
.401,088
.297 Loss
.034.028
(068.034
.294,00
Logistic-T7 IT ordered
ST
364 .07
395 142
.444 152
418.141
.552.10 .19
.410.ss3
.37 .922
.044,22
.044.os
.27
EOT
.34
.345.10
.413, 140
,37
.528
128
.397 .778
.94
.032.o22
.061 o30
.292.037
Logistie-AT AT ordered
MT.ST
LB
.38 203
.300 230
152 .385
335 125
476 137
A45. 149
.47..10
A15,194
.522
.633 167
158
.420 081
.422 aso
.291 .291.097 .097
.881o78
.043.010
.000 oas
060 oas
.303.04 .443
.303 0422
FOT
308 197
.355.18
.389 130
.376.38
.520 143
.408 08c
.284
.034 026
.077
.988
OLR-NLL ordered
MT.ST
LB
.372.12 212
384 243
393 170
.30 129
.467 132
.455
144
.418
.44,140
.548 146
.577.18 .167
.422 081
.406 079
.881 074
.94.084
.043.036
.43..00
.063.om 034
.063 034
.299 042
.301 .045
FOT
.34
352 132
.393 148
.371,922 1192
.490.149
410,88
.291 .070
.031 .225
.058 028
.294_040
Learning Labeling
BAS
EUQ
LEV
ERA
SWD
WQR
CAR
|MRPPH|
CACD
|
AFAD
NNT EOT
080 025
385 044
.35 026
.754 027
.411 o28
.392 026
.014.o0g
.875.005
.927 002
.882 002
AD
.067.029
.388 044
.384.027
.733.002
.412,025 .22
.395.023
.013,008
.874,003
.925 .001
.878,002 ,002
non-ordered
Hinge-IT
EOT
MT
ST
.031 021
.027 .11
001. 021
012 .402
.385,00
.402 042
.387
.402 028
.402 028
.o2s
.732,030
.739
.739 034
034
.413 031
.416,025
413.0a.
.395 027
.395,020
.395 027
.008 .00
.009 005
.009 oos
882,003
.881 003
.882,033
.939.om .039
.939 002
.937,o00
.909
.902 003
.89 .002
BT ordered
ST
032 022
389 048
382 382
734.oas
.417 000
.409 023
.007 oo5
.82..o
.958 002
.905
EOT
025_p13
.375.0s5
.35.025
.732.030
.414.026
.399
.22
.007.005
.880.003
.949 ,002
.886 .005
Hinge-TT ordered
MT.ST EOT
055 036
.355 .477
.38 029
.750 027
.413 032
.396 027
.007
.873 005
.928 002
882 002
.031.016 .166
.386.47
.38
386
.728,029
414.027
-402,04
.007,005
.874.004
.926,001 000
.878
.002
MT
.044 020
.420 051
.390 030
.744.029
.414 1,6
.393 ,29
.017 .007
.884 oo0
.977 0011
.87 .004
Logistie-IT non-ordered
ST
.02 020
.20 a61
390 030
.744.029
.414.36
.394 021
.018 008
.885.003
.966.02 .006.002
895 0022
EOT
.015,014
.388 ooss
.387 .26
.734,033
.414.031
.384,00
.010 000
.83.0004
.966 002
.882 ,002
Logistic-TT ordered
MT.ST EOT
.024 023
.407 043
.385 .299
.725 031
.416 oss
.390 024
.011 .008
.833 004
.947 .022
.811 002
.016,01
.389 (850
386 0226
.729
.412
.389
.010
.800
9-16
.82
Logistic-AT ordered
MT.T
LB
.019.017
.018.012
.414.099
.399
386 029
383.0z 022
.71,,20
.758 026
.413.034
.418 p31
.397 022
.397.022
.012.008
.011 000
.844 o06
884.005
.300 o05
.299 004
.880 ,002
.83 002
EOT
.016.011
.383.46
.89.025
.732.030
.412,u2
.393,020
.011.008
.872.006
.999.004
.879 .003
OLR-NLL ordered
MT
LB
.017.012
.018.011
.395 045
.414.051
.383 .29
.30 025
.747 027
.725.0a5
.419 030
.411.00 .033
.392.221 021
.394 .25
.010,00 005
.010.005
.871 .005
.870,005
.926 003
.26
.876 .022
.878 .022
EOT
.015.012
.376,047
334.026
.733,82
.410
.394
.010
.871,04
.95,.022
.876 .008
Acknowledgments
This work was supported by Grant-in-Aid for JSPS Fellows, Number 20J23367.
References
Shivani Agarwal. Generalization bounds for some ordinal regression algorithms. In Algorithmic Learning Theory, pp. 7 21, 2008
Alan Agresti. Analysis of Ordinal Categorical Data, volume 656. John Wiley & Sons, 2010.
Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Slatistical Associalion, 101 (473):138-115) 2006.
13

---

Published in Transactions on Machine Learning Research (1/2023)
Table 3: A counterpart of Table 2 regarding MAE for Task- A and RW datasets.
Learning
Labeling
COL
PAS
SQ1
SQ2
BON
TAE
AUT
NEW
TOY
ESL
AD
NNT EOT
.352. 292.272
333.171
.382.154
176
.587 122 .550.117
.365 129.129
.035 024
.102 045
.321 048
512.272
338,142
.311166
.369.14
.535.170
.39,105
.037 .024
.085.022
.303,07
non-ordered
IT
EOT
MT
ST
.164.257
.000 266
500.268
.407 195 .345.153
.07..15
.188 187
.38,150
.418.187
.371.499 149
.360
.39
142
.880.19
189 .527
.580
.372.10
.601
.601 138
.391 3351
.406.08
.403 168
.039 .29.022
.034
.039 021
.09
.080.ss
080 oss
.294,03
.333 052
.333
Hinge-IT
MT.ST
.92. 269 .68.261
.400 164
.440 184
.409 151
.593 203
.582. 1882.137
.380 133
.033 028
.065.os7 .037
.320 048
ordered
EOT
.33,,14 146
.398.43
.387.152
.555, 103
.545.121
0.35
064,
.300,0s
Hingo-AT ordered
MT ST EOT
504 281
.372.
.407
.359 143 .364147
.558 187 .200
.567 119 .552,
.322 1330 .379,12
.032 025
074 033
.317 040
448,275
.338
.398 1108
.034 0334.02
.072
.034
.304,000
Logistic-ITT
MT
.420.21 216,27
.484.199
.456 156
.630 244
.544 153
.420 109
.039 029
.07 .033
.32 061
non-ordered
ST
516,274 .48,220
420.215
.84 192
.56
.277 238
544 508,119
384 .088
009 029
07
Lo33
320
EOT
.360,151
.409,105
.369,100
.575.195 195
.380,121
034.028
.068 .034
.303 .440
Logistic-TT
ST
5088 303
.403 117
484 188 .425,174
440 156
.600 192
.51 129
.300 125
003 025
006 038
.312 043
ordered
EOT
.6642290 ,20
.365,171
.385,152
.565_16
.519 126
389,130
.032 .222
.01 _039
.304,039
Logistio- AT MT.
.44. 2911
.410,198
.484 184
.436 144
.573 200 .563.90
.588 124
.309 110
043.o00
.060 o600oss
.314 314.044
ordered
EOT
.20
155
.407
389
.21,1114
.364.117
.034.026
057 .031
.307 .042
OLR-NLL ordered
MT.TT.L. ST..SLLB LB EOT
.417 210
.489 185
.440 161
.597 184
.519 119
.366 110
.043 036
.063 034
.310 043
.644,81
.352,149
.413,108
.384.10
.555.185
.509.1
.380
.03
.058 .2.8
.304 .037
Learning Labeling
BAS
EUQ
LEV
ERA
SWD
WQR
CAR
MORPH
CACD
AFAD
NNT
.109 439
.424.088
.413.a1
1.236 o69 1.231
.429 035
.430.z2
.014.0o9
2.931 022
5.243 022
3.352 024
AD
EOT
.080,os3
.423.054
.413.000
.428
128.031
.432,028
.013,00
2.918
.33
5.238 ,229
3.3:19,o20
non-ordered
Hinge-IT
EOT
MT
ST
.033 .223
.035 026
.035
.424,061
.453
.43
.412,029
.45.
oas .45
1.232 .08
1.243 .73
1.243 073
.432 0334
.431 .0.3
.32
429 .311
.488 031
128
.008 .005
.009 .05
009 n05
3.099.02
3.183
3.173 .177
022
5.742.005 .205
5.939
5.937 213
3.7 ,27
3.979 n63
4. 4.029
Ilinge-IT ardered
MT.TT
.035 024
.431 058
.414 029 413,030
1.245 074
.437 037
.446,27
.007 005
3.349 040
8.052
279
4.694, 116
FOT
0229.019
.17,,55
1.229,075
430,,27
437.027
.007 .005
3.204
.032
6.58 .067
3.761
1107
Hinge-AT ordered
ST
.066.088
.435 059
.417 031 413
1.222 .02
.433 038
.433.031
.007 .004
2.881
033
030 5.264
3.582 080 3.390.o1s
EOT
.043.mm
.422.
1.226
063
.431 .0.4
435.02a
.007 .005
2.863 022
5.242
.344
Logistic-IT non-ordered
MT
.026.024
.463.007
.425.36
036
1.262.cm 075
.438 037
.300 o26
.017 .077
3.189 049
5.758 101
3. 3.796 oso
ST
.26
.4633
.425 .455.06
073 1.262
438
.429
.018 009
3. 388 049
103 5.757
3. 732
EOT
.017.01
.425 .64
.114
1.228.a7m
.466 034
.21 Lo26
.010,006 0,006
3.124 .033
5.631 .nna
3.538 .050
Logistie-ITT
ST EOT
.27
.45
.416 030
076 1.226
1.239
.440 033
.426 026
.01 .006
3. 3,742 n84
6.685 131
4.082
ordered
0199,066
.432,062
.113,066
069
.436,031
.428,027
.010 .006
3.540 .038
6.359
3.619.022
Logistic AT MT.ST.LD
017
.409 100
.422 029
1.225 068
.33 032
429 027
.010.0.
2.874 037
5.381 138
3.375 .33
ordered
FOT
.020.014
.438.052
.31
1,233.063
.437 034
.431
.023
.01 .o06
2.856 m33
5.342 128
3.367,025 .025
OLB-NI, ordered
MT..BB.LL EOT
.018.015
.407 .Gs
.416.00
1. 1.230 009
.444 .35
432
.01 1 .00G
2.873
.43
5.243 .044
3.345..00
.019,015
.33.057
.417 029
1.234.067 067
.436 035
.27. 022
0100.00
2.845 e37
5.228. 041
3.343.017
Paul-Christian Bürkner and Matti Vuorre. Ordinal regression models in psychology: A tutorial. Advances in Methods and Proctices in Psychological Science, 2(1):77 101, 2019.
Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters, 140:325 331, 2020.
Bor-Chun Chen, Chu-Song Chen, and Winston H Hsu. Cross-age reference coding for age-invariant face recognition and retrieval. In Proceedings of the Europeon Conference on Computer Vision, pp. 768-783, 2014.
Wei Chu and Zoubin Ghahramani. Gaussian processes for ordinal regression. Journal of Machine Learning Research, 6(Jul):1019 1094,,,009-100 2005.
Wei Chu and S Sathiya Keerthi. New approaches to support vector ordinal regression. In Proceedings of the International Conference on Machine Learning, pp. 145-152, 2005.
Wei Chu and S Sathiya Keerthi. Support vector ordinal regression. Neural Computation, 19(3):792 815, 2007.
Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20 (1):37-46, 1960.
14

---

Published in Transactions on Machine Learning Research (1/2023)
Table 4: A counterpart of Table 2 regarding RMSE for Task-S and RW datasets.
Learning
Labeling|
COL
PAS
SQ1
SQ2
BON
TAE
AUT
NEW
TOY
ESL.
NNT EOT
.779
.593 144 .676.161
.613 193
.630.145
.821 194
.883 116 .828,109
.730 164 (748.19
.167 oss .173.084
.310
.580 .056
AD
827 324
.633,667
.617.14
1797.77
.281,075
.574,045
non-ordered
Hinge-IT
EOT
MT
ST
.760 326
799,.19 s19
799.319
.584
.652 191
.652 194
.614 109
.676
.676 190
190
.601 .188
.019.71
.609.71
.793.21
.866,21
866 214
.857,110
.884 121
.884.122
.753,
.765 200
.667.19
.179.084
.160,091
179
.275.008
.275 068
.258,074
.604,0or
.570 o0a6
.604 067
Hinge-IT
MT.
.789 831 763
158 633
.644. 156
.649 156
.837 2044
.885 140
.745 164
154.095
.243. ors
.591
ordered
EθT
571
.642,
.688 186
.791 .17
.848 105
.736 151
.159.10
.422 .711
.570.033 0311
Hingo-AT ordered
MT ST
786 307
596 194
646 172
6288 .144 .601,166
.817.199
866 116
.741.17 171
.15..og 092
.24..066 006
.598 032
EOT
.752,43
569
209
.38
165
757.160
.834,091
.750,158
.15T .085
.20
.569 ,44
Logistic-IT non-ordered
MT
812 330
656
222
.74
193
.65 .142
.87 242
.846 132
.789 186
.175 094 .175.00
.25 064
.580 .655
ST
.812.300
656
222
193 .734
.655 142
.870 233
.846 132
.774.192
.288 064
.586.00
EOT
.754.345
.597
168
.650.77
.608.
.806.19 194
.807 100
.745,146
.161 008
.252 068
566.p49
Logistic-TT IT ordered
MT ST
.700 350
644 182
.733 188
680 144
842 18G
.837 124
.753 159
.159 0091
.239 .082
.583 035
EOT
.779,342
.611
199
.633.77
.624.14
.793,160
.809.110
.742,164
.160,082
.230,081 091
.572,047
Logistic-AT ordered
MT.ST
LB
.757 361
7722 368
66555555 202
.614 187
.29..88
.655 176
.80
.699 144
134 .623,
.855199 197
.797 205
.812 112
.85..11 117
.726, 153
.736 150
.182 098
.182 094
.234,000
.230 n82
.582.049 0449
.585 .046
EOT
.732.346
601,11
1,6
.802 199
.813
.720
.162 o8G
.224 p80
.565 .047
OLR-NIL ordered
ST
LB
.755 347
795
634 158
.651
221
.766 177
7735
182
.640 153
6800146
.830 168
8800 180
.833 111
.821 118
.736.15 153
.741 157
.184 .097
.166.00 .96
.210.073
.239
.578
.582
.049
EOT
.754 951
.587 161
.655,174
.620.159
.787 188
.812 107
.751, 156
153.089
.231 .66
.569 .48
Learning
Labeling
BAS
EUQ
LEV
ERA
SWD
WQR
CAR
|ORPH|
CACD
AFAD
AD
NNT EOT
.341.00
3991 075
.719,067
.712. 7712.062
.608 006
.677 036
1.589.077 .077
1.623.cov
.676 .330
.684 .035
.710,a30 .721.033
.108.044
.113,00z
3.962 000
3.942 039
7.395 .227
7.410 033
4.571 a,22
4.534.027
non-ordered
Hinge-IT
FOT
MT
ST
.187 097
.187 097
190.per
.711
7788
.748 ons
.58
.711 .039
.695.aa5
7711.099
1.586.072
1.657 .92
1.657.cov 092
.681
.689 .036
,331
036
.708 .28
.712.030
.708 028
.081
.088
.088 033
033
4.137 .843
4.273.02 0223
4.272 024
7.721.060
8,009 277
8.003 276
4.927 .024
5.301 072
5.435, .ss
Hinge-IT ordered
MT.ST
188 oss
777 .008
.67 033
1.644 o83
.692 .37
.722 025
.078 036
4.529 041
276 10,098
6.374
EOT
180 p092
.713.000
.93.096
1.588,e0 .r5
.679,um .229
.716.029
.077.036
4.288 .033
8.475 .000
4.958 ,116
Hinge-AT ordered
MT.ST EOT
277 118
726.072
.988 036
1.607 .oa6
665 .34
.707 .027
.079 .33
3.874 046 3.867
7.443 044
4.902 130
160
.714.071
.933034
1.591 .077
076.027
.711,025
.077.06
.331
7.442 048
4.550.124
Logistic-IT non-ordered
MT
.151 103
.55
706
1.655.oem
696 .035.035
.713.
.133..m
4.257 .70
7.870 109
5.151 .078
ST
151.103
.756 074
706..41 041
1. [.655 088
.966 035
.710.09
.135 036
4.255 068
7.860 110
5.058 065
EOT
.119 101
.708 .062
.694 .034
1.585
.684 0300
.702
.094.0x
4.172
7.620
4.706
Logistic-IT ordered
MT ST
.148 112
.748 062
.688 037
1.644 090
.694.oo
.033
.707 .29
.102.032
5.016 100
8.902 182
5.720 .622
FOT
.134 096
.708
.693
.33
1.592.om
.684
.27
.709
.095
4.726
8.320
4.798
Logistie-AT ordered
MT.ST
LB
.137
.140 p6s
071
.731 .088
.706,ee0
.96 035
.696
1.585.o0 .077
1.604.08)
.689
686 031
.04
.710
.708 027
.102 033
.102 .03
3.859 043
3.858 .433
7.520 696
7.497 .880
4.517,040
4.580
EOT
130
.693.0o7
.694.036
1.587 Loso
.682
032
.713,029
.099.037
3.841 .34
7.494.033
4.517 7
OLR-NLL ordered
MT.ST
LB
.129 071
.144 .04.
706
721
.666 .022
.697
1.616 .085
1.592.oa .081
.690 034
.683.cm
.710 o29
.710.02s
.099.02 025
.100.025
3.870.og o3r
3.869 .336
7.437 061
7.423 060
4.574
4.508 .220
EOT
.128 091
699 889
.693
1.588
.682 026
.708
.096
3.852 044
7.417 o5e
4.510.ma 022
Jacob Cohen. Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213, 1968.
Joaquim F Pinto da Costa, Hugo Alonso, and Jaime S Cardoso. The unimodal model for the classification of ordinal data. Neural Networks, 21(1):78 91, 2008
Stephen E Fienberg and William M Mason. Identification and estimation of aee-period-cohort models in the analysis of discrete archival data. Sociological Methodology, 10:1-67, 1979.
Eibe Frank and Mark Hall. A simple approach to ordinal classification. In Proceedings of the European Conference on Machine Learning, pp. 145- 2001.
Philip Hans Franses and Richard Paap. Quantitative Models in Marketing Research. Cambridge University Press, 2001.
Pedro Antonio Gutierrez, Maria Perez-Ortiz, Javier Sanchez-Monedero, Francisco Fernandez-Navarro, and Cesar Hervas-Martine Ordinal regression methods: survey and experimental study. IEEE Transactions on Knowledge and Data Engineering, 28(1):127-146, 2015.
15

---

Published in Transactions on Machine Learning Research (1/2023)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Paltern. Recognilion, pp. 770-778, 2016.
R Herbrich, T Graepel, and K Obermayer. Support vector learning for ordinal regression. In International Conference on Artificial Neural Networks, volume 1, pp. 97-102, 1999.
Stefan Kramer, Gerhard Widmer, Bernhard Pfahringer, and Michael De Groeve. Prodiction of ordinal classes using regression trees. Fundamento Informaticae, 47(1-2): 2001.
Ling Li and Hsuan-Tien Lin. Ordinal regression by extended binary classification. In Advances in Neural 2007.
Information. Processing Systems, pp.
Hsuan-Tiee Tien Lin and Ling Li. Large-margin thresholded ensembles for ordinal regression: Theory and practice. Algorithmic Learning Theory, pp. 119-333. Springer, 2006.
Tisuan-Tien Lin and Ling Li. Reduction from cost-sensitive ordinal ranking to weighted binary classification. Neural Computation, 24(5):1329-133 2012.
Tic- Hie-Yan Liu. Learning to Rank for Information Retricval Springer Science & Business Media, 2011.
Peter McCullagh. Regression models for ordinal data. Journal of the Royal Statistical Society: Series B (Methodological), 42(2):109 127, 1980.
Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output cnn for age estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4920 4928, 2016.
Fabian Podregosa, Francis Bach, and Alexandre Gramfort. On the consistency of ordinal regrossion methods. Journal of Machine Learning Research, 18(Jan) 1803, 2017.
Sebastian Raschka. Mixtend: Providing machine learning and data science utilities and extensions to python's scientific computing stack. Journal of Open Source Software, 3(24):638, 2018.
Karl Ricanek and Tamirat Tesafaye. Morph: A longitudinal image database normal adult age-progression. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, pp. 341- 345, 2006.
Christos Sagonas, Epameinondas Antonakos, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic. 300 faces in-the-wild challenge: Database and results. Image and Vision Computing, 47:3-18, 18, 2016.
Amnon Shashua and Anat Levin. Ranking with large margin principle: Two approaches. In Advances in Newral Information Processing Systems, pp. 961-968, 2003.
WA Thompson Jr. On the treatment of grouped observations in life studies. Biometrics, pp. 463-470, 1977.
Richard Williams. Generalized ordered logit/ logit/partial partial proportional odds models for ordinal dependent vari- ables. The Stata Journal, 6(1):58-888 82, 2006.
Ryoya Yamasaki. Unimodal likelihood models for ordinal data. Transactions on Machine Learning Research, 2022. URL
Shipeng Yu, Kai Yu, Volker Tresp, and Hans-Peter Kriegel. Collaborative ordinal regression. Proceedings of the International Conference on Machine Learning, pp. 1089 1096, 2006.
16

---

Published in Transactions on Machine Learning Research (1/2023)
A Proof of Consistency of Statistical Methods
We here give proof of the theorem on the interpretation of the learning procedure for statistical methods.
Proof of Theorem 2. We can characterize the surrogate risk minimization for the NLL loss as maximum likelihood estimation for the statistical model (10) for multi-class classification problem through the equation
K
min (10mn(((X),b,);00))) a6
min ae7,beB
P
y=1
(15)
min aEALeeB
B
(ogyl))oogPP Pooy:)))oggP(xxooo.a)....))
According the method of Lagrange multiplier, one solution of point-wisc (at cach X x) minimization problem
K
min
Pr(y log Pr(y)æ), subject to Σ Pr(ylz)=1 =1
(16)
(Pr(k (m))
is Pr(y]az) = Pr(y)a) = o(y;or,ã())) b), y= I,...,, where the existence such {ā(a:), b) is assumed in the statement of the theorem. This solution applies for any x E Rd, and one can see that a solution of (15) is {ā, b), which completes the proof of the statement for the NLL loss.
Also, for the ANT.CL loss, can provide the following characterization:
(())) 6.
Pr(y X
log ((k:o,,((X))0) Σ log (())),
(17)
K-1
Σ
E Pr(Y 2(1x Pr(Y s y/X)) log -(0
where Q(y;o,,(x)) b) Σx P(k; or a(x), b) and the expectation value E[-] is taken for X. On the ground of the binary version, 'y less' vs. 'more than y' = l..... K 1), of (16), one can prove the statement similarly.
One may consider the IT loss (7) with p(u) lo), which we call immediate negalive log cu- mulative likelihoods (INLCL) loss function. However, it is difficult to characterize the surrogate risk minimization with the INTCL loss as a problem with a known solution unlike those for the NLL and ANLCI. losses, and the optimality condition for the INLCL loss is unknown.
Proof of Relationships between Labeling Functions
This section provides proofs Theorems and 4 regarding the relationships between the LB and threshold labelings. Propositions and would be trivial, so we omit proofs of them.
First, we prove Theorem 3.
Proof of Theorem 3. We introduce the functions
K-1
(u) = Σ (o u) ( -)) = ((j,))+ Σ o -0)(())))-((A... for j 1,...,, K.
k=1
(18)
17

---

Published in Transactions on Machine Learning Research (1/2023)
with bo -oo and bx t00, where the equation holds, since o(-20) The classifier based on the LB labeling, f(x) migminjelln Σ P(k: ook;or,āee, (,)), b) is equal to arg j Ry(a(z)) According to Proposition 1, the LB labeling is a certain threshold labeling if and only if arg ( condition holds if the situation
and o 1.
S (K for any 41 E R such that u1 $ u2. The latter
(u) > (u) for u E ($1, and Rx (4) < (u(4) for u E (52, s3) with k < 1, $1 < $2 < 53
(19)
does not occur. In the following we assume I for the indices k, E [K].
Proof of (i). Under the assumption described in the statement of the theorem, the difference
R₂(1) - (₁(1) =
Σ
o(b,-u) -0)--000000)00-00/-------------- -C,)
(20)
non-nrgative constant
nou-negative non-increeasing
non-positive constant
is non-decreasing with respect to u. Thus, (u) s Ri(u) for S p and (x(n) 2 Ru( for 2 for some point p, (u) s (i( for any u, or (u) 2 (u(u) for any u, which implics that the above-mentioned situation (19) does not occur. Note that, for the instances = Cad. leqs one has that
-2. 1(k s jsl-1) 2(k - l),
for l = lad. for l = lzq.
((.))
(21)
This completes the proof of the statement (i).
Proof of (ii). For & E2o,e with c e [o, [K/2]) where lym = E20,0; the function Ry(u) reduces to Ry(1) 1 (o(b, -u) o(aaaaa1)
(22)
with aj == b max and by == bmin(jtc.K) where aj by. Lemma 1 (described after the proof of Theorem 3) shows the shape of the function Ry(u): Under the assumption of Theorem 3 (ii), Rý(u) is minimized at (aj + b2)/2 == cj. symmetric in around = cjz non-increasing in for < cj. and non-decreasing in when cj2 from Lemma (i) and (ii). Also, assuming that ej is fixed, then Ry(n) is non-decreasing in bj 10--a aj, from Lemma 1 (iii).
When by - 0g = bj a,, the translated two curves (u) and Ru( have just one intersection point at u == (cx + c2) and it holds that Re(u)= s Rt(u) for u S (ck + c1)/2 and (x(u) 2 Ri(u) for u 2 (C* + c2/2. Therefore, the situation (19) does not occur if by ag = bb aj.
Then, assume by ag bi aj (the following proof strategy for this setting can be applied to the other setting by ax bj 11-aj ar). In this setting, R+(u)= (u(u) for 2 due to the shape of the functions Re and R,. Also, within [c), they can have one intersection point p at most such that ( < R₁(4) for E [ck, p] and (u) 2 (u) for u E [p.cı], since Rx(n) and (i(u) are respectively nen-decregiing and non-incrcasing in u. Therefore, the situation (19) can be satisfied only in such a situation that there exists a point p satisfying
(x(p) R((p), R((p) Rj(,), and p Ck.
(23)
The existence of such a point p implics that
a'(a - p) o'b P))
- (obip) with a S al, by bb, as bu. a S bj. p S ck. (24)
o(ax - p) o obbx - p)
However, the assumption that ((())-o'((2) is non-increasing in v, with fixed v2 and in V2 with fixed v1 when (v)))oo(v))
Vi < V2 shows that
a'(a p) o' - p)
0'(a - o' o'(b- o'(a- - o o(ax - p) - o(bi - p) o(ai - p) o(bi p)
2
2
(25)
o(ax p) - o(b; pp)
18

---

Published in Transactions on Machine Learning Research (1/2023)
which contradicts to equation (24). Therefore, the situation (19) does not occur also when by ak bs a1. Note that, especially when Nogistie: one can show that
(vv))- (((2) ('1) 0' (V2) Mogistic (v1) Ologistic (v1)) (1)) logistic (v2) (Togissic (12))
Nogistic (v1) Nogistic (V2)
(26)
=1 =1- -ooossitie + Nogistic (12)).
is docreasing in V1 with fixed V2 and in V2 with fixod V1. Morcover, when o o gaauss one has that
o'(v1) - o' o Ugauus (v1) Ugauss (12) == fi(v1,12), ((v1) (v2)
e --
(27)
that the derivative of fi fi(1.112 12) with respect to V1,
-vie 12/ gauss (v1) o gauss (v2) - e-9122
f(11
v2
(28)
dv1
{o (v1) ggauss (v2))² )2
has the same sign as
=-i P2( == -vi gagass (v1) (ggans((22) (P2))
1
-1/12
12/2
(29)
V2T
and that the derivative of f2(v1. v2) with respect to V2 is
fe(vi. v2) = (v1
(11 - v2)
1///2
(30)
8v2
V2T
Since a Tuy f2(11, < 0 when 12 and f2 v1) = 0, it holds that f2 v2), which has the same sign (((())-0((11) de ((v))-o(())) is negative when V1 V2, that is, ((())))0(22) is decrcasing in with fixed v2 when < V25 monolonicity in v2 with fixed v1 can be proved by the same discussion.
a (v)))
Proof of (iii). Regarding the MT and ST labelings, let y == hthr (u: b) under the assumption bi ... S bx-1, which implics that b1 S .... S by-1 S u I by I ..... S bk-1- Regarding the LB labeling for the likelihood model (10), one has that, with the abbreviations o% o(bb- u) for k I,...,K,
K Σ k=1
R2(u)= (ox - -
= -
OK-1}
K-1
(31)
Σ
= =-j- - i| σo +
ok
ok +11 - Kl ok
k=j
1
Σ o obb k=1
Σ (1 obb u)}.
for every j E [K]. Simple calculations show that obb - u) $ 0.5 for k= -=,,...,y- and (1 (1-o(bx- - -4) s 0.5 for = y....... 1, from b, .... $ by-1 S u S by S .... s bx-1 and the assumption on the shape ar. One would see that objective function (31) is minimized at j = because some summands are replaced by ones of 0.5 or more if j deviates from y, which concludes the proof.
The following is an auxiliary lemma for the above-described proof of Theorem 3. Lemma Suppose that a decresscg and satisfies o--co) and (++00) Define S(u; a, b) orb u) (aaa- u) for b. Then, has that
19

---

Published in Transactions on Machine Learning Research (1/2023)
(i) S(u; with fixed ond b is symmetric in around u ath differentiable and o' is even.
2 of 1 o(), if a is
(ii) S(u; a, b) with ftxed and b is mazimized with respect to al ath 2, non-decreasing in for u 2, and non-increasing in u for u atb if o is differentiable and o'(u) is even and non increasing in if u >
(iii)
S(u; a. b) with fized u and wth is increasing with respect to (b a).
2
Proof of Lemma 1. Proof of (i). The assumptions that (-(o)) 0, (++00) = 1, and o' even imply that
((-.)==1 =1)((). On the basis of this result, one then has that
a+b b S 5(u+
a+ b
a+
a b
b
11 +
a
11 +
2
2
2
(32)
-
b
-
b h-
-1+0
+
2
2
2
2
which implies that
a a+b
a+b b
u S
a. ba
= =S
11
;a,b.
(33)
2
2
Proof of (ii). The above equation (32) shows that
ö
a b
b 0
b -
uu
2 aa
b o¹
=0,
at 0.
(34)
du
2
2
Also, one can show that
ô
a+b
b
b - a
- +
;a b
+
0
Ou
2
(35)
o' b-u ul) 0' (bme u)
2 0,
for < 0,
2
b 2 u) - o (1 u|) s 0, for u 0.
Here, for u < 0, used the fact that of is even, which implies that o' عج + 01 + +ul), and o'()) is non-inereasing for > 0 and b= u 1 u| > 0; for и > 0, we used the fact that of even, which implies that r'( hhe u) = o' d' - u)), and o'(v) (v) is non-increasing in v for v > 0 and b-a + u > 1 -ul 0.
Proof of (iii.. With change of variables == b-a 2 v a+b we introduce function
T(r: = -(uvv (v - -
(36)
For this function, one has that
u- 0.
(37)
since is non-decreasing (i.e., ('))) 2 0 for any u).
Next, we give proof of Theorem
Proof of Theorem 4. If < 1, the convexity shows that
[k [] (k (k-
(1- - (k-j) ((I.)
((k. <
((k, j +1)+++++++++++
(1 (1- j} - (k (j+1)}
[1-j} - (k (j +1)
(38)
1
1-k
1-k+1 [(k,) + 1) - 1-k+1* ((..).
20

---

Published in Transactions on Machine Learning Research (1/2023)
and that
(1,j+1) (1-()tt)((k-(+1 [k.j+ +)+ --])-(1-(++) ((.))
(1- - (k - (j 1)}
(k-
(39)
1-k
1- [k.j+ + 1-k+1 ((1,)).
These inequalities imply that &.11 is non-positive:
(x,(())
(((k.)) ((,]++1)) (+
1-k -(((k,))+ + ((1,++1)) 11 1- k+1 (1) 1- 1-k+1 e(1,)) +
1-k
1
[(k,) [t.]+]]] +)+
1 1 k ((I,))
11-+1
= ((CK.)) 1- [(k, 1)+ + 1- 1 ((1.1)
1
1-k
[(1,
- 1+ [(k.j( + k ((1.))
S..
(40)
Similarly, one can show that lx.1 is non-negative if > I.
0
McCullagh (1980, Section 6.1) has proposed the heteroscodastic extension of the emulative link model (10).
P2(); b, s(x)) P(y;o, Ply:o,.dda)///x)bb//.a
(41)
with the scale model s Rd oo), and statistical OR studies, Thompson Jr (1977); Fienberg & Mason (1979) and Agresti (2010, Section 4.2), have also considered another model
Ps(y; = o(b, - a(x))
a ((1-o(b--a(()))
[
(42)
k=
We obtain the following theorem that is similar to Theorem 3 and suggests the efficiency of the EOT labeling for statistical methods adopting these other likelihood models:
Theorem 5. Suppose that is non-decreasing and satisfies (--o0) and (++00)) and that Rdd R, b e K-1 and 5 :Rd c0,c
(i) arg migminjelk Σk_ b()) (anr((();;))) if E lad, (00) = 0.5, and bi s ...
bk-1-
(ii) arg argminje(k] M
Pa(k; a(w), ()((j, k) (atx(((();;)) for some t R-1 if Ead.
Proof of Theorem 5. Proof of (i). The statement (i) of Theorem is trivial from the statement (iii) of Theorem 3.
Proof of (ii). Regarding the LB labeling for the likelihood model (42), one has that, with the abbreviations o == 1-bbbb a(x)) for k = I,
(a(ā())) = Σ
o (((x), b)a)).))
k=l
Σ
(1 )r:)
07-1 1/ kk,
R=1
1=1
(43)
= lj 11(1 δ1) + jj 2/0 -2) -0-11-0-111-- rjj11)
+ + 0r (1+11) + +1j K+ 11
0k xx + lj Klōn ..
1-1 k
K-1 k
-)-1)
Σ
- o (b) - -(x))) +
Σ
-al--abbe (bb - -d(x)))
1=1
k=j
21

---

Published in Transactions on Machine Learning Research (1/2023)
for every j E [K]. One has that
R)+1(a(z)) ( = =1 [ -b)))())((()),, a(z)))
(44)
1-1
is non-decreasing in with fixed a(x). Therefore, arg argminje(k Σk=1 P3(k; or a(x), (j, is the first index such that R11 (ā())) - (((ā())) & O, or K if R/+1 (a(x)) R((ā(z)) > 0 for all 1= 1,..., K 1. Also, R1+1(c (a(x)) (((((x)) is non-increasing in a(x), for each / = I....,
These facts show that arg [ 2k=1 P3(k; o, a(x), (a hthr (ā());; t) with the threshold parameters lx, I,...,,.
satisfying Rk+1(1R) (x(+)) =0
C Optimality Guarantee of Algorithm for Empirical Optimal Threshold Labeling
Lin & Li (2006) do not describe the optimality guarantee of Algorithm 1 in their paper. As a supplement to their development, we write here the optimality guarantee of Algorithm 1.
Theorem 6. For any task loss c: [K] - [0, oo), 1DT a, and training data D, = (ar;,;)) the threshold parameters & oblained by Algorithm minimize the empirical task risk for a classifier based on the threshold labeling: E arg min Σn (a(a))((()));) t).
Proof of Theorem 6. First, prove 'statement( (f)' that, for each k E [K], Lj, is the minimum task risk for task such that IDTs (ā(œ;) a(x;) = ái)...., {a(x;) a(æ;) aj-1} are predicted any of 1...., in a non-dccrcasing manner, and 1DTs (ā(x;) ((x;)) = ā,} are predicted as k:
Ljkk
min beE s- LE[] Yoo EM
Σ
Σ
Fm)
(45)
The statement(1), which is the starting point for mathematical induction, is trivial. Also, according to the equation,
L,418 = min L y
Σ y(k,y))
[e[k]
min [e[k]
min hy....heel Ell
Σ
Σ
((h.
Σ e(k.3i) M
s.t. hy shy [E[T] you E.
(46)
min
Σ
Σ
ym) with 1' arg min [c[R]
min hy..... ell s.. s.. shy-s /eliil Yim 34
Σ
Σ
E(hi (i))))))
In hasekk s.b. by S... shy=t" skya1=k |elj+1] Now c.
min RmE St Shy [E[i+1] Em
Σ
Σ
E(hi.mm)
one can find that the statement (f + 1) holds with j 2 as well.
The statement(N) shows that 1DTs (ā(x;) a(xx)) áv) } should be labeled as min minjak) LN.i) M. Also, for
(h....,,Nx)
arg min hy e[M s.t. kt. S-yy
Σ
Σ
E(hry
(47)
1E[N] Vm E./
it will also be clear that 1DTs (ā(x;) a(x;) a.)...., (((..,,((((;))) ) a(x,)) ĀN-13 should be labeled as bi...., ., hN-1- The index or J in Lines 9-15 tracks hv (= M). hN-1..... h1. Therefore, can be found that the obtained threshold parameters optimal.
22

---

Published in Transactions on Machine Learning Research (1/2023)
Table 5: Dataset properties, the total sample size Ntot, and the dimension of the explanatory variables,
of classos of the target variable, of the benchmark datasets used for the experiments in Appendix D. Note that AMP originally has 6 missing values and excluded them.
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
Boo
74
159
167
186
194
209
392
506
950
d
27
15
4
60
32
6
13
9
ABA
A12
KRA
CO1
PU1
BA1
CO2
PU2
BA2
EL2
4177
7129
8192
8192
8192
8912
8912
8192
8192
9517
5
21
32
POT
AI1
EL1
CAL
CE1
CE2
2DP
FRA
MVA
Buot
13750
15000 48
16599
20640 8
22784 8
22784 16
40768 10
40768 10
40768
d
40
18
10
D Additional Experiments with Benchmark Datasets
Purpose Many OR studies use datascts, which are generated by discrctizing a real-valued target of bench- mark datasets commonly used for evaluation in a regression task, in their experiments. For example, Frank& Hall (2001) applied regression benchmark datasets summarized in https://www.cn www.fdc..f..com up.pt/ html, and experimented with datasets generated via the discrctization into 3/5/10 equal-frequency bins. Also, Chu & Ghahramani (2005) tried the discretization into 5/ 10 equal-length bins in addition to the 5/10 equal-frequenc discretization. Therefore, performed similar experiments additionally to enforce our claim, and this section describes results of these experiments.
Settings In the experiments, regression benchmark datasets, use 29 datasets, DIA (Diabetes), PYR (Pyrimidines), APR (Auto Price), SER (Servo), TRI (Triazines), WBC (Wisconsin Breast Can- cer), CPU (Machine CPU), AMP (Auto MPG), BOS (Boston Housing), STO (Stocks Domain), ABA (Abalone), AT2 (Delta Ailerons), KRA (Kinematics of Robot. Arm), CO1 (Computer Activity (1)), PU (Pumadyn Domain (1)), BA1 (Bank Domain (1)), CO2 (Computer Activity (2)), PU2 (Pumadyn Do- main (2)), BA2 (Bank Domain (2)), EL2 (Delta Elevators), POT (Pole Telecomm), All (Ailerons), EL1 (Elevators), CAL (California Housing), CE1 (Census (1)), CE2 (Census (2)), 2DP (2D Planes), FRA (Friedman Artificial), MVA (MV Artificial), which are obtainable in https: http://wwwwc.ee dcc. be/:Ltoorg//eegrrssiio///aaaass html, a researchers' site of Chu & Ghahramani (2005) (http://www. /
html), our GitHub reposilory (https://github com/ yamasakiryoya/01L)) see also Table 5. As the discretization manner, we tried e/wa///uualllreeeegeeeeeeeeeeaee frquuenrr /length discrctization; we denote these generated datasots, for example, as EF3 and EL10 datascts. We adoptod the same neural network model applied for the RW datasets. All other settings are the same as those in Section 6.
Results Tables 6 23 show the mean and standard deviation of the test errors. Table 24 summarizes all the results: the column 'SUM' shows that the EOT labeling was superior to existing other labelings for all learning methods and tasks, and reinforces our claim regarding the usefulness of the EOT labeling.
23

---

Published in Transactions on Machine Learning Research (1/2023)
Table 6: A counterpart of Table 2 regarding MZE for Task-Z Z and EF3 datasets.
Learning
Labeling|
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
|
STO
NNT EOT
558 163
.388 119
.150.os .145,065
206.076
5533 090
.596 082
.261 062
.204
.200
.079.om
AD
161
.372 129
.198,700
.518,00
.589,000
.260,062
.185,041
.248.008
.080.022
Hinge-IT
MT
534
.388 107
.159
.188
.501.ec
.580.081
.264.663
.206,048
.245.03
.079.018
non-ordered
EOT
ST
568 135
534 161
.388
.379 110
107
.154,os
.159.060 .660
.198 070
.208 0087
.488 oos
.501 089
.579 .077
.580 081
.254
.264.063
.184,0m
.206 o1s
.255 038
.247 o08
.075.015
.079 018
Hinge-IT
MT.ST
.56 153
.391 102
152.052
.211.00 1770
.51 .881
.583 .900
.256 065
-197,013
.24.009
.079.017
ordered
EOT
544. 127
365
11.6
.148 .022
.199,081
.510 083
.575 082
.256 060
.100,037
.250
.066.019
Hingo-TT ordered
MT ST
.822 155
.38 100
.152 ono
.207 ooss
.49 .093
.595.0s os6
.252.633
.201.aa5
.23..a7 oay
.077.017
EOT
554
363
108
.147 ,053
.212,085
.505 0000
.584 ,999
.265,062
.193,010
.245,03
.077 .200
Logistic-IT non-ordered
MT
.58 167
384 114
.158 668
.277 .600
.483 092
.587 .91
242 063
.196
.411 036
.077
ST
5538
167
114 384
158 ons
.27.0080
.483
.587 .944
2412 .63
.196
241.036 o36
.077.018
FOT
.538.71
385,119
044
.215.047
.481,079 .099
.568 088
.247
.178.009
.234.037
.222
Logistic-TT IT ordered
MT.ST ST EOT
536 156
400 000.099
.153.
.229
.475 1715
.571 0922
.255.070
.195,044
.232 038
.073.012
.580 158
.37
.90
.134,,04
.219,060
.486 082
.559 082
.252,065
.183,033
.229
040
.074 .019.019
Logistic-AT ordered
MT.ST
5-22 148
397 106
144 063
.221.771
.487 .79
.584 085
.249
.191 .191.44 o45
.20 032
.074 0118
1,14
.580
159
.35 104
.148 064
.218 .84
.491 0811
.575
o8s
2445.63
.192.0s
.221
034
.075. .19
EOT
536
167
385
115
136
.215.063
.503
082
.580 o8s
.252 065
.181 038
.277
040
.076 .16
OLR-NLL ordered
MT.
LB EOT
.552 176
599 136
.45 104
388
105
.155 064
.1 1146.0s.
.218 067
214.063
.489 0so
.487 096
.576 .oo
.591
095
.250 066
.245
.190 n40
.197 041
.228 n36
.23 035
.073 020
.073 .2200020
560.172
.32. 105
.366,49
.212.065
.480 .088
569 .00
.253,069
.177,037
.227 .039
.076.018
Learning
Labeling
ABA
A12
KRA
CO1
PU1
BA1
CO2
PU2
|
BA2
EL2
NNT FOT
.311 018
.350 012
.179 o08
.171 .08
.321 012
.107 oos
.2000
.167 oos
.339.om 011
.350 .000
AD
351_
330.011
.176.009
.171 .008
.323,013
.107
.199,009
.166,008
.340.00
.348,000
Hinge-IT non-ordered
MT
ST
347 015
347 o15
.333 0
3353.o1
.157 o10
.157 o10
.168
.168 oos
o08
-318 011
.318 011
.102 oo8
.102 oos
.201 011
.201.o11
.163,009
.163 009
.340.o00
.44 o10
.347 011
.347 011
BOT
.31
.350
1566,09
167.
.318,010
.102.008
.198,010
.160
.33.,00
.34,000
Hinge-IT ordered
ST
.319 017
.354 011
.157 010
.169 oos
.321 011
.101 .oos
.200 0099
.159 009
.340
.348
EOT
319
.350_am
.157.00
168 .08
.318.0m
3318,000
.101.0os
.198.o10
.158.009
.388
.348.p0
Hinge-AT ATT
ST
.388 017
.352 011
.16
.169 nos
.322 012
.101 oos
.199 010
.160 008
.341
.30
ordered
EOT
.350.017
.352.11
.158 011
.168 .088
.110 .320
.100
.198 .009
1558,09
.339,010
.347 _011
Logistic-IT non-ordered
MT
347
348 010
.154 150.
.166
.319 011
.101 007
.195 o0s
.174 009
.342 011
.348 811
ST
347.017
348,010
.154 op0
166 .000
.319 011
.101 .007
.195 .008
.174 009
.342 011
.348 011
EOT
.347 .01
.347 .0.0
.153 008
.166 ,008
.319 .099
.101 oos
.195, 0008
.157 .008
.44 011
.347
Logistic-TT ordered
MT
349.o17
.37 010
.153.010
.167 .09
.318 010
.101 oo7
.196 o009
.160 009
.342 011
.349 012
FOT
.348 .16
.348 3000
.152,008
.319 .167,006 .006
.100 .os
.1941,09
.159,009
.399
.346
Logistic-A'T ordered
MT
LB
351 .17
348.017
338,010
347,000
.154
.154.009
.167 .099
167.
.319 010
.319
.009
.101 .os
.101 ooos
.197 007
.197
.158.010
.158 009
.342 012
.42.013
.349 _100
.35 _011
EOT
.350
.349 .100
.153,010
.168 .099
.319 .009
$100,0s-
.196,008
.156,009
.34.001
.348
MT.ST
338 017
348 .10
.152
.167 .00
.318 010
.101 007
.195 oos
.158 ons
.41 012
.349 011
OLR-NIL ordered
LB
351 017
347 010
.152 009
166 .00
.318
.101.o0
.158 0009
.440.011
.350
FOT
.349.15
.349,010
155,.0
1667 .008
.318 001
.100
.194,007
.156,009
334 341o12
.34
Learning
Labeling
POT
A11
EL1
CAL
CE1
CE2
2DP
FRA
MVA
AD
NNT
EOT
335 115..08
.315 oo6
.243
.243 007
.300 0099
.299
.257.
.258 .06
.290 .00
.292 .07
.266
.267 .o.6
.134.003
.134,003
.118,003
.119 o03
.010 ons
.009 oos
non-ordered
Hinge-IT
EOT
MT
ST
.365
.359
.365 015
.239
.241.008
.239 .o0
.2977 .008
.298 007
.298 007
.217
.28..00
.248 .06
.321,
.325 006
.325 .006
.301 0os
.296
300..os
.137 .o03
.149
.149.008
.120,004
.121.004
.121
.007.001
.007 ,001
.007 001
Hinge-IT
MT.TT
.311
.239 0339.00e
.298 o08
.248 180.007
.288 .os
.266
.134,00
.120,oom
.007 001
ordered
EOT
.311.007
.240.00s
.297 007
2477007
.287 .077
.265.006
.133,003
.119.00
.007 .011
Hingo-TT ordered
ST
314..o
.239 0399.0s
.299 oos
.25 027.
.290 290.007
.669 007
.134.003
.120.004
.009.000
EOT
.314,008
.239.007
.297 .007
.252.00 000
.288 .077
.267 .00r
.134,003
.119,004
.008,007
Logistic-IT non-ordered
MT
.333 pos
235 007
.296 o0s
.238 007
.882 006
.200 006
.133,008
.121,004
.006 .006.001
ST
313. 0os
.35.000 007
.296 .08
228 007
.82 .06
.290 ao6
1333.003
.121
.006.000 001
EOT
.311 .07
.235
.296
.277 0007
.281 .07
.286 000
.133
.119 0044
.005 ,001
Logistic-IT ordered
MT.ST ST
.313 oos
.33 00.8
.277 .007
239 007
.282 .066
.263 no7
.133 004
.116 004
.009 p12
EOT
.311 .00l
2235,00
.296 _00
.238 .007
.281
,007
.262,au7
.133,000
.116,004
.008 .012
Logistie-AT ordered
MT.ST
.312 007
.33 .00
.298 oos
.21. 1o10os
.282 .006
.297 .00_
.133 003
.119.000 .044
.021 .24
LB
312
236
036
298 ons
241.07
.883
D0G
.297
00s
.133
.120.000
.021 0224
EOT
.310 007
.235.007 007
.298 008
.210 007
.280
006
.295 007
.133,003
.119 004
.020 023
OLR-NLL ordered
MT.ST
.313 007
.235 007
.298 ons
.239 009.
.281 007
.290 g07
.134.003
.119 004
.017 022
1,18
312 007
.255 008
.297 oo8
.299 .07
.282 .06
.290 207
.134.003
.119 004
.017 7 022
EOT
311 007
236 go8
.28 008
.239
.281 007
.288 606
.133
.118 004
.016,021
24

---

Published in Transactions on Machine Learning Research (1/2023)
Table 7: A counterpart of Table 2 regarding MZE for Task-Z Z and EF5 datasets.
Learning
Labeling|
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
|
STO
NNT EOT
.122 116
.568 132
.397 059
.326.079
.677 065
.757 .741.062
.455
.319,018
.347 o03
.179.00
AD
.88_120 126
1577162
.399,08
.322.77
.685.00
.022
.45,065
.318,04
.34,,02
.176,023
non-ordered
Hinge-IT
EOT
MT
ST
.96 154
.944 195
.668 127
.587
597.114
114
.385
.18,057
418.087
.338 .85
.355.11
.35.110
.675
.655 069
.675 .709
.760
.751,80
.751.s0
.45..87
.451,87
.448
.311,051
.315,011
.311.01
.349.0a 047
.353,04
.349.oa7
.158 .266
.207
.207 039
Hinge-IT ordered
MT.ST
666 153
.588, 134
403.077
.323 078
.685 .06
.751 .79 .749
.431or5
.307.017
042 .333,oc
.155 .26
FOT
.558 131
557
.333
.379
.328 077
.679 .04
.447
.35,05
.345.44
.154
Hingo-AT ordered
MT ST
.728 159 662.132
.69 135
.37 .091
.300 oos
.676.60 .799
.800 063
422.a78
330,051
.340 p46
.156 025
EOT
557.122 129
.383
.355.011
.685 .83
.750,a7
.443,77
.320.047
.345.00 .422
.151 .225
Logistic-IT non-ordered
MT
.600 157
.53 127
.17 082
.66 117
.62 078
.765.om 070
.47 070
323,00 017
.353 040
.164 028
ST
692
593
127
.177 .82
.366 117
.62 078
.765.070
447,070
.323
.353 .000
.164 025
FOT
.668
138
.52,112
.385,m
.320,ra
.672 080
.751,063
.454.77
.305,09
.152.a2m .26
Logistic-TT ordered
IT
MT ST EOT
.656 142
581 11.9
405
.37 307.092
683 .80
.761
.40.069
.308 056
.345
.136 ..2
.664.12
.577 118
.35 088
.311,0m
.668 071
.746 076
.451,072
.310,06
.341 .48
.136 1136,02
Logistic-AT ordered
MT.ST
.656.10
547 137
423 085
.309 065
.677 .70
.737 .07
.443 083
.298 645
.330 p44
.136 022
1,14
.730 146
.565 150
A18 085
.304 .63
.671
074
.765 065
.422,077
.305,08
.335 051
.136 023
EOT
.6600.81
.577
142
.399
.299
.687 004
.735
.60
.455 085
.308
.342 051
.139 025
OLR-NLL ordered
MT.ST
.664 135
569 116
.04 ose
.29 994
.663 072
743 075
.443.81 081
.299 p48
.331 p43
.134 022
LB EOT
.718 142
.585
106
396 o58
.08
0BG
.652 067
.713
061
.439
.303
.337 042
.136
684 125
.575 120
.382.ss .00
.308 .78
.664.77
.740.am
.448.s0 080
.309 044
.342.044 044
.134..23
Learning Labeling
ABA
A12
KRA
CO1
PU1
BA1
CO2
PU2
BA2
EL2
AD
NNT EOT
.517,01
533_o17 o17
o17
.489 014
.02..13
.222 2912.11
.277,011
.277
.482 388.008
.483 010
.204.01
.206.010
.331.011
.331,ou1
.271 .009
.268.012
.522,008
.526.012
.547 ,011
.555 012
non-ordered
Hinge-IT
EOT
MT
ST
.17.016
.577 016
.19..18
A84.o12
481.01
484.012
.277.011
.279.012
279.012
.27T,011
.277.013
.277.o11
.485.00
.495 013
.495 01s
.205 oo9
.205 .09
.203.008
.331011
.329,011
.331.011
.262.010
.274.010
214
.551 .520.01
.521.o11
520.o13
.545.m
.551 0011
Hinge-IT ordered
MT.ST
521.1oo
484.o01
.79..01
.27..110
.491
.203 .00
.299
.268.011
.224.013
555
EOT
516.016
.480,010
.279,013
.276,011
.485,009
.205_009
.328,01
2228.111
.265.010
.521.12
.546 .011
Hinge-TT ordered
MT.ST
531 017
.503 013
.279 011
276.011
.489 o10
.204 00.
.330 001
.63
.26
.554 013
EOT
.515.m1
.478.010
.277.010
.276,011
.486,m
.205.010
.328,012
.263.010
.523,01
.548.010
Logistic-IT non-ordered
MT
514.017
494.014
281,013
.78 011
.466 .100
.204
.330,00
.300,022
523..11
559 0013
ST
514.017
.494
081
.288,013
.486 010
220
.330 010
.300 012
.523.014
.549 013
EOT
.14.019
.482.01
.274.01
.25
.486 0100
.203 010
.328
.268
.500
.546
Logistic-TT ordered
MT ST EOT
.493,011 .515.o18
.271 .o10
.25.0112
.484.o0 .0.9
.202.009
.327.010
.08,,122
.222.10
.551 012
.14.018
.483,012
.272.11
.274.010
.485 .00,000
.204.010
.326.011
.268.01
.519.01
.546 ,111
Logistie-AT ordered
MT ST
.521.1r
.486 010
274.011
275.011
.488 010
.203 009
.329 011
.259 011
.244
.551
LB
.532
o17
.493 0L3
.23.,011
.275.011
.489
.203
.328
.258
.009
.52T.o11
.554
EOT
.521mm
.482.01
.271.011
.275.011
.491.o1
.203 .00.000
.326,010
.258.009
.525.012
.549.011
OLR-NLL ordered
ST.ST
LB
.526 016
.512.o1
.900 012
485
222.012
.275.010
.27.0111
.487 .009
.486 006.
.202 009
.022.10
.328 ,000
.327 329.009
.259,011 0011
.260.011
.527 014
.520.014
.553.01 014
.548 .12
EOT
.511.1
.479.01
227
.274.011
.487
.202 .09
.25.011
.261
.221
.57
Learning Labeling
POT
AI1
EL1
CAL
CE1
CE2
2DP
FRA
MVA
AD
EOT
NNT
.486,m
.513 011
.397 .009
.398 008
.486 010
.479.00s
.389 0099
388 .09
.497 .007
.502 007
.479,00
.485 pos
.241.00
.242 005
.220 ons
.219,005
.017 .006
.021 004
ITnge-IT
MT
.474.010
.412
.476 ooa
.448 nos
.496 007
.476 oos
.245 005
.217
.089
non-ordered
EOT
ST
.471.013
477.0100
.412.007
.412 .412.009
.472 pc9
.476 009
.448 .008
.488
o08
.492.mm
.496 007
.476 808
.473.0m
.241 ooos
.245 005
.217 ons .214.0m
.089 018
.083
Hinge-IT ordered
MT. ST
.469 .09.
.397 .009
.474 007
.447 .008
.444 007
.474 oos
.246 004
.214 004
.015 003
EOT
.466 011
.395 0009
.47 1.08
.477 .009
.492 .08
.472 008
.241 000
.213 .004
.013 .003
Hingo-TT
MT.ST ST FOT
.88 011
412.008
.482 010
.451 .099
.505 po8
.484 ons
.245 005 .240
.216 004
.072 022
ordered
.482,012
.410 1000000
.476 ,09
.450
o00
.497 .007
.477 .007
.2 2114,004
.065 .020
Logistic-IT non-ordered
MT
.699 p09
.414.008
.479 .100 .479
.449 .009
.497 oos
.473 010
.244.005
.218,006
.076 007
ST
.470 010
.414 008
.449
.497 008
.473 010
.24 005
.218
.076
EOT
.465.009
.12.007
.477,010
.48
.077
.496 .007
.470.007
.241 .014
.211,005
.066,005
Logistic-IT IT ordered
MT.T
.465 007
.395, 399,009
.478 009
.447 148
.497 oo8
.472 .09
.244 004
.212
.015 003
EOT
.462.mm
.355.08
.475,009
.oo8
.496
.007
.469 .008
.241 .004
2111.04
.013.00
Logistic-AT ordered
MT.
LB
.470 011
.468 pos
.412 008
.412 1n2
.483 010
.480,e10
.454 ous
.451 L,00
.505 008
.501 .007
.484
.488 009
.241 005
.245
.212,006
.211 .oos
.087 0:15
.086 .44
EOT
.460
.412,008
.478,010
.450 o07
.499 ,077
.482
.242
.210,005
.08 1,44
OLR-NIL ordered
S'T
LB
.463 po9
.465 011
.411 .o09
.110 100.000
.477 .088
.45..no
.450
.449 o10
.496 mna
.501 007
.472 ons
.478 0os
.215 005
.245.005
.212,005
.211
.071 .036
.071 036
FOT
.460,0m
.410,008
.474.009
.448 .009
.495,mm7
.471.m7 .m07
.241 .004
.211.005
.066 0666.99
25

---

Published in Transactions on Machine Learning Research (1/2023)
Table 8: A counterpart of Table 2 regarding MZE for Task- Z and EF10 datasets.
Learning
Labeling|
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
|
STO
NNT EOT
892 096
.728 110
.647 069
.508 119
.843 057
.875 .58 .859.09
.647 072 .660,08
.581 o66
.577 o43
.340
AD
868 095
.713,115
.622,07
.506,081
.838 063
.596,051
.543,048
.337 .337
Hinge-IT
MT
.920 0777
.756 103
.649.700
.568
841.00
.884,057
.673.77
.594.07
.549.oas
.372.04
non-ordered
EOT
ST
.866,001
.912 081
.731 131
.756 103
.639
641.o76
.516
$51_082
.851 .05
841 .59
.879 050
.876 055
.665 064
.666
.398 ose
.594.067
.561
.549 008
.349 .034
.368 041
Hinge-IT ordered
MT. ST
800 091
.727 103
636,074
.538 09s
.837 .04
.868 .53
.676 065
.597.55
.553.00 040
.335
1335.041
EOT
868 076
.731
641.822
.504.086
.839 .63
.860.046
.656 .67
.597 .058
.563
.329
Hingo-AT
ST EOT
894 114 8T2
775 118
632 073
.006 104
.23 065
.877 053
663 073
.54..72
.557
.319 p33
ordered
.727 112
637
,772
495,095
.831 .051
.868 .055
.656,077
.586,066
.561,044
.316,om
Logistic-IT non-ordered
MT
.906 099
.T37 122
.653 086
.559 104
828 .59
.822 041
.61 083
.011 .63
.565 046
.355
ST
.908 093
.732 (192.199
647. .079
.55
.824.00
.880
.658 .081
.601 0633
.565.0a
.353 .045
EOT
.868.m
.752 106
661.o82
.336.09
.827 .01
.871.05
656.069
.594.62
.564.44
.343 .039
Logistic-TT IT ordered
MT.ST ST EOT
856 pos
.777 130
635 073
.52.
822 060
869 648
.659.r
.593.66
.556 os3
.315 315.032
.872 .823
.420
.636,776
.33,,95
.823
823,088 .08
$63,068
.658 .66
.595,057
.573,013
.320,o3
Logistic-TT ordered
MT.ST
LB
878 090
866 099
779..10
.739 119
.655077 077
626 059
491. Loss
.05 080
.836 .83
821 .088
.67 o53
888 646
636.074
669.070
.85.070
.585.08
.552 .46
.574 052
.301 101.
.301 .031
EOT
862 106
.732.122
.699
639,050
.422
.831,000
.868.om
.651,663
.590,065
.575,050
.311 1033
OLR-NIL ordered
MT.ST
LB
.878 106
.10 .078
.751 105
717. 122
.637
.677.77
.631 071
.422 192.084
.95 082
.866
.27 .080
.007
.88 .48
.60 .ass
.651 .071
.600 0069
.573.0m
.57T
.553.0a .048
.567 od6
.308 .035
.309 .37
EOT
866 Da7
.736,118
.635,078
.76.07
.829 069
.867 .522
.659.076
.585.057
.558 053
.303 031
Learning
Labeling
ABA
A12
KRA
CO1
PU1
BA1
CO2
PU2
BA2
EL2
AD
NNT EOT
.722.017
.366 614
.886.1
701 .13
.93.015
.94 013
.468 0133
.64.o11
.711.008
.721 0100
.405.013
.403.ox4
.527.012
.528 011
.458.0
.465,017
-743.01
.741.10
.736.m
.764 011
non-ordered
Hinge-IT
FOT
MT
ST
.719.ou 016
.220.016
.22
.702
.690.am
.702.013
.422 014
.42
.486.014
014
.471,012
.475.012
.455.012
.713 .099
.711.0m
.713 009
.404 o14
.404.014
.400.a1
.534
.533.013
.534.011
.456.010
.457
.45..11
.742.010
.742 010
.742.010
.751
.736
.751.12
Hinge-IT ordered
MT.TT
719
695 011
490.015
467.010
.719 011
.399.ax
.529.o
.48..05
.741 009
.742,
EOT
.724
.691.012 .012
490.o13
.67..112
.709,009
.401.a1
.531.033
.458.012
.741.01
.734,005
Hinge-AT ordered
MT.ST EOT
732 013
.716 013
.477 015
.476
.725 011
.401
.532 013
.449 .044
.748
.782 012
.722.015
.686.o13
488,013
.473,012
.715.009
.400,012
.530,011
.146.013
.743,010
.739 .719
Logistic-IT non-ordered
MT
722 722.014
711
487
.478 478.012
.714 1,00009
.404.0z
536
.459
.743.011
755..13
ST
722.014 014
.711
487.012
.478,012
.714 0009
.404 012
.536 012
459.016
.743
.752 013
EOT
.721 .16
.686
.85,015
.172.m
.710 010
.400 011
.535
.156.013
.740
.738
Logistic-IT ordered
MT ST EOT
.717.016 .725.00
701.111
.94.014
.471,013
.709 .009
.397 .013
.532.012
.464.033
.742.012
.744 0099
.692,012
490.014
.67.012
.708 .099
.399
.535.011
.664.12
.741
.735 ,010
Logistie-AT ordered
MT ST
.720.014
.696 011
489 014
.77 0122
.711 oo08
.397 011
.531 .009
.451 014
.743
.14 .750
.743 009
LB
015 734
.712
.92
.480,012
.720
.398 .011
.538 013
.450,014
.767 .010
EOT
.723.mm
.687.a12
.899,15
.473,011
.715.010
.399,o1
.532.m
.47.
.743,010
.738 .mmm
OLR-NLL ordered
MT
LB
.718.016
734 015
.708 012
.999 .03
A86,014 014
482 .25
.469.02 0122
.65.011
.717 .111
.709 .08
.397 .133
.398 .91
.527.010
.57.
.449 014
.45..122
.737 .010
.747 012
.739 .009
.764 011
EOT
.721.17
6877
483.013
466.014
.710,mm
.398
.524.oo
.477 0122
.742.011
.738 .009
Learning Labeling
POT
AI1
EL1
CAL
CE1
CE2
2DP
FRA
MVA
AD
NNT EOT
.652.007
662 010
.612 010
.609_.op
.655_ees
.676 o08
.598 .099
596
.678
.685 007
.659.mm
.665 oos
.447 000
.446.00
.423.om 007
.420,006
.155.060
1899 069
non-ordered
HInge-IT
MT
ST
010 682
682 010
.229 oos
6229
008
.661 oo0s
666
008
.656 0506.00
.666 009
.719 007
.719 007
.705 007
.705 007
.451 005
.451
.420
.420 002
.300 072
.300
EOT
.675.m
.627 .007
.647 .000
.652.008
.711 1 Lonn
.698 on6
.447 .oos
.416 nno
.220 .688
Hinge-IT ordered
MT. ST
643 007
.612 0
650 opo
.65 0008
.721 .077
.710 009
.452 oD6
.418 0oG
.140 0z0
EOT
643 009
.614.000
4,00
.649 .009
.654
.715 0,07
.703
.000
.447
.005
.416 1,06.006
.115 ,000
Hingo-TT
MT.ST ST
650 009
.67 009
.675 008
667 .008
.721 007
.711 gos
.400 oo6
.417 006
.179 069
ordered
FOT
009 646
.629
.009
.65 pos
.06 .655
.7 7 7.007
.704 .00s
.446 ,00
.41 4,00
.138 .033
Logistic-IT non-ordered
MT
689 010
631 .008
670 .000
6664 .oo8
.700 ooc
.709.cm .10
.448 oos
.424.0m 000.
.222 070
ST
010 688
.31
Loo8
660
664
oos
.720 006
.709 010
.488
006
.424
.22
EOT
.673.010
.627.006
.648
.000
.653.000
2712,000
.700,006 000
.447 .00
.417 .006
.182,058
Logistic-IT IT ordered
MT.T
6417 pos
.614.000
.656
.677 007.
.687 0o8
.661 oos .660.077
.488 006
.418 007 .415.00s
.128 032
EOT
.644.nm
.614.099
654.010
604_009
.86 .007
.447,00
.114.030 L030
Logistic-AT ordered
MT.T
LB
.654 009
.655 p09
.634 oos
638 011
660 o10
676 009
.668 008
.588 ,00
.723 007
.720 .007
.714 007 .706
.707 _mn0
.449 006
.449
.417 006
.418 016
.191 064
.187 .56
EOT
.649.mm
.629 _on8
.657 .oos
.656
.7 199 ann6
.446
.415,005
.156 045
OLR-NIL ordered
STT
LB
.675 po9
688 009
.634 000
.628 .nss
.670 008
667 _009
.651 Loos
.664 p09
.719 007
.714.00r
.695 n07
.703 007
.448 008
.448 006
.416.005
.416
.166 069
.164 oss
FOT
.6666.n9
.628_ons
.648p007
.652 .nns
.714 4,0
.695_mm
.447,00s
.415,005
.145.0m
26

---

Published in Transactions on Machine Learning Research (1/2023)
Table 9: A counterpart of Table 2 regarding MZE for Task-Z ZZ and EL3 datasets.
Learning
Labeling
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
|
STO
AD
NNT
EOT
.330.100
144
.175.101
.183 121
.139 o65
.122.060
.067 .042
.070 oss
.319,074
.326 076
.523,0s
.527
.045,033
.046
.177,042
.199 011
.200,am .59
.22 053
.043,016
.041
non-ordered
Hinge-IT
EOT
MT
ST
346
380
147
.185 129
.181
115,123
.122,0ss
.147 057
.147
.067 018
.068
.067 048
.336 .022
.312 om1
.312 071
.510 062
.523
.510
.047 034
.04
.047 034
.167
.190 017
.190,047
.197 o1l
.97 641
189.004
.036
.032 .12
.022 012
Hinge-IT ordered
ST.
.324.1m
.180 105
.144.051
.070
.010 .071
-311 .066
.516 063
.048 004
195,010
. 198 041
.03 .012
E0T
382.149
.176 110
.119,oss
1071.00
.330 .778
.527 .044
.045 ,001
.165,010
.192 036
.036
Hinge-AT ordered
MT.TT
.30 156
.172 098
.147 059
.077 043
.305 072
.526 oss
.048 034
.191 043
.207 051
.030 .144
FOT
376
.179 .994
.119,04
.074.0m
.318 065
.541 .094
.044. .044.004
.163,042
.201 .444
.044_o1
Logistic-IT non-ordered
MT
330 145
.209 112
.1 1442 065
.065
.319
.547 oo5
.052 029
.168.00
188 p4G
.029
ST
3990.15
.209
112
.142 065
.065 042
.319 071
.547 065
.052 029
.168.04
.185 046
.029 011
FOT
142
.187 100
.120
.069.035 040
.31,,77 197
0544
.048,031
.160,013
.182,015
.030,o11
Logistic-IT ordered
MT. ST
.392 140
.215 128
.142 055
0633 041
.313 .030
.522 .78
.053 029
.166.038
.622 034
.029 .011
EOT
.426 137
.203, 114
.119,054
.03
,445
311
001
.548 070
.052 029
.165
.173 p42
.03 _011_12 _012
Logistic-AT ordered
MT.
LB
.438 143
.408 128
.205 11.
.211 121
.143
.148 052
056
.069
.069 040
039
.304 084
.308 083
.540 082
.525 081
.055 030
.055 .29
.167 013
.162 034
.171
.167,000
.0300
.030 009
EOT
.408 141
.184, 108
.119
0068.01
.307
.077
.544
.052.
.160,o00
.178,039
.030
OLR-NLL ordered
MT.S
LB
.426 148
396 134
.204 128
2000 119
135. 033
.140.050
.067 044
.08 044
.320 073
.319
071
.528 076
.537
074
.054 .299
.054.00
.166
.166.08
.174 046
166
.031
.031 012
EOT
.488.31
.125.112
.1 115,044
.072.015
.317 3,000000 .0000
.535,080
.051,030
.144,099
.882,04
.030,010
Learning Labeling
ABA
A12
KRA
CO1
PU1
BA1
CO2
PU2
BA2
EL2
NNT FOT
237 013
.000 o04
.198 011
.012.o04
.276 010
.065 ono
.015 003 .015,003
.126 o08
.088 ou7
.062 004
AD
236 032
.030 go5
.193
.01 1 Lous
.276
.055 .005
.120 .007
.082 .092
.061_004
non-ordered
Hinge-IT
MT
ST
236 014
236 614
.030 004
.030 004
.150 1510
.150.a
.010 .022
.010.
.270 010
.270 010
.056 007
.056 o0z
.013 003
.013 003
.110 0099
.110
.855 o0o
085 006
.062 004
.062
EOT
.235_015
.030,004
.149,011
.011,003
_269,o1
1055,006
.013,003
.113,010
.083.005
.061.004
Hinge-IT ordered
MT.T
236 014
.030 004
.147,012
.011.00
.269 009
.054 oo6
.013 002
.110 .oos
.85
.062 000
EOT
.235.112
.030.on4
.145,01
.011.002 .002
.269 0100
0544.066
.013,003
.114.008
.844.006
.061.004 _014
Hinge-AT ATT ordered
MT. EOT
235 234
.030 004
144 .000
.010.0os
.269 .010
.055
.013
.115,010
.084 pos
.062 0662.004
.355.12
.030 004
.1 .143,010
.010,o0s
.268 0111
.051_000
013,,03
.117,010
.083.omm 0006
1661_004
Logistic-IT non-ordered
MT
232 016
.029 oo4
.13 008
.011.oo
.265 009
.053 0o5
.014.o0
.098 008
083 007
.061 .044
ST
232 016
.029 004
.135 008
.011
.002
.265 .099
.053 oo5
0004
.098 008
083 007
.061 .004
EOT
.230 013
.030
9o4
.138
009
.011
1. .008
.267 0100
.054 005
.013 002
.098 008
.083 006
.061
Logistic-TT ordered
MT.
.230 p12
.029 904
.133 np8
.011 .002
.267 009
.053 004
.013 003
.092.07 .07
.883 n06
.061 .004
EOT
.229 .012
.030 0005
.131,
.010.000
.265
,0000009
.0541 1151_005
.013
.091 .007
.083
.061
Logistic-ATT ordered
MT.ST
.232 016
.029 004
.133 009
.011 .022
.265 010
.053 oos
014.,003
.092 006
.081 007
.061 .004
LBB
230.o14
.029
.133,009
.01 0,02
.265
.053.000
.014,003
.092.00
.084 Q84_006
.061 [06]_004
EOT
.220.014
.030.004
.004
131.o0
.010.002
.264.010
.054.006
.014,003
.092.00G
083..00
.061.004
OLR-NLL ordered
MT.ST
LB
230 013
.230 .300.13
.029 .004
.029
.004
.132 1.32.007
.132 007
.010.002
.011.00 1o0s
.267
.267 .09
.010
.054_005 0005
.053.oom
.0133 0022
.013
.093, 007
.093.om
084 007
084.0o0
.061 1011001
.061
EOT
230 014
.030 .04
132 o08
.01 1 .o02
.265 .010
.053
.013
.093
083
.061
Learning Labeling
POT
A11
EL1
CAL
CE1
CE2
2DP
FRA
MVA
NNT EOT
.061 001
.068 005
.019 002
.224 oos
.033 002
.039 8o3
.094 003
.125.om 011
.002 0011
AD)
.060.000
.069.000
.018.00z
.224.006
.033.000
.039.ao3
.095,003
.123,008
.001.000
non-ordered
Hinge-TT
EOT
MT
ST
.061.000 .004
.070
.070
004
001
.068
.068
0668 005
.00
005
.018
.019
019.,02
002
o02
.213,006
214.000
.214 214.000
.031 .002
.031 .002
.031 002
.039,aos
.039 .022
.039
002
.095,003
.095 0033
.095.003
.092.000 .002
.094 007
.094.007
.001 .000
.002.00 .022
.002.00 0011
Hinge-IT ordered
MT.ST ST
.071 003
.069 0os
.019 002
.214 o06
.032 .088
.038 003
.095.003
.091. oos
.002 001
EOT
.062.nn .004
.068 005
.018 p02
.212,000
.031 .002
.035 .012
.095,003
.090,006
.002,001
MT ST
.088
003
.068 oos
.019 002
.18 .07
.032 002
.034.000 0834.00
.044.003
.091 .005 .090.005
.002.001
EOT
062
000
.069.oo
.018 p022
.217 .007
.032 .002
.035 .000
.095 003
.001 .000
Logistic-IT non-ordered
MT
044 001
.071.000 0011
.019 002
.200 .066
.031 .002
.038 0o3
.095 003
.084.oo o03
.002 001
ST
049
004
0071.00 .004
.019.002
.200
.031 .002
.038 .033
.055
.084 003
.002.00
EOT
.036 .004
.069.omm
.018 .022
.199.000 .00.
.032 .002
.038 .033
.095.003
.083.005
1001,000
Logistic-IT IT ordered
MT.ST ST EOT
.050 004
.070 gos
.018 002
.200 .000,06
.032 002
.035 ,00
.094 003
.081 .04
.002 0000000
.035.004
.069.00 .000
.017 .002
.199 .006
.032 .002
.035 .000
.095
.081 0003
.001 .000
Logistic-AT ordered
MT.T
.041 .004
.070.00
.019 002
.200.005
.031 .0100002
.035
.095
.082 082.003
.002
LB
.038
004
.05 .069
.019 0022
.201.00 oo0
.032 0022
002 .035
.095 003
.082.003
.002.000
EOT
.034,0m .003
.069 1669,004
.018 pe2
.199
.032
.035
.095 000
.081
.001 .000
ST
.038 001
.069 0o5
.019 002
.201 .006
.032 .002
.038 0o3
.095 003
.082 003
.002 000
OLR-NIL ordered
L.B
.037 004
.070 305
.019 001
.210 000
.032 .002
.037 003
.095.000
.082
.012
EOT
.034.004
.069.co 060.000
.018.am
.199.
.031 1/31.002
.037 003
.095.003
.081.063
.001.000
27

---

Published in Transactions on Machine Learning Research (1/2023)
Table 10: A counterpart of Table 2 regarding MZE for Task-Z Z and EL5 datasets.
Learning
Labeling
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
|
STO
NNT EOT
582.113
.463 137
.172,066 .191,70
114.055
.528 071
.714,070 .694.073
.087 040
.265.050
.268
.169 025
AD
483,128
.118,080
.549,0a ,083
.090.044
.261.082
.264.049
.167,031
non-ordered
Hinge-IT
EOT
MT
ST
5544
518.179
518.179
.476
.468 171
.468 171
.185.074
.183,772
.195,084
.124.050
.119.055
.119,066
.532 009
.530.073 073
.529.ors
.702 074
.706 o69
.707
.097
.092.
0090,00
.252.0m
.279.0a
279.051
.288 .888.40 040
.268 .000
.28
.164
.209 034
.209.030
Hinge-IT
MT.T
.542 174.170
.471 130
176.077
.130 o5o
.530 079
.704.770 .70
.098 .199
.266 014
.260 040
.147 .21
ordered
EOT
.548 143
.489 124
.179,067
.129,052
_547 ,778
.699
.095
.253
.260
.146
Hingo-TT ordered
MT ST
.588 142
.452 152
.168 075
.221,50
.530 .83
.400 067
.094.048
.206.0a5 .262
.85 052
153 .30
EOT
148 .538
.881,
.181,770
.128,050
.553 076
.705 0011
.100.048
.272
.148
Logistic-IT non-ordered
MT
532 142
.469 133
.201 071
.266 052
.546 079
775.075
.105 044
.258 015
.283 045
.145 023
ST
.532 148
469
163
.204 .044074
.15.049
.546.07
.715.078
.107 .44
.258
.283 046
.145 .223
EOT
.532
163
.467 119
.196 066
.124.044
.545.079
.106.043
.25T .54
.264.089
p27
Logistic-TT IT ordered
MT ST
5556 151
463. 127
193 067
.12..s
.532 077
.766 006
.105.041
.245.43
.25.
.128 021
EOT
159 .538
.41.131
.192
.128,052
.537
.076
.701,078
.110,0m
2056
.265.044
.127 .223
Logistic-TT ordered
MT.ST
LB
.592
.536 128
135
.57 147
484 131
.155.711
.187 oos
1200.o88
.121 055
.551 o86
.549 073
.722.061 061
.708 065
.107.044
.108
108,001
.250.522
.248 044
261
.265 233
.130,019
.131 015
EOT
.538.157
.422 107
.205.07
.124.042
.557 .83
.701 076
.106.04
045
.253,o0
.274 040
.130 020
OLR-NIL ordered
MT.ST
LB
.62 135
.60 128
485 133
.881 120
.189 064
.194
.122.0m
.124 051
-547 .800
.544 077
.714 077
.728
.77
.108 1018
.108
043
.249
.20..44
.257
.25
.129 .211
.129
EOT
534 152
.471, 120
.191.072
.121.054
.548 080
.708 088
.110.04
.250. .54
.265.044
.129.019
Learning
Labeling
ABA
A12
KRA
C01
PU1
BA1
CO2
PU2
BA2
EL2
AD
NNT EOT
.223.m
.249 026
.140,00
.154, 009
.220.012
.223 013
.049..06
.052 006
.469.07
.470 o11
.110.00 007
.113 609
.069.oos .068.008
.160.010
.1641
.175.008
.176.00s
.166 .n8
.174 00s
non-ordered
Hinge-IT
FOT
MT
ST
232 012
.223.mm
232.o12 0322
.138,00
.152,009
.152.009
.229.
.233 016
.233 016
.050.aoG
.053
.053 005
005
.471.02
.496 015
496.015
.111 aos
.108
.111,0o
.068 o07
.066.0os
.068 oor
.164.om
.202 011
.202.011
.178 009
.177
.178.009
.174.00s
.166
.174
Hinge-IT ordered
MT.ST
.23 014
.145.01
.20 014
.052 D0G
.483 011
.111 oos
.068 .06
.169.000
.174
EOT
.221.014
.137,009
.217.014
.050.006
.470,012
.108,000
.067,000 0067
.161,00
.177.00
.166 .008
Hinge-AT ordered
MT.ST EOT
.330 014
.146
.228 013
.052 005
.476 011
.111 o0s
.068 007
.164
117 008
.173 005
.223.01
.137.ous
.223,012
.050.006
.469,m2
.109,00s
.066.0o6
.158,009
.177,099
.166,00s
Logistic-IT non-ordered
MT
217.011
.148 .000
.216.013
054
.474.01
.106..o0
.06 .00
.209
.16.
.168 .0.00
ST
217.013
.137,000 0009
.216.013
054.000
0114.01
.106 oos
.067 006
.209 010
.776.009
.166 p08
EOT
.218 2118,14
.138 .099
.212.012
.049 oos
.470 010
.107 .007
.064
.160
.178,110
.166
Logistic-IT ordered
MT ST
216.
.137.010
.22..014
.053 005
.473 0111
.107 o07
.065 oo8
.161 012
.176 009
.167 .008
FOT
.218
.136
.210
.050
.472
.107
.063 006
.157
.179
.166
Logistie-AT ordered
MT.S1 ST
LB
215.011
216
.137 o08
.136 oos
.218.m1
227..014
.062 006
.053
.005
.474.012
.474.012
.108 007
.108 o07
.064
0044.007
.159.
.160
.166 oos
.178 008
.166
.166.6 p09
FOT
2216.012
.136 030
.215,013
.049,am
.474.012
.107,00s
.063.007
.157,011
.177,007
.165 1665_008
OLR-NLL ordered
MT.ST
LB FOT
215.013
215
.137 o10
.137 .00
.212
.212.014
.051 005
.052.om
.471
4471.10
.106
.106 007
007
.065.000
.064.006
.160.010
1600.10
.179. 010
.178
.166 on9
.166 1666_0s
217.
136 g08
.008 013
.050 oos
.473,012
.106
.063
.158
.179.00
.165 .000008
Learuing Labeling
POT
A11
EL1
CAL
CE1
CE2
2DP
FRA
MVA
NNT EOT
.043 004
.173 006
.045 003
.349 .o88
.090 004
.104.004
.172 006
.142.006
.004 001
AD)
.041.000
.170,00c
.046
.347,009
.090,004
.105,004
.167,000
.1400,004
.003 ,001
non-ordered
Hinge-TT
EOT
MT
ST
.108 0oc
.095.007
.107 D00
.170,007
.173 o008
.173
000
.051.001
.061 0077
.061 .007
.386 .099
.375.007
.386 009
.093 005
.092 .005
.089.00
.102.a0
.104 000
.104 004
.177,004
.177,000
.177
.152.009
.163 000
.163 009
.006,001
.032 .05
.032
Hinge-IT
ST
.035 pos
.172 006
.050 op0
.379 008
101 007
.104 004
.172 o06
.143 007
.004 p011
ordered
EOT
.034.00 000
.169_000 000G
.047 0014
_375.007
.089 p044
.103 ..0
.165 .60
.137 .044
.003 .011
Hinge-AT AT
MT. ST
.030 oo3
.172 006
.054 oos
.376 000
.091 004
.101 004
.175 005
.146 00G
.004 001
EOT
.029
p03
.169,006 .o06
.049 no4
.355.08
.090.mm mm4
.098 mn4
.167 .005
.138 .044
.003 .011
Logistic-IT
MT
.083 004
.175500 007
.056 003
.382 000
0089 004
.092 004
.172 .03
.144.00
0043 008
non-ordered
FOT
ST
.064.00
004 .083
.173
.169_007
006
.056
.050.004
005
.378.006
.382
.009
.088 .044
.088 .044
.093 204
.092 004
.172
.163 on3s
.136.00
.144 ons
.007 .002
00:3
Logistic-IT IT ordered
ST
.20 p03
.167 .006
.046 003
380
.089 .044
.092 004
.163 004 .163,004
.137 004
.003
EOT
.028.0mm _008
.166,
0o6
.046 003
.377 0008
.088
.044
.092. 009
.135 .004
.003.001
Logistie-AT ordered
MT.ST
.028 .033
.172 .008
.052 004
.380 .oos
.089 .044
.094
.163 004
.137
.004
1.13
.028
003
.169 .0oG
.052
004
339 .009
.089
004
.091
.163,004
.137 004
.004 001
EOT
.028 po2
.169 .ous
.049 pc3
.378 .088
.089 004
.094.044
.162 003
.135 ,033
.003 ,011
OLR-NIL ordered
MT.ST
1.B3
.028 003
.028 003
.170
.172 007
.049 000
.050 0o33
.376 .008
.376 .07
.088 .044
.088 004
.091 .04
0092.o0
.163 004
.163
.136,004
.1366 004
.010 002
.010
EOT
.028 004
.168,007
.048.om
.376.000
.088 .044
.091..00
.162.004
.136.004
.007 .001
28

---

Published in Transactions on Machine Learning Research (1/2023)
Table 11: A counterpart of Table 2 regarding MZE for Task- ZZ and EL10 datasets.
Learning
Labeling
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
AD
NNT EOT
.788
756 146
.632.113
.617 101
.387 069
.404.071
.274.089
.272.06
.707,07
.711 067
.853 059 .833,ma
.182.06
.189 o5s
.470,044
.473,073
.442,018
.460.644
.280,025
.284 029
non-ordered
Hinge-IT
EOT
MT
ST
.738 131
.722
712..44 144
144
.62
.220 113
.333 103
110
.387 076
.393
.397
.257
.345 070
.315.00
.708 070
.720
.708 074
.850 050
.829 060
.843 a50
.194.058 058
.190
.183
.454.063
.490 067
.486
.462 050
.447
.463 n40
.376 027
.376 027
.289 ,029
Hinge-IT ordered
MT
.998 146
.599 116
.392 0022
.244 .05
-713 .866
.22 064
.188,588
.488 084
.441.0au
.281 2811029
EOT
.744
6008 113
.388 074
.263 052
.716 .044
.826 071
.184 055
.447
.011
.443 ,033
.279 .228
Hingo-TT ordered
MT.ST FOT
.750 143
.616 118
.35 077
.291 079
.725 063
.852 058
.197 058
.471 .60
.435 049 .444,012
.284 028
.730
.612.112
.300
.284
.710,063
.837
.677
.186,007
.461 .082
.273 .026
Logistic-IT non-ordered
MT
.742 137
.631 111
.405 078
.296 073
.707 .66
.848 061
.197 054
.483 069
.441 .os
.306
ST
742
187
.29
103
403 403.075
358 086
.706 072
849 a5s
.206
.485
.41 D42
.306 031
EOT
.452.145
.607
.998
.874,077
.269,774
.711,070
.820,066
.194,054
.456,053
.41,,0t
.279 ,027
Logistic-TT ordered
ST
740.147
.616 122
.96 069
.322 067
.705 077
.829 048
.189 oss
.45..62
.446.00
.273 027
EOT
.744.120
.612.
395
.239 001
.722 .099
888,066
.188,062
.456,63
.43,009
.273.027
Logistic-AT ordered
MT. ST
LB
.718.19
748
149
.611 130
593, 115
393 072
387 079
.259 075
.248 076
.703 082
704.075
852 061
.822 .000
.193,055
.187,053
.477 .73
.65,071
446
.44.
.66 n26
.266 024
EOT
.718 138
6601.10
336,,ss
.245
.713 1173
.823,0m
.184,060
.456,055
.455,04
.263
OLR-NIL ordered
MT.T
LB EOT
711
770
.604 182
.615.111
378 050
396.091
.235 050
229.077
7000 D81
.701 067
.85 .58
853 050
.188 .50
.191 o55
.468 oas
.666.66
.366
444.043
.266 027
262
.788 146
.617.101
.392 0086
.229,082 .082
.717 .084
.837 .056
189,059
.453,05
.54.050
.264.025
Learning
Labeling
ABA
A12
KRA
CO1
PU1
BA1
CO2
PU2
BA2
EL2
AD
NNT EOT
.43 0155
.34 017
.205.01
.212 011
.00,014
.399 013
.166 160
.166 00e
.649
.652 011
.013
.250 012
.248 010
.209
.212.o1
.321.13
.325 014
.307 ,011
.313 011
.294_008
.244 on9
non-ordered
Hinge-IT
EOT
MT
ST
.137.015
.471
.470.016
.212.011
.203.01
.212.o1 011
.396,016
.477 015
015
.161.006
.144,os
164..077
.655,011
.662 014
662 014
249.013
.245.01
.249 01s
.204.010
.208 011
.008 011
.327,014
.347 016
.348
.306.011
.305 012
.305.012
.294 po9
.295
294
Hinge-IT ordered
MT.ST
.32 017
212.010
.410
.160 one
.670 012
.245 011
.20 010
.340
.308 011
.293 .0.0
EOT
432.016
.202.o1
.404,012
.159 1159,09
.656,010
.245.011
.204.010
.338,014
.304,010
.294_009
Hinge-AT ATT' ordered
MT EOT
.431 .16 .432.014
209 012
.066
162 102.00s
.655 0555.012
.266 011 .246.011
.200
.324.013
.311 .011
.24
.201.n1
.403,014
.162.009
.654.012
.205.011
.321.011
.308.014
.294.009
Logistic-IT non-ordered
MT
ST
.433 020
444
014
.206 010
.02 012
.397 015
408 014
.163 007
163 163.007
.652 011
.652 011
.245 012
224 012
.204
.244
.351 016
.35Lo17
.310
.300
.291 .008
.291 .088
EOT
.199 .430
.391
.158,010
.653
.243 o11
.196.010
.325.014
.38
.293
Logistic-TT ordered
MT.
.198.011 .426.ms
.38 0133
.158 .099
.65 o13
.243.011
.196,010
.377,013
.306,012
.21 .008
EOT
.433
.199
.396
156,007
.652 .022
.242
.196,011
.333,013
.309
.292 _009
Logistic-ATT ordered
MT.ST
.429 013
.198 010
.38 017
.161 o06
.655 011
.244
.00
.32..0 012
.306.011
.233 010
L.B
.433 .333
.199.01
396.017
1660,07
.653.012
.244.011
.198
.320,012
.315.013
294.01
EOT
.437 .066
.199,011
.394.015
.157 .008
.654.012
.245.013
.197,010
.317.m
.308,012
.293.009
OLR-NIL ordered
MT.
LB
.430 0155
.426 013
.198 010
199..01
.389,05
.392 005
.055
.157 .008
.157 007
.652.00 0011
.652 011
.242 011
.242 011
.197,012
.197
.321 Lo13
.320,012
.315.012 115.
.307 .012
.293 .099
.292 009
EOT
.34.
199.
389
117.008
.654.012
.243
.197
.300
.38
.293
Learning
Labeling
POT
A11
EL1
CAL
CE1
CE2
2DP
FRA
MVA
NN'
.141 .oos
.286 007
.267 005
.539 013
.218 .07
.211 .000
.318 005
.263 ons
.050 004
AD
EOT
.130,006
.283.007
.268
.528,0m
.216.006
.210.005
.S15.00
.262.005
.041 .044
non-ordered
Hinge-TT
EOT
MT
ST
.114,00s
.119 009
119
009
.305
.286.007
3305 011
.289.00s
.300
310.0o0
.583.00s
.587 nos
588 o08
.222.00 .06
.226 007
.226 007
.227 .006
.226 005
.226 oo5
.330 oos
.316.00s
.344 0o5
.269 oo5
.303
.288 ons
.039 ,003
.066 006
.074
Hinge-IT ordered
MT. ST
.151 005
285,007
.60 oos
_22__o0
.227 .006
.239 018
.323 oos
.269,004
.047 p05
EOT
.129,005
2833
2600.00
.517 .008
.225 .006
.229 .006
.315 .005
.268.005
.038 .004
Hingo-TT
AT
MT. ST EOT
.120 oos
284 007
.59 no6
.535 011
.235 pos
.237 007
.324 o06 .3115,001
.270 oos
.051 006
.106,00s
283
007
.260.
.526,00 g06
.223 000
.230 oo6
.268 .005
.043 ,004
Logistic-IT non-ordered
MT
.104 007
.92 .008
.303 0099
.586 008
.225 .066
.225 p06
.342 o06
.304. 009
.077 .005
ST
104
007
291 .008
303, .099
586
.225 000
2225,00
.329
.293
.071
EOT
.097,00 .008
283
2855,00
.582.00
.222 .006
.224,006
.315 .044
.268 .05
.038 ,003
Logistic-IT IT ordered
MT.TT
.135 007
.288 007
.28 007
.555
.207 .07
.226 006
.318 004 .3166.00s
.260 0000,04
.038 004
EOT
.115,007
270
.258 .07
.14.007
207 .006
2225.06
.260.004
.033,000
Logistic-AT ordered
MT.ST
.137 .37_006
287 287.007
.281 1007
.591.om 1,007
.229 2229.007
.234 008
.321 _004
.209
.052 .022
LB
.124 007
285 008
.277 .006
.04 008
.231 .066
.232 006
.320 005
.268 005
.043 004
EOT
.107 .00
283 080
.272,000
.587
.225
.231
.315
.266
.037
OLR-NIL ordered
MT.ST
LB
.110 007
.126 007
288 007
285
.255.077
2275.007
.584.000 .007
589 0008
.223.00 .0TT
.226
.225 .06
.25
.319 004
.319
.268 o06
.267
.042 001
.040
EOT
.100,00s
.282.007
.272.005
.582,00s
.222,00
.223 606
.315,005
.266,00 2266,005
.034,003
29

---

Published in Transactions on Machine Learning Research (1/2023)
Table 12: A counterpart of Table 2 regarding MAE for Task-A and EF3 datasets.
Learning
Labeling
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
NNT EOT
658 183
.404 125
.500.60 .145,06
.14.077
.631
.721 105
.271
.209 048
.63 037
.080 0119
AD
.668 189
.373 135
.226.09
.637 100
.748
102
.268.065
.183,009
.256.037
.079.021
non-ordered
Hinge-IT
EOT
MT
ST
.670 219
.644 184
219
.396 115
.392, 123
.396 115
.154.0s
.159
.159.o60
.211.79
.215,077
.211.799
.616
.644.123
.644.12
.711,112
.730
.730 104
.271
.267
.271.67
.184.04
.209.051
.209.oa
.254
.255.041
.255.0a
.075.01 018
.080
.080 018
Hinge-IT
MT.ST
666 204
.401 114
.152 .022
.218 071
.648 109
109 .706
.710
.262. 266.068
.199 o4s
o41 .250
.079.0xz
ordered
FOT
656171
.355 135
.1 1148,o
.204
.618 109
.264
.190
.25
.076
Hinge-A1 AT
ST
696 178
.396 104
.152 oso .147 053
.218 077 .210.057
.621.1 .124 .629
.706 113
.256 o65
.203 046
.251 .a99
.077
ordered
EOT
666
194
.355 109
.717 122
.267 066
.194 040
.248 038
.077
Logistic-IT non-ordered
MT
622 194
.416 134
.158 o66
.228 066
.625 099
.725 123
.249 065
.19T.047
.251 p40
.077.019
ST
194 622
.116.15
158 066
.228 066
.625 099
.725.12 1225
2219
.197 017
.251 o010
.077.019
EOT
.660181
.351.155
. 146,,47
.226,052
1604.094
.714,112
.250,064
.179.eas
.241.038
.081.22
Logistic-IT ITT
ST
636 199
.423 114
153 o6s
.237 .833
600 100
.721 131
.263.cm 267.077
.14..43
.239 p40
.074 .018
EOT
180
3855,06
.134,014
.224,059
.611 106
.721,119
.260,066 ,066
.182,033
2368,00
.074,018
Logistie-AT ordered
MT.T
1,1
.638 208
.638 208
.11.115
A11.115 .15
.148,064
.48 064
.223 067
2223008
.616 102
.616 102
-715.122
.715 122
.256,068
.256 066
.193,010
.20 637
.230 037
.075.022
.075 .199
EOT
690 192
381
136
.221
.616.110
.715
.261.70
.182.038
.233
.076.016
OLR-NIL ordered
MT. ST LB EOT
.658 188
658
188
.409 120
.409
120
.146,084
164
.234 083
.24..08
.611.112
.611 .12
.713 137
.713 137
.211
.251.071
.196 050
.196,050
.200 oay
230 037
.074 021
074.021
.688 186
393,101 101
.136,09
.217,0s8
600_104
.733
.26-1.03
.177.00 .009
.234.042
.076.017
Learning Labeling
ABA
A12
KRA
CO1
PU1
|
BA1
CO2
PU2
|
BA2
EL2
NNT
.35 020
.30 012
.180 o08
.173 on8
.332 012
.107 oos
.202 0009
.167 .09
.361 o013
.365 .612
AD
EOT
338
.38,010
.177,010
.173,008
.335
.107,00s
.202.010
.166,008
.300
.35,,02
non-ordered
Hinge-IT
EOT
MT
ST
.314.015
.35 020
375..20
.359,011
.369 012
3369.12
.157.010
.157,00
157.010
.170.009
.169,009
.170.009
.332.01
-330 -330.012
-330 012
.102.00
.102 oos
.102 oos
.201
.203,011
.203.011
.160
.163,009
163
.34.033
.364.01 .364.013
.359,014
.364.011
.365_011
Hinge-IT ordered
MT.TT
.376 020
.370 013
.158 010
.170 000
.332 012
.101 .os
.202 009
.159 009
.344
.363 p11
EOT
.37..017
.358,01
.157
.170,008
.332.o11
.101.0os
.200,009
.158.009
.35..02
3363,00
Hinge-AT AT ordered
ST EOT
376 o11
364.o15
.157
.170 .170.00s
.332 012
.101 .oos
.201 _009 .200,009
.160 oos
3G1
365 0000
.376 021
.359,o1
1188,01
.170
.332.00
.012
.100,000
.158
.359,013
.365
Logistic-ITT non-ordered
MT
ST
368 019
3G8.o19
355.011
.35 011
.144.00
.154
.667 .099
167 o0e
.329 010
.329
010
.101.007
.101 001.007
.197 oos
.19.00s
.174.009
.174 o09
.64 063
364.013
.363.012
.363 012
EOT
.368 .18
.352.012
.153,o08
.167 .000
.329 .009
.101
.197,008
.157 .008
.350.014
.364.01
Logistic-TT ordered
MT ST
.39 .199 .371,017
.35.mm 011
.153,00
.168 009
.329 011
.101 007
.198 008
.160 009
.363 014
.363 o11
EOT
.352
.152,008
.168 .009
.330
.110
.100 .os
.196 .009
.159
.361
.364,011
Logistic-ATT AT ordered
MT.ST ST
1.13
019 3368
.368 019
.333 010
353 010
.154
.154.00
.168
.168 .08s
.328
.328 0011
011
.101 aos
.101.0os 801.0os
.199
.199 .09
009
.158 009
.158 009
.361
.361 0144
.364
.364 o012
EOT
.370.pss
.35.g10
.153.o0
.168 000
.329
.000
.100
.197.
.156.009
.360.014
.363.pn
OLR-NLL ordered
MT.ST
LB FOT
.37 017
367 017
353 012
353
012
.152 009
152 .00
1667 oos
.167
.329
.329 011
011
.101 007
.101 o07
.197 008
.197 oos
.158 009
.158
.66
.361 0133
.363 .122
.363
.39 p18
.33.gg
0
.153.m
169 .000
330
.009
.100 oos
.197 .008
.157 .099
360 o15
.363.p11 0111
Learning Labeling
POT
A1l
FL.
CAL
CE1
CE2
2DP
FRA
MVA
NNT BOT
316.007
246 007
.311.0os 10110008
.268 oo8
.308 oos
.283.007
.134.003
.119
.010.00s
AD
316,000
216 .0.8
.31 .009
.268 007
.308 00s
.282
.134
118.,00
.009
HInge-IT non-ordered
MT
ST
.333 nz6
.393.o2c 026
242 .000
242 .09
.31
.311.00 1o.0oos
.258 our
258
.350 ons
.350
.324.009
332. 000
.149.000
.149
.121
121
121.004
.007.001
.007
BOT
.363,014
243.0000
.311,008
.257.008
.338,007
.314,008
.137 .005
.120.00
.007,001
Hinge-IT ordered
MT.ST
312 007
242 008
.311 .0.008
.258 008
.303 ,08
.82 .0.0
.131 003
.20 000
.007
BOT
312 20.008
242 .008
3310.oos
.256.oo7
.33.op8
.280
.133.003
1119
.007 .001
Hinge-AT A'T'
MT.ST BOT
314.
.242 .008
.311 .oos
.260.oo
.302,
.282 .007
.134.003
.100.0o
.009 .008
ordered
.314,008
.211 .008
.309 po0s.oss
.259.0o 000
300_,08
.280 007
.134.003
.119.004
.008 .007
Logistic-IT non-ordered
MT
.313 omo
233 007
.307 oos
.245.0
.293
.306.oo
.133.003
.12
.006
ST
.313.00 00.
238 007
.307 000
.245.007
.293. 007
-306 007
.133 003
.121 001
.006 001
EOT
312 2000
239 008
.307 .009
244
.922 .07
.299
.133 003
.119
.005
Logistic-IT
MT.ST EOT
.313 .008
238 0o8
.308 .oo8
.246.om o46.oo7
.244.007 n07
.255.007
.133 004
.116.004
.009 n12
ordered
.32,
.238 .000
.306 .009
.24. 244
.292 007
.24.0006
.133
.116,004
.008
Logistic-AT
MT.ST.LB BOT
.313 o0s
.239 pos
.310 oos
.247 000
.92 007
.310 000
.133 003
.120.0000 00.
.021 .244
ordered
311 1,07
238 .008
3309,oo
246,,007
.22,,006
.38
.133,008
1119,004
.220,233
OLR-NEL
MT..TTLL BOT
313.007
238 oos
.309 .009
246 0400
293
304.oom 007
.134 00.033
119 00.
.077 022
ordered
.32.008
238 p09
308 .09
.25.007
L9
.301,007
.133,00g
1118,004
.016,021
30

---

Published in Transactions on Machine Learning Research (1/2023)
Table 13: A counterpart of Table 2 regarding MAE for Task-A A and EF5 datascts
Learning
Labeling
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
AD
NNT EOT
1.146.279
1.204.251
.755.246
.740.229
.414.000
401
.48.112
423.107
1.049. 147
1.038.
1.225.1s5 1.220.195
.496 093 .468,073
.328
.328 .088
.390,oct
.344.05 .955
.176,023
.180.024
non-ordered
Hinge-IT
FOT
MT
ST
1.104.321
1.090
1.126,278
.781.212
.795,242
.781242
.399,88
.37 098
437 096
.424.198
.471 156
.471 156
1.084 171
1.084 171
1.042,176
1.263
1.259.20
1.243.192
219
.45.
.507 106
.507 106
.320 o52
.320 052
320.040
394
.344 .077
390.po8
.158,0a6
.207 0039
.077 oss
Hingo-IT ordered
ST EOT
1.092.290 292
.805.25
430 oes
.422 126
1.095.1ta
1.287.221
.504.13
.317 .000
.379.064
.155.28
1.154.298 298
.792 266
.366,08
122
1.045,161
1.238 196
.880
.325 055
.385 .033
.154.027
Hinge-AT
MT.ST EOT
1.170
.749 2311
427 103 .388,04
.111 124 141
1.075.144
1.217 109 1.178.186
.471 1098
.318 052
.377 007
.157 025
ordered
1.152,251
.764.264
1.035,191
.451 .o85
.331 .052
.387 .004
.152.027
Logistic-TT
MT 1.114,310
.813.25 253
.52 097
.459 159
1.087 157
1.230 222
.495
.333 o50
405 .056
.165 165,080
ST
1.108.301
.813.252
452 097
139 .459
1.087 157
1.230.222
.495.109
.333 050
.405 066
.165 030
non-ordered
EOT
1.110.266
.759.229 239
.408 .82
.422.44
1.052,175
1.240.
.178,082
.316 053
.392 .06
.151,024
Logistic-IT ordered
MT.ST
1.092 293
.816.22
432 099
.379 119
1.084 153
1.240 200 1.259
.43 .994
.310 oss
.378 .666
.136 025
FOT
1.136
256
.763.24
.404 099
.392
1.049.
144
.485 1885.085
314 314.000
385,,00
.137
Logistic-Ar AT MT.ST.LL LB
1.118 270
.768
.433 101
.361 o88
1.075
1.188 189
.67 .97
.308 048
.365 o69
.137 .24
ordered
EOT
1.154
265
.757 256
.420 n88
.368 118
1.046 148
1.196.180
.49 n91
.388 .488
.367 .053
.139 025
OLR-NLL ordered
MT.ST ST LB EOT
1.002.
.772.210
.45.111
.380 11
1.073. 137
1. 1.205.198
.479 os8
.310 .499
.367 .046
.136 022
2.140,262
.739,250
.407 .96
.381.19
1.058 148
1.211
.467 .770
.318,08
.373 373.054
136,024
Learning Labeling
ABA
A12
KRA
CO1
PU1
BA1
CO2
PU2
BA2
EL2
NNT EOT
668 021
.559.017
.302.02
.97.
.589
.206.010
.361 061
2774.010
.650.o17
.666.17
AD
622.a24
.560.01
.30
.296 011
.592,014
.204
.362.m3
.274.1
.658,016
.665.016
non-ordered
IlingeII
EOT
MT
ST
.661.nm
.688 .244
.688 024
.500
.160.016
.560.m
.500
.287 0122
.284 .13
2287 012
2966
.296.012
.292 013
.589,a12
.623 019
.623 o19
.205 .099
.356 .204
.25..o
.361 ff1lo122
.361
.266
.280 o11
.880
.656.a18
.672 .188
.672 o18
.709
.660.a15
.709 020
.200
Hinge-TT ordered
MT ST EOT
.688 029
286 .559.mm
.244
.617 018
.203.om
.357 .122
.73 p11
.673 020
.722 020
.662.oz
.50..05
.286.113
.293.o1 012
.590,004
.205
.354,.12
.20.o11
.658,017
.662.on
Hiige-AT ordered
MT.ST
659..2 023 6658.029
.55
284 o100
292 01
.591
.009 .204
.205
.355 .133
267
.658 .088 .65
.668 .155
FOT
.559.om
.285.011
.291 po9
.588.014
.35,012
.277
210
.660.a16
Logistie-TT non-ordered
MT
ST
.688 024
678 024
.84.017
017
.200
290..14
.944
294 012
.594 013
.594
.204
204
.357 013
.35 013
309 013
300 013
.665 017
665 007
.683 016
683 01.
FOT
.659. .24
.559,013
.28 .012
.293.011
.589,014
.203,009
.354,012
.273
.657 .117
.66 _616.066
Logistic-IT ordered
MT ST EOT
.679,22 n27
.582.015
228 011
.291 013
.595 013
.212,010
.353.o12
014 .273
273.
.667 020
.686 015
.660 034
.561.a13
.278
.290
.589,n12
.204.010
.35 .122
.857.m9
.662.a16
Logistic-Al AT MT. MT.ST.L. LB EOT
.658 023 .658.021
.555.15 .558
.278 012
.291 0010
.590.om 015
.203.om
.353.m1
261.p10
.656.019
.663 .16
ordered
.279
.290,000
.589
013
.204 0o08
.351 .011
2611 _011
.656 016
.663 01.
OLB-NLL ordered
MT.ST.L. LB FOT
.65..221 .657
.562.016
.277 277.
.291 .900
.589 013
.202 0099
.353 012
.644.013
665..19
.662 .055
.58
.278
.290
.588 016
.202
.349.0
2644 .012
.656_o18
.663 063.
Learning Labeling
POT
A1l
EL1
CAL
CEI
CE2
2DP
FRA
MVA
AD
NN'T EOT
.57.011
.549.0 001
.411.12
439,011
581 oo13
582,013
.11
.448.011
.617.028
.616,011
.592.01
.592,009
.24..0n
.241.00
.219 2119,0m
.221 .005
.019,0o5
.022 005
non-ordered
Hinge-TT
EOT
MT
ST
606
.606
.288.0s
.454.010
A644
061
.572.02
.539 o116
.616
.539 012
.527 .009
.612.0m
.634 015
.634 015
.575.011
.596
.596.o12
.242
246 on5
.266 o05
.21 .214,00
.277 0o0
.217 p05
.097 .188
.09 1911.014
.097 0088
Hinge-IT ordered
MT ST
.644
.44..11 o01
.617.00 013
.535
.632.oc .13
.589.013
.247 .004
.244 004
.015 0044
EOT
.518
437
371o12
.577.11
.610.011
.375.011
.241 004
.213 0004
.01 .014,00
Hingo-AT ordered
MT.TT EOT
.55 019
.56
.577 012
.530 010
.615 010 .613,010
.579 012
.266 o05
.216 004
.073 023
.549 018
.54.011
376,012
.529
009
.37,,111
.241
.215 .004
.067 .021
Logistie-TT
MTT'
6662.22 .20
.664.02
.617.01
.536 0336.012
.630.oo 011
.583.014
.25..00 245.000
.219 oo6
.77 .008
non-ordered
EOT
ST
.519,012
662
.54.000
064
.576.013
617.01 015
.527
536
.009
.614.01
.30 011
.568.0n
583 583,014
.242,00
245
.211 .005
.29 onG
.068 .005
078 00.
Logistie-IT
MT.T
660,as0
437
.616 014
.533 010
.630 012
.578 012
.444 004
.212 .004
.015 0033
ordered
EOT
.515.m
434.010
.577 .012
.526,00
.614.010
.566.009
.242.004
.211 0004
.01 0144,00
Logistic AT MT.TT.LB LB
.523.013
455.009
.58 0011
.53 .009
.614.011
.581.013
.245 0455.00
.212.c0
088 .408
ordered
EOT
.00,011
453,010
.577
009
.613,01
.579,011
.242.005
.211 005
.882005
OLR-NLL ordered
MT, LL. EOT
.20 012
454 010
.575.01
.57 010
.612.oo.
.570,o11
.245 005
.212 005
.072 037
.506.011
.452.10
.57.013
.526.010
.61 .611.010
.567,00 .099
.241.004
.211 .005
.068,038
31

---

Published in Transactions on Machine Learning Research (1/2023)
Table 14: A counterpart of Table 2 regarding MAE for Task-A A and EF10 datasets
Learning
Labeling
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
NNT EOT
2.466 .89
1.397 .353 1.396,379
.852.134 .200.30
.783 216 .778.187
12.297.347 .347
2.524 .23
.995 157
.721 .994
.791 .097
.362.ose .035
AD
2.554.527
2.278
2.533
1.000.149
.746
.805 .003
.361,ss
Hinge-IT
MT
2.436.a51
1.413. 404
.85..32
.911 232
2.314.351
2.479
L.066 173
.742 744.110
826 .00
.414,055
non-ordered
FOT
ST
2.444.561
2.448 533
1.424 402
1.433 3994
872
.817 .12
.819.95
.874 238
2.304 354
2.277 .255
2.498.30 316
2249
1.000.1
1.008.148
.422
.733.087
.226 001
.81 814_006
.374,043
.406 049
Hinge-IT
MT.ST
2. 2.426 534
1.456 3446
.874 1445
.830 186
2.328 .077
2.566 .558
1.085 166 1.025.149
.731 peo
.812 .844
.353 .43
EOT
2.542 480
1.468 376
.233
133
.798 178
2.259 .033
2.505
272
.739
.76
.813,8s7 0087
.347 .088
Hinge-AT
MT.ST EOT
2.406 499
1.427
384 1.504 330
844 140
.742 160 _7211161
2. 2.244 .14 2.219,320
300 2.499
2.541
.989 1.02 .992,140
.702 .044
.802 .07
.330 0300
ordered
2.476.507 507
144
336
.731 0082
.807 .092
.329,035
Logistic-TT
MT
2.356 496
1.456
411
8833 152
.825 205
2 2.303.mm 285
2.531 .08
L.042 160
.737
092
.244 .092
.384 .059
non-ordered
EOT
ST
2.522,343
2.308 476
1.447 379
1.451 407
.818,116
.865 144
.817 211
.7833191
2.239.33
2. 2.295 2887
319
2.336
2.489
.77
:397
1.033,150
.995
-737
.743
082
.002
.824 .002
.810 .099
.383 .088
.365 0433
Logistic-IT ordered
MT.ST FOT
2.330.545
1.453 365 1.487
.838 151
.841 210
22.02 204
2.606 430
1.069. 182
.732 084
.811 088
.331 .032
2.530.50 .551
.40G
.827
198 .783
2.234 233
2.607,347 .91
1.026.144
.735 pa8
.813 0087
.335,01
Logistic- AT MT..TTLL LB
2.324
1.465 419
.837 .130
.733 171
2, 2.288 270
2.520 350 2.511
.995 144
.719 093
.796 106
.312
ordered
EOT
2.526 .444
1.471.122
888..31
.732 187
2.233
2.
323
L.002. 139
.727 .092
799.006
315.037
OLR-NLL ordered
MT MT.ST.I LB EOT
2.362
465
1.465 435
842
744
2.252
2.508
318
1.008. 145
.695 087
.797 .088
.317 $17.003
2.174.621 621
1.465.408
129
.720
2.206 283
2.548 .81
L.004,311
.717 .095
.805 .06
.314,030
Learning
Labeling
||
ABA
A12
KRA
CO1
PU1
BA1
CO2
PU2
BA2
EL2
NNT
1.375.000
1.149 o26
.599 025 .598,021
.606.on
1.249 .27
.24.004
.722 222.022
.542 026
1.379.com .039
1.335,
AD
EOT
1.373,037
1.148.000
,025
_604.o15
1.251.02s
.125,012
.730,022
.532,023
1.380,o2 .029
1.333,022
MT
1.404 014
1.200 o32
.595 .595.020
.615.or
1.272 026
.426.013
.746
.541 141
1.384, .031
1.377 0227
non-ordered
Hinge-IT
EOT
ST
1.382.a41
1.403 045
1.152 .24
1.200
002
.595.020 020
.581.m
.615.017
.611 017
1.248.024
1.272 026
.419.om
.126
.736
.746
.541 019
.532
1 1.382
1 1.38/Lo31
1.33-Lo2s
1.377
.27
Hinge-TT ordered
MT.ST
1.463.on 053
1.218 032
.593.zz 023
606 016
1.395 041
.20..122
.71 021
.543 020
1.415 034
1.366
026
EOT
1.394
1.163 .244
.500000 o233
.607.00 .01
1. I.253,02
.23,,014
.733
.539
.220
1.391 .aa0
1.339
.026
Hinge-AT ordered
MT EOT
1.366 oas
1.177 026 1.149.azz
586
.596
1. 1.258 025
420,013
.720
.521
1.387 .032
1.365 022
1.375.090
384.020
1. 1.248.024 .599.
.20.o11
.719 719.020
.17.01 .17
1.382.os
1.333
Logistie-TT non-ordered
MT
1.397
1. 1.226 030
588
.611 o16
1. 1.266 026
424.o13
744.221
.51
.23
1 1.386
1.380
ST
1.397 0044
1.226 030
.588 022
.611 016
1.266 026
.24.o13
744.221
028 541
1.386 .034
1.382 028
FOT
1.383 044
1.151 024
.578.021
.609 014
1.249 .27
.420
.740
.528
1.385,032
1.333 .035
Logistie-IT IT ordered
MT.ST ST EOT
1.437 054 1.398
1.244 027
556.o22
.603 017
025 1.249
1.297
.19.013
.736.22
.555 018
035 1.402
1.363, 027
1.169
.591 022
.601 017
.119.122
.735.220
.550
1.387 .029
1.335 .024
Logistic-AN AT MT.ST LB EOT
1.363
039
1.151 025 1.148
.500 o1s
.598 014
024 1.250
1.251
418.
.722 021
.520 018
1.386
029
1.339
027
ordered
1.371
1 .099
024
.587
.598.016
.411 421.03
.720 020
.516
019
1 1.385
1.333
OLB-NLL ordered
MT.ST. LB
1.362.ou o01
1.150
024
.577 557.
.594.ox. .014
1.245.020
.18,,13
.721 722.020
.522 .018
1.383.oc. .029 1.381
1.334.027
.227
EOT
1.374.042
1.143
022
.74.022
.94.016
1.248 025
.18,012
.T17
.523.07
1.331
Learning Labeling
POT
A11
EL1
CAL
CEI
CE2
2DP
FRA
MVA
NN'T EOT
1.155.025
.899.014
1.196 019
.915.o2
1.084.015 1.142.017
1.144.016
.496.007
.458 009
.249 007
AD
1.145.022
.898.015
1.191
.913.oss
1.082.015
.455.00
.454.007
.217.002
non-ordered
ITnnge-IT
EOT
MT
ST
1.329
1.197 .024
1.329
011
041
.947
.942.015
015
1.234
1.177.021
1.234 021
024
1.094.ms
1.112
1.112 017
017
1.294.01
1.323
1.323
022
022
1.237
1.210..is
1.237 o20
020
.495,007
.502 006
.502 o06
.456 006
.450 poG
.466 006
.373 094
.295.0s8
.373 091
Hinge-IT ordered
MT. ST
1.401 o5s 1.121
.92..15
1.266
02s
021 1.138
1.421 o00
1.341
oa2
.504 007
.422 p07
1171 .075
EOT
.901 011
1.100 1.191 022
014
1.301
.119
1.230
.199
.495,000 ,006
.448 ,077
.148 ,062
Hinge-AT ordered
MT.TT EOT
1.125 021 1.126.020
.942.om 044.017
1.186
021 1.185
1.105 015
o20 1.288
019 1.205 1.277.017
1.210
.500 ons .494000
.449 007
.210 081
.40..14
11800000
1.102
.447 .077
.169.00
Logistie-TT
MT
1.447 .35
.945.01
1.237 .22
1.113,017
1.314.z22
o22
1.233 021
499 .077
.461 .010
.299 105
non-ordered
EOT
ST
1.446
1.189 ,29
035
.940,,06
.945.01
1.183.02
1.237
022
1.095.01
1.113
1.314
1.291..0
oz2
.018
1. 1.233 021
1.212.01s
.499 007
.495,006
.451,00r 1007
.461 010
.232,09
.79 105
Logistic-I'T
MT
1.279
032
1.256 025 .900.015
.943 022
1.219 020
1.124 019
.496 007
.451 .oo
.153 044
EOT
1.119 m21
.905.015
1.214
.933
1.172.018
1.093.m14
.494.006
.007
.138,063
Logistic AT MT.ST.LB.
ordered
EOT
1.128,020 020
1.132
020
.939,016
.442 m14
1,191
021 1.188
020
1,107 015
1.103
.19 1.280
1.287
.20
1.214.018
1.221 .24
494,006
.488 p07
.446 4,460000
.450 p07
.197,071 ,071
.27 080
OLR-NLL ordered
ATTSS.LB
1.198
023
.940 015
1.183 022
1.096
016
1.287 018
1.197 019
.498 007
.448 006
.190 087
EOT
1.167 o21
.938,077
1.175.020
1.092 011
1.284.020
1.194.015
.495.006
.447 .006
.18 .o5s
32

---

Published in Transactions on Machine Learning Research (1/2023)
Table 15: A counterpart of Table 2 regarding MAE for Task-A A and EL3 datasets.
Learning
Labeling
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
AD
NNT FOT
.330 .10 .32
.184.122
176
138 .88.
126.065
.069 p47
.073 .02...222
.351 1884
.352.092
.621.116
.631.
.046
.048 039
.181,48
.201 .040
.210.04
.226 054
.041.017 017
.043
non-ordered
Hinge-IT
FOT
MT
ST
346
147 147
346
.193.
.189.085
1193.
150.082
150 062
124.062
.070.oa
.070 053
.075.047
.355,084
.342 086
.342 086
.612 103
.631 093
631 .093
.048.077
.048 037
046.038
.168,042
.193 048
.133 .48
.199 041
.193.o83
.199 941
.036,012
.032.012
.032 .012
Hinge-IT
MT.ST
.324.15
.185.111
141.1ss .553
.073 045
.336 .79
.613
.050 007
.17 041
.201 L002
.031 .012
ordered
EOT
165
.183,115
1221,000
.072 049
.348. 078
.620,00 106
049..03
.170,0a3
.194,036
0336,013
Hinge-AT
ST EOT
340 158 371
.179..00 102 .187.088
147 002 121.062
080 046 0800.47
.329 079
.633 107
050,.06
.194 043 .168,044
.077 .48
.030 ,014
ordered
.34
.073
626
.047 .039
205 044
.044.011
Logistic-IT
MT
.392 145
.23 122
151 157
.068 045
.365 098
649 107
056.0a7
.168.o41
186
.029 .019
non-ordered
EOT
ST
.322 145
.102
.196.109
.223 .22
.124,00
151 075
.078 .049
.068 045
.365 096
.352.88
649 107
.625
$225,09
.056 037
.53
.168 041
166 1661.64
.144,066
186 046
.029 0011
.030
Logistic-IT ordered
ST
.394
.28 140
146 063
006 045
.349 074
649 .997
0566.02
.168 039
162 162.035
.029 .011
EOT
428.140
.212.123
.122
.079.051
.345 0,86
621,100
.05 055.035
.162.02 ..22.02
.175.045
.031,012
Logistic- AT MT. LT. EOT
408 128
.219 124
146 061
.072 043
.341 096
626 106
.058 035
.166 .433
.169 .039
.030 000
ordered
.106.18
.199.114
.121 .550
.071.043
.37 .098
.620.10
.055 .006
.161,00
.176 .038
.030
OLB-NIL ordered
MT..T.. LBB EOT
339 135
.213.129
.139 064
.071 047
.349 096
.625 109
.058 036
.167 .099
.175.04
.031 .012
118
.207.118
.119,051
.079.051
349.094
.612.093
.054.0a
165.039
.183,045
.030,a10
Learning Labeling
ABA
A12
KRA
CO1
PU1
BA1
CO2
PU2
BA2
EL2
AD
NNT EOT
.337 013
2338,013
.030,o05
.030.om 004
118.011
.194.014
.012.004 004.
_011.003
.285.011
.287 012
.055.o0 .055 nes
.015 o03
.015.003
. .120 p007
.226 pos
.084.nom
.092 009
.062.004
.061,00
non-ordered
Hinge-IT
FOT
MT
ST
.366
.236
236
.030 .044
.030,000 .000
.030 004
.150,o00
119..11
150
.010 002
.011
.010
002
003
.273 011
.273 011
.274.00
.056 007
.055
.056 o07
.013
.013 o03
.013.o08
.110.000
.110 110,000
113
.085_oo
.888 000
.088 .0.6
.062 000
.06 161l
.062 .044
Hinge-I1 ordered
ST
.366
.030
147..12
.011
.02
.274 o10
.054 0G4_006
.013 0022
.11 .000
.087 .006
.062
EOT
.236.012
.0300.00 004
145.011
.011
002
.274.010
.054.006
.013.0os 0033.003
114 po0
.086
00.0066
.061,004 0004
Hinge-AT
MT.TT EOT
.34.om 001
.030
144 009 143,010
.010 003
.272 011 .273,012
.055 .06
.013 002
.15 010
.88 2005
.062 202.004
ordered
.235,012
.030.004
.010.003 0003
.054,005
.013,003
.117 .010
.085.00
.061 1004
nou-ordered
Logistie-IT
EOT
MT
ST
.231
.222 .13
.222 2322,13
.029 004
.029 004
.030,044
.138 008
113 o0s
1138
.011
.011
.011 0033
.003
003
.270 00.9
.269 0090.o1
.269 010
.053 005
.054
.053 0005
.044 003
.013
004 .003
.098 00.
.098
.098 00s
008
086. .07000777
086 10.0066
086 .007
.061 .004
.061 .o04
.061 .004
Logistie-IT
ST
013
.029.0m 0029.004
.133 008 131
.011 002
.269 010
.053 004
0013 003
.092 .07
0085 00.
.061.00 1004
ordered
EOT
.229.012
.300.o5
.010 0022
.270.010
.054.005
.033,002
099
_007
.86 D866006
.061 1661100
Logistic- AT MT..ST.LL.
.231 .144
.029.004 .030.044
.133 009
.011 002
.269 .199
.053.00 .005
.014 4.0.
.092 .006
088 0o.
.061 .004
ordered
EOT
.24
131
.010 002
.270 .09
.054.005
014.,03
092
086 1866007
.061.00
OLR-NLL ordered
MT.T LB EOT
.231o11 .231,113
.029.004
.132 007
.011..00 L003
.271.010
.013.002 .054.005
.093 007
.07.000
.007
.061..0
.030.o04
132 008
.011 .000
.270
.053 063,,05
.013 113,,o3
.093 .007
086 .00
.061
Learning
Labeling
POT
AI1
EL1
CAL
CE1
CE2
2DP
FRA
MVA
AD
NN'T EOT
.07..om
.075.006
.069 .05
.0688
.018 .022
.019 002
.222 006
.229 pos
.035 002
.035 003
.043 0044
.044 004
.095
.094 Lo03
,003
.125 .113
.123 .008
.002
.001 0000
non-ordered
Hlinge-IT
EOT
MT
ST
1087 005
.077 .005
.087 005
.069 005
.069 005
.069
.05
.019
.019 002
.018 002
.022
.2 28 006
.219 0o6
219 006
.033 002
.033 002
.033 003
.044 oo3
.044 0044.003
.044.on
.095 003
.095 .033
.095 ,033
.092.n0 .00
.094 007
.094 007
.001.000
.002 0011
.002 001
Hinge-IT ordered
MT. ST
.889 gos
.069 005
.019 002
229 oo6
.033 003
.04 004
.095 003
.091 D05
.002.co 001
EOT
.077
.068 0068.08
.0 8
.218 007
.033 003
.038
.095 .003
.090
.066
.002 2011
Hinne-AA ordered
MT EOT
.085 005
.068 o05
.019 002 .018 8
.222 006 .222 222_.07
.033 002 .033
.037 003
.094 003
.091 .005
.002 001
.077
.069
.04
.038
0338,003
.095
.090
.00
Logistie-TT
MTT'
.55..00 004
.071 .004
.019 10...02
.204 oos
.033 002
.043 003
.095 .003
.084 003
.002 .001
non-ordered
ST
052
.071 004
019 002
.204 005
.033 002
.043
003
009 .033
.04
003
.022 .001
EOT
.037 .004
.069.065
8,002
.203.000
.033,002
.041 1,03
.095.033
0833003
.001,000
Logistic-l'T
.054.00
.070 o05
.018 002
.204 006
.033 002
0038 003
.09 oon3
.081 [881.004
.002 .000
EOT
.038.00 005
.069 005
.017 002
.203 006
.033 002
.038 003
.095 003
.081 1000
.001_.000
Logistic AT MT,TT.LN LB
.040 go5
.070 .055
.019 002
.204 pos
.033 002
.037 o03
.095
.082 003
.002 000
ordered
FOT
.036.004 004
.069 000
.018 .02
.203 2203.005
.033 (33.
.038 003
.095.083 0033
.081 10001
1001.000
OLB-NLL ordered
MT.ST.LB EOT
.039 004
.070 005
.019 002
.204 006
.033 002
.041 0033
.095 0033
.082
p00
.002 000
.035 004
.009 .00
.018 n02
.202 006
.033 002
.041 L003
.055 1035
.081 1003
.001 .nnn
33

---

Published in Transactions on Machine Learning Research (1/2023)
Table 16: A counterpart of Table 2 regarding MAE for Task-A and EL5 datasets.
Learning
Labeling
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
AD
NNT EOT
.520.155 2654218
.98.182
.48,1667
.233 0er 213,0s8
.152.993 093
.150
.757.140
.70
1.126.156
1.123.170
.103,059
.102 .054
.268 1685
.269 .o3
.288 2888
.29 .499
.167,030
.168.025
HTnge-IT
MT
658 270
.521 221
216 092
.168 p90
.743 147
1.109.16 164
.144 oos
.283 054
.321 .499
.008 0s3
non-ordered
FOT
ST
2656,21
.658.22
.532,13
.12.200
.288.088
219 092
168 089
186.112
.739 142
.742, 136
1.123.179
1.111 .64
114 .o50
121 .073
.259.mm
.283 054
.321,49
.297.042
.208 0333
.164.024
Hingo-IT ordered
MT.ST
.670.219
.531.61 161
210.007
174 086
.744
1.121.135
119 .09
.270 770.047
.922
.147 .22
EOT
.668.21
.329.16 164
.205 n86
.176 083
.759.135 135
1.0994.42
.112 .66
.262 D49
2900,42
.145.025
Hinge-AT ordered
MT ST
.722.215 215
.500 153
195 ons .205.092
..6 .07.077
.732 132
1.115
113 .633
.267 D46
.305 ons
.153 030
EOT
6644.199 198
327,,166
1688,88
.749,
122
1.133,178
114,080
.264 po5
.295.04 .445
142,025 .025
non-ordered
Logistic-TT
EOT
MT
ST
.672 203
.540.17 .670.11 .525.172
.600.18
.565 214
.236 097
.35 097
.219 088
.174.77
.181 o88
.757 134
.763 145
.734
1.151 .004
1.136.194
1.152.204
135. 073
.139 .981
131 .007
.267 D47
.26 052
.267 047
.318 048
.318 .488
.288,000
.145 .145.023 .023
.145 152.
.141 (,27
Logistic-IT ordered
MT.T
.676 202
.531 169
.237 098
.168 081
.741 1:30
1.141. 165
.144 068
.200 D45
.279 043
.129 021
FOT
2676.201
.517
.216.085
[185,10
.737 .737.139
1.121
.130,067
.249 .033
.287 .047
.127 127.029
Logistic-Al AT ATTTTTLL
680
.528
.221 .933
.161
.747 127
.14 1.119.165
1.114
.132 070
.252 044
.279 .042
.130 01.9
ordered
EOT
266
.51
.225 n87
.168 077
.763 126
.127 .066
259 p57
.29 Lo61
.130
OLR-NILL ordered
ST..IN EOT
658
.553
.225
.622 085
.745 140
1.161 .661
.135 071
.254 044
.283 .042
.129 .221
.652.117
515,165
.25.008 $15.093
1668.01
.744 120
1.134.10
.133 1333,66
.253 .500
.284 0,42
.129 .1.9
Learning
Labeling
ABA
A12
KRA
CO1
PU1
BA1
CO2
PU2
BA2
EL2
NNT EOT
.284 Lo2s
.157 009
.223
.052 006
.521 .014
.113 009
.069 .00G
.164 p09
.207 3007.09
.178 .099
AD
.211.199
.140,009
.221 m12
.050,000
.519.014
.110.00
.067 .006
.160.a10
.204.009
1168.0s
non-ordered
Iline-II
EOT
MT
ST
.268
.688
.238.015
.154.010
.138
.11..o1
.234 o01
234
.231 .144
.054 006
.051
.054 006
.565 0:0
.522.n12
.565
.111 .008
.108
.111 111.008
.069 .007
.067 .006
.069 .07
.164.am
.202 011
.202 o11
.227 .012
.206.a1
.227 .022
.178 00s
.167
.178 ons
Hinge-IT ordered
MT.T
268
.146.mo
.221 014
.052 oos
.544 013
.111 oo8
.069 00G
.169 po9
.226 012
.177 oos
EOT
.234.01
.137
.218 015
.051 005
.521 2018
108 .009
.067 0006
.161 1,00
.205
.167.00
liige-AT ordered
MT.ST
FOT
.237 .35
2600
.147
.137
.28 013
.224 .114
.05 .006
.053
013 .519
.525
.111 .008 109
.068
.007 .066
.158
.164 010
.213 012 .204
.176 109
.167
non-ordered
Logistie-TT
FOT
MT
ST
.233.016
236
.236
.137 009
.138,009
.151
221 014
.217 2017
.213.012
.050 J00
.054 005
.054 oos
.520,012
.525
.525 015
.107 .007
.106
.106 106.008
.66 006
.064.006
069 .00G
.160
.209 010
.209
.204.010
.208 .011
208 011
.171
.167 .008
.168 ona
Logistic-IT ordered
MT.ST ST EOT
.322 .16
.137 010
.213,014 o14
.053 pos
.525 014 .522.013
.107 000
066 09G
.161 o12
.008 .100 .204.a1
.168 008
.232.016
.137 008
.210
.050
n06
.107 o077
.063 n96
.157 .010
.167 onn
Logistic-AN ordered
AT
MT ST. LB EOT
.230.015
.137 008
.219 015
005 .050
.052
.524 014
.108 007
0044.oc
.160 1110.11
.205.011
.167 ons
.231
.137 0099
.116,03
.522
.107 008
.063.00
.157 0111
.207 .012
.167,008 0008
OLB-NLL MT.ST.L. LB FOT
.230.a18 .231
.137,010
.212.014
.051
oos
.522. .522.013
.106.007
.065 .07
.159 1.59.o10
.205 o10
.167,009
ordered
.137
.209
.050 p05
.521 2014
1006
.063
.158 .011
.007 .013
167 an8
Learning Labeling
POT
A11
EL1
CAL
CEI
CE2
2DP
FRA
MVA
NNT EOT
.049 044,,01
.174.007
.045 on3
.387 0099
.106 no8
.140 0008
.172
.142 006
.004 001
AD
.046.004
.171,006
.046.004
.35
.105
.132.000
.167.00
140
.003.0m
Hinge-TT
MT
ST
.17oor
a17
.175.008
.175 006
.062 007
.022 007
.45.o11
445.
112 007
.111 .008
.140 0o0
.140,000
117 .o03
.177
.163 009
.163 0099
0032 005
.32 .055
non-ordered
EOT
.150, 0020
.170
.07
.051,004
.421 0008
.103 .005
.124.00s
117. .,044
.152,000 000
.006
000
Ilinge-IT ordered
MT
.038,000
174 006
.051 0044
431 009
.133 014
.1400 ooo
.172 .066
_013 .1433
.004 001
EOT
0036.00
170 D06
.048 .044
.420 nn8
.103
.125.000 _oos
.165,000
.137
.003
Hingo-AT ordered
MT.ST
.031.004
173 00G
.055 005
.421 009
.106 005
.139 .00
.175,005 000
.146 006
.004 001
EOT
.030,003
170
.0499.00
4400
.103
.118
.167 .005
.138
.003 ,011
Logistie-IT
MT
.100.008
.17. 177.ons
.057 o05
.42 009
.105 008
.112 .00
.172 .005
.144 005
.043 .008
non-ordered
EOT
ST
.075 .007
100
.169.00 .007
174.
.050,004
.057 005
.421.007
.42
.102. 004
.103 oos
.109 ..09.0es
.112 000
.163,000
172
.136 1336,00
144 005
.007 .007
044
Logistie-IT
MT.T
.030
.167 .00
.046 003 .016.003
.43 010
.104 0os
.112 005
.163 .004
.137 .004
.003 001
ordered
EOT
.029.004
0006
.422.008
.102.005
.110,0es
.163,004
.135,004
.003.001
Logistic AT MT.TT.L LB
.029,000
.171 Loos 1169.00
.053 004
.42 .099
.1022 005
.111.00 |loos
.13 .004
.137 .044
.004 .001
ordered
EOT
.028.003
.049,000 .000
008
.102,005 005
.111,0s
.162,003
.135 ,00
.003,001
OLR-NLL ordered
WIT.ST.LBB EOT
.029 .03
.77 007
.050 004
.40 0os
.101 005
.109 0o5
.163 004
.136 004
.010 003
.028,004
1669,07
.048 003
.420,007
.101_005
.108 .108,005
.162 000
.136 1.36,004
.007 .001
34

---

Published in Transactions on Machine Learning Research (1/2023)
Table 17: A counterpart of Table 2 regarding MAE for Task-A and EL10 datascts.
Learning
Labeling
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
AD
NNT EOT
1.412,376
1.348,309
.852.252 .888261
.523.14
336.149
.391 150
.431
1.361 224
1.351,247
2.412.ca 2.366
289
.261.117 .288.10
.558 .911
.563
.537,077
.547.000 .009
.289
.286.229
non-ordered
Hinge-IT
FOT
MT
ST
1.354.416
1.412,374
1.35-4.416
.81.31
.699 295
.872.26
.553.66
.513,17
564.162
.414.16
.561 170
.523 155
1.343 216
1. 1.336
.22
2.369 .70 2.378 333
2.349
322
.265,116
.272.128
.278.131
.542.779
.579.077
.585 077
.555,am
.582
.583 o77
078
.295_ax3
.382
.322 .029
Hinge-IT
MT.ST EOT
1.376.422
.841 1211
.522.17
.428 171
1.343
2.380
.309
.275
.571 Leo
.553 088
.282
ordered
1.366.391
.851 248
.515,166 166
.452 168
1.335.260 260
2.341
298
.259, 100
.537.067 .087
.550.00 .004
.281 281,g30
T
MT.ST EOT
1.322. 363
.844 224
.525 147 .493,14
.47 188
1.334 231
2.350.31 .122
.287 120
.552 D83
.542 072
.285 028
ordered
1.358,40 .01
.868,22 255
.456.12
1.339.25 ,29
2.365,320
.265,111
.549 ps8
.555
.276.02
Logistic-TT
MT
1.366 407
.944 261
.578 158
.583 200
1.385
230
2. 2.368 359
.294 144
.559 .or7
.552 .73
.307 .001
non-ordered
EOT
ST
1.352.41 411
1.35-4
.897 241
.840,256 256
.572 155
.513,18
.538 153
.526 227
1.378 255
1.35 |50
2.354.358 358
2.394
.318, 1442
.269
113
.561 078
.535
.551
.537
.79
.04
.280
.307 001
Logistic-IT ordered
MT.T
1.350 385
.876.2m 284
.537 .537.15
.378 187
1.383 203
2.383 .347
.277 115
.552 .90
.557 .078
274.o27
FOT
1,368 .644
.833 244
.510.15
.407 208
1.372
2.395.30
.275,140
.536 D86
.544.080 0000
.274.020 .226
Logistic-Ar AT MT..TLL.
1.300
.811
.530 530..48
.46
1.349 .23
2. 2.298 271
.275.129
.542 093
.546 083
.267 027
ordered
EOT
1.342 .372
.843.20
.14.141
.446 207
1.345 215
2.307
280
.208 121
540 .74
5600074
265.025
OLR-NLL ordered
MT.ST ST.LLT
1. 1.362. 364
827.
.518
.368
1.356
200
2.377 .004
.269 269..30
544 096
.544 .73
.263 025
EOT
1.360,380 3,9
.857,200
.486,135
.398 214
1.361 219
2.
2.365.34 ,319
.267 211
529 088
.5444 6-44.066
266.027
Learning
Labeling
ABA
A12
KRA
CO1
PU1
BA1
CO2
PU2
BA2
EL2
AD
NNT EOT
.34.o21
534.026
.215,012
.228
.17
417.017
.167 009
167
.995 020
.994.om
.251.o00
.253,o13
.21..10
2114.11
.322.014
.327 .014
.516,022 6,22
527 .024
.326,01
.331
non-ordered
Hinge-IT
EOT
MT
ST
.607 a30
6666.29
.549.am
.229,012
.229 .122
.213.0m2
.417.02
.135.016
.45.015
.164
.66 160.010
.168 oos
1.068 oao
1.068 030
1.000.
.247.on
.253
.253.o13
.21
.207
214
.329
.353 017
.351 015
022 .5413
.512.02
.543 022
.326,a1
.332 012
.332 .122
Hinge-TT ordered
MT.ST ST EOT
.582.o2 029
.28..1
436.019
1611 0os
1.130 n31
.247.012
.208.om
.342 013
.639 926
.332 0112
.550,0m
.210 001
.311 o20
161 009
1.003,az
.247,012
.207 001
.312,014
.511,020 1,20
.325.00
Hinge-AT
ST FOT
.557.08
.224 013
427
163 oos
1.014 .255
.249 .19
.08 012
.326 012
.524 .119
.332 .011
ordered
.533.m
.211.o13
019 .47
116.
.992 nos
.248.011
.207 o01
.324
.510,m9
.323.012
non-ordered
Logistie-TT
MT
ST
.562 029
588
.213
21T 012
442 019
421
019
165 007
167 009
1.016 024
1.016 024
247 012
.247
.008 010
.088
352 015
.355 016
.21 023
552 .023
323.oL1
.25
FOT
.536 .24
.208 011
.10
.66 m10
.989 ,011
.246,011
.198
.327 .114
.51
.22
.321 .011
Logistic-IT
MT.ST EOT
.548.0z 027
.206.022
421 .o18
159
009
1.028 023
.245
.199.01 .19.011
.31 .113
.547 024
.321,a11 _111
ordered
.540 n27
.207 011
420
158 007
.989
.243.o11
.200.011
.336pp1
.520 ,21
.320 011
Logistic-Al
AT MT.ST LB
.529 .20 .535
.206.11
.477 021
161 007
.993 022
246.0111
201_n10
.322 .12
.511.21
.30 .012
ordered
EOT
.008
.216.20
159
.991
.245,012
.200,100
.320 014
.51 .,22
.321
OLB-NLL ordered
MT.ST. LB EOT
.526,0m .220 .534.02
.206.010
.408.20
.15 oos
.989 020
.244.011
.000.02
.322 .144
.513 021
.320.011 .011
.206.011
406.019
1158
.990 021
.244
.202
.323
.510 021
.321an
Learning Labeling
POT
A11
EL1
CAL
CEI
CE2
2DP
FRA
MVA
AD
NN'T EOT
.321 033
312.a37
.300.om
.298.008
.271.os
.271 .006
.741.14
.746.oz
.285.010
.284.010
.283,009
.283.ou
.316,004
.319 o05
.262.
.63 005
.041.004
.050 001
Hinge-TT
MT
ST
.209 022
.211 023
.329
.329 014
.3227
.327
.867 013
.870
.314
.314
.388 012
.348 018
.332 006
.388 005
.308 007
280 006
.067 007
080 oos
non-ordered
EOT
.196,02
.300,ess
.301,008
.855,mm
.300
.324.01
.317 7,04
.269 poo
.039 ,003
Hinge-IT ordered
MT. ST
.392 n18 .257
.300,om o09
.25..00
.772.015 o15
.330
.376 oss
.325 006
.69 004
.047 005
EOT
.298 9o8
.26.00G
.726.012
.307 .011
.339.0m ,010
.316 ,055
.268 006
.038 .004
Hingo-AT ordered
ST
.195 .163
.300 o07 .297
.263 006
.737 014
.312 011
.324 o10
.326 006 .3 1316,000
.270 pos
0051 006
EOT
264.006
.733
.295
.315,010
.268 .005
.043,00
non-ordered
Logistie-TT
EOT
MTT
ST
.137 .013
.153.015
155
.316.011 .311,010
.298 0o8
.309
.292.00 .077
316
.83
.862.oz o11 864
.302 010 .322 .298
.313,0
.324.011
321
.35, .007
.316 3316,004
331 OoGG
.94 000
.309 o10
f268,005
.038 .003
.079 .007
.071 |_1100
Logistie-IT
MT.T
.235,019
.292 ons
.262 007
.756 013 .723
.283 009
.342
.319 004
.260 .044
.038 .004
ordered
EOT
.165,014
.202.007
.261.077
.272.009
.316,01
.317 .98G
.260 .004
.033.003
Logistic AT MT.ST.LB LB
.183,014
.300.08
.844 007
.85L.111 o01 .849,010
297..10
.33.,00
.32 .05
.268 .005
.043 .004
ordered
EOT
.155,01s
.29T 007
.275,008
.295,010
.310,000
.316 ,00
.266 005
.037 _037.005
OLR-NLL
AT.SSTLB EOT
.151
.208 007
281 007
.888 012
.295
.307 307,o01
.32 .04
.267 pos
.040 pos
ordered
.135,011
.297.008 .08
.275.007
.866.01
.295.010
.304,011
.31,,004
.266,005 0,0
.034.005
35

---

Published in Transactions on Machine Learning Research (1/2023)
Table 18: A counterpart of Table 2 regarding RMSE for Task-S and EF3 datascts.
Learning
Labeling]
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
NNT EOT
877.174
.612 121
.380 075
.489 .o8
.897 100
.935 111
.535.076
.459 053
.533.047 047
.282 .336
AD
884
.622.
135
.372,085
.486.106
.889
.930
.521.72
.429,04
.532
.279 0337
uon-ordered
Hinge-IT
FOT
MT
ST
888
887 200
887 200
.639
.628
.639
118
106
118
.391
.31.o76
.387 .065
.470
.470 o98
.463,102
.878
.896 .95
.966 .95
.919.103
.937
.937.122
.524.070
.528,
528.0r4
.431.am Lase
.459.0a
.49..
_517.06
.520
.520 649
.282 6335
.273 .273.035
.282 035
Hinge-IT
MT.ST EOT
.894 190
.649
11.9
.382 .80
.478 .99
.91 1. .112
.918.
.520
.447
.515.0s0
.282 .036
ordered
.97
.613
100
.375,05
.450,77
.884.114
.9141.997
.522.065
.434.044
.518,04 044
.275 236
Hinge-AT ordered
MT.ST EOT
.908 1655 135
.639 102
.381 078 .377 .770
.883
.900. 118
.915.1 112
.510.074
.450.053
.517 p45
.277 .336
.598.10s 108
.475,087
.893,005 100
.931,119 119
,521,075
.436,018
.513,ou8
.274, .274,036
non-ordered
Logistic-TT
EOT
MT
ST
843 183
.871.14
843 183
.672
.672 141
631.
100
141
388
.377
.388 086
065
086
.884,07
491.oss
19ssoss
.872,00
.914 4 097
.914 0194
.938 118
.938 118
.920.115
.508 077
.504,ora
073
.422.045
.44
.444,088
.516
.502.01
.278 .335
.278 .:35
.279 .035
Logistic-TT ordered
ST
.819 178
.61 119
339 097
.96 085
896 105
.920 127
.519 077
.411 052
.502 048
.69 pa5
FOT
887
.36
0997
358 077
.473,07
.878 100
.933 117
.515,074
.425,042
.500
.271 .o35
Logistic-ATT ordered
MT
828 130
6645 116
369 095
.85 .94
.900 099
.918 107
.513 075
434.0a
.491 049
.273
LB
855 199
654
11.5
373
093
.85 088
.903 093
.919 119
516 .517,083
.441.48 .o48
.493 048
.72 035
EOT
180
663
102
.60,077
.76,082
.883 107
.925.116
.427,06
.966,53
.275.030
OLR-NIL ordered
MT.ST
840 129
663 116
368 092
491 .90
.892 091
.908 099
.506,8s
.29.08r
.455.099
.270 p36
LB
867 187
.63
129
370.094 094
.98 110
.82 099
.900 118
580,00
.44.056
.493 045
.270.040
EOT
878 146
633 098
.35..81
.45,0722
.882 112
.912 109
.520.885
.422
.493
.275.032
Learning
Labeling
ABA
AI2
KRA
CO1
PU1
BA1
CO2
PU2
BA2
EL2
AD
NNT EOT
.640.on .61
651 021
.610.011
.613.011
.22.o11
.46
.419.010
.119,10
.597 .012
.597 0144
.320 012
.327 ,012
.466.10
.456,012
.009.01
.408,010
.624.014
.30 014
.67.
.623.o1
non-ordered
Hinge-IT
EOT
MT
ST
.637.01
660 022
660 022
.610.00
631 013
633 013
.396,012
.39..012
.97 397
.416.m
.416.011
.115,011
.593,012
.593 011
.593 01l
.318,01
.319
.319 01s
.456.012
.454,01
.456 o12
.404 012
.399
.404.02
.624.015
640 017
.64 017
.622.010
.629 013
.629
Hinge-IT ordered
MT.ST ST
655 023
631 Q14
397,.13
.166.00
.594 011
.318 012
.455.01
.399 o11
640 614
.226 013
EOT
.636,01
.611.010
.366011
.11,0010
1594,01
.317 013
.454.11
.398,012
.624,014
.623
Hingo-TT ordered
MT ST E0T
638
018
620 014
.355,03 013
.16..11
.591 011
.318 01s .316,012
.544 010 .53,010
.401 o10
.628 013 .623
.624 o11
638
018
.608.010
.38
0013
.16,011
.592 0011
.398
.623
Logistie-IT non-ordered
MT
.64 p23
.088 011
.93 012
.13.om
.593
010 .593
.318 01
.448 009
.418
.636 o14
.625 .111
ST
023 643
608 011
393
.13,011
.318 011
.488
009
.418
.66
.25 011
EOT
.634.016
.604.010
.91,011
A11.011
.591
.317
.448,010
.397 .100
.623,014
.622 2011
Logistic-ITT I'T' ordered
ST
640 020
607
.391 0012
.413.011
.593.o11
.318 011
.449.00
.401 .011
.635
.62 001
EOT
.633.a17
.04.000
.39,,00
.412.011
591,009
.316,012
.46,010
.399
.623,013
.621.o11
Logistie-AT ordered
ST
LB
.32 020
632 018
604.010
604 604.011
.322.12 .122
333 012
.43.010
.13,0000
.589
588 010
012
.317,012 0012
.317 012
.450
449.01
.398
.397 o0
.627 0155
624
.624 .624.111 011
.62 010
EOT
632
602.010
392.013
.13,012
.589
.316
.47.009
.35
.63
.622
OLR-NIL ordered
MT.STT ST
LB
.32 019
633 018
.044.012
605
390
390.o12 010
.412.01
.413.011
.590 .100
.590
.318,012
.317
.488 .09
448.010
.39T
.366.11
.23..13
.2T.o14
.621 .011
.622
EOT
633.o17
6603009
390,012
.11.,01
.591.01
.316,013
.447,010
.395,02
.623.o15
622.o11
Learning Labeling
POT
A11
EL1
CAL
CE1
CE2
2DP
FRA
MVA
NNT EOT
564 [G74
501.oos
.766.09
.57
5883 .099
.559 007
.366.om
.345.0
.055.ozz
AD
564_0o0
.500,009
.575,00
.535,008
.580 .008
_557,007
.367,005
.344.005
.093,024
non-ordered
Hinge-IT
EOT
MT
ST
.671 Loas
.607.am
.671 pos
.49..00
.47. .00
.47 .09
.578 oos
.575.00
.578 o08
.524.010
.526
.526.010
.604.ons
.630 009
.630 009
.609
.588,mm
.609
o10
010
.370
.386 011
.346
.348 one
348
.084.00r
.084 .084.00 007
.080
Hinge-IT ordered
MT.ST ST
.600 pos
.497 .00
.578 ons
526.010
.576 009
.557 oos
.366.0o 005
.347 00G
.086 o08
EOT
.559 .007
497,000
.575,009
.24,009
.573.om
.555
.365,004
.345 .006
.082,00
Hinge-AT AT ordered
MT.ST ST
562 007
.466 .0.0
.576 010
.525 .00
.571 .009
.555 .os
.366 004
.347 o00 .34,.005
.090
8OT
.661 .007
496.009
.75,000
.23
.569
.008
.553
6653.009
.365,005
.086
Logistic-IT
MT
561 .000
493 .0.008
575.010
.507 007_008
562 .oo
582 .os
.3555 065.005
.38 .00
.076
non-ordered
FOT
ST
.500 008
561 Do8
493.ons
4944,099
.570,p00
.55 5575.10
.505.008
507.008
.558 p07
.622 008
.582 008 .569 pos
.365,004
.36..00
.345 oo6
348 006
.073 .007
.076 D07
Logistic-IT ordered
MT
560 007
193.o09
.575 009
.100 009
.561 pos
.547.00s
.365 065
.341 .005
008 035
EOT
.500 p07
.493 o93
.570
pcc
.506
.558
000
.545 007
.35,004
.341 .oo5
.083 .36
Logistie-AT ordered
MT ST
LB
560 007
560 pos
.433 008
494..00 304_009
553
.570 op9
009
509
.088 009
009
.588
.588 .07
008
009 .881
.579 0007
.365 004
.365 .044
.345 000
.316, 006
.125 076
.255 0775
FOT
559 po7
493 030
.570.ns
.507
.007.00
.557 .oo8
.576.ons
.365.004
.34,006
.122 224
OLR-NLL ordered
MT
LB
560 007
560 007
.493 0099
493
.571 .007
.574.00
.508 0008
509
009
.557 .007
.559
007
.575
.573 oos
008
.366
.366 004
.346 006
.345,006
.113 068
113
FOT
659.007
493.009
.572,088
.507 .09
.556 007
.570,00s
.365,004
.344.005
.108 .67
36

---

Published in Transactions on Machine Learning Research (1/2023)
Table 19: A counterpart of Table 2 regarding RMSE for Task-S and EF5 datasets.
Learning
Labeling|
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
AD
NNT EOT
1.435 284
1.399.
1.021.26s 268
1.051.2r4 .25
.657,088 0085
.64.6oc
008
.776.131
763.133
1.407 196 1.385.215 215
1.454.1s. 181
1.445.17
.728
.769 09s
,788
.54.065 .604
.712,033
.714.08
.419.03 031
.425 025
non-ordered
Hinge-IT
EOT
MT
ST
1.444.29 291
1.339. 250
1.431.29) 291
1.073.272
1.068.306
1.073,272
.702
.662.0
702
103
103
.793,128
.825.150
825.150
1.362.194
1.457 245
1.457 245
1.539
1.444.88
1.539
243
.802 104
.802 104
7166,09
.592.o55
.83.054
583.054
.695.0a 6.95.665
.702
.702 oas
.454.043
.397 .033
.454
Hinge-IT ordered
MT.ST
1.426 289
1.059.
.685 057
.826
1.467 221
1.532 220
.803 118
.580
.677 077
.393 035
EOT
1.377,209
1.069 .044
.651.ss
.771,127
1.393_203
1.4644 181
.736
.379.089
.693.064
.392.038
linge-AT AT
MT.ST EOT
1.430.29e
1.027
.691 100
.758 147
1.425 192 1.373.175
1.442. 160
.750,00
.580,057
6600.oo
.395
ordered
1.415,271
1.060.326
.666.084
.743,133
1.419. 185
.724.004
.500,60
.692.662
.390,037
Logistic-IT
MT
1.420.281
1.089.29
722 102
.829.
1.465
1.480
.779 115
.02.oom
.715.ee
.406
non-ordered
FOT
ST
1.413. 277
1.378
1.089 204
1.025
.659
.722
es7
102
.774
.829 190
1.465 205
1.391 .214
1,480 251 1,458
.733
.779 115
.602.om 6602.061
.580
.715 083
.697
.390 .038
.406 0.3
Logistic-IT ordered
MT.T EOT
1.439 283
294 1.100..20
.692
110
.730 161
1.471
198 1.377
1.518 217 1.441
.774. 110
-573.062
.675.or 074 .677
.369 .035
1.371.248
1.031. 260
.663,
103
.737.137
.742
090
.385
.369
Logistic-AT ordered
MT.S7 ST
267 1.339
1.008 .100 1.089.2m0
703 087
706.104
1.386 183
1.415 145
.730 088
.57T.066
.670 oss
.369 .322
LB
1.374.270
.99
.715
712,10
.697.139
1.426.199
1.447 204 1.450,16 168
.998 .746
.56T .864
.655.00
.369 .033
EOT
1.421,263 263
1.054.292
.680
100
.715,152
1.370.
7,88
.378.088
.681 .067
.373.030
OLR-NLL ordered
MT.
LB FOT
1.312 247
1.385
263
1.028 247
1.041 .15
.703
.700 104
112
.714
.722 143
1.387 .193
1.452 195
1.452 192
1.440 141
.738 a7o
.755 094
.575.064
.571056
.676 066
.666 060
.368 031
.369 .033
1,400 281
1.012
.665.100
.737.135
1.378 200
1 1,436 163
.729.07
.587
.675.064
.368.om
Learning
Labeling
ABA
A12
KRA
C01
PU1
BA1
CO2
PU2
BA2
B1.2
NNT EOT
.922 026
.843 0188
.570
.579 011
.906 017
.455 011
.647 014
.532 014
.955
.9455 017
AD
.969.024
.835.m1
066
.573.mm
.894.01
.453,013
.648.013
.533,014
.967.019
.923
non-ordered
Hinge-IT
EOT
MT
ST
1.040.c0
1.040 033
.966,024
.832.016
8849.01
8849.19
.545,012
549..13
.49..1
.572.013
.566.022
.76.o12
.889.m7
.952 023
.952 o23
.152
.453
.453 a10
.640,
.651 014
.51.o14
.522,012
.538
.538
1.011 L11
1.011
.965 .017
019
1.023 023
1.023
.92 1. .012
022
IlingeIT ordered
MT.ST
.039 1.043,
.847 o18
.549 013
.54..11
.916 021
.451.0
.646 014
.530,012
1.013 o19 .964.m7
1.044 .222
EOT
.964.02
.831.m
.588.13
.570
.891.m
.453 009
.639
.588.13
.922.mm
Hingo-TT
MT.ST ST EOT
024 .961,o2)
.968.um
.87 017
548 012
570..12
.901 017
.453 010
.639 014
.524.012
.988 022
.935 017
.833
.5 .446
.569 .61
.889 010
.452
.637 0033
.24
963,.19
.920 ,012
Logistic-TT
MT
1.026 031
.883gm .222
553,014
.570.00 .122
.913 .199
.452 011
.641 1015 641
.370,014
.993.22 022 .933
.988 .220
non-ordered
ST
1.026.08
883 022
553.014
.70.012
.913 019
.452.01 .451.m
.70
.988
EOT
.962.025
.828,016
.540,014
.68.012
.889,a17
.636,112
.529 012
.963,gm
.920 .013
Logistic-IT IT ordered
MT.ST
1.030.03
875 018
.41 0122
.567
.910 018 .889,m
.451
.638 015
.33 016
.955 023
.986 .016
EOT
.965_u21
.829.014
.38.012
.563,012
.451.11
.633
.332.14
.963,18
.919.o12
Logistic-AT ordered
MT.ST
.959.um
.832.o
542 2220012
.68 .133
890 .016
.452.010
.633 013
.515.01
.959 959..017
.920 9220.013
LB
.970 024
840 a18
541.014
566.o12
894.017
.451.10
.635.014
.518.o1
.988 019
.936 016
EOT
.959,uz
.827
.5 5400012
.566,01
.89 o115
.451 .009
.633
316
.960,oxz
.918
OLR-NIL ordered
MT.ST
LB
.960.o20
.971.227 027
882..1
840
538 035
.57.001
.013
066.011
565.011
891 .066
.896 019
.150.010
451.11
.636
.634.115
.519.om
.20.01
.962.cm .017
.970
.922 ,012
.938
EOT
.961 .22
887 .015
.366013
566,010
.888 016
.451.010
.634.113
.520,014
.963,18
.920 .014
Learning
Labeling
POT
A11
EL1
CAL
CE1
|
CE2
2DP
FRA
MVA
NNT EOT
840 015
7288 013
.883 012
.768 013
.937 014
.921 014
.493 005
.471 o06
.158 017
AD
.833.015
.721,01
.876.012
.762,012
.929,012
.907 0111
.492.005
.469,00 000
.148,019
Hinge-IT non-ordered
MT'
.910.22
759.014
946.017
8668 014
.983 019
.942 016
.498 008
.467,o05
.334.028
ST
.910 028
.759 014
.966 017
868 014
.983 019
.942 016
.498 006
.467 005
.334 028
EOT
.800,015
.732,013
.868,012
.927,01 .841,000
.891
.492.00 .992.00
.464,005
.316 .023
Hinge-IT ordered
MT
.956 020
.730 012
.959 017
861 013
.981 017
.934 015
.499 005
.463 ons
.131 016
EOT
.788,01
.719.013
187.
.840,010
.926.m2
.890,012
.492
.462.004
.126,014
Hinge-AT AT ordered
ST EOT
837 023
7736 013 .732.o1
883 012
844 012
.929 018 .920
896 011 .885,010
.498 0o5
.465 ons .464,004
.273
.821,019
.899.11
.840,000
.492
.262,00
Logistic-IT non-ordered
MT
ST
.955 020
020 .955
.755.L44
.75 014
.936 016
936.016
859 014
859,014
.973 016
.973 016
017 .922
.922 017
.497 008
.497 o06
.468 007
.468 007
.285 o15
2285_115
EOT
.788,012
.734.11
.87101
.838,10
.881,00 .923.o11
.493 .014
.460 o05
.267 ,011
Logistic-IT ordered
MT ST EOT
.967 021 .784,012
.723 012
.957 017
854 013
.944 016 .922.o11
.913 015
.496 004
.461 o0s .460,04
.129 013
.7 .715,011
.870,012
.837 .011
.877 .100
.493 .os
.123
Logistic-ATT ordered
ST
LB
.791 012
790 013
737 013
.73 .012
.872.012
888 013
841.00 0111
843
.917.og 001
.925 014
.886 o16
.866 014
.496 005
.496 o05
.461 00G
.461. 161
.295 089
.95 069
EOT
.776.011
.730.01
.870,012
840,010
.915.a11
.883,00
.493,005
.460,006
.285
OLR-NIL ordered
MT.T
LB
.789 012
792 015
.730.o11
.737 013
.868,012
884_014
838 011
841.012 012
.929 014
.922.012
.879
.889 013
.496 005
.496 o05
.461 .05
.461.
.268 059
.669 060
EOT
.774.01
.732,012
.867
.839 011
.920
.876
.492 .05
.460
.261
37

---

Published in Transactions on Machine Learning Research (1/2023)
Table 20: A counterpart of Table 2 regarding RMSE for Task-S and EF10 datasets.
Learning
Labeling|
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
AD
NNT EOT
2. 2.899 576
2.922 529
1.882 521
1.872
1.152.1 147
1.102,148
1.339 289
1.327.24
2.900 403 2.816
352
2.983 356
2.945.320
1.379.179
1.378.189
1.030 1007
1.055
1.280 163
1.287
150
.649
.40 037
Hinge-IT
MT
ST
2.861.560 563
1.878.54 2.871.509 .00
1.919 600
1.182.134
1.188
1.433
1.413
309
2.956
134 2.956
2.945
2.961 373
1.480
1.468
1.044 107
1.306 1.044.107 161
1.306 161
.707 .073
.695
non-ordered
EOT
2.811.581 581
1.860.455
1.073.145
1.386
2.853.422
2.936 .62
1.396.1s
1.067.11 .1067.113
1.263
156
.657.0m
Hinge-IT ordered
MT.ST EOT
2.977. s61
1.961.543
1.170.
1.354.300
304
2.041
2. 2.934.ss
1.531
1.045
1.285 137 1.277.1400
.629
2.901.544
1.920,500 5000
1.099.149
1.34152611
2.876
.34
2.923 3368
1.398 166
1.054
.624_044
Ai
MT.ST EOT
521 2.845
1.923.568
1.1441.771
1.260.244
352 2.922
2.887
1.390.
1.014.10p
1.224 123 1.220
.598 .042
ordered
2.782. 633
1.279,522
1.112,150
1.258
252
2.791
333
2.910.367
1.358, 182
1.052,107 107
130
.598,044
Logistic-IT non-ordered
MT
2.767
495 2.770
2.014 618
1.163.
155
1.366
295
2.944 976
2.925 355
1.458
203
1.035 102
1.259
138
.678 084
ST
499
623 2.028
1.151.161
111
1.348
300
2.940
.372
2,934 403
1.447 199
1.035
102
1.259 138
.676
EOT
2.843
1.886.500
1.104,.14
1.319
.220
2.874
2.925
1.381
1.048
1.242. 120
648
Logistic-TT ordered
MT ST
2. 2.853.cm 513
146 1.911,515
503 1.961.
1.158,
244 1.390
2.972 .44
2.947 365
1.503 191
1.031 .994
1.262 142
.04.00440 033
EOT
2.899.50s 568
1.107
1.317.247
.247
.378 2.849
2.937 403
1.412
1.034.106
118 1.238
.597 $597,335
Logistic-A'1 AT ordered
MT ST
LB
495 2.802
2.740 477
1.918
562 1.981 567
133 1.143.
1.141.1154
1.267 .29
1.286.251
2. 848 .31
2,906
.341
2. 2.894.4 3112
2.890 .309
1.373 175
1.400.177
1.009
1.017 .96
096
1.212
123 1.218.
1.216
119
.584 L043
.577 .337
EOT
2.889
567
1.931.517
1.108,145
1.2666
2.789
360
2.911 1.00
1.369.167 167
1.033 ons
128
.580 0338
OLR-NLL ordered
MT.ST
2. 2.792.cmm 472
1,888 .611
1.159.112
1.241 260
2.876 363
2 1..811 204
1.400 4000.201
1.006 097
1.227
126
587.033
2.812 494
1,963 627
1.150.146
1.231 249
2.
896 365
2 8888 352
1.413.210 210
1.002
103
1.233
137
.584 032
FOT
2.848
1 1.959.54 546
1.129
1.277
2,864
2.913
1,368 168
1.027 .103
1,224
.582
Learning
Labeling
ABA
A12
KRA
C01
PUT
BA1
CO2
PU2
BA2
F1.
NNT EOT
1.857 o5e
1.558 032
.926 081
.979
1.698 .344
.688 .11
1.124 028
.842 026
1.865 .33
1.738 0331
AD
1.838
1.535.0z
.917.ea
.970.a19
1.690
a0s1
.688,013
1.119.027
.838,088
1.857
.037
1.723.n2
non-ordered
Hinge-IT
EOT
MT
ST
1.915.
1.915 067
1.533 1.849,04
1.626 028
1.626 028
.22
.898
.917
.917,02
026
.975.017
993.o21
.993.22
1.685
1. 1.739 o33
1.739 033
o03
.682.01 .122
687 .113
.677 013
1.148
1.121 .023
1.148.026
.837 .220
.854
884..20
1.877
1.854,000 .008
.33 1.877 035
1.721
1. 1,766 031
1.766 031
1.29
Ilinge-I ordered
MT.ST 1.973 o6a
1.641 .30
.920 030
.987 020
1.912
063
.683 .013
1.145.030
.859 023
040 1.929
1.784 .331
FOT
1.861.0m 044
1.544
.906.22
.971.220
1.695 .84
.013 .681
1.117 .036
.850,z22
1.856 n0s
1.725.0mm
Hingo-TT ordered
MT.ST EOT
1.844.oc 0GT 1.826,
1.570
028
.905 023
.951.o18
1.709 035 1.685 .035
.683 013
1.102
.27
.821 .20
1.860 o33
1.747 028
1.532
.028
.904,o22
.951 .166
.680.mm
1.099
024
.819 018
1.851 0036
1.720
Logistic-TT
MT
1.893
063 1.893
1.619
029 1.619
.905.026
.981 .088
1.726
2226. 1.726
.144 .685
.685
1.137.030
.856 026 .856
1.869 .33
1.761 .034
non-ordered
EOT
ST
1.847 .447
1.535.02a
.890,029
.905
.967.a17
991
1.686.cm 000
.681.o3
1.118 .025
1.137 020
.833.018
1.853.nm.
038 1.869
1.720.a2
1.761 .044
Logistic-IT ordered
IT
MT.ST EOT
1.934 063 1.863
020 1.629.com
.920 028
.975.o18
1.787 037
660 .14
1.129
031
.876 021
040 1.892
1.769 033
1.552 .300
.909.as
.960,ms
1.688 .92
.679.o13
1.116 0,2
.860,017
1.856 ogs
1.723.uz
Logistic-AT ordered
MT.ST
1.824.0a
1.533.0m
.903
.951 .16
1.683
.033
.680 .113
1.095.02
.820,0m
1.850
.344
1.721
LB
1.844 061
1.545.028 028
.905.022
.49.017
1.693
033
.800.12 o012
1.097.o23 .22
.83.o22
1.862 038
1.732 a30
EOT
1.826 044
1.531 oz2
.901 .044
.917
1. 1.683
.680
012
1.094 .222
.817
1.849
1.720
OLR-NIL ordered
MT.TT
LB
1.831
.550 1.848
1.532.0 .22
1.549.00
.893 .300
893
.953 953.018
.957
1.693
1.684.as3
.033
.681
.681 .044
013
1.097.02. 0026
1.102 024
.827
826
019
1.85
1.86
1. 1.720.0c .027
1. 7.734 031
EOT
1.833,047
1.529gaz m2
899,,90
.949.014
1. 1.682,082 032
.600.01
1.098.025
$27,020
1.851.0r
020
Learning
Labeling
POT
A11
EL1
CAL
CE1
|
CE2
2DP
FRA
|
MVA
NNT EOT
1.569 024
1.291.oo .20
1.669 024
1.386 025
1.620 025
1.574.om 017
.770 007
.727 oos
.632 632.104
AD
1.557. 02
1.287,021 021
1.649.01
1.378.024
1.608 .20
1.56 .020
.766.00 .006
.724,007
.601 ,105
Hinge-IT non-ordered
MT'
1.813,04
1.348
024
1.723.029
1.611 [,611.021]]
1.865.033
1.758 o28
.779,007
-727.006
-739 139.110
ST
1.813,044 044
1.348
024
1.723.029
1.611.024
1.
1.865.034 .034
1.758
028
779 007
006
.739
EOT
1.605
1.322
.22
1.641
1.564,019
1.786 .04
1.692
.022
.767 .006
.720 .00
.665 ,099
Hinge-IT ordered
MT.ST
1.868 045
017 1.314
031 1.740
029 1.657
2.000 040
1.904 0.10
.780 o08
.723 007
.487
EOT
1.533
1.286 020
1.653.azz
1.566.018
1.783
.011
1.709.00 .024
.767 .007
.718
.44,0966
Hinge-AT AT ordered
ST EOT
020 1.540.cm
1.329
024 1.655.00
021 1.565.00
1.576
1.763 024
1.688
024
.766
.719,00
.222
1.538 020
1.316 020
1.639.ga
.20
1.746
.21
1.668 .025
.767
.006
.715.007
.480 .085
Logistic-IT non-ordered
MT
ST
031 1.597
1.893
1.892
1.341
1.341
024
024
1.711.026 026
1.711
1.600
1.600.024
030 1.847
1.847
030
1.748
1.748 .299
029
.775 007
.775 007
.732 010
.732 010
635 1339
633 139
EOT
030
1.319
010
1.641.01
1.564.019
1.778
023
1.686
025
.006 .767
.719 .006
.576,00
Logistic-IT ordered
MT ST EOT
1.824 044 1.529
021 1.288
1.308
026 1.670
1.718
1.416
031
1.743 031
1.653
026
.771 007
.722 009
.456 074
019
oz2
1.391,00
.24
1.635
.22
1.572 .200
.767
.007
.716 ,007
.437
Logistic-ATT ordered
ST
1.541 020 1.545,021
1.318 001
1.643.o22
021 1.568
1.751 .221
1.677
.025
.773
008
.719 007
.541 101
LB
1.327
1.664
1.577
1.767
1.698
028
772 202
.720 ons
.538
EOT
1.541.022
1.316
m21
1.638
o21
1.563.g0
.20
1.747 .200
1.669
.02
.767 .006
.716,006
.514,100
OLR-NI,L ordered
MT.TT
LB
1.596
1.620,22
026
1.319
1.328
021
023
1.639 ozs
1.667
028
1.562.02
1.572
024
1.767
1.782
027
.022
1.668
.025 7,687
024
.772 007
.772.000 007
.718,007
.717 o06
.55.114
511.114
EOT
1.575 020
1.316
021
1.634.02
1.560
o20
1.766
1.665
.22
.766 .07
.716,007
.492112
38

---

Published in Transactions on Machine Learning Research (1/2023)
Table 21: A counterpart of Table 2 regarding RMSE for Task-S and EL3 datasets.
Learning
Labeling|
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
AD
NNT EOT
.607 119
560 126
.400
.407 105
.352,096
.322 083
.258,118
259.108
.634.098 0998
.29 629.102
.859 113 .863.11
.197 104
.205 104
.433,s5 oss
.452 04G
.481 ons
.468,56
.203
.198 .400
non-ordered
Hinge-IT
EOT
MT
ST
574.129
.608 130
129
.421 180
.432 138
.421 180
.347,02
384
3844065
2264,006
228..20
.288 140
.625 100
.625 100
.640 .906
.922 106
.922 106
.853
115 .198
.199
.199
105
105
.416,053
.43
443.058
.442.042
.449 019
.49.049
.176,
.186 .333
.176
Hinge-IT ordered
MT.ST
.552.187
.414 156
.380 075
.263 09s
.618 092
.936 102
.207 106
.477 o1s
.448 051
.172 .054
EOT
1610,14
.405
.34.0944
2288.1
.633 .osis
.870.11
2200,114
.418,006
.444.047
.186 .032
Ai ordered
MT.ST EOT
567
137
.407 164
.379 056 .343,083
.276.00
.10 .0as
.822
.208 100
.445,0.
.458.ox
.167 .n44
.597
133
1430,14
.287 .108
.625 .083
.861.112
119
4100,000
.44.050
.183 .330
Logistic-IT non-ordered
MT
.617 126
.473 159
.392 113
.255.115
.664 110
.887 112
.232 105
.408 05.
.430,os5
.168 0a3
ST
.617.126
.473 159
.392
113
.25..115
.664 .100
.877 112
.232 105
.408,553
.430
.168 08.003
EOT
6621
.449.
.348 .099
2266,144
.96 .631
.833
.218 121
.401
.427
.170.033 0333
Logistic-TT ordered
MT.ST
.688 127
.473 177
.380 094
.244 .11
6677 097
895 106
.231 093
.499 oso
.401 044
.168 .32
EOT
646
.422,
.18
.345,00
.269.122
.632 .944
.835
.228 1004
.401
.415,058
.173_033
Logistic-A'1 ATT ordered
MT.
.604 140
.456 171
.381 0900
.263 096
.630 096
.801 .o8
.232 099
401,o50
.417 001
.169
LB
.229 100
.67.
380 088
262.099
630.092
864 120
.344.09
.406,000
.410.048
.170
EOT
.626.118
.446
.36,052
.271.111
.637 1041
.837 105
.225, 103
.400,019
.419,07
.170,030
OLR-NII ordered
MT.ST
LB EOT
.618 119
623,120
.455 161
1559.72
.37
366 087
.099
.254.10
.256 156
.633
.640 108
103
859 103
.792 oss
.227 111
.266 100
.407 o07.082
.408.00
.166.55
..05.00
.172.033
.171 028
636 113
.461,78
.344
.263,122
.631 103
.822 002
.224 104
.405,019
.455
.170
Learning
Labeling
ABA
AI2
KRA
CO1
PU1
BA1
CO2
PU2
BA2
EL2
AD
NNT
EOT
.488 D14
480 .044
.173,013 013
.172 o12
.445
.400.16
.106,014
.107.016
.539,o12
.554 615
.233,o11
.255 019
.121,14
.120.014
.346,010
.355
.297 .012
.317
.247 .008
.248
non-ordered
Hinge-IT
MT
ST
486 012
486 012
.172
.172.012
.387
.38 0013
.101.om
.101.o1
.530 012
_530 012
.235 014
235.014
.115,012
.115.012
.331
.331 .01
.305
.305 012
0005.012
.248 oo8
.248
EOT
487 0015
.172,013
.385.014
.102,012
.530 011
.233,013
.113,om
.336,015
.301
.246 p07
Hinge-IT ordered
ST
.85 012
.172.012
.383.015
1102.000
.530 011
.232 012
.114.om
.331.01
.305 011
.248 oos
FOT
486 _813
.173
.381 181,00 .015
.102
.530 .122
.233
.114,011
.337 0357
.299 .11a
.247
Hinge-AT AT ordered
MT EOT
.866 012 486
.172.012 1772.012
.379 0112
.101 .013
.529 011
.231.012
.115,010
014 .338
-304 011
.248 pos .247
.172.012
.378.11
.101
.530.o1
.233
.114
.342.014
.299.012
Logistic-IT
MT
.483 D15
.169..r 212
.322 m10
.108 014
.526 o10
.231oo1
.117,013
.301 .33.0m
246 pos
non-ordered
ST
483
169 162
332
.108
.526.on
.231.n
.117
.313
.301
2416 1166
EOT
_481_p13 .013
.173,012
.372.112
.102.012
.526 010
.231
.114.01
.33
301,o12
.246 pos
Logistic-IT ordered
MT.ST ST
481 013
.170.013
.34
104.012
.526 011
.230.a10
.115.01
.303.012
.300.013
.246 008
EOT
.800
013
.172,014
.362,012
.102,012
.527 001
.231.01
.114,1
.301.012
.300,012
.247 .007
Logistie-AT ordered
MT.ST
1,14
.481 .14
481 014
.170.012
.171.012
364.012
3664.012
.103.012
103..13
.526 010
.526 010
.230.010
.231 230
.116,012
.115.112
.304.00
.304.000
.300 .013
.300.014
.247 DOS
.247 008
EOT
.478 014
.173.013
.62.012
.101
.526 009
.231 011
.116
.303
.301
.247
OLR-NIL ordered
LB EOT
ST
.480 D14
.481 .044
.170.012
.170
363 010
.36
010
.101.o
104
.527 010
.529
012
.231 011
.230
.115,010
.115,110
.355.01
.306
.301 .133
.301 0133
246 008
2466 DOs
.881014
.173,013
.03,011
.102.011
.527 001
.230
.115,01
.06,012
.300,033
.247 .007
Learning
Labeling
POT
A11
EL1
CAL
CE1
|
CE2
2DP
FRA
MVA
AD
NNT BOT
.323 323_015
.327 015
.262.009
.62 009
.138 007
113
.488 0007
.499 on6
.197 .009
.197 0os
.224.m1
.234 013
.308,005
.307 000
.353.om
.351,012
.038.000
.040.00
Hinge-IT
MT
355_.12
.62 009
.138 007
.478 007
.189 007
2355000
.308 oo5
.306,010
.041 2007
non-ordered
EOT
ST
.325,014
.351 012
.262.
262 000
.138
.135,000 1,35,009
007
.477,00
.788
007
.189.00
.189 007
.225,010
.235 010
.308 _008,005
.308 005
.303
.306 o10
.038 .006
.041 007
Hinge-ITT ordered
ST
.353 012 .327.m1
.262 009
.138 on8
.477 .007
.191 .oos
.232 013
.307 oo5
.302 009 .300,009
.041
EOT
.262.009
.134,007
.466.08
.189,00
.210,00
.308,004
.039.cc 039.007
Hinge-AT AT ordered
ST EOT
345 o13
262 000 .263 263_008
.139 nos
.481.oo
.190 oos
.208 one
.307 .05
.302 009 .300,008
.042 006
.326.014
.135,066
481.008
.190,00 .07
.208 po8
.308,.00
.005
.038 .005
non-ordered
Logistic-IT
EOT
MT
ST
.202.013
.42
242 013
013
263 0099
.266 gos
260.008
.133 pos
.138 008
.138 00s
.459 007
.459.00 000
.459 ,007
.190 .08
.189
.189 007
.077
.217 .000
.226 010
.226 010
.308 005
.088
.308 005
000
.289 006
.20 o06
.200 o06
.035 .000
.039 .00
.039 007
Logistic-IT ordered
MT
246 011 .201
265 009
.132 008
.460 o08
.189 007
.207 609
.307 005
.285 o07
.039 006
FOT
263 .099
.131 _008
.459
4459.06
.189 .007
.208 ,09
.308
.284
.036 0006
Logistic-ATT ordered
ST
LB
.110 014
203 203.013
264 008
264.008 004.
.137 007
.137 008
.58
.459 .07
006
.188 007
.188
008
.077 009
.206 609
.308 005
308
005
.286 o06
286 006
.039 006
.039
EOT
.199.012
263.gos
.133.00
.588 .088
.189 .007
.208 .099
.308 .05
.284.006
.037 ,006
OLR-NIL
ordered
MT.T
EOT
LB
.197,01
.199.015
2007 013
.263,010
264
264.009
.133 pos
.137 008
.137 008
.460 (60.007
.460 006
.49.007
.188
.188
.189 .00
007
.216 g09
.216 gos
2277 .0.
.308
308
330 005
005
.287 006
.285
.287 o06
.037 .005
.040 006
.040 006
39

---

Published in Transactions on Machine Learning Research (1/2023)
Table 22: A counterpart of Table 2 regarding RMSE for Task-S and EL5 datasets.
Learning
Labeling
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
NNT EOT
.975 218
.788 203
.98. 168
.481 235 .498,210
1,089 186
1.375.170 170 1.362
.343 130
.525.om 054
.603 059
.408 831
AD
.901.181
820
.522
1,084.
.343,18
.524,059
.605.ee
.408_039
non-ordered
Hinge-IT
EOT
MT
ST
.867.10 1192
.991
.911 308
308
2814.255
287 775
.789 2G3
.514 168
.05, 158
524
168
.520 22G
.20
.24
1.107 .19
1.115.172
1.108
1.36T
1.390 162
1.389 160
.386
-371
.365
1111
159
164
.514,053
.537.057
.537 057
.599,000
.63 o611
631
.455
.455
.405 .025
Hinge-IT ordered
MT.ST
.965 231
.813,218
.502.167
.524,217
1.093. 175
1.408 159 1.375,160
.384 1557
.529.
.600
.383 .333
EOT
898_194
.811,206
1491,
508,210
1,080.157
3354_13
.326,060
.604,060
.381.88 3881.033
Hinge-AT ordered
ST EOT
.998 245 .871,185
787.215
.866 178
.514.19
1.077 160
1.364. 179 1.356,174
.364.
.523,05
.601 L.76
.390
.802 201
.488
165
.513,209
1.069.158 158
.363,
124
.324,051
.655.822 .82
.384_032 032
Logistic-IT
MT
.973 225
856 268
.288 160
.525 181
1.123 187
1.400 224
.424 186
.525.om
.624.89
.380 p30
non-ordered
EOT
ST
.875.189
.969 226
825
808
.540 163
.499,
.529.186
.522.150 .500
1.115.117 15.167
1.074.158
1.370.173
1.399 223
.433 198
.400,171
-525.050
.510,052
.581 .022
624,,59
.380 .33
.374.036
Logistic-TT ordered
MT ST EOT
.947 180
.815 216 .798
.541 157
.505 190
1.101
1.452
.418 175
.506 o5o
.581 065
.358 .300
.903
196
.501
.511
1,085 .151
1.387 .167
A41,166
.512.om
.586,065
.356_00 .336_033
Logistic-A11 AT ordered
MT ST
LB
.929 211
.920 227
232 8899
.82 214
.422 160
.10
.481. 176
.84 182
1.085
1.090 152
1.362.
1.365 163
.408.17
.399 160
.517
.513,061
.579.0m
.573 057
.360 .027
.359 .27
EOT
883.211
.812,216
.488 142
2966
1.080 148
1.302, 196
166
.515.053
.586.065
.359 .227
OLR-NLL ordered
ST.ST
LB FOT
.931 233
.934 174
.837 224
821
.515 162
512
.491. 199
.46. 476.184
1.078
1,108 163
163
1.369 185
1.399 201
.420 179
.421 179
.511.48
.509.04
.579.06
.584 060
.358 030
.356.03
888 191
800 225
.490
.502.207
1.084
1.393
.403 156
.517
.588
.359
Learning
Labeling
ABA
A12
KRA
C01
PUT
BA1
CO2
PU2
BA2
F1..
NNT EOT
.596 p32
.404.012
.474.013
.228 013
.791 016
.336 013
.262 012
.405
.522 018
.430
AD
.524.02
.376.012
.471
.224.0ma
.785.01
.331.01
.260,012
.400,012
.505.016
.413
non-ordered
Hinge-IT
EOT
MT
ST
.521.om
583
583
027
D27
.371.01
398
398 014
014
.83,005
.86
.86.017
.228
.235
.238
.785,mm
8443 o20
843 020
.329,01
.333 012
.333 012
.633.012 .261,o11
.644
.405,013
.450
.450.133
.507 .016
.580
.580 .20
.412 ,010
.430
.430 011
IlingeIT ordered
MT.ST
.583 027
.385 014
471,
.230
.800 015
.333 012
.26,,011
.411 111
.574 .22
.429 011
FOT
.512.02
.371m1
.468.0
.227
.786.m
.329.
.261 .011
.401.o
.501.m7
.412.a
Hingo-TT ordered
MT.ST EOT
567
029
.387 014
.880
.231 011
.793 o14
.333 012
.262.o12 .259,013
.405 012
5336 022
.428 011
.511,021 .011
.371.01
.477,014
.226
.783
.330.022 012
.397 ,10
.503 ,01
.412 ,011
Logistie-TT non-ordered
MT
.522
026
.394.013 0994
.468 016
.236 013
.794.015
.326 .122
.265.013
.458 011
.524.019
.420 012
ST
.522
026
.371.01
.688 016
.236
.794.05
.326 012
.662 012
.188.1
.24
.412
EOT
.509.01s
.372.01
A63.014
.226,012
.785,012
.327 .111
.255,012
.400,013
.502,016
.412 ,010
Logistic-IT ordered
IT
MT
513 023
.370 011
.633
.233 011
.795 014
.328 .110
.259 013 .252.012
.401 o15
.229 .018
.413
EOT
.510.019
.371.12
.61..16
.224.012
.786,ma
.326,011
.396.012
.501,016
.112.010
Logistic-AT ordered
ST
507 .20
370
470.017
.227
.787 014
.328 0111
254.0122
.399.0
.55..0
.133.100
LB
510.022
.30.a11
.669.16
.229.012
.791 016
.328 01
.256.013
.100
.508 616
.412.010
EOT
5588
019
.370
.468
.224,011
.788 o13
.327
.252
.397 013
.505
.412
OLR-NLL ordered
MT.ST ST
1,8 EOT
5088
509 019
023
.370 .122
370
Q14
.462
462.01e
.229.013
.22
.226 022
014 .789
.787 .012
.326
.326 011
.566
.256.012 256.012
.399
.399
013
.505 014
.508
.412 .412
.412
412.010
508 018
.31
.59,015
.224.011
.786
.325
.252.o11
.39,,033
.500,018
.12.010
Learning Labeling
POT
A11
EL1
CAL
CE1
CE2
2DP
FRA
MVA
NNT EOT
.50 p15
.419 109
.13.007
.888 .10
.384 017
.492 017
.415.007
.376,00
.055.00
AD
.241.015
.415,007
.214.008
.683,009
.373,014
.438,01
.409 0o6
.374,006
.057 ,008
non-ordered
Hinge-IT
EOT
MT
ST
.589 D47
.415,m .534_om
592 D48
.41 .oos
421 421.0s
.252 015
.226,00
2252.11
.718,0m
.763 o13
.763
.401 022
.397 .21
.369.m
017 .492
.421 .423
.492 017
.421
.204004
.40-L011
.390
.404.01
.177,014 014
.076
.17 014
Hinge-IT ordered
MT.TT
.007 014
.420 ons
.229 008
.741.00
.468 042
.492 017
.415 007
.378 009
.065 007
EOT
203
.414_007
.219,009
.718
.369,mz
.422,012
.407 ooo6
.370 3000
.056 .006
Hinge-AT AT ordered
MT.ST
.880 012
.419 oo8
.237 010
.722 .oss
.373 018
.489 a1s
.188 oos
.381 ons
.066
FOT
.177
.414,0m
.223.010
.715.010
.361.a2
.410ms
.409 ooc
.372.00
.058,00 8008
Logistic-IT
MT
ST
.37 821
370
.119 pos
.24.009
.242.12
242 .122
.622 762_o12
762
014 .372
.380 o15
.405 .14
.402 014
.414 oos
.14.008
.380.om 000
.3S0 006
.06
2006 019
non-ordered
FOT
.308
.414
.225
.718,010
.364.m
.392 m13
.403 004
.369 ooG
.083 .1.0
Logistic-IT ordered
ST
.176. 1176.011
.412.00
.214. o0o6
.744 .100
.381 013
.407 014
.404,005
.370 o0G
.059 000
EOT
.172.012
.411 oons
.214.
.718,008
.365.mz
.392.m4
.403 005
.368 nn6
.051 .007
Logistie-AT ordered
MT.ST
L.B3
.174.012
.174.01
.413 3,.00s
.116 aos
.233 o08
.235 010
.718,009
.721
.360 .14
.357 012
.393 .133
.391 014
.404 oos
.404.00
.370 000
370.005
060
060 007
EOT
.171.100
-13.omo
.222
.716,00x
.357
.391,122
.402 004
.367 .00G
.053
OLR-NLL ordered
MT.ST
LBB
.173.012
.173.012
.11.000
.414
117.008
.223
.224 008
.717 .009
.720 .099
.359
.361.012
.390 .14
.393,11
.4003,000 003
.403 005
.369
.369
066
.00G
.098 013
.098 013
EOT
.171.013
.414.008
.219 pos
.715.00
.357.012 .12
.389.014
.403,004
.368,005
.082 .009
40

---

Published in Transactions on Machine Learning Research (1/2023)
Table 23: A counterpart of Table 2 regarding RMSE for Task-S and EL10 datasets.
Learning
Labeling
DIA
PYR
APR
SER
TRI
WBC
CPU
AMP
BOS
STO
AD
NNT EOT
1.741 419
1.736
1 1.309,534
1.276.54 514
.888
.961,300
1.034.439
.935 411
1.943 315 1.927.29.
290
2.858 206 2.817.317
.674 286 .633
.867
.878,098
.983 135
.997.137
.544,
.539 028
Hinge-IT
MT
ST
1.831 450
1.831.450 .40
1.3 1.312 569
1.310,
1.0 1.20
.984 299
1.192.373
1.107 361
1.941 289
1.936 304
7,7 330 2.813,350
2.
2.856.42 424
.714
.699
.87. 092
.04 089
1.005 137
1.007 137
.627 025
.627 028
non-ordered
EOT
1.731. 400
1.276 497
.916 266
1.019_328
1.9440_309
.691,31
.873,166
.970.1a5
.550,004
Hinge-IT
MT.ST
1.801.
1. 1.282.513
.958 306
1.057 100
1.939.. .144
2.821 .40
.718 304
.884
.981
.535 033
ordered
EOT
1.705_367
1.301 .73
896,272
1.050,372
1.915,317
2.761.311 .317
.692.285
.853,99
.963,14
533.029
AT
MT.ST EOT
1.690
1.285,51
.950 293
1.032.4m
1.895
2.769..0m .69
.711,31
.873
.966.15
.535
ordered
1.703.345
1.277 .487
.888 263
1.120.421
1.898,200 295
2.808 .385
1899,29
.871,111
.948,135
.531.030
Logistic-IT non-ordered
MT
1.713 467
1.325
539
1.033 290
1.237 389
1.944 293
2. 770 348
.750 376
.868 oa5
.973 144
.557 030
ST
1.708
460
1.310.52e 528 1.284
1.042.300 .300
1.114.300 304
1.920 303
2. 7,.85.00 .622
.801.374
.867,997 .07
.972.144
.557 0300
FOT
1.732.
.881,266
1.099
1.907
2.825
.672 278
.855 10G
.971.
.533 .266
Logistic-T7 IT ordered
MT ST EOT
1.758 411
1.312.522 527
.970 278
.942 .81
1.959 328
2. 80.80..mm .376
.715,as0
854.106
.93.3aa 111 .947.
.525 .299 .526
352 1.733.
1.276 .486
.890,239
.959.340
1.927.311 3188
2.780 .374
.706.361
.866
Logistic-A'1 AT ordered
MT.S1 ST
LB
1.679
1.658. 412
1.242
1.252.500 .225
.935 256
.940
1.0/00330
.995 368
1.898
1.930 284
2.715 269
2.745
.707 365
.699.
858 098
.83,
.938 .143
.22 134
.519 .127
.519 026
EOT
1.715. 392
1.328. 500
.910,242
1.089 384
1.911 279
2.793.326 326
.661,300
.864.11
.956.140
.518,029
OLR-NLL ordered
MT.ST
1.678 3G8
1.213 490
.944.274
890 890.321
1.914.29 277
2. 7.. 258
.705.341
.442 204
.928 .28
.516 516.029
1.7041 438
1.234.566 566
.919 278
.912.49
1.915 298
2.7 781 348
.700 351
.855 109
.943. 1412
.515 026
FOT
1.720,
1 1.284
.833 303,233
.957,779
1.907 .27
2.794 .932
.648 267
.842.101
.946.13
.519.022 .,32
Learning
Labeling
ABA
A12
KRA
C01
PUT
BA1
CO2
PU2
BA2
F1..
AD
NNT EOT
.887 .030
886 D37
.188,018
.512 019
.676,199
.679.o9
.412,01
.412 012
1.392 022
1,383
.225
.509.012
.509.am 011
.468,013
.469.
.571,013
.575.013
1.064 033
1.004.00m
.621 ,016
.639 017
MT
1.003,c 0448
.517,017
703..04
.118.17
1.503 033
.511.o1
.477
.600.o15
1.112 039
.6:10. .018
non-ordered
Hinge-IT
EOT
ST
.913.0a
.981 D47
.482.016
.517
.679.o2
701
.433,016
.46.o17
1.503 039 1.389
025
.502,m .122
.511a15 a15
.472 016
.463,01
.578,013
.600
1.112 039
.999 .033
.622 .015
.640 018
IlingeIT ordered
MT.ST
1.005 ..0
.515 111
700 o20
.088 014
1.598
033
.501.12
.462 014
.590,013
1.349 .42
.6:11 o19
FOT
.915.nas
.478.m
664 o02
.407
013
1.390 m28
.500,012
.463,05
.590
.994.029
6621.m
Hingo-TT ordered
MT.ST EOT
o53 .883
.939
.507 019
692. 022
.410 0002
030 1.379 .24
1.413
.504.om .112
.463.on 464
.577 .017
1.060 o3c
6443 018
.478,016
.689
023
.409
002
.503 011
.462.13
.374
.993
.030
.616 ,015
Logistic-IT
MT
982 041
493.020
.715.00
1,00
.413 .01.
1.422
.29 1.422
.502 013
.466.014
.601 .011
1.043
035
.631 .021
non-ordered
EOT
ST
.881.081
.933
040
.475.m
.491 OL17
.670.om
.687 021
.411
.006.15
111
1.375.0. .04
.502 .133
.50
.54.012
.463 014
.601 015
.578.014
1.000.a27
035 1.043
.613.n14
.624
Logistic-IT ordered
IT
MT.TT EOT
.32 036
.476 016
689 023
.422 002
1.450
.498 012
.54.014
588.ms
1.120 044
.622 016
.884.aa
.473,013
685,025
.03,012
1.377 .22
497.012
.455,015
.584.012
1.012.gm .20
.613,015
Logistic-AT ordered
MT.TT
1.B
875 031
866 027
.473.014
473 015
680
680 026
025
.404
404.o10
404.000
1.377
1.384
.024
.26
.500.a1
.500.oz
.455.013
.45.
.572.12
.571 .122
1.001 .227
1.019 032
.612.015
.612.orz
EOT
.873 029
.474.014
.677
.24
.003.01
1.378 023
.199.012 012
.155.033
.571
.991 _029
.614
OLR-NLL ordered
MT.ST ST 1,8 EOT
.811 030
.74.004
668
026
.40..01
1.378
.23
.499 014
.456.13
.571 012
1.000 030
.614 015
874
030
.475
015
.670
024
.401 011
1. 1.384
.498 012
.456
.572
1.015
.61
877 .20
473.014
668 025
403,00
1.375,025
498,012
.457,012
.52..11
.997 .000
.63.p14
Learning Labeling
POT
A11
EL1
CAL
CE1
CE2
2DP
FRA
MVA
NNT EOT
1.084 112
555..10
528 007
1.174.017
.710.oz 020
.748.
.567 .00
.513.00s
.223.oo
AD
1.046.102
.574.01
.529.077
1.169.016
.702,020
7746.30
.564.001
.512.00
.202, .010
Hinge-IT
MT
.750 250.075
.620 020
.607 014
1.324.01s
.787 024
.910 a31
.597 o00
.562.om oni
.286 .286.115
non-ordered
EOT
ST
.745 073
.577.am .698_oms
.620,o2
.572.1
.607 014
1.288
1.324
.787 024
.739
.910 031
.827 a00
.580 oo8
.565 .014
.519,0ms
.539 00G
.257 012
.197 .0n8
Hinge-IT ordered
MT.ST ST
1.194.047
.577 1
523.oo 003
1.250 021
.833 029
.977 063
.572 0o5
.519 004
.26 012
EOT
.838,am
574.010
521.007
1.155.015
.757.a26
.845
.564.gou
.518 004
.196 .011
Hinge-AT AT ordered
MT.ST
.679 052
.575 011
521 .006
1.164.019
.765 026
.817 .27
.573 005
.320 ons
.225
FOT
.604.0m
.544
519.0n6
1.150.o17
.715.026
.789
.564.0o
.519,065
.207 .009
non-ordered
Logistic-IT
FOT
MT
ST
.514
558 p41
558 041
.574.01
590
593 003.o11
.558,010
586 012
586
1.312
1.284.017
1.312,019
.726
.745 024
.745 .224
.794.02s
.836 .32
.255
030
.565 .044
.578 on6
592 .08
.518,00
.545.008
563..12
.194 0008
.267 011
.883 012
Logistic-IT ordered
MT ST
.744 o5o
.699 o10
518 007
1.245.
.752 028
.899 036
.567 0004
.510,004
.191.
EOT
.567 .411
.688
0
.18 .077
1.151.016
.688.022 022
.804.031
.565 .044
.510,004
.181 .088
Logistie-AT ordered
MT.ST
L.B3
.616 pas
.567.08 .oss
.75 011
.54.010
.577 0009
.455 008
1.276.017
1.289
.724 026
.712 024
.774 o2s
.787
029
.569 004
.569 004
.319
.519 005
.210 000
.08
FOT
.555,084
.544
.532
1.275.01s
.708
.773
026
.564.o04
.516.0m5
.192.0 1.92.008
OLR-NLL
ordered
MT.ST
FOT
LB
.488,40
502 042
.299 043
.573.011
574.010
.574
.531.00
.55.000
544 008
1.278 018
1.275.017
1.288 018
.721.026
.732 026
.7222 024
.780
7700 o29
.791
029
.588 004
.564.o4
569
.516.ms
.518 005
.518.008
.183 .088
200
199 008
41

---

Published in Transactions on Machine Learning Research (1/2023)
Table 24: The number Nz (resp., NA. Ns) of times that each method was best for Task-Z (resp., Task-A, A, S) in each dataset group in the form 'Nz, Ns', and their summation in the column 'SUM. The larger the number, the better than method is for that dataset group and that task.
Learning Labeling
RW
EF3
EF5
EF10
EL3
EL5
EL10
SUM
AD
BOT
NNT
0,2,0
2.1.1
0.0,0
1,0,0
0,0,0
4,1,1
9,5,3
1.2.0
1,2,3
1.1.1
3,3,3
4.3.1
0.1.0
3,1,3
21,14,13
8,8,3
MT
0,0.0
1.0.0
0,0,0
0.1.0
2,1.1
1.0.0
0,0.0
4.2.1
Hinge-IT
ST
0.0.0
1.0.0
0.1.0
0.0.0
2.1.1
1,0.1
0,0.0
4.2.2
non-ordered
EOT
3,3,1
0,0,0
2,0,2
1,0,2
1,3.1
0,0,1
0,0,0
7,6,7
Hinge-IT
MT,ST
0.0.0
0.0,0
1,0.0
2.2.1
0.0.0
1,0.0
4.2.1
ordered
BOT
2,2.1
1,1,1
2,3,3
1,0,1
1,0,0
0,1.0
2,0,0
9,7.6
Hinge-A4
MT.ST
0.0.0
3.3.3
2,2.1
1,2.1
6,10.6
ordered
FOT
2,3,4
1,2,1
2,5,0
2,1,2
0,0,1
1,1.3
0,1,1
8,13,12
Logistic-ITT non-ordered
MT
0.0.0
2,4.1
0,0.0
0.0.0
1.2.1
0.0.0
0,0.0
3.6.2
ST
0.0.0
2.4.1
0.0.0
0,0.0
12.1
0.0.0
0.0.0
3.6.2
EOT
2,3,4
5,4,5
0,0,0
0,0,1
2,2,2
1,1,1
2,2,1
12,12,14
Logistic-TT ordered
MT.TT
1,0.0
1.1.1
1,0,0
2,0.0
3,5.2
1.0.1
3,1.0
12,7.4
BOT
1,0,0
8,7,6
4,4,3
2,2,3
5,5,4
4.6.4
7,8.6
31,32,26
Logistic-AT ordered
MT.
LB
0.0.0
0.0.1
0.2.1
1.2.3
2.4.3
3.4.5
1.3.4
0.0.1
1,0,1
0,1,0
0.1.3
0.3.3
3.3.3
8, 8,13.20
5,13,8
EOT
2,4.5
3,2,5
4,2,3
1,2,5
2,1.2
4,4,4
2,2,2
18,17,26
OLR-NLL ordered
MT.ST
LB
1,0,0
1.0.1
2.2.1
1.2.2
1,3,1
133.1
2.12
1,0.1
0.0.1
1.1.0
2,4.2
9.14.
9.14.9
BOT
4,4,0
2,3,2
3,6,7
2,8.5
3,4.3
6.6.5
1,4.4
21,35,26
42


Reviews:
[{"review_id": "YB2rWfZU0Q", "reviewer": "Reviewer_RVhG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper nicely describes different 1DT methods under the same notation. I find the rewritten Sections 2 and 3 to be much easier to understand than in the previous version.\n- The proposed EOT approach beats competing methods in most cases. \n- Not any expert in the area, but related works seem to be sufficiently covered and discussed, the previous comments of the reviewers regarding this aspect seem to be correctly addressed by the authors in this revision.\n- The paper is sound and well-written. I see significant improvement over the previous version. I didn't notice any mistakes.\n\nWeaknesses:\n- The EOT method could be better explained in the text, as in the previous version IOT was explained. Since proposition of using EOT still seems to be one of the main contributions of this work among theoretical results on sub-optimality of some methods and unification of notation. At the moment, Algorithm 1 is not described in the text, the reader needs to refer to Lin & Li (2006).\n- Because of the lack of explanation, it is not that clear what is authors' contribution and what is the contribution of Lin & Li (2006) is in terms of the EOT approach. Is it only the task under consideration, as explained in Section 5?\n- If I remember correctly, the previous version of the manuscript was criticized for using a significance level\nof 0.1, the authors changed it to 0.05 again in this version using 0.1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider the setting of ordinal regression/classification (OR), where the task is to learn a mapping from X to Y, which is a set of classes with ordinal relations. The author focuses on the group of one-dimensional transformation (1DT)-based methods that first map X to a continuous score and then apply the labeling function that uses thresholds to decide on the final class by checking to which interval between thresholds the score belongs. The authors discuss a few methods for learning a mapping from X to score by using different surrogate losses (since task losses considered in OR have a discreet nature) and different labeling functions for mapping scores to Ys under unified notation. The authors demonstrate that discussed 1DT methods can become sub-optimal for the minimization of task risk. Because of that the authors propose to apply Empirical Optimal Threshold (EOR) labeling procedure that aims to find a proper threshold by minimizing empirical task loss on available data. It is done by finding optimal thresholds turning the problem into a combinatorial problem, and solving it using dynamic programming (an algorithm proposed by Lin & Li (2006)). The proposed approach is evaluated for different methods for learning scores and compared with other labeling methods on eight datasets. The proposed approach improves mean test task loss in most of the cases, mainly being worst only on the WQR dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe that description of the EOT method and Algorithm 1 would be a nice addition that would make the paper more complete.", "broader_impact_concerns": "I have no concerns about the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qmWryCQGTY", "reviewer": "Reviewer_Z9Q8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The submission no longer presents a new algorithm, but focusing on the existing optimal algorithm instead seems a good choice. It may also be useful to have this existing algorithm spelled out in detail.\n\nThere is a nice discussion of all the various approaches from the literature that fit into the threshold-based framework, and it is interesting to see that some likelihood-based methods can also be covered.\n\nThe submission includes experiments on a fairly extensive collection of real-world ordinal regression datasets.\n\nThe observed improvements in predictive performance are rather small.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission has been changed to focus on an (existing) dynamic programming algorithm that finds the optimum thresholds for the discretization problem considered in the paper. The previously proposed algorithm has been removed. \n\nThe empirical results indicate that by applying the dynamic programming algorithm, it is possible to obtain small improvements in zero-one loss, absolute error, and squared error, for some datasets and some ordinal regression methods considered. \n\nThere are also some theoretical results (already present in the previous version) showing that certain types of likelihood-based ordinal regression methods can be interpreted in the threshold-based framework.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The submission seems close to being ready for publication. However, I am disappointed to see that the significance level for the Wilcoxon rank sum test has been changed back to 0.1 (after this had been changed to 0.05 in the previous version following my concerns). Moreover, and I hadn't spotted this previously, a one-sided test is used instead of a two-sided one, which makes Type I error even more likely! Please use a 0.05 level with a two-sided test.\n\nThe dynamic programming algorithm (Algorithm 1) appears to depend on the fact that the loss function can be decomposed into a sum over subsets of instances. What about target losses such as the quadratic weighted kappa, which has been used in competitions involving ordinal regression problems (https://www.kaggle.com/c/prudential-life-insurance-assessment/overview/evaluation)? \n\nThe impact of K on the runtime of Algorithm 1 should be made clear. The algorithm appears to be quadratic in K, which could be relevant for problems with a large number target values. The min operation in Line 8 is linear in k.\n\nWhy are the results for SVOR no longer included in the table with results (Table 2 in the new version)?\n\nTypos, etc.:\n\n\"imagine that\" -> \"because, generally,\" ?\n\n\"replacing ... to\" -> \"replacing ... by\"\n\n\"can predict it as\" -> \"one can predict it as\"\n\n\"minimization of sum\" -> \"minimization of the sum\"\n\nTheorem 3(i): \"such as\" -> \"such as it is for\" ? \n\nTheorem 3(ii): Clarify what $\\sigma^\\prime$ is? Also, \"such as\" -> \"such as it is for\" ? \n\n\"is non-positive and non-negative\" -> \"non-positive (resp. non-negative)\"\n\n\"a threshold labeling ... have\"\n\n\"we changed a learning rate\" -> \"we changed the learning rate\" ?\n\n\"we adopted a model at the timing with the smallest error\" -> \"we adopted the model at the time step with the smallest error\"\n\n\"the one-sided Wilcoxon rank sum test with p-value 0.1\" - why is the p-value back at 0.1? Also, and I didn't spot this last time, why is the one-sided version of the test used, which makes \"significant\" differences even more likely?\n\n\"as its design\" -> \"as its designed to do\"\n\n\"were tie each other\" -> \"were tied with each other\"\n\n\"able to take\" -> \"able to undertake\"\n\n\"the EOT labeling will serve as the other ... components\"\n\n\"contradicts to the equation (24)\" -> \"contradicts equation (24)\"\n\n\"for a some point\"\n\n\"j >= as well\"\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OqyJHFUUmQ", "reviewer": "Reviewer_UiW4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is well written and draws a broad picture of the ordinal regression problem\n- related work is well discussed and theoretical guarantees are also given for already existing methods\n- method based on a computationally efficient computation of the empirical optimal thresholds (quasi-linear time complexity of O(n*log(n)) with logarithmic factor due to a sorting step)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the ordinal regression problem which consists in affecting an input to a certain discrete category. Further, all categories are assumed to be ordered: for instance, the 3 categories \"good\", \"medium\" and \"bad\". The 1-dimensional transformation (1DT) approach relies on two components. First, a mapping $g$ transforming any multidimensional input $x \\in \\mathbb{R}^d$ to a scalar value. Second component, a labeling function $h$ mapping any real number to a discrete category ranging from 1 to K. A natural and popular labeling approach consists in using threshold to discretize the real line.\nBoth components ($g$ and $h$) can be learned from an i.i.d. dataset of input-target pairs $(x_i,y_i)$ valued in $\\mathbb{R}^d \\times [K ]$ so as to minimize an empirical loss $\\frac{1}{N}\\sum_{i=1}^n loss(h\\circ g(x_i), y_i)$.\nThis paper focuses on the learning of the second component, namely the labeling function $h$, given some 1DT $g$.\nAfter reviewing and providing theoretical guarantees for existing labeling methods (minimum threshold, summation threshold, nearest-neighbor threshold, labeling based on likelihood ), the authors describe their own method consisting in choosing the threshold parameters based on an empirical risk minimization procedure. From a computational point of view, the authors recall that these empirical optimal thresholds can be efficiently computed via dynamic programming.\nNumerical experiments on several datasets (MORPH-2, CACD,AFAD ) illustrate the efficiency of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "minor concern: lack of discussion about the full joint optimization problem, over both components $g$ (1DT) and $h$ (labeling function)", "broader_impact_concerns": " ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The author studies the one-dimensional transform (1DT) methodology to solve the ordinal regression problem, which projects the data by some scoring function and then quantizes the scores with respect to some loss function of interest. The author discusses the methodology with a decent review of the field, solve the optimization problem with a pre-existing dynamic programming method (that was not deeply studied), and demonstrates that the proposed method achieves superior performance on several data sets.

All reviewers agree that the paper is significantly improved over the previous version and should be accepted. The contribution is on revisiting a simple solution for the task, gluing existing work together with the 1DT perspective, and sufficient experiments to back the solution. The author is encouraged to address all remaining issues pointed out by reviewers in the camera-ready version.

While one reviewer recommended the survey certification, the AE checked the paper and believed that the survey part was insightful but arguably not broad enough for this mature field. Hence the survey certification is not recommended.
Decision: Accept (poster)