Unimodal Likelihood Models for Ordinal Data
Authors: Ryoya Yamasaki
Venue: Accepted by TMLR ()

Abstract: Ordinal regression (OR) is the classification of ordinal data, in which the underlying target variable is categorical and considered to have a natural ordinal relation for the explanatory variables. In this study, we suppose the unimodality of the conditional probability distribution of the target variable given a value of the explanatory variables as a natural ordinal relation of the ordinal data. Under this supposition, unimodal likelihood models are considered to be promising for achieving good generalization performance in OR tasks. Demonstrating that previous unimodal likelihood models have a weak representation ability, we thus develop more representable unimodal likelihood models, including the most representable one. OR experiments in this study showed that the developed more representable unimodal likelihood models could yield better generalization performance for real-world ordinal data compared with previous unimodal likelihood models and popular statistical OR models having no unimodality guarantee.

Published in Transactions on Machine Learning Research (10/2022)
Unimodal Likelihood Models for Ordinal Data
Ryoya Yamasaki
Deportment of Systems Science
Graduate School of Informatics, Kyoto University 36-1 Yoshida- Honmachi, Sakyo-ku, Kyoto 606-8501 JAPAN
Reviewed on OpenReview: https: // openreview. net/ forum? id- id 110sCLLiPe
Abstract
Ordinal regression (OR) is the classification of ordinal data, in which the underlying target variable is categorical and considered to have a natural ordinal relation for the explana- tory variables. In this study, we suppose the unimodality the conditional probability distribution of the larget variable given a value of the explanatory variables as a natural ordinal relation of the ordinal data. Under this supposition, unimodal likelihood models considcred to be promising for achieving good gencralization performance in OR tasks. Demonstrating that previous unimodal likelihood models have weak representation ability, we thus develop more representable unimodal likelihood models, including the most repre- sentable one. OR experiments in this study showed that the developed more representable ble unimodal likelihood models could yield better generalization performance for real-world or- dinal data compared with previous unimodal likelihood models and popular statistical OR models having no unimodality guarantoc.
1 Introduction
Ordinal regression (OR, also called ordinal classification) is the classification of ordinal data, in which the underlying target. variable is categorical and considered to have natural ordinal relation for the explanatory variables; see Section 2 for detailed formulation. Typical examples of the target label set of ordinal data are sets of grouped continuous variables like age groups {'O to 9 years old', '10 to 19 years old', years old', 'over 100 years old'} and sets of assessed ordered categorical variables like human rating { ('strongly agrec', 'agrec', 'ncutral', 'disagrec', 'strongly disagrec') (Anderson, 1984), and various practical tasks have been lackled within the OR framework: for example, face-age estimation (Niu et al., 2016; Cao et al., 2019; Anonymous, 2022), information retrieval (Tiu, 2011), credit or movie rating (Kim & Ahn, 2012; Yu et al., 2006), and questionnaire survey in social research (Chen et al., 1995; Bürkner & Vuorre, 2019).
'90 to 99
Consider example of the questionnaire survey about support for certain idea that requires subjects to respond from ('strongly agree', 'agree',... Here, it would seem possible that subjects, who have features specific to those who typically respond 'agree', respond 'neutral', but unlikely that they respond 'disagree'. Such a phenomenon in OR tasks can be rephrased as the unimodality of the conditional probability distribution (CPD) of the underlying target variable given a value of the underlying explanatory variables. The hypothesis "many statisticians or practitioners often judge that the data have natural ordinal relation and decide to treat them within the OR framework, with unconsciously expecting their unimodality" may be convincing,
will experimentally confirm in Section 6.2 that many ordinal data, treated in previous OR studies that do not consider the unimodality, tend to have a unimodal CPD.
Commonly, the generalization performance of a statistical model or classifier (OR method) based on that model depends on the underlying data distribution and the representation ability of that model. Recall that the generalization performance can be roughly decomposed into bias- and variance-dopendent terms (well- known birs-variance decomposition); model in which the representation ability too strongly restricted will result in large bias-dependent term if it cannot represent the underlying data distribution, and model

---

Published in Transactions on Machine Learning Research (10/2022)
ACL (s..) & a
THE
m1.9
Unimodal
VS SLea
DRs'
Unimodality Constrained
Full-FDF
12
ORD-ACL 1840)
OH-VS-SLesa
OH-ORD ACDDD-ACLI (822)
mh.31
2-FDF
OHI-PO] 83
O s0.221
1-FDF
PO-ORD-ACL (85.12
12.14
Th.34
1h.20
POL 1832)
BIN (a.
Figure 1: Relationship among the representation abilities of 12 likelihood models that we consider in this paper (except for Section 5.3). We describe the name of a model and the section number the section which the model is studied in rounded box. Models in rounded boxes with gray background are previous ones, and models in rounded boxes with white background are proposed in this paper. The directed graph with 'rounded box' box'-shaped nodes implies that models in parent node are more representable than models in a child node in the sense of Definition 5. Also, we describe the theorem number of the thcorem that shows the relationship in a square box.
that is unnecessarily flexible to represent the underlying data distribution will result in a large variance- dependent term especially for a small-size training sample, and such models at both extremes may yield bad generalization performance. Therefore, assuming the unimodality of the CPD a natural ordinal relation of ordinal data, unimodal likelihood models, which adequately represent such unimodal data and are compact with respect to the representation ability than unconstrained statistical models such as multinomial logistic regression model, are considered to be promising for achieving good generalization performance in the conditional probability cstimation and OR tasks.
The existing studies (da Costa et al., 2008; Tannario & Piceolo, 2011; Beckham Pal, 2017) were inspired by the shape of the probability mass function (PMF) of elementary categorical probability distributions such binomial, Poisson, and uniform distributions and developed unimodal likelihood models. In this paper, we introduce several notions that characterize the shape of CPDs in Section 2.1 and notions that characterize the representation ability of likelihood models in Section 2.2. Then, the basis of these notions, clarify characteristics of data distributions that their likelihood models can not represent well and show that their likelihood models have a weak representation ability, in Section 3. We thus propose more representable unimodal likelihood models in Sections and 5. Those proposed unimodal likelihood models bridge the gap between previous unconstrained likelihood models (e.g., ACI and SL models in Figure 1) that have no unimodality guarantee and may be unnecessarily flexible and previous unimodal likelihood models (e.g., POI and BIN models in Figure 1) that may be too weakly representable. In particular, VS-SL model described in Section 4.2 is the most representable one among the class of unimodal likelihood models.
We performed experimental comparisons of 2 previous unimodal likelihood models, 2 popular statistical OR models without the unimodality guarantee, and proposed unimodal likelihood models; see Section 6 and Appendix C. Our empirical results show that the proposed more representable unimodal likelihood models can be effective in improving the generalization performances for the conditional probability estimation and OR tasks for many data that have been treated in previous OR studies ordinal data. On the basis of these
2

---

Published in Transactions on Machine Learning Research (10/2022)
results, this study suggests the effectiveness of the proposed unimodal likelihood models and OR methods based on those models.
2
Preliminaries
2.1 Ordinal Regression Tasks and Ordinal Data
The OR task is classification task. Denoting explanatory and categorical target variables underlying the data as X E Rd and Y E [K] == (I.....K), we formulate the OR task as searching for classifier f : Rd - [K] that is good in the sense that the task risk B[((f(X),))) becomes small for a specified task loss e: (: [x]K]² - [0), where the expectation B[-] is taken for all included random variables (here X and Y). Popular task losses in OR tasks include not only the zero-one loss (z)(i.k)= = 1(j # k}, where 1(c) values if a condition c true and 0 otherwise, but also V-shaped losses reflecting one's preference of smaller prediction errors over larger ones such as the absolute loss fabs(j.) k) == --j-kt k| and the squared loss (a = (j (j- k)?.
In the OR framework, it is supposed that the underlying categorical target variable y of the data is equipped with ordinal relation naturally interpretable in the relationship with the underlying explanatory variables X, like examples described in the head of Section 1. We here assume that the target labels are encoded to I....,K K in an wyder-preserving manner, like from 'strongly agree', 'agrec', 'neutral', 'disagroe', 'strongly disagree' to ,,...,5 or 5,...,1. The OR framework considers the classification of such ordinal data. Note that, like most previous OR studies have discussed the OR without formal common understanding of what constitutes ordinal data and their natural ordinal relation, it is difficult define ordinal data any more rigorously, and in this paper refer to the data discussed in previous OR studies ordinal data.
As we declared in Section 1, this study basically assumes, natural ordinal relation of ordinal data, the unimodality in the theoretical discussion or the almost-miinodality to real-world ordinal data, precisely defined in the following:
Definition 1. For vector p (Pk)KE(K) E RKK define M(p) == min(arg (px, and say that p is unimodal if it satisfies
P1 S ... S PM(p) and PM(p) 2 .... PK.
(1)
Also, we call M(p) the mode of p if p is a PMF satisfying p E AK-1, where AK-1 is the (K -idimensio probability simplez {(p))e([]] ΣK Pk 1,px E [0,1] for k = I.....KI. Moreover, if the CPD (Pr(Y yl )])) is unimodal al any in whole the domain Rd or in ils sub-domain X S Rd with a large probabilily Pr(X E X), we say that the data is unimodal or almost-unimodad
This paper presents a qualitative discussion on the representation ability of various statistical OR models. It aims to clarify whether each model is suitable for representing data that follow distribution with various structural properties that we consider to be expectable in unimodal ordinal data. As a preparation for that discussion, we further introduce notions that are related to structural properties of the data distribution.
We first define decay rates of a PMF as the rate at which the probability value of that PMF decays from the mode toward both ends (1 and K):
Definition 2. For a PMF
(pe)Re(K) E Âx-1 having mode m E [K], we call Pt for k = 1...,m Pa-1
(if m + 1) and Pk P1-1 for k = m + 1, 1....,K (if m + K) the decay rates (DRs) of p, where AK-1 == ((x))<(K) Σ Pk 1. E (0,1) for 1. K).
Note that we here introduced Âx-1 to avoid the situation where probability value becomes exactly 0 and technical problems associated with such a situation. It will appear several times in the following discussion.
Commonly, scale of a PMF refers to the degree of spread of that PMF, The following definition gives a numerical characterization of the scale used in this paper and introduces related notions.
arg min and arg max may return a set, and so min applied convert it a point and simplify the discussion.
3

---

Published in Transactions on Machine Learning Research (10/2022)
El 0.01 Ei Ps
Ee Ps = 0.29, h
El Ph 82 0.18, Si 0.60
1.0
4.0.
2..4
(nche
0,0
Figure 2: Instances of the unimodal 10-dimensional PMFs (px)εetoo) in which the mode 5, in black solid line, and their DRs [ Pt 14 and { P1-1 k in red dotted line.
Pa-i k==
Definition 3. For PMF p (pR)<e(K) Ax-1 hoving mode m E [K], call Σm=l Pk ΣK Ex Pk 11-pm the scale of p, where we define the summation notation Σ such that E fk = as far as > regardless of (fc). If the scale of the CPD p(æ) = (PrYY = y| x)) is similar or dissimilar over whole the domain Bd of the explanatory variables X, we say that the data homoscedastic or heteroscedastic. In particular, we refer to heteroscedastic data as mode-wise heteroscedastic or overall heteroscedastic data, if the scale of their p(a) is similar or can be dissimilar in domain where the conditional mode M(p(x)) is the same.
Commonly, skewness of distribution refers the asymmetricity of that distribution (around its mode). The following definition gives numerical characterization of the skewness of the PMF used in this paper and introduces related notions.
Definition For a PMF p = (PR)RE[K) Ag-1 having a mode m E [K], we call Em-1 Pk -EK Ek-mal Pk the skewness of p, and we say that p is skew less skew when the absolute value of its skewness (absolute skewness) is large or small. If the absolule skewness of the CPD p(æ) (Pr(Y yl x))ye(K] is large or small over whole the domain Rd of the explanatory variables X, say that the data is skew less skew.
For unimodal PMF, its DRs are smaller than or equal The scale and skewness of PMFs are typically treated as qualitative notions, but mcasure them numerically for the sake of clarity (sce Σx=l Pk + Ex-m=1 Pk and Σx= Pk - Σt=m+1 Pk in Definitions and 4). The numerical measure in each definition is intended just to further clarify our qualitative discussion, and we do not see any particular importance in that choice of the measure (other mcasures can be used instead). The readers should understand our treatment of them by referring to the description of Definitions 2, 3, and and Figure 2. The figure displays 3 instances of the unimodal PMFs and their DRs: we say that the right PMF has a larger scale than the left PMF and that the center PMF is less skew and the right PMF is skew.
2.2 Likelihood Models and Ordinal Regression Methods
Suppose that one has a set of observations (ordinal data) {(2;,))) each of which drawn independently from an identical distribution of (X,Y). Every statistical OR method covered in this paper (except for AD tried in Section 6) assumes, as model of the conditional probability Pr(Y = y| x), certain likelihood model Pr(Y y| x)) P(y: g(z)) with fixed part P (say, the softmax function in multinomial logistic regression model) and learnable part g E (say, certain neural network model), where we call P the link function and the learner model in a learner class G. Note that the distinction between P and g is to aid in understanding relation between multiple models, and this paper does not emphasize a strict mathematical distinction (note that, for example, (P(:)-).9) and (P(:;2-), (g/2 E G)) yield an equivalent likelihood model).
The key notion in the discussion of this paper the representation ability of the likelihood model. Formally, the representation ability of the likelihood model we discuss is defined as follows, based on the inclusion relation of the function space corresponding to the likelihood model:
4

---

Published in Transactions on Machine Learning Research (10/2022)
Definition 5. We say that the likelihood model based on (P1.91) has stronger representation ability (or is more representable) in the formal sense than that based on (P2. G2), if
D0Par Ig G1} 2 ((20::90)aa
G2).
(2)
The representation abilities of two different likelihood models may not be formally comparable according to Definition 5, but even in such cases we in this paper discuss the relationship between the representation abilitics from a qualitative understanding of those likelihood models as much as possible. In the following part of the paper, we describe results obtained via such a discussion as "one likelihood model has a stronger representation ability (or is more representable) in the informal sense than the other", and distinguish them from results in the formal sense that follow Definition 5.
Additionally, we introduce the functional degree of freedom:
Definition 6. The functional degree of freedom (FDF) of a vector-valued function fi Bd - R to the Rd - R. required for fi(.) f2(8:(().....88M)) hold with
minimum number of functions 81.....8M 8M
certain vector-valued function f2 RRM - B.
In this paper, we discuss the FDF mainly for CPDs or likelihood models: For example, general CPDs (Px(Y = y|X = yy]K]]] including those underlying unimodal data have up to (K - 1)-FDF (or called full- FDF) (it is not K since Σy=1 Pr(Y = y|X = 1, and can be smaller than d when d < K), while likelihood models in Figure 3 are 1-FDF. We use the FDF as one simple indicator of the representation ability of statistical likelihood models; we consider that the larger its FDF, the stronger the representation ability of the statistical model tends in the informal sense.
In this paper, we set up statistical OR method as learning model g from the class G through the maximum likelihood estimation maxgeg 1 > log P(y;; g(æ;)) (which we call conditional probability estimation task), and then constructing classifier under the task with the task loss & as f(æ) = (((((():(=))))441) with f(((p))<())) == min(arg min, PI: p))(j,)))eek) and obtained model 9.
Therefore, the only difference of statistical OR methods in this paper appears in the link function P and learner class G of the likelihood model. Under these settings, generic principles based on the bias-variance tradcoff suggest that a compact likelihood model that can adcquately represent the data is promising for an OR method with good generalization performance. On the other hand, we believe that subsequent discussions hold as well even if changing the loss function used in the parameter fitting procedure and the decision function: As options for the parameter fitting other than the maximum likelihood estimation, we apply robust alternatives (Bianco & Yohai, 1996; Croux et al., 2013) to every likelihood model treated. Anonymous (2022) has developed a decision function that allows misspecification of the likelihood model.
3
Existing Unimodal Likelihood Models
3.1
Binomial Model
The (shifted) binomial distribution (Pi(k: ())ce(]) is unimodal any p E [0,11), where
Po(k; = K -
k- x (1 x)* for k E [K], p E [0.1].
(3)
A! and where (9) MK-DD is the binomial coefficient. Inspired by the shape of the PMF of the binomial distribution, da Costa et al. (2008) considered a unimodal likelihood model based on the link function
| u) = Poly; T- ly
K 1K=11 1
(4)
l+
and an R-valued learner model g(x) (applied to in (4)). Thereafter, Beckham & Pal (2017) introduced the scaling factor: in other words, they proposed the link function
Prin(y;uss) = clog(Po6::1nd- 1//x
Σk (e for y [K]. E R, E (0.
(5)
5

---

Published in Transactions on Machine Learning Research (10/2022)
0.2
1.0
0.6
0.1
0.2
0.0
Figure 3: Instances of the BIN model with K 10.2
and applied a az-dependent R-valued model and æ-independent positive parameter to We call this likelihood model, which consists of the link function Pbin and learner class G {(g(+)s)) g Rd R oo)), the binomial (BIN) model.
The BIN model (Poii((gg(()))..)))))))))))))()) y((k]] is parametrically constrained with the 1-FDF. The mode of the binomial distribution
(((Pu(k: (K)pa((k:)))))))) = min([[Kp]][[K]) [[]]][]]]]
(6)
where [.] and [.] are the ceiling and floor functions, and hence max(1, Kp) s M((Pb(k; p))eek)) s min(Kp (Kp + 1,K3. This property and simple calculations can show the mode, unimodality, and DRs the BIN model: Theorem 1. holds that
Pon = (K-yip) 1/8 with p I-e-a for all y [K 1], u e R, s (0).
Poinyya.))
Then, for any R. and s o0), it holds that, for
that is (6) with p T+e
(ii) (bin(1;,s)) S ... Pbiu(m;; u. if m + 1, and (m; 2 Poin(K; u, s) if m # K,
(iii)
Pain(asss) Poin(2ar..)
S
Puin(m=1:0. lan,s) (mmn())
(s 1) if # 1, and (12)
Puig(m+ Poml(mass))
(oin(Ksu,s) Poin 11:.)) if K.
First note that Theorem (ii) can be seen direct corollary of Theorem (iii). Theorem (ii) implies that the BIN model is guaranteed to be unimodal. We call the constraint on the sequence of DRs like Theorem
(iii) as the DRs' unimodality (constraint), considering that (a.... Pm-1 Pm-1 Pm Pm ..... PK-1
PK is unimodal if a
PMF (Pи)kε[K) having a mode m satisfies that constraint. One can find that, owing to the DRs unimodality
constraint, the BIN model cannot exactly represent unimodal MPFs in Figure 2. In addition to the DRs unimodality constraint, the BIN model (Ppinly: g(z)). s)){ek) is always less skow especially when its mode is close to the labels' intermediate value 1*K of the binomial distribution (Py(k; p)) are close (Kaas Buhrman, 1980)), and tends homoscedastic (see Figure 3; there, the scale maxy (Pbin (y; u, s))ye(10) for each is similar for many except at both ends). We will relax the restriction of the representation ability regarding the scale of the BIN model in Section 5.2, but that relaxed model cannot avoid the restriction regarding the DRs and skewness.
2 (recall that the mode (6) and median, [(K-1)p]+] or [(K-1)p]++ +1,
3.2 Poisson Model
The (shifted) Poisson distribution (Pp(k; d))ket is unimodal at any A 0, where
- for E de (0,00).
Pp(k;)) (k - 1)!
(7)
da Costa et al. (2008) truncated (7) within [K] and normalize it to develop unimodal likelihood model:
Pr(Y y| = æ) P)(r: P,kk; g(x)) with positivo-valued learner model g(x)) As in the case for
2The display style in Figure is like showing a PMF created by cutting curves Figure 3 each

---

Published in Transactions on Machine Learning Research (10/2022)
0.2
1,5
0.6
0.4
0.2
0.0
logim)
Tog(n)
logia)
Figure 4: Instances of the POT model with K 10.
the BIN model, Beckham & Pal (2017) developed a scaled link function
Ppoi(Y) u) = e'vls Σx evnts for E [K], u,s E (0,,0), with V2 = (k - 1) 1))lg(u)) - log((k 1)!) for k E [K], (8)
which is generalization of (ori(y; of (da Costa et al., 2008). They applied æ--lependent positive- valued learner model g(x) to and æ-independent positive parameter (i.e., ((s(), g Rd (0, oo), (0)) we call this likelihood model the Poisson (POT) model.
The POL model (P (Ppo s))) is also parametrically constrained with the 1-FDF. We find that the POI model is a special instance of PO-ORD-ACL and PO-ACL models discussed later (in Sections 4.1, 5.1, and 5.2) and has a weaker representation ability in the formal sense; see Theorem 15. The POT model (Ppoi(y; g(z) has DRs' unimodality constraint similar to the BIN model. Also, always has mode- wise heteroscedasticity especially when is small and K is large: its scale tends small and large respectively when M((P (y: (y) s))(k) is close to 1, K, smaller side of (2..... K - 1} and larger side; see Figure 4 with s = 0.2. Therefore, it is not suitable for representing homoscedastic or overall heteroscedastic data.
4
Novel Unimodal Likelihood Models
4.1 Ordered Adjacent Categories Logit Model
The previous BIN and POI models are ensured to be unimodal, but have just 1-FDF and may be too weakly representable for some data. We thus developed more strongly representable unimodal likelihood models that have up to (K - 1)-FDF. This section describes ordered adjacent categories logit (ORD-ACL) model developed by modifying cxisting ACL model that does not have the unimodality guarantee.
The (naïve) ACL model designed to model the relationship between the underlying conditional probabilities of the adjacent categories, Pr(Y=y x=x) P=(
as
Pr(Y = y|xx
Pr(Y E (y,++1)|X =x) 1 (y( for E [K -1]
(9)
with an (KK-I)-valued learner model gi see Simon (1974); Andrich (1978); Goodman (1979); Masters (1982), and Agresti (2010, Section 4.1). Considering the normalization condition Σ Pr(Y y| x)) 1, it be seen that the ACL model depends on the ACL link function
Σ e- = e -1- m
(act =
for y y [K], u K
(10)
Σk_ e -
together with the learner class G (g Rd - RK-1), where we define the product notation Il such that
IT fk = 1 as far as i >> regardless of [fi)
The ACL model can represent arbitrary CPD in the following sense:
Iin (Beckham & Pal, 2017). va (8) defined with minus u, but which irrelevant after the softmax transformation.

---

Published in Transactions on Machine Learning Research (10/2022)
Theorem 2. It holds that
(Paal):g(Weee Ig: Rd - -
= {p: Rd Ak-1)-
(11)
Theorem 2 implics that the naive ACL model does not have the unimodality guarantee, while analyzed properties of the ACL link function and then found simple condition for its learner class under which likelihood model based on the ACL link function gets the unimodality guarantec:
Theorem 3. It holds that
(i) Paaly+ = = for all [K - 1], u R--
If u E k-1 satisfies un(:= -00) S ... S 4m-1 S S Um S ... S.. (x(=== oo) for some m [K], holds that
(ii) (aci(1; u) S ... E (m; u) if m # 1, and Par u) ... (acl(K; if m K;
(iii)
Paci(l:))
Pag(m-11:))
(s 1) if 1, and (12) Pacl(m+1a))) (ac(();))
Pac(Ksu) Paol
K.
(an((:
(ac(mma)
On the basis the unimodality guarantee stated in Theorem 3 (ii), we propose the ORD-ACL model (Pacily; ((x))))(1 applying the ACL link function (10) and ordered learner model g(x) satisfying that 81(2) S S (x)) for any x Rd. Here, the ordered learner model ğ can be implemented
for (x)x) = ( + p(gx(x)). for k K
(2)),
(12)
with another model g : Rd R-1 and a fixed function that satisfies
(p(u) u R) (0,
(13)
such as Psq(u)= = u2 (we denote the procedure (12) as ğ = p[gg))
The ORD-ACL model can represent arbitrary unimodal CPD that has the DRs' unimodality constraint:
Theorem 4. For any function p satisfying (13), holds that
((Pact):; (lo()) g Bd
RK-1
(14)
{p: Rd Âx-1
(((()) P2(x)" (M))pa))(())) PMripaal(e) pk-il
PMiplan)1(0)) (()) PMiplesn1(()
(m))
is unimodal for any Rd).
It will be a good aspect in the modeling of the ordinal data assumed be unimodal that the ORD-ACL model has the unimodality guarantee. In contrast, the ORD-ACL model incidentally imposes a constraint the DRs; even so it is important that the ORD-ACL model has stronger representation ability in the formal sense than the BIN and POI models, which also have DRs' unimodality constraint.
4.2 V-Shaped Stereotype Logit Model
We next describe V-shaped stereotype logit (VS-SL) model, which is unimodal, full-FDF, more representable than the ORD-ACL model in the formal sense (see Theorem 10), and developed by modifying existing SL model (also called multinomial logistic regression models).
The SL model (refer to Anderson (1984) and Agresti (2010, Section 4.3)) attempts to model the conditional probabilities of multiple categories paired with certain fixed (stereotype) category, PH|==1X===)
as P=
PI(Y = 1 = x) Pr(Y E (1,y) = (.(m) for E [K]
115)
4If alternative Pexplu) e is used the function p, inequalities in the ordering condition 9 and the unimodality condition reduce the strict inequalities. Using such a p function daes cause serious computational problems.
8

---

Published in Transactions on Machine Learning Research (10/2022)
with an valuul learner model g such that gI(x) 0 for any a Rd. Considering the normalization
condition, we introduce the SI. link function
Pa(y; ==
M e for y [K], u EER..
(16)
The SL model is based on the SL link function Pst and learner class G = {g Rd - RK 181()) = =).
The SL model can represent arbitrary CPD:
Thcorem 5. holds that
{(Pal): Myakk g Rd R 00 - 1g Rd - R = {p- Rd - Ax-1). (17)
The ACL and SL link functions have only minor differences in parameterization, but looking at the SL link function will reveal another simple way to create unimodal likelihood model as follow:
Theorem 6. If R satisfies that i1 2 ... 2 Um if m # 1 and um ..... I uK if m # for some m [K], it holds that Pa(11: u) S ... I Pa(m;; u) if
and Pa(mm u) 2 2.... Par(K; u) if m # K.
On the ground of this theorem, proposed VS-SL model (Pa(y; y==))) based the SL link function Pst and a V-shaped learner model ğ = r(g) (described below) with و p(g] and another R RK -vauued learner model g. Namely, the VS-SL model applics the learner class G (t(pgg) | g Rdd - RK) for the link function Pat First, transforms an arbitrary model to an ordered model 9 (so g1(#) < ... I (x(x)) for any æ E Rd). Next, transforms an ordered model g to a V-shaped model r(ğ). Here, the notation r(u) for an & -vauued object u implies the element-wise application a [k)k=]]]]] and the function T is supposed satisfy
T()) is non-increasing and (v(u) s 00 (T(u) # 0} (o),
and non-decreasing u O,
(18)
such as (u) = |u| and (n) == n². If r = Tabs Taqo it holds that (1())) 2 ... 2 (a) and (m) s ... s (x)()) with m(x) min(arg min; (l&x (x)),,)))) as required in Theorem Finally, the SL link function transforms ğ to unimodal likelihood model.
The VS-SL model (Paly; yy=)) is ensured to be unimodal and further can represent arbitrary unimodal CPD:
Theorem 7. For any functions p satisfying (13) and satisfying (18), it holds that
((Pa0;:(oll0)))aa)) g Rd R (p Rd AK-1 p(x) is unimodal for any xr Rd). (19)
Accordingly, we propose the VS-SL model the most representable one among the class of the unimodal likelihood models.
5 Variant Models
5.1
Proportional-Odee Models
The ACL model (aa(()g(())) ORD-ACL model plox)))) SL model (Paly: (())))(())))))))))))) gy=))) and VS-SL model (Paly: (() have up to full-FDF and may be too flexible and difficult to learn for some data especially when the sample size is not large enough. In the statistical research (e.g.. see McCullagh (1980)), the proportional-odds (PO) constraint that constrains the FDF of the likelihood model to often applied avoid such a trouble. We call (Pactly; g(z) 1))x<(k] the PO-ACL model, (Pectly; ((6]-g(x)) 4)4= the PO-ORD-ACL model, and (Paly;((plbl ((x)) ())))1))) the PO-VS-SL model, where b and are learnable parameter and all-1 vector with appropriate dimension ((K-1) 1) for ACL and ORD-ACL models, or K for VS-SL models) and g is an R-valued learner model.5
The PO-SL model Paty;; b- bby / bb is constant regarding æ and may be weakly representable. Therefore, Anderson (1984) considered to use learner model g(rr) a- -og(g))eee with learnable parameters e. R here) s.t. 0j = cj =
9

---

Published in Transactions on Machine Learning Research (10/2022)
(0, 3,
(0, 3,,,2,2,,,,,,44,,,,,,,,
1,,,,,,1,,,,3 7.4.88
1.,,
1.0
03.5
0.2
0.0
Figure 5: Instances of the PO-ACL model with K = 10.
1, 4,5, 7,8,8 0.2
(0, 4,5,6, 6.6.T. 7,8%
b = (0. 1,2,3,43333
Pelgr 1orlb-ur (1-1--1)-00)) 11/51
0.69
0.4
0.2
3
Figure 6: Instances of the OH-ACL model with K 10.
Recall the POI model reviewed in Section 3.2. The equation
Ppoily; u) Pactly; b- v. 1) for all y [K], with b (log(k)/s)<(x-1) log(u)/s,
(20)
which formalized in Theorem 15, shows that the PO-ACI. and PO-ORD-ACT. models are more repre- sentable in the formal sense than the POI model. The PO-ACL and PO-ORD-ACI. models further represent homoscedastie data and more various mode-wise heteroscedastic data because they can adjust the parameter b or p|b] according to the model fit to the data, compared with the POI model; compare Figure for the POI model and Figure 5 for the PO-ACL and PO-ORD-ACL models.
5.2 Overall Heteroscedastic Models
The BIN, POT, and PO models have just 1-FDF. The strong constraint of the representation ability of these models can be interpreted as incidentally assuming the homoscedasticity mode-wise heteroscedasticity. We in this section describe their overall homoscedastie (OH) extension, OH models.
McCullagh (1980) studied an OH extension of a certain PO model. According to his idea, we propose the OH-ACL model (Pactly; (b (b- - 11)))))))s/()))(/llll and OH-VS-SL model (Pa(y;;(p(b)] - g(x)) 1)/ y(=))) where and 1 are learnable parameter and all-1 vector with appropriate dimension, and and s are R- and positive-valute learner models. The positive- valued scale model be implemented as s(x: = + + p(t(x)) with constant for avoiding zero-division, fixed non-negative function p, and another R-valued learner
OH-ORD-ACL model (Paclyy; {p[b] - g(x) (oz)) 1//s(=)))
The significance of OH models is that their scaling factor can vary depending on æ. Extending previous BIN and POI models by Beckham & Pal (2017), we also introduce the OH-BIN model (Poinyy g()), s(x))) and OH-POI model (Ppol();((z). (()))))))))))))))))))))))))))))) s(x)))y<[K] For the implementation s(x) = r + +(((x)) of the scaling factor enables to represent a wider class distribution. Note that the
for these models, the use of a small OH-POI model is special instance of the OH-ACL and OH-ORD-ACL models in the formal sense, Theorem 15 shows.
and R-valued learner model Rd (we call g rotatable-odds (RO) learner model) for the SL link function: call this model the RO-SL model. We also consider novel RO-ACL model: PO-ORD-ACL) model, but has unimodality guarantee the RO learner model g(æ) g(x) does necessarily satisfy the ordering condition (a))) s ... (x) (depending on c).
representable than the PO-ACL (and
10

---

Published in Transactions on Machine Learning Research (10/2022)
These OH models can represent overall heteroscedastic data and are more representable in the formal sense than their corresponding 1-FDF model (see Theorem 14), but their representation ability is parametrically constrained with the 2-FDF; compare Figures 3, 4, 5, and 6.
5.3 Combinationwwitt-nnffom Models
When a likelihood model (Pr(Y = y| =)))y[[]] is unimodal, the
(CU) likelihood =)])e[K] with e.g., g(x) = I+e-n for an R-valued learner r, = y| ((P()
model, q(z) (+)K + (1=)(z))-
_
is also unimodal. For example, Piccolo (2003); Jannario & Piccolo (2011) studied a combination of uniform and binomial model. The CU extension of low-FDF (like PO and OH) models can increase their FDF by 1.
6 Experiments
6.1 Experimental Purposes
From the bias-variance tradcoff, can be expected that compact likelihood models that can adequately represent the data will yield better generalization performance in the conditional probability estimation task, and accordingly that OR methods based on such likelihood models will yield better generalization performance in the OR task. We performed numerical experiments in order to verify whether the proposed likelihood models, developed based on this working hypothesis, can provide better performances than previ- ous unimodal likelihood models with a weak representation ability and popular statistical OR models with no unimodality guarantee.
6.2 Experimental Settings
We selected 21 real-world datasets of those used in experiments by the previous OR study (Gutierrez et al., 2015) with the total sample size not that is 1000 or more, and used them for our numerical experiments. AB5, CE5' (resp. AB10, CE10") are datasets generated by discretizing a real-valued target of datasets, which are often used benchmark regression methods, by (resp. 10) different bins with equal proportions. SW, ...., CA originally have categorical target, and the authors of (Gutierrez et al., 2015) judged that their targets have a natural ordinal relation.
Table shows the dataset name, dataset properties Noot d, and K, and mean and standard deviation (STD) of 100 test MUs and test DRs MUs. The mean unimodality (MU) is numerical criterion to evaluate the unimodality of the conditional probability distribution model Pr(y- = -|X )) and n used data points, as E [(Pi(Y y|X = a;)))<[K] is unimodal.). As that likelihood model, we used the test SL model under the task-P (described below) that can represent any conditional probability distribution. We trained likelihood model with training sample of size Bur 800, and evaluated the MU with an obtained likelihood model and remaining test sample of size Bies Aut Aim- We repeated this procedure 100 trials with randomly-set different sample setting and initial parameters of the likelihood model to obtain 100 test MUs. We also give a relative reference value in Table 2: It shows a ratio that 108 samples of the PMF uniformly and randomly drawn from the probability simplex in R were unimodal for K 3...., 10. Comparing the test MU in Table and the relative reference value of the same K in Table 2, one can find that many woal-world data treated as ordinal data by existing OR research tend strongly unimodal. Additionally, we introduced the DRs MC n Σn 1((P((Y y|X x;))ye[K] satisfies the DRs unimodality.), and took similar procedure to that for the MU; see Tables 1 and 2. The DRs' MU for ordinal data were larger than those for uniform random PMF, but their difference appears to be relatively smaller than that for the MU.7
the data, and it is defined, for a likelihood
OOn get the datasets from researchers's site (attp://wws.uc) www.:/ccc.eeeeeeeeeee htwpgrpps//aaae htt.gp/rssrraa////eree orreview) of (Gutierrez et al., 2015), or our GitHub repository (Attpe://github com//yamsakrryyya/u.. together with our used program codes.
FFor notions related to the scale and skewness, we introduced numerical measures in Definitions 3 and just for the sake of clarity of the qualitative discussion. Hlwwever did not introduce metrical meaning into these mcasures (and it is quite difficult): example, a gap between scales 0.1 and 0.3 and a gap between scales 0.7 and 0.9 would not have the same meaning. Therefore, we could not evaluate the heteroscedastticity without misleading.
11

---

Published in Transactions on Machine Learning Research (10/2022)
Table 1: Dataset name; dataset properties niot, d, and K, and mean and STD (as 'mean + STD') of test
MUs and test DRs' MUs.
dastsnt name
Mlod
MU
Ditt' MU
datases name
Plot
K
MU
Bift MU
AB5 (shalen5)
1177
10
8915 0662
.4042 1 0745
CO10 (com (computari00)
8192
12
10
.0066 1. 14411
.0935 .0683
BA5 (bankss)
5192
.944
00877
.4288 11588
C010' (computari0)
8192
21
10
.8189 1. 11617
.0517 1 1.737
BAS' (banks))
5192
32
.967 .0167
.7041 s 1.220
CHEO (cal beustngg)))
20640
10
.3311 x. 1173
.0051 0009
005 (computer5)
8102
12
9906 .133
.5623- = 1061
CE10 (consus10)
22784
8
10
.2022 + 1017
.0008 .0018
CD5' (comppaeer3))
8182
21
1.0000+ 0001
.5851 + +117
OFTO (comeust O')
29764
16
10
3151 + 1153
.0727 + -0053
CHS (cal housing)
20840
.9065 2 .0027
.5993 2 2129
SW (SWD)
1000
10
4
.9993 = .0027
..651..920
CE6 (scuvus5)
22784
.7887 00929
.5221 = 1105
LE (LEV)
1000
4
6
.9547 = .0656
.2208 2 1555
CES' [com)us5))
22784
16
.8332= 0736
.3827 = 0978
ER (ERA)
1000
4
9
.7900
+ 1045
.2126 1.400
AB10 (shalento)
4177
10
10
.3218 1331
0076 0164
Wn (mimaquxiity-red)
1898
11
.98941 0047
1.1181.1288
BA10 (bank10)
5192
10
.8923 14463
0226 = 0432
CA (carr)
1728
21
.8736 2037
(bank10")
8102
.3101 12261
0019
Table 2: Ratio of 10 PMF samples uniformly and randomly drawn from Ax-1 that satisfied the unimodality (resp. the DRs unimodality) in the row MU (resp. in the row DRs' MU).
MU
6008
33300
1329
0444
01:25
0032
.0007
.0001 .0000
DR MU
8968
3330
1329
0276
0035
0009
.0011
We considered 4 tasks: conditional probability estimation task (task-P), and 3 OR tasks with the zero-one task loss Ez0 (task-Z), absolute task loss fabs (task-A), and squared task loss bag (task-S), Results for the lask-P, -Z, -A, and -S are respectively evaluated based on the negative log likelihood (NLL), mean zero-one error (MZE), mean absolute error (MAE), and mean squared error (MSE). Note that, for a likelihood model Pr(y- = +| :) and n used data points, the NLL is defined as - Ei= log Pi(Y > = æ;), and the MZE, MAF, and MSE are respectively defined 1 E/=1 (f(x)). yı) with f(x)= f(((P((y = y|X =())) for l = Ezo. Eab A method that yields smaller error value is better for the corresponding task.
We tried 12 likelihood models and OR methods based on those likelihood models: previous 1-FDF BIN and POT models; proposed 1-FDF PO-ORD-ACT. and PO-VS-ST. models; proposed 2-FDF OH-BIN, OH- POL, OH-ORD-ACL, and OH-VS-SL models; proposed full-FDF ORD-ACL and VS-SL models; previous full-FDF ACL and SL models with no unimodality guarantee. See Figure 1 for the relationship among the representation abilities of these models. Here, we used link functions p = Pexp ensure the negaatityyv and for positive learner model for POI and OH-POI models and r Tsq to ensure the V-shape, and a positive constant
0.01 in the scaling model s(x) r p(((z)) for OH models. We implemented all learner models with a 4-layer fully-connected neural network model that shares weights in except for the final layer and has 100 nodes activated with the sigmoid function in addition to bias nodes in every hidden layer. Note that, for the OH models, we implemented their real-valued learner and scaling models (g and in the notation in Section 5.2) with two isolated networks (each of which is described above), because their performance was significantly degraded when the real-valued learner and scaling models were implemented with single weight-shared network. We trained model with a training sample and Adam optimization for 1000 epochs according to the maximum likelihood estimation, and evaluated the NLL, MZE, MAE, and MSE with a remaining test sample at the end of each epoch. Then, for the task-P, -7, -A, or -S, we adopted a model at the timing when the test NLL, MZE, MAE, or MSE got minimum as the test model under the coresponding task.
As a bascline method, we also tried a regression- based OR method (AD) (Agarwal, 2008). This method solves least absolute deviation regression task with targets {yi), min Meyo > Ig(a;) - yıl with a regression predictor g implemented with 4-layer fully-connected neural network model, and predicts a class label for X = x by label value closest to g(x) among [K]. For AD, we evaluated only the MZE, MAE, and MSE.
We experimented with 6 training sample size settings Pura = 25, 50, 100, 200, 400, 800, to see the dependence of behaviors of each method on the training sample size.
For all combinations of 21 datasets, 4 tasks, 12/13 likelihood models, and 6 training sample size settings, we repeated above-described procedure 100 trials with a randomly-set different training sample and initial parameters, to obtain 100 test errors under the corresponding task.
12

---

Published in Transactions on Machine Learning Research (10/2022)
Table 3: A cell for a method and training sample size Atra a sub-table for an error shows the total (over the 21 datasets and 11/12 other methods) number #win (resp. #lose) of times that that method wins (resp. U-test with p-value 0.05 regarding the error for each pair of that method and
loses) in the Mamn-Whilme other method (as #win, #lose"). A method with larger #win and smaller #lose is better. In each block for Bira in sub-table, the 1st, 2nd, and 3rd best results are highlighted in the red, green, and blue colors.
Ptra 25
Rera
Pera
Pora
Ntça
Riza -00
Ptra 22
"tra
Nitza
Rtza
Aara
Aitra -500
BIN
90
93.
86,
78,
61.111
48.142
BIN
77. 22
75, 34
44
68.
62.
62, 71
POI
110.
111. 76
96,
82,
64,125
41.152
POI
92. 25
43
67
38.
36,101
38, 99
PO-ORD.ACL
47,152
41,150
31,147
33,140
42,161
34.149
FO.ORD-ACL
56, 48
66, 49
02. 3G
54, 60
40. 70
50, 77
PO-VS-SU SL
10.145
65.135
57.134
44,132
49,132
55.121
PO-VS-8L
08, 36
52. 37
T7: 48
06. 4T
56. 55
65, 64
OH-BIN
194, 46
134. 40
100, 80
73. 76
71.110
71.102
OH-BIN
74, 25
81, 28
T2. 43
73. 57
66. 71
52, 914
OH-POI
147. 23
140. 38
116, 63
92. 78
68.113
00.118
OH-POI
183, 5
96, 24
85, 53
69. 83
54. 95
40,110
OH-ORD-ACL
77. 92
74, 93
78. 74
78, 74
72, 96
77. 96
OH-ORD-ACL
36, 83
31. 97
$3.102
21.110
ST. 95
49, 08
OHL VS-S...L
83, 90
92
91, T3
84,
88,
138. 83
OR-VV-SL.
31,
33,109
38, 103
35,102
54,
51,102
ORD-ACE
181.
142.
138,
115, 48
134.
38
141.
ORD-ACL
ST. 24
107.
133.. 13
130. 21
136. 322
63.125
63.130
00,117
92.
147
40.
61. 44
142. 12
145.
ACL
119,
28
162
87,125
85.
101,
195,
09,145
71.
100,
MZB
MSE
Mtra-22 25
Rera-50
Pera 100
Pera- -200 Nirra Ntra,-400
Reza Neza--800
Ptra -25
Mtrr 150
Nitro
Reyo
Peera
BIN
$1. 39
67. 53
70. 76
66,102
40.127
30. 30.186
BIN
$1.
12. 22
66, 22
67. 23
87.
80, 37
POI
84, G1
G4.111
34.153
17.158
18,185
15.184
POJ
94, 14
53, 28
60. 36
58. 36
63.
82. 45
PO-ORD. ACL
58. 59
01. 60
45. 96
37. 17.125
135
13.133
PO-ORD- ACL
43. 48
51. 48
02: 44
07:
76.
02. 32
PO-VS-SL
62. 39
74, 50
68,112
47.124
51.124
PO-V3-SL
52. 63
58. 42
74.
81.
95,
104,
110,
23
155. 24
159, 42
132.
OH-BIN
91. 11
84, 17
25
32,108
OH-POI
129.
45
105. 73
100,
83, 98
OH-POI
112,
184.
OH-VS-SL
OH-VS-SL.
ORD-ACL
85
29
167.
29
185,
135,
ORD-ACL
23
126,
VS-SL.
76
53
42
197
29
151,
145,
VS-SL
54
37
191, 1.5
124,
ACL
32.159
53,140
116, 57
130, 54
128,
ACL
33,110
35,106
84,
92,
89,
SL
0,240
12.221
22,197
64,149
80,106
99, 86
EL
0,226
2,213
3,194
26,151
39,121
65,107
AD
88, 37
94, 44
96, 68
84, 96
75,100
$2.117
AD
01, 45
79, 67
66, 90
28,143
20,196
18,196
6.3
Experimental Results
6.3.1
Outline of Results
Table shows the results of a comparison of all the methods. In summary, the OH-BIN, OH-POT, and ORD- ACL models were especially good when Mera was small, the ORD-ACL and VS-SL models were especially good when Mera was large, for every task. These results imply that methods based on likelihood models with weak (resp. strong) representation ability more effective when the training sample size is small (resp. large), and that unimodal models yielded better performances than the ACL and SL models, which are more representable but have no unimodality guarantee, under our implementation of the models (neural network architecture) and within the sample sizes that we considered (Mura s 800). These results clarified the significance of the proposed more representable unimodal likelihood models.
6.3.2 Detail of Results
For better understanding of the obtained results, we further provide considerations about comparisons within each set of methods that have interesting differences while maintaining commonality. We believe that these considerations will be helpful for future development; see also Table 4.
POI v.s. PO-ORD-ACL. and OH-POI u.s. OH-ORD-ACL The POI model is special instance of the PO-ORD-ACT, model with fixed mode-wise heteroscedlattciity and has a weaker representation ability in the formal sense (see Theorem 15). This relation might make the POI model work better when Rua was small and the PO-ORD-ACL model work better when Atra was large, in every task. The situation for OH-POI v.s. OC- is similar to that for POI e.s. PO-ORD-ACL.
BIN u.8. POI v.s. PO-ORD-ACL v.8. PO-VS-SL (1-FDF) The PO-ORD-ACL and PO-VS-SL models can adjust the mode-wise heteroscedasticity, while the BIN and POI models cannot. In this regard, the PO-ORD-ACL and PO-VS-SL models tend more representable in the informal (or formal) sense than the BIN and POI models. For this reason, it can be considered that the BIN and POI models worked better when Bara was small and the PO-ORD-ACT. and PO-VS-ST. models worked better when pima was large.
13

---

Published in Transactions on Machine Learning Research (10/2022)
Table 4: cell for an error and training sample size Attr in a sub-table for a group of specified methods shows the total (over the 21 datascts) number of times that cach method wins in the Mann- Mann-Whitncy U-test with p-value 0.05 regarding the error for each pair of that method and other method in that group. In each cell in a sub-table, methods that win the most times are highlighted in the corresponding color.
Pol PO-ORD-ACE
PO-ORD-ACL OH-ORD-ACL ORD-ACE
Pera- Psra-25 -25
Psra- -50
Psra-
Psra-
Mara 4000
Plsra -800
Merra 25
Pera-30
Mtra
Hero 200
Rera 1400
Ptra-800
NLL
12.5
12.5
13,4
14,5
11,8
7,7
NLL
7,10,29
6,13,28
1,14,26
3,16,22
4,34,30
1,14,38
M28
6,5
5,7
3,11
2,10
3,11
1,11
MZE
6,2.17
0,2.22
3,7.28
2,10,34
8,13,27
3,16,27
MAE
6,2
4,2
1,2
1,4
2,6
2,5
MAE
7,2.18
9,1,22
8,2:28
9,1.27
7,5,22
6,7,28
MSE
7,1
6,2
3,2
1,2
2,2
1,4
MBE
2,0,15
6,0,15
3,1,23
9,1,24
8,4,18
12.4.19
BIN n... POT n.v. DO-ORD-ACE, n.e. PO-VS-8L.
PO-VS-81, es OHI-VS-8L via. VS-S1.
Pera -25
Para -00
Perra -100
Pera 200
Mara 400
Perra 8800
Ntya 26
Peza R0za-50
Neya 110
Rira 200
Ptyo 400
Rtya-
1opa-BB0 -800
NLL
33 37,.66,7.20
97,96,34,28
38, 29..44,8.20
22.33,18.20
25,30,16,20
28.24.16.2
NLL
11,22,17
12,24,19
11,26,16
3,20,21
5,15.29
4.19.29
MBB
16,20,6,10
16,14,9,16
23,7,15,20
25,6,13,28
24,7.38,66
22,6,19.26 26
MZE
9,5,10
8,6,10
4,0,37
2,14,25
0,18,28
1,21,24
MAR
11,13,2.5
7,8.4,7
6,3,8,11
12,2,5,12
13,4,9.11
15,3,9,10 5,4,9,9
MAE
19.1,9
21,0,31
14,1,17
11:3,22
7,5,28
10.5.25
MSE
15,15,2,4
8,13,2,5
8,8,4.6
10,2,4,5
5,4,3,5
MBE
13,1,10
13,0,12
11,3,13
13,3,15
8,3,17
12.5.16
BIN x... OHI-IIN
GRD-AGT maa ACI,
Ptra 22
+50
-100
-200
4000
Plrra =800
-25
=00
110
220
Ptya -400
- 800
8,13
4,14
4,11
4.7
7,0
9.5
8.5
6.0
2.7
0,6
0.0
1,15
1,37
16.0
11.0
8:2
6,8
1,1
2,0
5,5
6,6
MAB
15.0
14.0
12,1
8,1
8,1
0,1
5,2
7,1
POT OH-POI
VS-SL SL
Para 25
Piza +50
Pera 100
Pira 200
Pira 400
Poraa -800
Riza 25
Rizza
Neoa 100
Rtza 200
Piza 400
Piza- 1800
NLL
2,7
4,10
5,30
6,11
6,11
4,14
NLL
14,2
15.3
15.8
15.65
12,3
12.4
MZE
1,8
0,12
0,18
0,18
1,27
1,18
MZE
21,0
19.0
18.0
17:2
14,2
10,2
MAE
1.6
0,7
4,0
6,8
5,8
8,0
MAE
21.0
20.0
10.0
17.0
15,1
12.2
MSR
0.3
2,6
4,6
6,3
9,3
13.1
MSR
20.0
20.0
19.0
15,0
15,1
13,1
OH-POI m.a. OB-ORD-ACL
ORD-ACL s.e. VS-SL n... ACL m...
"in "Am=~2 -25
Pizu -50
"ira
Piz -200
Pin 100
Rira -800
"Lex 25
Pizza 50
"Lox 100
Piiza "iiea-00 200
Pizz -400
"Izx-500 -500
NLL
13.1
10,4
5,5
6,31
NLL
44,16,40,4
46.16,36,3
41,14,38,8
24.16,36,9
14,36,30,5
14.31,26.11 26
MZE
10.1
9,3
10.4
8,4
6,8
0,10
M2E 43,37,21,0
44,33,20,0
40, 40.32,18.0
30,29,18.4
22,27,20.4
21,22,21,
MAE
8,0
10.0
11,1
5,2
5,5
6,7
MAE
42,29,20,0
42.29,19.0
ST. $9.29.19.0
20. 29.25,19.0
26,26,15,2
26,22,18.3
MSR
13,0
9,0
9,0
5,3
4,5
1,10
MSR
10,28,19:0
36,30,19.0
34,26,14,0
28,21,15,0
27,25,14,2
24,34,10,3
OH-BEN u.. OH-POI us. OH-ORD-ACL n.. OB--VS-8L
Pin -25 -25
Pirx +50
Pan 1000
Paax 2200
Pin 100
Par -800
28,87,8,10
32,35,0,18
28,28,18.18 14,21,177
15,15,24,30 16,14,29.36
36,18,8,18
19,22.3.2
25,29.2.1
17,22.2.2
15,14.7.7
1.12.1.111
9,10,26.18
BIN v.s. OH-BIN, and POI u.s. OH-POI The OH-BIN and OH-POI models are respectively OH- generalizations of the BIN and POI models, and have a stronger representation ability in the formal sense. Thus, OH models were better regarding the NLL that was directly optimized. However, OH models were bad for some tasks (especially, task-A and -S) when Atta was large, which is counter-intuitive result and requires further analysis but may be because the difference of the restricted struetures of the two likelihood models makes a difference in the compatibility between the model and task.
OH-BIN v.8. OH-POI v.8. OH-ORD-ACL 0.8. OH-VS-SL (2-FDF) The situation is similar to that for the 1-FDF models, BIN v.s. POI u.s. PO-ORD-ACL v.s. PO-VS-SL.
PO-ORD-ACL u.s. OH-ORD-ACL v.s. ORD-ACL, and PO-VS-SL v.s. OH-VS-SL u.s. VS-SL The VS-SL, OH-VS-SL, and PO-VS-SL models (and the ORD-ACL, OH-ORD-ACL, and PO-ORD-ACL models) are more representable in the formal sense in that order. Presumably for this reason, the PO- and OH-VS-SL models worked better with small Atra and the VS-SL model worked better with large Mtra-
ORD-ACL v.s. ACL The ACL model can represent any data (see Theorem 2), and the ORD-ACL model can represent unimodal data with the DRs unimodality constraint (see Theorem 4). The uniodaaity of the ORD-ACL model might improve the generalization performance when Atra was small, but the ORD-ACI model was not good when Arr was large perhaps because its DRs' unimodality constraint restricted its representation ability too much for our tried data; see again DRs MU in Tables and 2.
14

---

Published in Transactions on Machine Learning Research (10/2022)
VS-SL v.s. SL The SL model can represent any data (see Theorem 5), and the VS-SL model can represent any unimodal data (see Theorem 7). Unlike the comparison between ACT, and ORD-ACT. models, the VS-ST, model gave better results than the SL model, probably thanks to the validity of the unimodal hypothesis.
ORD-ACL v.8. VS-SL v.s. ACL U.S. SL (Full-FDF) The ORD-ACL model was the best in many cases, and the VS-SL model was better when Mera was large. This is thought due to the unimodality guarantce of these models and the DRs' unimodality constraint of the ORD-ACL model.
For further details of the experimental results, refer to Appendix C.
7 Conclusion
In this paper, we pointed out that previous unimodal BIN and POI models have a weak representation ability from the perspective of the DRs unimodality constraint, scale, skewness, and FDF, and then developed more representable unimodal ORD-ACL and VS-ST. models as well as their PO-constrained version and Oll-extension. In our experiments, 1-FDF or 2-FDF OH models worked better when the training sample size was small, and full-FDF ORD-ACL and VS-ST, models worked better than low-FDF unimodal models when the training sample size was large. Also, full-FDF ORD-ACL and VS-SL models were better than full-FDF models with no unimodality guarantee, due to their unimodality guarantce reasonable for many real-world data that were almost unimodal.
Finally, we have to mention one caution, for future research. We did not encounter non-uninn non-unimodal ordinal data in this study, but not all ordinal data would be almost-umimodal. If you feel that the data you are interested in have natural ordinal relation and consider to apply unimodal likelihood models, it would be better to first test the unimodality of the data as did with the MU in our experiments. As is obvious, unimodal likelihood models would not work well if the data do not tend unimodal.
Acknowledgments
This work was supported by Grant-in-ae Maaut-in-Aic Aid for JSPS Fellows, Number 20.22336..
References
Shivani Agarwal. Generalization bounds for some ordinal regression algorithms. In International Conference on Algorithmic Learning Theory, pp. 7-21, 2008.
Alan Agresti. Analysis of Ordinal Categorical Data, volume 656. John Wiley & Sons, 2010.
John A Anderson. Regression and ordered categorical variables. Journal of the Royal Statislicnl Sociely:
Series B (Methodological), 46(1):1 22, 1984.
David Andrich. A rating formulation for ordered response categories. Psychometrika, 43(4):561 573, 1978.
Anonymous. Modified threshold method for ordinal regression. Submitted to Transactions on Machine Under review.
Learning Research, 2022. URL
Christopher Beckham and Christopher Pal. Unimodal probability distributions for deep ordinal classification. In International Conference on Machine Learning, pp. 411-419, 2017.
Ana M Bianco and Victor J Yohai. Robust estimation in the logistic regression model. In Robust Statistics, Data Analysis, and Computer Intensive Methods, pp. 17-34. 1996.
Paul-Christian Bürkker and Matti Vuorre. Ordinal regression models in psychology: tutorial. Advances in Methods and Practices in Psychological Science, 211):77- 2019.
Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Consistent rank logits for ordinal regression with convolutional neural networks. ar Xie preprint or 1rXin:1901.0700 2019.
15

---

Published in Transactions on Machine Learning Research (10/2022)
Chuansheng Chen, Shin- Ying Lee, and Harold W Stevenson. Response style and cross-cultural comparisons of rating scales among east asian and north american students. Psychological Science, 6(3):170-177) 1995.
Christophe Croux, Gentiane Haesbroeck, and Christel Ruwet. Robust estimation for ordinal regression. Journal of Statistical Planning and Inference, 143(9):1486-1499 1499, 2013.
Joaquim F Pinto da Costa, Hugo Alonso, and Jaime S Cardoso. The unimodal model for the classification of ordinal data. Neural Networks, 21(1):78-91, 2008.
Leo A Goodman. Simple models for the analysis association cross-Classfications having ordered cate- gories. Journal of the American Statistical Association, 74(367):537-552 1979.
Pedro Antonio Gutierrez, Maria Perez-Orliz, Javier Sanchez-Monederg Francisco Fernandez-Navarro, and Cesar Mervas-Martinee Ordinal regression methods: survey and experimental study, IEEE Transactions on Knowledge and Data Engineering, 28(1):22-1166, 2015.
Maria lannario and Domenico Piccolo. Cub models: Statistical methods and empirical evidence. Modern Analysis of Customer Surveys: with Applications using R, pp. 231-258 2011.
Rob Kaas and Jan M Buhrman. Mean, median and mode in binomial distributions. Stotistica Neerlandica, 34(1):13-18, 1980.
Kyoung-Jae Kim and Hyunchul Alm. A corporate credit rating model using multi-class support vector machines with ordinal pairwise partitioning approach. Computers & Operations Research, 39(8):1800- 1811, 2012.
Tie Yan Liu. Learning to Rank for Information Retrieval. Springer Science & Business Media, 2011.
Geoff N Masters. A rasch model for partial credit scoring. Psychometrika, 47(2):149- 174, 1982.
Peter McCullagh. Regression models for ordinal data. Journal of the Royal Statistical Society: Series B (Methodological), 1212))109-127,, 1980.
Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output enn for age estimation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4920-4928, 2016.
Domenico Piccolo. On the moments of mixture of uniform and shifted binomial random variables. Quaderni di Statistica, 5(1):85-104, 2003.
Gary Simon. Alternative analyses for the singly-ordered contingency table. Journal of the American Statis- tical Association, 699349) 1974.
Shipeng Yu, Kai Yu, Volker Tresp, and Hlans-Peter Kriegel. Collaborative ordinal regression. Proceedings of the International Conference on Machine Learning, pp. 1089 1189-1096, 2006.
A Proof of Theorems in Main Part
BIN and OH-BIN We first prove Theorem about properties of the BIN and OH-BIN models:
Proof of Theorem 1. First, prove Theorem (i): Letting The one has that
Pbin(y l;u,s)
dog(Phly+11ph)
Jog(P/brp))/s
Pokp)) = (bgpP
Pbiu (y;4,) s)
R clog(
M
(21)
[log( (K-11| (1-p))
k-p))=--11----- -ogt (K-18
(1-p)
(K - -)pp
1/s
y(1
16

---

Published in Transactions on Machine Learning Research (10/2022)
Next, we prove Theorem 1 (ii) and (iii): Let r(y.p) (K-ylp X(-p) so that Panny+1a,s))
Painly {r(y,p)) 1/. (6) shows
max(1,Kp} s ) s min{Kp+ + 1,K} with p= T+r L. Since (Pb(k; ())k]]])][]]]]] = (P (1; y)kk and the scaling factor does not change the mode, one has that max(1, S m = M((P bin u, u. y(())) S [kp + L.K). This implies Kp s m S Kp + (or equivalently (m 1) K 2 p S m/ K). Since r(y.p) is decreasing in y E 1J and increasing in p E (0,1), one has that
rly,p) (K y)p
(K y)p
=1 for y l....,m--
y(1 p)
y(1 - p) (m-
(22)
r(y.p) (K - y)p
(K -yp
= for y m...., K 1,
y(1 p)
y(1 - p) y=m.p=m/K
i.., Theorem 1 (ii). Also, these results and the fact that r(y,p) is decrcasing in y E [K - 1] prove that
r(L.))* S ((2p))) -(2.p)-- .... ..... r(m L,p) 1, and 2 r(m.) p) 2 r(m L,p) N r(K - 1,p).
(23)
which imply Theorem (iil).
ACL and ORD-ACL and their PO and OH versions We next provide proofs of Theorems 2, 3, and 4 about properties of the ACL and ORD-ACL models and their PO and OH versions:
Proof of Theorem 2. The construction of the ACL model trivially shows {p Rd - ÁK-1 2 ((Pact(); ( Dy) 9 Rd Rx-1), namely, that (Paal);:(a)))) and Rd. Thus, we here show {p : Rd AK-13 S {(Paelly) g ())(k[]]] g : Rdd RK-1), namely, that there exists g Rd - RK-1 such that p(-) (Paal(y:g))))p for any p Rd - Ag-1. Considering the proportional expression
ÅK-1 for any g Rd RK-1
Pi(x): :2(x): px(x) Paci(1;(()) Paci (())())
Paci g(x))
(24)
Σl=l )ı(2) : E7-i si(a)
Ex- si(w)
can find that the equation p(-) (Pacily; g()) y[[]] is satisfied by using g such that
gx(x) log (x)))) Pk-1 (x)
E E for I....,K- K K-1.
(25)
This concludes the proof.
Proof of Theorem 3. First, Theorem 3 (i) is proved by
Pactly 1; m e
IT e" IT e-
My
(26)
Pactly; u) ΣK [1--1 e-
Σ
k- e-
Next, we prove Thcorem 3 (ii) and (iii): Under the assumption that un(== -00) ... S Um 50 um S (=x(==on oo) for some me E [K], one has that
Pacil;; u)
e (n n2 S... Paci(m l;u) Paci(3; u)
Mm-1 S 1,
(2c u)
Paci(m; u)
(27)
1 Paci(m + 1; u)
Paci(m + 22;4)
Paci(K; (K
-m 2 Pac(m + 1; Paci(m; u)
= e umil ...
= = -K-1
Paci(K - 1;
These results show Thcorem 3 (ii) and (iii).
Proof of Theorem 4. Theorem and the fact that 9 p[g] is ordered (É1(x) S ... S (x=1(x)) for any x e Rd) show 'left-hand side of (14)' g 'right-hand side of (14)'. Thus, we here show lefft-hand side of (14)² 2
17

---

Published in Transactions on Machine Learning Research (10/2022)
'right-hand hand side of (14)', namely, that there exists g. R - RK-1 such that p(-) = (Pac((::())) with ý =p[g] for any e 'right-hand side of (14)'. Considering the proportional expression
Pı(z): p2(x): : (aci(1; ((=)) Pact(2: ((x))
aac (K; g(x))
(28)
=e 2 k((m)
e 27-1 (r(m)
- ki(m))
can find that the equation p(-) (Pactly; : ()))))[KK) is satisfied by using g such that
g1(x) = (xx) log p1(x) (() = (x(x)) - (2) = log (x( Pk-1( for k = 2....,K 1.
p2(x)) E R.
(29)
log
pk+ px+1(x)
px(x))
The question here is whether there exists gt Rdd - R satisfying the latter half (29). The DRs unimodality
shows that (k-1(m)
(x( S px(m)
pa(m) pe-1l()) Thus, log pk-1(a)) - log pa(a) E [0),0), and there exists 8k Rd - R satisfying
()-1(m)
the latter half of (29) for a non-negative function P satisfying (13). This concludes the proof.
SL and VS-SL and their PO and OH versions We finally provide proofs of Theorems 5, 6, and 7 about properties of the SL, VS-SL, and PO- and OH-VS-SL models:
Proof of Theorem 5. The equation (Pa():g()))) = (Pa())g())) + c 1))yy[K)) c E R shows the former equation of (17). We thus prove the latter equation of (17) below. The construction of the SL model trivially shows {p Rd Ag-1) 2 ((Pa(o::g))))err) g Rd RK), namely, that (Pa(): ((x)))(( Åg-1 for any 9: Rd - R and Rd Thus, we here show (p: Rd Ax-1) ((Paly)gg)))) 9 Rd RK), namely, that there exists g Rdd R such that p(-) = (Pa(y: (Pa(ygg)))))) y)(K for any p Rd Ax-1- Considering the proportional expression
(1x)): p2(2)) px()) (((1:(())) Pa(2; :.... Pa((Kg(())) e Bl(w) e -
sk(ee) (30)
we can find that the equation p(-) (Pa(y;; @ M){[K]] is satisfied by using g such that
gx = log (x for k = l,...,K-1 K-1.
(31)
This concludes the proof.
The first equation of (17) suggests that the presence or absence of the constraint g1(-) in the SL model is not related to the representation ability of the ST. model. So we adopt the formulation of the SL. model without the constraint 81(+)= 0 in the following.
Proof of Theorem 6. Theorem would be trivial. We omit the proof.
Proof of Theorem 7. Theorem 6 and the fact that ğ = T(pgg) is V-shaped (there exists m(x) [K] such
that (1(x) 2 ... 2 (x) and (m( I ... (x(x) for any x E Rd) show 'left-hand side of (19)* g 'right-hand side of (19)'. Thus, we here show 'lefthhand side of (19)' 2 'right-hand side of (19)', namely, that there exists g Rd - R such that p(-) (Pact)::()) (())eKK] for any p E 'right-hand side of (19).. Considering the proportional expression
(x : P2(x): : ..... (x) = Pa(1; ((x)) Pa(2g((z)) ..... Pa(K: ğ(x)) = e Bi(x)) 82(0)) : (x(x) (32)
we can find that the equation p(-) (Pa g g(-)) is satisfied by using such that
T()++1())) T(É)(x)) = ((g1(z) + - -
log (1(x) p)(
k=
(33)
(((x(() + (())+1 (x)) log pa
k 2....,K K
18

---

Published in Transactions on Machine Learning Research (10/2022)
For simplicity, let (M(p(a))(2) (which is achievable for the VS-SL model). Since p is non-negative and r(u) is continuous and non-increasing in O, one has that {τ(c) - r(c p(u)) E RJ = [T(c) r(c v) v 2 0} = (-o0,0] a log Paripimn-r(a) (< 0) (here с {M(p(m(())), which implies that there exists {M(n(=)) Paippay)a)
also prove the existence of .. M satislying (33). Since is non-negalive and T(u) is continuous and non-decreasing in 0, one has that PMroeen
satisfying (33). By repeating the same argument,
2 0} = [0,00) 9 log PMip(a))s 0) (here c {M)(())(())) which implics that there exists 8Mipla))- (z) satisfying (33). By repeating the same argument, can also .x
(r(c + p(4) - T(C) u E R (T(C + 1) -
prove the existence of (x) satisfying (33). This concludes the proof.
B Theorems on Relationship among Likelihood Models and their Proof
We here provide theorems on relationships among likelihood models, which are shown in Figure 1. Many of the theorems are direct corollaries of Theorems 1-7.
ACL SL Theorems 2 and 5 show the following theorem:
Theorem 8. The ACL and SL models have on equivalent representation ability the formal sense: it holds thal
I Rd - RK--1 ((Pa():: I Rd - R*
(34)
Proof of Theorem 8. This theorem is a direct corollary Theorems and 5.
VS-SLC SL and ACL The following theorem would be trivial from Theorems 2, 5, and 7:
Theorem 9. The VS-SL model has a weak representation ability in the formal sense than the ACL and SL models: for any functions p satisfying (13) and satisfying (18), it holds that
((Pa());(0lg())))) g Rd - R ((Paciyygg ())))( Rd - -RK- RK-1),
(35)
((Pa(); Rd R S ((aa)))))))))) g Rd - R
(36)
Proof of Theorem 9. The trivial result, (Pa(y; ((pg(())))) Ag-1, and Theorems 2, 5, and prove this theorem.
ORD-ACL VS-SL Theorems 4 and 7 show the following theorem:
Theorem 10. The ORD-ACL model has weak representation ability in the formal sense than the VS-SL model: for any functions p salisfying (13) and satisfying (18), it holds that
(l pla(-) g Rd - RK- S ((Pa))) ((pgg)))0a | : - R*).
(37)
Proof of Theorem 10. Since p Âx-1 satisfies
(P) .....
Parp-11 Paripit
PK
is unimodal = p unimodal,
(38)
PMipi PMin
PK-1
Theorems and show this theorem.
OH-BIN & ORD-ACL Thcorems and show the following thcorem: Theorem 11. The OH-BIN model has weak representation ability in the formal sense than the ORD-ACL model: for any function p satisfying (13), it holds that
((P g: Rd - R.s: R...dd - (0) s ((Pac1);; p(g()) g Rd R-1)).
(39)
Proof of Theorem 11. Theorem (iii) shows that an instance of the OH-BIN model has the DRs' unimodality constraint, and Theorem shows that the ORD-ACL model can represent arbitrary CPD with the DRs unimodality constraint. According to these results, this theorem is proved.
19

---

Published in Transactions on Machine Learning Research (10/2022)
OH-ORD-ACL S ORD-ACL Theorems 3 and 4 show the following theorem:
Theorem 12. The OH-ORD-ACL model has weak representation ability in the formal sense than the ORD-ACT. model: for any function p satisfying (13), it holds that
{(Pactly; S (ll)) Rd - R
11/5(())) b bERR-1 R - R. -
- (0,00)
(40)
Proof of Theorem 12. Theorem 3 (iii) shows that an instance of the OH-ORD-ACI model has the DRs unimodality constraint since [p[b]-g() ())((()) ordered, and Theorem shows that the ORD-ACL model can represent arbitrary CPD with the DRs' unimodality constraint. These results prove this theorem.
OH-VS-SL & VS-SL Theorems 6 and 7 show the following theorem:
Theorem 13. The OH-VS WH-VS-SL model has weak representation ability in the formal sense than the VS-SL model: for any functions satisfying (13) and a satisfying (18), it holds that
((Pa())7((()) g(-) 1)/(())((K)) R .g : Rd - R.s R (0.
(41)
S ((Pa():))pll(())))) 9 Rd - R*
Proof of Theorem 13. Theorem 6 shows that an instance of the OH-VS-SL model is unimodal since T(p[b] (()-1)/s( is V-shaped, and Theorem 7 shows that the VS-ST model can represent arbitrary unimodal CPD. According to these results, this theorem is proved.
BIN C OH-BIN, etc. The following theorem would be trivial:
Theorem 14. For any functions p satisfying (13) and satisfying (18), it holds that
The BIN model has a weak representation ability in the formal sense than the model:
((Pbsa(): Rd - +RSE (0,)0)}
(42)
C ((Poin();: RR RRs: (0,co)).
The POI model has a weak representation ability in the formal sense than the OH-POI model:
{(P ((Ppa((r: Rd - RE (0,c0))
(43)
(((Ppoi(y; g0 80. gg Rd Rd-Rs Rd (0,
The PO-ORD-ACL: model has a weak representation ability in the formal sense than the OH-ORD-
ACL model:
((Pac1(): g() 1)) b K-1 8 :R - R)
(44)
S ((Pa((y: [p[b) 8(-) 1)/s()))e(k)) bE RK R R Rd (0))
The PO-VS-SL model has weak representation ability in the formal sense than the HS-SS
model:
K . Rd +R) R) ((Pa)y);([(b)-g) s0) 1))) be ç ((Pa(r g(-) 1)/s(O))) b b .g Rd - .R.: - co))
(45)
Proof of Theorem 14. The BIN, POI, PO-ORD-ACI, and PO-VS-SL models are respectively special in- stance of the OH-BIN, OH-POI, OH-ORD-ACI, and OH- VS-SL models with fixing s(x) a æ-independent constant (s or 1).
20

---

Published in Transactions on Machine Learning Research (10/2022)
POI S PO-ORD-ACL, etc. The following theorem formalizes (20), and show the relationship between POT and PO-ORD-ACT, models and its OH version.
Theorem 15. (20) holds. Also, for any function satisfying (13), it holds that
The POT model has weak representation ability in the formal sense than the PO-ORD-ACI. model:
(() 1 Rd - R,,S (0,
(46)
S p 1 8() 1)) be BEERRLL. . -
The OH-POI model has weak representation ability in the formal sense than the OH-ORD-ACL
model:
((Poo)(); Ig: R. Rd - (0,
(47)
S ((Pa)))) [p[b] 8(-) 1)/s(-))) b bERK-1 .8 Rd RR, :d - (0))
Proof of Theorem 15. For = (bx) satisfying by log(k)/s for k = I.....K w (w*)ke[K-1) = b- V. 1, one has that
K - 1, v = log(u)/s, and
Pacily; w) = 1
e Ei (log())
(10-))lopuut-loo(y-1111
(48)
Σx= 1=1 e
Σ Et- llog())-og(iu)//
Σx=1 el(k-1)loon (okK))))ooo--oo((--1)))
which is equal to u), in (8). This proves (20). Also, since (log(k)/s)e((K-1) is ordered, (20) proves the latter results.
C Supplement of Experimental Results
We here show details of the experimental results. Figures 7, 8, 9, and 10 respectively show errorbar-plots of mean and STD of the test NLIs,, MZFs, MAFs, MSFs for all methods and datasets.
21

---

Published in Transactions on Machine Learning Research (10/2022)
abalones
bane5
bane5
140
10
16
2.50
1.0
2.
SoMIAANN 5
compatie
Sallersingg
231
3.0
1.0
3.
CETGUSS
OETGUUS
abalonst0
3.
38
1.0
14
b0nkLV
bani30
computer10
12
38
43.
3.4
12
4.1
36
3053
computer11
calhouenggD
22
3.6
2.0
1.8
1.4
1.8
415
300
403
SM1
LEV
4.0
130
1.00
2.0
1.0
15
1.55
413
300
320
200
E6A
Minoqua Rorred
2.6
244
2.0
3.
3.8
300
222
200
Figure! 7: Errorbar-plots of mean and STD of the test NLLs of all methods for each dataset.
22

---

Published in Transactions on Machine Learning Research (10/2022)
140
abalones
banks'
0.75
barkd"
150
3.10
2.5
3.6
2.54
2.50
3.00
125
120
putarS
calhooanu
142
310
1.0
335
2.34
2.56
3:00
3.0
0.45
ORTGUSS
CenSAnS
abalone10
2.604
1.6
0.70
0.000
2.34
8.50
2.25
3.60
bark30
computer10
3.57
4.5
3.70
0350
491
100
computar1
Ca chousingg0
3.2
1.00
330
2.50
2.5
1.8
355
300
250
0.00
3005.11
SND
LEV
1.000
0.411
3.825
3.400
5900
0.75
2445
5 3.00
5.500
1.500
400
802
100
382
400
100
FRA
Wit wireouelityrre
3.50
3.40
6.77
0.15
2.40
0.30
2.00
8.00
-20
220
302
Figure 8: Errorbar-plots of mean and STD of the test MZEs of all methods for each dataset.
23

---

Published in Transactions on Machine Learning Research (10/2022)
abalones
banks'
banc5
1.00
2.35
35
325
120
Companie
1.50
Salleruingg
2.505
2.455
0.
3..00
2.40
377
5.400
0.-
2.375
130
OETGUUS
abalonst0
8.60
1.0
bank1V
20011V
computer100
2.6
1.2
0.0
0.4
computer11
2.0
1.0
2.0
5.7
12
1.
415
300
3.0
330
260
SNND
LEV
3.2
3.555
0.45
3.42
1.6
2400
3.4
300
100
382
400
100
E8A
2.500
wosqualityrrdd
0.55
025
$1.00
125
1.000
2.35
3.5
230
-20
220
302
202
400
Figure 9: Crrorbar-plots of mean and STD of the test MAEs of all methods for each dataset.
24

---

Published in Transactions on Machine Learning Research (10/2022)
abaloned
bane5
banc5
13-
12-
1.4
3.0
putarS
Salleruingg
1.00
255
1.55
2.50
140
0.6
135
COTGUSS
OETGUUS
abalonst0
13.
3.0
15
1.4
2.3
b0nk1V
bana30
computer10
2.6
2.5
1.5
2.0
491
35
300
computer11 102
caltessing10
2.4
3.2
3.66
12
35
300
25
100
200
330
200
600
SM10
3.00
250
21
3.50
3.45
3.40
8.40
100
000
E6A
witeouelltyrom
Ch
3.4
2.0
2.50
2.5
3.0
3.3
413
-22
200
Figure 10: Errorbar- cooooba---lossssooo of mean and STD of the test MSEs of all methods for each dataset.
25


Reviews:
[{"review_id": "UP5T61jPgV", "reviewer": "Reviewer_qPrL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The theoretical contribution is limited. The authors used quite some space showing existing definitions and results, but did not do a good enough job explaining the motivation / intuition behind using unimodal likelihood functions for ordinal regression tasks. \n\nThe experiments might possess some value, in the sense that it might point to some useful directions for future research. But the limited theoretical contributions make it disqualified for publication at TMLR.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors studied unimodal models in the context of ordinal regression. Through a set of experimental results they showed that their \"more representable\" unimodal models obtained a better generalization performance, compared to previous unimodal likelihood models. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Motivations for introducing unimodality in OR tasks;\n2. More systematic analysis on what \"representability\" means;\n3. More rigorous results on why representability is important for obtaining good generalization performance for OR tasks;\n4. Design experiments to verify your theoretical findings, if any.\n5. Correcting several typos in the paper.", "broader_impact_concerns": "No concerns on ethical implications of the work. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HFnepjiHqY", "reviewer": "Reviewer_ofxX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The problem of taking advantage of the ordinal nature of the target variable seems to be very interesting.\n- The experimental analysis sounds quite strong to me. Many alternative methods are considered and the results clearly advocate the advantages of the new methods.\n\nWeaknesses\n- I am not an expert of this field, but I have the impression that not all the existing literature is discussed.\n- The (qualitative and empirical) justification of the advantages of the proposed approaches might appear a bit weak\n- The discussion of the results is very hard to follow \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a paper about the classification by probabilistic methods with ordinal target variables (i.e., there is an ordering relation in the set of possible values and we expect the corresponding conditional distribution to reflect this order by being unimodal. The authors derive a (mostly qualitative) characterisation of such notion of unimodality and show by extensive experiments that existing approaches on likelihood models based on binomial and Poisson distributions are less effective than the ones presented here.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I would definitely add to the paper a related work section where all the existing approaches in the field are discussed.\nIn the experimental section I would better use a summary to better outline the main conclusions.\nA toy example to be considered over the different sections of the paper would help (while the figures are often a bit hard to follow).", "broader_impact_concerns": "No specific issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HxiabzrCzB", "reviewer": "Reviewer_CPCb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+: a set of new models (maybe too many?) from which to choose in order to perform ordinal regression\n+: some interesting discussions about these model properties\n+: a comprehensive experimental studies, considering many models and settings\n\n-: no proof of provided theoretical results\n-: difficulty to really know when to apply which model, or one needs so many alternatives and what problem they solve\n-: unclear whether such a large experimental setting is needed (why using a given metric to predict and another to measure the prediction quality?)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces different models fitted to ordinal regression, when one wants the predicted output conditional probability to be unimodal, an assumption that may seem responsable in ordinal classification contexts (although one may easily image input samples for which there is a polarised response, such as a movie that most people either love or hate). \n\nProperties of the new models are discussed, as well as their associated flexibility. Finally, some experiments are performed (using neural networks as ways to estimate the mapping from input space to predictive probabilities) to show the behaviour of the different models, under different training regimes. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I will denote my most pressing requests by a plus **(+)** in front of the label. Note that addressing the most pressing ones would most probably make the paper well over 12 pages. \n\n* Section 2.1., end of second paragraph: I do not understand the point of mentioning that ordinal data cannot be more rigorously defined? Ordinal data are what they are, and their rigorous definition is to be ordered categories. What more would be needed? \n\n* Definition 3 and definition 4: here, authors provide some characterisation of the distributions. However, while some qualitative comments are indeed given when studying each models, such characterisation and their impact in possible experiments is not discussed further. Similarly, in the definition, authors speak of heterogeneity or homogeneity of these features across the input space, but do not provide any (statistical) test to see if this is the case in data sets, nor do they discuss such aspects in the experiments where real data sets are used. \n\n* Definition 5 is a very strange way to state that the DoF of a parametric model correspond to the number of parameters it depends on? Why invoking manifold here and the function $g$, when we are speaking about a probability vector? \n\n* **(+)** Theorem 1 and everywhere in the paper where formal results are stated: while I acknowledge that the result may not be difficult to prove, A formal statement in a paper (and even more in a journal paper) should definitely be proven in order to check the validity of the paper. If I am correct, papers in TMLR are supposed to be self-contained papers, which is not the case if proves of statements are missing. If space does not allow it, then authors can still go to a venue where space is unconstrained. \n\n* **(+)** Theorem 2 really needs a proof, even more so than theorem 1. Also, it would be desirable  to explicitly state the condition mentioned just before Theorem 2, in the style \"If The ACL model satisfy THIS condition, then it is a unimodal distribution\". Right now it is totally similar to theorem 1, and I cannot really identify the mentioned condition, especially as this condition seems to have to do with the $\\'{g}$ function defined later, that is absent from the theorem? \n\n* **(+)** P7, top: it is stated that the ORD-ACL generalise both the POI and BIN model. This would need to be proven, or at least to mention for which choices of parameters in ACL we do retrieve these two models (leaving the exercice to the reader to check it, even if a proof wold be better). I get that theorem 3 somehow states it for ORD-ACL and POI, but the theorem is again not proved. \n\n* **(+)** Why is the presentation for the VS-SL model not following the same format than for the previous models? I gather that the added flexibility comes from the additional V transformation, nevertheless I would have expected unimodality, decrease rate etc. to also be expressed as theorems. \n\n* **(+)** In the experiment, I will not really question the considered model/neural network (as this is beyond my scope of expertise), however I am wondering what is the use of using the probabilities to minimise a given loss, and then to assess the models according to another metric? For example, why not using the MAE only when considering $\\ell_{abs}$, for instance? \n\n* Finally, one could wonder why proposing different models with extra degrees of flexibility (such as the $\\phi$ or $\\tau$ functions, if it is to choose fixed ones in the experiments (for instance, the added flexibility of the VS-SL model does not seem to be explored here?)\n\n* **(+)** Last but not least, the paper english could largely be improved, as several places contains forms of sentences that are not proper english. For example (this is not an exhaustive list):\n- Models are expected promising (abstract)\n- Our took OR experiments (abstract)\n- Symplex --> simplex (Definition 1)\n- The readers should recognise our treatment of them with reference (P3)\n- To every our treated likelihood-models (P4)", "broader_impact_concerns": "No Broader Impact statement needed. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes several novel models for ordinary regression (OR), which interpolates existing methods in terms of representation ability.  The authors defined several measures of the model's property, with which flexibility of models are theoretically analyzed.  Experimental results with several datasets demonstrate the usefulness of proposed models.

Most reviewers acknowledged the contributions of the paper, and recommended acceptance of the revision.  I agree with them that the paper contains significant contributions, and the paper should be published.  However, the revision still seems unnecessarily hard to read.  Therefore, I suggest the following modifications before publication.

- I see Figure 6 very important and it can make readers easily understand the main contributions of the paper.  This figure explains which kind of models were missing before this paper, and readers will acknowledge that this paper fills many nodes of the tree structure of models with theoretical justification and empirical performance improvement.  
To make full use of this figure, I suggest the following modifications:

1. This figure should already be referred to in Section 5, where many models are introduced and the relation is explained.  I strongly felt that this kind of figure is necessary when reading Section 5, and found it later in Section 6.  The authors could show this figure even in the introduction to motivate readers, by saying that there are many missing models in the tree, and the current paper fills them with new models, and will be later shown to outperform existing methods in some situations.
2. Visually distinguish between the proposed new models and the existing models (for example, bold-faced for the new models and the normal font for the existing models.  This should be mentioned in the caption).
3. If I understand correctly, all the 11 arrows indicate the superset/subset (SS) relationship (i.e., the children are special cases of the parents for each arrow).
Then, it would be much easier to understand the contributions of the paper if the authors provide the 11 theorems that state the SS relation of all 11 arrows, and prove all of them in the appendix.  For proving SS relation, you can simply shows what set of the parameter values corresponds to a child model.  Showing the theorem numbers next to each arrow in Fig.6 would be also helpful.  The SS relations might be trivial for the authors but not for readers, so explicit statements will improve the readability.

- In Sections 4 and 5, many claims are made as plain text, and it's not clear if they are mathematically rigorous claims or some intuitive, non-rigorous arguments.  For example, "ORD-ACL fully encompass the BIN and PO models", "VS-SL models can be viewed the most representable ones among unimodal likelihood models..."  If they are rigorous claims, it should be stated as theorems and proved.  To prove "VS-SL models can be viewed the most representable", you could, for example, define the region spanned by the most general unimodal models in the density function space, and show that it matches the image of the parameter domain of VS-SL onto the function space.   When you claim for example that "The PO-ORD-ACL model generalizes the BIN model" is incorrect in general, you should also prove it by, for example, showing that a particular instance of BIN cannot be expressed as PO-ORD-ACL.  Those things are not trivial for all readers.

- Definition 5 (the term "stronger representation ability" can be only used for the superset relation) is confusing because this term tends to be used more casually when comparing two models which are not in the SS relation.  Why don't you simply says model 2 is a special case of model 1?  You could also use the SS terminology.

- I don't understand Definitions 3, 4, and 6, which must be rewritten.  What do you mean by "zero irrelevant"? Why f suddenly appears?  For Definition 6, why don't you define FDF as the manifold dimension of the model?

- "The PO-ORD-ACL and PO-VS-SL models are novel. The significance of these models we expect lies in the theoretical guarantee of the unimodality, not in the improvement of probability prediction and classification performances"
If this would be true, the authors should explain why unimodal guarantee is useful even if it won't improve the performance.
I think the authors think that they could improve the performance, thanks to the unimodality on some data, which the authors didn't find.
If so, modify the sentence.

- I expect that many readers of TMLR are not expert of OR, and they (including myself) would wonder what happens if we simply train a regressor with the ordered labels treated as continuous values, e.g., {'strongly agree' = -2, 'agree' = -1, 'neutral' = 0, 'disagree' = 1, 'strongly disagree' = 2}, and discretize the output in the test phase.  Mentioning this point in the introduction would be helpful for non-experts.  Including this regressor as a weak baseline in one of the experiments would be also nice.

- As motivation, the authors argue that statisticians implicitly expect the unimodality, and showed that this expectation is right by analyzing existing datasets.  However, the unimodality can be easily violated by combining bins: even if the labels {'strongly agree', 'agree', 'weakly agree', 'neutral', 'weakly disagree', 'disagree', 'strongly disagree'} are unimodal, a different binning {'agree' = {'strongly agree', 'agree', 'weakly agree'}, 'neutral', 'disagree' = {'weakly disagree', 'disagree', 'strongly disagree'}}  can be not unimodal.  In this sense, one can't say most data is expected to be unimodal because most benchmark data is unimodal.  It should be cautioned that unbalanced grouping of the labels can violate the unimodality, and in such cases the proposed unimodal methods shouldn't be used.

- In Table 3, "ranking of the methods" are shown.  I wonder why don't you show for example average NLL, MAE,..., and apply for example the Wilcoxon signed rank test and highlight the best methods and those which are not significantly outperformed by the best method?  If showing ranking is standard in OR, the authors don't need to change but explain why showing ranking is better with citations.

- MUs are shown Table 1. Can you show MU of samples drawn from a model (for example SL model) without unimodality, which could help demonstrate that SL is not a good model to fit the datasets in the table.


Minor but important comments:
- The paper is understandable, but the choice of words is not very typical, which degrades readability.  I strongly recommend the authors to ask someone (or some company) for proof-reading.

1. In the last paragraph of Section 1, "We took experimental comparisons" -> "We perform experimental comparisons"
2. "As we show the experimental results and considerations in Section 6.3 and Appendix B, we confirmed that ..."-> "Our empirical results show that"
3. The last paragraph of Section 2 is hard to read.  I understand it very vaguely.
4. Footnote 3 in Page 5: "meaningless"->"irrelevant"
5. Captions in many tables and figures:  "It shows the ***".  The first sentence in captions can be the title of the tables/figures, so remove "It shows".
6. In the beginning of Section 6.3: What is "simultaneous comparison"?
7. In Section 6.3, having "detail of results" and "POI v.s ..." in parallel doesn't make sense.  Maybe "outline of results" and "detail of results" could be subsubsections?
8. Use "citet" when the reference is a part of the sentence.

Please consider all the comments above, and revise the paper accordingly.  If the authors disagree on some comments, please rebut.  Although many, most of my suggested modifications are about how to present the theoretical/empirical results that the authors already have.  Accordingly, my recommendation is accept with minor revision.

Decision: Accept (poster)