Learning Monotonic Probabilities with a Generative Cost Model
Authors: Yongxiang Tang
Yanhua cheng
Xiaocheng Liu
Jiaochenchen
Yanxiang Zeng
Ning Luo
Pengjia Yuan
Xialong Liu
Peng Jiang
Venue: ICML 2025 poster (2025)
Keywords: monotonic model
generative model
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: In many machine learning tasks, it is often necessary for the relationship between input and output variables to be monotonic, including both strictly monotonic and implicitly monotonic relationships. Traditional methods for maintaining monotonicity mainly rely on construction or regularization techniques, whereas this paper shows that the issue of strict monotonic probability can be viewed as a partial order between an observable revenue variable and a latent cost variable. This perspective enables us to reformulate the monotonicity challenge into modeling the latent cost variable. To tackle this, we introduce a generative network for the latent cost variable, termed the Generative Cost Model (**GCM**), which inherently addresses the strict monotonic problem, and propose the Implicit Generative Cost Model (**IGCM**) to address the implicit monotonic problem. We further validate our approach with a numerical simulation of quantile regression and conduct multiple experiments on public datasets, showing that our method significantly outperforms existing monotonic modeling techniques. The code for our experiments can be found at [https://github.com/tyxaaron/GCM](https://github.com/tyxaaron/GCM).

Learning Monotonic Probabilities with a Generative Cost Model
Yongxiang Tang' Yanhua Cheng' Xiaocheng Liu'
Chenchen Jiao
Yanxiang Zeng' Ning Luo
Pengjia Yuan
Xialong Liu" Peng Jiang'
Abstract
those between individual's height and weight be- tween a company's 's stock price and its yearly earnings. For both types of monotonic problems, it's necessary to use model that can predict the monotonic probability based on particular inputs. We call these input monotone variables as revenue variables, where higher revenue correlates with an increased probability of a more positive response.
In many machine learning tasks, it is often nec- essary for the relationship between input and out- put variables to be monotonic, including both strictly monotonic and implicitly monotonic re- lationships. Traditional methods for maintaining monotonicity mainly rely on construction or reg- ularization techniques, whereas this paper shows that the issue of strict monotonic probability can be viewed as partial order between an observ- able revenue variable and latent cost variable. This perspective enables us to reformulate the monotonicity challenge into modeling the latent cost variable. To tackle this, we introduce gener- ative network for the latent cost variable, termed the Generative Cost Model (GCM), which in- herently addresses the strict monotonic problem, and propose the Implicit Generative Cost Model (IGCM) to address the implicit monotonic prob- lem. We further validate our approach with a numerical simulation of quantile regression and conduct multiple experiments public datasets, showing that our method significantly outper- forms existing monotonic modeling techniques. The code for our experiments can be found at https: http:/ggtttttttt httm//yxaaroo//ano GCM.
Common deep learning methods to address the monotonicity problem can be broadly categorized into two types (Runje & Shankaranarayana, 2023): monotonic by construction and by regularization. The construction approach maintains strict monotonicity through customized structures in deep neural networks, such nonconvex or nonconcave mono- tonic activation functions, positive weight matrices, and min-max structures (Sill, 1997). On the other hand, the reg- ularization approach promotes monotonicity by designing specific loss functions (Sill & Abu-Mostafa, 1996); however, these approaches do not guarantee strict monotonicity.
Unlike conventional approaches, this paper introduces a generative framework to tackle the monotonicity problem. In our approach to estimating p(y|x, r). where y E is response variable that maintains monotonicity with respect to the revenue variable but not necessarily with we em- ploy a two-step process. (i) The regression task is converted into a classification task via variable substitution, reducing to a binary state (0 or 1). (ii) We proved that solving the monotonic binary classification problem is equivalent to identifying the latent cost variable c, such that JL r
1. Introduction
and {x,r] d I(c < r) (x. Notably, 4 indicates that two variables share an identical distribution, < denotes the partial order in the vector space, and is the indicator function. Consequently, we shift focus to modeling c and climinate the need to design a strictly monotonic function, granting us increased flexibility modeling c.
Many machine learning problems exhibit a monotonic rela- tionship between inputs and outputs. These problems can be divided into two types: the first strict monotonicity, where an increase the input assured to yield an increase in the output. For instance, the relationship between equipment usability and its age, or the relationship between auction winning rates and bidding prices. The second type is im- plicit monotonicity, often shown through correlations, such
This paper introduces a combined generative process for r, and y. Although direct co-generation of x. and via single latent variable is not feasible since must be conditionally independent of r given x, i.e., c JL r x. To address this, we independently draw two different latent vari- ables z and w from Gaussian priors and independently sam- ple x, c, and r from p(x|z), p(c|z), and p(r[w, x), ensuring .ll x. Subsequently, is generated as y II(c r).
Kuaishou, Beijing, China. Correspondence to: Yongx- iang Tang <tangyongxiang otareyooggianguuuusuuuooooomo kekaiishuu.com. Yanhua Cheng <chengyanhua kuaishou.com>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Monotonic Probabilities with Generative Cost Model
For estimating the posterior p(z]x, r, r),,, employ a vari- ational model (z(x))), which is learned by optimizing the evidence lower bound (ELBO). This approach is termed the generative cost model (GCM).
Definition 2.2 (Strict Order in Vector Space). 01 02 if and only if 01 I 22 and 01 o 222-
Note that 1 < U2 is equivalent to v2 U1 but not equiva- lent to _ 22
Additionally, to capture implicit monotonicity, we alter the GCM generation procedure by integrating a latent kernel variable k. This guarantees that both r and y exhibit mono- tonic behavior concerning k. We refer to this refined method as the implicit generative cost model (IGCM). While this approach does not uphold strict monotonicity, it is better aligned with practical modeling applications.
In the last part, we conduct two types of experiments. First, we design numerical simulation for quantile regression where the predicted r-th quantile increases monotonically with respect to the variable r E (0, 1). We evaluate the performance of traditional techniques versus GCM ap- proach, demonstrating that our method exhibits enhanced accuracy. To further evaluate the monotonic problem with a multivariate revenue variable r, we conduct experiments on six publicly available datasets: the Adult dataset (Becker & Kohavi, 1996), the COMPAS dataset (Larson et al., 2016), the Diabetes dataset (Teboul), the Blog Feedback dataset (Buza, 2014), the Loan Defaulter dataset and the Auto MPG dataset (Quinlan, 1993). In most tasks, our model outper- forms existing approaches. Further ablation studies and time complexity analysis are available in the appendix.
Figure 1. An example of partial order between vectors, where #1 < r2 and rı < 13. The partial order hetween r2 and i's not determined.
Definition 2.3 (First-Order Stochastic Dominance (Hadar & Russell, 1969)). For random variables yı and y2 defined in RR, say that y2 first-order stochastically dominates Y1 (denoted yı Arv. M2) if and only if Pr(y1 > t) < Pr(y, > t) for any vector t E R".
epiFi
The main contributions of our paper are summarized as follows:
We introduce a universal technique that reformulates the problem of monotonic probability into modeling problem for latent cost variables, avoiding restrictions in conventional monotonic neural networks.
Fi(y))
F2(22)
Fs(ya)
We address the modeling of the cost variable using generative approaches, including the generative cost model (GCM) for strict monotonic problems and the implicit generative cost model (IGCM) for implicit monotonic problems.
An illustration of the probability distribution functions for three random variables Y1. Y2. and Y3, given that yı <r.v. Ya and y, <r.v. Ya. is presented. It is apparent that epifi is subset of both epil' and epidf3. However, there is no containment relationship between the sets epi F2 and epi Fs, indicating that the variables y2 and yg cannot directly compared.
Figure 2.
We evaluate our method through simulated quantile re- gression and machine learning tasks on multiple public datasets, demonstrating that our method consistently outperforms traditional monotonic models.
It is significant to highlight that [y <r.v. V2l repre- sents a statement, whereas ly < y2l denotes an event, with its probability density expressed as p(y: x y2) [f x p(y1 = y1 )y(,, = y2)dy,dy2.
2. Background
In the context of one-dimensional random variables. the relationship y, <1 Y2 is equivalent to having Fi(t) Fz(t) for all real numbers / (or epi/,, C epiFF)). where F; denotes the cumulative distribution function (CDF) of the random variable y, and epi/:; indicates the epigraph of this CDF. This concept is illustrated in Figure 2.
Definition 2.1 (Partial Order in Vector Space). For vectors vn and U2 in RR",, 01 < U2 if and only if u1th < (.))) any < < n.
(k) for
This relationship is illustrated Figure 1. Note that V1 3 02 equivalent to 02 ž V1.
For example, y Ny;; (Y:HH, nt E), where the mean l also
2

---

Learning Monotonic Probabilitics with Generative Cost Model
random variable, then we find that y is monotonic with respect to Ja. Similarly, if y (ernoulli()), then y is monotonic with respect to B. In these cases, f and 9 are referred to as monotonic parameters.
regularization strategies, which can be formulated as:
minimize C(y, Go(x, r)) - -RGaa).
Here, the regularization R(Ga) > 0 if Gg is not monotonic at some points, while R(Go) = 0 if monotonicity is pre- served. Crucially, the entire term R(Ga) is differentiable with respect to 6, ensuring that the model adheres more closely to monotonicity. This direction includes monotonic- ity hints (Sill & Abu-Mostafa,, 1996). which employ hint samples and pairwise loss to guide model learning. Certi- fied monotonic neural networks (Liu et al., 2020) certify monotonicity by verifying the lower bound of the partial derivative of monotonic features. (Gupta et al., 2019) pro- posed a pointwise penalization method for negative gradi- ents. Furthermore, counter example guided methods were introduced by (Sivaraman et al., 2020).
Definition 2.4 (Monotonic Conditional Probability). A con- ditional probability (()rr) is defined as monotonic if and only if y {r- = [r= Kev. y {r rr2) for any pair of vectors 11 - 2-
Alternatively, p(y(r) is monotonic if and only if Pr(y > t/r = r1) < Pr(y > t/r = T2) for any vector and vectors r1 < r-- Specifically, in the Bernoulli case, p(y/r) is considered monotonic if and only if Pr(y- = 1|r-=- = rI) Pr(y = 1/r r2) for any pair of vectors 11 < r2. Within this paper, we refer to this relationship between y and as: y being (conditionally) monotonic (increasing) with respect r Throughout this paper, all discussions of monotonicity are assumed to be of the increasing type; for decreasing cases, the original variables can be substituted with their negatives.
Additionally, lattice networks (Garcia Gupta, 2009) ad- dress the monotonic problem via construction or regular- ization techniques; extensive works have been carried out in this area by (Milani Fard et al., 2016; You et al., 2017; Gupta et al., 2019; Yanagisawa et al., 2022).
3. Related Work
Monotonicity is a significant concept in various machine learning domains. (Ben-David, 1995; Lee et al., 2003; van de Kamp et al., 2009; Chen & Guestrin, 2016) integrate monotonicity into bree-based models. The QMIX method (Rashid et al., 2020) incorporating monotonic value func- tions in multi-agent reinforcement learning. Additionally, (Lam et al., 2023) introduce multi-class loss function uti- lizing the monotonicity of gradients of convex functions. Moreover, (Haldar et al., 2020) and (Xu et al., 2024) explore the application of monotonicity in online business.
Monotonic Modeling. In numerous machine learning tasks, we hold the prior knowledge that the output should be mono- tonic with respect to certain input variables. A straightfor- ward idea is to identify a monotonic function and optimize its parameters to approximate the desired monotonic output. can be formulated as follows:
minimize Cly, r)(
subject to (Go r) Ör O.
Generative Model via Variational Inference. Variational inference (VI) (Peterson, 1987; Saul & Jordan, 1995) serves as a potent method for generative models, and has recently made considrrabe progress (Kingma, 2013; Rezende et al., 2014; Burda et al., 2016; Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021). 1). VI simplifies the challenge of Bayesian inference by transforming into a more feasible optimization problem, where latent variables are approxi- mated within chosen family of distributions. This is done by maximizing the evidence lower bound (ELBO) rather than the actual evidence.
The Min-Max architecture (Sill, 1997) represents a founda- tional advancement in monotonic neural networks, utilizing a piecewise linear model to approximate monotonic target functions. This architecture maintains monotonicity through (i) positive weight matrices, (ii) monotonic activation func- tions, and (iii) a Min-Max structure.
Continuing in the direction of monotonic by construction, (Nolte et al., 2022) introduced the Lipschitz monotonic network, enhancing robustness through weight constraint. (Igel, 2024) proposed the smoothed min-max monotonic network by substituting the classic Min-Max structure with a smoothed log-sum-exp function, which prevents the neu- rons from becoming silent. (Runje & Shankaranarayana, 2023) developed the constrained monotonic neural network, refining the approximation of nonconvex functions through adjusted activation functions. Furthermore, (Kim & Lee, 2024) addressed the scalability limitations present in tradi- tional monotonic models.
Recently, there has been an emerging interest in examining generative models under specific conditions. For instance, significant contributions have appeared in text-to-image gen- eration (Ramesh et al., 2021; 2022; Saharia et al., 2022: Rombach et al., 2022). Typically, these generative models initiate the generative process with predefined conditions (such as text, image, or video), generally represented as p(x, = Ex,=eex,((x))(((>>> z), where x is the condition, z is the latent variable, and is the target. Consequently, these models focus mainly on predicting the posterior p(z)x)
An alternative approach of monotonic modeling is through
3

---

Learning Monotonic Probabilitics with Generative Cost Model
and the conditional density p(y]x,2) 2). Since directly infer- ring the posterior p(z|x) is computationally challenging, the posterior is often approximated by the variational model g(z)x), optimized through the ELBO. This paper adopts this framework to build our generative cost model.
T1
4. Method
4.1. Problem Formulation
Pr(c Ri)
In the case of general monotonic problem involving r,), where the output variable y E IR is continuous, the model is structured as follows:
y rx, ~ Fly; r)x,
(1)
Figure 3. Within the density contour plot for the cost variable the shaded area represents the event C < r. This event indicates that the probability of the cost variable falling within this shaded region is expressed as Pr(c - r). Thus, for any vectors Tı X V2, follows that Pr(c ri) Pr(c r2).
where F denotes the probability family for y. The function G' is monotonic with respect to r and produces a monotonic parameter for F. As a result, preserves monotonicity with respect to r. As an illustration, if F is a Gaussian distri- bution V (y: p(x. r). (x))²) with G p(x, r) predicts its mean parameter, then G' must be monotonic function of to ensure that y is monotonic with respect to
The distribution of the Bernoulli variable y is defined by:
y 1 {x, r) ~ Bernoulli (y; G(x,
(3)
This general monotonic probability problem can be trans- formed into the binary case by the following lemma.
According to Definition 2.4, the function G is required to be monotonic with respect to Learning the function G poses significant difficulties, particularly when opting for complex structure, such as deep neural network. A considerable amount of prior research has concentrated on designing or learning an appropriate neural network for G. Our research, however, takes a different approach by introducing a latent cost variable, thereby eliminating the necessity of directly learning the function G.
Lemma 4.1. For any random variable E R defined over Dom(y), the domain of y, and that y TL t| {x, r}. Define y" = -I(y > t) {0,11 1} and r" = [-t, Then y {x, r) is monotonic conditional probability with respect to r, if and only if {xx r"} } is monotonic conditional probability with respect to r"
The proof of this lemma is available in Appendix B.1. This lemma demonstrates the equivalence between the prob- lems of (y",,, r)) and xy, As result, the mono- tonic modeling task for the triplet (y, x, r), where y E I., can be simplified to the binary problem xyr, Here, y* [x, r" ~ Gernoulliy"";G(x, - t, rl)). Given that Pr(y I s s|x. r)- = 1 Pr(y > s|x, r) = 1 - G(x, [-s,)), the probability density function of y is given by:
4.2. The Cost Variable
In this section, we examine the binary problem. The con- ventional approach, as defined in Equation (3), involves
identifying a strictly monotonic function G(x. r) with re- spect to In this paper, instead of searching for a suitable function G, we introduce a random variable to model y. This random variable e fulfills the following conditions:
ply s(x,r)
as
Ille c x, y (xxx)) _ I(c Ix,,,
This equivalence illustrates the transformation of a gen- eral monotonic probability problem into binary mono- tonic one. Additional information regarding this transfor- mation is provided in Appendix D.3. As result, our inves- tigation now centers on the binary classification problem Hn represents the nonmonotonic
where d denotes that the two variables share the same dis- tribution. The existence of the cost variable e is guaranteed by the following lemma.
for (x, r, y), where variables, r E RT signifies the monotonic variable, and y E (0.1) }} is the binary outcome that retains monotonicity with respect to r. We refer to as the revenue variable of y. The reasoning is that when y is considered a decision vari- able, a praiit-maximizing decision will favor higher values of r, thereby ensuring the monotonicity between y and r.
Lemma 4.2. Let y be binary variable inffuenced by and with the constraint that Pr(y = 1 1|x, r) is continuous for E D, where D is bounded. Then Pr(y = 1|x, r) is monotonically increasing with respect to r if and only if there exists a random variable e in D such that r iL c x and y {x, d T(e <r) 1 (x,
4

---

Learning Monotonic Probabilities with Generative Cost Model
The proof of this lemma is available in Appendix B.2, and also address the case of unbounded in Appendix C. Here, we define as the cost variable, with its intuitive interpretation akin to that of r: it represents the resistance of a profit-mxxmizing decision. As shown in Figure 3, the likelihood of y = 1 corresponds to the probability that the revenue r domains the cost c. This lemma implies that once the cost variable с is obtained, the initial probability Pr(y = 1[x, r) varies monotonically with r. Consequently, the task of identifying the monotonic function G essentially transforms into determining the cost variable However, since с is not directly observable, we must infer it from the observable variables x, r, and y, posing a significant unresolved challenge.
W
r
X
I[c - r) =
y
(o(z/xx
------
Z
Figure 4. The probabilistic graphical model of the generative cost model ensures a monotonic conditional probability p(y|x, r) with respect to r. Within this diagram, observable variables are depicted as gray nodes, whereas latent variables are shown white nodes. Solid arrows illustrate the generative model pe. and the dashed arrow indicates the variational model qo. The thick arrows depict the estimator for p(y x, r).
4.3. Generative Cost Model
In addressing the challenge of modeling the cost variable c, the distribution of с can be complicated, making it challeng- ing to select an appropriate distribution family. To bypass the necessity of picking specific distribution family, we employ generative approach capable of automatically ap- proximating intricate distributions. A direct idea involves first sampling latent variable z, and subsequently generat- ing x, r, and с independently, conditional on z. However, this approach does not ensure с Il x, as z still serves as a shared factor influencing both c and r. To achieve IL r x, we design a generative model using an addi- tional latent variable w with the following process:
side of this inequality corresponds to the evidence lower bound (ELBO) of our model, with equality occurring only if DK I (z(x))) pl (x)/x,,,, r, y)] O. The detailed derivation can be found in Appendix D.1. Comprehensive details on computing (rø Yy r/z, r) can be found in Appendix D.2.
In the course of optimizing the ELBO, our model employs the reparameterization trick (Kingma, 2013) as 2(n) = uxx) + o (x) O c(7), where e(m) is sampled from the stan- dard normal distribution (00, E). Following the IWAE (Burda et al., 2016) framework, the final loss function of GCM is given by:
(2), x~ Cr>poxll),ccoom c po (clzz)) r- po(()))))
pe(w),
(4)
y = T(cc r).
Here, po(z) and (w) are standard multivariate Gaussian distributions, while po(x/z). Po (clz.) and po (r),,)) are conditional Gaussian distributions. This entire framework is referred to the generative cost model (GCM) and is illustrated in Figure 4. It is evident from Figure 4 that x blocks the path between r and c, and that is a collision node. Therefore, r I c x holds, ensuring the strict monotonicity of with respect to r, as specified by Lemma 4.2.
LOCM (r,,,,
1 N
Pro (c yy r/2 = z(1))
pe (z = 2 (1)
= - log
go zz 2 21))
x x
(6)
Here, we drop the term (x r|z) in Equation (5), since we do not need for predicting y. The sampling number N influences the loss function, with detailed ablation study N provided in Appendix F, In scenarios where the output variable is continuous, we additionally formulate the cor- responding loss function, which is elaborated in Appendix D.3.
For learning this generative model, we employ the vari- ational inference method, utilizing the variational model gs x2)x) to approximate the posterior po(2)x,r,y). As result, the variational bound is formulated as:
During inference, the aim is to estimate y while given x and r. We leverage the approximation pe r. y) (z(x) to acquire:
rx) Pozzg log Pr Yy Y, r(z)po(z)
+ log loggoo(xxrl2) (5)
g0
Pro(y = 1|x= x; r) Pro (y = 1 1/z,r= =
BELBOGOM (0,))
N
Here, Yy denotes x if = 1, and A if = (note that A is distinct from 2 in the vector space). The right hand
Pro c rczrlz=22 2(
5

---

Learning Monotonic Probabilities with Generative Cost Model
where 2(n) is a random sample from (o(z)x a². This estimation is strictly monotonic with respect to r, since Pro (c < rlz z(") is a strict monotonic function of r. Consequently, we have established strict monotonicity be- tween r and without the necessity of developing a mono- tonic function Gi r), thereby reducing the complexity involved designing a strict monotonic neural network.
monotonic
k
r
(k(rx)
qe (x
X
I(c - k)
y
4.4. Implicit Generative Cost Model
While we have effectively established the generative cost model to address the strict monotonic problem, a challenge remains that not all monotonic relationships indicate cau- sation; some merely reflect correlation. For instance, both child's height and weight increase monotonically with age, which serves as the underlying monotonic factor. In considering the monotonic relationship between the output variable y and the revenue variable r, our goal is to identify the latent variable k, with which both y and r exhibit mono- tonicity. This model does not guarantee strict monotonicity of p(y) x, r); however, it introduces monotonic correla- tion between y and r, owing to the strict monotonicity of p(y|x, x, k) and p(r(k). We designate k as the kemnel revenue variable.
Z
C
Figure 5. The probability graphical model for the implicit genera- tive cost model (IGCM), which doses not ensure strict monotonic between and y. but introduces monotonicity from k and from k to The thick arrows represent the path for the prediction of ply x, r).
sponding loss function becomes:
LIGCM o, 2.,
N
Bk Pro (o Y-u k/z = =2(n))
= - -og
The generative framework is specified as follows:
N n=1
po (z. = =z==())) z(n))
~~ po(o k ~ (k), x ~ (x c~ ~ (clz)) Ar ~ (Ar()),
4a (z = z(1)| x = x)
1 M
loog M m=1
Σ pr r (m))
r = Ar + Wk. W > 0,
y= 1(c k).
+ DEL 1 (k'x = x, r-r==)))pp(l
The assurance of a monotonic relationship between and r is given by the positive projection matrix W, while the monotonic relationship between y and k is guaranteed by y= I[c < k) and the independence condition .IL k x. This model is referred to as the implicit generative cost model (IGCM), as shown in Figure 5.
Since the conditional distributions pa(c)z), po (r(k), and no (k|x, r) are
Gaussian, the expressions Ek Pra (c Yy=y k/z =(")), (r r|k (m)) and DKL lg (k|x, r)] rr)(lp((k) (k)] can be evaluated in closed form. The vectors 2(1) and ((m) represent samples from their respective variational distributions (z(x)) and go (k|x). Consequently, the reparameterization trick can be employed to optimize this objective effectively. The estimator for within the IGCM framework ultimately represented as:
To learn the IGCM, we propose an additional variational model qu (k) r) to estimate the posterior distribution Pe pe(,,, y). Close to Equation (5), the ELBO of IGCM
can be derived as:
Praly = 1|x = =|x==xxxx xr
ELBO (0, Φ, v)
B. Pro = 1|2,
F. Bazgg kngg log Pr Y (x)))o (r
k/z (c<k(z=<(n)) =(t))
po (z) 2o (z
pe (k) (k) q (k|x, r)
E Exeggo Pro
- Egggg log
E gx log
in which (x)) are sampled from qs (2)x), and the variable k is estimated using the variational distribution qu (k|x, r). In contrast to the GCM, the IGCM estimator does not maintain strict monotonic property.
Similarly to Equation (6), we drop the term po (x/z) in ELBOJGCM (0, &, v). while keeping the term pe (r kk to capture the latent monotonicity between r and k. The corre-
6

---

Learning Monotonic Probabilities with Generative Cost Model
Table MAE (with 95% confidence interval) of the quantile regression experiment.
MAE
Method
r=
0=3
0
r=0.7
r=0.9
PosNN
0.2103 #0.0464
0,1174 =0.0327
0.0900 #0.0140
0,1173 #0.0291
0.1898
+:00332
MM
0.2055 #0.0623
0.1202 #0.0339
0.0819 #0.0236
0.1236 #0.0376
0.1781
#0.0551
SMM
0.2499 #0.0799
0.1310 #0.0410
0.0846 #0.0274
0.1420 #0.0378
0.2144 #0.0620
CMNN
0.1575 #0.0349
0.1096 =0.0253
0.0799 =0.0287
0.1165 #0.0348
0.1685 #0.0428
SMNN
0.2184 #0.0486
0.1132 10.0319
0.0870 #0.0143
0.1140 #0.0313
0.1777 #0.0622
Hint
0.1536 +0.0241
0.1166 +0.0153
0.1091 +0.0181
0.1280 +0.0211
0.1379 +0.0164
PWL
0.2083 +0.0217
0.1587 =0.0138
0.1436 #0.0131
0.1608 #0.0162
0.1720 +0.0187
GCM
0.0961 10.0261
0.0817 #0.0318
0.0742 #0.0233
0.0792 #0.0159
0.0899 #0.0155
5. Experiment
This paper evaluates various traditional methods alongside our generative cost model (GCM). The strictly monotonic characteristics of this context exclude the IGCM method from consideration. All methods share an underlying three- layer pereeptron network architecture with tanh activation functions. For training, network parameters are optimized using the stochastic gradient descent algorithm. The meth- ods being compared include: (i) the positive weighted neural network (PosNN), which employs solely positive weight matrices (refer to Appendix E); (ii) the Min-Max net- work (MM) (Sill, 1997); (iii) the smooth Min-Max network (SMM) (Igel, 2024); (iv) the constrained monotonic network (CMNN) (Runje Shankaranarayana, 2023); (v) the scal- able monotonic network (SMNN) (Kim & Lee, 2024); (vi) the monotonicity hint model (Hint) (Sill & Abu-Mostafa,, 1996); (vii) the pointwise loss method (PWL) (Gupta et al., 2019). It should be noted that both the Hint and PWL techniques are weakly monotonic, aiming to enhance mono- tonicity without ensuring it. Training data are generated using simulation configured as follows:
5.1. Experiment by Simulation
Quantile regression is a common problem in statistics, aim- ing to estimate the r-th quantile of y given x, based observations of both x and y. The r-th quantile, represented as Qy)x()) is defined by (r)x(r) = y (r), where Fy is the conditional cumulative distribution function of y given
x. Due to the monotonic nature of F. its inverse, Qy)((), also strictly monotonic with respect to r. The standard objective of linear quantile regression (Koenker, 2005) is formulated as:
3 arg min Σ P: -y00-/0 o.").
8,
where (A) = r + (1 (1-1) r))-)))
Here, A, A O, otherwise D+ O. (o(0))()) is the i-th example pair and /¹ = (0) denotes the lin- ear prediction of the quantile 2ykx=(() (r). with 3, as its parameter. For nonlinear relationships of given x, neural networks can be leveraged to automatically capture these dynamics. Additionally, by integrating into the network, it is possible to predict the r-th quantile of y given x using 9. = D2, r). for any E (0, 1). This can also be represented in a generative format:
X~ UI ((15515,11))
<~ (~N(0,1) 1),
y 0.3 + 0.8)) 0.4 1.3))
+0.3 (5x) + 0.2 0.6)c.
(8)
y, po(,,),,r),
For each example (x,y) (2(0) we additionally sam- ple (+) ~ U(0, 1]) and optimize our models with Equation (7). It is important to note that the sampling of 7()) operates independently of c. Our training procedure employs a batch size of 20 over 5, 000 iterations, yielding a total 100, 000 training samples. The testing phase involves 1, 000 sam- ples per quantile, with r E {0.1, 0.3, 0.5,0.7, 0.9). Table provides the mean absolute error (MAE) results for all methods involved in the quantile regression analysis. We conduct the experiment 10 times, varying the random seeds each time, and present the results with 95% confidence inter- vals. According to the results, GCM surpasses all competing methods.
fr Ey,
However, this problem diverges from the original monotonic modeling, since the variable r is unobservable, preventing the direct implementation of the GCM method. To address this, we reformulate the quantile regression problem as fol- lows:
sample ~~ u( 1J), minimize (5y-- y,),
which becomes an optimizable problem.
7

---

Learning Monotonic Probabilities with Generative Cost Model
PosNN
MM
SMM
CMNN
SMNN
Hint
PWL
GCM
Step
=1000
Step
=2000
Step =3000
Step =4000
Step =5000
Figure 6. Validation of quantile regression accuracy. The background scatter points represent the true examples sampled from the distribution specified in Equation (8), while the red curves depict the estimated quantile curves obtained by each model at various training stages ranging from 0 to 5, 000.
We define the r-th quantile curve as r, {xx, }r)}; thus, for any r2 > T1, the quantile's monotonic property en- sures that rr,, is positioned above rr.. Figure presents the quantile curve prediction outcomes across the train- ing phases, where each column denotes a distinct model approach and each row corresponds to training stage. Every panel depicts five predicted quantile curves r, for {0.1, 0.3, 0.5, 0.7, 0.9} illustrated red curves, while the training data (Ir, 14) are represented as underlying scat- ter points, demonstrating the prediction efficacy of each model at different training stages. Ideally, the predicted quantile curves should closely approximate the true quantile curve, defined as:
5.2. Experiments Public Datasets
To further evaluate the GCM and IGCM methods with mul- tidimensional revenue variables, we conduct experiments using six public datasets, which include the Adult dataset (Becker Kohavi, 1996), the COMPAS dataset (Larson et al., 2016), the Diabetes dataset (Teboul), the Blog Feed- back dataset (Buza, 2014), the Loan Defaulter dataset and the Auto MPG dateset (Quinlan, 1993). The main statisties of each dataset presented in Table 2.
Table 2. Details of the datasets.
dataset
examples
x dim.
dim.
target
yr =0.3 0n. + 0.8)) + 0.4 sin((r 1.3))
Adult
48,842
33
classification
+ 0.3 sin (5x) + 0.. 0.6)) (r),
COMPAS
6,172
9
4
classification
Diabetes
253,680
105
4
classification
where denotes the CDF of the standard normal distribu- tion. Conceptually, the predicted quantile curves ought to be distinct from each other, with the upper curve To.9 and the lower curve Fo.1 positioned towards the edges of the scatter plot, with limited outliers.
Blog Feedback
54,270
268
regression
Loan Defaulter
488,909
23
classification
Auto MPG
392
4
regression
The models we compare are identical to those presented in Section 5.1, while the evaluation metrics are switched to root mean squared error (RMSE), classification accuracy (ACC) and area under the curve (AUC). For regression tasks, we apply the RMSE metric, while for classification tasks with positive sample ratio between [0.3, 0.7]. utilize the ACC metric; otherwise, the AUC metric is employed. The datasets are divided into training and testing sets in 4 1 ratio. We adhere to the data preprocessing methods specified by (Runje & Shankaranarayana, 2023) for the COMPAS, Blog Feedback, Loan Defaulter, and Auto MPG datasets.
Inspection of Figure 6 indicates that: (i) the quantile curves for PosNN, MM, SMM, and SMNN, which are primarily built using monotonic construction methods, tend to clus- too closely in certain regions; (ii) CMNN, Hint, and PWL, which mainly employ monotonic regularization tech- niques, have surplus of outliers beyond To.1 and To.9: and (iii) GCM demonstrates noticeable separations between the quantile curves and the fewest instances of outliers. Ulti- mately. GCM yields the most precise quantile curve predic- tions contrast to traditional approaches.
8

---

Learning Monotonic Probabilities with Generative Cost Model
Table 3. Experimental results on public datasets.
Method
Adult AUC1
COMPAS
Diabetes AUC1
Blog Feedback
Loan Defaulter
Auto MPG
ACC1
RMSEJ
ACC1
RMSE4
PosNN
0.7597 10.0016
0.6921 13.0013
0.8254 =0.0002
0,1580
10.0017
0.6519
10.0003
3.1473 10.1417
MM
0.7791 =0.0010
0.6976:0.0011
0.8263 =0.0003
0.1591 =0.0005
0.6526 #0.0003
3.4517 #0.1311
SMM
0.7819 =
469444100001
0.8263 =0.0001
0.1579 =0.0006
0.6527 #0.0002
3.3534 =0.0923
CMNN
0.7725
=0.0009
#0.0012
0.8261
=0.0003
0.1685
=0.0030
0.6525
#0.0002
3.2813 =0.109
SMNN
0.7729
=0.0033
0.6935
10.0015
0.8245
=0.0002
0.1776 =0.0097
0.6527
=0.0001
3.3640
=0.1649
Hint"
0.7509
=0.0009
0.6723
+0.0013
0.8184 =0.0002
0.1590
10.0011
0.6477
+0.0003
3.3872
+0.1689
PWL'
0.7661
=
0.6916 #0.0014
0.8251 =0.0003
0.1606
=0.0031
0.6522 #0.0003
3.4017 =0.1302
GCM
0.7844 =0.0025
0.6945
10.0011
0.8263
=0.0004
0.1584
=0.0004
0.6528
#0.0004
3.2240
=0.0801
IGCM"
0.7891 +000018
0.6934 +.00012
0.8285 0.0002
0.1569 +0.0005
0.6528 +0.0004
3.1000 -0.1268
Non-strict monotonic model.
The hyperparameter configurations for GCM and IGCM can be found in Appendix E. Each experiment is repeated 10 times with different random seeds, and the results are reported with 95% confidence intervals, as shown in Table To better understand the experimental results, we analyze the test results in three aspects: (i) comparison between GCM and IGCM, (ii) comparison between strict monotonic methods, and (iii) comparison between non-strict monotonic methods.
improved outcomes compared merely modifying the net- work structure.
Comparison between non-strict monotonic methods. Hint, PWL,, and IGCM are non-strict monotonic approaches, with Hint and PWL serving as regularization techniques. In contrast, our newly developed IGCM tackles the non-strict issue via an innovative implicit monotonic modeling. This involves addressing two strict monotonic challenges: p(r(k) and p(ylk), thereby creating non-strict monotonic rela- tionship between and r. This new technique offers an in- novative probabilistic perspective for non-strict monotonic modeling. Experimental findings indicate enhancements when compared to conventional non-strict monotonic meth- ods.
Comparison between GCM and IGCM. Our analysis re- veals that within public datasets, the IGCM outperforms the GCM. This improved performance is attributed to the fact that real-world scenarios do not always adhere to strict monotonic relationships; for instance, while BMI is corre- lated with diabetes, it is not necessarily causative. Typically, the selection of monotonic features is informed by expe- rience or statistical analysis. Under these cireumstances, implicit monotonic assumptions tend to be more applica- ble. Our experimental data support this assertion, as IGCM demonstrates superior performance to GCM in five of the six datasets.
In conclusion, our GCM and IGCM methods deliver supe- rior performance in most tasks, highlighting the efficacy of our generative approach. Further analysis of time com- plexity is provided in Appendix G and ablation studies are presented in Appendix F.
6. Conclusion
Comparison between strict monotonic methods. The strict monotonic methods evaluated include PosNN, MM, SMM, CMNN, SMNN, and the proposed GCM. Our evalu- ation indicates that the basic PosNN performs inadequately on most datasets, implying that simply implementing a pos- itively weighted neural network is insufficient for handling monotonic problems. Notably, the traditional MM method, along with its successor SMM, demonstrates considerable effectiveness, indicating that the fundamental Min-Max ar- chitecture successfully optimizes the monotonic network. Recent advancements with CMNN and SMNN introduce novel monotonic network structures that surpass MM in sev- eral tests. Our proposed GCM method consistently achieves strong performance across all datasets, suggesting that re- framing the monotonic problem as a generative one yields
This paper introduces novel generative approach to mono- tonic modeling. We reformulate the monotonicity problem by incorporating a latent cost variable c. Additionally, we propose two generative methods: GCM, which targets the strict monotonic problem, and IGCM, which focuses on the implicit monotonic problem. Our experimental find- ings indicate that GCM and IGCM successfully tackle the monotonicity challenge and substantially surpass traditional methods across multiple tasks.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal
9

---

Learning Monotonic Probabilities with Generative Cost Model
consequences of work, none which feel must be specifically highlighted here.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In The Third International Conference on Learning Representations, 2015.
References
Koenker, Quantile regression. Cambridge Univ Pr, 2005,
Becker, B. and Kohavi, R. Adult.
UCI Ma- DOI:
Lam, K. H., Walder, C.,, Penev, S., and Nock, R. Legen- dretron: uprising proper multiclass loss learning. In In- ternational Conference on Machine Learning, pp. 18454- 18470. PMLR, 2023.
chine Learning Repository, 1996.
https//dbo.or/1024433xXxXX
Ben-David, A. Monotonicity maintenance in information- theoretic machine learning algorithms. Machine Learn- ing, 19:29-43, 1995.
Larson, J., Mattu, S.. Kirchner, L., and Angwin, J. compas-analysae 2016. URL https://githib. com/ conn/ropublica. aarpppbbicccccompsaaaaoiie coompas-analysinn
Burda, Y., Grosse, R. B., and Salakhutdinov, R. Impor- tance weighted autoencoders. In The Forth International Conference Learning Representations, 2016.
Lee, J. W., Yeung, D. S., and Wang, X. Monotonic de- cision tree for ordinal classification. Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, volume pp. 2623-262222222 2628.. IEEE, 2003.
Buza, K. BlogFeedback. UCI Machine Learning Repository, 2014. DOI: http://cbb.ogg1122433//55533
Liu, X., Han, X., Zhang, N., and Liu, Q. Certified mono- tonic neural networks. Advances in Neural Information Processing Systems, 33:15427-15438 2020.
Chen, T. and Guestrin, C. Xgboost: A scalable tree boost- ing system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785-794, 2016.
Milani Fard, M., Canini, K., Cotter, A., Pfeifer, J., and Gupta, M. Fast and flexible monotonic functions with ensembles of lattices. Advances Neural Information Processing Systems, 29, 2016.
Garcia, E. and Gupta, M. Lattice regression. Advances in Neural Information Processing Systems, 22, 2009.
Gupta, A., Shukla, N., Marla, L., Kolbeinsson, A., and Yellepeddi, K. How to incorporate monotonicity in deep networks while preserving flexibility? arXiv preprint 1exiv:/909.00 10662, 2019.
Nolte, N., Kitouni, O., and Williams, M. Expressive mono- tonic neural networks. In The Eleventh International Conference on Learning Representations, 2022.
Peterson, C. A mean field theory learning algorithm for neural network. Complex systems, 1:995-1019, 1987.
Hadar, J. and Russell, W. R. Rules for ordering uncertain prospects. The American economic review, (9(1)235,34, 1969.
Quinlan, J. R. Combining instance-based and model-based learning. In Proceedings of the tenth International Con- ference on Machine Learning, pp. 236 243, 1993.
Haldar, M., Ramanathan, Sax, T., Abdool, M., Zhang, L., Mansawala, A., Yang, S., Turnbull, B., and Liao, Im- proving deep learning for airbnb search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2822-2830, 2020.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text- to-image generation. In International Conference on Machine Learning, pp. 5821-8831 PMLR, 2021.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text- coxt-conditional image generation with clip latents. arXiv preprint arXiv:2204.066 25. 1(2)33 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in Neural Information Process- ing Systems, 33:6840-6851, 2020.
Rashid, T., Samvelyan, M.. De Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
Igel, C. Smooth min-max monotonic networks. In Interna- lional Conference on Machine Learning. PMLR, 2024.
Kim, H. and Lee, J.S. Scalable monotonic neural networks. In The Twelfth International Conference on Learning Representations, 2024.
Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep gen- erative models. In International Conference on Machine Learning, pp. PMLR, 2014.
Kingma, D. Auto-encoding variational bayes. arXiv preprint X3 / 2013.
10

---

Learning Monotonic Probabilities with Generative Cost Model
Rombach, R., Blattmann, A., Lorenz, D., Esser, and Ommer, B. High-resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
You, S., Ding, D., Canini, K., Pfeifer, J., and Gupta, M. Deep lattice networks and partial monotonic functions. Advances in Neural Information Processing Systems, 30, 2017.
Runje, D. and Shankaranarayana, S. M. Constrained mono- tonic neural networks. In International Conference on Machine Learning, pp. 17338-29353,, PMLR, 2023.
Saharia, C., Chan, W. Saxena, S., Li, I.., Whang. J., Denton, E. L., Ghasemipour K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in Neural Information Processing Systems, 35: 36479-36494, 2022.
Saul, L. and Jordan, M. Exploiting tractable substructures in intractable networks. Advances in Neural Information Processing Systems, 1995.
Sill, Monotonic networks. Advances in Neural Informa- tion Processing Systems, 10, 1997.
Sill, and Abu-Mostafa, Y. Monotonicity hints. Advances in Neural Information Processing Systems, 9, 1996.
Sivaraman, A., Farnadi, G., Millstein, T., and Van den Broeck, G. Counterexample-guice learning of mono- tonic neural networks. Advances Neural Information Processing Systems, 33:1 1933-11448, 2020.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynanics. In International Conference Machine Learning, pp. 2256-2265. PMLR, 2015.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In The Ninth International Conference on Learning Representutions, 2021.
Teboul, A. Diabetes health indicators dataset. URL https:
//www.a.. www.aklgg............kkkkk.k..e com http:///ee.o/eaaaaattataatttaaba aiaaetess-ealtthee hwwp::/iiniatttttttttttattaaeaaa
de Kamp, R., Feelders, A., and Barile, N. Isotonic clas- sification trees. In Advances in Intelligent Data Analysis VIII, pp. 416. Springer, 2009.
Xu, X., Wu, H., Yu, W., Hu, L.., Jiang, P., and Gai, K. En- hancing interpretability and effectiveness in recommen- dation with numerical features via learning to contrast the counterfactual samples. In Companion Proceedings of the ACM on Web Conference 2024, pp. 453-460, 2024.
Yanagisawa, H., Miyaguchi, K., and Katsuki, T. Hierarchi- cal lattice layer for partially monotone neural networks. Advances in Neural Information Processing Systems, 35: 11092-11103, 2022.

---

Learning Monotonic Probabilities with Generative Cost Model
A. Notations
Table 4. Notations.
notation type
examples
scalar
2, V,
vector
a,
matrix
W
random variable
X, I. y, c
random vector
x, V,
model parameter
0, dz sp
function
F, G
distribution
F.N
density function
P.
B. Proofs
B.1. Proof Lemma
Proof. Since y {x, r). for any ró rol. it always holds that
Pr(y" = 1|x, r = rū)
= Pr(y t/t to ro)
(9)
-Pr(y tolt to. ro)
= =Pr(y to|x,r = ro).
y (x,r) is monotonic conditional probability with respect For any rī <r2, where r. [-t;, rz), it follows that ri 3 and 2 tz. By employing Definition 2.4, the following inequality holds:
Pr(y" 1|x, =r))
-Pr(y > f f1/xrr r
(by Equation (9))
<rr( > 4 11,x,r =T)
(by ri I r2)
Prr( > 12/x,r = T2)
(by t 2 12)
Pr(y" =1|xrr' r
(by Equation (9)).
Equality occurs if f2 and Ti r2. which is inconsistent with rí < r Therefore, according Definition 2.4. y" rx, is strictly monotonic with respect r".
If y" (x,r") is monotonic with respect to r* = [-t, r). then for any vectors x r2 and E Dommyy, the inequality Pr(y* 1|x, = [-s, ril) < Pr(y" = 1|x, = [-s, r2]) holds, thereby ensuring that Pr(y > s|x, r ri) < Pr(y >
s/x, - p===-, r2), which confirms the monotonicity of y with respect to r.
B.2. Proof of Lemma 2
Proof. IF Pr(y = 1|x, r) is monotonically increasing with respect to r. We define G(x,r) = Pr(y = 1|x, r). then the function G.., (r) = G(x = Zo r) (0, 1) is continuous and monotonically increasing with respect to within the bounded set D. Therefore, Gz. (x) is a fraction (by D) a cumulative distribution function, and we still use Geo to denote this CDE Define the random variable e such that (c. x = mo) ~ Gao (c), which is conditionally independent of the variable r, i.e.
12

---

Learning Monotonic Probabilities with Generative Cost Model
IL c x. Furthermore, we find that
Pr(c - r|x = 2o. r = = ru)
Pr(c x xo x = 20. r ro)
-Pr(c x ro = 2n)
(by 11 c)
(ro(ro)
G( ro)
Pr(y 1/x LD, ro)
thus rx d 1(c <r) rx
Conversely, suppose that there exists a variable such that r TT c and y [x,r] lI(c - r) {x, r). For any rı - r2 where Tı; T2 E D, it follows:
Pr(y 1|x,1 r1)
Pr(c x x(xxr=-)))
( (x,r) $ Il(c x r) (x,r)) (by r 1l c x)
Pr(c < x//)
Pr(c x r2/x)
(by r < 12)
Pr(y = =1(x,r=r2).
confirming that Pr(y r) is monotonically increasing with respect to
C. Discussion Bounded Revenue Variable
The bounded revenue variable issue can arise in practice, for instance, when our model trained with |||]| < 6, but is used for inference with revenue whose elements r; > This discrepancy can cause an out-of-distribution (OOD) problem, resulting in Pr(y 1[x,r) r) Pr(r c|x) - 1. We address this by hypothesizing non-zero probability
Pr(c +x0 po 0, which yields bounded estimate:
Pr(y = 1|x, r) = Pr(r > c|x) < 1 - Po-
This upper bound is independent of r. Alternatively, we can replace the revenue variable with a bounded invertible monotonic function h(r), for instance, the sigmoid function. Consequently, the probability Pr(y 1 1 h(r)) will be monotonic with respect to hrr), effectively transforming the original unbounded problem into bounded one.
D. Details of GCM
D.1. Derivation ELBO
The ELBO of GCM is expressed as follows:
log Po (x, r,
= llog E2 2<zzz(())00 Xx, r, y/z)
Po Xx, r, y 122)
= =log Ez-ga
(z x)
pa(x, r, y/ (/z)po(z)
>E log
(2 x)
Pro (c Y, r/z, r () r (z)
-Eg-gs log
gs (z
Pro(e Yy (l))
= -E -Ezzga log
+ log po r/z).
(x(2(x)
13

---

Learning Monotonic Probabilities with Generative Cost Model
D.2. Calculate the Likelihood
We assume that c is element-wise independent. Thus, can decompose the conditional likelihood follows:
Pro(e Yy -1- [ pe po(cc/z)des
(10)
where c; the i-th element of c and r, the i-th element of r.
D.3. GCM for Continuous Regression
When y is a continuous variable, the regression problem can be transformed into a binary classification problem, as demonstrated in Lemma 4.1. Consequently, this allows us develop the generative model for the latent variables and c, so that
Pr(y > (/x, r, Pr(r > c|x,r,)) Pr(r c)r,z).
(11)
We assume that y and adhere to conditional Gaussian distributions: specifically, pe(y|x, r, z) N (y; r, 2), oz (x)) and po (t/x, r, z) = N (t: pı(x), of (x)). Here, oy(x), ((xx), and (x(x) are functions we defined, while My (x, r, z) is determined via Equation (11). We recognize that
My (,) -x(x)
Prry (/x,r, = Pr(r > c)r,
vof(x) + ox()
enabling us to define Myy
My r. =)
oo) -xx (0- (Pr(r > c(r, + (x()..
The established definitions of and ensure that is conditionally independent of given and r, as stipulated by Lemma 4.1. By employing variational inference, the loss function corresponding to the continuous problem is expressed as:
Po z r)po (z) (x(z)
C Euzag log
9s (z|x)
(y 12y) Exzng
Pe (z) Pe (x|z)
log oy loog
20y² 2
2o (z)x)
The estimator for with respect to and is expressed as:
(x,,,,, 2(1)),
where (") is sampled randomly from the variational distribution (z)2)) x).
E. Experiment Details
F. Positive Weight Matrix
In both the baseline models and our IGCM approach, positive matrices are essential ensure monotonic linear projections. Through our experiments, discovered that the transformation:
W+ softplus(1OW)
yields better positive matrices compared to using operators like square, softplus, leaky relu, or exp.
14

---

Learning Monotonic Probabilities with Generative Cost Model
E.2. Hyperparameters
Table 5. Hyperparameters the experiments.
dataset
hidden dim.
sample number
latent dim.
max epcoh
optimizer
batch size
learning
rate
Adult
16
32
4
40
ADAM(Kingma
256
0.001
& Ba, 2015)
COMPAS
32
200
ADAM
64
0.01
Diabetes
32
20
SGD
256
0.3
Blog Feedback
32
40
ADAM
256
0.001
Loan Defaulter
16
32
20
ADAM
512
0.003
Auto MPG
6
32
120
ADAM
16
0.01
F. Ablation Studies
F.1 Ablation on Latent Dimension and Sample Number
We conduct ablation studies on the GCM and IGCM methods using the Adult dataset, focusing on two key hyperparameters: D, representing the latent dimensionality, and N, the sampling amount. The values for D and N are chosen from the sets {4. 8, 12, 16] and {8, 16, 24, 32} respectively, and each experiment is repeated 10 times with distinct random seeds. The results indicate that both GCM and IGCM perform optimally in the parameter space's 's northeast region.
Table Experimental results (AUC) of GCM on the Adult dataset with multiple D and N settings.
D =
D 12
D
N 0.7834 #0.0026
0.7840 #0.0028
0.7842 #0.0025
0.7857 +0.0022
N 16
0.7833
#0.0024
0.7824 +0.0023
0.7835 10.0015
0.7851 #0.0024
N 24
0.7833 #0.0014
0.7825 #0.0021
0.7816 10.0022
0.7842 +000021
N 32
0.7844 -0.0025
0.7822 #0.0030
0.7836 +0.0024
0.7843 #0.0034
Table 7. Experimental results (AUC) IGCM on the Adult dataset with multiple and settings.
D 4
D =
D 12
D 16
N 0.7892 +000019
0.7900 +-00022
0.7907 +-0.0155
0.7914 +.00010
N 16
0.7894
=0.0024
0.7911 #0.0012
0.7909 +0.0009
0.7918 #0.0009
N 24
0.7908
=0.0018
0.7901 +0.0020
0.7924 10.0007
0.7893 +0.0019
N = 32
0.7891 +0.0018
0.7899 #0.0018
0.7889 #0.0022
0.7917 #0.0011
G. Comparison of Time Complexity
One of the primary benefits of our GCM model its efficiency during the inference phase, For given x, the model efficiently computes (y(y x, rς) across various values. This efficiency stems from the fact that the GCM model predicts the latent variables z and based solely on x. This enables it infer y using and as per Equation (10) with minimal computational time. Consequently, we bypass the extensive time required to process each pair (x æ, r:) through deep neural network, as is common with traditional approaches. We assessed the inference efficiency for different counts of r while maintaining a constant with results detailed in Table 8. As illustrated, the GCM emerges as the fastest method when the quantity of r surpasses 64, highlighting its efficiency in multi-revenue prediction contexts. When the count of r reaches peak value of 1, 024, GCM can reduce computational time by up to 72% compared to the quickest baseline model.
15

---

Learning Monotonic Probabilities with Generative Cost Model
Table 8. Inference time (ms per batch) of different models with different numbers of r the COMPAS dataset.
Method
Number of
2
4
8
16
32
64
128
256
512
1024
MM
1.51
2.35
3.33
4.83
9.27
17.36
31.24
58.53
112.65
306.57
308.33
CMNN
3.39
5.17
9.02
15.87
28.95
51.96
102.01
198.07
394.63
869.76 877.47
PWL
1.02
1.67
2.47
3.73
7.86
13.89
26.01
47.86
92.95
280.70 285.48
GCM
11.66
11.55
11.98
12.89
13.88
16.85
20.14
28.89
43.88
76.23 79.63
16


Reviews:
[{"review_id": "FGbNHQO1X9", "reviewer": "Reviewer_rxdb", "summary": "The paper introduces a new generative framework to model monotonic probabilities by reformulating the traditional problem into learning a latent cost variable. Instead of directly designing a monotonic function, the authors propose that for a binary outcome, the probability is given by the event that a latent cost variable is dominated by a revenue variable. Two models are presented: (1) GCM (Generative Cost Model): Targets strict monotonicity by modeling the latent cost variable via two independent latent variables. (2) IGCM (Implicit Generative Cost Model): Extends the approach to capture cases where monotonicity reflects correlation rather than a strict order.\n\nThe paper validates the approach through simulated quantile regression and experiments on multiple public datasets. Results indicate that GCM/IGCM achieve superior performance compared to existing monotonic neural network methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the reviews and valuable suggestions. Here are our answers to the concerns raised:\n\n# Answer 1: Bounded Revenue Variable\n\nThis issue can arise in practice, for instance, when our model is trained with $r \\\\in (-10, 10)$, but is used for inference with values $r \\\\gg 10$. Such a discrepancy can lead to an out-of-distribution (OOD) problem, resulting in $\\\\Pr(y=1|x,r)=\\\\Pr(r>c|x)\\\\to1$. We address this by hypothesizing a non-zero probability $\\\\Pr(c=+\\\\infty|x)=p_0>0$, which yields a bounded estimate:\n$$\n\\\\Pr(y=1|x,r)=\\\\Pr(r>c|x)<1-p_0.\n$$\nThis upper bound is independent of $r$. Alternatively, we can replace the revenue variable $r$ with a bounded monotonic function $h(r)$. Consequently, the probability $\\\\Pr(y=1|x,h(r))$ will be monotonic with respect to $h(r)$, effectively transforming the original unbounded problem into a bounded one.\n\n# Answer 2: Conditional Independence\n\nThe conditional independence $c\\\\perp r\\\\mid x$ is crucial for solving the strict monotonic issue of $p(y|x,r)$ by achieving $\\\\Pr(c\\\\prec r|x,r)=\\\\Pr(c\\\\prec r|x)$, with $r$ representing the revenue variable and $y$ as its monotonic outcome. As demonstrated in Lemma-2, for any strictly monotonic probability $p(y|x,r)$, such a latent variable $c$ is guaranteed to exist, allowing us to construct the $c$ model without worrying about its practical interpretation.\n\nConversely, in non-strict monotonic problems, finding such a $c$ is impossible. For example, a consumer might prefer buying a product at a lower price, but if the price is excessively low, it might seem dubious, resulting in the consumer not buying it, illustrating that $c$ (consumer cost) is affected by $r$ (inverse of the product price). In this case, an implicit monotonic model, detailed in section-4.4, can be utilized. Here, the concealed kernel variable $k$ denotes the product's authenticity, and then $r$ shows monotonicity with respect to $k$, enabling us to identify a cost variable $c$ so that $c\\\\perp k\\\\mid x$, instead of $c\\\\perp r\\\\mid x$. Here, $c$ and $k$ are both latent variables that the modeling for them will not obey the practical interpretation of $r$ and $y$.\n\n# Answer 3: Ablation on Hyperparameters in Variational Inference\n\nThe ablation of $D$, the latent variable dimension, and $N$, the sampling number, is detailed in Table-5 and Table-6 within Appendix C. Testing results for GCM using the Adult dataset are as follows:\n|      | D=4 | D=8 | D=12 | D=16 |\n| ---- | ------ | ---- | ----- | ---- |\n| N=8  | 0.7834 | 0.7840 | 0.7842 | 0.7857 |\n| N=16 | 0.7833 | 0.7824 | 0.7835 | 0.7851 |\n| N=24 | 0.7833 | 0.7825 | 0.7816 | 0.7842 |\n| N=32 | 0.7844 | 0.7822 | 0.7836 | 0.7843 |\n\nThe findings indicate that $D$, the latent dimension, impacts the test AUC, whereas the sampling number $N$ plays a lesser role. Another critical hyperparameter is $\\\\beta$, utilized in the ELBO as per the $\\\\beta$VAE approach, where the modified ELBO is expressed as follows,\n$$\n\\\\text{ELBO}_\\\\beta=\\\\mathbb E_q \\\\log {\\\\text{Pr}( c\\\\curlyvee_y r| z, r)p({x},{r}| z)}-\\\\beta \\\\log \\\\mathbb E_q \\\\frac{q( z| x)}{p( z)}.\n$$\nThe test outcomes for varying values of $\\\\beta$ are:\n|model |AUC |\n|----|-----|\n|$\\\\beta=0$ |0.7844|\n|$\\\\beta=1$ |0.7828|\n|$\\\\beta=2$ |0.7844|\n|$\\\\beta=3$ |0.7824|\n|$\\\\beta=4$ |0.7841|\n|$\\\\beta=5$ |0.7830|\n\nThe results reveal that $\\\\beta$ values within $\\\\{0,\\\\cdots,5\\\\}$ yield comparable results, with $\\\\beta=0$ achieving the best performance. This is attributed to the test AUC metric being primarily focused on the accuracy of $y$, rather than the whole generative model $p(x,r,y,z,c)$.\n\nWe hope these clarifications address your concerns and demonstrate the validity and applicability of our proposed method. Thank you again for your valuable feedback.", "claims_and_evidence": "I think authors argument are clear and convincing.", "methods_and_evaluation_criteria": "I think both the proposed methods and the evaluation criteria are sensible for addressing monotonicity in probabilistic predictions.", "theoretical_claims": "The proofs are logically coherent, I think they make sense to me.", "experimental_designs_or_analyses": "I think the experimental design and analyses are sound and effectively demonstrate the strengths of the proposed models.", "relation_to_broader_scientific_literature": "I think the authors successfully relate their contributions to prior findings and articulate how their work extends and improves upon existing methods.", "essential_references_not_discussed": "Although I am not familiar with the relevant literature, the references cited in the paper provide sufficient context to help me understand its contributions.", "other_strengths_and_weaknesses": "- Strengths : The paper’s main strengths lie in its originality and rigor. It introduces a novel formulation by recasting the monotonic probability problem as one of learning a latent cost variable, thereby addressing both strict and implicit monotonicity within a unified generative framework. Their contributions supported by solid theoretical derivations and extensive experiments on simulated and real-world datasets.\n\n- Weaknesses:  Assumptions such as a bounded revenue variable and conditional independence, may not always align with practical scenarios. I am wondering how sensitive of the methods to these assumptions, what would happen if the model assumptions are misspecified?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Could the authors clarify the practical implications of the conditional independence assumption? In particular, what types of real-world datasets or scenarios might violate this condition, and how does the model’s performance degrade in such cases?\n\n2. The model involves variational inference with multiple latent variables, and the results appear sensitive to hyperparameter choices. Could the authors provide additional ablation studies on how these choices affect both training scalability and performance of the method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f2PZ49z0dV", "reviewer": "Reviewer_fPoj", "summary": "The paper studies the problem of monotonic regression where the target variable should maintain a monotonous relationship with part of the input variables. It first establishes some analytical properties for the probability model underlying the monotonous regression and then it proposes a Bayesian network model to capture the relationships between different classes of variables. It provides numerical experiments to support the discussion.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and suggestions. We appreciate the opportunity to address the concerns raised in the review.\n\n# Answer 1: Discriminative vs. Generative Models\n\nWe respectfully disagree with the assertion that generative models are inherently more challenging than discriminative models. The modeling difficulty depends on the task and the structure used. For example, naive Bayes, a generative model, is widely used for classification tasks and is comparable to discriminative methods such as logistic regression. In our paper, we have shown that a discriminative model $\\\\Pr(y=1|x,r)=G(x,r)$ can be transformed into a generative model $p(x,c)=p(c|x)p(x)$ using the relationship established in Lemma-2:\n$$\n\\\\Pr( c \\\\prec  r| x) =\\\\Pr( c \\\\prec  r| x,r) = G(x,  r),\n$$\nand consequently\n$$\np( c | x) =\\\\partial G(x,  c) / \\\\partial c.\n$$\nThis demonstrates that in monotonic modeling tasks, these approaches can be interconverted, so it is unfair to say that generative methods in monotonic modeling are more difficult arbitrarily.\n\nMoreover, traditional monotonic networks face challenges such as requiring all weight matrices to be positive [1] and all activations to be monotonic and not fully convex or concave [2]. Common normalization techniques like layer-norm and batch-norm are also not allowed since they are not monotonic operators. As a result, it is hard to train a deep monotonic network. However, with the cost generative model, we can easily build a deep network for generating the cost variable $c$ and obtain the target variable $y$ via $\\\\{y=1\\\\}=\\\\{c\\prec r\\\\}$. This allows us to use unconstrained weight matrices, non-monotonic activations such as silu and gelu, fully convex activations such as relu and softplus, and normalization techniques widely used in deep networks such as LN and BN. Training this generative network is also straightforward following the loss function demonstrated in (7).\n\n[1] Monotonic Networks. [Link](https://proceedings.neurips.cc/paper_files/paper/1997/file/83adc9225e4deb67d7ce42d58fe5157c-Paper.pdf)\n\n[2] Constrained Monotonic Neural Networks. [Link](https://arxiv.org/pdf/2205.11775)\n\n# Answer 2: Challenge of Learning the Joint Probability Distribution\n\nPrevious studies show the outstanding performance of learning the joint probability in machine learning tasks, such as learning the joint distribution of an image via generative methods. For simple tasks such as the MNIST dataset, a simple four-layer VAE [3] is sufficient, while more complex tasks like CIFAR10 might require deeper models such as DDPM [4]. This paper has shown that the cost variable $c$ is essential for all monotonic problems $\\\\Pr(y=1|x, r)$, where $r$ is the monotonic revenue variable, $x$ is the non-monotonic variable, and $y$ is the response variable determined by $c\\\\prec r$. Since $x,r,c$ are all real vectors, we can adopt model structures similar to image generative models.\n\nTo design the generative model for $x,r,c$, we have chosen a basic structure similar to the VAE, as the tasks in our paper are not overly complex. Although $c$ is not directly observable, we can evaluate the model for $c$ using the observable variable $y=\\\\mathbb I(r\\\\succ c)$, and the testing AUC/ACC of $y$ indicates the performance of our generative model of $c$. Additionally, the ELBO $\\\\mathbb E_{z\\\\sim q}\\\\log p(x,r,y|z)-D_{KL}(q(z|x)\\\\|p(z))\\\\leq \\\\log p(x,r,y)$ serves as a lower bound for the evidence $\\\\log p(x,r,y)$, providing another evaluation metric.\n\n[3] Auto-Encoding Variational Bayes. [Link](https://arxiv.org/abs/1312.6114)\n\n[4] Denoising Diffusion Probabilistic Models. [Link](https://arxiv.org/abs/2006.11239)\n\n# Answer 3: Choice of Functions in (4) and (8)\n\nWe employ the standard normal distribution for the priors of $z_1$ and $z_2$. Each conditional likelihood $p(a|b)$ is expressed as:\n$$\n\\\\mu_a, \\\\log \\\\sigma_a = \\\\text{MLP}(b), \\\\ \\\\ \na \\\\sim \\\\mathcal N (\\\\mu_a, \\\\sigma_a^2).\n$$\nIn the GCM model, we use the reparameterization trick to derive the variable $a$, or we apply the CDF function of the normal distribution to compute the probability $\\\\Pr(a<a_0)$. The MLP here consists of one or two layers. We will provide further details in the appendix of future revisions.\n\n# Answer 4: Sufficiency of a Three-Layer Network\n\nThe number of layers is not a constraint for generative models. As mentioned in Answer 2, the VAE applied to the MNIST dataset uses only two layers with tanh activation functions and two affine layers without activations, yet it successfully generates high-quality handwritten numerals. In our experiments, the dimension of the revenue variable (see Table 3 on page 8) is less than 10, much smaller than the pixel count of an MNIST image. Therefore, we chose not to use an excessively deep generative network for our study.\n\nWe hope these clarifications address your concerns and demonstrate the validity and applicability of our proposed method. Thank you again for your valuable feedback.", "claims_and_evidence": "Yes. I think all the claims are supported.", "methods_and_evaluation_criteria": "The evaluation criteria make sense to me. \n\nBut I disagree with the proposed method. If we step back, the monotonous regression can be viewed as a discriminative task, though there is the challenge of maintaining monotonicity. Yet the proposed solution essentially relies on a generative model, or, a probability model for the joint distribution of input variables and output variables, together with some latent variables. Generally, it's perceived that generative modeling tasks are more difficult than their discriminative counterparts. So I think the proposed method is to some extent misleading for future works or practical uses, in that it is too ambitious to assume that the joint probability distribution can be approximated/learned by the proposed probability model, and there is no way to know/check when it cannot be.", "theoretical_claims": "Yes, the claims are correct based on my check.", "experimental_designs_or_analyses": "The paper didn't discuss much about the choice of functions in (4) and (8). \n\nIt mentions \"All methods utilize the same foundational architecture: a three-layer perceptron network utilizing tanh activations\". I think this part is done too casually. Is a three-layer network sufficient to serve as a generative model?", "relation_to_broader_scientific_literature": "Nan.", "essential_references_not_discussed": "Nan.", "other_strengths_and_weaknesses": "Nan.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "POzTuRZT4z", "reviewer": "Reviewer_6X1f", "summary": "The paper tackles the problem of enforcing monotonicity in predictions as a function of some variables if the true underlying function also abides by the monotonicity argument. Given the prediction target $y$ that is supposed to be monotonic with respect to the variable $r$, the paper relies on intuitive observation that this monotonicity condition can be translated to some additional auxiliary variable as long as that variable satisfies conditional independence condition. The paper then proposes a generative model to learn this auxiliary variable using variational inference. Experiments are done on synthetic dataset as well other real-world datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the reviews and valuable suggestions. Here are our responses to the concerns raised:\n\n# Answer 1: Clarification on Figure 6\n\nThe scatter plot in the background represents the training instances $(x_i, y_i)$, with $p(y|x)$ defined in the southeast corner of page 7. Consider $y_r|x$ as the r-th quantile of $y$ given $x$; thus, $y_r$ is a function of $x$, and $(x, y_r)$ can form a curve $\\\\Gamma_r$ on the x-y plane. For $r_2>r_1$, the monotonicity of quantiles ensures that the curve $\\\\Gamma_{r_2}$ lies above the curve $\\\\Gamma_{r_1}$. In Figure 6, five estimated quantile curves (red curves) are plotted for $r$ in $(0.1, 0.3, 0.5, 0.7, 0.9)$ using different monotonic modeling methods at different training stages. Each column signifies a distinct modeling approach, and each row denotes a training stage. An accurate quantile value approximation should result in each estimated quantile curve being close to its actual quantile curve. Therefore, the estimated curves should:\n- Not have excessively narrow gaps between them.\n- Have $\\\\Gamma_{0.1}$ and $\\\\Gamma_{0.9}$ adjacent to the edges of the background scatter plot, with limited outliers.\n\nObserving Figure 6 reveals that:\n- The red curves for PosNN, MM, SMM, and SMNN (mostly monotonic by construction methods) are clustered in certain areas.\n- CMNN, Hint, and PWL (mostly monotonic by regularization methods) exhibit too many outliers.\n- GCM displays distinct gaps between the red curves and the fewest outliers.\n\n# Answer 2: More Intuition for the Experiments\n\n**Comparison between GCM and IGCM**: Our analysis reveals that within public datasets, the IGCM outperforms the GCM. This improved performance is attributed to the fact that real-world scenarios do not always adhere to strict monotonic relationships; for instance, while BMI is correlated with diabetes, it is not necessarily causative. Typically, the selection of monotonic features is informed by experience or statistical analysis. Under these circumstances, implicit monotonic assumptions tend to be more applicable. Our experimental data bolsters this assertion, as IGCM demonstrates superior performance to GCM in five of the six datasets.\n\n**Comparison between strict monotonic methods**: The strict monotonic methods evaluated include PosNN, MM, SMM, CMNN, SMNN, and the proposed GCM. The basic PosNN shows poor performance across most datasets, indicating that merely establishing a positively weighted neural network is insufficient for monotonic problems. Interestingly, the traditional MM method performs commendably, and its successor, SMM, exhibits a correlation, indicating that the core min-max architecture proves effective in optimizing the monotonic network. Recent developments like CMNN and SMNN introduce innovative monotonic network architectures, outperforming MM in several experiments. Our proposed GCM method consistently exhibits strong performance across all datasets, suggesting that reframing the monotonic problem as a generative one yields enhanced results compared to simply altering the network structure.\n\n**Comparison between nonstrict monotonic methods**: Hint, PWL, and IGCM are non-strict monotonic methods, yet only IGCM addresses the non-strict issue through implicit monotonic modeling. This approach consists of two strict monotonic challenges: $p(r|k)$ and $p(y|k)$, establishing a nonstrict monotonic link between $y$ and $r$, which proves beneficial when compared to traditional non-strict monotonic methods.\n\n# Answer 3: Other Potential Improvements\n\n**Rapid Decision Making**: As illustrated in Appendix D, GCM and IGCM provide the quickest inference for multiple revenue variables, providing a clear advantage in decision-making models. For instance, consider a robot tasked with determining a sequence of actions for a particular mission, where each action is directly proportional to total energy usage. If there is a model capable of forecasting energy consumption to help the robot minimize energy loss, this model must rapidly predict multiple action options. Here, the GCM method proves beneficial for swift decision making.\n\n**Scalability**: GCM is inherently scalable. Unlike previous monotonic models that struggle to scale due to restrictions on non-monotonic normalization techniques (like LN, BN) and limitations on activation functions or linear projections, GCM can incorporate any structure to model $p(c|x,z)$ and $q(z|x)$. Thus, with more extensive datasets, GCM's scalability advantages may become more evident.\n\n**Monotone Multitask Modeling**: For a set of tasks $y_1,\\\\cdots,y_n$ that are monotonic such that $(y_i=1)\\\\subset(y_{i+1}=1)$ and each $y_i$ is monotonic with respect to $r$, we can apply the monotonic cost variable sequence $c_1\\\\succ \\\\cdots \\\\succ c_k$ and let $y_i=\\\\mathbb I(r \\succ c_i)$ to address it.\n\nWe hope these clarifications address your concerns and demonstrate the validity and applicability of our proposed method. Thank you again for your valuable feedback.", "claims_and_evidence": "1. Theoretical claims are well supported, and the paper is generally well-motivated in that sense. \n2. I like the synthetic experiment on quantile regression to tractably study the problem setting, however I could not verify the insights in Figure 6. The paper claims estimated quantile values maintain strict monotonicity; I think I'd be helped if authors can provide an elaborate and annotated version of Figure 6 so that I can compare the violations of monotonicity for baselines methods. \n3. In a similar sense, I think I'd be helped if more intuition can be provided for the real-world dataset experiments in Section 5.2. The paper currently compares standard metrics like AUC and RMSE, but I'd appreciate if authors could clarify how the monotonicity assumption or requirement factors in that improves these metrics as the paper claims or the experiments suggest.", "methods_and_evaluation_criteria": "I like the synthetic experiment to tractably study, and it makes a lot of sene to study quantile regression as a test-bed for monotonicity (however, I believe the experimental insights can be presented more meaningfully), and would appreciate clarification on the evaluation on real-world datasets (see above).\n\nI'd personally also like to see comparisons on the efficiency or the computational aspects of the inference mechanism as proposed in the paper compared to the baselines. I see Table 2, and to me, the reported numbers live close to each other, or the improvements are not that significant; I don't want to play the devil here as I don't necessarily think that's a strong concern, but I'd love to get some more insights as what are other potential improvements of the proposed methodology; computational, ease of applicability, etc.", "theoretical_claims": "While I do get the intuition behind their theoretical claims (Lemma 4.1 and Lemma 4.2), I haven't very rigorously verified the notational description of the proofs. But to me the results are basic (in a positive sense) and correct.", "experimental_designs_or_analyses": "Some questions remain (see above).", "relation_to_broader_scientific_literature": "The paper presents an intuitive insight and builds an inference mechanism around it to enforce monotonicity in the probabilistic predictions. Compared to the literature in the paper, it", "essential_references_not_discussed": "Not that I'm familiar of.", "other_strengths_and_weaknesses": "The paper is generally written well, but maybe proofs can go to appendix, to make space for more experimental insights. In addition to that, the paper is bit heavy on the notation, and could use some intuition to motivate the main insights. And maybe positioning it more broadly it into the current literature could help strengthen the contributions.", "other_comments_or_suggestions": "None.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies a regression model where the response variable is stochastically monotonic with respect to one ($r$) of the two covariates ($r$ and $x$). They first reduce this monotonic regression model to a monotonic classification model and then show a simple yet interesting and quite powerful result that states that the monotonic assumption is equivalent to the existence of a latent variable that is independent of $r$ conditionally on $x$. This result allows them to learn the model without introducing regularization techniques, which are costly and introduce a possibly large bias in practice. They adopt a generative approach to model and learn this latent variable using variational inference. Numerical experiments suggest that their method outperforms traditional ones. 

Two reviewers are positive/very positive about this work, both for its clarity and its content, which also provides good intuition and smart ideas. They raise important questions related to the assumptions that may be quite restrictive (e.g., boundedness of the covariate $r$), which were discussed in the rebuttal. 

One reviewer was very negative, but more on the general idea of using generative modeling rather than on the paper itself. Therefore, it does not outweigh the other positive reviews and I recommend accepting this paper. Please, note that the rebuttal discussions should be included in the final version of the paper.
Decision: Accept (poster)