Hierarchical Lattice Layer for Partially Monotone Neural Networks
Authors: Hiroki Yanagisawa
Kohei Miyaguchi
Takayuki Katsuki
Venue: NeurIPS 2022 Accept (2022)
Keywords: partially monotone regression
partially monotone neural networks
monotonicity constraints

Abstract: Partially monotone regression is a regression analysis in which the target values are monotonically increasing with respect to a subset of input features.   The TensorFlow Lattice library is one of the standard machine learning libraries for partially monotone regression.  It consists of several neural network layers, and its core component is the lattice layer.  One of the problems of the lattice layer is that it requires the projected gradient descent algorithm with many constraints to train it.  Another problem is that it cannot receive a high-dimensional input vector due to the memory consumption.   We propose a novel neural network layer, the hierarchical lattice layer (HLL), as an extension of the lattice layer so that we can use a standard stochastic gradient descent algorithm to train HLL while satisfying monotonicity constraints and so that it can receive a high-dimensional input vector.  Our experiments demonstrate that HLL did not sacrifice its prediction performance on real datasets compared with the lattice layer.

Hierarchical Lattice Layer for Partially Monotone Neural Networks
Hiroki Yanagisawa IBM Research Tokyo
Kohei Miyaguchi IBM Research Tokyo
Takayuki Katsuki IBM Research Tokyo
IBM Japan, Ltd.
IBM Japan, Ltd. Tokyo, Japan miyaguchi www.cchhhiiiioooooooooooo com
IBM Japan, Ltd. Tokyo, Japan kats@ jp ibm. com
Tokyo, Japan yanagis@jp.. ibm. com
Abstract
Partially monotone regression is a regression analysis in which the target values are monotonically increasing with respect to subset input features. The TensorFlow Lattice library is one of the standard machine learning libraries for partially monotone regression. consists of several neural network layers, and its core component is the lattice layer. One of the problems of the lattice layer is that requires the projected gradient descent algorithm with many constraints to train it. Another problem is that it cannot receive high-dimenn sigh-dimensional input vector due to the memory consumption. We propose a novel neural network layer, the hierarchical lattice layer (HI.L), use a standard stochastic gradient descent algorithm to train HI.. while satisfying monotonicity constraints and that it receive high-dimensional input vector. Our experiments demonstrate that HLL did not sacrifice its prediction performance real datasets compared with the lattice layer.
extension of the lattice layer so that
Introduction
In regression analysis, we often have prior knowledge that the target values are monotonically increasing (or decreasing) with respect to subset of imput features. For example, when we estimate house prices from house data (e.g., the area size, the number of rooms, and the distance to the nearest supermarket), we know that the target value (i.e., the house price) is monotonically increasing with respect to some of the input features (e.g., the area size) if the other features are equal PPotharst and Feelders 2002
Partially monotone regression is regression analysis that exploits such prior knowledge. The input of this regression is feature vector Rdd, which be partitioned into (p,q) such that p E IR IRd is non-monotone features and g E I is monotone features. The goal of this regression monotonically increasing on q. Partially
to find a function / such that f(x:) = I' [p, q) E monotone regression models have advantages over the standard regression models. For example, they
have a better regularization capability [Dugas et al.) [2000] Fard et al. 20166; You er al. [017] and
better interpretability [Gupta et al. 2016). They can be used for fair machine learning [Wagg and Gupta 2020
Many machine learning models have been developed for partially monotone regression. For example. there are decision-tree based models (Potharst and Feelders) 2002]. SVM models (Lauer and Bloch 2008), boosting models [Bartley et al. 2018), and random forest models [Bartley et al. Many popular open source libraries for gradient boosting can handle partial monotonicity (e.g., CatBoost, LightGBM, and XGBoost). See survey paper [Cano et al. 2019 and the references in [Gupta el al. 2016 for comprehensive surveys.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

As for neural network models, the TensorFlow Lattice library [Canini et al. 2016; Gupta et al. 2016; You et al,, 2017] is a standard model for partially monotone regression, and available as open source library. It consists of several neural network layers, and its core component is the lattice layer [Cupta et al.) [016]. Hereafter, we call this layer TI. Lattice. TI. Lattice constructs hypercube with k vertices as an internal structure, where k 2 2 is an integer hyperparameter for specifying the granularity of the hypercube, and d is the dimension of the input vector æ E IR One of its problems is its memory consumption in storing kd parameters, and therefore, it cannot receive a high-dimensional input vector with a large d. This means that we need to combine multiple neural network layers to handle an input vector with large d (e.g., by using small lattices [Canini er You et al. 2017). Another problem its requirement for the projected gradient descent algorithm with many constraints to train it to handle monotonicity constraints. This means that cannot use a standard stochastic gradient descent algorithm to train it. The details of the training algorithms used for the TensorFlow Lattice library described in [Gupta et al. 2016 You et al. 2017].
al. 2016
In this paper, we propose novel neural network layer, the hierarchical lattice layer (HLL), as an extension of TL Lattice. The internal structure of HLL is similar to that of TL Lattice in that both layers construct hypercube structures. However, they are different in several aspects. For example, HLL constructs an m-dimensional hypercube, whereas TL Lattice constructs a d- dimensional hypercube, where d and are the dimensions of the input vector E R and its monotone feature q E Rmm, respectively. Since m is usually a small integer in practice, the number of vertices knnh in HLL is significantly smaller than that of TL Lattice (i.e., k²). Therefore, HIL can receive a nigh-dimensional input vector as long as m is small. For another example, the parameterization of HIL is designed to represent class of partially monotone functions (i.e., HIL cannot represent a function f(p 9) that is not monotonic on q) by using the hierarchical relationship between the vertices of the hypercube. Therefore, the internal structure of HLL ensures partial monotonicity regardless of the training algorithm used. In contrast, the internal structure of TL Lattice is lookup table, and the table itself can represent any function [(p q). which means that the parameterization of TL Lattice is not designed to satisfy monotonicity constraints.
In our experiments, we demonstrate that HLL did not sacrifice its prediction performance compared with TL Lattice. Note that HLL is designed to satisfy monotonicity constraints and is not designed to improve the prediction performance compared with TL Lattice. Therefore, the purpose of our experiments was not to show improvements in prediction performance compared with TL Lattice but to show that HLL did not sacrifice the prediction performance to satisfy the constraints. In addition, we demonstrate that we could use HLL for high-dimensional input vectors with a small m.
2 Partially Monotone Regression
Regression analysis a method for finding a function f : R + R such that f(x;) approximates y: for given dataset D {< 91) where 2; Rd is a feature vector, and ys IR is target value. In partially monotone regression,
assume that I = (p. q) E dd- x RTT where 0 < m < d, such that f(z) e R is monotonic on q. A function f(x) for x (p.4) is monotonic on q if this inequality holds:
partition feature vector æ E Ed into
S(, q) S f(p, d'), Vp, V4 s',
where q S q' denotes the inequality for all the elements (i.c., q/il S q' for all S i m, where q[i denotes the i-th element of g).
Partially monotone regression has many applications in various fields, and it has important applications even m = 1. An example of such application with m = is quantile regression [Koenker and Basseti 1978 Zhou al. 22000, which used to estimate the conditional quantiles of target variable distribution. We show that the output ((x) q(<',7) of the quantile regression for an input x = (x', r) is monotonic on t, where x' E Rd is a feature vector, and T E [o, 1] is a quantile level. In quantile regression, we estimate the quantile value g(x', r) defined by
q(2',7) inf F(]| = x')
where F(y|X= x²) is the conditional cumulative distribution function of Y given X z':
F(y|X = x') Pr(Y s y = z').

---

Since the function g(z', T) is monotonically increasing on this problem can be modeled as partially monotone regression with m Note that, if we use standard multi-layer perceptron without any monotonicity constraints to estimate the quantile value g(=',)) it sometimes fails to satisfy the monotonicity constraints, which is known as the crossing problem (see e.g., (Tagasovska and Lopez-Paz, 2019)). In quantile regression, many algorithms have been proposed to resolve the crossing problem (e.g., [Par et al. 2022)).
Another example of partially monotone regression with m = 1 is survival analysis, which is also known as meeeto-event analysis (Wang et al. 2019, Rindt al.) 2022] We show that the output f(z) = (a', t) of the survival analysis for an input a feature vector of a patient, and E R is the time of an event of interest of this patient. In survival analysis. the event of interest typically corresponds to the death of a patient. The goal of the analysis to estimate a survival function S(a',)) 1) for patient x', which represents the probability that the event does not occur until time t for patient x'. By setting f(x', .t) = S(x', t), we can model the survival analysis as a partially monotone regression with m = 1, where the value f(a', t) shows the mortality rate of patient x' at time t, and f(x', t) is monotonically increasing on t.
(a', t) is monotonic on t, where z' E
3 Hierarchical Lattice Layer
In this section, explain new hierarchical lattice layer (HLL) for neural network models for partially monotone regression. It can be seen as an extension of TL Lattice [Gupta et al. [016]. Note that the input of HLL is a d-dimensional vector I E [0, 1j and its output f(z) is in [0, 1J, where the input vector can be partitioned into two subvectors p [0, _jd-m and q [0, 11 such that f (ax is monotonic on g.
We create the mn-dimensional unit hypercube [0, 11 as an internal structure of HIL, and we decom- pose it into (Λ1 1) x (k2 - 1) xx..xx x (km - 1) subhypercubes by dividing the interval [0, 1] of the i-th dimension into k; - subintervals, where each k; 2 2 is an integer hyperparameter that specifies the granularity of the hypercube. Unless otherwise stated, we assume that k h2
km and the interval [0, 1] is divided into k equal-length subintervals just for simplicity. This means that the length of an edge of a subhypercube
1/(k- 1). and we have the kndh vertices in the unit hypercube. We denote the set of the vertices in the unit hypercube by Vn. In this paper, we do not distinguish between vertex V E Vn and its coordinate q. E [0, 11m This means that the value [(, q.) for coordinate qu is written as (V)
3.1 Key Idea of HIL
Before describing the details of HLL, explain the key idea of HLL by using a simple case with m = 1 and k = 3. In this case, HLL is supposed to learn a function f(p. q) such that S f(p S fp. m1) S f(p 92) S 1 holds for any p E Rdd- and q1 = 0, 92 1/2, and gs 1. Hereafter, we use v1. U2, and 03 to represent @ı, 02, and g3. respectively, to emphasize that they correspond to the vertices in Vn..
In HLL, we associate a function g(p, v) for each vertex E Vm so that the function [p, v) for Vm can be represented by using the function g(p v). We show how the function g(p, v) is defined here. First, the function g(p, vo) is defined as g(p, vo) f(p, Second, the function g(p, v1) defined as the function that satisfies this equation:
f(p v1) = (1 - g(p, 01 v)))s(p (p, vo) + (p(
Note that 0 < g(p, v1) < always holds because fpp un) S f(p, v1) < 1. Similarly, the function g(p, 02) is defined as the function that satisfies this equation:
f(p 02) = (1 g(p, ( 01) + 9 v2).
See Figure for an illustration of these definitions. By using these definitions, the task of learning
the function [p, v) with monotonicity constraints 0 < f(p vn) < f(, U1) < f(p 12) < 1 is casted as the task of learning the function (p, v) such that 0 < g(p, v) I without any monotonicity constraints. We generalize this idea for m 1 in the following.
We note here that HILL for the case with m survival analysis (see e.g., [Ren el al.) 2019 Zheng al. 2019). More specifically, the survival function is decomposed into a product of hazard rates in these approaches, and the survival function and the hazard rate can be represented by using f(, v) and g(p, v) in HLL, respectively.
is similar to some existing approaches used in
3

---

g(p. vo) -pp vo)
g(p v1) : 1 - g(p
f(p. 02)
[(, vo)
f(p, v1)
g(p, U2) 1 11- U2)
Figure 1: Illustration of parameterization of HLL for m = I and & = 3.
3.2 Hierarchical Relationship between Vertices
In HLL, we assume that there is a total order (vo, U1...., Vvll1) of vertices in Vm. We can use any total order a to construct HLL. but unless otherwise stated, we assume that a is determined by the increasing order of the Manhattan distance (i.e.. L1 distance) from the origin vertex ti = (0, ......0) E [0, 1) of the umit hypercube with an arbitrary tie-breaker. See Figure 2 for an example of a decomposed unit hypercube with
and 4 and the total order a = (Vo, u1...., U1s) determined by the Manhattan distance.
We define the domination relationship between vertices in Vm. and we define the minimal dominated and dominating sets based on a.
Definition 3.1 (Domination.) A vertex v E IR is said to dominate another vertex u E R if and only if' if'll [[]]] v(i] holds for all E (1,2...., m. and there exists an index i' { m) such
that uli'] < vl]'].
Definition 3.2 (Minimal dominated set.) The set of vertices Lu.a denotes the minimal set of vertices that satisfy three conditions (i) every vertex u E Leço is dominated by v. (ûi vertex u precedes v in o, and (iii) other vertex Lugo dominates
Definition 3.3 (Minimal dominating set.) The set of vertices U2,a denotes the minimal set of vertices that satisfy three conditions (i) every vertex E Uv. dominates v, (ii) vertex precedes u in and (iii) no other vertex w Uv, is dominated by u.
Intuitively the domination relation means that [(p u) S f(p, v) holds for any p Rdd-m if vertex dominates vertex u. Regarding the minimal dominated and dominating sets, there can be multiple minimal sets that satisfy the conditions. Throughout this paper, dominated set Le and an (arbitrary) minimal dominating set U..o for each v E a. Note also that we have | S m and !U,,, 0 is determined by the Manhattan distance. This is why we use the Manhattan distance to determine because we have weaker upper bounds Le S [Vml and |U=,=| [Vm] if a is an arbitrary ordering.
fix an (arbitrary) minimal
Example. In the hypercube shown in Figure 2 vertex Uy dominates the vertices in (uo, v1, 22 Ug }s} and is dominated by the vertices in (111, U12, V13, U14, V1s). By the definition of the minimal dominated set, we have L = {v4, vs} because vertices un. 6i, and vo are dominated by va or ug. By the definition of the minimal dominating set, have Ung.o 0 because all the vertices that dominate Ug appear after vg ordering a.
3.3 Auxiliary Neural Network
We explain the function g(p, v) associated with each vertex u E Vm- functions ((p, v) and u(p, v) forp E IRd-m and a vertex v E Vm as
To define it, we define two
max f(p,0') if Lv.o # 0, ((,,0) = ve
(1)
if L = 0,
min f(p.0') r' d'elu
if Ug # 0,
u(p,v)
if U v. = 0,

---

(0,1)
(1,1)
F2
C4
V10
Uo (0
P0
66 (1,0)
Figure 2: Example of vertices in unit hypercube in HLL with 2 and ls 4.
where To and Un. are the minimal dominated and dominating sets, respectively. Note that ((p v) and u(p, v) are defined so as to satisfy the inequality [(p, v) - f(p, v) S u(p, v). By using this inequality, we define g(p, v) as a function that satisfies
f(p, v) = (1 - g(p, v))((p, v) + g v)u(p, u).
(3)
Since we have ((,,v) S f(p, u(p,v), v). there always exists 0 S g(p, v) S 1 that satisfies Eq. (3)
We construct an auxiliary neural network go (p. v) HLL learn the function ((p v). To satisfy the constraint 0 S ((, v) S 1, we use the sigmoid function in the last layer of go v) as the activation function. Note that, by the definitions of the three functions ((, v),, u(p, v). and g v), we can compute (p, v) for any p and v by using Go v).
Example of Parameterization of HIL. We show an example of the functions v),,),, u(p, v). and g(p, v) for simple case with m = 1 and k = 3 (see Figure In this case, the hypercube has three vertices Vm {up, 21, 02}, and we use the total ordering æ (Vo, 01, 02) here. For the first element to of a, we have ((p, 2o) = 0 and u(p, vo) = because Lug.o UvD.a = @ and g 2b) (p, uo)., which satisfies [(p, un) S f(p, vo) S u(p, vo) and S g(, vo) < For the second element v1 of a, we have (pp, vn) = (p. vo) and u(p, v1) = because L = {wo} and Un.. = 0. The function g(p, 01) defined as a function that satisfies
f(p v1) = (1 g(, v1)) 1(,,11) v1) + g(p. 01) v1).
Note that this definition satisfies l(p, v1) < f(p, v1) S u(p, 01) and 0 < g(p, v1) < 1, which means that the monotonicity constraint fp, oo) S fp, vi) I 1 is satisfied. Similarly, for the third element U2 in a, we have l(p,v) = f(p vi) and u(p, v2) = 1 because Lug.o { and Ung.rr 0. The function g(p 22) is defined as function that satisfies
f(p, 12) = (1 ((, 02)) 1(p, v2) + g(p, 02) u(p, U2).
Note that this definition satisfies U(p, v2) f(p, I u(p 02) and 0 S g(, v2) S which means that the monotonicity constraint (p, vn) I fp, v2) S is satisfied.
3.4 Inference and Training Algorithms
The inference algorithm of HLL is similar to TL Lattice, and Figure Jillutraess it. Given an input x (p. g). we first calculate [p, v) for all V, by using (p v), where V,, S Vm, the set of the vertices that surround in the hypercube. Then, we estimate [' 9) as an interpolation of the values f(p, v) for e E Vg. As for the interpolation algorithm, we can use multilinear interpolation or simplex interpolation [Gupta et al. 2016). Note that the size of V, depends algorithm (i.c., VV, = 2 we use multilinear interpolation, and [Vl = O(m log m) if we use simplex interpolation).
the interpolation
The training algorithm of HLL the same as that of the standard neural network layer. For example, if we train neural network model that consists of single HLL, the parameters of HLL (i.e. the parameters & of the auxiliary neural network go) are trained to minimize the empirical loss
min
Evv. f(x:)),
5

---

f(p,v)) v7) f (p. Us)
f(p vz)
Compute (1, v) by using go
Interpolate
I (p, q)
f(p 4)
-1
f f(p v4) (p, 02)
f(p.
Figure 3: Illustration of inference algorithm of HLL for case with m = 3, and @ is surrounded by
vertices V, {v1, U2. .,,,...,.. nz).
where {(T, )=) is dataset, f(x;) is the output of the inference algorithm of HLL for input wi. and l is loss function.
3.5 Partial Monotonicity
As already proven in [Gupta et al. 2016], the function f(p 9) computed by an interpolation algorithm satisfies monotonicity constraints if the values corresponding to the vertices of the hypercube satisfy the constraints. Therefore, show that have fp. u) S f (p, v) for any pair of vertices E Vm such that u is dominated by v. Note that this proposition holds for any vertex ordering a and is not restricted the vertex ordering determined by the Manhattan distance.
Proposition 3.4 In HIL., f(p, u. S u)) holds for any p Rd-m and any pair of vertices u. U. E Vm such that u is dominated by
Proof. Suppose that precedes v in the total ordering σ. By the definition of the minimal dominated set, there exists a sequence of vertices uo(= u), U1 .g..... ... u-(= v) such that uj-1 E Lay holds for j 1.2,....JJ Since uj-1 Lug.or we have f(p, 1z-1) 1(p, u,) I (p, Uz) by Eq. (I) (w,
Hence, we have [(p, u) = [(p uo) S f(,, u1) < ... S(,, uy)
Similarly, suppose that precedes u in the total ordering σ. By the definition of the minimal dominating set, there exists a sequence of vertices vo(= v), U1, ........ (-(= -= u) such that 2 e U holds for j == 1.2,...., Since V E Uny.or we have [p, U2-1) 2 u(y, 2 f(p, u/) by
Eq. ( Hence, we have f(p u) f(p vo) f(p,v1) ..... f(p,os) f(p,u).
3.6 Complexity Analysis
We show that the time complexity of the inference algorithm of HLL is ((( + m)) where T(ge) is the time complexity of computing (p, v) for an input (p. v) E Rd-m x RT First, the time complexity to compute [(p v) for all vertices v E Vm is (((T(g)) + m)] V., since it takes (gT(g)) + |Tr. + (U,=()) time to compute (p, v) for a single vertex v E Vn.. and we have [Te.o] JUr.al S m. Then, the time complexity for the interpolation algorithm is 0(2m) if we multilinear interpolation and (m log m) if we use simplex interpolation [Gupta et al. 2016). Since we have nm < [, the time complexity for the interpolation algorithm is smaller than that for the computation of (p, v). Hence, the overall time complexity remains O((T(g9) ) Vm)V If can assume that m a small integer (i.e., is constant), the computation time for inference can be written simply as ((T(g))). Note that, if the time complexity T(ge) ) is a linear function on d, the time complexity of the inference algorithm for HLL smaller than that for TL Lattice because the inference algorithm of TL Lattice takes O(2d) and Odd d) times if we use multilinear interpolation and simplex interpolation, respectively (Gupta et al. 2016).
The space complexity of HIL. is O(ggol + Vm|V if the vertex ordering a is determined by the Manhattan distance, where denote the size of the auxiliary neural network ge (p, v) by [gel, and the (m Vmml) term is the space required to store L and Uo.o for all E Vm. Note that, if we use an arbitrary vertex ordering a, we require O((Vml space to store Ly, and U..o for all E Vn, because we know only that [Twrol S [Vm and |Ur.r < /Vmm]- Since VV,'" 12 is much larger than mi V.nl. we recommend using the Manhattan distance to determine in HIL.

---

4 Experiments
In our experiments, we demonstrate that HLL did not sacrifice its prediction performance compared with TL Lattice. Note that HLL is designed to satisfy monotonicity constraints and is not designed to improve the prediction performance compared with TL Lattice. Therefore, the purpose of our experiments was not to show such improvements but to show that HLL did not sacrifice the prediction performance to satisfy the constraints. In addition, we demonstrate that we could use HLL for high-dimensional input vectors with a small m. All our experiments were conducted on a virtual machine with an Intel Xeon CPU (3.30 GHz) processor without any GPU and 64 GB of memory running Red Hat Enterprise Linux Server 7.6.
4.1 Neural Network Models
In our experiments, we compared these four neural network models.
MLP. A multi-layer perceptron model with three hidden layers. We used the ReLU acti- vation layer for the hidden layers and the sigmoid function for the output layer. Note that this neural network is not designed for partially monotone regression and cannot handle monotonicity constraints.
HLL. A neural network model consisting of single hierarchical lattice layer. We used an MLP model with three hidden layers for the auxiliary neural network go (p. v), and we used multilinear interpolation in the inference algorithm.
TL Lattice. A neural network model consisting of single TL Lattice layer Gupta el al. 2016). This layer uses O(k=) space, and therefore, can be used only for low-dimensional inputs (i.e., d is small integer). We used multilinear interpolation in the inference algorithm.
TL RTL. A neural network model consisting of a single RTL layer [Canini el al. [0166) from the TensorFlow Lattice library, which is random ensemble of tiny lattices (i.e.. TL Lattices). This layer consists of [d/r] tiny lattices, and each tiny lattice receives an r-dimensional input vector, where is an integer hyperparameter. Hence, this layer uses ((d/r)kk" space, which is significantly smaller than that of TL. Lattice when r < d. Note that theoretical analysis in (Canini et al. (016) shows that TL RTL with d/r tiny lattices is equivalent to the single d-dimensional high-dimensional inputs (i.e., is a large integer).
TL. Lattice. We used this model for datasets with
We used Python 3.8.12 and PyTorch 1.8.1 to implement MLP and HLL., and the implementation of HLL is available at httph:: http::///gtthub.... com/ IBM/ BtM//IMMM/ppm///e pmlayer. As for TL Lattice and TL RTL, we used TensorFlow 2.3.0 and TensorFlow Lattice 2.0. 10, which is the implementation provided by the authors [Canini el al. 2016 Gupta et al. 2016) and available as an open source library under Apache License 2.0 at https:/ /www//wwwc tensorflow org argllattice. The TensorFlow Lattice library also provides an implementation of the Kronecker-Factuos
lattice layer [Morioka et af.) (201]. which uses smaller amount of memory than TL Lattice. However, we did not include the experimental results for this layer because TL RTL significantly outperformed it for the datasets we used in terms of prediction performance.
Throughout our experiments, all problems were solved as regression problems, and we used mean squared error (MSE) for the evaluation metric. For the training of the neural network models, we used the Adam optimizer [Kingma and Ba 2015). In HIL, TL Lattice, and TL RTL, the number of vertices in each dimension of the hypercube was set to k by default. However, the numbers of the vertices in the dimensions that correspond to small categorical values were set to the numbers of the categories. For example, if the i-th dimension corresponds to binary feature, then set k; 2 instead of K; = k.
4.2 Real Datasets
We used 12 real datasets taken from the UCI Machine Learning Repository [Dua and Graft 2017 7 in
our experiments. We used six small datasets: Energy Efficiency [Tsanas and Xifara 2012]. QSAR Aquatic Toxicity [Cassotti el al. 2014], Concrete Compressive Strength YYeh] 1998 Contraceptive Method Choice [Lim et 2000), Abalone [Nash el af.) 1994), and Shill Bidding [Alzhhaani and Sadaouï 2020), and six large datasets: Online Shoppers Purchasing Intention (Sakar al. 2019
7

---

Table 1: Datasets obtained from UCI Machine Learning Repository.
Name
d
Monotone features
Energy Efficiency
768
8
X3, X5, X7
QSAR Aquatic Toxicity
908
6
MLOGP, SMI Dz(Z)
Concrete Compressive Strength
1 1030
8
Water
Contraceptive Method Choice
1 1473
12
Number of children ever born, wife age
Abalone
4 176
10
Shell, shucked, viscera, and whole weights
Shill Bidding
6321
9
Winning Ratio
Online Shoppers Purchasing
12330
74
Bounce ExitRates
Adult
30 162
88
Age
Online News Popularity
39 644
58
oLtokens content, o_tokens_tin tttlle
Facebook Comment Volume
40949
52
Column 37
Bank Marketing
41 188
63
cons.conf.idx
Blog Feedback
86864
276
Columns 51-53
Adult, Online News Popularity [Fernandes et al. 2015). Facebook Comment Volume (Singh et al. 2015), Bank Marketing [Moro al.] 2014). and Blog Feedback [Buza 2014). We confirmed that none of these datasets contained any personally identifiable data points nor any offensive content.
Table shows summary of these 12 datasets. Some of the datasets were already split into training and test data, and the others were unsplit (i.e., single dataset was given). Column n of this table shows the number of data points in the training dataset for the split datasets and the number of data points for the unsplit datasets. In our experiments, we removed all the data points that had missing feature values.
Column d of Table T shows the number of features after our feature encoding. We briefly explain our feature encoding here. We normalized all the continuous values (including the target values) to range between 0 and by using the min-max scaling algorithm (i.e. a simple linear transformation). We used label encoding to encode the ordinal categorical values. For example, we replaced the ordinal categorical values {low, med, high} with low = 0, med = 0.5, and high = 1 so that the replaced values were in the range between 0 and 1. We used one-hot encoding for the other categorical values. When we used TL Lattice and TL RTL, we renormalized the input feature vectors from [0, 1] to [0, k - 1]d, where ie is hyperparameter, to conform to the input specifications of these two models.
The last column of Table shows the monotone features we used in our experiments. The italic feature names represent monotonically decreasing features, and the others represent monotonically increasing features. We should choose monotone features by using domain knowledge for each dataset, but we used the SHAP values [Lundberg and Lee 2017 find candidates of monotone features for the datasets for which we had little domain knowledge We expected that there would be an almost monotonic relationship between the SHAP value and the i-th feature value (i.e., a(i) if the target value was monotonic on the i-th feature. In our experiments, we constructed an MLP model for each dataset and computed its SHAP values. Then, we checked for each feature if the SHAP values were monotonically increasing (or decreasing) with respect to the feature values by using Spearman's 's rank correlation coefficients. Note that such correlation between SHAP values and feature values does not mean monotone relationship, but we found this approach helpful to find candidates of monotone features for datasets with little domain knowledge.
4.3 Experimental Results
In our experiments, we split the data points into training, validation, and test data points. For the unsplit datasets, we divided the data points into training (60%), validation (20%), and test (20%). We used single random split to search for the best hyperparameters and five random splits to obtain the average prediction performance. For the datasets that were already split into training and test datasets, we further divided the data points in the training dataset into training (80%) and validation (20%) and kept the test dataset unchanged. We used a single random split search for the best hyperparameters and five random splits to obtain the average prediction performance without changing the test dataset.
8

---

0.020
Icaan Squared Etror
0.008
MIP
0.03
Hi
MIP
0.006
Squared
Squared
0.02
0.004
0.010
0.01
0.002
0.005
(a) Energy Efficiency
(b) QSAR Aquatic Toxicity
(c) Concrete Compressive Strength
0.012
0.03
Mean Squared Error
Tu Le
Tu Lattoon
0.3
0.010
Squared
Mean Squared
0.02
0.008
0.2
0.006
0.004
0.002
0.00
3
5
2
(d) Contraceptive Method Choice
(e) Abalone
(1) Shill Bidding
Figure 4: MSE (lower is better) of the neural network models with various & on small datasets.
Regarding the hyperparameters, chose the best hyperparameters for each combination of a neural network and dataset; the learning rate was chosen from {1, 10 1002 10-3 10-4, 10 10-5), the batch size was chosen from {8, 16, 32. 4096), ), the number of neurons in the hidden layers was chosen from {16, 32, 64, ..., 512] for MLP and HLL, and the hyperparameter r was chosen from {4, 5, 6, 7} for TL RTL. We used only the training and validation datasets determine the best hyperparameters, and we used Optuna [Akiba et al, 2019 to search for the best hyperparameters.
Results for Small Datasets. We compared the prediction performances of MLP, HLL, and TL Lattice by using the six small datasets (i.e., d was small). We trained these neural network models for 1000 epochs with certain early stopping criteria, and Figure44show the results for various k. The red dashed lines show the average of the MSFs for five runs of MLP, and we regard these values as baselines of the prediction performance. The blue and orange lines show the average MSEs together with their standard deviations for five runs of HLL and TL Lattice, respectively. These results showed that our HIL performed comparably to MLP for all six datasets. Somewhat surprisingly, we note that using finer-grained lattice (i.e., increasing k)) did not necessarily improve the MSEs. This phenomenon was already mentioned in Guppta et al. 2016) for TL Lattice, and we observed this phenomenon both for TL Lattice and HLL in our experiments. performance of HLL were also comparable with TL Lattice. Although TL Lattice showed worse MSEs than MLP and HLL for some datasets, these differences in prediction performance were supposed to come from two factors. One was the difference of the parameterization; HI.I. used the
4 shows that the prediction
hypercube of size k. whereas TI. Lattice used hypercube of size kd, The other difference of the neural network training algorithm; MLP and HIL used the standard stochastic gradient descent algorithm, whereas TL Lattice used the projected gradient descent algorithm. In addition, we compared the prediction performace of HLL and TL Lattice with the calibration layer, and the results are shown in Appendix A.4.
Results for Large Datasets. For the datasets with large d, we could not use TL Lattice due to its space complexity ((k4), so we used TL RTL instead. TL Lattice requires that at least 252 parameters be stored even if A 2 since d > 52, but we cannot store such large number of parameters in memory. In contrast, we could use HLL since m is small in the datasets we used.
We compared the prediction performance of MLP. HLL, and TL RTL by using the six large datasets. trained these neural network models for 100 epochs with certain early stopping criteria, and F shows the results for various k. The red dashed lines show the average MSEs for five runs of ML.., P. and we regard these values as baselines of the prediction performance. The blue and green lines show the average MSEs together with their standard deviations for five runs of HLL and TL

---

0.30
0.0004
Mean Squared Error
Mcan Squared Error
Mean Squared Error
0.20
0.25
0.0003
0.15
0.20
0.0002
0.10
0.10
0.05
0.05
0.0001
(a) Online Shoppers Purchasing
(b) Adult
(c) Online News Popularity
0.004
Mcan Squared Error
RTL
TLTT
0.06
TLm
Squared
0.125
0.100
0.002
0.075
0.001
0.050
0.02
0.025
3
(d) Facebook Comment Volume
(e) Bank Marketing
(1) Blog Feedback
Figure 5: MSE (lower is better) of the neural network models with various it large datasets.
RTL, respectively. Note that the result for TL RTL with = 5 on the Blog Feedback dataset was missing due memory limitation in computer environment. These results showed that HLL performed comparably to MLP and TL RTL regardless for the six large datasets. We think that HLL performed better than MLP for some datasets thanks to the regularization effects coming from the monotonicity constraints.
Related Work
There are some neural networks for partially monotone regression other than the TensorFlow Lattice library. One of the simplest approaches is to enforce non-negativity constraints neural network weights that are related to monotone inputs (c.g., [Daniels and Velikova 2010 You et al. 2017)) Another simple approach
use regularization Monteiro et at.) [202]] but a prediction model that uses this approach may not always satisfy the monotonicity constraints. More complex approaches are proposed in (Liu et al. 22000] Sivaraman
enforce monotonicity Guupt al. 2019
al. 2020). These approaches use mixed integer linear programming solver or a satisfiability modulo
theories (SMT) solver ensure the monotonicity of the neural network. The disadvantages of these
approaches are their computational complexity and the costs of using the solvers.
Conclusion
We proposed the hierarchical lattice layer (HLL) for partially monotone neural networks an extension TL Lattice (Gupta el al. 2016). We demonstrated that HLL resolved two problems with TL Lattice: the memory consumption and the requirement for the projected gradient descent algorithm with many constraints. In our experiments, we showed that HLL did not sacrifice prediction performances resolve these problems. Note that our experiments were limited comparison between single layers (i.c., HLL and TL Lattice). The prediction performance these models might be able to be improved by combining multiple layers (e.g.. You et al, 2017 any other techniques (e.g., changing feature encoding and increasing the number of layers in MLP) and the memory consumption might also be reduced by using smaller lattices, but note that the requirement for the projected gradient descent algorithm with many constraints remains for TL Lattice.
A limitation of our HIL is that it cannot be used for the applications with large If m is large, we combine small HILL layers to construct a large neural network as demonstrated [Canini
al. 2016; You et al. 2017 for TL Lattice.
10

---

References
Akiba, Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of KDD 2019, pages 2623-2631, 2019.
A. Alzahrani and S. Sadaoui. Clustering and labeling auction fraud data. Data Management, Analytics and Innovation, pages 269-283, 2020.
C. Bartley W. Liu, and M. Reynolds. A novel framework for constructing partially monotone rule ensembles. In Proceedings of ICDE 2018, 2018.
C. Bartley, W. Liu, and M. Reynolds. Enhanced random forest algorithms for partially monotone ordinal classification. Proceedings of AAAI-19, pages 3224-3231, 2019.
K. Buza, Feedback prediction for blogs. Data Analysis, Muchine Learning and Knowledge Discovery, pages 145-152, 2014.
K. Canini, A. Cotter, M. R. Gupta, M. Milani Fard, and J. Pfeifer. Fast and flexible monotonic functions with ensembles lattices. In Proceedings of NIPS 2016, pages 2927-2935, 2016.
J.-R. Cano, A. Gutiérrez, B. Krawezyk, M. Wožniak, and S. García. Monotonic classification: An overview algorithms, performance measures and data sets. Neurocomputing, 341:168-182, 2019.
M. Cassotti, Ballabio, Consonni, A. Mauri, I. V. Tetko, and R. Todeschini. Prediction acute aquatic toxicity towards Daphnia magna using GA-èNN method. Alternatives to Laboratory Animals, 42((1)311-4) 2014.
H. Daniels and M. Velikova. Monotone and partially monotone neural networks. IEEE Transactions Neural Networks, 21(6):906- -97, 2010.
D. Dua and C. Graff. UCI Machine Learning Repository, 2017.
C. Dugas, Y. Bengio, F. Bélisle, C. Nadeau, and R. Garcia. Incorporating second-order functional knowledge for better option pricing. In Proceedings of NIPS 2000, pages 472-478, 2000.
M. Milani Fard, K. Canini, A. Cotter, Pfeifer, and Gupta. Fast and flexible monotonic functions with ensembles of lattices. In Proceedingss of NIPS 2016, pages 2927-2935, 2016.
K. Fernandes, P. Vinagre, and P. Cortez. A proactive intelligent decision support system for predicting the popularity of online news. Proceedings of EPIA 2015, pages 535-546, 2015.
M. Gupta, A. Cotter, Pfeifer, K. Voevodski, K. Canini, Mangylov, Moczydlowski, and A. van Esbroeck. Monotonic calibrated interpolated look-up tables. Journal of Machine Learning Research, 17(1):370-3386,, 2016.
A. Gupta, N. Shukla, L. Marla, A. Kolbeinsson, and Yellepeddi. How to incorporate monotonicity in deep networks while preserving flexibility? In Proceedings of NeurIPS 2019 Workshop Machine Learning with Guarantees, 2019.
D. Kingma and Ba. Adam: A method for stochastic optimization. In Proceedings of ICLR 2015, 2015.
R. Koenker and B. Bassett, Jr. Regression quantiles. Econometrica, 46(1):33-50, 1978.
F. Lauer and G. Bloch. Incorporating prior knowledge in support vector regression. Machine Learning, 70:89-118, 2008.
T.-S. Lim, W.-.. Loh, and Y.-S. Shih. comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Machine Learning, 40:203-228, 2000.
X. Liu, X. Han, N. Zhang, and Q Liu. Certified monotonic neural networks. In Proceedings of NeurIPS 2020, pages 15427-15438, 2020.
M. Lundberg and S.-I. Lee. A unified approach to interpreting model predictions. Proceedings of NIPS 2017, pages 4768-4777, 2017.

---

J. Monteiro, M. O. Ahmed, H. Hajimirsadeghi, and G. Mori. Not too close and not too far: Enforcing monotonicity requires penalizing the right points. In Proceedings of XAI Debugging Workshop at NeurlPS 2021, 2021.
N. Morioka, E. Louidor, and W. T. Bakst. Monotonic Kronecker-factored lattice. In Proceedings of ICLR 2021, 2021.
S. Moro, P. Cortez, and P. Rita. A data-driven approach to predict the success bank telemarketing. Decision Support Systems, 62:22 21, 2014.
W. J. Nash, T. L. Sellers. S. R. Talbot, A. J. Cawthorn, and W. B. Fort. The population biology of Abalone (Haliotis species) in Tasmania. I. Blacklip Abalone (H. rubra) from the north coast and islands of bass strait. Technical report, Sea Fisheries Division, 1994.
Y. Park, D. Maddix, F.-X. Aubet, Kan, J. Gasthaus, and Y. Wang. Learning quantile functions without quantile crossing for distribution-fee time series forecasting. In Proceedings of AISTATS 2022, pages 81 27-81550, 2022.
R. Potharst and A. Feelders. Classification trees for problems with monotonicity constraints. ACM SIGKDD Explorations Newsletter, 4(1):1-10, 2002.
K. Ren, J. Qin, L. Zheng, Z. Yang, W. Zhang. L. Qiu, and Y. Yu. Deep recurrent survival analysis. In Proceedings of AAAI-19, pages 4798-4805, 2019.
D. Rindt, R. Hu, D. Steinsaltz, and D. Sejdinovic. Survival regression with proper scoring rules and monotonic neural networks. In Proceedings of AISTATS 2022, pages 1190-12000 1205, 2022.
C. O. Sakar, S. O. Polat, M. Katircioglu, and Y. Kastro. Real-time prediction of online shoppers' purchasing intention using multilayer perceptron and LSTM recurrent neural networks. Neural Computing and Applications, 31:669666908, 2019.
K. Singh, R. Kaur, and D. Kumar. Comment volume prediction using neural networks and decision trees. In Proceedings of UKSIM 2015, pages 15-20, 2015.
A. Sivaraman, G. Farnadi, T. Millstein, and G. Van den Broeck. Counterexample-guidee learning of monotonic neural networks. In Proceedings of NeurlPS 2020, pages 11936-1199 1948, 2020.
N. Tagasovska and D. Lopez-Paz. Single-mont model uncertainties for deep learning. In Proceedings of NeurlPS 2019, pages 6425, 2019.
A. Tsanas and A. Xifara. Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools. Energy and Buildings, 49:560-567, 2012.
S. Wang and M. Gupta. Deontological ethics by monotonicity shape constraints. In Proceedings of AISTATS 2020, pages 2043-2054, 2020.
P. Wang, Y. Li, and C. K. Reddy. Machine learning for survival analysis: A survey. ACM Computing Surveys, 31(6):1-36, 2019.
I.CC. Yeh. Modeling of strength of high performance concrete using artificial neural networks. Cement and Concrete Research, 18912):1777-1800 1998.
S. You, D. Ding, K. Canini, J. Pfeifer, and M. R. Gupta. Deep lattice networks and partial monotonic functions. In Proceedings of NIPS 20/7, pages 2985-2993, 2017.
Zheng, S. Yuan, and X. Wu. Safe: A neural survival analysis model for fraud carly detection. In Proceedings of A9, pages 1278-1288 2019.
F. Zhou, J. Wang, and X. Feng. Non-crossing quantile regression for deep reinforcement learning. In Proceedings of NeurlPS 2020, pages 15909-15919, 2020.
12


Reviews:
[{"review_id": "jXlChWc6BAJ", "reviewer": "Reviewer_hkrw", "summary": "This paper proposes a novel neural net layer that improves on the TensorFlow lattice layer by being able to take advantage of standard neural net training procedures (the existing lattice layer requires a special training procedure) and also being able to handle high-dimensional inputs (whereas the existing lattice layer would run into memory issues).", "questions": "Please address the points that I listed as weaknesses above.", "limitations": "Yes, the author(s) do briefly address the limitation of their approach (i.e., it doesn't handle large $m$), and I found their response to the question on potential negative societal impact in the checklist to be adequate.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I would say that overall, the technical advance is very incremental. For example, much of the same ideas are already used in survival analysis for estimating the monotonically decreasing survival probability function over time (e.g., Kvamme and Borgan 2021, although the same ideas in parameterizing the monotonic survival function appear in earlier papers, some of which are referenced within Kvamme and Borgan), where standard neural net training procedures suffice. One could perhaps view the proposed HLL method as a generalization of what is already done in survival analysis that does not need a specialized training procedure (although really I'd be surprised if the same ideas used by survival analysis researchers haven't already been used by researchers in other fields that need to enforce monotonicity). That said, I think a paper that neatly shows that there's a general idea that subsumes existing approaches is valuable.\n\nStrengths:\n- The key technical ideas are actually not that complicated although the presentation perhaps make them sound more complicated than they actually are.\n- The author(s) provide sufficient empirical evidence that the proposed hierarchical lattice layer (HLL) is better than the existing TensorFlow lattice layer in the case of partially monotone regression.\n\nWeaknesses:\n- I found the presentation at times to be more complicated than it needs to be. I would suggest adding a simple running example (could be very low-dimensional) throughout the paper that already clearly shows why the proposed method clearly works and we really don't need a specialized training procedure.\n- It would be helpful relating the proposed method to possibly special cases of how people have previously enforced monotonicity without resorting to specialized training procedures (I mentioned already that this is done in survival analysis but I'd assume that it's also done in other fields too).\n- Minor: Please appropriately switch between using \"citep\" and \"citet\" for references (currently, the paper basically uses the equivalent of \"citet\" too often so that the text suddenly switches to author names in a manner that is not grammatically correct and detracts from reading the paper). As a few examples:\n    - In line 19-20, I'd suggest you use \"citep\" so that the text says \"... if the other features are equal [Potharst and Feelders, 2002].\"\n    - In lines 25-28, if you use \"citep\", you would get: \"For example, they have a better regularization capability [Dugas et al., 2000; Fard et al., 2016; You et al., 2017] and better interpretability [Gupta et al., 2016], and they can be used for fair machine learning [Wang and Gupta, 2020].\"\n\nReference:\n\nKvamme H, Borgan Ø. Continuous and discrete-time survival prediction with neural networks. Lifetime Data Analysis. 2021 Oct;27(4):710-36.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ctbL-Y3T5h_", "reviewer": "Reviewer_utRT", "summary": "This paper introduces a new model architecture to represent partially monotonic functions that better scale to a larger number of inputs compared to current baselines. Monotonic functions are ubiquitous in real-world ML applications, with many examples included in the paper. The new technique is based on a reparametrization of lattice models (interpolated lookup tables). Instead of explicit parametrization of Gupta et al. [2016], the proposed method uses an auxiliary (non-monotonic) neutral network to linearly interpolate in the space of upper and lower bounds of each lattice site/vertex. The auxiliary model is thus implicitly defining the value of the vertices on the lattice while satisfying monotonicity constraints.\n\nNote: I reviewed an earlier version of this work. The new submission is improved in several aspects, including the presentation of the method and empirical studies.\n", "questions": "Questions:\n* The specific parametrization of the lattice described in the work results in a compound influence of the first output of the auxiliary model through the hierarchy of calculations, whereas the last output of the auxiliary model is only used in the calculation of the value of the last vertex. One would expect the gradients to flow very differently in the backward path for different outputs of the auxiliary model (and possibly issues vanishing gradient if D is large). Did the authors notice any significant effects related to this asymmetry? \n\nNotes and suggestions:\n* The citations in the paper have incorrect formatting. The authors should choose the correct citation latex macros that would include parentheses when inlined in text.\n* IMO, “Explicitly Parameterized Lattice” (EP Lattice) might be a better name for the baseline.\n* Projected gradient descent used for baseline lattice models is very much a standard training algorithm, with projections supported as first class citizens in several ML libraries (e.g. TF/Keras, PyTorch). The calculation of the projection is of course non-trivial and specific to the modeling. It would be better if the authors are more explicit about this distinction in their description of the baseline training algorithms.\n* Line 170: … hold for any ordering sigma: … any ordering sigma consistent with the partial ordering defined by the monotonicity constraints.\n* Please check plots in a black and white print. Some line formattings are hard to tell apart at the current resolution.\n", "limitations": "IMO the authors have sufficiently discussed the limitations of their work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The lattice reparametrization idea presented in the paper is very interesting, as it makes it possible to model monotonic functions via a generic non-monotonic auxiliary function. This makes it possible to use generic off-the-shelf models on non-monotonic features (e.g. a CNN applied to an input image), while making sure the model is monotonic w.r.t. other features (e.g. quantile value in a quantile regression task).\n* The proposed implicit representation of the lattice parameters forces monotonicity by construction. This avoids the need for the expensive parameter projection step during training, which is needed for explicitly parameterized lattice models.\n* The method should be easy to implement in any standard ML framework.\n\n\nWeaknesses:\n* The calculation of vertex values in the proposed method adds extra compute compared to explicit representations when the number of features in each lattice is small. If simplex interpolation is used (Gupta et al. [2016]), the extra computation needed for the proposed method will change the overall evaluation time from O(D logD) to O(K^D) for D features used in a lattice with K vertices along each dimension. This is not a big concern since the main use case argued in the paper is when only a small number of features are monotonic, and the proposed method only needs to apply lattice interpolation to the monotonic features. However, I believe simplex interpolation could have been studied in the empirical studies at least for the baseline lattice models.\n* Unlike the studies of baseline lattice models (Gupta et al. [2016], Fard et al. [2016], You et al. [2017], Wang and Gupta [2020]), empirical studies in this paper do not use a trainable calibration layer before passing the input to the lattice. Since the vertices of the lattices are fixed at equidistance locations, standard linear normalization is likely suboptimal for features with non-uniform distributions. Increasing the lattice resolution (as the paper concurs with previous work) will not solve the issue since it is adding parameters uniformly across the input domain where they might not be needed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_kzzcO2yLQK", "reviewer": "Reviewer_GeFP", "summary": "This work proposes a novel neural network layer to model partial monotonicity in regression analysis. To enforce monotonicity w.r.t. a subset of input features, the authors define a hierarchical lattice structure over a unit hypercube spanning the space of monotonic features. The authors then employ an ordered set (based on the coordinates) of vertices of the lattice structure to define upper and lower bounds of the target regression function for a given set of input features. Given the upper and lower bounds corresponding to an input, the authors propose to approximate the target function by using interpolation between the bounds. They learn an auxiliary neural network with sigmoid head to output the interpolation weight given input features. The authors perform complexity analysis of the proposed method and in the numerical experiments, they demonstrate that their method is competitive or better than compared methodologies, which also includes a generic multi-layer perceptron model.", "questions": "Have you analysed if for instance in your experiments a generic model like an MLP also learns (partially) monotonic relationships between input and target variables, even though it is not enforced by design?", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Compared to previous approaches, the proposed method reduces the spatial complexity to the dimensionality of monotonic features\n- The lattice layer can be trained using standard optimisation methods such as back propagation.\n- Numerical results are competitive or better than compared approaches\n\nWeaknesses:\n- The approach requires prior knowledge of monotonic features and the layer cannot yield optimal results if monotonicity assumptions do not hold.\n- Time and space complexity of the proposed method increases exponentially w.r.t. the lattice granularity and the number of monotonic features, hence it's application to a large number of monotonic features can become infeasible and the performance may suffer due to a coarse granularity of the lattice structure.\n- Although the method is competitive to generic methods like MLP, it can be simply down to the choice of architecture used in the comparison.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept