Constrained Monotonic Neural Networks
Authors: Davor Runje
Sharath M Shankaranarayana
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Neural Network
Monotonicity
Deep Learning

Abstract: Deep neural networks are becoming increasingly popular in approximating arbitrary functions from noisy data.  But wider adoption is being hindered by the need to explain such models and to impose additional constraints on them. Monotonicity constraint is one of the most requested properties in real-world scenarios and is the focus of this paper.  
One of the oldest ways to construct a monotonic fully connected neural network is to constrain its weights to be non-negative while employing a monotonic activation function. Unfortunately, this construction does not work with popular non-saturated activation functions such as ReLU, ELU, SELU etc, as it can only approximate convex functions. We show this shortcoming can be fixed by employing the original activation function for a part of the neurons in the layer, and employing its point reflection for the other part. Our experiments show this approach of building monotonic deep neural networks have matching or better accuracy when compared to other state-of-the-art methods such as deep lattice networks or monotonic networks obtained by heuristic regularization. This method is the simplest one in the sense of having the least number of parameters, not requiring any modifications to the learning procedure or post-learning steps.
Finally, we give a proof it can approximate any continuous monotone function on a compact subset of $\mathbb{R}^n$.

Constrained Monotonic Neural Networks
Anonymous Author(s)
Affiliation
Address
email
Abstract
Deep neural networks are becoming increasingly popular in approximating arbitrary functions from noisy data. But wider adoption being hindered by the need explain such models and to impose additional constraints them. Monotonicity constraint one of the most requested properties in real-world scenarios and is the focus of this paper. One of the oldest ways to construct monotonic fully connected neural network is to constrain its weights to be non-negative while employing a monotonic activation function. Unfortunately, this construction does not work with popular non- saturatedd activation functions such as ReLU, ELU, SELU etc, be
2
6
it can only approximate convex functions. We show this shortcoming fixed by employing the original activation function for a part the neurons the layer, and employing its point reflection for the other part. Our experiments show this approach of building monotonic deep neural networks have matching better accuracy when compared to other state-of-the-art methods such deep lattice networks or monotonic networks obtained by heuristic regularization. This method the simplest one in the sense of having the least number of parameters, not requiring any modifications to the learning procedure or post-learning steps. Finally, we give proof can approximate any continuous monotone function on a compact subset of R"
10
12
14
15
18
17
18
Introduction
20
Deep Learning has witnessed widespread adoption in many critical real-world domains such as finance, healthcare, etc [21]. Predictive models built using deep neural networks have been shown to have high accuracy, Incorporating prior knowledge such as monotonicity in trained models help in improving the performance and generalization ability of the trained models [25] 9.. The introduction of structural biases such as monotonicity makes models also more data-efficient, enabling a leap in predictive power smaller datasets [42] Apart from the requirements of having models with high accuracy, there is also need for transparency and interpretability, and monotonicity helps in partially achieving the above requirements (14). Due to legal, ethical and/or safety concerns, monotonicity of predictive models with respect to some input or all the inputs required in numerous domains such as financial (house pricing, credit scoring, insurance risk), healthcare (medical diagnosis, patient medication) and legal (criminal sentencing) to list just few. All other things being equal, a larger house should be deemed more valuable, bank's 's clients with higher income should be eligible for larger loan [36], and an offender with a longer crime history should be predicted as more likely to commit another crime 33.. etc. model without such monotonie property would not, and certainly
21
22
23
24
25
26
28
30
32
34 should not, be trusted by society to provide a basis for such important decisions. However, the monotonicity of deep learning models is not guaranteed property when trained monotonic
36 data, let alone when training on noisy data typically used in practice.
Although monotonicity is important and increasingly often even required property, there is no 38 simple or easy method to enforce It has been an active area research and the existing methods on the subject can be broadly categorized into two types:
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

40
1. Monotonic architectures by construction: This line of research focuses on neural architec- tures guaranteeing monotonicity by construction
41
42
2. Monotonicity by regularization: This line of research focuses on enforeing monotonicity in neural networks during training by employing a modified loss function or heuristic regularization term (38 13
43
We give more detailed account of the existing methods in the next section.
46
The simplest method to achieve monotonicity by construction is to constrain the weights of the fully connected neural network to have only non-negative (for non-decreasing variables) only non- positive values (for non-ascending) variables when used conjunction with a monotonic activation function, technique known for almost 30 years 22. However, this method does not work well in practice. When used in conjunction with saturated (bounded) activation functions such as the sigmoid and hyperbolic tangent, these models are difficult train, i.e. they do not converge to good solution. On the other hand, when used with non- non-saturated (unbounded) convex activation functions such as ReLU [26]. the resulting models are always convex [2] severcly limiting the applicability of the method in practice.
48
50
52
54
55
The main contribution of this paper is simple modification of the method above which, in conjunction with non- not-saturaeed activation functions, capable of approximating non-convex functions as well: both concave and convex monotone activation functions are used in neural network with constrained weights, it regains the ability to approximate monotone continuous functions that are either convex, concave or nothing of the above.
56
58
There are several possible ways to satisfy this condition, with the simplest one using both an activation function أ and its point reflection with respect to point (0, 0) defined as:
p(z) = -
60
Assuming the original activation function I both monotonic and convex, the properties holding for ReLU, ELU, SELU, and Softplus, the proposed modification uses the original activation function on a part of the neurons in the network and its point reflection á on the rest of the neurons.
61
62
69
The resulting model is guaranteed to be monotonic, can be used in conjunction with any convex monotonic non-saturated activation function, doesn't have any additional parameters compared to non-monotonic fully-connected network for the same task, and can be trained without any additional requirements the learning procedure. Experimental results show it exceeding the performance of all other state-of-the-a art methods, all while being both simpler (in the number of parameters) and easier to train.
64
65
86
67
68
89
Our contributions can be summarized as follows:
1. We propose modification existing constricted neural network layer enabling it to model non-convex functions when used with non-saturated monotone convex activation functions such as ReLU, ELU, SELU, and alike.
71
72
2. We perform comparisons with other recent works and show that our proposed novel neural network block can yield comparable and in some cases better results than the previous state-of-the and with significantly fewer parameters.
74
76
3. We prove the universal approximation property for the ReLU activation function, showing that the proposed architecture can approximate any monotone continuous function on compact subset of T....
77
78
2
Related work
80
Before dwelling on the methods employed to build monotonic models, we give an overview of activation functions as they form crucial part of our work.
81
2.1 Activation functions
85 Right from its inception in perceptron 31 non-linear activation functions have historically been one of the most important components of neural networks. Previously, the saturated functions such

---

85
as the sigmoid [33], the hyperbolic tangent [27,, and its variants were the most common choice of activation functions. Currently, one of the most important factors for state-of-the-ee results accomplished by modern neural networks is the use of non-saturated activation functions. The use of Rectified Linear Unit (Rel.) [26 12] as activation function was instrumental achieving good performance in newer architectures. The ReL.U has since become defacto choice of activation in most practical implementations and continues to be widely used because of its advantages such as simple computation, representational sparsity, and linearity. Later, number of activation functions were proposed to deal with solving problems of dead neurons and aid in faster convergence 23
86
87
86
89
90
91
92
93
45
30 (0)].
94
The idea of using both the original activation function and its point reflection in the same layer has been proposed in [35] where both outputs of ReLU and the negative value of its point reflection were used in the construction of concatenated ReLU (CRel.U) activation function. The proposed modification outputs two values instead of one and therefore increases the number of parameters. In the authors propose negative concatenated ReLU (NCReLUU) flip the sign and use the point reflection directly. Notice that the proposed architectural change could be applied to other non- saturated, monotonic activation functions as well, but with an unknown impact on their performance.
95
96
97
96
44.
99
100
101
In 10 the authors propose bipolar ReLU which consists of using ReLU on the half of the neurons in the layer and the point reflection of ReLU on the other half. The same construction could be used with other ReLU family activation functions as well. However, the focus of their work was to alleviate the need of normalizing layers and improve the performance of deep vanilla recurrent neural networks (RNNs) by using bipolar ReI.U.
102
103
104
105
106
Both NCReLU and bipolar ReLU could have been used in the construction of a constrained neural network capable of representing non-convex functions, but to the best of our knowledge, they have not been so far.
107
108
2.2
Monotonicity by construction
110
Apart from the approaches mentioned in the introduction (section I. another approach to building monotonic neural architecture is Min-Max networks where monotonic linear embedding and max- min-pooling are used (37]. In [8] authors generalized this approach to handle functions that are partially monotonic and proved that the resulting networks have the universal approximation property.
111
112
113
114
However, such networks are very difficult to train and not used in practice. Their construction does not allow for replacement with other activation functions.
115
116
Deep lattice networks (DLN) 43 use a combination of linear calibrators and lattices 24 for learning monotonic functions. This is the most widely used method in practice today, but not without its limits. Lattices are structurally rigid thereby restricting the hypothesis space significantly. Also, DLN requires very large number of parameters to obtain good performance.
118
120
Given a model with a convex output function, it is possible to use backpropagation (34 to make a monotonic model by computing the derivation of the output function. One simple way to construct convex function is to use an unsaturated monotonic activation function in a fully connected layer as mentioned above. but we could also use more elaborate architecture such as the input convex neural networks (.] Although possible, these kinds of constructions are computationally more complex than the simple solution proposed here.
122
124
126
2.3
Monotonicity by regularization
127
In the second category, the research works focus on enforcing monotonicity during the training process by modifying the loss function or by adding a regularization term.
129
In 3 the authors propose modified loss function that penalizes the non-monotonicity of the model. The algorithm models the input distribution as a joint Gaussian estimated from the training
130
131
data and samples random pairs of monotonic points that are added to the training data. In [33, the authors propose a point-wise loss function that acts soft monotonicity constraint. These methods are straightforward to implement and can be used with any neural network architecture. However, these methods do not guarantee the monotonicity of the trained model.
133
3

---

Undrabrated Neuu
Condiniied Rell
Creraneed bipdar Rdd)
1.00
(a) Unconstrained ReLU
(h) Constrained ReL.U
(c) Constrained bipolar ReLU
Figure 1: Approximations of the cubic function f(x) = x
135
Recently, there is an increasing number of proposed methods to certify or verify monotonicity obtained by regularization methods. In [22). the authors propose an optimization-base technique for mathematically verifying, or rejecting, the monotonicity of an arbitrary piece-wise linear (e.g.. ReLU) neural network. The method consists of transforming the monotonicity verification problem
136
137
138
139
into a mixed integer linear programming (MILP) problem that can be solved using an off-the-shelf
140
MILP solver.
141
In 39,, the authors propose an approach that finds counterexamples (defined as the pair of points where the monotonicity constraint is violated) by employing satisfiability modulo theories (SMT) solver [3] To satisfy the monotonicity constraints, these counterexamples are included in the training data with adjustments to their target values to enforce the next iterations of the model to be monotonic.
143
145
Both methods [22 39] have been shown to support ReLU as the activation function only and there is no obvious way how to extend them to other activation functions. More precisely, they rely on wise linearity of ReLU to work, the property not satisfied by other variants such as ELU, SELU, GELU, etc. Last but not least, the procedure for certifying/verifyye using MILP or SMT solvers is computationally very costly. These approaches also require multiple reruns or iterations to arrive at certified/verifeed monotonic networks.
147
149
3
Constrained neural networks
151
152
Most of the commonly used activation functions such as ReLU, ELU, SELU, etc. are monotonically increasing, convex, non-polynomial functions. When used in fully-connected, feed-forward neural network with at least one hidden layer and with unconstrained weights, they can approximate any continuous function on compact subset. The simplest way to construct monotonic neural network is to constrain its weights when used in conjunction with a monotone activation function. However, when the activation function is convex well, the constrained neural network is not able to approximate non-convex functions.
154
156
158
159
To better illustrate this, and to propose simple solution in this particular example, we refer the readers Figure where the goal to approximate a simple cubic function I using neural network with single hidden layer with either or 32 neurons and with ReLU activation. A cubic function is 1, 0J and convex in the interval
160
162
apt for our illustration since is concave in the considered interval
[0,
Fig.
la
An unconstrained ReLU network with neurons approximate both concave and convex segments of the cubic function using at most n + piecewise linear segments. Increasing the number of neurons will provide a better fit with the function being approximated. Notice that even though the cubic function is monotone, there is no guarantee that the trained model will be monotone well.
165
187
189
Fig.
1b
If we constrain the weights of the network to be non non-negatn negative while still employing ReLU activation, the resulting model monotone and convex. We can no longer approximate non-convex segments such as the cubic function on [-1,0] 0] in the figure. and increasing the number of neurons from 2 to 32 does not yield any significant improvement in the approximation.
170
172

---

P(.)
X
W
P()
b
Figure 2: Proposed Monotonic Dense Unit or Constrained Monotone Fully Connected Layer
Fig.
Ic
The proposed solution both convex and concave activation functions in the hidden layer, in this case, bipolar ReLU, to gain the ability to model non-convex, monotone continuous functions. Notice that increasing the number of neurons increases the number of piecewise linear segments to approximate the cubic function. The resulting network is monotone by construction even when trained on noisy data.
175
178
177
178
179
The schematic block diagram of our proposed solution (which we refer to as Constrained Monotone Fully Connected Layer or Monotonic Dense Unit interchangeably) is shown in the figure Fig. 2 The individual components of the proposed solution are defined and described in the subsequent subsection.
180
182
189
3.1
Constrained monotone fully connected layer
0f
A function is partially monotone with respect to its parameter if Ox; (x1, ................................ zn,) is non non-negative non-negative for all ri,-.., F... A set S R is called compact if every sequence in S has subsequence that converges to point in S. One can easily show that closed intervals [a, b] are compact, and compact can be thought of generalizations of such closed bounded intervals.
186
187
188
Our construction is preconditioned on priori knowledge of (partial) monotonicity of a multivariate, multidimensional function f. Let K + R. be defined on compact segment K R... Then define its n-dimensional monotonicity indicator vector element wise as follows:
190
0f(x))
2 for each j E {1,...,m m]
0r: if
4
f((x)
0 for each j E {1....,
dr: otherwise
Given an (n x -dddiemnsionaal matrix W and n-dimensional monotonicity indicator vector t, we define the operation |t assigning an (n m) m)ddienniinaal matrix W' W to W as follows:
192
waysl
if t, = 1
w
if t; - otherwise
(2)
195 Definition (Constrained linear layer). Let Ruxm E R., x R. and E Ra. The output h E Th of the constrained linear layer with monotonicity indicator vector t, weights W, biases b
195 and imput is:
h
W/ b
3
5

---

196
Lemma 1. For each E {I.....m) n. and fl...... .......,, m. we have:
Jh1
197
ift; = 1, then
2 0, and
dz;
Dhi
198
ift; = -1, then
< 0.
Oz;
199
Let ý be a monotonically increasing convex function such as ReLU, ELU, etc. Then its point reflection I around the origin is:
200
x(x) = -)(=x)
201
Notice that is monotonically increasing concave function.
Definition 2 (Combined activation function). Given monotonically increasing convex function
203
its point reflection P and m-dimensioona activation selector vector s [0, 1]m,, the output of the combined activation function p² is weighted sum of and p::
p"(h) @ p(h) + (1 - @ p(h)
(5)
205
where @ the element-wise (Hadamard) product and 1 is m m-ddimensional vector with all elements equal 1.
θyj 0hg
207
Lemma 2. Let = (h)). Then for each j (1....,m)) we have
z 0. Moreover
208
ifs 1, then P is convex; and
ifs= then oj is concave.
Definition 3 (Monotone constrained fully connected layer). Let and m be positive natural numbers, a monotonically increasing convex function. an n--dimensional monotonicity selector vector and 212 s an m-dimensional activation selector vector. Then the monotone constrained fully connected layer with m neurons a upple (ø, t, s), denoted MFFrsst-
211
213
Moreover, let weights W be an (n x m) mi)-dimensional matrix, biases b an m-dimensional vector and 215 input x an n-dimensional vector. The output function of the monotone constrained fully connected layer, denoted MFC W p.s,t (x). assigns an m nn-dimensional vector y to (W, b, x) as follows:
214
216
MMFCWW. B.st = (W,, x+
(6)
Notice that M.F.A.s.t determines architecture of the layer, while MFCW W.b p.a,t. is fully instantiated layer with all of its internal parameters W and b defining function from input to output values.
219 From Lemmal [ and Soudddiree directly follows:
Corollary 3. For each E {1....,n) n} and each m, we have:
Oy
ift, = 1, then > 0;
ifs 1, then y, is convex;
Dx.
0yj
if4 1, then 0z.
< 0;
if's 0, then y, is concave.
222 On the layer level, we can control both monotonicity, convexity and concavity of the output with respect to chosen input variables. The following section discuss how we can use such layers to build practical neural networks with the same properties.
223
224
3.2
Composing monotonic constrained dense layers
226
As mentioned before, the main advantage our proposed monotonic dense unit is its simplicity. We build deep neural nets with different architectures by plugging in monotonic dense blocks. The figures
228
two examples neural architectures that can be built using the proposed monotonic dense block.
229
230
The first example shown the figure Fig 3 corresponds to the standard ML.P type of neural network architecture used in general, where each of the input features is concatenated to form single input

---

Monstmnk
Dano:
Maroomii: Dame: Block
Fizal Mumotooiic
Final
y
Axtivation
A Almbluiity
6
4
4-1
Figure 3: Neural architecture type 1
232
feature vector x and fed into the network, with the only difference being that instead of standard fully connected or dense layers, we employ monotonic dense units thorughout. For the first (or input layer) layer, the indicator vector t, is used to identify the monotonicity property of the input feature with respect the output. Specifically, t set 1 for those components the input feature vector that are monotonically increasing and is set to -1 for those components that are monotonically decreasing and set to if the feature is non-monotonic. For the subsequent hidden layers, monotonic dense units with the indicator vector always being set to 1 are used in order to preserve monotonicity. Finally, depending on whether the problem hand regression problem or classification problem (or even multi-task problem), appropriate activation function (such as linear activation or sigmoid or softmax) to obtain the final output.
233
234
235
238
237
239
241
242
Figure Fig. shows another example of a neural network architecture that can be built employing proposed monotonic dense blocks. The difference when compared to the architecture described above lies in the way input features are fed into the hidden layers of neural network architecture. Instead of concatenating the features directly, this architecture provides flexibility to employ any form of complex feature extractors for the non non-monotonic features and use the extracted feature vectors as inputs. Another difference is that each monotonic input is passed through separate monotonic dense units. This provides an advantage since depending on whether the input is completely concave or convex or both, we can adjust the activation selection vector s appropriately along with an appropriate value for the indicator vector t. Thus, each of the monotonic input features has separate monotonic dense layer associated with Thus as the major difference to the above-mentioned architecture, we concatenate the feature vectors instead of concatenating the inputs directly. The subsequent parts of the network are similar to the architecture described above wherein for the rest of the hidden monotonic dense units, the indicator vector is always set to 1 to preserve monotonicity.
243
244
245
247
249
251
253
255
3.3
Universal approximation
The classical Universal Approximation Theorem 7 17 28 states that any continuous function closed interval can approximated with a feed-forward neural network with one hidden layer if and only if its activation function is nonpolynomial. In [1), authors prove the approximation property holds for arbitrary deep neural networks with bounded number neurons in each layer holds if the activation function is nonaffine and differential at at least one point.
257
259
In authors shows the universal approximation property for constrained multivariate neural networks using sigmoid as the activation functions: any multivariate continuous monotone function on a compact subset of IR. can be approximated with a constrained neural network with the sigmoid activation function of at most & layers (Theorem 3.1). However, the proof of the theorem uses only
262
263
284
265
the fact that the Heavyside function defined as
{
if 0 otherwise
H(x)
can be approximated with the sigmoid function on a closed interval. is important to note that this 286
provides an upper bound on the number of monotonic layers in the general case and the number of
288
layers for a particular function should be determined experimentally in practice (in our experiments in Section 4 got the best performing networks using 2-3 layers).
7

---

S/,
Dirne
Bick Aiitt
Kimitee
Beck
Cancrada
Menkeri Biack (Usilbe
Fied Moncioii Daess Hloes Alliiszilly
Thal
12
Nea- lepul
Netwalk
Figure 4: Neural architecture type 2
We provide simple condition for arbitrary activation function for the universal approximation property to hold:
271
272
Theorem Let be a monotone activation function. If constrained neural network using with single hidden layer can approximate the Heavyside function on closed interval, then any multivariate continuous monotone function on a compact subset of R can be approximated with a constrained neural network of most k layers using activation functions p.
274
276
Since we can approximate any continuous monotonic function with monotonic piecewise linear segments, we have:
277
Corollary 5. A constrained neural network of at most layers using ReLU the activation function can approximate any multivariate continuous monotone function on a compact subset of R
279
280
The same property can be shown using the same way for activation functions other than ReLUU.
Experiments
281
282
In order analyze the practical utility of the proposed method, we experiment with various datasets and compare them with the recent state-of-the of-the-art. For the first set of experiments, we use the datasets employed by authors in [22] and use the exact train and test split for proper comparison. We perform experiments on 3 datasets: COMPAS [18. which is classification dataset with 13 features of which are monotonic; Blog Feedback Regression 4 which is a regression dataset with 276 features of which 8 are monotonic; Loan Defaulte which is classification dataset with 28 features of which are monotonic. The dataset contains half million data points. For comparison with other methods, compare with Certified monotonic networks (Certified) [22 and other methods described in it.
284
286
288
290
For the second experiments, use datasets: Auto MPG (which is regression dataset with monotonic features) and Heart Disease (which is a classification dataset with 2 monotonic features) as employed the work [39] and once again use the exact train and test split for proper comparison. We compare with the method COMET described in [39 along with Min-Max Ner [8] and Deep
292
294
Lattice Network (DLN) 43 as described in
285
For the classification tasks, we use cross-entropy and for the regression tasks, we use mean-squared- loss functions. We employ gridsearch to find the optimal number of neurons, network depth or layers, batch size, activation function and the number of epochs. For training, we first find the optimal learning rate using learning rate finder 440 and then train with one cycle policy [41]
297
298
299
4.1
Results
The results on the dataset above are summarized in Tables 2 It shows that our method tends either match or surpass the other methods terms of test accuracy for classification tasks and root mean
301
http://www..agggeeommm/aaaaaaaaa loandaa-aa

---

302
squared eтror (RMSE) for regression tasks. For each of the datasets, we run the experiments ten times after finding the optimal hyperparameters and report the mean and standard deviation of the best five results. Experiment results show that networks learned by our method can achieve better results with
303
305
fewer parameters, than the best-known algorithms for monotonic neural networks, such as Min-Max Network [8] and Deep Lattice Network 43 It should be noted that the recent state-of-the-art works-
306
307
Certified [22] and COMET [39] require multiple runs in order to even satisfy monotonic constraints whereas monotonicity is guaranteed by simply employing the proposed monotonic dense units.
309
The most important advantage of our solution is simplicity and computational complexity. Our models have slightly better performance (accuracy or RMSE) on all datasets we tested them on, but it
310
311
important to note they have significantly fewer parameters and the simplest training procedure. As
312
such, they reduce the carbon footprint when used in data centers and also aid in the easier adoption of edge computing applications.
313
COMPAS
18
Blog Feedback 4
Loan Defaulter
Method
Parameters
Test Acc f
Parameters
RMSE +
Parameters
Test Acc f
Isotonic
N.A.
67.6%
N.A.
0.203
N.A.
62.1%
XGBoost 5
N.A.
68.5% + 0.1 %
N.A.
0.176 + 0.005
N.A.
63.7% :: 0.
Crystal 24
25840
66.3%
0.1%
15840
0.164 = 0.002
16940
65.0% = 0.1%
DLN 4R
31403
67.9% 0.3%
27903
0.161 + 0.001
29949
65.1% = 0.2%
Min-Max Net
42000
67.8% 0.1% %
27700
0.163 + 0.001
29000
64.9% = 0.1%
Non-Neg-DNN
23112
67.3% + 0.9%
8492
0.168 + 0.001
8502
65.1% = 0.1%
Certified 22
23112
68.8% + 0.2%
8492
0.158 + 0.001
8502
65.2% = 0.1%
Ours
101
68.9% + 0.5%
1101
0.156 + 0.001
177
65.3% + 0.001 %
Table Comparison our method with other methods described in 22
Method
Auto MPG
Heart Disease
RMSE
Test Acc t
Min-Max Net g
10.14 + 1.54
0.75 0.04
DLN 4
13.34 + 2.42
0,86 + 0.02
COMET 39
8.81 + 1.81
0.86 + 0.03
Ours
3.04 1 0.13
0.86 0.02
Table 2: Comparison of our method with other methods described in 39
5
Conclusion
315
In this paper, we proposed simple and elegant solution to build constrained monotonic networks which can approximate convex as well as concave functions. Specifically, we introduced constrained monotone fully connected layer which can be used as a drop-in replacement for fully connected layer to enforce monotonicity. We then employed our constrained monotone fully connected layer to build neural network models and showed that we can achieve either similar or better results to the recent state-of-the-art ([39 22) in addition to the well-known works such Min-Max networks
317
319
321
8 and DLNs 43). However, the main advantage of the proposed solution not higher accuracy but its computational and memory complexity: we use orders of magnitude fewer parameters and computation which makes the resulting neural networks more energy efficient.
323
324
One drawback of our proposed method is that we are restricted by the choice of activation functions ie., we can only employ monotonic activation functions. the future, we would like to build simple monotonic counterparts for other standard neural layers such as convolutional neural networks, recurrent neural networks and their variants, and attention models. Last but not least, we proved such networks can approximate any multivariate monotonic function.
325
327

---

References
329
330
[1] Brandon Amos, Lei Xu, and J. Zico Kolter. Input convex neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 146-155, PMLR, 06-11 Aug 2017.
332
334
[2] Norman P Archer and Shouhong Wang. Application of the back propagation neural network algorithm with monotonicity constraints for two-group classification problems. Decision Sciences, 24(1):60-75, 1993.
335
337
[3] Clark Barrett and Cesare Tinelli. Satisfiability modulo theories. In Handbook of model checking, pages 305-343. Springer, 2018.
338
339
[4] Krisztian Buza. Feedback prediction for blogs. In Data analysis, machine learning and knowledge discovery, pages 145. Springer, 2014.
340
341
[5] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16 pages 785-799, New York, NY, USA, 2016. ACM.
342
343
344
[6] Bjor--Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter, Fast and accurate deep network learning by exponential linear units (ELUs). In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
345
348
347
348
[7] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems (MCSS),
349
December 1989.
350
[8] Hennie Daniels and Marina Velikova. Monotone and partially monotone neural networks. IEEE Transactions on Neural Networks, 21(6)990--917, 2010.
351
352
[9] Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incor- porating second-order functional knowledge for better option pricing. Advances in neural information processing systems, 13, 2000.
353
354
355
[10] Lars Hiller Eidnes and Arild Nøkland. Shifting mean activation towards zero with bipolar activation functions. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 2018, Workshop Track Proceedings, 2018
356
357
John H Gennari, Pat Langley, and Doug Fisher. Models of incremental concept formation. Artificial intelligence, 40(1-3):11-61, 1989.
359
360
[12] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 315-323. JMLR Workshop and Conference Proceedings. 2011.
362
363
[13] Akhil Gupta. Naman Shukla, Lavanya Marla, Arinbjöm Kolbeinsson, and Kartik Yellepeddi. How to incorporate monotonicity in deep networks while preserving flexibility? arXiv preprint arXiv: 10662, 2019.
385
366
[14] Maya Gupta, Andrew Cotter, Jan Pfeifer, Konstantin Voevodski, Kevin Canini, Alexander Mangylov, Wojeiech Moczydlowski, and Alexander Van Esbroeck. Monotonic calibrated interpolated look-up tables. The Journal of Machine Learning Research, 17(1):370038866, 2016.
368
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-leve performance on imagenet classification. In Proceedings of the IEEE international conference computer vision, pages 1026-1034, 2015.
371
[16] Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with Gaussian error linear units. CoRR, abs/1606.0844 2016.
374
375
[17] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251257,, Mar 1991.
10

---

377
[18] S. Mattu J. Angwin. J. Larson and L. Kirchner. Machine bias: There's ' software used across the country to predict future criminals. and it's biased against blacks. ProPublica, 2016.
378
379
[19] P Kidger and T Lyons. Universal approximation with deep narrow networks. In Proceedings of the 33rd Annual Conference on Learning Theory (COLT 2020), pages 2306-2327, 2020.
380
381
[00] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing
382
neural networks. Advances in neural information processing systems, 30, 2017.
383
[21]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521((553):436 444,
May 2015.
385
[22]
Xingchao Liu, Xing Han, Na Zhang, and Qiang Liu. Certified monotonic neural networks. Advances in Neural Information Processing Systems, 33:15427-15438, 2020.
387
[23] Andrew L Maas, Awni Y Hannun, Andrew Ng. et al. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop Deep Learning for Audio, Speech and Language Processing. Citeseer, 2013.
389
390
[24] Mahdi Milani Fard, Kevin Canini, Andrew Cotter, Jan Pfeifer, and Maya Gupta. Fast and flexible monotonic functions with ensembles of lattices. Advances in neural information processing systems, 29, 2016.
391
393
[25]
Tom M. Mitchell. The need for biases in learning generalizations. Technical report, Department of Computer Science, Laboratory for Computer Science Research, Rutgers Univ. New Jersey, 1980.
394
395
396
[26] Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML, pages 807-814, 2010.
397
398
[27] Radford M Neal. Connectionist learning of belief networks. Artificial intelligence, 56(1):11-113, 1992.
389
400
[28] Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta Numerica, 8:143-192 1999.
402
[29] J Ross Quinlan. Combining instance-based and model-based learning. In Proceedings of the tenth international conference machine learning, pages 236 1993.
404
[30] Prajit Ramachandran, Barret Zoph, and Quoe Le. Swish: a self-gated activation function. arXiv preprint 1701v:1710.059411, 7(1)) 7(1:5, 2017.
406
[31] Frank Rosenblatt. The perceptron: probabilistic model for information storage and organiza- tion in the brain. Psychological review, 65(6):386, 1958.
407
408
[32] Cynthia Rudin, Caroline Wang, and Beau Coker. The age of secrecy and unfairness in recidivism prediction. Harvard Data Science Review, 2(1), 3 2020.
409
410
[33] David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back-propagating errors. nature, 323(6088) 1986.
[34] David F. Rumelhart, Geoffrey F. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323:533-536, 1986.
413
[35] Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. CoRR, abs/1 0abs/1603.00 603.05201,
415
2016.
417
[36] Arna Sharma and Heike Wehrheim. Higher income, larger loan? monotonicity testing of machine learning models. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 200-210, 2020.
419
[37] Joseph Sill. Monotonic networks. Advances in neural information processing systems, 10, 1997.
11

---

[38] Joseph Sill and Yaser Abu-Mostafa. Monotonicity hints. Advances in neural information processing systems, 1996.
422
423
[39] Aishwarya Sivaraman, Golnoosh Farnadi, Todd Millstein, and Guy Van den Broeck. Counterexample-guidee learning of monotonic neural networks. Advances in Neural Informa- tion Processing Systems,
425
11948, 2020.
426
[40] Leslie Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference applications of computer vision (WACV), pages 464 464-472. IEEE, 2017.
[41] Leslie N Smith. A disciplined approach neural network hyper-parameters: Part I-learning rate, batch size, momentum, and weight decay. arXiv preprint 1rXiv:1803.09820 2018.
429
430
[42] Peter Veličković. The resurgence of structure in deep neural networks. PhD thesis, University of Cambridge, 2019.
431
432
[43] Seungil You, David Ding, Kevin Canini, Jan Pfeifer, and Maya Gupta. Deep lattice networks and partial monotonic functions. Advances in neural information processing systems, 30, 2017.
[44] Sergey Zagoruyko and Nikos Komodakis. Diracnets: Training very deep neural networks without skip-connections. CoRR, abs/1706.00388, 2017.
435
436
[45] Hao Zheng, Zhanlei Yang, Wenju Liu, Jizhong Liang, and Yanpeng Li. Improving deep neural networks using softplus units. In 2015 International Joint Conference Neural Networks (IJCNN), pages 1-4. IEEE, 2015.
437
438
12

---

Checklist
439
440
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]- - You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
441
442
443
444
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprietary.
446
447
Did you include the license to the code and datasets? [N/A]
448
Please do not modify the questions and only use the provided macros for your answers. Note that the
449
Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
451
1. For all authors...
452
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
454
(b) Did you describe the limitations of your work? [Yes]
455
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
456
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
458
2. If you are including theoretical results...
459
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
460
(b) Did you include complete proofs of all theoretical results? [Yes] In Appendix
461
3. If you ran experiments...
462
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] Provided in supplementary
463
465
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Provided as part of code in supplementary. Also, it should be noted that we build our experiments based on two previous ofate-of-the-are Certified Monotonic Network [22] and COMET (39] therefore for fair comparison, we employ the exact splits the authors employed in their repective works
486
467
469
470
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See -
472
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Our code can be run on Google's Collaboratory and with default settings. A GPU instance uses a NVIDIA Tesla T4 GPU with 16GB RAM
473
474
475
476
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
477
(a) If your work uses existing assets, did you eite the creators? [Yes]
478
(b) Did you mention the license of the assets? [No] Our work has also been applied as patent.
(c) Did you include any new assets either in the supplemental material The code is included in the supplemental material and will be released on github
URL? [Yes]
481
482
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
483
484
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
485
486
5. If you used crowdsourcing or conducted research with human subjects...
487
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
13

---

489
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
490
491
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14

---

493
A
Detailed proofs
494
We restate all lemmas from the main text here are give detailed proofs of them.
485
The following is well known result, proved here for completeness only:
Lemma 1, For each E (1.....n) n. and j E {1,... m) we have:
496
Ohy
497
ift; = 1, then
2 0, and
dz
Th
i44 = -1, then
< 0.
0x:
0hg Oz,
Proof. From h | b get h2 Ei w Fs by. Hence
wh.s- Finally, from
500
equation 2 have
0h Ōr;
|wasl 1
1f = if = -1
-/w03 50
Oyj Ōhg
502
Lemma 2. Lety = 1² (h). Then for each j E {L...., we have
2 0. Moreover
503
iss 1, then 9² is convex; and
504
i = 0. then 0] is concave.
Proof. From equation
(*(h) -00 + (1-
- s) o p(h)
have:
V) s,b(hz) + (1 s)))p(n))
Oyj (hys) + (1 sk)))
Th, 0p(hy)
ap(hy)
85 oh,
8;) Th,
Since both / and f are monotonically increasing and 8j E [0, 1] we have:
dy: 0
Th
508
it's 1 or O, we have p²(h) p(h) or /² (*(h) p(h), which is convex or a concave function in all components of the output, respectively.
507
508
For completeness, we repeat the Theorem 3.1 from g and its proof here:
Theorem 3.1 For any continuous monotone nondecreasing function f K R. where K is compact subset of TR. there exists a feedforward neural network with at most hidden layers, positive weights, and output O such that |O, f(x) < f.r any K and
510
The proof is derived by induction the number of input variables k. Without loss of generality, we may assume that f> 0 (otherwise, we add a constant C and approximate f + C with the network output O, then modify O with negative bias the output node). First, assume that f is strictly increasing and C In case of k = 1, we write
Proof.
513
515
f(x) = (f(x) - u) du
(7)
15

---

where H is the Heavyside function:
(xx) {
i 2 0
otherwise
516
Since continuous and increasing, invertible and therefore the right-hand side of 7 can be written as
517
[(x) / - f (v) du
(8)
518
The integral can be approximated arbitrarily well by a Riemann sum
Σ 1141 vi) H (a- f (v:)
(9)
where [u:]_=] is a partition of the interval [f(),, f(b)] This expression corresponds to a neural network with input z, one hidden layer with N neurons all connected the input with weight of
519
520
bias term in the hidden neurons f (u;), and the weights connecting the hidden layer with the output
521
522
14411 vi > Note that the Heavyside function H can be replaced by sigmoid activation function using a standard approximation argument.
523
Assume that Theorem 3.1 holds for k - imput variables. We now combine the integral representation in / with the induction assumption. For a given v, we may solve the equation of the level set corresponding to for æt
f(x,, x*) n.
524
By the implicit function theorem, there exists a function In such that
:.. xk) #x-
(10)
Note that is decreasing in all arguments Tg. This can be seen by taking the partial derivative of TO with respect to z4. We will now show that
H (f(x)) v) H (xx (z 1))
analogously to for the 1-D case. Note that is sufficient to show that
f(x) if and only if I& < (1 TT. TR-1)
and
f(x) > u and only vx (Z ( II-)).
525
But this follows easily from 10 and the fact that f increasing in all its arguments. We now approximate the integral in with Riemann sum, leading to the following equation analogously to
526
527
N
Σ
(4-1 1;) H (I Iv.
xk-1)
(11)
Since Iv, is decreasing in its arguments -r, is increasing. By the induction assumption, we can approximate -9;; with feedforward neural network O with Z1...., #k-1 as inputs, hidden layers, and nonnegative weights, such that
N
U Σ 1=1
H H( - O, (II....,Zk-I)) - REE <E
528
because the sum finite. Expression corresponds a feedforward neural network with & inputs and & hidden layers. Here k - hidden layers are needed to represent -o; and the k-th hidden layer needed combine N neural networks with outputs O and the input Fy. The weights on the connections between the last hidden layer and the final output are (U++1 v.) 0. The input vx is directly (skip-layer) connected to the k-th hidden layer.
530
532
We can now easily generalize the proof to continuous nondecreasing functions. For continuous functions, we define the convolution f with mollifier K₃ by
fa
16

---

533 Then, fs is Criftyy and fs - as 8 & 0 uniform on compact subsets. Furthermore, fa is also 534 increasing since K3 O. Now choose ó such that ff-fa fal 2 and approximate fs with feedforward
neural network O such that Iʃa O < 5. Then. /s O <
If is nondecreasing, then approximate by f8
fa ʃ + ò (2) + ok)
which is strictly increasing and let 5 4 0.
537
B Datasets Description
538
539
The descriptions of datasets used for comparison are detailed below. As mentioned in the section 4 the datasets are chosen from [22] and [39] for proper evaluation. The train-test splits of 80% 20% are used for all comparison experiments.
541
542
COMPAS [8] is a binary classification dataset, where the task to predict risk score of individual committing crime again two years, based the criminal records of individuals arrested in Florida. The risk score needs to be monotonically increasing with respect to the following attributes number of prior adult convict ctiinsti,, number of juvenile felony, number of juvenile mi misdemanno, and number of other convictions. It should be noted that there have been ethical concerns with the dataset [88 32
5M
546
548
549
2. Blog Feedback 4 is a regression dataset where the task is to predict the number comments in the upcoming 24 hours from feature containing 276 features of which 8 (A51, A52, A53, A54, A56, A57, A58, A.59) are monotonic features. The readers are suggested to refer link with targets smaller than the 90th percentile are used since the outliers could dominate the mean-squared-errov metric.
550
551
P for more details. As mentioned by the authors of [22), only the data points
559
554
555
3.
Lending club loan dateBis is classification dataset, where the task is to predict whether the individual would default on loan, from a feature set having 28 features containing data such as the current loan status, latest payment information etc,, The probability of default should be non- non-decreasing with respect to number of public record bankruptcies. ratio, and non-increasing with respect to credit score, length of employment, annual income.
556
557
558
559
580
4. Auto M 29 is regression dataset where the task is to predict city-cycle fuel con- sumption in miles per gallon (MPG) from a feature set containing features which the monotonic features are weight (W), displacement (D), and horse-power (HP)
561
562
564
5. Heart Disease 11 is classification dataset, where the task is to predict the presence of heart disease from a feature set containing 13 features of which the risk associated with heart disease needs to be monotonically increasing with respect to the features trestbps (T), cholestrol (C))
565
566
567
C
Additional Experiments and Results Details
588
569
For all our experiments, we adopt simple architectures the types depicted Figure. or Figure. More often than not, we have seen from our experiments that the neural architecture type 2 (Figure. performs better than the neural architecture type (Figure. 3). The number of hidden layers (apart from the input layer) is either set to be 1 or 2. The number of neurons in the hidden layer is selected from 4, 8, 16, 32, 64. The activation function is set to be either exponential linear unit (ELU) or Rectified Linear Unit (ReLU).
571
573
hhtps:/www..aggee/ http:///wwwwwwaoa///////n//oolla loan-dta
"hurps:/archive.icss http:///wwwiie.....i.......aaaea sec.ccctuui///ddnancttattttnnnn
17

---

(a)
(b)
(c)
(d)
Figure 5: Results for function fitting experiments where (a) is the ground truth, (b) is the result
obtained from a simple unconstrained neural network (with ELU activation) and (c) is the result obtained from constrained neural network (weights be positive and ELU activation) and (d)
shows our results with constrained weights and bipolar ELU activation
575
Apart from the experimental results in the main part the paper, here we highlight the usefulness of the our monotonic dense unit. In a scenario where the dataset is noisy and very small and also contains monotonic features, our neural network constructed using our proposed monotonic dense unit tends to perform better because of the inductive bias. To illustrate this we consider synthetic dataset generated by function f(x, 3) = sgn(ax) x² + b sin(cy), a, b, E {0.5, 0.35, 3.3]. We sample points uniformly in the range 2-2., 2.5) for both and y, and add zero-centered Gaussian noise. We then sample only 100 points for training and train three neural networks having the same kind of architecture but having difference in constraints
577
579
581
weights mentioned in section but activation function used is ELU instead ReLU. We test the three networks Unconstrained, the task of fitting the aforementioned function. The results be seen in the Figure. [S] It is evident that the unconstrained neural network does not preserve monotonicity and although constrained neural network does preserve monotonicity, it cannot faithfully reproduce the concave part of the function, whereas ours (constrained bipolar ELU) can fit both concave and convex part of the function.
683
Constrained ELU and Constrained bipolar ELU,
585
587
588
18


Reviews:
[{"review_id": "_jO1Yem6xnb", "reviewer": "Reviewer_8Y4o", "summary": "The authors proposed a constrained monotonic dense layer architecture that is compositable to form a monotonic neural network which can approximate non-convex function and has strong empirical results.", "questions": "\n1. On the limitation of the stacking multiple layers, would it be possible to adapt this method to build monotonic layers other than dense layers (such as self-attention or convolutional layers)?\n\n2. In the experiment section, grid search is performed to find the best hyper-parameters for the current method. Would it be possible to perform a similar search for one of the other methods in Table 1 to verify the advantage of the new architecture?", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed monotonic dense layer employs simple, off-shelf components in today's deep learning frameworks, making it very easy to use;\n2. The design of the monotonic layer allows composition, making it extensible to deep neural networks;\n3. Strong empirical results (models with fewer parameters and better quality).\n\n\nWeakness:\n\n1. Several definitions are missing from the writing: in section 3.1., definition of a partially monotonic function and a compact segment should be given for clarity.\n\n2. If I understand correctly, in the composed monotonic model with both monotonic and non-monotonic inputs, the layers after the concatenation layer can only be monotonic dense layers. This seems to pose a limitation on stacking multiple layers, such as a stack of transformer blocks, or a stack of convolutional layers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "W4TlLcjsN8", "reviewer": "Reviewer_ShEg", "summary": "In the current paper, the authors propose a deep neural network that produces monotonic functions. It is easy to produce a convex function (constrain the weight to non-negative values) but it is not trivial to get monotonic function.   To that end, the author introduce a so called Monotonic Dense Unit that combine convex and concave activation functions in order to output a provably monotonic function.                                                                   The authors then justify their approach on 3 benchmarks.", "questions": "How do you compute the vector t (monotonicity indicator) in practice?\n\nWhat is the advantage of your method over constrained networks? Do you have examples that highlights the benefits of Monotonic Dense Unit?\n\nAs mentioned, the question of universality is particularly interesting, could you describe the reach of your algorithm in term of descriptive power?\n\nWhat would prevent to use any other concave decreasing activation function instead of the point symmetric?", "limitations": "The author must discuss their usage of COMPAS dataset, as well as the comment l29-l33 as already stated in the main review.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nModeling functions with specific constraints such as monotonicity or convexity is an interesting and broad topic of ML with many application (despite the thoughts of the authors l139). The proposed method is sounds and provably outputs monotonic function in selected components.\n\n# Weaknesses\n\nThe paper lacks proper definition about what are increasing and decreasing features and how do they interact with each other.                                                                                                               The propose method is very incremental and the paper lacks of a proper develop theory. The author conjecture l189 the universality of their approach but to not provide elements. Also, the PMU seems a bit over-complex and tend to look like an engineer approach.       \n                                                                                                                                                                                                                                                                                                                                                                                               * Global style issues\n\nThe paper is written in a somehow bad style. Nothing is properly defined (the core of the paper is \"defined\" in the caption of figure 2), the text is full of pleonasms such as \"monotonic increasing functions\" and often contradicts one paragraph with another (e.g. l36 claims it is not trivial, l49 asserts it is).                                                                                                                                                               A lot of points are repeated over and over, sometimes with useless information (e.g. the name of bozillions activations function from the literature) and it often feels like the author just wanted to fill 9 pages. This gives room to properly introduce what is the code of the paper and all the needed notions.                                                                                                                                                                   Whether an equation is a definition or a proposition is often unclear (e.g. l237 seems to be a proposition (because of 'then') but is in fact the definition).                                                                              Two names are proposed for the global algorithm \"Monotonic Dense Unit\" or \"Constrained Monotone Fully Connected Layer\" (figure 1, l.219-220) none of those being really used in the later. Definition 3 is yet another \"Monotone Constrained Fully Connected Layer\".\n\n* Ethics\n\nAnd seriously what the actual heck is the following (l29-33):                                                                                                                                                                               \"All other things being equal, a larger house should be deemed more valuable, a bank's client with higher income should be eligible for a larger loan, an offender with a longer crime history should be predicted as more likely to commit another crime, etc. A model without such a monotonic property would not, and certainly should not, be trusted by the society to provide a basis for such important decisions.\"                                                              This has NOTHING to do in an ML paper.                                                                                                                                                                                                                                                                                                                                                                                                                                                  The use of COMPAS dataset should be ethically discussed.    \n\n* More precisely among much more\n\nHow to \"constrain\" the weights is never properly said (only in the introduction, I guess the author mean that the weights are >=0 elsewhere too).\n\nDefinition 1: h is defined twice in an inconsistent manner (let h be [...], h=...).\n\nSection 2.4: the question of universality is indeed interesting but this section is somehow very weak if the authors have nothing to say on this topic.\n\nFigure 2: why are W \"*kernel* weight\"\n\nI do not understand Figure 4 and the explanation (l280-288) is really confusing.     \n                                                                                                                                                                                                                                                                                                                                                                                                   Eq (1): the case = 0 is ill-defined\n\n* Global remark\n\nA major limitation of the proposed approach is that the assumed monotonic components of the input are dealt in an independent manner.\n\n* Experiments:\n    - the experimental protocol is insufficiently described;                                                                                                                                                                                    \n    - the networks used are never properly described;                                                                                                                                                                                                   \n    - dataset are not described well enough, and the hypothesis (monotonicity) never discussed;                                                                                                                                                 \n    - not enough experiments, I would also have expected some ablation studies and hand-crafted problems;                                                                                                                                       \n    - the use of COMPAS dataset must trigger some ethical concerns that are ignored in the paper;                                                                                                                                               \n    - the choice of datasets and the chosen \"monotonic\" target features should be accurately discussed;                                                                                                                                         \n    - the proposed model should at least be compared with \"constrained NN\" in order to assert the improvement of the proposed algorithm over naivety.\n\n* Typos\n\nPunctuation is missing in many if not most equations.\n\nReferences: many if not most references are incomplete (arxiv instead of peer reviewed journal/conferences)                                                                                                                                                  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inadequate Data and Algorithm Evaluation"]}, "extra_text": {}}, {"review_id": "E_GS-fV7_au", "reviewer": "Reviewer_k32L", "summary": "The paper proposes a differentiable neural network layer which is monotonic in its input by construction. It is defined as a convex combination of a convex and a concave monotonic functions on the input. The layer is simple, both conceptually and to implement, and may be useful to reduce parameter count when the modeler has prior knowledge that the function to be learned is monotonic. This layer is also computationally efficient: it has the same amount of computation as a typical feed forward layer followed by a nonlinearity, up to a constant factor.\nAfter describing the layer, the authors provide empirical validation on some tasks, showing that models using the proposed layer achieve better accuracy with smaller parameter count vs monotonic layers in the literature.", "questions": "- The authors discuss convex monotonic networks [1]. On the other hand, they do not discuss Input convex neural networks [2]. Taking the derivative of an input convex neural network wrt input would also yield an nondecreasing function. How would this compare to the proposed model?\n\n[1] Xingchao Liu, Xing Han, Na Zhang, and Qiang Liu. Certified monotonic neural networks. NeurIPS 2020\n\n[2] Amos, B., Xu, L., & Kolter, J.Z. (2017). Input Convex Neural Networks. ArXiv, abs/1609.07152.\n", "limitations": "The paper proposes a class of parametric functions to approximate monotonic functions. It does not provide a guarantee that this class can approximate any monotonic function. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nConstraining a model to be input monotonic is an important problem in ML. Monotonicity can stem from either domain knowledge (price of a house should be increasing wrt surface all else fixed), or functional constraints (estimating a cdf).\nThe proposed method is simple, which is a strength. Although the specific shape of the layer seems to have been explored by previous works, it has not been used in the specific context of monotonicity. \nThe paper is well written and clear, although perhaps more space could be used for empirical results, and discussing settings in ML where monotonicity is important.\nThe context of the work is clear.\n\nMain criticism:\nThe paper does not provide an approximation theoretical result. Are there increasing functions that cannot be approximated by the proposed method?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6WQBgSD-9cI", "reviewer": "Reviewer_QsCV", "summary": "The authors propose a simple scheme based on the convex combination of a function and its point reflection in order to produce a monotonic building block from which input monotonic (or partially input monotonic) neural networks can be built.  The construction is validated both theoretically and experimentally.", "questions": "See strengths + weaknesses.", "limitations": "Seem to be adequately addressed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:  There is quite a bit of work on monotonicity in neural networks, and at least to my knowledge, the authors seem to do a good job of positioning their work.  While point reflections have been used in the past, their application here does appear to be novel.\n\nQuality:  I didn't find any serious errors with the technical content.  However, I think that the experiments fall a bit on the short end:  the number of data sets is limited and they are somewhat smallish and the performance improvements in many cases are either not statistically significant or too small to get really excited about.  As a result it feels like a more robust set of experiments, including on problems where monotonicity isn't even required, would significantly bolster the case for acceptance.  Additionally a careful critique of the competing methods in the experiments should be done to emphasize the strengths and weaknesses of the present approach.\n\nClarity:  I found the article easy to read for the most part.  Some of the definitions could be written more clearly with less fluff, e.g., in Definition 1 use R^n, etc. to simplify the presentation, and there are a few minor typos here and there, e.g., line 268.  I also feel like way too much of the main text is devoted to background work (some of which is repeated way too much -- I got the point the first time).  I'd prefer more experiments to supplant it.\n\nSignificance:  The construction is simple and the resulting networks are seemingly easy to train.  As a result, I could see this work being of practical significance.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject