Scalable Monotonic Neural Networks
Authors: Hyunho Kim
Jong-Seok Lee
Venue: ICLR 2024 poster (2024)
Keywords: neural networks
monotonicity
scalability
conventional error-backpropagation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: In this research, we focus on the problem of learning monotonic neural networks, as preserving the monotonicity of a model with respect to a subset of inputs is crucial for practical applications across various domains. Although several methods have recently been proposed to address this problem, they have limitations such as not guaranteeing monotonicity in certain cases, requiring additional inference time, lacking scalability with increasing network size and number of monotonic inputs, and manipulating network weights during training. To overcome these limitations, we introduce a simple but novel architecture of the partially connected network which incorporates a 'scalable monotonic hidden layer' comprising three units: the exponentiated unit, ReLU unit, and confluence unit. This allows for the repetitive integration of the scalable monotonic hidden layers without other structural constraints. Consequently, our method offers ease of implementation and rapid training through the conventional error-backpropagation algorithm. We accordingly term this method as Scalable Monotonic Neural Networks (SMNN). Numerical experiments demonstrated that our method achieved comparable prediction accuracy to the state-of-the-art approaches while effectively addressing the aforementioned weaknesses.

Published as a conference paper at ICLR 2024
SCALABLE MONOTONIC NEURAL NETWORKS
Hyunho Kim, Jong-Seok Lee"
Department of Industrial Engineering
Sungkyunkwan University Suwon, Republic of Korea {retna319, jongseok 8skku. edu
ABSTRACT
In this research, focus the problem of learning monotonic neural networks, as preserving the monotonicity a model with respect to a subset of inputs is erucial for practical applications across various domains. Although several methods have recently been proposed to address this problem, they have limitations such not guaranteeing monotonicity in certain cases, requiring additional inference time, lacking scalability with increasing network size and number of monotonic inputs, and manipulating network weights during training. To overcome these limitations, we introduce a simple but novel architecture of the partially connected network which incorporates a 'scalable monotonic hidden layer' comprising three units: the exponentiated unit, ReLU unit, and confluence unit. This allows for the repetitive integration of the scalable monotonic hidden layers without other structural constraints. Consequently, our method offers ease of implementation and rapid training through the conventional brroo-backpropagation algorithm. We accordingly term this method Scalable Monotonic Neural Networks (SMNN). Numerical experiments demonstrated that our method achieved comparable pre- diction accuracy to the state-of-the-art approaches while effectively addressing the aforementioned weaknesses.
INTRODUCTION
The field of neural networks has made significant progress in recent years, leading to notable ad- vancements in domains like computer vision (Redmon & Farhadi (2018), natural language processing
(Vaswani et al. 2017). and speech recognition (Baevski et al. 2020). Neural networks have been
successfully applied to a wide range of practical applications, including prediction (Chen et al. 2018
Arik & Pfister 2021). generation (Goodfellow et al.) 2020 Ho et al. 2020 Kobyzev et al.) 2020).
out-of-distribution detection iLiang et al. 2017). and more. However, the incorporation of prior domain knowledge into neural network models remains open question. One important inductive bias sought in neural network models is monotonicity, ensuring that the output consistently increases decreases with changes in certain inputs. In domains where model credibility is critical, such as finance, healthcare, legal, and engineering, it is essential to consider monotonicity when designing neural network models. For instance, in the financial domain, accurately reflecting the positive influence of factors on price is crucial. Similarly, in engineering, predictive models must account for monotonic relationships between control factors and responses to ensure their safe use.
In traditional machine learning, various approaches have been employed to enforce monotonicity in trained models, including decision trees (Potharst & Feelders) 2002), random forests (Bartley et al. 2019), boosting (Chen et al. (015). support vector machines (Doumpos & Zopounidis 2009
and traditional statistical regression (Dykstra al.) 2012) Kalai & Sastry] (009) et al. 2015).
Similarly, there have been research efforts in the past and more recently to incorporate monotonicity
as an inductive bias into neural networks (Archer Wang 1993 Sill 1997 Lang 2005 Daniels &
Velikova 2010 Milani Fard et al. 2016 You et al. 2017). which can be categorized into groups.
Approaches with customized architecture: The first group of studies focuses on modifying neural networks to achieve monotonicity or partial monotonicity in specific input features. Hand dend-ne
"Coresponding author

---

Published as a conference paper at ICLR 2024
architectures heuristic algorithms are employed for this purpose. For instance, the non-negative approach (Archer& & Wang 1993) restricts weights on monotonic features to be positive. The Min-Max Network (Daniels & Velikova 2010) utilizes three specially designed layers (linear layer. Max-pooling layer, Min-pooling layer) to enforce monotonicity. The Deep Lattice Network (You al. 2017) constructs network using lattice layers and combines them in ensemble. Additionally, the Hierarchical Lattice Layer (HLL) method (Yanagisawa et al. (022)) extends the existing deep lattice network to address its limitations, particularly in handling high-dimensional inputs. However, HLL still faces the challenge of significantly increased computational complexity as the dimension of the monotonic features grows. It has been recognized that common drawback of the aforementioned studies lies in the often excessive constraints imposed, which result in restricted hypothesis space for weight parameters. Consequently, models in this category tend to exhibit relatively lower performance when compared to the other group. However, recent works have demonstrated their ability to overcome this limitation. The Lipschitz Monotonic Networks (LMN) method (Nolte et al. 2022) has exhibited enhanced performance by constraining the norm of weights, ensuring the trained network maintains Lipschitz continuity. Nevertheless, it is important to note that if the Lipschitz constant À remains smaller than the inherent monotonicity scale within data, LMN may struggle to effectively learn from the data. This limitation arises because LMN cannot approximate any function that holds a gradient exceeding the given Lipschitz constant. Another notable study demonstrating promising performance is the Constrained Monotonic Neural Networks (Constrained MNN) method (Runje& & Shankaranarayanal 2023). In this work, a monotonic dense layer was introduced to enforce monotomicity by converting updated weights into absolute values during the training process. The main strength of the approaches within this category lies in their capacity to ensure monotonicity. However, limitations exist. Earlier studies exhibited low performance, while recent advancements come with their own constraints, notably the non-optimality of trained networks resulting from weight manipulation during the training process.
The second group aims to achieve monotonicity through regular- ization techniques. One approach is to impose penalties on negative gradients directly penalize negative weights of the network as a form of regularization loss (Gupta et al. 2019 [iu et al.] 2020
Approaches with regularization:
Monteiro et al., 2021). The Certified Monotonic Neural Networks (Certified MNN) method (Liu et al. 2020) penalizes negative gradients for each feature in the data. It leverages the piece-wise linear property of ReLU to verify the monotonicity of the trained network using mixed-integer linear programming (MILP) solver. However, it requires additional processes to enforce and verify monotonicity. As a regularization method, this method does not always ensure monotonicity. There- fore, in scenarios where the guarantee of monotonicity is essential, opting for the approaches with customized architecture may be preferable. Moreover. as the network depth increases, satisfying the monotonicity constraint becomes more challenging, resulting in significantly higher computational costs. Another regularization approach involves data augmentation. The Counterexample- Guided -uuidd Learning of Monotonic Neural Network (COMET) method (Sivaraman et al. 2020) iteratively injects monotonicity into the model through guided learning with the augmentation of counterexamples for samples violating monotonicity. Monotonicity guarantees are achieved by finding the upper and lower envelopes of the model using iterative algorithms proposed in this research. The COMET method utilizes the satisfiability modulo theories (SMT) solver (Barrett & Tinelli 2018) to find coun- terexamples. Similar to Certified MNN, COMET does not employ end-to-end learning, and as the network size grows, the solver becomes computationally intensive. Both Certified MNN and COMET have the advantage of exploring weights within an unrestricted hypothesis space. However, they also share the vinadvantage of requiring additional steps in the inference or prediction process, making them not fully scalable to high- gigg--iimnsionall monotonic features, As the network size increases, the computational cost of the solvers for both methods becomes prohibitively high.
Table I presents a comparison of different monotonic neural network methods and their respective capabilities. While some existing methods demonstrate competence in certain aspects, others fall short. While addressing the limitations in learning monotonic neural networks, it is important to prioritize scalability concerning the increasing number of monotonic inputs and network size, along- side achieving good model performance. To address these challenges, this research proposes new learning approach. Specifically, our study introduces a simple but novel architecture of the partially connected network which incorporates 'scalable monotonic hidden layer' comprising three units: the exponentiated unit, ReLU unit, and confluence unit. This allows for the repetitive integration of the scalable monotonic hidden layers without other structural constraints. Consequently, our method offers ease of implementation and rapid training through the conventional error- eroorbbackpropggation
2

---

Published as a conference paper at ICLR 2024
Table I: Comparison of monotonic neural networks with their capabilities
Method
end end guarantecing monotonicty scalable trainable tradiional gradient deseent
Appreaches Regulurization
COMFT (ivaramna
Approaches with Cusomizzd Architectures
HLL
Yapasioge alll 2022
LMV
Notle
Constraimed MNN
Kanjy
2023
algorithm. We accordingly term this method as Scalable Monotonic Neural Networks (SMNN). Furthermore, our method enhances function approximation capabilities. Through numerical experi- ments, we demonstrate the superiority of our approach when compared to the state-of-the-art methods. Additionally, our method effectively incorporates monotonicity as an inductive bias into the model, leading to improved generalization performance. In matters of scalability, even as monotonic inputs and network size expand, our method demonstrates efficient training with minimally required time increments.
2 MONOTONICITY AND PARTIAL MONOTONICITY
This section provides the definition of monotonicity for functions, starting with the definition for a univariate function. Subsequently, we extend the definition to cover monotonicity and partial monotonicity for multivariate function.
Monotonicity: Let f(x) be continuous and differentiable univariate function, which maps normalized input x E X = [0. 1] y of IR. The function f(x) said to be monotonically non-decreasing iff x' = f(z) < f(x') holds for any two points x, x' X. Similarly, it
monotonically non-increasing iff x x' + f(z) 2 f(x') satisfied. Alternatively, f(z) is
monotonically non-decreasing iff Vx. 0f(x)
dz 2 0 is met, and it is monotonically non-increasing iff
Vx, (f) < 0 satisfied. Bx
Monotonicity and partial monotonicity: Let f(x) be continuous and differentiable multivariate function, which maps a normalized input x E X = [0, 1]d to y of R. Suppose that the input vector is partitioned into f(x is said to be partially monotonically non-decreasing on Xm. which ith element is Tú, iff Vi, x4 S 24, Vj * i, xz = d' + f(x) S fxx) holds for any two points x, x' E X. Similarly, monotonically non-increasing iff Vi, H; x/ Vj i, Ij x'j
(Xm: x-m) Rm Rd-m where 0
d. The function
f(x) > f(x) is satisfied.
Alternatvely, fxx is partially monotonically nen-decrgg for x iff Vx; Xms dz; is met, and it is partially monotonically non-inereasing iff Vz; X. 8f(x) ar.
0f(x)
< 0 is satisfied.
As observed, represents monotonic features while signifies non-monotonic features. Si- multancously, m denotes the number of monotonic features. In the multivariate case, when m = partial monotonicity can be treated monotonicity. Within the neural network training context, the function f can be interpreted as a neural network.
SCALABLE MONOTONIC NEURAL NETWORK
In this section, we introduce our proposed SMNN method. Our method incorporates three key elements: the utilization of exponentiated weights to ensure monotonicity, the adoption of ReLU-n activation for universal function approximation, and partially connected network structure that preserves monotonicity for monotonic inputs while capturing interactions among all input features. Because of these characteristics, method achieves scalability by eliminating the need for any pre- or post-procsssin steps. Instead, it simply employs conventional error- buroo-backroopggatiio to train a neural network efficiently.
3

---

Published as a conference paper at ICLR 2024
Exponentiated weights: Exponentiated formulations have been widely used in various machine learning studies for different purposes. For example, Neural Additive Models (NAM) (Agarwal et al.) 202T) introduced the Exu-centered layer to increase the flexibility of univariate sub-networks. The use of exponentiated weights in this layer allows even small changes in the input to result in significant changes in the output within a specified range. In the FLOW-hased generation model REAL-NVP (Dinh et al. 2016). exponentiated formulations were employed to ensure that the determinant of the Jacobian matrix of the affine coupling layer remains positive. In line with these previous works, but with different purpose, we propose a hidden node that utilizes exponentiated weights for monotonic input features, as defined by
Mexp((n.) = (exp(ww) Xm + b)
where a is an activation function and b is a bias. It becomes evident that when the activation function is monotonically non-decreasing, the hidden node preserves its monotonic behavior. Therefore, we exploit the ability of exponentiated weights to preserve the monotonicity of the hidden node, making them a good choice for learning monotonic neural networks. While the concept is similar to an early study (Zhang & Zhang, T999), our research is distinguished by the use of different activation function and a specially designed network structure. These will be described in detail below.
Activation functions: In modern deep learning, ReLU commonly used as an activation function due to its advantages such as alleviating gradient vanishing (Nair & Hinton 2010 He et al. (016)) 6). Another benefit of this activation function is that it allows neural network training to be formulated as
an integer programming problem (Liu et al., 2020) other optimization problems (Sivaraman et al. 2020), simplifying the training process. However, it is known that a ReLU network with all weights constrained to be positive is either convex concave, depending on the direction of ReLU (Liu et al. 2020). This means that the neural network lacks the universal approximation property. To overcome this limitation, we employ the ReLU-n activation function (Liew al. 2016), which is given by
n
+ if n. x x i
ReLU-n(x))
J 8 K n. x
0 if < 0, 2
A previous research has demonstrated that neural networks with positive-constrained weights and a threshold activation function are universal approximators of partially monotonic functions with at least a depth of hidden layers (Mikulincer & Reichman [0022). The nature of the ReI.U-n function in equation 2 is a piecewise-linent function with a threshold 0 and n. Unlike ReLU, this function exhibits both convexity at zero and concavity at n. Importantly, it is a monotonically non-decreasing function, thus ensuring the preservation of monotonicity of the hidden node in equation [ Alternative activation functions offering the similar benefit, including both convexity and concavity, can be substituted for the ReLU-n function (See our ablation study in Appendix
Network structure: Prior introducing the proposed network structure, we define three types of hidden nodes. The exponentiated hidden node, denoted in equation 3. is designed for monotonic features (x,,) ) based on equation (and equation 2 For non-monotonic features (x-m)) the well-known ReLU hidden node, which is given in equation
4 is employed. Lastly, the confluence hidden node is defined in equation [ Each hidden node forms individual hidden unit comprising multiple nodes, resulting in three types of hidden units, namely the exponentiated unit, ReLU unit, and confluence unit, as also defined in e equation 4 and equation (quation[3respettiiiii respectivey. To maintain a consistent activation function throughout the network, it is acceptable to use the ReLU-n function for the ReLU unit (See Appendix C.I). It is worth noting that the hidden nodes in equations 3 4 and 5 represent the nodes in the first hidden layer, which receive the inputs (Xm, x -m). As we progress to the next hidden layer, they receive the outputs from the preceding hidden nodes.
Exponentiated unit := (Nezp)) where heap (xm) ) = ReI.U-((exp(w) xm + b)
(3)
ReLU unit == }, where hyratu (x-m) ReLU(w" x-m + b)
(4)
Confluence unit = (Ihmnf) ), where heon (x (x-m) = ReI.U-n(W' T x.m + b)
(5)
Based on these units, we present our proposed SMNN structure, where each hidden layer comprises the three units. We prepared Fig. [ illustraaing network with three hidden layers as example to
4

---

Published as a conference paper at ICLR 2024
unit
unit
Esp
Conflumnce unit
Conflumnce
Conflumnco unit
fc layer
unt
Rel.U
unit
ReL.i
unit
Rel
unic
Concatenation
hidden layer
hidden layer
hidden layer
Figure 1: SMNN Structure. The features are divided into two groups based on whether each variable requires monotonicity (Xm) not (x-m). The monotonic features are directed exclusively through the exponentiated unit and fully connected layer to ensure monotonicity, as indicated by the bold line.
help readers better understand. Let
= (Cheap) th (d
(6)
be the i-th hidden layer, which named scalable monotonic hidden layer. The units in this layer serve specific roles: the exponentiated unit captures interactions among monotonic features (xm) while preserving monotonicity, the ReLU unit captures interactions among non-monotonic features (X-m), and the confluence unit combines the outputs of the exponentiated and ReLU units to capture interactions between both types of features (See Appendix C.3 for the justification of the confluence unit.). The purpose of using ReLU-n activation functions in the confluence unit nodes is to align the output magnitudes of these nodes with the output magnitudes of the exponentiated unit nodes in the preceding layer. When i 1,
(KOP)) = (heap (xm)), (hoony) = (heons (X-m)) (heelaa = (hretu (x-m))
When 2 2,
(xxp) } = (heep ((((6i-2) (1(6-111) H1
(h (i)
fonnf) = (heon (1
(1-1)
(8)
relu
(1 (1,0) = (bratu
(5-1) retu })};
where [ implics the concatenation of two vectors. Let In. which is the output vector at the n-th hidden layer, and fxx be the output node in fully connected output layer. Then,
f(x) = cxp(wx+1) h b,
where Wx+1 is the vector of weights between the n-th hidden layer and the output layer. The output node also employs exponentiated weights to ensure monotonicity, without an activation function. is important to note that while we present the proposed SMNN for a single output node, this expression can be easily extended to accommodate multiple output nodes. Notably, the monotonic features follow distinct path through the exponentiated unit and the fully connected layer, depicted by the bold line in Fig. ensuring the monotonicity of the proposed SMNN.
We now prove the monotonicity of SMNN by following the monotonicity definition outlined in Section 2 This involves taking the partial derivative of SMNN with respect to monotonic feature and showing that the derivative is positive. It is worth noting that SMNN is inherently monotonically non-decreasing function, and for monotonically non-increasing feature, we can simply multiply the feature by -1 and incorporate it into SMNN to maintain its property of monotonic non-increase.
Monotonicity SMNN:

---

Published as a conference paper at ICLR 2024
Theorem 1. Let fx) be the proposed SMNN with scalable monotonic hidden layers and Xm = (01,.... Tp) a vector of monotonic input features. For 2,, {I....,p).
(x(x) Ox, 0.
Proof. As Fj follows a distinct path through the exponentiated unit and the fully connected layer and aim find the partial derivative of f(x) with respect Iy without loss of generality, consider a continuous function (z)) given by
g(x) = cxp( bn+1,
where
ReeLU--((epp((t))) bı), i 2,...,
and
t =ReLUnn(exp(ux)e b1).
By showing dg(=)
da 2 O, we can show dz. 2 O. It is quite straightforward to illustrate dg(a) 2 0.
xx(x)
We first describe the derivative of ReL.Un(()) function as follows:
if > n, 5 8-n n
R' = ReLU-n'(s)
=
0 < 8 +nn 0+
0 if
Then,
dh = R' exp(w1) 2 O. dx
Likewise,
dt2 R' exp(w2)
dt = R' ex-p(xy))) R' exp((1) 2 0. dr
dr
and
dt Rt' exp(wa) dr We finally show that
dtn-1 It' exp(wn) Rt exp(wn-1) ... Rt' exp((x)) 2 0. dx
dg(x) dx = exp(wn+1)
dla 2 0.
dr
NUMERICAL EXPERIMENTS
4.1 EXPERIMENTS ON SYNTHETIC DATASETS
We first assessed the effectiveness of our proposed method by conducting experiments on synthetic datasets. In addition to verifying the monotonicity aspect, we thoroughly examined the scalability and generalization performance of our method.
Scalability test: In this experiment, we used two synthetic datasets evaluate the scalability
with respect to the network size and the number of monotonic features. For the first aspect of the scalability, we used a two-dimensional example obtained from (Liu al., 2020). The function is defined in equation 10, where monotonicity is imposed y only between and y. We created dataset consisting of 1000 instances with æ and values generated from uniform distribution within the range of and 1. The dataset was divided into 70% training set and 30% test set.
25. b 0.5) + exp(y) + +
[ v) a sin
(10)
a, b. {0.3, 0.6, 1.0), 2. E [0,
Fig. [[a) demonstrates that our proposed method (with two scalable monotonic hidden layers) accurately fit the original function, ensuring monotonic increase of the function with respect y.

---

Published as a conference paper at ICLR 2024
0.14
Moan MSE
Moan Time (s)
NSE
Mean
175
6:22
6:00
Moan Timefol
Mealh Timee 66)
Nean
3.5
6.00
Number of Monotonic Features
(a)
(b)
(c)
Figure 2: (a) Upper: Contour plot of original function with a, b, c and æ, y E [0, 1]. Lower: Contour plot of the fitted model by our method (MSE=0.0007). (b) Scalability test with respect to increasing network size: We tested the proposed method for all combinations of a, b, c. Blue line represents the average MSE from all 27 runs. Orange bar indicates the average training time required for all 27 runs. (c) Scalability test with respect to increasing number of monotonic features: Blue line represents the average MSE from five repetitions; Orange bar indicates the average time required for five repetitions.
Furthermore, Fig. bb) illustrates that as the number of parameters increases, the average mean- squared error (MSE) of the model decreases while the computation time remains nearly constant. This indicates the high scalability and extensibility of our proposed method to accommodate larger networks.
In addition, to investigate the scalability with respect to the number monotonic features, we designed 40-dimensional dataset, excluding the output feature, comprising 20 uniform distribution- based dummy inputs and 20 monotonic inputs determined by equation
10
20
f(x)
Σ 0g3 2
Σ
a exp F4
Σ a1(x; + sin x.))
Σ
az(2; cos x; +0c, (11)
6=
1=16
32 00
(i
20), 6 N(0,1)
{a:} 4,3, 6, 8, 5, 45344446653434443222 442.22225544332.33.2
20)
We generated 1000 instances, allocating 700 instances for training and 300 for the test set. The study involved systematically increasing from 20. It is worth noting that monotonic input either belonged to Xm. or X-m Fig. 2[c) illustrates that as the number of monotonic inputs increased from 1 20, SMNN exhibited consistent training times (at around 20 seconds), while decreasing its MSE performance, See Appendix C.4 for the case of 200 monotonic features.
We conducted a series experimental evaluations to assess the generalization performance of the proposed SMNN method with guaranteed monotonicity, using well-known Friedman function (Friedman) 1991). ). To introduce variability into the original function, Gaussian noise N(0, 1) was incorporated, and the scaling factor À applied to the noise was gradually increased. The Friedman model, defined in equation imposes monotonicity on IA and Ø5.
Generalization performance test:
f(x) = 10sin(T2112)) + 20( - 0.5)² + 10ma + 5.5 + λe, 24 E [0, (i 1,... ) (0,
(12)
In our analysis, compared MSE across three different networks: the proposed SMNN with monotonicity enforced on I4 and øs, a structurally identical network with SMNN but using only ReLU activation functions (without exponentiated weights and ReLU-n), and typical Multi-Layer Perceptron (MLP) with matching layer depth and node count. To ensure robust evaluation, we employed five-fold cross validattn strategy, using 80% of the data for training and 20% for testing.
7

---

Published as a conference paper at ICLR 2024
For each folding, conducted five independent runs, which resulted in total of 25 experimental trials.
SKHNN (dum))
SMMW LOurs]
SKNN (Oun) Sa Stracture
Structure
Same Structure
602
MLP
330
100
(a) Training MSE
(b) Test MSE
(c) Denoised Test MSE
Figure 3: Results from the Friedman function experiments
The experimental results are presented in Fig. (For more detailed results, please refer to Appendix B} The training MSEs are shown in Fig. 3 a). where SMNN exhibited the highest error, while MLP achieved the lowest. This suggests that the imposition of monotonicity constraints significantly affected the learning dynamics of SMNN. Conversely, in terms of the testing MSE shown in Fig. (b),, MLP had the highest error, followed by the structurally identical network, while SMNN yielded the lowest MSE. For MIP, the testing MSE was nearly seven times greater than the training MSE when reached 20, indicating overfitting to the noise. In contrast, the testing MSE of SMNN remained consistent with the training MSE across different À values, indicating robustness to noise. Notably, Fig.Bcc) 3(c presents an interesting observation where the noise of the test dataset was eliminated. In this scenario, the MSE of SMNN becomed significantly smaller than that of the other networks. These results suggest that incorporating monotonicity as an appropriate inductive bias enhanced the generalization performance of the network, as demonstrated in our experimental findings. There- fore, we conclude from this experiment that imposing monotonicity could improve generalization performance.
4.2 EXPERIMENTS ON REAL-WORLD DATASETS
Having established the validity of our method in the previous subsection we proceed to compare it with the state-of oftte--o--hearr methods discussed earlier. We conducted both regression and classification tasks using several publicly accessible datasets that were used in previous studies. The Auto and Blog Feedhack (Spiliopoulou et al.) 2014) datasets were used for regression tasks, while the COMPAS (Angwin et al. 2016), Heart Diseasef) and Loan ( were employed for classification. Similar to the synthetic data experiments, we applied a 5-fold cross-validation strategy, performing 25 runs for each dataset. Appendix A provides details regarding the data description and training configurations. The experimental results are presented in Tables 2 and B respectively. The reported metrics include model complexity (number of parameters) if available, and prediction performances such as MSE and RMSE for regression and accuracy for classification, with mean and standard deviation obtained from repetitions. The benchmarking methods included in the tables for comparison are Isotonic in KKlla & Sastry] 2209)), XGBoost in (Chen et al.] 2015). Crystal in Milani Fard et al., (2016), DLN in You et al. 2017). Min-Max Net in (Daniels & Velikova (2010).
Beg-Neg-DNN in (Liu et al. 2020), Certified MNN in (Liu et al. (020). COMET in (Sivaraman
et al.
2020 LMN in (Nolte et al. 2022), and Constrained MNN in (Runje & Shankaranarayana
2023
Tables 2 and present comprehensive performance evaluations of the proposed methodology with the comparative techniques. Table 2 shows the experimental results and model complexities with the dataset featured in the previous studies (Liu et al. [020 Nolte et al., 2022, Runje & Shankaranarayana
2023). Our proposed approach demonstrated superior performance on both the COMPAS and Blog Feedback datasets in comparison to existing methodologies. In the case of the Loan Defaulter dataset, while our method did not achieve the top position, showed comparable performance to recently
8

---

Published as a conference paper at ICLR 2024
Table
Comparison of our method with other methods described in (Liu al., 2020 Nolte et al.
2022
Runje & Shankaranarayana 2023 ("1" indicates the statistical tie with the best.)
COMPAS
Blog Feedback
Loan Defaulter
Method
Parameters
Test Ace
Parameters
RMSE +
Parameters
Test Acc
Isntonic
N.A.
67.6%
N.A.
0.203
N.A.
62.1%
XGBoost
N.A.
(68.5 1 0.11%
N.A.
0.176 = 0.005
N.A.
(63.7
Crystal
25840
(66.3 0.1)%
15840
0.164 1 0.002
16940
(65.010.1)5
DLN
31403
(67.9
27903
0.161 = 00000
29919
(65.1
Min-Max Net
42000
0.1.%%
27700
0.163 = 0.001
29000
0.1.9
Non-Neg-DNN
231 12
(67.3 0.9]%
8492
0.168 + 0.001
8502
(
Certified MNN
23112
(68.8 t: 0.9)%
8492
0.158 0.001
8502
(65.2 0.1)%
LMN
37
(69.3 + 0.1)%
2225
0.160 0.001
753
(65.4 = 0.0)% %
Constrained MNN
2317
(69.2 + 0.2)%f
1101
0.156 = 0.001
177
(65.3 = 0.1)
SMNN (Ours)
2657
(69.3 + 0.9)%
1421
0.150 + 0.001
501
(65.0 - 0.1)%
Table 3: Comparison of our method with other methods described (Sivaraman et al.
2020 Nolte
al. 2022
Runje & Shankaranarayana 2023)
(" indicates the statistical tie with the best.)
Auto MPG
Heart Disease
Method
MSE 4
Test Acc f
Min-Max Net
10.14 + 1.54
0.75 0.04
DLN
13.34 + 2.42
0.86 + 0.02
COMET
8.81 + 1.81
0.86 + 0.03
LMN
7.58 - 1.20f
0.90 + 0.02
Constrained MNN
8.37 0.08
0.89 0.00f
SMNN (Ours)
7.44 1.20
0.88 + 0.04f
proposed methods, with a relatively small network size. Table 3 also includes the experimental results obtained from the previous works (Sivaraman al. 2020; Nolte al., 2022 Runje & Shankaranarayana] 2023). In the context of the Auto-MPG dataset, our proposed method and LMN outperformed all others in the comparison, while, for the Heart Disease dataset, our approach did not achieve the level of performance attained by LMN, which secured the top position. When it came to the classification tasks, our method achieved the top result a specific case, while in other cases, performed comparably to the benchmarks.
5 CONCLUSIONS
In this study, we introduced SMNN, a simple but novel learning method designed to ensure the monotonicity of trained models. Our approach combines the exponentiated unit, ReLU unit, and confluence unit in a hidden layer to achieve both monotonicity and effective function approximation. Being an end-to-end learning approach, our method exhibits scalability with respect both network size and number of monotonic features. Through extensive experiments on synthetic and real-world datasets, we demonstrated the scalability and comparable performance of our method against other benchmarking methods. For future work, we will focus mathematically proving the benefits of incorporating monotonicity as an inductive bias in neural networks to enhance their generalization performance. Additionally, we plan to apply SMNN to a broader range of real-world applications that require both monotonicity and scalability.
REPRODUCIBILITT
To facilitate reproducibility, we have provided sufficiently detailed descriptions of the implementation of the proposed method both in the main text and appendices. In particular, in Section 3 we clearly
present the complete structure of our proposed method along with its components. Appendix A provides detailed experimental descriptions and training configurations. All implemented code be found in the supplements, which also available at btttt::: htap::///ithtbb com/ /ttp:/tennnn/9111 SMNN Experiments were conducted using the provided code.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported part by the National Research Foundation of Korca(NRRF) funded by the Korean Govermment (Ministry of Science and ICT (MSIT) under Grant 2020R 1019 649 and in part by the Institute for Information and Communications Technology Planning & Evaluation (ITTP) funded by the Korean Government (MSIT) under Grant 20210002920012022.
REFERENCES
Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems, 34:4699-4741 2021.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias: There's 's software used across the country to predict future criminals. and it's biased against blacks. propublica, 23 may, 2016.
Norman P Archer and Shouhong Wang. Application of the back propagation neural network algorithm with monotonicity constraints for two-group classification problems. Decision Sciences, 24(1): 60-75, 1993.
Sercan Ö Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6679-6687, 2021.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for supersuue learning of speech representations. Advances in neural information processing systems, 33:1 21-22244922442 2020.
Clark Barrett and Cesare Tinelli. Satisfiability modulo theories. Springer, 2018.
Christopher Bartley, Wei Liu, and Mark Reynolds. Enhanced random forest algorithms for partially monotone ordinal classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3224-3231, 2019.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Tianqi Chen, Tong Hc, Michael Benesty, Vadim Khotilovich, Yuan Tang. Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, et al. Xgboost: extreme gradient boosting. R package version 0.4-2, 1(4):1-4, 2015.
Hennie Daniels and Marina Velikova. Monotone and partially monotone neural networks. IEEE Transactions Neural Networks, 21(6):906 99700 2010.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. arXiv preprint arXiv:1605.0 08803, 2016.
Michael Doumpos and Constantin Zopounidis. Monotonic support vector machines for credit risk rating, New Mathematics and Natural Computation, 5(03):557-570, 2009.
Richard Dykstra, Tim Robertson, and Farrol T Wright. Advances in Order Restricted Statistical Inference: Proceedings of the Symposium on Order Restricted Statistical Inferecce Held in lowa City, lowa, September 11-13, 1985, volume 37. Springer Science & Business Media, 2012.
Jerome H Friedman. Multivariate adaptive regression splines. The annals of statistics, 19(1):1-67, 1991.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144 2020.
Akhil Gupta, Naman Shukla. Lavanya Marla, Arinbjöm Kolbeinsson, and Kartik Yellepeddi. How to incorporate monotonicity in deep networks while preserving flexibility? arXiv preprint arXiv: /909. 10662, 2019.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Adam Tauman Kalai and Ravi Sastry. The isotron algorithm: High- High-dimensional isotonic regression. In COLT, 2009.
Ivan Kobyzev, Simon JD Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on paltern analysis and machine intelligence, 43 (11):3964 3979, 2020.
Rasmus Kyng, Anup Rao, and Sushant Sachdeva. Fast, provable algorithms for isotonic regression in all Lp-norms. Advances in neural information processing systems, 28, 2015.
Bernhard Lang. Monotonic multi-layer perceptron networks as universal approximators. In Artificial Neural Networks: Formal Models and Their intiicattios-ICAAN 2005: 15th International
Conference, Warsaw, Poland, September 11-15, 2005. Proceedings, Part 15, pp. 31-37. Springer, 2005.
Shiyu Liang Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribttion image detection in neural networks. arXiv preprint
2017.
Shan Sung Liew, Mohamed Khalil-Hame and Rabia Bakhteri. Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems. Neuro- computing, 216:718 187744, 2016.
Xingchao Liu, Xing Han, Na Zhang, and Qiang Liu. Certified monotonic neural networks. Advances in Neural Information Processing Systems, 33:15427-15538, 2020.
Dan Mikulincer and Daniel Reichman. Size and depth of monotone neural networks: interpolation and approximation. Advances in Neural Information Processing Systems, 35:5522-5534, 2022.
Mahdi Milani Fard, Kevin Canini, Andrew Cotter, Jan Pfeifer, and Maya Gupta. Fast and flexible monotonic functions with ensembles of lattices. Advances in neural information processing systems, 29, 2016.
João Monteiro, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Greg Mori. Not too close and not too far: Enforcing monotonicity requires penalizing the right points. eXplainable Al approaches for debugging and diagnosis., 2021.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Niklas Nolte, Ouail Kitouni, and Mike Williams. Expressive monotonic neural networks. In The Eleventh International Conference on Learning Representations, 2022.
Rob Potharst and Adrianus Johannes Feelders. Classification trees for problems with monotonicity constraints. ACM SIGKDD Explorations Newsletter, 4(1):1-10, 2002.
Joseph Redmon and Ali Farhadi. 1801021804.02767,1 2018.
Yolov3:
An incremental improvement. arXiv preprint
Cynthia Rudin, Caroline Wang, and Beau Coker. The age of secrecy and unfairness recidivism prediction. Harvard Data Science Review, 2(1):1, 2020.
Davor Runje and Sharath M Shankaranarayana. Constrained monotonic neural networks. In Interna- tional Conference on Machine Learning, pp. 29338-29352 PMLR, 2023.
Joseph Sill. Monotonic networks. Advances neural information processing systems, 10, 1997.

---

Published as a conference paper at ICLR 2024
Aishwarya Sivaraman, Golnoosh Farnadi, Todd Millsteinn and Guy Van den Broeck. Counterexample- guided learning of monotonic neural networks. Advances in Neural Information Processing Systems, 33:11936-11948 2020.
Myra Spiliopoolouu, Lars Schmidt-Thieme and Ruth Janning. Data analysis, machine learning and knowledge discovery. Springer, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Hiroki Yanagisawa, Kohei Miyaguchi, and Takayuki Katsuki. Hierarchical lattice layer for partially monotone neural networks. In Advances in Neural Information Processing Systems, 2022.
Seungil You, David Ding, Kevin Canini, Jan Pfeifer, and Maya Gupta. Deep lattice networks and partial monotonie functions. Advances in neural information processing systems, 30, 2017.
Hong Zhang and Zhen Zhang. Feedforward networks with monotone constraints. IJCNN' International Joint Conference Neural Networks. Proceedings (Cat. No. 99CH36339), volume 3, pp. 1820-1823. IEFE, 1999.
12

---

Published as a conference paper at ICLR 2024
A DETAILED EXPERIMENT DESCRIPTIONS
A.1 TRAINING CONFIGURATIONS
All experiments were conducted on a system equipped with an Intel(R) CTr)(TM) i7-8700 CPU @ 3 Hz20GHz processor and 32.0GB of DDR3 RAM, running the Windows 10 operating system. The experiments were implemented in Python, utilizing the Py Torch tibrary 1.10.2). Stochastic optimization for training networks performed using the Adam optimizer. The best hyperpa- rameters were determined through grid search, considering different batch sizes (128, 256, 512), learning rates (0.05, 0.005, 0.002, 0.0005). The neural network architecture consisted of two scalable monotonic hidden layers. We consistently set the hyperparameter n in the ReLUU~ activation func- tion throughout all our experiments, because the primary purpose with this activation function was to utilize the convex and concave properties, making detailed tuning of this value unnecessary. For regression tasks, the mean squared error loss was employed, while for classification tasks, the cross-entropy loss was used. The number of epochs was set to either 1000 or 500 considering datasets.
A.2 DESCRIPTIONS FOR REAL-WORLD DATASETS
Table provides the detailed descriptions for the real-world datasets used in our experiments. It includes the tasks, number of instances, number of features, number of monotonic features, and names of the monotonic features. Monotonically non-increasing features are denoted in bold, while monotonically non-decreasing features are non-bold. For monotonically non-increasing features, we multiplied the features by -1 ensure monotonic non-decrease. Additionally, we normalized all inputs to a 0-1 scale. The best hyperparameter configurations for each dataset are presented in Table 5 This table shows the network structure with two scalable monotonic layers, previously mentioned. The numbers within parentheses represent the node counts for each exponentiated unit, confluence unit, and ReLU unit.
Table 4: Descriptions for real-worle datasets
Datasel
Task
Instances
Features
Monotonic features
Monotonic features
number of prior adult convictions,
numher juvenile felony, number of juvenile misdemeunor, number of other convictions
COMPAS
Classification
6172
13
4
Heart Disease
Classification
303
13
2
trestbps, chol
number of public record bankruptcies, dept-to-income ratio. credit score.
Loan defaulter
Classification
488909
28
5
length of employment, annual income
Blog feodback
Regression
54270
276
8
A51.A522.533A4444555.555
Auto-MPG
Regression
398
7
3
weights, displacement, horse power
Table 5: Hyperparameters of SMNN for real-world datasets
Dataset
Parameters
Network structure
Learning rate
Batch size
COMPAS
2657
(32, 16, 16), (32, 16, 16)
0.005
128
Heart Disease
7905
(64, 12), (64, 16, 32)
0.002
128
Loan Defaulter
501
(16, 4,4). (8, 2,4
0.0005
512
Blog Feedback
1421
(16, 2,2). 2,4)
0.0005
256
Auto-MPG
14193
(128, 16, 64). (64, 16,32) 32)
0.005
128
The COMPAS dataset is classification dataset containing the criminal records in Florida. The task is to classify whether an arrested individual will reoffend or not within two years.
COMPAS:
https://ppytrrch.org//
13

---

Published as a conference paper at ICLR 2024
This dataset consists of 13 features, with four of them considered as monotonic features. Specifically, the following features exhibit monotonic non-decreasing behavior: number of prior adult convictions, number of juvenile felon,, number of juvenile misdemeanor and number of other convictions. It is worth mentioning that this dataset has been used in several published papers (Angwin et al., 2016) Rudin al. 2020), although with minor ethical concern. While few individuals perceive it as possibly problematic, the majority do not share the same concern.
The Heart Disease dataset used to classify whether a person has heart disease or not. The presence of heart disease monotonically increases with respect to fresthps (resting blood pressure) and chol (cholesterol level).
Heart Disease:
Loan Defaulter: The Loan Defaulter dataset includes complete loan data issued between 2007 and 2015 and addresses a classification task aimed at predicting loan defaulters using 28 features. Among these 28 features, five exhibit monotonic behavior. Specifically, the number of public record bankruptcies and dept-to-income ratio are monotonically non-decreasing, while the credit score, length of employment, and annual income are monotonically non-increasing.
The Blog Feedback dataset is regression dataset used to predict the number of comments within hours. It comprises 54,270 blog post records and includes 276 features. Eight of these features (A51, A52, A53, A54, A56, A57, A58, A,59) exhibit monotonically non-decreasing behavior.
Blog Feedback:
The Auto-MPG dataset regression dataset used predict miles per gallon (mpg). consists of features, and among them, weights, displacement, and horse power need to be enforced as monotonically non-increasing.
Auto-MPG:
A.3 WEIGHT INITIALIZATION AND UPDATE STRATEGY FOR SMNN
In the proposed SMNN method, the exponentiated unit incorporates the exponential value of weights and utilizes the ReLU-n activation function, which has a threshold at and n. This design can lead to node inactivation when the activation function's gradient becomes zero at high input values due the influence of the exponentiated weights. To mitigate this potential risk, we have implemented specific weight initialization and weight update strategies. Regarding the initialization strategy, instead of generating initial weights from the typical Gaussian distribution, we opted for a uniform distribution spanning from negative lower limit Consequently, the exponentiated values of these weights are distributed in proximity to zero, creating skewed long-tail ttil distribution. This distribution shape effectively prevents the occurrence of zero gradients in the activation function. Concerning the update strategy, we intentionally set the gradient of inactivated nodes a very small non-zero value, such as 0.01. By ensuring non-zero gradient, we enable the exploration of alternative optimal weight configurations and minimize the risk of getting trapped in local optima. These strategies are designed to enhance the search for better optimal weights by mitigating the possibility of node inactivation caused by large input values and zeto gradients.
small positive upper limit, such as uni form(-20, 2).
14

---

Published as a conference paper at ICLR 2024
B
SUPPLEMENTAL REPORT OF EXPERIMENTAL RESULTS
B.1 TABULATED NUMERICAL RESULTS FOR FIGURE B)
Figure bb was created using the data in the following table.
Table 6: Mean MSE at varying number of parameters
Network
# Parameters
Test MSE
Time (s)
SMNN (size 1)
165
0.111 + 0.147
28.220 + 1.219
SMNN (size 2)
337
+ 1047=0.11
28.370 + 1.114
SMNN (size 3)
1025
0.019 1 0.038
29.180 1 2.019
SMNN (size 4)
3585
0.018 0.050
29.296 1 1.170
SMNN (size 5)
13697
0.009
31.889 + 1.948
B.2 TABULATED NUMERICAL RESULTS FOR FIGURE 2
Figure c) was created using the data following table.
Table Mean MSE at varying number of monotonic features
Network
Monotonic Features
Test MSE
Time (s)
SMNN (m 1)
(oia}
12.80 = 1.78
21.4 + 1.14
SMNN = =2)
{ {a1 25, #13. Ø17}
10.61 4.57
0.00
SMNN (m =5)
9.83 4.32
21.8 1.09
SMNN =10)
{I1, F3, 25, 27, 2g, #11, T13 I15, T17, #19
6.31 #
20.8
SMNN (m 200)
(x1...,, T20
1.51 + 1.03
2.64
B.3 TABULATED NUMERICAL RESULTS FOR FIGURE
Figure was created using the data in the following table.
Table 8: Results from the Friedman function experiments
Network
Training MSE
Test MSE
Denoised Test MSE
SMNN (A = 0)
0.68 + 0.42
0.78 + 0.50
0.78 + 0.50
SMNN (A 2)
4.85 0.23
5.42 0.40
1.25 0.28
SMNN (A = 5)
26.53 + 0.65
28.89 3.05
2.22 + 0.47
SMNN (A 10)
100.64 = 1.73
107.46 6.45
4.02 1 0.79
SMNN (A -0)
412.44 = 12.98
427.92 = 41.84
7.73 1.55
Same Structure (A
0.02 +.02=0000 0.01
0.03 0.02
0.02
Same Structure (A =2 2)
3.27 0.17
4.63 0.41
0.784..24
Same Structure
18.81
31.46
6.03
Same Structure (A 10)
71.69 + 6.68
134.16+ 10.80
25.69 5.883
Same Structure 20)
273.58 27.75
530.27 + 60.01
118.02 28.31
MLP (A = 0)
0.02
0.04 0.02
0.04+0.02
MLP (A 2)
2.53 0.29
5.18 1 0.46
1.28 0.227
MLP (A 5)
11.77 L 2.13
37.01 1 4.10
12.73 2,45
MLP (A- 10)
32.92 8.60
184.95 + 21.64
63.61 9.63
MIP (A 20)
118.95 + 30.99
687.04 + 74.25
270.01 = 42.59
15

---

Published as a conference paper at ICLR 2024
C ABLATION STUDIES
C.I DIFFERENT ACTIVATION SETTINGS
In our experiments, we applied the ReLU-1 function to both the exponentiated and confluence units, and the standard ReLU function to the ReLU unit (denoted by 'Proposed (Ours)" in Table We believe that it is acceptable to use the ReLU-1 function for the ReLU unit maintain consistent activation function throughout the network (denoted by 'ReLU-1' in Table Since the ReLU-n function offers convexity and concavity, other activation functions providing the similar benefit can replace ReLU-n. In this ablation study, we explored the use of the sigmoid and hyperbolic tangent functions for three units (identified as 'Sigmoid' and 'Tanh', respectively, in Table Based on the results, we cannot assert performance differences among the activation functions.
Table 9: Performance comparison among different activation settings (* indicates the statistical tic with the best.)
Auto MPG
Heart Disease
Method
MSE 4
Test Acc f
Proposed (Ours)
7.44 1.20
0.88 = 0.04
ReLU-1
= 1.36+1.00
0.85 = 0.04f
Sigmoid
13.96+3..1
0.85 + 0.07f
Tanh
7.83
0.87 = 0.05f
C.2 DIFFERENT VALUES IN RELU-n
While an optimal value for the byperparameter n in ReLU-n may exist for specific training dataset, to demonstrate its insensitivity, we conducted experiments with varying based on the Auto MPG and Heart Disease datasets. The results are presented in Table TO, showing no performance differences across different n values.
Table 10: Performance comparison among different n values in ReLU-n t"t" indicates the statistical tie with the best.)
Auto MPG
Heart Disease
Method
MSE
Test Acc
ReLU-1
7.44 + 1.20
0.88 1 0.04
ReLU-2
7.40 1.31
0.87
ReLU-6
7.38
0.87
ReLU-10
7
0.87 0.04
C.3 EFFECT OF CONFLUENCE UNITS
We conducted ablation study validate the impact of the confluence units in learning interactions. The function designed for the experiment is defined in equation [3] The output is determined by T1 and 22- To control the strength of the interaction effect, we varied the coefficient a of the interaction term from 0 to 20. We imposed monotonicity on only. even though 22 is monotonic (0. We generated 1000 instances, allocating 700 instances for training and 300 for the test set. The number of hidden layers for both networks, with and without confluence units, was set to two. However, the number hidden nodes was assigned differently: {(64, 64, 64), (32, 32, 32)} for SMNN with confluence units, {128,, 0, 64), (64, 0, 32)} for SMNN without confluence units. The MSE comparison in Table illustrates that with increased interaction effects, more prominent performance differences emerged, consistently favoring SMNN with confluence units. This validates the effectiveness of employing the confluence unit to capture interactions between monotonic and non-monotonic features.
16

---

Published as a conference paper at ICLR 2024
f(11, = 11 + 2t. 22,
(13)
31, 32 E [0, a E [0, 2,5, 10, 20]
Table 11: Performance (MSE) comparison between with and without confluence units
Siee via carf jnls
Woan
ar
with confluence units
without confluence units
a 0
0.46 0.05
0.96
1
0.58 0.05
8.35 0.20
2
0.95 0.15
27.11 = 1.05
5
3.87 1.08
232.58 1.78
a 10
5.72 1.19
703.62 +: 19.29
Figure 4: Visualization of the results in Table
a 20
22.99 15.52
2341.43 52.46
C.4 ADDITIONAL SCALABILITY TEST WITH RESPECT TO NUMBER OF MONOTONIC FEATURES
We conducted an additional scalability test with respect to the number of monotonic features, for the case where the number of monotonic features is dominantly larger than the number of non- monotonic features. Specifically, we created a 220-dimensional dataset, excluding the output feature, consisting of 20 uniform distribution-basee dummy inputs and 200 monotonic inputs achieved by expanding each group of features of equation TT from 5 to 50 features. Notably, in this experiment, the dummy features exert much less influence during training compared our original experiment.
The training time and MSE results are shown in Table 12 and Fig We observed the consistent trend of training times remaining constant while MSE decreased as the number of monotonic inputs increased.
Table 12: Scalability test with respect increasing number of monotonic feature when m 200
Network
# Parameters
Test MSE
Time (s)
SMNN (m =10)
29414
0.204
78.76 2.22
SMNN (m =20)
29977
0.216+0.12
78.40 : 2.57
SMNN (m = =50)
31657
0.175+0.009 0.09
78.76 1.51
SMNN (m 100)
34457
0.117++000
79.08
2.06
SMNN (m 200)
40057
0.013
78.84 + 2.01
0.40
Mean MSE
Mean Time (s)
0.35
0.30
Mean Time (s)
Mean MSE
0.25
60
0:15
-00
20
0.05
0.00
Number of Monotonic Features
Figure 5: Visualization of the results in Table
12
17


Reviews:
[{"review_id": "wKSnTNd7U9", "reviewer": "Reviewer_tddM", "summary": "This work proposes Scalable Monotonic Neural Network (SMNN) to learn neural networks that preserve monotonicity of a model w.r.t. a subset of inputs. Existing works require additional overhead in terms of inference cost, weight constraints during training, scalability issues w.r.t. network size and number of monotonic inputs, etc. This work bakes in the monotonic properties directly into the network architecture by designing hidden layers with three different units ( exponential unit, ReLU unit and confluence unit ). Exponential unit preserves the monotonic nature of its input and are explicitly tied to the monotonic inputs. Remaining inputs go through the confluence and the ReLU unit, while the output of the confluence unit also goes through addition with the input for the next exponential unit (see Figure 1). Thus, monotonic inputs pass through a unique path in the network that preserves their monotonic nature. Empirical evaluation demonstrates that this method achieves comparable performance to the existing state-of-the-art and helps eliminate many issues arising in those works.", "questions": "- Does restricting interaction between monotonic and non-monotonic inputs hurt expressivity of this network?\n- How does one incorporate other activations/non-linear operators in this architecture?\n- How do you select the parameter n in ReLU-n activation?\n- Do you have any comparison on train/test time for the baselines with the SMNN architecture? Does it take longer to converge compared to existing methods?\n- Why are parameters missing for XGBoost and Isotonic methods in Table 2?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Interaction of the monotonic inputs is segregated from the other inputs through a dedicated path in the neural network. This ensures the monotonic nature as these inputs pass through exponential units in the hidden layers \n- Empirical experiments show the viability of such a simple approach to enforcing monotonicity", "weaknesses": "- Only performs on-par as the existing methods for enforcing monotonicity in neural networks \n- Its unclear if there's any computational advantage (in terms of training/inference cost) compared to existing baselines\n- Limited discuss on extensions to other activations or expressivity of the network", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eGGfdC8jAb", "reviewer": "Reviewer_sRQA", "summary": "The paper presents an architecture for a neural net that guarantees the monotonicity of the output wrt a subset of predefined inputs, and still allows interactions between features derived from all inputs.", "questions": "1. To disentangle the effects of the connectivity pattern from the ones of monotonous parametrization, I'd be curious to see the performance of a network composed only of Exponential units (all fully connected), but where the non-monotonous inputs would be duplicated as [x_¬m, -x_¬m]. Is that something you tried?\n\nUpdate after discussion\n--------------------------------\nMany points have been addressed, so I'm moving my score from 6 to 8.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n--------------\nThe method is related to several existing pieces of work, but the design of the specific architecture, enabling the integration of information from the non-monotonous inputs in a principled way, at every level, is original to my knowledge.\nThe architecture is generic enough that it should be applicable in a wide variety of settings.\n\nQuality\n----------\nExperiments are well designed and match well with the (implicit) research questions, demonstrate well the behaviours of the algorithm, in particular:\n1. when scaling up the number of parameters\n2. when scaling up the number of monotonic features\n3. when increasing the noise (compared to less-constrained or regular MLPs)\n\nExperiments on real datasets show that this method is at least competitive with state-of-the-art methods.\n\nClarity\n---------\nThe paper is clearly written, does not pose any notable challenge for comprehension. The method is defined clearly enough to be reimplemented from the paper, even without the provided source code.\n\nSignificance\n-----------------\nA scalable, end-to-end learning method guaranteeing the monotonicity wrt some inputs would have an impact for ML application where interpretability, trust, or predictability are necessary.", "weaknesses": "Quality\n----------\n1. Regarding the theoretical aspect, and the proof of Theorem 1, an issue is that ReLU-n is not a differentiable function, contrary to the definitions in section 2. Its derivative is not defined at 0 or n. I think the conclusion is still correct due to it being continuous, and that both the left-derivative and the right-derivative at every point exist and are >=0, but the proof and definition should be updated.\n2. The scaling of compute time wrt number of monotonic features cannot really be extrapolated from varying the dimensions from 1 to 20. It's likely that at this scale, the execution time is dominated by constant factors, and it's unclear how it would actually scale to \"high-dimensional monotonic features\", compared to Certified MNN or COMET for instance. Due to the architecture, it is likely that the scaling time _per training step_ would scale up as well as a regular MLP, however optimization issues could happen and slow down convergence, for instance. It would be more effective to demonstrate the scalability of this method by applying it on a problem too big for Certified MNN or COMET.\n3. In Tables 2 and 3, statistical ties should be bolded, and be assessed by a statistical test taking into account the variance of _both_ distributions. Assuming the ± numbers indicate 95% CIs, there should be many more ties, and the conclusions about SMNN outperforming all other methods on regression tasks to not really hold. For instance, on AutoMPG, 7.44±1.20 and 7.58±1.20 should clearly overlap.\n4. The COMPAS dataset is used despite ethical concerns, since these features were used by the original COMPAS system to produce a score that was unfair and biased. This work uses the dataset despite not explicitly aiming at producing unbiased decisions, or examining critically the trained system.\n\nClarity\n---------\n1. In Table 1, maybe indicate which methods are categorized as \"regularization\", and which ones are \"customized architectures\". Or, if \"customized architecture\" is synonymous with \"end-to-end learning\", maybe indicate it in the caption.\n2. The circled \"+\" signs in Figure 1 is confusing if the operation is concatenation, not addition. Maybe it would be clearer to have both arrows point directly at the next \"Exp unit\"? (Idem for the fc layer)\n3. Why are Table 2 and 3 separate? Both have a mix of regression and classification datasets, but they have a different set of methods, and those in Table 3 do not report parameter counts.\n4. Not clear at first what \"subjected to denoising\" (S. 4.1) means.\n5. Clarify what the \"±\" numbers in the table means: standard deviation, confidence interval?\n\nSignificance\n-----------------\nThe small scale of experiments is a limitation. Given that the ambition of the algorithm is to be \"scalable\", I think it should try to demonstrate scaling where some other algorithms are limited.\n\nMinor points\n-----------------\n1. Datasets should have citations or footnotes when they are introduced in the main text, not only in the appendix. Maybe also refer to the appendix explicitly for which features were considered monotonic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vHxl9lg4Dm", "reviewer": "Reviewer_iEFi", "summary": "The paper tackles the inductive bias of monotonicity in Neural Networks. The authors propose a module based on nonnegative weights with a ReLU-n activation to enforce monotonicity. The proposed architecture comprises three types of hidden layers to handle monotonic and non-monotonic inputs.", "questions": "- Could you clarify what this sentence means: “Nevertheless, it should be noted that an optimally trained network is not always guaranteed with respect to the Lipschitz constant λ, particularly for larger constants”?\n- I am unsure why scalability is considered such a central point in the paper. Some of the other methods are clearly just as scalable, including LMNs and constrained monotonic networks. Am I missing something?\n\n**********Nits:**********\n\n- Missing slash for $\\exp$ instead of $exp$ in the proof of theorem 1 on page 5.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is very clear and easy to follow.\n- The method itself is simple and easy to implement.\n- Variety of experiments across several domain-specific datasets and toy problems.", "weaknesses": "- It could be argued that the novelty of the approach is limited since Mikulincer & Reichman (2022) already describe a very similar architecture and prove the universality of 4-layer threshold networks with nonnegative weights in approximating monotonic datasets. ReLU-n is a continuous relaxation of the threshold function, so it is likely that universality holds here as well for the same reasons and might possibly offer better convergence properties, but this is not explored.\n- There seems to be a big focus on scalability, but it’s unclear how this approach is better than current state-of-the-art models in this respect. The experiments do not compare against other models, and for good reason, I imagine; there is little difference in the scalability of the proposed method and existing works in the literature. So, this begs the question of why scalability is a central thesis in the paper.\n\nOverall, this paper feels like yet another approach to the inductive bias of monotonicity in neural networks. It has no big flaws, but the advantages do not appear to be substantial compared to current methods. The approach is simple enough and is worth considering, but the paper is slightly below the acceptance threshold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aW8GfAmW8L", "reviewer": "Reviewer_y34F", "summary": "Authors propose a new method, Scalable Monotonic Neural Networks (SMNN), to learn monotonic neural networks. Monotonic features are propagated forward using exponentiated units, which guarantees monotonicity. Non-monotonic features are propagated forward through confluence units and ReLU units, where confluence units capture interactions between monotonic features and non-monotonic features, and ReLU units capture interactions among non-monotonic features. The authors provide a theoretical result to verify the monotonicity of SMNN and experimental results to demonstrate the performance of SMNN.", "questions": "1. On page 2, the authors claim that the assurance of monotonicity is not always guaranteed as a regularization method. This is true but I think this should not be treated as the drawbacks of regularization methods. The monotonicity usually comes from experience and may not be the truth. When we prefer to believe the experience, we can use methods in the first group to guarantee monotonicity. But when the experience contradicts collected data, then regularization provides a chance to balance between past experience and new data. Thus, I think it is more suitable to say these two groups of methods have their applications in the real world rather than treating not guaranteed monotonicity as a drawback.\n2. Authors provide two definitions for partial monotonicity, one using function values and one using partial derivatives. However, it seems that the definition using function values is not correct. I think the correct one should be \"The function $f(\\boldsymbol{x})$ is partially monotonically non-decreasing on $\\boldsymbol{x}_m$ iff $\\forall i, x_i \\leqslant x_i' , \\forall j \\neq i , x_j = x_j' \\Rightarrow f(\\boldsymbol{x}) \\leqslant f(\\boldsymbol{x}')$. Take a 2-dimensional case as an example. Let the 1st coordinate be the monotonic feature, and the 2nd is not. Let $f(x_1 , x_2) = x_1 + x_2^2$. Then $f$ is partially monotonic according to the definition using partial derivatives but is not partially monotonic according to the definition using function values.\n3. On page 4, the authors claim that \"ReLU is commonly used as an activation function due to its advantages such as avoiding gradient vanishing\". ReLU helps address the problem of gradient vanishing (compared with sigmoid) but it cannot \"avoid\" gradient vanishing. When the weight parameters are sufficiently small and the number of layers is large, ReLU also faces the problem of gradient vanishing, and other techniques are needed to help address the problem of gradient vanishing. Thus, it would be better to use the verb \"alleviate\" rather than \"avoid\".\n4. In Eq. (5), $h_{con}$ should be $h_{conf}$.\n5. In the current SMNN structure, both the exp unit and conf unit use the ReLU-n activation function for the purpose of universal approximation and aligning output magnitudes. But in the ReLU unit, ReLU is used rather than ReLU-n. I am wondering what will happen if all these three units use the ReLU-n activation function. I think a unified activation function makes the structure more succinct.\n6. In theorem 1, $x$ belongs to $\\boldsymbol{x}_m$. To my understanding, $\\boldsymbol{x}_m$ is a vector rather than a set, which indicates that the belonging relation is not rigorously defined. I think $x$ is a coordinate of the vector $\\boldsymbol{x}_m$. A suitable way is to define the subscript set of $\\boldsymbol{x}_m$ as $M$, and then use $x_i$ with $i \\in M$.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The investigated problem, learning monotonic neural networks, is important in many fields, where some features are believed to have a positive or negative impact on the concerned output.\n2. To the best of my knowledge, the proposed SMNN is novel.\n3. The proposed SMNN is succinct. It is easy to understand as it only consists of commonly used activation functions, exponentiated weights, and a two-group architecture. Thus, SMNN can be trained by traditional backpropagation algorithms. It is also intuitive that SMNN can guarantee monotonicity.", "weaknesses": "1. The authors review previous methods in detail in the introduction part but the introduction might be too long and Table 1 does not fully distinguish SMNN and other methods. In Table 1, LMN, constrained MNN and SMNN have the same characteristics. It would be better to summarize the difference between SMNN and other methods in Table 1, which helps readers grasp the advantage of SMNN without reading through the long introduction.\n2. The authors provide many experiments but it seems that the proposed SMNN has a similar performance to LMN. There are 5 datasets. SMNN wins on 2 of them, loses on 2 of them, and is comparable on the last one.\n----\nThe authors' answers are convincing and I have updated my ratings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)