Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
Authors: Davide Sartor
Alberto Sinigaglia
Gian Antonio Susto
Venue: ICML 2025 poster (2025)
Keywords: Monotonic
Neural Network
Primary area: deep_learning->theory
TL;DR: This paper proves that constrained MLPs are universal approximators for a broader class of activations.  This includes most modern activations, even convex ones like ReLU. A novel parametrization is introduced, reducing sensibility to initialization.

Abstract: Imposing input-output constraints in multi-layer perceptrons (MLPs) plays a pivotal role in many real world applications. Monotonicity in particular is a common requirement in applications that need transparent and robust machine learning models. 
Conventional techniques for imposing monotonicity in MLPs by construction involve the use of non-negative weight constraints and bounded activation functions, which poses well known optimization challenges.  
In this work, we generalize previous theoretical results, showing that MLPs with non-negative weight constraint and activations that saturate on alternating sides are universal approximators for monotonic functions.  
Additionally, we show an equivalence between saturation side in the activations and sign of the weight constraint. This connection allows us to prove that MLPs with convex monotone activations and non-positive constrained weights also qualify as universal approximators, in contrast to their non-negative constrained counterparts.  
This results provide theoretical grounding to the empirical effectiveness observed in previous works, while leading to possible architectural simplification.  
Moreover, to further alleviate the optimization difficulties, we propose an alternative formulation that allows the network to adjust its activations according to the sign of the weights. This eliminates the requirement for weight reparameterization, easing initialization and improving training stability.  
Experimental evaluation reinforce the validity of the theoretical results, showing that our novel approach compares favorably to traditional monotonic architectures.

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
Davide Sartor" 11
Alberto Sinigaglia 2 Gian Antonio Susto
Abstract
25
Conventional techniques for imposing monotonic- ity in MLPs by construction involve the use of non-negative weight constraints and bounded ac- tivation functions, which pose well-known opti- mization challenges. In this work, we generalize previous theoretical results, showing that MLPs with non-negative weight constraint and activa- tions that saturate on alternating sides are univer- sal approximators for monotonic functions. Ad- ditionally, we show equivalence between the saturation side in the activations and the sign of the weight constraint. This connection allows us to prove that MLPs with convex monotone acti- vations and non-positive constrained weights also qualify as universal approximators, in contrast to their non-negative constrained counterparts. Our results provide theoretical grounding to the em- pirical effectiveness observed in previous works while leading to possible architectural simplifica- tion. Moreover, to further alleviate the optimiza- tion difficulties, we propose an alternative formu- lation that allows the network to adjust its activa- tions according the sign of the weights. This eliminates the requirement for weight reparam- eterization, easing initialization and improving training stability. Experimental evaluation rein- forces the validity of the theoretical results, show- ing that our novel approach compares favourably to traditional monotonic architectures.
26-
24-
Figure I. Monotone MLPs with weight-constraint and bounded activations (pink) and our proposed approach based on ReLU (blue). The former can only represent bounded functions and, thus, cannot extrapolate the data trend, which is important in many domains, such as time-series analysis and predictive maintenance.
1. Introduction
Monotonic neural networks represent pivotal shift in deep learning. They bridge the gap between high-capacity non- linear models and the need for interpretable, consistent out- puts in various applications. Monotonic MLPs preserve monotonic input-output relationships, making them particu- larly suitable for domains that require justified and transpar- ent decisions (Gupta et al., 2016; Nguyen & Martínez, 2019). Furthermore, monotonie MLPs have been exploited to build novel architectures for density estimators (Chilinski & Silva, 2020; Omi et al., 2019; Tagasovska & Lopez-Paz, 2019), survival analysis (Jeanselme et al., 2023), and remaining useful life (Sánchez et al., 2023). In general, the enforce- ment of constraints on the model architecture guarantees certain desired properties, such as fairness or robustness. Furthermore, explicitly designing the model with inductive biases that exploit prior knowledge has been shown to be fundamental for efficient generalization (Dugas et al., 2000; Milani Fard et al., 2016; You et al., 2017). For this reason, the use of monotonic networks can help both in improv- ing performance (Mitchell, 1980) and data efficiency (Veličković, 2019).
"Equal contribution 'Department of Information Engineer- ing. University of Padova. Padova (PD), Italy HHman In- spired Technology Research Centre, University of Padova, Padova (PD). Italy. Correspondence to: Davide Sar- tor <davide, sartor.4@pll curtoor4@phd.unip.... pnd.unipd.its Alberto Sinigaglia <al- berto. berto. songgiiiiigilaeee @ und.unipd.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
Recent works in this field usually fall into one of the follow- ing two categories: "soft monotonicity' and 'hard mono-
1. We show that constrained MLPs that alternate left- saturating and right-saturating monotonic activations can approximate any monotonic function. We also demonstrate that this can be achieved with a con- stant number of layers, which matches the best-known bound for gressold-activacee networks.
tonicity'. Soft monotonicity employs optimization con-
straints (Gupta et al., 2019; Sill & Abu-Mostafa, 1996), usu- ally as additional penalty terms in the loss. This class of ap- proaches benefits from its simple implementation and inex- pensive computation. They exploit the power of Multi-Layer Perceptrons (MLPs) to be able to approximate arbitrary func- usually applied to dataset sam-
2. Contrary to the noo-negative-onsstainn formulation, we prove that an MLP with at least 4 layers, non- cositive-constrained weights, and ReLU activation is a universal approximator. More generally, this holds true for any saturating monotonic activation.
tions. Since penalties ples, monotonicity is enforced only in-distribution, therefore struggling to generalize the constraint out-of-distributio
Hard monotonicity instead imposes constraints on the model architecture to ensure monotonicity by design (Wehenkel & Louppe, 2019; Nolte et al., 2023). The simplest way to do so is constrain the MLP weights to be non-negative and to use monotonic activations (Daniels & Velikova, 2010). The methods proposed in the literature that exploit this parametrization (Daniels & Velikova, 2010; Wehenkel & Louppe, 2019) require the usage of bounded activations, such as sigmoid and hyperbolic tangent, which introduce well-known optimization challenges due to vanishing gra- dients (Dubey et al., 2022; Ravikumar& & Sriraman, 2023; Szandala, 2021; Nair & Hinton, 2010; Glorot & Bengio, 2010; Goodfellow et al., 2013). This shortcoming is even more evident in monotonic NNs with non-negative weights, where bounded activations make the initialization even more crucial for optimization. As discussed in Appendix A.2, poor initialization can lead to saturated activations at the beginning of training, thus significantly slowing it down. Furthermore, the use of bounded activations leads to MLPs that can only represent bounded functions, which may hin- der generalization, shown in Figure 1.
3. We propose a simple parametrization scheme for mono- tonic ML.Ps that (i) can be used with saturating acti- vations; (ii) does not require constrained parameters, thus making the optimization more stable and less sen- sitive initialization; (iii) does not require multiple activations; (iv) does not require any prior choice of alternation of any activation and its point reflection.
Our discussion will primarily focus on ReLU activations. which are widely used in the latest advancements in deep learning. However, the results apply to the broader family of monotonic activations that saturate on at least one side. This includes most members of the family of ReLU-like activa- tions such as exponential, ELU (Clevert et al., 2016), ScLU (Klambauer et al., 2017), CELU (Barron, 2017), SReLU (Jin al., 2016), and many more.
2. Related work
Indeed, most recent advances in NNs use activations in the family of rectified linear functions, such as the popular ReLU activation (Vaswani, 2017; He et al., 2016). However, the use of ReLU activations in MLPs with non-negative weights is problematic. In fact, an MLP that uses con- vex activations (such as ReLU) in conjunction with non- negative weights can only approximate convex functions, which severely limits applications (Daniels & Velikova, 2010; Mikulincer & Reichman, 2022). For this reason, many approaches in the literature still rely on including bounded activations in the architecture in order to ensure universal approximation abilities of the network.
Monotonicity in neural network architectures is an active area of research that has been addressed both theoretically and practically. Prior work can be broadly classified into two categories: architectures designed with built-in constraints (hard monotonicity) and those employing regularization and heuristic techniques to enforce monotonicity (soft mono- tonicity). Our contribution falls into the former category.
2.1. Hard Monotonicity
Hard-monotonicii aims at building MLPs with provably monotonicity for any point of the input space. They do so by constructing the MLP so that only monotonic functions can learned. Initial attempts were Deep Lattice Networks (You et al., 2017) and methods constraining all weights to have the same sign exemplify this approach (Dugas et al., 2009; Runje & Shankaranarayana, 2023; Kim & Lee, 2024). However, constraining the parameters to be non-negative violates the original MLP formulation, thus invalidating the universal approximation theorem. Indeed, the universal ap-
The primary aim of this work is to extend the theoretical basis of monotonic-constraieee MLPs, by showing that it is still possible to achieve universal approximation using activations that saturate on one side, such as ReLU. To show that these new findings are not just theoretical tools, we create a new architecture that only uses saturating activa- tions with performances comparable state-of-the-art. Our contributions can be summarized as follows:
Code available at githuh.com/AMMCCO-iiiPDi
2

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
proximation capability of the architecture is proven only under the condition that the threshold function is used ac- tivation and that the network is at least 4 layers deep (Runje & Shankaranarayana, 2023). Furthermore, the non non-negative parameter constraint also creates issues from an initializa- tion standpoint, as the assumption for popular initializers might be violated for positive semidefinite matrixes. Only recently, new architectures have been proposed with novel techniques: adapting the Deep Lattice framework to MLPs (Yanagisawa et al., 2022), working with multiple activations (Runje & Shankaranarayana, 2023), and constraining the Lipschitz constant (Raghu et al., 2017). However, Runje & Shankaranarayana (2023) requires the usage of multiple ac- tivations, and an a priori split of the layer neurons between them, which might be suboptimal or require additional tun- ing, and Nolte et al. (2023) relies on very specific activa- tions to control such property, reported by the authors. In contrast, our work aims to overcome these drawbacks by im- proving flexibility without compromising the monotonicity constraint.
by simply inverting the sign of the inputs. 1
An MLP is defined as paramctrized function fo obtained as composition of alternating affine transformations i and non- non-linear activations og.
fo(x) 1 0 ...0g 09 - 1g.
(3)
A straightforward approach to building& provable mono- tonic MLP that respects Equation (1) is to impose con- straints on its weights and activations, forcing each term in Equation (3) to be monotonic. For affine transformations ig (z) W + 6t), we only need to enforce the Jacobian to be non-negative, which is simply the matrix W while monotonic activations are, by definition, monotonic.
To optimize the resulting MLP with unconstrained gradient- based approaches, the non-negative weight constraint is ob- tained using reparametrization. i.e. (ax) g(W")x + 8 for some differentiable g R + R_. Typical reparametriza- tions use absolute value or squaring.
2.2. Soft Monotonicity
3.1. Known universal approximation conditions
coft-monotonicity aims to build monotomic MLPs by mod- ifying the training instead of the architecture, either using heuristics or regularizations. Techniques such as woint-wise penalty for negative gradients (Gupta et al., 2019; Sill & Abu-Mostafa, 1996) and the use of Mixed Integer Linear Programming (MILP) for certification (Liu et al., 2020) have been proposed. These methods maintain consider- able expressive power but do not guarantee monotonicity. Additionally, the computational expense required for certifi- cations, such as those using MILP or Satisfiability Modulo Theories (SMT) solvers, can be prohibitively high.
Despite their surprising performance, one critical flaw of ex- isting MLP architectures based on weight constraints is the narrow choice of activation functions. Constrained MLPs have been shown to be universal function approximators for monotonic functions, provided the activation is chosen to be the threshold function, and the number of hidden layers is greater than the dimension of the input variable (Daniels & Velikova, 2010). Just recently, this result has been drasti- cally improved to constant bound, proving that four layers are sufficient to have universal approximation properties (Mikulincer & Reichman, 2022), when using the threshold function as activation. Therefore, practical implementations still resort to bounded activations such sigmoid, tanh, or ReLU6.
3. Monotone MLP
A function R: + R is said to be monotone non- decreasing with respect to v,, if given af of E &Rie [1, d), has the following property:
On the other hand, the use of convex activations like ReLU in a constrained MLP severely limits the expressivity of the network. To understand why this is the case, consider that: Proposition 3.2. The composition of monotonic convex functions itself monotonic convex.
of < m! f(x1, Li ................................ ma) < f(z), , a
ma)
(1)
And similarly, a function f Rd + R is said to be monotone non-increasing with respect to x; if:
Since affine transformations are simultaneously convex and concave, if the activation is chosen to be monotonic convex function, then the constrained MLP will only be able to approximate monotonic convex functions. Despite this clear disadvantage, there is interest in ReLU activated monotonic MLPs due to their properties and their perfor- mances shown in the unconstrained case (Glorot & Ben- gio, 2010; Hein et al., 2019). Runje & Shankaranarayana (2023) propose way to introduce ReLU activation in the
o < x S(x,..... ................................ 2. ... ma) 2 I [(m
ra).
(2)
Remark 3.1. Given (xxx), (zx) monotonic non-decreasing, and h(z), u(z) monotonic non-inereasing, f o g is mono- tonic non-decreasing, f fol h is monotonic non-increasing, and u o h is monotonic nonrdecra...
In this work. we will only focus on parametrizing non- decreasing functions, as the monotonicity can be reversed
'In the rest of the paper will use "monotonie" shorthand for "monotonic non-decreasing" ", unless otherwise specified
3

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
network. The architecture uses multiple activation functions derived from primitive activation (I)), its point reflection d'xx)= = --(x), and, in particular, a bounded sigmoid- like activation. While the ReLU activations are ignored in the subsequent theoretical analysis, this last bounded activa- tion function is used to ensure the universal approximation property.
the result derived in Mikulincer & Reichman (2022), while applying broader class of activation functions, Specif- ically, we will prove that 4 layers are sufficient, provided that the activations alternate saturation sides.
Definition 3.3. Given a function a : RR R saturates right/left if the corresponding limit exists and is finite. That is, a is right- segurattuu if (++x)) = lim_-++0 o (x) E R,
An upper-bound on the minimum required number of lay- ensure universal approximation of their architecture is derived from the result in Daniels & Velikova (2010). Although this bound scales linearly with the number of input dimensions, the authors observe good empirical per- formance with just a few layers. In this work, we extend the result of (Mikulincer & Reichman, 2022), showing that a constant number of layers is indeed sufficient, even when using oon-threshold activations. This result explains the em- pirical observations of Runje & Shankaranarayana (2023), and suggests that one of the three activations employed might not be necessary.
and it is saturaattu if (--00) = lim--+-oe (zx) E R.
We will denote the set of right-saturating activations as S+ and the set of left-saturating activations as S
Proposition 3.4. For every MLP with non-negative weights and activation o (x)). and for any E RR,bb E IR, there ex- ists an equivalent MLP with non-negante non-negative weights and acti- vation ao (x) + b.
In the following proofs, thanks to Proposition 3.4, we will only consider activations that saturate to zero.
The main result we will prove is the following.
Theorem 3.5. An MIP go .Rd - R with non-negative weights and 3 hidden layers can interpolate any mono- tonic non-decreasing function fxx on any set of v. points, provided that the activation functions are monotonic non- decreasing and alternate saturation sides. That is, in addi- tion to monotonicity, either of the following holds:
3.2. Universal approximation theorem for non-threshold activations
From Proposition 3.2, it follows that a constrained MLP with ReLU activations cannot be a universal approximator for monotone functions. However, two MI.P layers can approximate the Heavyside function arbitrarily well if the activations alternate between ReLU and its point reflection, as shown in Figure 2. From this observation, it is possible to use the result of (Daniels & Velikova, 2010) directly, show- ing that alternating activations between ReLU and its point reflection is sufficient to construct universal monotonic ap- proximators. However, the resulting bound on the required number of layers is linear in the input dimension.
af ESSSS, ,00 ES, a0 ES
(4)
ath Eest 00th ES 00 EST
(5)
The first step is proving that hidden units in the first layer can approximate piecewise-constant functions on specific half-spaces.
Lemma 3.6. Consider an arbitrary hyperplane defined by af (x - B) 0, E Rd and 3 E Rd, and the open 0}, = {x a" (I B) < 03. The i-th neuron in the first hidden layer of an MLP with non non-nieaeevv weights can approximate 2:
In Appendix A. use similar reasoning and the results of (Mikulincer& & Reichman, 2022) to achieve loose but constant bound of 8 layers. However, in this section, we will derive a tighter bound that instead matches and generalizes
half-spaces At = {z a" (x-
(+o0),
if A+
Activation function
Heav Side approximation
(x) (-oo), E
(0), otherwise
Proof. Denote by the weights and by b the bias associated with the hidden unit under consideration. Then, setting the parameters to w = Xa" and b = >aT B and taking the limit we have that:
0.4
02
0.0
(x)
lim 80 (wz + b) t
lim のt (Aa" (x B))
Figure 2. Constructions of Heavyside function using a com- position of ReLU and its point reflection ReLU' to obtain Rel.U(ReL.U (az 0.5) + 1) ReLU (Re.U(( +.5) 1)
A x
The limit is either "I' (+00), 'i' (-oo) or o (0) depending
NNote that a0 (+:ro) need not be finite.
4

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
on the sign of a (x 3), proving that:
Proof of Theorem 3.5. Assume, without loss of gener- ality, that the points 21,-..; ordered that < = f(x;) S f(zz), with ties resolved arbitrarly, We will proceed by construction, layer by layer.
0 (+00)
a (x - -6-8)
(x) の (->0),
a (I -I- 8) <)<00 (0), if of (x 8)
Layer 1
Since the function to interpolate is monotonic, for any couple of points Tı. z, with i < J. is possible to find hyperplane defined by o 3//4 (x - -B//4)
for some B3/e E' and some non-negative normal az// IR such that xy E A// E A5 where A and A denote the positive and negative half spaces respectively.
The second step is to prove that one hidden layer can per- form intersections of subspaces under specific conditions. In our construction, these will be either half-spaces or inter- sections of specific half-spaces.
Lemma 3.7. Consider the intersection A = Nizo Ar, for A , An subsets of TR. For any 7 in the image of ot single unit hi in the k-th hidden layer of an MLP with
non-negative weights can approximate:
(2)x)=1AA)
0
provided that h ((x) x 0 for x E Ag and either:
llh E S and hi (xx < 0 for $ A,
h lh olh e St and
(x) 0 for & A;
Figure 3. Example of representable functions layer 1.
Proof. Denote by the weights and the bias associated with the hidden unit under consideration. Then, setting the weights to T and taking the limit we have that:
Using Lemma 3.6, we can ensure that it is possible to have:
(z) R ( = 0, if E A5/s Vi.j
(6)
h (x) R Td (+00) > 0, otherwise
11 (x) R3 lim Tth (uuh""(("""
(x) b)
x
A visual example is shown in Figure 3.
a
lim Tth b + D h (xx)
A
Layer Let us construct the set A = =N Af/e Note
that the sets A," always contain x; and do contain any By for j > í. Using Equation (6), we apply Lemma 3.7, which ensures that it is possible to have the following":
Note that in any case, A E he be- -xx) x and the limit simply reduces o (b). On the other hand, for & Ni= As, the limit can be either al (=o) depending
x E E A,, then
the sign of ha (z). When Tt E S and "z)" < O, the limit is simply d (-oo) = 0. Similarly, when ṭ E S and h (x) > 0 the limit is dh (+oc) =
(z) 2
if & A
h1" < O,
otherwise
A visual example is shown in Figure 4.
In both cases, for any y the image of ( we can find a bias value 6 so that:
Layer 3 Consider A = =Njiee A. where A is the complement of A.. Using Equation (7) we can once again apply Lemma 3.7, which ensures that it is possible have the following*.
( (b) 7
i E ni= A,
(x) = 71)())) の (+o))= 0, otherwise
1 R T Ag
(8)
Thanks to Lemma 3.6 and Lemma 3.7, we can now prove the main result, that is Theorem 3.5. We will only prove the case Rl S-, opposite case follows the same structure and is reported in Appendix A.4.
Now, we will show that A represents a level set, i.e. Ty E A f(x,) 2 f(x)). To do so, consider that
, T E St, T E S The proof for the
IIn this case 7 < 0 since by assumption dt saturates right.
*In this case 7 since by assumption " saturates left.
5

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
3.3. Non-positive constrained monotonic MLP
0
Consider the simple modification of the standard constrained MLP approach described in Equation (3). However, instead of constraining the weights to be non-negative, they constrained to be non-positive. Although this simple modi- fication might seem inconsequential, we will show that this is not the case. Indeed, we will show that non-positive constrained MLP satisfies the conditions of Theorem 3.5, as long as the activation function saturates on at least one side. This includes convex activations like ReLU, which provably do not yield universal approximators in the non-negative constrained weight setting. Note that by Remark 3.1, an MLP defined according to Equation (3) is still monotone for an even number non-positively constrained layers; therefore, is still possible to construct provably mono- tonic networks using non-positive weight constraints. We will only discuss networks with however, the result also valid for an odd number of layers.
+1
Figure 4. Example of representable functions at layer
A = = Uptee Af. Since vi E Afr, then zy A for Similarly since Iy is the largest point contained in Aft, A cannot contain 2; or any point larger than 2₁. This shows that ATd contains exactly the points {æy f(())) 2 (x:)). visual example shown in Figure
number of layers;
A first crucial observation is that a and o' have the same monotonicity, but saturate in opposite directions.
Proposition 3.8. If (x:) is monotonic non-decreasing. then its point reflection o (x)) is also monotonic non-decreasing. If (z) saturates, then a (x) also saturates but in the oppo- site direction.
From Proposition 3.8 we can obtain an immediate corrollary of Theorem 3.5 which will prove useful:
Proposition 3.9. An MLP with at least 4 layers, non- negative weights, and alternating activation a and o' universal monotonic approximator, provided that satu- rates least side.
Figure 5. Example of representable functions at layer
Layer To conclude the proof, simply take the weights the fourth layer be:
The second observation is that imposing non-positive con- straints in two adjacent layers with an activation function in between is equivalent to imposing non-negative constraints in the two layers and using point-reflectes activation func- tion between them.
w f(x1) b f(22) f(x1) 700)
f(xn) f(xn-1) 7000
7/39
Since the points are ordered, this ensures that w contains all neg-tegative terms, when bias term taken be b S (x Defining f(x)) b, the output of the MLP can be expressed
Proposition 3.10. An MLP with W < O, W < 0 and (x) = (x). equivalent to an MLP with W 0,
IaID and o (x)) d'(x) -
From this, it follows that an MLP with an even number of layers, non- non-positive weights, and activation is equivalent to an MLP with non-negative weights that alternate activations between o' and a. This equivalence can be achieved using Proposition 3.10 by "flipping" the weight constraints two layers at time, which also changes the activations at odd-numbered layers from a to a'.
(0(x) = n (x+ b+ Σ (f(zz))((zz-1 f(zg-1)) a (z(x)
all layers
(9)
Evaluating Equation (9) at any of the points v,, it reduces to the telescopic sum:
= f(x1)( Σ (f())) f(zz-1)) f(x;) (10)
Thanks to Proposition 3010, this also shows that: Proposition 3.11. If E S US+ an MLP with 4 lay- ers, non pos-pooieive weights and activation o, is universal approximator for the class of monotonic functions.
1=2
Thus proving that the network correctly interpolates the target function.
6

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
Algorithm 1 Forward pass of Monotonic MLP with post-
activation switch
Input: data x E IR4, weight matrix W Rd bias
vectors E RRd" monotonic activation function
Output: prediction
IRd
W max (0, W) min (0. W)
W = max(W.0) 0)
W = -m.0
at = w+r(()
W
5
2 = W (xx)
g = 2t + 2 +b
Figure 6. Computation graph of single layer of a monotonic NN with the proposed learned activation via weight sign.
Similarly, we can apply the observations of this section show that the structure proposed in Runje & Shankara- produce universal monotonic approxi-
for the construction proposed in Section 3.3, we would still be left to decide the sequence activations that should be used for the MLP, which might be unclear or necessitate further hyperparameter tuning. However, by slightly rear- ranging the order of operations, it is possible to construct a monotone MLP that does not require manual tuning of the activation saturation side, while, at the same time, relaxing the weight constraint.
narayana (2023) mators using only point reflections without the need for the third activation class.
Although using Theorem 3.5 allows us to prove that a broad class of constrained MI.P architectures are universal mono- tonic approximators, does not necessarily translate into MLPs that are easily optimizable. Consider the computation for arbitrary input and an MLP with ReLU activation and biases initialized to zero. If x 2 O, then - -W| f s 0, and thus in the first layer ReLU(-IW[z))
Let us thus consider a single layer f(x) a[W]a + in constrained MLP, that uses the absolute value for weight reparametrization. Instead of constraining weights, we can separate W into its positive and negative parts W max (W, 0) and W min(W, (W,0). This allows us to express the affine transformation as
0. However, if 2 O and now the second layer
x S 0, then ReI.U(- will saturate instead. To allow for efficient and effective optimization, we must carefully tune the initialization of the bias term avoid having 0 gradient everywhere. Broadly speaking, the weight constraint makes the networks more sensitive to initialization.
W bb= w -
(11)
Applying the con-linearity to each term of Equation (11) individually instead of applying it to |W|a, and sharing the bias term, leads to the parametrization:
4. Addressing the weight constraint
Historically, the first works that proposed a monotonic neu- ral network formulation relied on forcing the parameters of the network be non-negative, specifically the matrices W in the affine transformations, combined with bounded mono- tonic activations, is a sufficient condition to guarantee that the overall function is monotonic (Daniels & Velikova, 2010; Sill & Aos-Mastaaa, 1996). Recently, Runje & Shankara- narayana (2023) showed a way to build effective monotonic MLPs with such technique by exploiting multiple activa- tions. However, even though the of constrained weights and bounded activation is easy to implement and can be optimized with any unconstrained gradient optimizer, poor initialization could lead to vanishing gradient dynamics, as further explored in Appendix A.2 and Appendix A.3. In- stead, we will show how to address this issue while also tackling the necessity of alternating the activation saturation have universal approximation capabilities.
j(x) = oW x + +b)
(12)
Proposition 4.1. Any function representable using an affine transformation with non non-negative weights followed or o' can also be represented using Equation (12), up to constant factor.
follweed by either
Proof. When all entries of W have the same sign, one of the two terms in Equation (12) collapses to (b). Specifically, when W > O, the expression reduces to a ( + 6) [bb). while when W S reduces to (b)) - W| instead. To conclude the proof recall that -000o(02 d'()).
The additional constant factor be accounted for in the bias term of the following layer. Therefore, Proposi- tion 4.1 covers both cases employed in Proposition 3.9. This shows that an MLP obtained by stacking at least 4 blocks parametrized as Equation (12) is a universal approximator for monotonie functions. Hence, the proposed formulation
4.1. Relaxing weight constraints with activation switches
Assuming that we used the weight-constrained formulation

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
Table Test metries across different datasets. The best-performing architccture per dataset is bolded.
COMPAS
Blog Feedhack
Loan Defaulter
AutoMPG
Heart Disease TTest Accuracy)
Method
(Test Accuracy)
(Test RMSE)
(Test Accuracy)
(Test MSE)
XGBoost
68.5% + 0.1%
0.176 + 0.005
63.7% + 0.1%
Certified
68.8% + 0.2%
0.159 + 0.001
65.2% + 0.1%
Non-Neg-DNo.
69.3% +: 0.1%
0.154 + 0.001
65.2% + 0.1%
10.31 + 1.86
89% = 1%
DLN
67.9% +: 0.3%
0.161 + 0.001
65.1% + 0.2%
13.34 + 2.42
86% = 2%
Min-Max Net
67.8% + 0.1%
0.163 + 0.001
64.9% + 0.1%
10.14 + 1.54
75% 4%
Constrained MNN
69.2% + 0.2%
0.154 + 0.001
65.3% + 0.1%
8.37 + 0.08
89% = 0%
Scalable MNN
69.3% 1 0.9%
0.150 + 0.001
65.0% 1 0.1%
7.44 1: 1.20
88% = 4%
Expressive MNN
69.3% 1 0.1%
0.160 + 0.001
65.4%
1 0.1%
7.58 1 1.20
90% = 2%
Ours
69.5% = 0.1%%
0.149 0.001
65.4%
= 0.1%
7.34 0.46
94% 1%
is more expressive compared to simple weight constraint, given that they are only special case of Equation (12).
second classification dataset considered is the Heart Disease dataset. It consists of 13 features, 2 of which are monotonic with respect to the output. Lastly, we also test our method on the Loan Defaulter dataset, comprised of 28 features, 5 of which have a monotonic dependency on the prediction. To test on a regression task, we use the AutoMPG dataset, comprised of features, 3 of which are monotonically de- creasing with respect to the output. A second dataset for regression is the Blog Feedback dataset (Buza, 2013). Con- trary to all other datasets, this dataset is composed of very small portion of monotonic covariates. In fact, the data set consists of 280 features, of which only 8 are monotonic with respect to the output, accounting for 2.8% of the total.
Alternatively, one could apply similar reasoning working backwards from the last layer of the network, which would lead to an alternative formulation, given by
f(x) o W o(=x)+b
(13)
We will refer Equation (12) and to Equation (13) as pre- activation switch and post-activation switch, respectively.
In Figure 6, we report only the post-activation switch's 's pseu- docode and computation graph since it will be the formula- tion that will also be employed for the experimental part of the paper. The pre-activation corresponding algorithm and computational graph can be found in Appendix A.5.
We compare our method with several other approaches that give monotonic guarantees by construction. In particular, we compare it to XGBooot(Chenn & Guestrin, 2016) Deep Lattice Network (You et al., 2017), Min-Max Networks (Daniels & Velikova, 2010), Certified Networks (Liu et al., 2020), COMET (Sivaraman et al., 2020), Constrained Mono- tonic Neural Networks (Runje & Shankaranarayana, 2023), Expressive Monotonic Neural Network (Nolte et al., 2023), and Scalable Monotonic Neural Networks (Kim & Lee, 2024). With Non-Neg-DNN we refer to a naive constrained monotonic MLP using sigmoid activations. Similar results are reported in (Liu et al., 2020), though in a narrower set of datasets. In Table 1, we report the final test set metrics. For the proposed method, little to no hyperparameter tuning has been performed, as the hyperparameters found by (Runje & Shankaranarayana, 2023) worked without the need for further tuning. The proposed method matches or exceeds the performance of all other recently proposed approaches.
Indeed, the simplicity of the approach can be appreciated: it shares most of the steps of the forward pass of a tradi- tional MLP due to the relaxation of the weight constraint and does not require additional special care for initializa- tions. In Appendix A.2 we provide additional details on the optimization properties of the proposed formulation. For all experiments, the default PyTorch (Paszke et al., 2019) ini- tialization was used without the need for additional tuning. A naive implementation can be achieved using second ma- trix multiplication. This additional operation can be easily parallelized and does not require additional data transfers. For the networks tested, we did not observe any overhead in practice.
5. Experiments
In this section, we aim to analyze the method's performance compared to other alternatives that give monotonic guar- antees. The first dataset used is COMPAS (Fabris et al.,
6. Conclusions and future works
In this work, we have relaxed the requirements achieve universal approximation in monotonic ML.s with con-
2022). COMPAS is a dataset comprised of 13 features, 4 of which have a monotonic dependency on the classification. A
8

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
strained weights. We proved that alternating saturation side in the activations is sufficient condition to achieve this property with finite number of layers. In addition, we show connection between the saturation side of the activa- tions and the sign of the weight constraint. This allows us to show that the non-positive weight constraint is, surprisingly, more expressive than the non-negative one, which can only represent convex functions. We then use this theoretical analysis to construct novel parameterization that relaxes the weight constraint, making the network less sensitive to initialization. The activation saturation side is learnable, which ensures universal approximation capabilities even when using monotonic convex activation, which was pre- viously not possible. MLPs built with our fully connected, monotone layer achieve state-of-the-ar performance.
units. arXiv preprint arXiv: 2017.
Buza, K. Feedback prediction for blogs. In Data analysis, machine learning and knowledge discovery, pp. 145-152. Springer, 2013.
Chen, T. and Guestrin, C. Xgboost: scalable tree boosting system. In Proceedings of the 22nd acm sigkdd inter- national conference on knowledge discovery and data mining, pp. 785-794, 2016.
Chilinski, P. and Silva, R. Neural likelihoods via cumulative distribution functions. Conference on Uncertainty in Artificial Intelligence, pp. 420-429. PMLR, 2020.
Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus). In ICLR, 2016.
Although this work proves that any monotonic saturating activation can be used to build monotonic MIPss, it is still an open question whether non-saturating activations, such as Leaky-ReLU, can be used to build monotonic MLPs. Furthermore, batch normalization has proven to be highly effective in the unconstrained case. Still, it has never been used possible solution to the initialization problem for the monotonic case.
Daniels, H. and Velikova, M. Monotone and partially mono- tone neural networks. IEEE Transactions Neural Net- works, 21(6):906-99) 9910997966999900 2010.
Dressel, J. and Farid, H. The accuracy, faimess, and limits of predicting recidivism. Science advances, 4(1 4(1))eaa05580, 2018.
7. Ethical Considerations
Dubey, S. R., Singh, S. K., and Chaudhuri, B. B. Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing, 503:92-108, 2022.
The use of the COMPAS dataset in this research acknowl- edges its status as a common benchmark within the field of machine learning fairness studies (Angwin et al., 2022: Dressel & Farid, 2018). Recognizing the complexities and potential ethical challenges associated with such datasets, we emphasize commitment to responsible research prac- tices. We prioritize transparency and ethical rigor through- out our study to ensure that the methodologies employed and the conclusions drawn contribute constructively to the ongoing discourse in AI ethics and fairness. This approach underlines our dedication advancing machine learning applications in manner that is conscious of their broader societal impacts.
Dugas, C., Bengio, Y., Bélisle, F., Nadeau, C., and Garcia, R. Incorporating second-order functional knowledge for better option pricing. Advances in neural information processing systems, 13, 2000.
Dugas, C., Bengio, Y., Bélisle, F,, Nadeau, C., and Garcia, R. Incorporating functional knowledge in neural networks. Journal of Machine Learning Research, 10(6). 2009.
Fabris, A., Messina, S., Silvello, G., and Susto, G. A. Algo- rithmic fairness datasets: the story so far. Data Mining and Knowledge Discovery, 36(6):2074-2155 2022.
Impact Statement
Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Pro- ceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. Maxout networks. In International conference machine learning, pp. 1319-1327, PMLR, 2013.
References
Angwin, J., Larson, J., Mattu, S., and Kirchner, L. Ma- chine bias. In Ethics of data and analytics, pp. 254-2644 Auerbach Publications, 2022.
Gupta, A., Shukla, N., Marla, L., Kolbeinsson, A., and Yellepeddi, K. How to incorporate monotonicity in deep networks while preserving flexibility? in NewrIPS, 2019.
Barron, J. T. Continuously differentiable exponential linear

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
Gupta, M., Cotter, A., Pfeifer, J., Voevodski, K., Canini, K., Mangylov, A., Moczydlowski, W., and Van Esbroeck, A. Monotonic calibrated interpolated look-up tables. Journal of Machine Learning Research, (7(109):-447 2016.
Nair, V. and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 807-814, 2010.
He, K., Zhang. X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Nguyen, A.p.. and Martínez, M. R. Mononet: towards interpretable models by learning monotonic features. In NeurIPS, 2019.
Nolte, N., Kitouni, O., and Williams, M. Expres- sive monotonic neural networks. In The Eleventh In- ternational Conference on Learning Representations, 2023. URL https: https:///ppeeeeee...ooo wtw.e/wuneee/ttt forum? id=w2P BM
Hein, M., Andriushchenko, M., and Bitterwolf, J. Why relu networks yield sigh-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 41-50, 2019.
Omi, T., Aihara, K., et al. Fully neural network based model for general temporal point processes. Advances in neural information processing systems, 32, 2019.
loffe, S. Batch normalization: Accelerating deep network training by reducing internal covariate shift. in ICMI., 2015.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Jeanselme, V., Yoon, C. H., Tom, B., and Barrett, Neu- ral fine-gray: Monotonic neural networks for competing risks. In Conference on Health, Inference, and Learning, pp. 179-392. PMLR, 2023.
Raghu, M.,, Poole, B., Kleinberg, J., Ganguli, S., and Sohl- Dickstein, J. On the expressive power of deep neural networks. In international conference on machine learn- ing, pp. 2847-2854. PMLR, 2017.
Jin, X., Xu, C., Feng, J., Wei, Y., Xiong, J., and Yan, S. Deep learning with s-shaped rectified linear activation units. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Ravikumar, A. and Sriraman, H. Mitigating vanishing gra- dient in sgd optimization in neural networks. In Interna- tional Conference on Information, Communication and Computing Technology, pp. 1-11. Springer, 2023.
Kim, H. and Lee, J.S. Scalable monotonic neural networks.
In The Twelfth International Conference on Learning Representations, 2024. URL https: // openreview. Bett/oorumiiG-jjIiee
Runje, D. and Shankaranarayana, S. M. Constrained mono- tonic neural networks. In International Conference on Machine Learning, pp. 29338-29353 PMLR, 2023.
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
Sánchez, L., Costa, N., and Couso. I. Simplified models of remaining useful life based on stochastic orderings. Reliability Engineering System Safety, 237:109321, 2023.
Liu, X., Han, X., Zhang, N., and Liu, Q. Certified mono- tonie neural networks. Advances in Neural Information Processing Systems, 33:15427-15453 2020.
Sill, J. and Abu-Mostafa, Y. Monotonicity hints. Advances in neural information processing systems, 9, 1996.
Mikulincer, D. and Reichman, D. Size and depth of mono- tone neural networks: interpolation and approximation. Advances in Neural Information Processing Systems, 35: 5522-5534, 2022.
Sivaraman, A., Farnadi, G., Millstein, T., and Van den learning of mono-
Broeck, G. Counterexample-giue tonic neural networks. Advances in Neural Information Processing Systems, 33:1 1936-11948, 2020.
Milani Fard, M., Canini, K., Cotter, A., Pfeifer, J., and Gupta, M. Fast and flexible monotonic functions with ensembles of lattices. Advances in neural information processing systems, 29, 2016.
Szandala, T. Review and comparison of commonly used ac- tivation functions for deep neural networks. Bio-inspired neurocomputing, pp. 203-224, 2021.
Mitchell, T. M. The need for biases in learning generaliza- tions. Technical Report 1BM-TR-117, Carnegie Mellon University, 1980.
Tagasovska, N. and Lopez-Paz, D. Single-model uncertain- ties for deep learning. Advances in neural information processing systems, 32, 2019.
10

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Veličković, P.
The resurgence of structure in deep neural networks. PhD thesis, University of Cambridge, 2019.
Wehenkel, A. and Louppe, G. Unconstrained monotonic neural networks. Advances in neural information process- ing systems, 32, 2019.
Yanagisawa, H., Miyaguchi, K., and Katsuki, T. Hierarchi- cal lattice layer for partially monotone neural networks. Advances in Neural Information Processing Systems, 35: 11092-11103, 2022.
You, S., Ding, D., Canini, K., Pfeifer, J., and Gupta, M. Deep lattice networks and partial monotonic functions. Advances in neural information processing systems, 30, 2017.

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
A. Appendix
The appendix is structured the following way:
Appendix A.1: in this section, we show the arguably simplest though loosest bound to prove that non-negative constrained MLPs with ReLU and ReLU" activations are universal approximators.
Appendix A.2: we discuss how parameterizing the network with non-negative weights leads to optimization issues, specifically vanishing gradients dynamics.
Appendix A.3: in this section, we will compare the proposed method to the bounded-activatioon counterpart, showing how the formulation with sigmoidal activation suffers from vanishing gradients.
Appendix A.4: in this section we prove the results of Theorem 3.5 for the opposite alternation case.
Appendix A.5: reported in Section 4, propose two possible parametrizations, pre-activation switch, and post-activation switch. In Appendix A.5, the pseudocode and the computational graph of the two can be found.
Appendix A.6 and Appendix A.7: in these sections, we report further information regarding how the results have been obtained and about the datasets employed for this work.
Appendix A.8: since Theorem 3.5 only requires the non-linearity to be saturating, in this section we report a brief overview of other activations that can be applied with the proposed method, in order to underline how is more general than just using ReLU activations.
Appendix A.9: the proof provided for Theorem 3.5 is different to the ones previously proposed in literature. However, it still ends with the result of requiring 4 layers to be a universal approximator, as previously shown in (Mikulincer & Reichman, 2022) for the heavy-side function. For readers that are already familiar with such proof, we also report in Appendix A.9 proof very similar to the one in (Mikulincer Reichman, 2022), trying to reuse as much as possible the original structure.
12

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
A.1. Naive bound for universal approximation of alternating MLPs
simpler, though looser, bound prove that MLPs with alternating ReLU and its point reflection ReLU activations are a universal monotonic function approximator can be achieved by building the proof of (Mikulincer & Reichman, 2022).
Two simple observations are sufficient.
Remark A.I. the composition of ReLU and its point reflections ReLU'(x) ReLLU-=>) can approximate the threshold function 2,20 arbitrarily well:
lim (az) + + 1020(z)
(14)
lim ReLU (ReL((a)) - 1) 1020(z)
(15)
00+00
A representation of Equation (14) is provided in Figure 2.
The reason why we can approximate non-convex functions using only ReI.U-like activations is reported in Proposition 3.2. However, considering Proposition 3.8, we can see how this limitation can be addressed.
Remark A.2. The formulas in Equation (14) can be implemented with a 2-layer constrained MLP, alternating ReLU and ReLU activations.
This is enough to leverage the existing results for threshold-activated MLP (Mikulincer & Reichman, 2022). This includes the best-known bound on the number of required hidden layers, which, however, doubles from 3 to 6 due to the need for two ReLU layers for the Heavyside approximation. However, this naive bound is unnecessarily loose, as shown in Theorem 3.5.
A.2. Initialization issues and Vanishing Gradient in Constrained MLPs
A.2.1. VANISHING GRADIENTS DYNAMICS IN CONSTRAINED MONOTONIC MLPs WITH BOUNDED ACTIVATIONS
As reported in Section 3, a naive approach to ensure monotonicity is to have monotonic activations and to impose monotonicity to the weights, constraining them to be non-negative. For this reason, the affine transformations of these networks are usually parametrized as ((x) = g (()) 1 + 6, for some transformation g R R... Note that the bias can be any value, as it a constant and thus does not affect the gradient.
Such networks employed bounded activations. like sigmoid, tanh, or ReLU6, to have convex- connexe activations. This peculiarity makes them very sensitive to initialization and potentially lead vanishing gradient dynamics (Glorot & Bengio, 2010). To see why constraining weights to be non-negative exacerbates this condition, consider a monotonic MLP with sigmoidal activations, initialized with random weights according to known, widely used initializers, such as Glorot, where each matrix is sampled from symmetric distribution around zero with some variance. Instead, the biases are initialized to zero, as is usually done. Let's assume using ((x) = |x]], but the same reasoning can be applied to any other mapping y. At this point, the MLP comprises layers of the following form (xx) = o[ [ + b). Now, let's consider the second layer of such MLP. Since the first layer has applied the sigmoid activation, then σ( (z) E (0, 1). Because of this, [W (a:) will be product of all non-negative terms. Therefore, its result can become significantly large. Then, when applying the sigmoid activation of the second layer, it will most likely saturate due to the large positive values returned from the affine transformation. Going with this reasoning for multiple layers, such behavior will be exacerbated. Appendix A.3 shows one example of such behavior for a very simple function. The same behavior occurs for ReLU6 MLPs, where the gradient might even become exactly 0, and for tanh MLPs if, for example, the dataset is normalized, which is one of the most commonly used petrppreproc.s..ng....
simple evidence of such dynamic can be seen in Appendix A.3, where even with a ID toy example, constrained monotonic MLPs with sigmoid with few layers show signs of vanishing gradient (Figure 8). However, such signs of vanishing gradient are evident since initialization, as shown in Figure 9
One possible solution might be using BatchNormalization layers (Ioffe, 2015). BatchNorm has already shown its ef- fectiveness in tackling initialization and optimization problems. Indeed, BatchNorm is comprised only of the following transformation:
BN I
E
7 +
Var [x]
+
Considering that V Var[a] + >> O, forcing 7 2 0 by construction, for example, using 7 = SoftPlus(?'), makes such operation monotonic. Usually, it initialized as 3 and 7 1. For this reason, used as a pre-activation layer, might
13

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
address exploding pre-activation values, standardizing them around zero. However, the investigation of this approach falls out of the scope of this work, and it's left future line of research.
A.2.2. INITIALIZATION OF SWITCH MONOTONIC NEURAL NETWORKS
The proposed activation switch parametrization shown in Equation (12) and Equation (13) aims at relaxing monotonic MLPs by the weight constraint required to ensure monotonicity, while also removing the need to pick a correct sequence of activations manually. As addressed in the previous chapter, that weight constraint easily leads to vanishing gradients used conjunction with bounded activation, extremized when using ReLU6 activations, which leads almost always to a completely dead MLP, as reported in Appendix A.3. Therefore, our parametrization tries solve this problem from two points: avoiding directly constraining weights and using unbounded activations.
Nonetheless, it still deviates from the well-studied original MLP formulation. In particular, separating the W matrix into positive and negative parts might still introduce initialization issues.
Unconstraired
Unconstrained
2.5
Constrained (Naive)
Constrained Naive
(Post Activiatiinnnn switch) (Pre Activation switch)
Our (Post-Actives Activation switch)
Our (PreArtivvtion switch)
1.0
0.0
100
200
300
400
500
790
Hidden Dimersion
Figure 7. First plot, the distrihution of the output of an MI.P with the different parametrizations. Second plot, the scaling law of the expected output after initialization of the different parametrization. In both images, can be how the naive constrained MLP has very different scaling behavior compared to the rest.
From an empirical perspective, in Figure 7 we can observe the distribution of the output of multilayer MLP with different parametrization. Unconstrained refers to a naive MLP, Constrained Naive refers to MLP with [W] weight parameterization, while pre/post activation switch refers to Equation (12) and Equation (13). Indeed, it can be seen how the naive MLP, matter the hidden dimension, have O-mean in predicted value. Instead, the switch activation has very slow tendency to increase the expected output from random initialization, as predicted by theory. However, such slight increase, is notably largely smaller than the one induced by naively constraining the weights to be positive.
However, empirically, the activation-swinn Table 1 have been obtained with the default Py Torch initialization. Furthermore, such results the aggregation of multiple seeds. Thus, the networks used have been initialized with different values.
formulation does not exhibit initialization issues. Indeed, the result reported in
Overall, we can see how, even though the behavior of activation-switch parametrization slightly deviates from the one by normal MIP,, such difference does not hinder many performance while still being less than the one obtained by the constrained counterpart that does not use such a trick.
14

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
A.3. Comparison of different parametrizations on trivial datasets
To showease the effectiveness of the proposed method to the bounded-activation counterpart, in Figure we compare them simple synthetic example. In particular, the models asked to approximate f(x) cos(z) + 2, simple 1D monotonic function with multiple saddle points. For this reason, it is fundamental for the approximation model to be very flexible. To showease the different performances, we will test 4 models. The first model to test is an unconstrained NN, which shows that an unconstrained model can learn such a function. The second model is monotonic NN with non-negative and ReLU activations, which shows that, as shown in theory, it cannot approximate a nonconvex function. The third model a monotonic NN with non-negative and sigmoid activations. This model, instead, is shown to be a universal approximator for monotonic functions but suffers from vanishing gradients. Lastly, the fourth model the proposed parametrization, specifically the post-activation setting, as described in Section 4.1.
In Figure 8, it can be seen how the model with non-negative and ReLU activations cannot learn the function predicted by theory, since the function that is asked to learn is non-convex, Instead, both the sigmoid model and our proposed approach successfully approximate it. Still, the sigmoid function struggles to be optimized due to the complications of using sigmoid activations. Instead, the proposed method exploits rectified linear activations, which, under regime where the number of dead neurons not too high, is much easier to optimize, as explained the original work that introduced such activation Glorot & Bengio (2010) and Raghu et al. (2017).
Such difference is also evident in analyzing the Negative Log Likelihood (NLL) loss of the training. We report in Figure 8 the various training losses obtained with two different sizes of layers. The naive monotonic ReLU, which cannot approximate such a function, is indeed the worst. However, even though the sigmoid monotonic NN is universal approximator, it is the slowest to learn, probably due to the vanishing gradient problem. Instead, the proposed method that uses ReLU activations the fastest to converge, almost catching the unconstrained model the setting with more neurons.
8-
10'
10a
6-
162
1.9
10-1
10
2-
10
10-4
10
10-0
10-2
10-4
-2
800
1000
Gradient steps
700) costx)
Unconstrained
Monotonic ReLU
Monotonic Sigmoid
Monotonic Split ReLU (ours)
Figure 8. First plot, approximation of f(z) using MLPs with layers of 128 neurons. Second plot, approximation of f(x using MLPs with layers 256 neurons. Last plot, training losses of the different methods (full lines represent versions with neurons, dashed lines represent versions with 256 neurons).
Generally speaking, as also reported at the end of Section 3.3, MLPs with constrained weights require careful initialization to avoid non-optimizable configurations. The proposed method in Appendix A.5 alleviates this behavior but is not indifferent to it.
In order to showcase the vanishing gradient problem exacerbated by the non-negatively constraining, in Figure we create a 128- 22-nneuron wide MLP with varying numbers of hidden layers, and we compare the average gradient of the output with respect the parameters on the same function approximation problem presented earlier in Figure 8. It can be observed how the sigmoid monotonic MLP, even with small number of layers, has order of magnitude less gradient magnitude; in particular, it has average gradient of 0.0019 for 4 layers and 0.00099 for 10 layers. Instead, the ReLU monotonic MLP has an exploding gradient due to the accumulation of activations induced by the pairing of ReLU activation and positive weight; in particular, it starts from a gradient magnitude of 3.54 for 4 layers and goes to 3311 $311.00 for 10 layers. Finally, the proposed approach keeps the gradient magnitude in a reasonable magnitude range, starting from a gradient of 0.010 for layers and going to 1. 1.259 for 10 layers. Results are averaged over 20 different random initializations, and plot shows +1g.
15

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
In order better analyze the optimization problems of these architectures, also report in Figure 10 the distributions of the gradients of 6-layers MLP with the various architectures. It can be seen that the sigmoid MLP has extremely low gradients for the initial layers, leading to slow learning. On the other hand, the ReLU MLP has exploding gradients for the final layers.
is worth noting that the same exact configuration, using ReI.U6 as activation, leads to dead network, as all gradients are zeroed out due the saturated section of the activation.
Average gradient w.r.t. network parameters with default initialization
ReLU constrained MLP
104
Sigmold constrained MLP
Proposed approach
10
10a
deld W-1
10
10
10
10-3
Layers
Figure 9. Average gradient from monotonic ML.Ps varying the number of layers. Data is shown in the log scale for the y-axis.
Signoidd
RelU
Proposed
1.0
1.0
1.6
Ieero
Kyer
koor3
Soer
1por4
3.6
2.6
0.5
32
32-
0.
275
-255
2000
3375
35.0
-12
-165
.35
-150
-2.
73
-10
logt
Figure 10. Distribution of gradients from monotonic MI.Ps for each layer (layer O is the final one, layer is the first after the input).
16

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
A.4. Proof for opposite alternation of activation for Theorem 3.5
In this section, we will conclude the proof of Theorem 3.5, considering the case with activations that alternate in the opposite direction from the one reported in the main text. Indeed, in Section 3.2 proved the result for the case with altll & oه St ah S while in this section we will prove the case with σf E S o E S , & St. The proof is extremely similar, with just a few opposite signs due to the opposite alternation. Thus, most constructions will be shared.
Proof of Theorem 3.5 with opposite alternation. Assume, without loss of generality, that the points .1,...; In are ordered so that <j - f(x))= S f(r)), with ties resolved arbitrarly. We will proceed by construction, layer by layer.
Layer Since the function to interpolate is monotonic, for any couple of points <j: < f(z) it is possible to find a hyperplane with non-negative normal, with positive and negative half spaces denoted by A3,, and Azjo such that Jy 2/0 E A/1-
Using Lemma 3.6, we can ensure that it is possible to have:
1 (x) x (+00) = 0, i E AJ/a
(16)
hf = (-o0) < 0, otherwise
Layer 2 Let us construct the set ATT =
Miscr 4//5 Note that the sets A always contain and do not contain any Tj
for j < i. Using Equation (16), we can apply Lemma 3.7, which ensures that it is possible to have the following":
A ~ 0, if E A (z) 00 otherwise
(17)
Layer 3 Consider ATT = =N AT where A is the complement of AT. Using Equation (17) we can once again apply Lemma 3.7, which ensures that it is possible to have the following
1 (z) Rz 7 101
A (x)
(18)
Now, we will show that A!" represents a level set, i.e. Iy E A S f(x)). To do so, consider that
A U امحل A.t Since r; E Al" then x, E A for j > i. Similarly since Ij is the smallest point contained in A4), A cannot contain 24 or any point smaller than n,. This shows that A contains exactly the points {x, : f(x,)) $ f(x)).
Layer 4 To conclude the proof, simply take the weights the fourth layer be
- -
f(zn-1) f(xn)
f(xn) 6 7/8)
w
708)
7/30
Note that compared to Equation (8), here 7 is now negative, and the terms in the numerators' difference are reversed. Since the points are ordered, this ensures that w contains all non-negative terms, when bias term is taken to be b > f(xn). Defining fIn+11) = b, the output of the MLP can be expressed as:
go(x) = 3 (x) b (f(x,)) f( a (x(x)
(19)
j=l
Evaluating Equation (19) at any of the points x4. reduces to the telescopic sum:
go(x;) = f(x,)) + (((()))((app ((x))
(20)
Thus proving that the network correctly interpolates the target function.
In this case " > 0 since are considering the case where " saturates left.
'In this case 7 < 0 since we are considering the case where a saturates right.
17

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
A.5. Algorithms
In Section show how we can parametrize the activation switch using the sign of the weights. For such a mechanism, propose two different parametrizations, Figure 6, report both the pscudo-code and the computational graph for the post-activation formulation. For completeness, in this section, we also report the pre-activation pseudo-code and computational graph, and for readability and to ease the comparison, we report them side by side, reporting again the post-activation formulation also reported in the main text. In particular, in Figure 11, we report the two pscudocode side-by-side, and in Appendix A.5, the relative pseudocodes.
where the switch is applied post-activation and another pre- pre-activation. In
y
W
min (0, W)
W max (0, W) W- min (0, W)
W
W
Figure 11. Computation graph of single layer of ReLU monotonic NN with the proposed learned activation via weight sign. The left plot reports the computational graph of the post-activation, and the right plot shows the pre rreacctivaionn switch.
Algorithm 3 Forward pass of Monotonic ReLU MLP with post-activation switch
Algorithm 2 Forward pass of Monotonic ReLU MLP with pre-activation switch
Input: data x E: R" weight matrix W RA 1, bias vectors E WR activation function
Input: data x E IR", weight matrix W E R bias vectors E R activation function
Output: prediction
R
Output: prediction ÿ E he
W == max(W, 0)
W+ = max(W.0) 0)
== min(W.0)
W- = min(W.0)
2t = W+o(x)
zt W
= o=o(-z)
= W=++b
= 2 +
= 02+t) oo(=)
A.6. Dataset description
For this work, the code was heavily based the code provided by Runje & Shankaranarayana (2023) in order to ensure that the used dataset matched exactly. For this reason, we will report short description of the employed dataset, but for further and more detailed description, refer to the original work (Runje & Shankaranarayana, 2023).
This dataset a binary classification dataset composed of criminal records, comprised of features, 4 of which are monotonic.
COMPAS:
This dataset is a regression dataset comprised of 280 features, 8 of which are monotonic, aimed at predicting the number of comments within 24h.
Blog Feedback:
Auto MPG: This dataset is a regression dataset aimed at predicting the miles miles-per-eee per-gallon consumption and comprised of features, of which are monotonic.
Heart Disease: This dataset is a classification dataset composed of 13 features, 2 of which are monotonic, aimed at predicting possible heart disease.
Loan Defaulter: This dataset is a classification dataset composed of 28 features, 5 of which are monotonic, and aimed at predicting loan defaults.
18

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
A.7. Experiments description
The following are the specifications used to obtain the results reported in Table The experiments were developed in PyTorch. The training was performed using the Adam optimizer implementation from the PyTorch Library. For all datasets except BlogFeedback, architecture to (Runje & Shankaranarayana, 2023). For BlogFeedback instead, special care was required, as the dataset is very large, but the features are very sparse and mostly unconstrained (only 3% are monotonic). Therefore, small hyperparameter tuning has been done to find the best setting. CELU has been used as an activation function for the smaller MLPs to avoid dead neurons.
hyperparameter tuning has been carried out. Instead, used similar networks in size and
Table 2. Hyper-parameters used for results reported in Table
Hyper-parameter
COMPAS
Blog Feedback
Loan Defaulter
AutoMPG
Heart Disease
Learning-rate
10 3
10-2
10-3
10
10 3
Epochs
100
1000
50
300
300
Batch-size
256
256
Free layers size
16
2
16
16
Number of free layers
2
Monotonic layers size
3
16
Number of monotonic layers
Activation
ReLU
CELU
ReLU
CELU
ReLU
19

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
A.8. Extension to other activations
In the rest of the paper, for all the practical examples, we assumed that ReLU was the activation chosen for the MLP. However, the results in Section 3.3 and Section 4.1 only require that the activation function saturates in at least one of the two sides, other than being monotonic. If ReLU falls in such category, it is not the only one, and many other widely used ReL.U-like activations satisfy the minimal assumptions of Theorem 3.5. For this reason, we will now analyze many other activations and report whether they comply with our construction. In particular, we report in Table 3 multiple widely used activations. With them, we also report the respective gradients, whether they are non-decreasing and saturating, and whether they can be used for the proposed approach.
can be seen that the proposed method allows the usage of most of today's widely used activations. However, it is crucial to notice that even though the proposed method allows for saturating activations, it also can be used with bounded activations, such as sigmoid and tanh, but that might bring almost no additional advantage over the weight-constrained counterpart. Any activation that saturates at least one side can be used, given that it is monotonic. Still, the real advantage comes from activations that saturate only one side.
Table Widely used activations with their corresponding properties, and whether they can be used or not.
Name
Function
Gradient
Monotone
Saturates
Usable
if x 2 0 otherwise
1
if 2 0
ReLU
/
0
0
otherwise
i 2 0 otherwise
1 if 2 0 otherwise
LeakyReLJU
x
x
ox
a
x ax (ar learnable)
if x 2 0 otherwisc
if 2 0 otherwisc
PReLU
/
S
= 2 otherwise
if 2 6 0 < 6 otherwise
ReLU6
f6
if 2 otherwise
1 ac"
ifa 2 0 otherwise
ELU
a(e" - 1)
SELU
X x1
i 2 0 otherwise
if 2 0 otherwise
a(E" -1)
1)
ae
CELU
x 1)
if x 2 0 otherwise
x
if 2 0 otherwise
a(= x
GeLU
x+(x)
(xx)
x
x
SiLU/Swish
(x)
(x4444441 (c*+1)
x
Sigmoid
1
e (146-2y2
/
146-=
Tanh
2 -
Exp
ex
/
SoftSign
[+]+I
[|]
Softplus
log(1 + c*)
d
LogSigmoid
-log(1 =
14c2
true only if parametrized in such way to guarantee 2 0
20

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
A.9. Alternative proof of Theorem 3.5
In this section, will construct proof similar the one proposed by Mikulincer & Reichman (2022) prove the constant bound of required layers for constrained MLP with Heavyside activations.
Figure 12. Examples of learnable functions at first hidden layer.
First layer construction First, let us show that the network can represent piece-wise functions at the first hidden layer.
Lemma A.3. Consider hyperplane defined by a (x 3) = 0. R and B E R and the open half-spaces:
A (x: a (xr () 0},
(21)
A = (x: a" (x B) < 0j.
(22)
A single neuron the first hidden layer of an MLP with non-negative weights can approximate ?
o (+00), e if E A otherwise
(x) T (-xo), (0),)
Proof. Denote by the weights and by b the bias associated with the hidden unit in consideration. Then, for any X E R+.
setting the parameters to w Aa and = Xa T 3 we have that:
h = ":" (wx + b) = l.') (Aa² (x - B))
the limit, we get:
(x) lim " (Aa (x 3))
*+000
The limit is either ('+(o)), (-oo) or o (0) depending on the sign of " (x B), proving that
(+)0),
if (a 9) > 0
(x)
(-
a (a 3)
(0))), if (r 3) = 0
For an easier interpretation of the just stated construction, we show in Figure 12 some samples from the family functions that can be learned with this first hidden layer.
21

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
Figure 13, Examples of learnable indicator functions at the second hidden layer.
Second layer construction Using Lemma A.3, we can show that alternating saturation directions in the activations is sufficient to represent indicator functions of intersections and unions of positive half-spaces.
Lemma A4. If " E S* ṭthh S there exists rescaling factor R4 such that single unit in the second hidden layer of an MLP with non-negative weights, can approximate:
1tth R (1)
for any A = - AT. Similarly, ( E Es E St, it can approximate
1 x R +7
for any A = U At.
Proof. Denote by the weights and by b the bias associated to the hidden unit in consideration at the second layer. For any A E R4 setting the weights to = A11 we have that
n (x) = l00 (wh + a
b b+1 b+ h
Taking the limit, the result only depends on the sign of E, h Using Lemma A.3, we ensure that it is possible have
σ (+00). if A ot (-oo), if E A
(z)
From here, there are two cases, depending the saturation of the activations. We will only prove the when the activations saturate to zero to avoid needlessly complicated formulas. However, the result holds even in the general case.
If we assume of E S+ Rt S For Ni= At', we have h.t (x) = " "": (+00) = O, while for æ & Ni=1 Att have (((()) < (+oo) = Therefore
E (b) = 7, T (-00) 0, otherwise
if E N A
lim (x)
where 7 can be any element of the image of th which is negative function. Therefore for A =Ni= A$
hx))): an
Note that nd (t:oo needs not finite.
22

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
If instead assume a! E _ S+: For N==1 A7, we have h,(()) (z) a (oo) O, while for Ui=1 A, have N (x) 0. T (b) = - if & U A 0 (+00) 0, otherwise
lim hea
where -y can be any element of the image of R that now non positive function. Therefore for A U²= A7 = ()))
(x) ~ (()
For more intuitive understanding of the class of functions that such constructed second layer can learn, in Figure 13, we report some samples from that class of functions.
Figure 14. Examples of learnable functions the third hidden layer.
Third layer construction Finally, let show that hidden unit in the third layer perform union and intersection operations when the second-layer representations are indicator functions of sets. Lemma A.5. If h (z) = A, there exists a rescaling factor E R. such that a single unit in the third hidden layer of an MLP with non-negative weights can approximate:
(x) R ())
for any = U A, when Th E St, and for any A = U A 0 S
We are finally ready to prove the main result.
Proof of Theorem 3.5. Since the function to interpolate is monotonic, for any couple of points v; <xz: f(x;) < f(az)) it is possible to find a hyperplane with non-negative normal, with positive and negative half spaces denoted by A,,, and Ajio
such that 2; E A 1/31 2y E A
Let us now construct the sets:
40
A+
(23)
Jiz, Ca
AU m;
A
(24)
This ensures that 2ÿ < R: = Iy & AD,- Also, since 42, is obtained from the intersection of positive half-spaces, Lemma A.4 ensures a hidden unit at the second hidden layer is able to learn (x) & Lag, (x). Now note that Ar, = =U - AG, contains only and all points z, such that f(x 2 f(x)) Moreover, from Lemma A.5, we know that hidden units in the third layer can approximate 1A.,'
As per the previous layers, we show in Figure 14 some samples of functions that the third layer, constructed as just reported, can learn.
23

---

Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
Fourth layer construction To conclude the proof, take the last
layer parameters
be
w/"h f(1++1) f(x)), = f(r)) This produces the following function approximation
E1 f(x) f(x1 + Σ LA, (((ttt)) -
f(x) evaluated any of the points 2; provides a telescopic sum where all the terms elide, leaving f(x,) = f(x;). For the opposite activation pattern, the same result can be obtained in similar fashion, considering intersections of AV, = Jarysn 1 instead.
24


Reviews:
[{"review_id": "N5FzLdEwLB", "reviewer": "Reviewer_4GJQ", "summary": "This paper constructs universal approximators for monotonic functions with MLPs with non-negative weight constraint and activations that saturate on alternating sides. Based on the result, the paper shows MLPs with convex monotone activations and non-positive constrained weights can also be universal approximators. Furthermore, the authors proposes the pre-activation and post-activation formulations for monotonic ReLU and ReLU' MLPs to address the optimization challenge of monotonic neural networks constructed with weight constraints.\n\n## update after rebuttal\nMy concerns are well addressed in the rebuttal. I have updated my score to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful comments and suggestions. Below, we address each of your points in detail:\n\n - _\"The two formulations are not equivalent to the original monotonic ReLU MLP.\"_:  \n   It is unclear whether the \"original monotonic ReLU MLP\" refers specifically to the setting described in Theorem 1 or to the \"naive\" ReLU with non-negative weights. For the former case, at the end of page 7 (around line 381), we illustrate how the proposed formulation can be mapped back to the setup used in Theorem 1, establishing that the MLP in Theorem 1 is indeed a special instance of the general formulation we propose. Another reviewer also noted that our description of the former formulation lacked rigor and clarity. We acknowledge this concern, and we plan to significantly revise this section to enhance its clarity and precision in the updated manuscript.  \n   In the latter case, if all weights are constrained to be non-negative, both formulations reduce equivalently to a \"naive\" ReLU MLP. However, this \"naive\" MLP lacks universal approximation capabilities, highlighting the generality and advantage of our pre- and post-activation formulation.  \n\n - _\"The weight constraints still seem to be required.\"_:  \n In Section 4.1, we demonstrate that by explicitly splitting the weights into positive and negative parts, the need for explicit weight constraints is effectively removed. This is crucial to simplify the implementation, avoiding the activation alternation, and relaxing the need for explicit weight constraints.\n\n - _\"Provide a concrete algorithm for optimizing the monotonic ReLU MLP with the proposed formulations.\"_:  \n The proposed parametrization can straightforwardly be optimized using standard gradient-based optimization algorithms, consistent with typical neural network training. This is explicitly demonstrated in the provided reproducible code accompanying our submission.\n\n - _Weakness 1 and \"the advantages of the paper's results and proof techniques.\"_:  \n  Existing proof techniques from [1] cannot directly provide a straightforward proof for constrained monotonic MLPs, primarily because they require negative weights in the proof construction$^*$. To overcome this limitation, we provided in Appendix A.1 an alternative, albeit naive, proof built upon the approach from Runje & Shankaranarayana (2023), in a similar spirit as the one you proposed. Although this alternative requires up to 8 layers (a loose bound), while our primary contribution (Theorem 1) significantly improves this by establishing that only 4 layers are necessary.\n\nWe sincerely appreciate your valuable feedback and hope the clarifications above resolve your concerns. We believe these refinements strengthen our manuscript and illustrate its novelty and practical value. We appreciate your thoughtful feedback and hope our clarifications have sufficiently addressed your concerns, potentially improving your assessment.\n\nThank you once again for your time and constructive suggestions.\n\n[1] Hornik, Kurt, Maxwell Stinchcombe, and Halbert White. \"Multilayer feedforward networks are universal approximators.\" Neural networks 2.5 (1989): 359-366.\n\n\n$^*$ from [1]: \"By adding, subtracting and scaling a finite number of affinely shifted versions...\". They take the difference of threshold functions to create Rect/Box functions. However, we cannot use the \"subtract\" part since it requires negative weights, while Theorem 1 necessitates non-negative weights.", "claims_and_evidence": "In Section 4, the paper claims the pre-activation and post-activation formulations can address the weight constraint challenge. However, it seems that the two formulations are not equivalent to the original monotonic ReLU MLP (line 344). Furthermore, to guarantee monotonity and apply Theorem 3.5, the weight constraints still seem to be required.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The proofs seem correct.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength:\nThe paper constructs universal approximators for monotonic functions with one-side saturating activations by simply modifying non-negativity to non-positivity, which is concise and insightful.\n\nWeakness:\nThe universal approximation result seems direct, considering the construction of Heavyside function (Figure 2) and the universal approximation results of NNs in [1]. The advantages of the paper's results and proof techniques need to be highlighted.\n\n[1] Hornik, Kurt, Maxwell Stinchcombe, and Halbert White. \"Multilayer feedforward networks are universal approximators.\" Neural networks 2.5 (1989): 359-366.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Can you clarify why the pre-activation and post-activation formulations are equivalent to the original monotonic ReLU MLP and address the weight constraint challenge? For example, give a concrete algorithm of optimizing the monotonic ReLU MLP with the proposed formulations.\n2. Can you clarity the advantages of the paper's results and proof techniques? (see Weakness)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A3MLFvOmub", "reviewer": "Reviewer_RAdq", "summary": "This paper proposes a novel Monotonic Neural Network as a universal approximator for monotone functions. Unlike previous works, this approach provides theoretical proof that the proposed Monotonic Neural Networks can serve as universal approximators and successfully removes the constraint of activation function boundedness. As a result, the proposed method enables ReLU-like activation functions to construct monotone networks with universal approximation properties.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your insightful review and constructive suggestions. Below, we address each of your points individually:\n\n - _\"Emphasizing the relevance and potential applications of Monotone MLPs would significantly enhance the paper’s impact.\"_:  \n We fully agree with your suggestion. Monotonic neural networks indeed have diverse applications beyond interpretability and fairness, such as their established use in quantile regression [1]. We will expand the paper to expand on the practical applications of Monotonic NNs, thereby better emphasizing the broader relevance of the topic.\n\n - _\"I strongly recommend further discussion on how the proposed parameterization does not unintentionally introduce new challenges in model stability and learning effectiveness.\"_:  \n Your point is well-taken and insightful. We will include additional analyses in the appendix examining this issue more closely. However, a thorough and detailed analysis of initialization methods would merit an entire standalone study, thus we will also note it in the future works. We also note this potential concern is common among state-of-the-art architectures employing constrained weights and ReLU activations [2, 3]. (for more details see next point)\n\n\n - _\"However, in your proposed MLP, if the input is zero-mean normalized, the network may fail to learn...\"_ & _\"This faster growth suggests that the proposed Monotonic MLP may suffer from instability and increased training difficulty.\"_:  \n We appreciate this important observation.\n The practical implementation provided in our paper employs PyTorch’s default initialization (Uniform Kaiming or Xe), which does not strictly follow the analysis you provided. Nonetheless, similar results to your one can be obtained on this variant.  \n In the revised appendix, we will add further details addressing your concern, also considering biases, and evaluating empirically our settings to normal MLPs. To anticipate the results that will be reported in the paper, we have analyzed the empirical expansion factor, and even though it can be seen that it's not the same as a normal MLP (as predicted by your analysis), it remains in a reasonable range.\n\n - _\"Additional experiments on more complex and diverse tasks are recommended.\"_:  \n As mentioned in the Introduction (around line 92), our primary aim is theoretical exploration rather than proposing a novel state-of-the-art architecture. While our theoretical results open opportunities for creating more expressive monotonic MLPs, we believe that the included benchmark comparisons are sufficient for validating our theoretical contributions. Nonetheless, we acknowledge the value of further experiments and agree they would enhance validation.\n\n - _\"The current discussion is somewhat informal and lacks the necessary theoretical depth to convincingly prove this property.\"_:  \n We agree that the discussion in Section 4.1 (around line 373) could benefit from a more rigorous presentation. We will carefully revise this section, within the constraints of available space, to ensure greater theoretical clarity and rigor.\n\nICML guidelines this year allow authors to submit revised manuscripts only during the second half of the review process. Accordingly, we will upload the revised version as soon as the submission system permits, with the requested improvements and analysis.  \nYour feedback is highly valuable and has helped us improve the manuscript significantly. We hope these clarifications address your concerns and reinforce your positive assessment.\n\nThank you once again for your constructive comments and support.\n\nReferences:\n[1] Chilinski, Pawel, and Ricardo Silva. \"Neural likelihoods via cumulative distribution functions.\" Conference on Uncertainty in Artificial Intelligence. PMLR, 2020.  \n[2] Runje, Davor, and Sharath M. Shankaranarayana. \"Constrained monotonic neural networks.\" International Conference on Machine Learning. PMLR, 2023.  \n[3] Kim, Hyunho, and Jong-Seok Lee. \"Scalable monotonic neural networks.\" The Twelfth International Conference on Learning Representations. 2024.", "claims_and_evidence": "The motivation behind studying Monotone MLPs is not clearly stated. The paper lacks a compelling explanation of the practical importance of this research, especially in the current popularity of LLMs and modern deep-learning frameworks. Emphasizing the relevance and potential applications of Monotone MLPs would significantly enhance the paper’s impact. Without this context, the work risks being perceived as a theoretical exercise with limited practical utility.", "methods_and_evaluation_criteria": "seems good.", "theoretical_claims": "The parameterization method described in Equation (12) effectively addresses the weight constraint and eliminates the need to alternate between activations and their point-reflected counterparts manually. However, this parameterization appears to alter the behavior and capacity of the resulting MLP significantly. Consider the following analysis:\n- Suppose $x$ is given, and $W_1, W_2$  are drawn from the standard Gaussian distribution. For a standard MLP, $f(x) = Relu(W_1x)$, By linearity of expectation, we have:\n$$\n\\mathbb{E}[f(x)] = ||x|||/\\sqrt{2\\pi}\n$$\nFor a two-layer MLP: $f(x) = Relu(W_2 \\cdot Relu(W_1x))$, It follows that:\n$$\n\\mathbb{E}[f(x)] = \\frac{||x|||\\sqrt{d}}{2\\sqrt{\\pi}}\n$$\nwhere $d$ denotes the hidden layer dimension.\n\n- Now consider the proposed parameterization: $f(x) = W^+_1 \\cdot Relu(y) +  W^-_1 \\cdot Relu(-y)，y = W^+_1 \\cdot Relu(x) +  W^-_1 \\cdot Relu(-x)$ Then:\n$$\n\\mathbb{E}[y]  = \\frac{\\sum_i{x_i}}{\\sqrt{2\\pi}}, \\quad \\mathbb{E}[f(x)] = \\frac{d \\cdot \\sum_i{x_i}}{{2\\pi}}.\n$$\n\nFrom this, two key issues arise:\n   1.  For a standard MLP, the network can inherently capture non-zero mean outputs. However, in your proposed MLP, if the input is zero-mean normalized, the network may fail to learn meaningful features in terms of $\\mathbb{E}[f(x)] $. \n   2. Even if we assume that the summation and the norm are equivalent in some intuitive sense, the proposed MLP expands the mean value magnitude by $O(m)$ from one layer to the next, whereas a standard MLP only scales by $O{\\sqrt{m}}$. This faster growth suggests that the proposed Monotonic MLP may suffer from instability and increased training difficulty.\n\nIn light of these concerns, I strongly recommend further discussion on how the proposed parameterization does not unintentionally introduce new challenges in model stability and learning effectiveness despite addressing gradient vanishing issues in some sense.", "experimental_designs_or_analyses": "The experimental evaluation is relatively small in scale. Although the authors compare their method with prior works that also use small-scale experiments, the diversity of datasets and task types is insufficient. To convincingly demonstrate the effectiveness and robustness of the proposed method, additional experiments on more complex and diverse tasks are recommended.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "Based on the above observations, the paper should provide a more rigorous theorem that formally establishes the universal approximation property of the proposed Monotonic Neural Networks under the parameterization described in Equation (12). The current discussion is somewhat informal and lacks the necessary theoretical depth to convincingly prove this property.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "COdIZJXfX6", "reviewer": "Reviewer_gyeZ", "summary": "This paper extends the body of work on monotonic neural networks. \nIt focuses on relaxing the existing constraints that limits the architecture, use, and performance of such networks. \nSpecifically, this work identifies limitations within the existing architecture including the use of threshold neurons, which limits the choice of activation functions and the expressivity of monotonic neural networks. \nIt also identifies potential benefits in reducing the required number of hidden layers for universal approximation with monotonic networks. \n\nIn addressing these limitations and others, this work presents theorems that carefully answer the research questions raised.\nIt proves that universal approximation theorem for non-threshold activation, thereby relaxing the constraint on the choice of activation functions for monotonic networks. \nThis work carefully reviews the non-negative constraint on the weights and claims that using a non-positive weight constraint is more expressive. It also presented a new parametrization method to better determine the weight constraint.\n\nThis work is a valuable contribution to the body of work on monotonic neural representation and its extension to the design of interpretable models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your constructive review and valuable comments. Below we address each of your concerns in detail:\n\n- _Typos & Figure 1_:  \nThank you for highlighting these issues. We have thoroughly revised the manuscript, corrected all identified typos, and will explicitly cite Figure 1 in the main text.\n\n- _\"How do non-threshold and threshold activations compare on the training datasets?\"_:  \nWe omitted this comparison since it has already been explored in [1] (under the name of \"Non-Neg-DNN\"), where threshold activations were demonstrated to outperform non-threshold approaches. Our proposed method, in turn, matches or surpasses the results obtained by [1]. In the revised version, given that the allowed space is enough, we will add such result in Table 1.\n\n- _\"Can you illustrate the effect of the non-negative and non-positive weight constraints on the training dataset?\"_:  \nWe appreciate your question, but we would kindly ask for clarification on this point to ensure we accurately address your concern.\n\n- _\"What is the additional computational cost of the new weight parameterization method?\"_:  \nThis aspect is addressed at the end of Section 4.1 (around line 403). Practically, our proposed method introduces negligible overhead and is often computationally cheaper compared to sigmoid-constrained monotonic MLPs, as it avoids the computational costs associated with sigmoid activations and their gradients.\n\n- _\"Please consider defining the hyperplanes... Also consider adding the explicit definition of...\"_:  \nThank you for noting these ambiguous definitions. We will clearly define these terms in the revised manuscript to ensure precision and readability.\n\nYour feedback significantly helps in improving our manuscript. ICML guidelines this year permit uploading revised manuscripts only during the second half of the reviewing process. We will therefore submit our revised paper as soon as the submission system allows. We hope our clarifications fully address your concerns and further strengthen your positive assessment. We hope these clarifications address your concerns and reinforce your positive assessment.\n\nThank you once again for your support and valuable suggestions.\n\n[1] Runje, Davor, and Sharath M. Shankaranarayana. \"Constrained monotonic neural networks.\" International Conference on Machine Learning. PMLR, 2023.", "claims_and_evidence": "The claims and evidence are clear and convincing.\n\nThe presentation is clear and easy to follow. \nThe paper provides sufficient background information to understand this work. \nIt carefully presents and discusses the focus of this work under subsection 3.1, including a clear description of the limitations of existing monotonic MLPs. \nSubsequently, the paper provides convincing evidence to support the central claim of addressing these limitations. \nSubsections 3.2 provides the proofs that address the use of non-threshold activations and the required number of hidden layers for universal approximation. \nSubsection 3.3 provides the justification for a new weight parametrization method. \nSection 4 presents the new method for the weight parametrization.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are relevant to the work in this paper.", "theoretical_claims": "Yes, I checked the proofs for Theorem 3.5 and the Lemmas.", "experimental_designs_or_analyses": "Yes, I checked the experimental analysis.\n\nThe experimental results are relevant and illustrate the benefit of the new approach proposed in this work.", "relation_to_broader_scientific_literature": "The results in this paper are important to the broader scientific literature because they offer a new perspective on the design of monotonic neural architectures. This in turn contributes to the development of explainable models.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "ORIGINALITY: This work is original because it clearly identifies the limitations of existing monotonic neural architectures and seeks to address the identified issues.\n\nSIGNIFICANCE: The potential impact of this work in designing easy-to-interpret models underscores its significance.\n\nCLARITY: This paper is well-organized, and its presentation is very clear.\n\n\nWEAKNESSES: \n(1) You should consider additional experimental information to further strengthen the central claim of this work. (Check my comment under Questions For Authors)\n(2) The definitions of some notations are not clear. Define the notations under the proofs. (Check my comment under Questions For Authors)", "other_comments_or_suggestions": "Here are some suggestions on typos:\n\nLine 426-426, Column 2, Section 6, Page 8 \n\"We then use this theoretical analysis to construct of a novel parametrization that relaxes the weight constraint making the network less sensible to intialization.\"\nI think \"construct of a\" should be \"construct a\". I think \"sensible\" should be \"sensitive\". I think \"intialization\" should be \"initialization\".  \n\nLine 055-056, Column 1, Section 2, Page 2\n\"Hard monotonicity instead gives guarantees by construction by imposing constraints in the model architecture\"\nThis sentence is not clear to me. Please review.\n\nI do not think the body / text of this paper references Figure 1 directly.", "questions_for_authors": "I think there are other experiments or discussions that could further strengthen the contributions and central claims of this paper.\n \n(1).  How do non-threshold and threshold activations compare on the training datasets?\n(2).  Can you illustrate the effect of the non-negative and non-positive weight constraints on the training dataset? \n(3).  What is the additional computational cost of the new weight parametrization method?\n\n\nPlease consider defining the hyperplanes $A_{i_2/i_1}^{+}$, $A_{i_2/i_1}^{-}$ under the Proof for Theorem 3.5 mathematically on pages 5 and 13 (Just like we have in Lemma 3.6). This will improve the readability of the proofs. Also consider adding the explicit definition of $A_{j/i}^{-}$ in equation (6).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PiKoiWYSmb", "reviewer": "Reviewer_C9i1", "summary": "Authores generalize previous theoretical results, showing that MLPs with non-negative weight constraint and activations that saturate on alternating sides are universal approximators for monotonic functions. Additionally, they show an equivalence between saturation side in the activations and sign of the weight constraint. This allows them to prove that MLPs with convex monotone activations and non-positive constrained weights also qualify as universal approximators, in contrast to their non-negative constrained counterparts.\n\nExperimental evaluation reinforce the validity of the theoretical results, showing that their approach compares favorably to traditional monotonic architectures", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your positive feedback. We appreciate your suggestion and will clarify the definitions of the terms you've highlighted, in our revised manuscript.\n\nIf you have any additional recommendations or improvements you'd like to see to elevate your rating further, please let us know—we would greatly appreciate your advice.\n\nThank you again for your support and constructive comments.", "claims_and_evidence": "Claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Proposed methods and/or evaluation criteria make sense for the problem or application at hand.", "theoretical_claims": "I checked all proofs in the main part and did not have any issues.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "I am not familiar enough with the broader Scientific Literature in thsi area to give a statement here.", "essential_references_not_discussed": "I am not familiar enough with the broader Scientific Literature in thsi area to give a statement here.", "other_strengths_and_weaknesses": "strengths:\n\n-problem considered is initerising\n\n-analysis is non trivial and intuitive\n\n-experiments support validity of the theoretical results\n\nweaknesses:\n\n-results are restricted to monotonic unctions.\n\n-analysis is fairly simple", "other_comments_or_suggestions": "For $h_j$ and $ A_{j/  i}$  it would be nice if $j$ would ne defined. Same goes for $n\\geq 4$) in our contributions", "questions_for_authors": "no questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper makes an important theoretical contribution by generalizing previous results on neural network approximation capabilities. Specifically, it demonstrates that MLPs with non-negative weight constraints and properly designed saturation activations can serve as universal approximators for monotonic functions.

The theoretical analysis seems correct and provides valuable insights into the representational power of constrained neural architectures. Now all reviewers found the results to be technically sound and conceptually interesting. I recommend acceptance.
Decision: Accept (poster)