Expressive Monotonic Neural Networks
Authors: Niklas Nolte
Ouail Kitouni
Mike Williams
Venue: ICLR 2023 poster (2023)
Keywords: monotonic
lipschitz

Abstract: The monotonic dependence of the outputs of a neural network on some of its inputs is a crucial inductive bias in many scenarios where domain knowledge dictates such behavior. This is especially important for interpretability and fairness considerations. In a broader context, scenarios in which monotonicity is important can be found in finance, medicine, physics, and other disciplines. It is thus desirable to build neural network architectures that implement this inductive bias provably. In this work, we propose a weight-constrained architecture with a single residual connection to achieve exact monotonic dependence in any subset of the inputs. The weight constraint scheme directly controls the Lipschitz constant of the neural network and thus provides the additional benefit of robustness. Compared to currently existing techniques used for monotonicity, our method is simpler in implementation and in theory foundations, has negligible computational overhead, is guaranteed to produce monotonic dependence, and is highly expressive. We show how the algorithm is used to train powerful, robust, and interpretable discriminators that achieve competitive performance compared to current state-of-the-art methods across various benchmarks, from social applications to the classification of the decays of subatomic particles produced at the CERN Large Hadron Collider.

Published as a conference paper at ICLR 2023
EXPRESSIVE MONOTONIC NEURAL NETWORKS
Niklas Nolte", Ouail Kitouni', Mike Williams
The NSF AI Institute for Artificial Intelligence and Fundamental Interactions Massachusetts Institute of Technology Cambridge, MA 02139, USA {nnolte, kitouni, mwill @mmit emittedu
ABSTRACT
The monotonic dependence of the outputs of a neural network on some of its inputs is crucial inductive bias in many scenarios where domain knowledge dic- tates such behavior. This is especially important for interpretability and faimess considerations. In broader context, scenarios in which monotonicity is impor- tant can be found in finance, medicine, physics, and other disciplines. It is thus desirable to build neural network architectures that implement this inductive bias provably. In this work, we propose weight-constrained architecture with single residual connection to achieve exact monotonic dependence any subset of the inputs. The weight constraint scheme directly controls the Lipschitz constant of the neural network and thus provides the additional benefit of robustness. Com- pared currently existing techniques used for monotonicity, our method is sim- pler in implementation and in theory foundations, has negligible computational overhead, guaranteed produce monotonic dependence, and is highly expres- sive. We show how the algorithm is used to train powerful, robust, and inter- pretable discriminators that achieve competitive performance compared current state-of-ie totte-of-theassdd the-art methods across various benchmarks, from social applications to the classification the decays of subatomic particles produced at the CERN Large Hadron Collider.
INTRODUCTION
The need to model functions that are monotonic in subset of their inputs is prevalent in many ML applications. Enforcing monotonic behaviour can help improve generalization capabilities (Mi- lani Fard et al. 2016 You al.) 2017) and assist with interpretation of the decision-maie making process of the neural network (Nguyen & Martinez 2019). Real world scenarios include various applications with fairness, interpretability, and security aspects. Examples can be found in the natural sciences and in many social applications. Monotonic dependence of model output on certain feature in the input can be informative of how an algorithm works-and some cases essential for real- word usage. For instance, good recommender engine will favor the product with high number of reviews over another with fewer but otherwise identical reviews (ceteris paribus). The same ap- plies for systems that assess health risk, evaluate the likelihood of recidivism, rank applicants, filter inappropriate content, etc.
In addition, robustness small perturbations in the input is a desirable property for models deployed in real world applications. In particular, when they are used to inform decisions that directly affect human actors-or where the consequences of making be extremely costly. The continued existence of adversarial methods is good example for the possi- bility of malicious attacks on current algorithms (Akhtar al. 2021). I) natural way of ensuring the robustness of a model is to constrain its Lipschitz constant. To this end, we recently developed an ar- chitecture whose Lipschitz constant constrained by design using layer-wise normalization which allows the architecture to be more expressive than the current ofate-of-the-art with stable and fast training (Kitouni et al., 2021). Our algorithm has been adopted to classify the decays of subatomic particles produced at the CERN Large Hadron Collider in the real-time data-pren bata-processing system of the LHCb experiment, which was our original motivation for developing this novel architecture.
unexpected and unwanted decision could
"Equal contribution

---

Published as a conference paper at ICLR 2023
In this paper, we present expressive monotonic Lipschitz networks. This new class of architectures employs the Lipschitz bounded networks from Kitouni et al. (2021) along with residual connections to implement monotonic dependence in any subset of the inputs by construction. It also provides exact robustness guarantees while keeping the constraints minimal such that remains a universal approximator of Lipschitz continuous monotonic functions. We show how the algorithm is used to train powerful, robust, and interpretable discriminators that achieve competitive performance com- pared to current state-of-the-art methods across various benchmarks, from social applications its original target application: the classification of the decays of subatomic particles produced at the CERN Large Hadron Collider.
RELATED WORK
Prior work in the field of monotonic models can be split into two major categories.
Built-in and constrained monotonic architectures: Examples of this category include Deep Lattice Networks You et al. 2017) and networks in which all weights are constrained to have the same sign (Sill 1998). The major drawbacks of most implementations of constrained architec- tures are a lack of expressiveness poor performance due to superfluous complexity.
Heuristic and regularized architectures (with or without certification): Examples of such methods IncludeSiill & Abu-Mostafa (1996) and and[G et al. which penalizes point- wise negative gradients on the training sample. This method works on arbitrary architectures and retains much expressive power but offers no guarantees to the monotonicity of the trained model. Another similar method is Liu et al. 22200), which relies on Mixed Integer Linear Programming to certify the monotonicity of piece-wise linear architectures. The method uses heuristic regularization to penalize the non-monotoniety of the model on points sampled uniformly in the domain during training. The procedure is repeated with increasing regularization strength until the model passes the certification. This iteration can be expensive and while this method is more flexible than the constrained architectures (valid for MLPs with piece-wise linear activations), the computational overhead of the certification process can be prohibitively expensive. Similarly, /iiaaaaan et al. (2020) propose guaranteed monotonicity for standard ReLU networks by letting a Satisfiability the monotonicity definition and adjust the
Modulo Theories (SMT) solver find counterexamples prediction in the inference process such that monotonicity is guaranteed. However, this approach requires queries to the SMT solver during inference time for each monotonic feature, and the computation time scales harshly with the number of monotonic features and the model size (see Figure 3 and in Sivaraman et al. )020
Our architecture falls into the first category. However, we overcome both main drawbacks: lack of expressiveness and impractical complexity. Other related works appear in the context of monotonic functions for normalizing flows, where monotonicity is key ingredient to enforce invertibility
(De Cao et al. 2020 Huang et al. 2018 Behrmann et al. 2019 Wehenkel & &Louppe 2019).
3 METHODS
The goal is to develop neural network architecture representing a vector-valued function f:R² Rd + R., d, E N,
(1)
that is provably monotonic in any subset of its inputs. We first define few ingredients. Definition 3.1 (Monotonicity). Let a E IRd, ass = 1g @ and the Hadamard product of æ with the indicator vector (e) = i E S
and 0 otherwise for a subset $ C [1.-.... .d).
We say that outputs Q C [1, n) of f are monotonically increasing in features $ if f( + mg)x I f(x + +g); Vi E Q and Val S as.
(2)
where 8 denotes the complement of S and the inequality on the right uses the product (or component- wise) order.
Definition 3.2 (Lip² function). g : d + R. is Lip" if it is Lipschitz continuous with respect to the L.P norm in every output dimension, i.e.,
[lg(x) - A|Iac all, Var, y R
(3)
2

---

Published as a conference paper at ICLR 2023
3.1 LIPSCHITZ MONOTONIC NETWORKS (LMN)
We will henceforth and without loss of generality only consider scalar-valued functions (n = 1). We start with model g(xr) that is Lip' with Lipschitz constant A. Note that the choice of is crucial for decoupling the magnitudes of the directional derivatives in the monotonic features. More details on this can be found below and in Figure [ The 1-norm has the convenient side effect that we can tune the robustness requirement for each imput individually.
With model g(x) we can define an architecture with built-in monotonicity by adding a term that has directional derivative À for each coordinate in &:
((a) g(x) + X(13 x) = g(x))
>
Vi-
(4)
BES
This residual connection X(1g a) enforces monotonicity in the input subset æg:
0g [-,, ViE N1.
(5)
dr.
0f 0g dr.
VoER",iES.
(6)
dr
The importance of the norm choice The construction presented here does not work with * constraints because dependencies between the partial derivatives may be introduced, The p 1-norm is the only norm that bounds the gradient within the green square and, crucially, allows the directional derivatives to be large 2 independently. When shifting the constraints by introducing the linear term, the green square allows for all possible gradient configurations, given that we can choose À freely. As counter example, the red cirele, corresponding to p 2 constraints, prohibits important areas in the configuration space.
Figure T
0f 0rg
II [h S
I SA
RCB
of dx
2X
2X
Figure 1: p-norm constrained gradients showing (red) p 2 and (green) 1. The gradient of function g(x)) that is Lip"=2 resides within the dashed red line. For a Lip function, the boundary is the green dashed line. Note that ø is taken to be row vector. The residual connection (in blue) effectively shifts the possible gradients to strictly positive values and thus enforces monotonicity. Note how the red solid circle does not include all possible gradient configurations. For instance, does not allow for very small gradients both inputs, whereas the green square includes all configurations, up to an element-wise maximum of 2A.
To be able to represent all monotonic Lip" functions with 2A Lipschitz constant, the construction of g(x) needs to be universal approximator of Lip' functions. the next section, we will discuss possible architectures for this task.
3.2
LIPP=1 APPROXIMATORS
Our goal is to construct universal approximator of Lip' functions, i.e., we would like the hypothesis class to have two properties:
1. It always satisfies Eq. 3 i.e., be Lip".
2. able to fit all possible Lip" functions. In particular, the bound in Eq. 3 needs to attainable æc, y-
3

---

Published as a conference paper at ICLR 2023
Lip' constrained models To satisfy the first requirement, fully connected networks can be Lip- schitz bounded by constraining the matrix norm of all weight matrices (Kitouni et al.) 2021 Gouk et al. 22000; Miyato al.) (018). We recursively define the layer of the fully connected network of depth D with activation a as
azz +b',
(7)
where x" === x is the input and f(x) D is the output of the neural network. It follows that g(x) satisfies Eq. 3
D I S
and a has a Lipschitz constant less than or equal to There are multiple ways enforce Eq. Two existing possibilities that involve scaling by the operator norm of the weight matrix (Gouk et al. 2020 are:
W
W
W W AID
W W
max
11)
max max (,, (1))
In our studies, the latter variant seems to train slightly better. However, in some cases it might be useful use the former to avoid the seale imbalance between the neural network's output and the residual connection used to induce monotonicity. We note that in order to satisfy Eq. 8 it is not necessary to divide the entire matrix by its 1-norm. It is sufficient to ensure that the absolute sum over each column is constrained:
W W d'ddig
(10)
max (1,A-1/.. E; [
This novel normalization scheme tends give even better training results in practice, because the constraint is applied in each column individually. This reduces correlations of constraints, in partic- ular, if column saturates the bound on the norm, the other columns are not impacted. While Eq. IO may not be suitable as general-purpose scheme, e.g. it would not work in convolutional networks, its performance training in our analysis motivates its use in fully connected architectures and further study of this approach in future work.
In addition, the constraints in Eq. and Eq. To can be applied in different ways. For example, one could normalize the weights directly before each call such that the induced gradients are propagated through the network like Miyato et al. (2018). While one could come up with toy examples for which propagating the gradients in this way hurts training, it appears that this approach is what usually is implemented for spectral norm in PyTorch and TensorFlow (Miyato et al. 2018). Alter- natively, the constraint could be applied by projecting any infcasible parameter values back into the set of feasible matrices after each gradient update as in Algorithm 2 Gouk et al. (2020).
Constraining according to Eq. 8 is not the only way to enforce Lip'. Anil et al. 2019 provide an alternative normalization scheme:
11, II Io0 SA
(1I)
Similarly how the I-norm of matrix is a column-wise maximum, the oc-norm of a matrix is determined by the maximum 1-norm of all rows and ||]],, simply equals the maximum absolute value of an element the matrix. Therefore, normalization schemes similar to Eq. 10 can be employed to enforce the constraints in Eq. [1 by replacing the column-wise normalization with row- or element-wiss normalization where appropriate.
Preserving expressive power Guaranteeing that the model is Lipschitz bounded is not sufficient, it must also able to saturate the bound to be able to model all possible Lip" functions. Some Lipschitz network architectures, e.g. Miyato al. (2018), tend to over constrain the model such that it cannot fit all Lip' functions due to gradient attenuation. For many problems this is a rather theoretical issue. However, it becomes practical problem for the monotonic architecture since it often works
4

---

Published as a conference paper at ICLR 2023
on the edges of its constraints, for instance when partial derivatives close to zero required, Figure [ As simple example, the authors of Huster et al. (2018) showed that ReLU networks are unable to fit the function f(z) |a| if the layers in fact that ReI.U, and most other commonly used activations, do not have unit gradient with respect to the inputs over their entire domain. While monotonic element-wise activations like ReLU cannot have unit gradient almost everywhere without being exactly linear, the authors of Anil et al. al.] (2019) explore activations that introduce non-linearities by reordering elements of the input vector. They propose GroupSort as an alternative to point-wise activations, and it is defined as follows:
conntconsta with À = 1. The reason lies
0G (x) (c = co + 501G++120(GG4120) +...
n/G-1
Σ
5011G44
(12)
where x E R", Di:5 = 15: O æ, and sort. orders the elements a vector from indices to j and leaves the other elements in place. This activation sorts input vector in chunks (groups) of
a fixed size G. The GroupSort operation has a gradient of unity with respect to every input, giving architectures constrained with Eq. 8 greatly increased expressive power. In fact, Anil al. (2019) prove that GroupSort networks with the normalization scheme in Eq. [ are universal approximators of Lip' functions. Therefore, these networks fulfill the two requirements outlined in the beginning of this section.
For universal approximation be possible, the activation function used needs to be gradient norm preserving (GNP), i.e. have gradient 1 almost everywhere. Householder activations are another in- stance of GNP activations of which GroupSort-2 a special case (Singla et al., 202T). The House- holder activation is defined as follows:
a = [2
20 > 0
(13)
(I-I 2CII-20'' 2v07)
Here, is the preactivation row vector and v is any column unit vector. Householder Lipschitz Networks naturally inherit the universal approximation property.
In summary, we have constructed neural network architecture f(x) via Eq. that can provably approximate all monotonic Lipschitz bounded functions. The Lipschitz constant of the model can be increased arbitrarily by controlling the parameter in our construction.
4 EXPERIMENTS
"Beware of bugs in the above code, have only proved it correct, not tried it" (nuthh). In the spirit of Donald Knuth, this section well in practice and gives competitive results, as should be expected from universal approximator.
algorithm many different domains to show that it works
4.1 ToY EXAMPLE
Figure shows a toy example where both monotonic and an unconstrained network are trained to regress on noisy mieddiensiooal dataset. The true underlying model used here is monotonic, though an added heteroskedastic Gaussian noise term can obscure this in any realization. As can be seen Figure 2 no matter how the data are distributed the edge of the support, the monotonic Lipschitz network is always non-decreasing outside of the support as guaranteed by our architecture. Such outoo-diistribttin guarantees can be extremely valuable cases where domain knowledge dictates monotonic behavior is either required or desirable.
4.2 REAL-TIME DECISION-MAKING AT 40 MHZ AT THE LHC
Because many physical systems are modeled with well-known theoretical frameworks that dictate the properties the system, monotonicity can a crucial inductive bias in the physical sciences. For instance, modeling enthalpy, thermodynamic quantity measuring the total heat content of system, in a simulator requires monotonic function temperature for fixed pressure (as is known from basic physical principles).
5

---

Published as a conference paper at ICLR 2023
2.5
Monotonic Unconstrainee
Monotonic
2.0
2.0
Unconstrained
train
train
1.5
1.5
1.0
1.0
0.5
0.0
0.0
0.5
-0.5
1
2
Figure 2: Our monotonic architecture (green) and an unconstrained network (red) trained on two realizations (purple data points) of a one dimensional dataset. The shaded regions are where training data were absent. Each model is trained using 10 random initialization seeds. The dark lines are averages over the seeds, which are each shown as light lines. The unconstrained models exhibit overfitting of the noise, non-monotonic behavior, and highly undesirable and unpredictable results when extrapolating beyond the region occupied by the training data. Conversely, the monotonic Lipschitz models are always monotonic, even in scenarios where the noise is strongly suggestive of non-monotonic behavior. In addition, the Lipschitz constraint produces much smoother models.
In this section, we describe a real-world physics application which requires monotonicity in certain features and robustness in all of them. The algorithm described here has, in fact, been implemented by a high-energy particle physics experiment at the European Center for Nuclear Research (CERN), and is actively being used to collect data at the Large Hadron Collider (LHC) in 2022, where high- energy potoonproton collisions occur at 40 MHz.
The sensor arrays of the LHC experiments produce data at rate of over 100 TB/s. Drastic data- reduction is performed by custom-buil read-out electronics; however, the annual data volumes are still (100)) exabytes, which cannot be put into permanent storage. Therefore, each LHC experiment processes its data in real time, deciding which proton-proton collision events should be kept and which should be discarded permanently; this referred to as triggering in particle physics. To be suitable for use in trigger systems, classification algorithms must be robust against the impact of experimental instabilities that occur during data taking-ann and deficiencies simulated training samples. Our training samples cannot possibly account for the unknown new physics that we hope to learn by performing the experiments!
A ubiquitous inductive bias at the LHC is that outlier collision events more interesting, since we are looking for physics that has never been observed before. However, uninteresting outliers frequently caused by experimental imperfections, many of which are included and labeled as background in training. Conversely, it is not possible to include the set of all possible interesting outliers priori in the training. A solution to this problem is to implement outliers are better directly using our expressive monotomic Lipschitz architecture from Section 3
Our architecture was originally developed for the task of classifying the decays of heavy-flavor par- ticles produced at the LHC. These are bound states containing a beauty or charm quark that travel an observable distance O(1 cm) before decaying due to their (relatively) long lifetimes. This example uses dataset of simulated proton-prott proton (pp) collisions in the LHCb detector. Charged particles recorded by LHCb are combined pairwise into decay-vertex (DV) candidates. The task concerns discriminating DV candidates corresponding to heavy-flavor decays from all other sources. Heavy- flavor DVs typically have substantial separation from the pp collision point, due to the relatively long heavy-flavoe particle lifetimes, and large transverse momenta, PT, of the component particles, due to the large heavy-flave particle masses. The main sources background DVs, described in Kitouni al. 12 mostly have small displacement and small pr, though unfortunately they can also have extremely large values of both displacement and momentum.
Figure shows simplified version of this problem using only the two most- powtrfl inputs. Our inductive bias requires a monotonic increasing response in both features (detailed discussion moti- vating this bias can be found in Kitouni al. (02T)). We see that unconstrained neural network
6

---

Published as a conference paper at ICLR 2023
raconstrained NN AUC-0.94
monotonic BDT
monotonic Lipschitz NN AUC=0.93
AUC=0.93
0.8
5 Pr [GeV]
0.2
6
log (miin Xip.)
Simplified version of the heavy-quark selection problem using only two inputs, which permits displaying the response everywhere the feature space; shown here heat map with more signal-like (background like) regions colored blue (red). The dark solid line shows the decision boundary (upper right regions are selected). Shown are (left) a standard fully connected neural network, (middle) monotonic BDT, and (right) architecture. The quantities shown on the horiztonal and vertical axes are related to how long the particle lived before decaying and how massive the particle was, respectively.
Figure 3: From Kitouni al. (202)):
0.8
0.8
Arbissary Ubits
0.6
monatonic Lipschitz NN
0.6
monotonic BDT
0.1
unconstrained NN
data distaiiation.
0.2
0.2
efficiency
0.0
0
10
29
30
30
10
20
30
Lifetime [ps]
Figure 4: From Kitouni et al. (2021): True positive rate (efficiency) of each model shown in Fig- ure versus the proper lifetime of the decaying heavy-quark particle selected. The monotonic models produce a nearly uniform efficiency above a few picoseconds at the expense of a few percent ffetime-integrann efficiency. Such trade off desirable as explained in the text.
rejects DVs with increasing larger displacements (lower right corner), and that this leads to de- crease of the signal efficiency (true positive rate) for large lifetimes. The unconstrained model vio- lates our inductive bias. Fad works here. However, the jagged decision boundary can cause problems in subsequent analysis of the data. Bigureaalso shows that our novel approach from Section [ectioon]suucesssfulll produces smooth and monotonic response, and Figure shows that this provides the monotonic lifetime dependence desire in the efficiency.
that monotonic BDT (Auguste et al. (2020) approach
In addition, we note that the added benefit of guaranteed Lipschitz robustness is major advantage for many real world applications. Specifically for particle physicists. this kind of robustness directly translates to important guarantees when considering experimental instabilities.
Due to the simplicity and practicality of method, the LHCb experiment is now using the proposed architecture for real-time data selection at a data rate of about 40 Tbit/s.
4.3 PUBLIC DATASETS WITH MONOTONIC DEPENDENCE
In this section, we follow as closely as possible the experiments done in Liu et al. (202)), and some experiments done in Sivaraman al. (2020) to be able directly compare to of-te-of-the-arn mono- tonic architectures. Liu et al. (2020) studied monotonic architectures on four different datasets:
COMPAS Larson & Kirchner 2016). BlogFeedback (Buza 2014), LoanDefaulter (Kaggle 2015)

---

Published as a conference paper at ICLR 2023
and ChestXRay (Wang et al., ).17). From Sivaraman et al. (2020) compare against regres- sion and one classification task: AutoMPG (Dua & Grafi 2017)) and HeartDisease (Gennari et al, 1989). Results are shown in Table
COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) refers to a commercial algorithm used by judges and police officers to determine the likelihood of reoffense.
Larson & Kirchner (2016) discusses that the algorithm is racially biased and provides dataset from two-year study of the real-world performance of COMPAS. The task here is to predict the reoffense probability within the next two years. The dataset has 13 features, 4 of which have a monotonic inductive bias, and contains total of 6172 data points.
BlecBee This dataset contains 54270 data points with 276 dimensions describing blog posts. The task is predict the number of comments following the post publication within 24 hours. 8 of the features have a monotonic inductive bias. Just like Liu et al. (2020), we also only consider the 90% of the data points with the smallest targets to not let the RMSE be dominated by outliers.
The version of this dataset available Kaggle updated yearly basis up to 2015. Kaggle (2015) contains a link that is, we believe, a superset the data used Liu eet al. (2020) Luckily, the authors have shared with the exact version of the dataset they used in their studies for an appropriate comparison. The data is organized in 28 features and the task to determine loan defaulters. The classification score should be monotonic in 5 features: non- decreasing in number of public record bankruptcies and Debt-to-Incom ratio, non-increasing in credit score, length of employment and annual income.
LoanDefaulter
ChestXRay This dataset contains tabular data and images of patients with diseases that are visible in chest x-ray. The task is predict whether or the patient has such a disease. Just like Liu et al. (2020), we send the image through an ImageNet-pretrained ResNet18 (He et al., 2016). The penultimate layer output concatenated with tabular data acts as input to the monotonic architecture. Two of the four tabular features are monotonic. In the bottom right table in [ there are two entries for our architecture. The E-E entry refers end-to-end training with ResNet18, whereas the other experiment fixes the ResNet weights.
AutoMPG (Dua Graff 2017) This dataset containing 398 examples of cars, described by 7 numerical features and the model name. The target, MPG, is monotonically decreasing with 3 of the features. The name is not used feature.
HeartDisease (Gennari et al. 1989) is a dataset of patients, described by 13 features. The task is determine whether or not the patient has heart discase.
As can be seen in Table our Lipschitz monotonic networks perform competitively or better than the state-of-ther the-trt on all benchmarks we tried.
It also immediately apparent that our architecture is highly expressive. We manage to train tiny networks with few parameters while still achieving competitive performance. Given that some of these datasets have a significant number of features compared to our chosen network width, most parameters are in the weights of the first layer. We manage to build and train even smaller networks with better generalization performance when taking only few important features. These networks are denoted with mini in Table [ Because all of the presented architectures are small in size, we show practical finite sample expressiveness for harder tasks and larger networks by achieving 100% training accuracy MINST, 10, and 1IFAR-100 with real and random labels well as an augmented version (i.e. with an additional monotonic feature added artificially) of CIFAR in A Appendix A
5 LIMITATIONS
We are working on improving the architecture as follows: First, common initialization techniques not optimal for norget networks (Arpit et al. 2019). Simple modifications to the weight
8

---

Published as a conference paper at ICLR 2023
COMPAS
BlogHeedbacc
Method
Parameters
11 RMSE
Method Certified
Parameters
T Test Acc (68.8 + 0.2)% (69.3 0.1)%
Certified
8492
.158 + .001
23112 37
LMN
2225
.160 + .00
LMN
LMN mini
177
.155 + .001
LoanDefaulter
ChestXRay
Method
Parameters
Test Acc
Method
Parameters
11 Test Acc
Certified
12792
(62.3 1: 0.2)%
Certified
8502
(65.2 + 0.1]%
Certified E-E
12792
(66.3 1 1.0)%
LMN
753
(65.44 0.03)% (65.28 0.01)%
LMN
1043
(67 6 + 0.6)%
LMN mini
LMN E-E
1043
(70.0 1.4)%
Heart Discase
Auto MPG
Method
I Test Acc
Method
II MSE
COMET
(86 + 3)%
COMET
(8.81 + 1.81)%
LMN
1 + 1.9)%
LMN
(7.58 + 1.2)%
Table 1: We compare method (in bold) against ofate-of-the-art monotonic models (we only show the best) on a variety of benchmarks. The performance numbers for other techniques were taken from Liu al. (020) and Sivaraman al. (2020). In the ChestXRay experiment, model with frozen ResNet18 weights (second to last) and another with end-to-end training (last). While models generally get quite small, taking a subset of all the features. These models are denoted with "mini".
train one
achieve smaller models when only
initialization might aid convergence, especially for large Lipschitz parameters. Secondly, we are currently constrained activation functions that have gradient norm of over their entire domain, such as GroupSort, to ensure universal approximation, see Anil et al. (2019). We will explore other options the future. Lastly, there not yet proof for universal approximation for the architecture described in Eq. 8 However, it appears from empirical investigation that the networks do approximate universally, as we have yet to find a function that could not be approximated well enough with deep enough network. We do not consider this major drawback, as the construction Eq. TT does approximate universally, see Anil et al. (20T9). Note that none of these limitations have any visible impact on the performance of the experiments in Section
6 CONCLUSION AND FUTURE WORK
We presented an architecture that provably approximates Lipschitz continuous and partially mono- tonic functions. Monotonic dependence enforced via an end-to-end residual connection to min- imally Lip constrained fully connected neural network. This method is simple to implement, has negligible computational overhead, and gives stronger guarantees than regularized models. Our architecture achieves competitive results with respect to current llate-of-the-art monotonic architec- tures, even when using a tiny number parameters, and has the additional benefit of guaranteed robustness due to its known Lipschitz constant. For future directions of this line of research, plan to tackle the problems outlined in the limitation section, especially improving initialization of weight-normed networks.
7 REPRODUCIBILITY STATEMENT
All experiments with public datasets are reproducible with the code provided at https:// github.
_estseesss The experiments in Section 4.2 were made with data that is not publicly available. The code to reproduce those experiments can be found /HLT 2Track and the data will be made avail-
mnder/httss : qithub. com/n wwmmmiikkasnnnc able in later years at the discretion of the L.Hbb collaboration.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by NSF grant PHY-2019786 (The NSF Al Institute for Artificial Intelli- gence and Fundamental Interactions, http://aiii..rrg)
REFERENCES
Naveed Akhtar, Ajmal Mian, Navid Kardan, and Mubarak Shah. Advances in adversarial attacks and defenses in computer vision: A survey. IEEE Access, 9:153161-155196 2021.
Cem Anil, James Lucas, and Roger Grosse. Sorting out Lipschitz function approximation. Kama- lika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Confer- ence on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 291- 301. PMLR, 09-15 Jun 2019. URLhttpp wwt.httttpppoceeediiiie 99 1a199//annnl1 anmilnnil99 html
mww. wtw.s//97
Devansh Arpit, Victor Campos, and Yoshua Bengio. How to initialize your network? robust initial- ization for weightnorm resnets, 2019.
Charles Auguste, Sean Malory, and Ivan Smirnov. A better method to enforce monotonic constraints in regression and classification trees, 2020.
Jens Behrmann, Will Grathwohi, Ricky Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks, 2019.
Krisztian Buza. Feedback prediction for blogs. In Data analysis, machine learning and knowledge discovery, pp. 145-152. Springer, 2014.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Block neural autoregressive flow. In Uncertainty in artificial intelligence, pp. 1263-1273. PMLR, 2020.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL htLtttt/p///cciee
ics. uci edu/ml
John H. Gennari, Pat Langley, and Doug Fisher.
Models of incremental concept forma- tion. Artificial Intelligence. 40(1):11-61, 1989. ISSN 0004-3702. doi: https:/doi.org! 00.101600004-37220))))0000 atticleep11100043770390000
URL http:::/t///// wttp:/wwwwwweteieeeeeeeecc.o conet.comsseee com.science////////cccccccccccec
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J. Cree. Regularisation of neural net-
works by enforcing lipschitz continuity, 2020.
Akhil Gupta, Naman Shukla, Lavanya Marla, Arinbjör Kolbeinsson, and Kartik Yellepeddi. How to incorporate monotonicity in deep networks while preserving flexibility? URL https://
1909.106622
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Chin-We Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Learning, pp. 2078-2087.2 PMLR, 2018
Todd Huster, Cho- Yu Jason Chiang, and Ritu Chadha. Limitations of the lipschitz constant defense against adversarial examples, 2018
Todd Huster, Cho- Yu Jason Chiang. and Ritu Chadha. Limitations of the lipschitz constant as defense against adversarial examples. In ECML PKDD 2018 Workshops: Nemesis 2018, UrbReas 2018, SoGood 20/8, IWAISe 2018, and Green Data Mining 2018, Dublin, Ireland, September 10- 14, 2018, Proceedings 18, pp. 16-29. Springer, 2019.
Kaggle. Lending club loan data. 2015. URL
https:/ www httpsss/wwww.aggl.cccoo www.ka................kk....... http:/ew..oo/ataaeaaeeeee
wordsfforthewi se/lendineeeeeeeee nordiing-ce clubl
10

---

Published as a conference paper at ICLR 2023
Ouail Kitouni, Niklas Nolte, and Mike Williams. Robust and provably monotonic networks. arXiv preprint 12.00:21/ 2021.
Donald Knuth. Donald Knuth's homepage. https://www cs
kruthh Saruth/faq.hee html Accessed: 2022-09-28.
SM Angwin Larson and Lauren Kirchner. There's software used across the country to predict future criminals. and it's biased against blacks, 2016.
Xingchao Liu, Xing Han, Na Zhang, and Qiang Liu. Certified monotonic neural networks, 2020.
URL https:: /apps///axiiva org/ abs/ 10219
Mahdi Milani Fard, Kevin Canini, Andrew Cotter, Jan Pfeifer, and Maya Gupta. Fast and flexible monotonic functions with ensembles of lattices. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 29, Cur- ran Associates, Inc., 2016. URL https: /ttpt:::///rocceiiggg.... nwwrips....... www.siieeeiinnn..uiii.......... htt.:ppp.ccppppp/21111
Sw1116555----Pee---e pdf
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. arXiv e-prints, art. arXiv:1802.05957 February 2018.
A. Nguyen and M. Martínez. Mononet: Towards interpretable models by learning monotonic fea- tures. 9 2019. doi: 0015550/rxiv.1999.1.333 URL httpp:://aaxiv.oo//////
13611v
Joseph Sill.
Monotonic networks. In M. Jordan, M. Kearns, and S. Solla (eds.), Advances in Neural Information Processing Systems, volume 10. MIT Press, URL
1998.
newriisss.ngg................... cc pdf
Joseph Sill and Yaser Abu-Mostafa. Monotonicity hints. In M.C. Mozer, M. Jordan, and T. Petsche (eds.). Advances in Neural Information Processing Systems, volume 9. MIT Press, 1996.
hww.eeipsw......iip.ec....... wttpp//p....cpppppr////////.. pdd
c850371 fda 68966822222
Sahil Singla, Surbhi Singla, and Soheil Feizi. Improved deterministic 12 robustness cifar-10 and cifar-100 In International Conference on Learning Representations, 2021.
Aishwarya Sivaraman, Golnoosh Farnadi, Todd Millstein, and Guy Van den Broeck. Counterexample-guieee learning of monotonic neural networks. Advances Neural Informa- tion Processing Systems, 33: 1911--199944.. 2020.
Xiaosong Wang. Yifan Peng. Le Lu, Zhiyong Lu, M Bagheri, and R Summers. Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of com- mon thorax diseases. In IEEE CVPR, volume 2017. URL lttpttttt.... //wp://wwwwww..... kaggle..ge.............kkk.....k com/
datasets chest wrwys/aaaa..... sxrays/samlle sample.
Antoine Wehenkel and Gilles Louppe. Unconstrained monotonic neural networks. Advances in neural information processing systems, 32, 2019.
Seungil You, David Ding, Kevin Canini, Jan Pfcifer, and Maya Gupta. Deep lattice networks and partial monotonic functions, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
11

---

Published as a conference paper at ICLR 2023
A EXPRESSIVE POWER OF THE ARCHITECTURE
Robust architectures like Lipschitz constrained networks are often believed to be much less expres- sive than their unconstrained counterparts Huster et al. (2019). Here we show that our architecture is capable of (over) complex decision boundaries even on random labels in a setup simular to Zhang al. 2021
We show the finite sample expressiveness of the architecture in https: qi thub. com/ Okitouni/Lipschitze wew.ttt-nnttowoebeeeeeeeeee by fitting MNIST, CIFARIO. CIFARI00 with nor- mal and random labels to 100% training accuracy. We also train on CIFAR 100 with an additional "goodness" feature x E [0, 1] showcase the monotonicity aspect of the architecture. This dataset is referred to as CIFARIO 101 below. The synthetic monotonicity problem is currently implemented such that samples with values above a critical threshold in the goodness feature I > Werit are labeled O. An alternative implementation is to take label 0 with probability æ and keep the original label (or summarizes the setup used for training. We
assign a random one) with probability x. Table 2
use Adam with default hyper-parametert im all experiments.
Task
Width
Depth
LR
EPOCHS
Batchsize
Loss
MNIST
1024
3
10-6
10
ALL
CE(r = 256)
CIFARIO
1024
3
10-5
105
ALL
CE(T 256)
CIFAR100/10)
1024
3
10-5
105
ALL
CE(T 256)
Table 2: Training MNIST and CIFAR10/100
100% training accuracy with Lipschitz networks.
12


Reviews:
[{"review_id": "q9CZqqhN3OM", "reviewer": "Reviewer_rnj5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a novel method for learning monotonic neural networks, meaning that its output can be constrained and guaranteed to always increase (or decrease) with respect to any subset of its inputs. It accomplishes this by starting with a standard feedforward architecture whose weights and activation function are bounded so that the function has a Lipschitz constant of 1, and then they simply add (or subtract) a residual connection from the monotonic input directly to the network's output, guaranteeing that the partial derivative w.r.t. the monotonic input is always positive (or negative).\n\nThe authors show empirically that this architecture performs favorably compared to an unconstrained network, as well as compared to a variety of prior work on monotonic machine-learned function classes.", "summary_of_the_review": "A very nice, clear, idea for training monotonic functions that seems relatively easy to implement and very effective.", "strength_and_weaknesses": "Strengths:\n* Achieves guaranteed monotonicity with a relatively simple function class that is easy and efficient to implement and train.\n* Side benefit of a regularized function class, due to the Lip_1 constraint. But this may be a weakness when highly-flexible function classes are desirable without needing to resort to very deep models.\n\nWeaknesses:\n* Uses a somewhat unusual activation function, \"GroupSort\". I wonder if there are alternatives that can satisfy the conditions for monotonicity and are more similar to more commonly-used activation functions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "h55lsN6ux7", "reviewer": "Reviewer_8Geq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper constructs a novel type of architecture for monotonic neural networks by adding a single residual connection to an expressive (weight-constrained) Lipschitz network. The resulting network architecture is simple in implementation and theory foundations, robust, and highly expressive. The algorithm is compared to state-of-the-art methods across various benchmarks.", "summary_of_the_review": "The paper addresses an important question. The approach is novel and of broad interest. The proposed structure is better designed and outperforms pre-existing approaches in constructing monotonic networks.", "strength_and_weaknesses": "### Strengths\n\n- The paper relies on a simple (but seemingly novel) idea: by adding a residual connection to a weight-constrained Lipschitz network, one can obtain a network that inherits the expressiveness of the Lipschitz network, while the bound of the Lipschitz norm can be used to specify the strength of the residual connection to ensure monotonicity. This is simpler and is certainly more powerful than two existing ideas in the field (ensure positive weight across the nodes of a layer, or penalize the negative gradient in the samples).\n- The main ideas of the work were developed based on well-founded rationales; the technical details (such as choice of the norm, or how datasets are obtained and analyzed) are carefully designed. The paper is well-written with a sufficient literature review. Visual information is intuitive.\n- The problem studied in this paper is an important and of great interest to a broad audience in the field. Its impact on applied science is highlighted by the motivating example (real-time decision-making at high frequency at the Large Hadron Collider). As stated in the paper, the LHCb experiment is now using the proposed architecture for real-time data selection.\n- The design of the experiments is very careful and rigorous. The results showcase that the new approach could outperform state-of-the-art methods across various benchmarks previously studied in the field.\n\n### Weaknesses\n\n- None noted. However, it could be argued that the main technical part of the work (technical structure, expressiveness, and universal approximation of Lipschitz networks) are all pre-existing results in the field and that the technical and theoretical contributions of the work are minimal.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KOv6KzPX-W6", "reviewer": "Reviewer_tc6i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to build neural networks where an output is provably monotonic with respect to certain inputs. This is intended for applications where domain knowledge exists for such monotonic relations. For such applications, the benefits are better robustness and better interpretability.", "summary_of_the_review": "This paper may get rejected for breaking blind review policy.\n\nIf not: this paper is working on a meaningful problem, however the proposed technique has significant limitations. It may work for very small neural networks but is not applicable to general architectures.\n", "strength_and_weaknesses": "This paper contains the following statements that breaks the double blind review policy. This may warrant rejection without review and I'll leave the decision to the Area Chair.\n\n-- \"Our algorithm has been adopted to classify the decays of subatomic particles produced at the CERN Large Hadron Collider in the real-time data-processing system of the LHCb experiment, which was our original motivation for developing this novel architecture.\"\n\n-- \"The algorithm described here has, in fact, been implemented by a high-energy particle physics experiment at the European Center for Nuclear Research (CERN), and is actively being used to collect data at the Large Hadron Collider (LHC) in 2022, where highenergy proton-proton collisions occur at 40 MHz.\"\n\n-- \"Due to the simplicity and practicality of our method, the LHCb experiment is now using the proposed architecture for real-time data selection at a data rate of about 40 Tbit/s.\"\n\n-- \"Those data will be made available in later years at the discretion of the LHCb collaboration.\"\n\n\nHaving said the above, I'll provide a brief review of the proposed technique anyway.\n\nThe motivation of this work is meaningful. The monotonicity guarantee is correct. \n\nHowever, whether the resulting neural networks are still expressive is very much in question.\n\nThe proposal is to first build a neural network g(x) that has a Lipschitz constant of lamda with respect to L1 norm, and then the overall model is\n f(x) = g(x) + lamda * sum_over_mono_set (x_i)\nwhere x_i's are the subset of input features that we have the domain knowledge about monotonicity. This can easily be extended to when f(x) and g(x) are vectors, and we can have a different set of x_i's for each dimension in f(x), depending on the domain knowledge.\n\nThe main problem is in building g(x). For a linear layer, the L1 norm of the weight matrix is the max over sum of absolute values across each row. For g(x) which may have multiple layers, its Lipschitz bound from linear layers is the product of the said max-row-sum values across layers. This is very restrictive for larger networks and much more restrictive than past works that work with L2 Lipschitz bounds.\nOn pages 4-5, the authors argue that using GroupSort instead of ReLU would help. That is not true, and choosing non-linearities cannot solve the problem of restrictive linear layers.\nIf, for a particular application, it is sufficient to have g(x) as one linear layer plus some non-linearity, the proposed method should work fine. If two layers, this becomes questionable. For deeper networks, hidden dimensions need to be very small for this method to be viable. For general neural network architectures, I see little hope with the proposal. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster