Monotonicity as a requirement and as a regularizer: efficient methods and applications

1 INTRODUCTION . Highly expressive model classes such as artificial neural networks have achieved impressive prediction performance across a broad range of supervised learning tasks and domains ( Krizhevsky et al. , 2012 ; Graves & Jaitly , 2014 ; Bahdanau et al. , 2014 ) . However , finding predictors attaining low risk on unseen data is often not enough to enable the use of such models in practice . In fact , practical applications usually have more requirements other than prediction accuracy . Hence , devising approaches that search risk minimizers satisfying practical needs led to several research threads seeking to enable the use of neural networks in real-life scenarios . Examples of such requirements include : ( 1 ) Robustness , where low risk is expected even if the model is evaluated under distribution shifts , ( 2 ) Fairness , where the performance of the model is expected to not significantly change across different sub-populations of the data , and ( 3 ) Explainability/Interpretability , where models are expected to indicate how the features of the data affect their predictions . In addition to the requirements mentioned above , a property commonly expected in trained models in certain applications is monotonicity with respect to some subset of the input dimensions , i.e. , an increase ( or decrease ) along some particular dimensions strictly imply the function value will not decrease ( or will not increase ) , provided that all other dimensions are kept fixed . As a result , the behavior of monotonic models will be more aligned with the properties that the data under consideration is believed to satisfy . For example , in the models used to accept/reject job applications , we expect acceptance scores to be monotonically non-decreasing with respect to features such as past years of experience of a candidate . Thus , given two applicants with exactly the same features except their years of experience , the more experienced candidate should be assigned an equal or higher chance of getting accepted . For applications where monotonicity is expected , having a predictor failing to satisfy this requirement would damage the user ’ s confidence . As such , different strategies have been devised in order to enable training monotonic predictors . These approaches can be divided into two main categories . The first one is monotonicity by construction , where the focus lies on defining a model class that guarantees monotonicity in all of its elements Bakst et al . ( 2021 ) ; Wehenkel & Louppe ( 2019 ) ; Nguyen & Martı́nez ( 2019 ) ; You et al . ( 2017 ) ; Garcia & Gupta ( 2009 ) ; Archer & Wang ( 1993 ) . However , this approach can not be used with general architectures in the case of neural networks . Additionally , the model class might be so constrained that it might affect the prediction performance . Alternatively , a second approach is based on encouraging monotonicity during training , i.e. , searching for monotonic candidates within a general class of models ( Liu et al. , 2020 ; Sivaraman et al. , 2020 ; Gupta et al. , 2019 ) . Such a group of methods is more generally applicable and can be used , for instance , with any neural network architecture . However , they are not guaranteed to yield monotonic predictors unless extra verification/certification steps are performed , which can be computationally very expensive . In addition to being a requirement as in the examples discussed above , monotonicity has also been observed to be a useful feature in certain cases . For example , it can define an effective inductive bias and improve generalization in cases where prior knowledge indicates the data generation process satisfies such a property ( Dugas et al. , 2001 ) . In such cases , however , it is not necessary to satisfy the property everywhere , since it is enforced simply as a desirable feature of trained models rather than a design specification . In this work , we tackle the problem of performing empirical risk minimization over rich classes of models such as neural networks , while simultaneously searching for monotonic predictors within the set of risk minimizing solutions . In summary , our contributions are two-fold : 1. monotonicity as a requirement : We show that previous methods can only satisfy monotonicity either near the training data or near the boundaries of the input domain . Then , we propose an efficient algorithm that tackles this problem . In short , we apply Mixup ( Zhang et al. , 2018 ) between the data and random points to populate the input space , which is shown to enforce monotonicity in a larger volume relative to previous methods in literature . To the best of our knowledge , this is the first work that studies the effect of the sample points used in calculating the monotonicity constraints . 2. monotonicity as a regularizer : We define a new notion of monotonicity which is shown to be useful when enforced for applications such as object recognition or generative modeling of images . In these cases , it is not necessarily required to enforce the property everywhere and , as such , the constraints that focus only on the actual data points are discussed ( i.e. , mixup is not needed ) . Models satisfying the property are compared with standard predictors across three applications , where it is shown the property is beneficial without compromising the original performance . 2 BACKGROUND AND RELATED WORK . We start by defining the notion of partial monotonicity used throughout the paper . Consider the standard supervised learning setting where data instances are observed in pairs x , y ∼ X ×Y , where X ⊂ Rd and Y ⊂ R correspond to the input and output spaces , respectively . Further , consider the function f : X 7→ Y , and let M indicate some subset of the input dimensions , i.e. , M ⊂ { 1 , ... d } , such that x = concat ( xM , xM̄ ) , where M̄ = { 1 , ... , d } \M . We further overload the notation of function calls to f such that f ( x ) = f ( xM , xM̄ ) . Definition 1 Partially monotonic functions relative to M : We say f is monotonically nondecreasing relative to M 1 , denoted fM , if f ( xM , xM̄ ) ≤ f ( x′M , xM̄ ) , ∀ xM ≤ x′M , ∀ xM̄ , where the comparison xM ≤ x′M is performed for every dimension . This definition covers functions that do not decrease in value given increasing changes along a subset of the input dimensions , provided that all other dimensions are kept unchanged . Several approaches were introduced for defining model classes that have such a property . The simplest approach restricts the weights of the network to be non-negative ( Archer & Wang , 1993 ) . However , doing so affects the prediction performance . Another approach corresponds to using lattice regression models proposed by Garcia & Gupta ( 2009 ) ; You et al . ( 2017 ) . In this case , models are given by interpolations in a grid defined by training data . Such a class of models can be made monotonic via the choice of the interpolation strategy and recently introduced variations ( Bakst et al. , 2021 ) scale efficiently with the dimension of the input space , but downstream applications might still require different classes of models to satisfy this type of property . For neural networks , approaches such as ( Nguyen & Martı́nez , 2019 ) reparameterize fully connected layers such that the gradients with respect to parameters can only be non-negative . Wehenkel & Louppe ( 2019 ) , on the other hand , 1Monotonically non-increasing f can be defined analogously . consider the class of predictors H : X 7→ Y of the form H ( x ) = ∫ x 0 h ( t ) dt + H ( 0 ) , where h ( t ) is a strictly positive mapping parameterized by a neural network . While such approaches guarantee monotonicity by design , they can be too restrictive or give overly complicated learning procedures . For example , the approach in ( Wehenkel & Louppe , 2019 ) requires backpropagating through the integral . An alternative approach is based on searching over general classes of models while assigning higher importance to predictors observed to be monotonic . Similar to the case of adversarial training ( Goodfellow et al. , 2014 ) , Sivaraman et al . ( 2020 ) proposed an approach to find counterexamples , i.e. , pairs of points where the monotonicity constraint is violated , which are included in the training data to enforce monotonicity conditions in the next iterations of the model . However , this approach only supports fully-connected ReLU networks . Moreover , the procedure for finding the counterexamples is costly . Alternatively , Liu et al . ( 2020 ) ; Gupta et al . ( 2019 ) introduced point-wise regularization penalties for enforcing monotonicity , where the penalties are estimated via sampling . While Liu et al . ( 2020 ) use uniform random draws , Gupta et al . ( 2019 ) apply the regularization penalty over the training instances . Both approaches have shortcomings that we seek to address . 3 MONOTONICITY AS A REQUIREMENT . Given the standard supervised learning setting where ` : Y2 7→ R+ is a loss function indicating the goodness of the predictions relative to ground truth targets , the goal is to find a predictor h ∈ H such that its expected loss – or the so-called risk – over the input space is minimized . Such an approach yields the empirical risk minimization framework once a finite sample is used to estimate the risk . However , given the extra monotonicity requirement , we consider an augmented framework where such a property is further enforced . We seek the optimal monotonic predictors relative to M , h∗M : h∗M ∈ arg minh∈H Ex , y∼X×Y [ ` ( h ( x ) , y ) ] + γΩ ( h , M ) , ( 1 ) where γ is a hyperparameter weighing the importance of the penalty Ω ( h , M ) which , in turn , is a measure of how monotonic the predictor h is relative to the dimensions indicated by M . Ω ( h , M ) can be defined by the following gradient penalty ( Gupta et al. , 2019 ; Liu et al. , 2020 ) : Ω ( h , M ) = Ex∼D [ ∑ i∈M max ( 0 , −∂h ( x ) ∂xi ) 2 ] , ( 2 ) where ∂h ( x ) ∂xi indicates the gradients of h relative to the input dimensions i ∈ M , which are constrained to be non-negative , rendering h monotonically non-decreasing relative to M . At this point , the only missing ingredient to define algorithms to estimate h∗M is how to define the distribution D over which the expectation in Eq . 2 is computed , discussed in the following sections . 3.1 CHOOSING DISTRIBUTIONS OVER WHICH TO COMPUTE THE MONOTONICITY PENALTY . In the following , we present and discuss two past choices for D : 1 ) Define D as the empirical distribution of the training sample : In ( Gupta et al. , 2019 ) , given a training dataset of size N , in addition to using the observed data to estimate the risk , the same data is used to compute the monotonicity penalty so that : Ωtrain ( h , M ) = 1 N N∑ k=1 ∑ i∈M max ( 0 , −∂h ( x k ) ∂xki ) 2 , where xk indicates the k-th instance within the training sample . While this choice seems natural and can be easily implemented , it only enforces monotonicity in the region where the training samples lie , which can be problematic . For example , in case of covariate-shift , the test data might lie in parts of the space different from that of the training data so monotonicity can not be guaranteed . We thus argue that one needs to enforce the monotonicity property in a region larger than what is defined by the training data . In Appendix B , we conduct an evaluation under domain shift and show the issue to become more and more relevant with the increase in the dimension d of the input space X . 2 ) Define D = Uniform ( X ) : In ( Liu et al. , 2020 ) , a simple strategy is defined so that Ω is computed over the random points drawn uniformly across the entire input space X ; i.e . : Ωrandom ( h , M ) = Ex∼Uniform ( X ) [ ∑ i∈M max ( 0 , −∂h ( x ) ∂xi ) 2 ] . Despite its simplicity and ease of use , this approach has some flaws . In high-dimensional spaces , random draws from any distribution of bounded variance will likely lie in the boundaries of the space , hence far from the regions where data actually lie . Moreover , it is commonly observed that naturally occurring high-dimensional data is structured in lower-dimensional manifolds ( c.f . ( Fefferman et al. , 2016 ) for an in-depth discussion on the manifold hypothesis ) . It is thus likely that random draws from the uniform distribution will lie nowhere near regions of space where training/testing data will be observed . We further illustrate the issue with examples in Appendix A , which can be summarized as follows : consider the cases of uniform distributions over the unit n-sphere . In such a case , the probability of a random draw lying closer to the sphere ’ s surface than to its center is P ( ||x||2 > 12 ) = 2n−1 2n , as given by the volume ratio of the two regions of interest . Note that P ( ||x||2 > 12 ) → 1 as n → ∞ , which suggests the approach in ( Liu et al. , 2020 ) will only enforce monotonicity at the boundaries of the space . In summary , the previous approaches are either too focused on enforcing monotonicity where the training data lie , or too loose such that the monotonicity property is uniformly enforced across a large space , and the actual data manifold may be neglected . We thus propose an alternative approach where we can have some control over the volume of the input space where the monotonicity property will be enforced . Our approach uses the idea of data mixup ( Zhang et al. , 2018 ; Verma et al. , 2019 ; Chuang & Mroueh , 2021 ) , where auxiliary data is created via interpolations of pairs of data points , to populate areas of the space that are otherwise disregarded . Mixup was introduced by Zhang et al . ( 2018 ) with the goal of training classifiers with smooth outputs across trajectories in the input space from instances of different classes . Given a pair of data points ( x′ , y′ ) , ( x′′ , y′′ ) , the method augments the training data using interpolations given by ( λx′+ ( 1−λ ) x′′ , λy′+ ( 1−λ ) y′′ ) , where λ ∼ Uniform ( [ 0 , 1 ] ) . We propose using mixup to generate points where the monotonicity penalty Ω can be computed and highlight the following motivations for doing so : ( 1 ) Interpolation of data points more densely populates the convex hull of the training data . ( 2 ) Extrapolation cases where mixup is performed between data points and instances obtained at random results in points that lie anywhere between the data manifold and the boundaries of the space . We thus claim that performing mixup enables the computation of Ω on parts of the space that are disregarded if one focus only on either observed data or random draws from uninformed choices of distributions such as the uniform .

Summary: The paper has two main contributions:  1. It takes a known monotonicity regularizer and trains with it using a new distribution that is roughly a mixing of uniform and the training distribution. It shows empirically that using this method increases the "size" of the input region in which the model is monotonic.   2. It defines several regularizers that are meant to encourage "monotonic behavior" of the model's output w.r.t the outputs of an intermediate layer or latent variable. It provides experiments that show how using these regularizers does not hurt the model's performance and two applications of the added structure: 1) detecting noisey/adverserial examples and 2) more controllable generation in generative models.