Monotonicity as a requirement and as a regularizer: efficient methods and applications
Authors: Joao Monteiro
Mohamed Osama Ahmed
Hossein Hajimirsadeghi
Greg Mori
Venue: ICLR 2022 Submitted (2022)
Keywords: Monotonic neural networks
Gradient penalties
Structural risk minimization

Abstract: We study the setting where risk minimization is performed over general classes of models and consider two cases where monotonicity is treated as either a requirement to be satisfied everywhere or a useful property. We specifically consider cases where point-wise gradient penalties are used alongside the empirical risk during training. In our first contribution, we show that different choices of penalties define the regions of the input space where the property is observed. As such, previous methods result in models that are monotonic only in a small volume of the input space. We thus propose an approach that uses mixtures of training instances and random points to populate the space and enforce the penalty in a much larger region. As a second contribution, we introduce the notion of monotonicity as a regularization bias for convolutional models. In this case, we consider applications, such as image classification and generative modeling, where monotonicity is not a hard constraint but can help improve some aspects of the model. Namely, we show that using group monotonicity can be beneficial in several applications such as: (1) defining strategies to detect anomalous data, (2) allowing for controllable data generation, and (3) generating explanations for predictions. Our proposed approaches do not introduce relevant computational overhead while leading to efficient procedures that provide extra benefits over baseline models.

where models Under review as a conference paper at ICLR 2022
MONOTONICITY AS A REQUIREMENT AND AS A REGU- LARIZER: EFFICIENT METHODS AND APPLICATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
We study the setting where risk minimization is performed over general classes of models and consider two cases where monotonicity is treated as either a require- ment to be satisfied everywhere or a useful property. We specifically consider cases where point-wise gradient penalties are used alongside the empirical risk during training. our first contribution, we show that different choices of penal- ties define the regions of the input space where the property is observed. As such, previous methods result in models that are monotonic only in a small volume of the input space. We thus propose approach that uses mixtures of training instances and random points to populate the space and enforce the penalty in a much larger region. As a second contribution, we introduce the notion of mono- tonicity as regularization bias for convolutional models. In this case, we consider applications, such image classification and generative modeling, where mono- tonicity is not a hard constraint but can help improve some aspects of the model. Namely, we show that using group monotonicity can be beneficial in several ap- plications such as: (1) defining strategies to detect anomalous data, (2) allowing for controllable data generation, and (3) generating explanations for predictions. Our proposed approaches do not introduce relevant computational overhead while leading efficient procedures that provide extra benefits over baseline models.
INTRODUCTION
Highly expressive model classes such artificial neural networks have achieved impressive pre- diction performance across a broad range of supervised learning tasks and domains (Krizhevsky al. 2012 Graves & Jaitly 2014 Bahdanau et al., 2014). However, finding predictors attaining low risk on unseen data is often not enough to enable the use of such models in practice. In fact, practical applications usually have more requirements other than prediction accuracy. Hence, devis- ing approaches that search risk minimizers satisfying practical needs led to several research threads seeking to enable the use of neural networks in real-life scenarios. Examples of such requirements include: (1) Robustness, where low risk is expected even if the model is evaluated under distribu- tion shifts, (2) Fairness, where the performance of the model is expected to not significantly change across different cub-populations of the data, and (3) Explainabiliyy/tteerretbb are expected to indicate how the features of the data affect their predictions.
In addition to the requirements mentioned above, property commonly expected in trained models in certain applications is monotonicity with respect to some subset of the input dimensions, i.e., an increase (or decrease) along some particular dimensions strictly imply the function value will decrease (or will not increase), provided that all other dimensions are kept fixed. As result, the behavior of monotonic models will be more aligned with the properties that the data under consid- eration is believed to satisfy. For example, in the models used to accept/reject job applications, we expect acceptance scores to be monotonically non-decreasing with respect to features such as past years of experience of a candidate. Thus, given two applicants with exactly the same features except their years of experience, the more experienced candidate should be assigned an equal or higher chance of getting accepted. For applications where monotonicity is expected, having a predictor failing to satisfy this requirement would damage the user's confidence, As such, different strategies have been devised in order to enable training monotonic predictors. These approaches vided into two main categories. The first one is monotonicity by construction, where the focus lies on defining model class that guarantees monotonicity in all of its elements Bakst al. 2021
be di-

---

Under review as a conference paper at ICLR 2022
Wehenkel & Louppe (2019); Nguyen & Martinez 2 You et al. (2017); Garcia & Gupta 2209) Archer & Wang (1993). However, this approach can not be used with general architectures in the case of neural networks. Additionally, the model class might be so constrained that it might affect the prediction performance. Alternatively, a second approach is based on encouraging monotonicity during training, i.e., searching for monotonic candidates within a general class of models (Liu al.) 2020 Sivaraman et al., 2020; [uupta et al., 2019). Such a group of methods is more generally ap- plicable and can be used, for instance, with any neural network architecture. However, they are not guaranteed to yield monotonic predictors unless extra perffcattio//cettffcctton steps are performed, which can be computationally very expensive. In addition to being a requirement as in the exam- ples discussed above, monotonicity has also been observed to be a useful feature in certain cases. For example, it can define an effective inductive bias and improve generalization in cases where prior knowledge indicates the data generation process satisfies such a property (Dugas et al. 200T). In such cases, however. it is not necessary to satisfy the property everywhere, since it is enforced simply desirable feature of trained models rather than design specification.
In this work, we tackle the problem of performing empirical risk minimization over rich classes of models such as neural networks, while simultaneously searching for monotonic predictors within the set of risk minimizing solutions. In summary, our contributions are two-fold:
1. monotonicity as a requirement: We show that previous methods can only satisfy monotonicity either near the training data or near the boundaries of the input domain. Then, we propose an efficient algorithm that tackles this problem. In short, apply Mixup (Zhang et al.. (018)) between the data and random points to populate the input space, which is shown to enforce monotonicity in a larger volume relative to previous methods in literature. To the best of our knowledge, this is the first work that studies the effect of the sample points used in calculating the monotonicity constraints.
monotonicity as a regularizer: We define a new notion of monotonicity which is shown to be useful when enforced for applications such as object recognition or generative modeling of images. In these cases, not necessarily required to enforce the property everywhere and, as such, the constraints that focus only on the actual data points are discussed (i.e., mixup is not needed). Models satisfying the property are compared with standard predictors across three applications, where is shown the property is beneficial without compromising the original performance.
BACKGROUND AND RELATED WORK
We start by defining the notion of partial monotonicity used throughout the paper. Consider the standard supervised learning setting where data instances are observed in pairs x, y X' x V, where X' C Rd and Y c TR correspond to the input and output spaces, respectively. Further, consider the function f X + y and let M indicate some subset of the input dimensions, i.c., A C {1, (L... such that x = concat(æ M x ). where M = {l,.... d} \ M. We further overload the notation of function calls to f such that f fIr) [xM
Definition 1 Partially monotonic functions relative to M: We say f is monotonically non- decreasing relative to M denoted fMr if f( 2x)) S f(''M T s2), V aM I I'r. V zzzz where the comparison MM S Jar is performed for every dimension.
This definition covers functions that do not decrease in value given increasing changes along a subset of the imput dimensions, provided that all other dimensions are kept unchanged. Several approaches were introduced for defining model classes that have such a property. The simplest approach re- stricts the weights of the network to be non-negative (Archer & Wang] 1993). However, doing so affects the prediction performance. Another approach corresponds to using lattice regression models proposed by Garria & Gupta| (2009); You et al. (0177). In this case, models are given by interpolations in a grid defined by training data. Such a class of models can be made monotonic via the choice of the interpolation strategy and recently introduced variations (Bakst et al., 2021 scale efficiently with the dimension of the input space, but downstream applications might still require different classes of models to satisfy this type of property. For neural networks, approaches such as (Nguyen& & Martínez, 2019) reparameterize fully connected layers such that the gradients with respect to parameters can only be non- negative.... Wehenkel & Louppe (019). on the other hand,
Monotonccally non-increasing f can be defined analogously.
2

---

Under review as a conference paper at ICLR 2022
consider the class of predictors H + y of the form (x) - h(t)dt + HO), where h(t) a strictly positive mapping parameterized by a neural network. While such approaches guarantee monotonicity by design, they can be too restrictive or give overly complicated learning procedures. For example, the approach in (Wehenkel & Louppe 2019) requires backpropagating through the integral. An alternative approach based on searching over general classes of models while assign- ing higher importance to predictors observed to be monotonic. Similar to the case of adversarial training (Goodfellow al. 2014), Sivaraman et 2020) proposed an approach find counterex- amples, i.e., pairs of points where the monotonicity constraint is violated, which are included in the training data to enforce monotonicity conditions the next iterations of the model. However, this approach only supports fully-connected ReLU networks. Moreover, the procedure for finding the counterexamples is costly. Alternatively, [iu al. (2020): Cupta et al. (2019) introduced point-wise regularization penalties for enforcing monotonicity, where the penalties are estimated via sampling. While Liu et al. (2020) use uniform random draws, Gupta et al. (2019) apply the regularization penalty over the training instances. Both approaches have shortcomings that we seek to address.
3 MONOTONICITY AS A REQUIREMENT
Given the standard supervised learning setting where é : : + IR is a loss function indicating the goodness the predictions relative to ground truth targets, the goal is to find predictor E such that its expected loss or the so-called risk over the input space is minimized. Such approach yields the empirical risk minimization framework once a finite sample is used to estimate the risk. However, given the extra monotonicity requirement, such a property is further enforced. We seek the optimal monotonic predictors relative to M, hy:
consider augmented framework where
hy E arg min Eay
(((()), (x.())))(((((()))
v)] + -0h, M),
where y is a hyperparameter weighing the importance of the penalty (2(h, M) which, in turn, is measure how monotonic the predictor h is relative to the dimensions indicated by M. 2(h, M) can be defined by the following gradient penalty (Gupta et al. 2019 Liu et al. 2020):
ah(u)) 2 ( Oz, )
1h, M) ELOD
Σ max 0, LieM
where 0h(z) 8z. indicates the gradients of h relative the input dimensions i E M, which are strained to be non-negative, rendering / monotonically non-decreasing relative to M. At this point, the only missing ingredient to define algorithms to estimate hi is how to define the distribution D over which the expectation in Eq. 2is is computed, discussed in the following sections.
3.1 CHOOSING DISTRIBUTIONS OVER WHICH TO COMPUTE THE MONOTONICITY PENALTY
In the following, we present and discuss two past choices for D:
1) Define D as the empirical distribution of the training sample: In (Gupta et al. 2019), given training dataset size N, in addition to using the observed data to estimate the risk, the same data used compute the monotonicity penalty that:
ah(x*)
2
Derain (h, M)
max 0,
N
dark
k=1iEM
where x indicates the k-th instance within the training sample. While this choice seems natural and can be easily implemented. it only enforces monotonicity in the region where the training samples lie, which can be problematic. For example, in case of covariate-shift, the test data might lie in parts of the space different from that of the training data so monotonicity cannot be guaranteed. We thus argue that needs to enforce the monotonicity property in region larger than what is defined by the training data. In Appendix B we conduct an evaluation under domain shift and show the issue to become more and more relevant with the increase in the dimension d of the input space X.
3

---

Under review as a conference paper at ICLR 2022
2) Define D = Uniform(X'): In (Liu et al. 2020 simple strategy is defined so that Q is computed over the random points drawn uniformly across the entire input space X;; i.e.:
ah(x))
Drandom (h, M) = E Uniffomm((2))
Σ max 0,
az;
REM
Despite its simplicity and ease of use, this approach has some flaws. In high-dimensional spaces, random draws from any distribution of bounded variance will likely lie in the boundaries of the space, hence far from the regions where data actually lic. Moreover, is commonly observed that naturally occurring high-dimensionn man et al. 2016) for an in-depth discussion on the manifold hypothesis). is thus likely that random draws from the uniform distribution will lie nowhere near regions of space where training/testing data will be observed. We further illustrate the issue with examples in Appendix [A] which can be summarized as follows: consider the cases of uniform distributions over the unit ni-phhee. In such a case, the probability of a random draw lying closer to the sphere's surface than to its center is P([|=112 2 > 5) = 2 as given by the volume ratio of the two regions of interest. Note that P(||=12 2) > 1 as n + oo, which suggests the approach in (Liu et al. 2020) will only enforce monotonieity the boundaries of the space.
data is structured in lower-dimensional manifolds (c.f. (Feffer-
In summary, the previous approaches are either too focused on enforcing monotonicity where the training data lie, too loose such that the monotonicity property is uniformly enforced across large space, and the actual data manifold may be neglected. We thus propose an alternative approach where we can have some control over the volume of the input space where the monotonicity property will be enforced. Our approach uses the idea of data mixup (Zhang et al.) 2018 Verma et al. 2019) Chuang & Mroueh 2021), where auxiliary data is created via interpolations of pairs of data points, to populate areas of the space that are otherwise disregarded. Mixup was introduced by ZZhang jt al. (2018) with the goal of training classifiers with smooth outputs across trajectories in the input space from instances of different classes. Given pair of data points (x', v"), (x" v"), the method augments the training data using interpolations given by (Az' + +1- A)aa", Ay' + (1- A)y"). where A Uniform [0, 1]). We propose using mixup to generate points where the monotonicity penalty Q can be computed and highlight the following motivations for doing so: (1) Interpolation of data points more densely populates the convex hull of the training data. (2) Extrapolation cases where mixup is performed between data points and instances obtained at random results in points that lie anywhere between the data manifold and the boundaries of the space. We thus claim that performing mixup enables the computation of Ω on parts of the space that are disregarded if one focus only on either observed data or random draws from uninformed choices of distributions such as the uniform.
3.2 EVALUATION
In order to evaluate the effect of different choices of Ω we report results on three commonly used datasets covering classification and regression settings with input spaces of different dimensions. Namely, we report results for the following datasets: Compas, Loan Lending Club, and Blog Feed- buck. Models are implemented using the same architecture as in (Liu et al., 2020). Further details on the data, models, and training settings can be found in Appendix [ For all evaluation cases, we sider the baseline where training is carried out without any monotonicity enforcing penalty. For the regularized cases, the different approaches used for computing Ω are as follows: (1) Prandom (Liu) ett al. 2020) which uses random points drawn from (niform(A). In this case, the sample observed each training iteration set to size of 1024 throughout all experiments. (2) Strain (Gupta al. 2019) which uses the actual data observed at each training iteration; i.e., the observed mini-batch itself's is used compute Q. And (3) Dmisup (ours), in which case the penalty is computed on points generated by mixing-up points from the training data and random points. In details, for each mini- batch of size N I, we augment with complementary random data and obtain final mini-batch of size 2N. Out of the 2N(2N
2 possible pairs of points, we take a random subsample of 1024 pairs to compute mixtures of instances. In this case, we use À ~ Uniform((O,1) and À is independently drawn for each pair of points.
Results are reported in terms of both prediction performance and level of monotonicity. The latter is assessed via the fraction of points within a sample where the monotonicity constraint is vio- ΣN
lated; i.e., given a set of N data points, we compute: p Mminees
Whiel <0]
0x!
such that p 0
4

---

Under review as a conference paper at ICLR 2022
Non-mon.
9 Drandows
Amiznp
COMPAS
Validation accuracy
09.1%=022%
68.5%=0.19%
68.59:10.1%
68.4%
Test accuracy
68.5%
00.2%
60.0%
68.3%
Prandom
$5.45%%-
0.01%-10.019 %
6,41%:2.5495
0.00%:20.005.
Ptrain
92.98%
2.08%-+2.21%
0.00%
0.00%-+0.00%
Prest
92.84%
2..5%%
0.00%:000000
0.00%:0.000%
Loan Lending Club
Validation RMSE
0,203:000000
0,2231:.0022
0.222=0.002
0,235=0.001
Test RMSE
0.221=0.001
0.230+:.001
0,229=0.002
0.228=0.001
Prondom
09.11%==1.7044
0.00%-110000%
14.47%:17.5555 555%
0.00% 0.00%
Ptrain
100.00%:++0.005
7.23%%
0.01 0.01%%
0.00% 0.0%
Prost
100.009%++000099
6.94% 7.43%
0.04% 0.03%%
1.000+0.005$
Blog Jeedback
Validtion RMSE
0.174:00.000
0.175:10.001
0.177=0.000
0.008=0.000
Test RMSE
0.139:10:011
0.139:10.001
0.142=0.001
0.143=0.001
Prondom
76.17%:::12.37%
0.05%-1.0.08%
4.86%:2..19%
0.00%=000015
Parain
78.67%:25.288
78.59%=2.3778
0.01%::0.01%
0.01%=0.011
Prest 76, 16.29%%
78.99%
0.02%::0.02%
0.02%
Table 1: Evaluation results in terms of 95% confidence intervals resulting from 20 independent training runs. Results correspond to the checkpoint that obtained the best prediction performance on validation data throughout training. The lower the values of p the better.
corresponds to monotonic models over the considered points. Moreover, in order to quantify the degree of monotonicity in different parts of the space, compute p for 3 different sets of points: (1) Prandom, computed on a sample drawn according to Uniform (A)). We used a sample of 10,000 points throughout the experiments. (2) Ptrain, computed the training data. And (3) Ptest: com- puted on the test data. Results are summarized in Table [ in terms of both prediction performance along with the metric indicating the degree of monotonicity of the predictor for each regulariza- tion strategy. Prediction performance is measured in terms of accuracy for classification tasks, and RMSE for the case of regression. Monotonicity, on the other hand, is quantified via the fraction p of points within a sample where the monotonicity constraint is violated. Results reported in the tables represent 95% confidence intervals corresponding to multiple independent training runs. Across all datasets, the different penalties do not result in significant variations in the performance of the final predictors. This indicates that the class of predictors corresponding to the subset of H that is mono- tonic relative to M, denoted H.M has enough capacity so to be able to match the performance of the best canditates within H. In terms of monotonicity, we observe a clear pattern leading to the following intuition: monotonicity is achieved in the regions where it is enforced. This is evidenced by the observation that Prandom is consistently lower for Prandom relative to Perain and Pymiruu while, on the other hand, Ptrain and Pteat are consistently lower for Derain and 9mizuu compared to Prandom- A comparison between Ptrain and Pmizup shows what we anticipated: enforcing mono- tonicity in points resulting from mixup yields predictors that as monotonic as those given by the use of Ptrain in actual data, but significantly better at the boundaries of X. Finally, the results demonstrate that our proposed approach Qmizup achieves the best results in terms of monotonic- ity for all the sets of points that we considered. Moreover, our approach introduces no significant computation overhead. Algorithm T in Appendix C details on how to compute Dmisup.
4 MONOTONICITY AS A REGULARIZER
In Section we presented an efficient approach to enforce monotonicity when it is requirement. We now consider a different perspective and show that adding monotonicity constraints during train- ing can yield extra benefits to trained models. In these cases monotonicity is not a requirement, and hence it is not necessary for it to be satisfied everywhere. As such, the penalties we discuss from now on are computed considering only data points, and no random draws are utilized. In the follow- ing sections, we introduce notions of monotonicity that will be enforced in our models, and discuss advantages of using monotonicity for different applications such as for the detection of anomalous data and controllable generative modelling. In Appendix [ we consider a further application for cases where one's s interest is to obtain explanations from observed predictions.
5

---

Under review as a conference paper at ICLR 2022
4.1 GROUP MONOTONICITY
First, we consider the case of K-way classifiers realized through convolutional neural networks. In this case, data examples correspond to pairs z. X' x V, but y (1, ..., K). K N. Models parameterize data-conditional categorical distribution over V/ i.e., for a given model h, h(z)y will yield likelihoods for each class indexed V. Under this setting, we introduce the notion of Group Monotonicity: we aim to find the models h such that the outputs corresponding to each class satisfy a monotonic relationship with a specific subset of high-level representations given by some inner convolutional layer. Let the outputs of a specific layer within convolutional model be represented by Ow, w E [1, 21,2,3,... ..., WI, where W indicates the width of the chosen layer given by its number of output feature maps. For simplicity of exposition,
consider the rather common of convolutional layers where each feature map Ow is 2-dimensional. We then partition such a set of representations into disjoint subsets, slices, of uniform sizes. Each subset is then paired with particular output or class, and hence denoted by Sk, k E V An illustration is provided in Figure [ where generic convolutional model has the outputs of specific layer partitioned into slices Sk which are paired with the corresponding output units.
Definition Group monotonic convolutional classifiers: We say h is group monotonic for input and its corresponding class label y if h((x), is partially monotonic relative to all elements in Sy-
We highlight that this case, unlike the discussion in Section monotonicity is not an applica- tion requirement, and it does not need to be satisfied everywhere. Intuitively, our goal is to "reserve" groups of high-level features to be monotonic with respect to different classes. Such an structure can add extra benefits to models, e.g., more accurate anomaly detection. For training, we perform mono- tonic risk minimization as described in Eq. | and the risk is given by the negative log-likelihood over training points. Moreover, we design a penalty that focuses only on observed data points during training and penalizes the slices of the Jacobian corresponding to given class, i.c., a cross- entropy criterion enforces larger gradients on the class slice. order to formally introduce such penalty, denoted by S2 Agroupp we first define the total gradient Ok, k E V of slice S₂ as follows: 0h(x)), where the inner sum accounts for spatial dimensions of aw. Given the set of total gradients, a batch of size m, and inverse temperature Sgroup will be:
(2) = Eawes ES, En, Jaw.s.y'
1
m
Ω Ggroup
log
(3)
m
K k==
4.2 EVALUATION
4.2.1
ASSESSING PERFORMANCE OF GROUP MONOTONIC CLASSIFIERS
We start our evaluation by verifying whether the group monotonicity property can be effectively enforced into classifiers trained on standard object recognition benchmarks. In order to do so, we verify the performance of the total activation classifier, as defined by: arg maxkey (x),, where Tk indicates the total activation on slice Sy: (z) = E..res ES, Σx, Aw,i,j (z). A good prediction per- formance of such a classifier serves as evidence that the group monotonicity property is satisfied by the model over the test data under consideration since it indicates the slice relative to the underlying class of test instances has the highest total activation. We thus run evaluations for both CIFAR- 10 and ImageNet, and classifiers in each case correspond to WideResNets (Zagoruyko & Komodakis 2016) and ResNet-50 50 (He et al. 2016). respectively. Training details are presented in Appendix D Results reported in Table and compared based on the top-1 prediction accuracy mea- sured on the test data. We use standard clas- sifiers as the baselines where no monotonic- ity penalty applied in order to isolate the effect of the penalty. In both datasets, the total activation classifiers for group mono- tonic models (indicated by the prefix mono) are able to approximate the performance of
Model
arg max l(()a
arg max (t(
CIFAR-I0
WideResNet
95.46%
16.35%
MonoWidcResNet
95.64%
94.95%
ImageNet
ResNet-50 MonoResNet-50 76.50%
75.85%
0.10% 72.52%
Table 2: Top-1 accuracy of standard and group monotonic models.

---

Under review as a conference paper at ICLR 2022
the classifier defined at the output layer, arg ma h(z)x. This suggests that the higher total activation generally matches the predicted class for group monotonic models, which indicates the property is successfully enforced. Considering performances obtained at the output layer, there were small variations in accuracy when we included monotonicity penalties, which should be considered in practical uses of group monotonicity. Nonetheless, results suggest that one can perform closely to unconstrained models while focusing on the set of group monotonic candidates. Additional experi- ments are reported on Table 8 on Appendix for cases with small sample sizes, where we show that the performance of the classifier defined at the output layer upper bounds that of the total activation classifier, i.e., the better the underlying classifier the more group monotonic can be made.
4.2.2 USING GROUP MONOTONICITY TO DETECT ANOMALIES
After showing that group monotonicity can be enforced successfully without affecting the prediction performance, we discuss approaches to leverage and introduce applications of the models satis- fying such a property. In partieular, we consider the application of detecting anomalous instances, i.e, those where the model may have made a mistake. For example, consider the case where a classifier is deployed production and, due some problem external to the model, it queried to do prediction for an input consisting of white noise. Standard classifiers would provide predic- tion even for such a clearly anomalous input. However, more desirable behavior to somehow indicate that the instance is problematic. We claim that imposing structure in the features, e.g., by enforeing group monotonicity, can help in deciding when not to predict. To evaluate the proposed method, we implement anomalous test instances using adversarial perturbations. Namely, we cre- ate Loo PGD attackers (Madry et al., 2017) and detect anomalies based on simple statistics of the features. In details, for given input 2, compute the normalized entropy H (x:) of the categor- ical distribution defined by the application of the softmax operator over the set of total activations
Ty))): (x) Σesy (x) log Pk: where K [}]| and the set py corresponds to the parame- log K ters of a categorical distribution defined by: Py (x) softmax(Ty(x))) Decisions can then be made by comparing H (z) with threshold 7 E [0, 1]. defining the detector L [[H">r]- We evaluate the detection performance of this approach on both MNIST and CIFAR-10. Training for the case of CIFAR-10 10 follows the same setup discussed on Section 4.2.1 For MNIST on the other hand, modify the standard LeNet architecture by increasing the width of the second convolutional layer from 64 to 150. This layer is then used to enforce the group monotonicity property. The resulting model is referred to as WideL.Nett. Moreover, 7 and fr are set to 1e10 and 1, respectively. Adver- sarial attacks are created under the white-box setting, i.e., by exposing the full model to the attacker. The perturbation budget in terms of Loo distance is set to 0.3 and 26 for the cases of MNIST and CIFAR-10, respectively. Detection performance is reported in Table terms of the area under the operating curve (AUC-ROC). The baselines are the models for which the monotonicity penalty is not enforced. They are trained under the same conditions and the same computation budget the models where the penalty is enforced. The results are as expected, i.e., for monotonic models, test examples for which the total activations are not structured very often correspond to anomalous imputs.
&
3 for the considered cases in
Finally, due to space constraints, we discuss the application of group monotonicity to explainability
in appendix F
Class
Model
AUC-ROC
Clovs 2
MNIST
WideLeNet
54.47%
Class J'
MonoWideLeNet
100.00%
CIFAR-10
Clors K
WideResNet 67.35%
Input Imuge
Com:. !
CAm e
Ourput Layer
MonoWideResNN
79.33%
Table 3: AUC-ROC (the higher the bet- ter) for the detection of adversarially perturbed data instances.
Figure I: Illustration group monotonic convolutional model. Representations output by certain convolutional layer are partitioned into disjoint subsets.
7

---

Under review as a conference paper at ICLR 2022
4.3 DISENTANGLED REPRESENTATION LEARNING UNDER MONOTONICITY CONSTRAINTS
We now discuss another application where using monotonicity as a regularizer can be beneficial. In particular, consider the case of learning disentangled set of representations. In this case, it is often assumed that the latent variables are independent, and hence control over generative factors can be achieved, c.g., one can modify a specific aspect of the data by modifying the value of a cor- responding latent variable. However, we argue that disentanglement is necessary but not sufficient to enable controllable data generation. We need latent variables that satisfy some notion of mono- tonicity to be able to decide their values resulting in desired properties. For example, assume we are interested in generating images of simple geometric forms and desire to control factors such as shape and size. In this example, even if disentangled set of latent variables is available, we cannot decide how to change the value of the latent variable to get a bigger or a smaller object if there is no monotonic relationship between the size and the value of the corresponding latent variable. We address this issue and build upon the weakly supervised framework introduced by Locctello et al. (020). This work extends the popular 3-VAE setting (Higgins et al. 2016) by introducing weak supervision such that the training instances are presented to the model in pairs (x², where only one few generative factors are changing between each pair. Here, we propose apply group monotonocity over the activations of the corresponding latent variables to have more controlable fac- tors. In the VAE setting, the data is generated according p(a 2 p))) given the latent variables 2. Approximation is then performed by introducing po (z](2) and ( (z),,, both parameterized by neu- ral networks. Our goal is to have = fully factorizable in its dimensions, i.e., p(z) = Pi [a] p(2:), which needs to be captured by the approximate posterior distribution qa (z|x). Training is performed by maximization of a lower-bound on the likelihood of the data, which yields the following loss:
CELBO = Exi,z²
Σ (E(1,)
(1 log(ps(>'18)) + 3DKx(
(4)
where (3((((z)) = go(2)|=²) for the latent dimensions 2; that change across x² and x², and (6(2)12') = ((90(2)l²)) + (0(2)1=²) for those that are common (i.e., the approximate poste- rior of the shared latent variables are forced to be the same for x: and x²²). The outer expectation estimated by sampling pairs of data instances (x+ x²²) where only a number of generative factors vary. In our experiments, we consider the case where exactly one generative factor changes across inputs. Moreover, we follow Locatello et al. (2020) and assign the changing factor, denoted by y, to the dimension j of such that y arg ja Dim[s] [2] enforces disentanglement, controllable generation requires some regularity in z so that users can decide values of z resulting in desired properties in the generated samples. We introduce OVAB to enforce such a regularity, i.e., a monotonic relationship is enforced for the distance between pairs of images where only particular generative factor vary and a corresponding latent variable. In other words, an increasing trend in the value of each dimension of should yield a greater change in the output along a generative factor. Formally, Py AE defined as follows:
While the above objective
Li 2e
AV
log
+ log
(5)
2m
K
K
2
where T. is given by the gradient of the mean squared error (MSH) between pairs of images that --factor away with respect to the dimension of assigned to the changing factor, i.e.:
aMSE (++ ..)) oz
L( x
(6)
In this case, & indicates the reconstruction of x. We evaluate such an approach by training the same 4-layered convolutional VAEs described in (Higgins et al. 2016) using the 3d-shapes datase The dataset is composed of images containing shapes generated from 6 independent generative factors: floor color, wall color, object color, scale, shape and orientation. All combinations of these factors are present exactly once, resulting in m = 480000. We compared VAEs trained with and without the inclusion of the monotonicity penalty given by Qy We highlight that the goal of
httpss://ii wtt.:/giithhhcooo.oo http:ttttttoooocoooomoemmo nd nd/3d- shapes
8

---

Under review as a conference paper at ICLR 2022
Onesaaion
Ohjereske
Shape
Hullowiir
Fioresder
Skap
Figure 2: Comparisons between data generated by standard and monotonic models. On the two panels on the left, we compare generations from a linear combination of the latent code of images which only differs in the object color. On the two panels vertically stacked on the right, we start from the same image but change latent dimension at time.
the proposed framework not to improve over current approaches in terms of how disentangled the learned representations are. Rather, seek to achieve similar results in that sense, but impose extra regularity and structure the relationship between the generated images and the values so that the generative process is more easily controllable. Qualitative analysis is performed and shown in Figure The two panels on the left represent the data generated by a linear combination of the latent code corresponding to two images that only vary in the factor object color. The panels stacked on the right present a per-dimension traversal of the latent space starting from a common image. It can be observed that disentanglement indeed achieved both cases. The monotonic model presents much smoother transitions between colors while the base model gives long sequences of very close images followed by very sharp transitions where the colors sometimes repeat (e.g., green-yellow- green transitions in the fourth row). As for the results per factor, the monotonic model provides more structure the latent space compared to the base model. This can be observed in the shape factor. The monotonic model provides a certain order: sphere, cylinder, and then cube. Visually inspecting many samples, the monotonic model is following this order for the generated shapes. This pattern is even more pronounced in the color factors. We have found that the colors generated by the monotonic model follows the order of the colours in the HUE cycle. So our model has ordered the latent space and know how navigate it to generate desired image. On the other hand, the baseline has no clear order of the latent space. For example, the baseline generates cubes at different ranges of z. Similarly, the colors generated by the baseline model do not have clear order. To further support the claim that Ωy induces regularity in the latent space, we introduce the analysis
shown in Table g We started by increasing (associated to floor color for both models), and recorded the sequence of the generated colors. We observed that for large fraction of the data, the monotonic models yield sequences of images where the color of the floor is ordered according to its corresponding HUE angle. Further details are available in Appendix
Model H1E structured rate
Base model Mon. model
0.00%
89.44%
Table 4: rate of examples where
colors are sorted according to hue.
5 CONCLUSION
We proposed approaches that enable learning algorithms based on risk minimization to find solutions that satisfy some notion of monotonicity. First, we discussed the case where monotonicity is design requirement that needs to be satisfied everywhere. In this case, we identified limitations in prior work that resulted models satisfying the property only in very specific parts of the space. We then introduced an efficient procedure that was observed to significantly improve the solutions in terms of the volume the space where the monotonicity requirement is achieved. In addition, we further argued that, when not required, models satisfying monotonicity present useful properties. We studied the case image classifiers and generative models and showed that imposing structure in learned representations via group monotonicity is beneficial and can be done efficiently.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Norman P Archer and Shouhong Wang. Application of the back propagation neural network algo- rithm with monotonicity constraints for two-group classification problems. Decision Sciences, 24 (1):60-75, 1993.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprini PrXiv:/409.043) 2014.
William Taylor Bakst, Nobuyuki Morioka, and Erez Louidor. Monotonic fronecker-factored
lattice. In International Conference on Learning Representations, 2021. URL https://
openreview net wwtp://oorrummicc00pp0eM
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient- based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV), pp. 839-844 847. IEEE, 2018.
Ching Yao Chuang and Youssef Mrouch. Fair mixup: Faimeess via interpolation. In International
Conference on Learning Representations, 2021. / http//pppneeeeeeeeeeeeeee new.eewwweeeeeeeeeeeeeeeeeeeeeee nww.:frrnttftu forum?
1d=DN1 5m5BXeBn
Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating second-orde functional knowledge for better option pricing. Advances in neural information processing systems, pp. 472-472777 478, 2001.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathmaatiaal Society, 29(4):983-1049, 2016.
Eric Gareia and Maya Gupta. Lattice regression.
Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (eds.), Advances in Neural Information Processing Systems, vol- ume 22. Curran Associates, Inc., 2009. URL https: htttpp://proceeddingg nwwrips. ca/ Paper pdf
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.660 2014.
Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In International conference on machine learning. pp. 1764-1772. PMLR, 2014.
Akhil Gupta, Naman Shukla, Lavanya Marla, Arinbjörn Kolbeinsson, and Kartik Yellepeddi. How to incorporate monotonicity in deep networks while preserving flexibility? arXiv preprint arXiv:1909.10662 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314-1324, 2019.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint 1rXvv:1602.0736) 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:14/2.69880 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
10

---

Under review as a conference paper at ICLR 2022
Xingchao Liu, Xing Han, Na Zhang, and Qiang Liu. Certified monotonic neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 15427-15438. Curran Asso- ciates, Inc., 2020. URL / neuri ppauripp... cc/ pttpppapp/222///1lee b139aeda cet.ocaalc2144ee557722 foc.eiaaaaa.ccceebb---a-ee.eee pdf]
Francesco Locatello, Ben Poole. Gunnar Raetsch, Berhaard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-superrised disentanglement without compromises. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learn- ing, volume 119 of Proceedings of Machine Learning Research, pp. (348-6355 PMLR, 13-18 mlr. press http://1111//1ll0llooooo html
Jul 2020. URL
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:/706.06083 2017.
An-phi Nguyen and María Rodríguez Martínez. Mononet: towards interpretable models by learning monotonic features. arXiv preprint arXiv:1909.144 13111, 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409, 1556, 2014.
Aishwarya Sivaraman, Golnoosh Farnadi, Todd Millstein, and Guy Van den Broeck. Counterexample-gudded learning of monotonic neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural In- formation Processing Systems, volume 33, pp. 11936-11948. Curran Associates, www.iieerrpsss cc/ppaper/222//ilee Ba17773lbb533111111aabbee 7e0b605-
Inc., 2020. URL
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp. 6438-6447. PMLR, 2019.
Antoine Wehenkel and Gilles Louppe. Unconstrained monotonic neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL htt.ps::/proceeddin htw.://eeedddnn......e.e.eeeeeee neurips....... wwweeeipp..ccpppeer///eeeeeeeeee ottp:://eeerrrrrrr.rrrrrr.ooopps (ee.44455c/bbbeeddaaleeeee..ee.e
Paper-appr... .pf
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 1492-1500, 2017.
Seungil You, David Ding, Kevin Canini, Jan Pfeifer, and Maya Gupta. Deep lattice networks and partial monotonic functions. arXiv preprint arXiv: 170.066600 2017.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks. arXiv preprint
1605::1605.07146, 2016.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv: 1212.502,, 2012.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lapez-Paz. mixup: Beyond empiri- cal risk minimization. In ICLR (Poster), 2018. URL /ww.tttpps://ppeeeeeiew..... wwtp//ween..ee.eepee/pe net forum? 1d=r1Ddp1-Rb
11

---

Under review as a conference paper at ICLR 2022
B
B
Figure 3: Illustration unit spheres B and B, on the plane.
A ILLUSTRATIVE EXAMPLES ON THE SPHERE: MIXUP HELPS TO POPULATE THE SMALL VOLUME INTERIOR REGION
To further illustrate the issue discussed in the item 2 of Section 3.1 as well effect our proposal, discuss simple example considering random draws from the unit n-sphere, shown in Figure i.., the set of points = {Ir R" ||1|| < 1}. We further consider a concentric sphere radius 0
given by B, = (x T ||0|| < r). We interested in the probability of a random draw from B to lie outside of B,, i.e.: P([ > r). ~ D(B), for some distribution D. We start by defining D as the Uniform(BB) which results in P([|=12 > r) r" In Figure 4al can see that for growing n, P[[[=]12 > r) is very large even if r R 1, which suggests most random draws will lie close to B's boundary.
We now evaluate the case where mixup is applied and random draws are taken in two steps: we first observe y ~ Uniform(B). and then we perform mixup between y and the origin) i.e., = Ay, ~ Uniform([0, 1)). In this case, P(|ær]22 > r) = 4b as a function of r for increasing n. We can then observe that even for large n, P([ > r) decays linearly with i.e., we populate the interior of B and æ: in this case follows non-uniform distribution such that its norms histogram is uniform.
r")(1 - -r), which is shown in Figure
1.0
n=2
n-10
nm2
n-34
0.6
n-18
0.6
n-
0:
0.4
0.4
4242
00
n=:82
n-66
n4
0.0
0.0
02
0.4
0.6
0.8
1.0
0.8
(b) P[[]e]| > r) as function of for various
(a) P( 2|12 > r) as a function various and x ~ Uni.orm(B).
In this case, Uniform (B).
Az. X Uniform( (0, 1)), y
Figure 4: Illustraiive example showing that uniformly distributed draws on a unit sphere in R concentrate on its boundary for large n. Applying mixup populates the interior of the space.
Similar conclusions hold for any fixed point within B3. The origin is chosen for convenience.
12

---

Under review as a conference paper at ICLR 2022
B PROOF-OF-CONCEPT EVALUATION
We start by describing the approach we employ to generate data containing the properties required by our evaluation. Denote design matrix by XN such that each of its N rows corresponds to feature vector within RP. In order to ensure the data lies in some manifold, we first obtain a low- dimensional synthetic design matrix given by N where each entry is sampled randomly from Uniform([100, 10). We then expand to TR. by applying the following transformation:
X X
where the expansion matrix given by Aax D is such that each of its entries are independently drawn from Uniform([0. (0,, 1J). Throughout our experiments, = [0.31]] was employed.
Target values for the function to be approximated are defined as sums of functions of scalar arguments applied independently over each dimension. We thus select a set of dimensions M E [D] with respect to which f is to be monotonic. i.e.:
f(x) (xx)) + E
REM
JEM
and every 9: IR + is increasing monotonic, while every h4 : R + R is not monotonic.
We then create two evaluation datasets. One of them, referred as the validation set, is identically distributed with respect to since it is obtained following the same procedure discussed above. In order to simulate covariate-shift, we create test set by changing the expansion matrix A to different one.
Xpol= Xoa A, Xtest = Xiest Atest:
where Atest will be given by entry-wise linear interpolations between A, used to generate the train- ing data, and newly sampled expansion matrix A': Arest = aA' + (1 a)A. The parameter [o, 1], set 0.8 in the reported evaluation, controls the shift between Ateat and Ateet in terms of the Frobenius norm, which in turn enables the control of how much the test set shifts relative to the training data.
We thus trained models to approximate f for spaces of increasing dimensions as well as for an increasing number of dimensions with respect to which f monotonic. Results are reported in Table 5 in terms of RMSE on the two evaluation datasets, and in terms of monotonicity in Table where P computed both on random points and on the shifted test set. Entries in the tables correspond to the centers of 95% confidence intervals resulting from 20 independent training runs.
We highlight the two following observations regarding the prediction performances shown in table 5 different models present consistent performances across evaluations, which suggests different penalties do not significantly affect prediction accuracy. Moreover, the pro- posed approach used to generate test data under covariate-shin is effective given the gap in perfor- mance consistently observed between the validation and the test partitions. In terms of monotonicity, results in Table/]ssugeest that 2yandom and Ptraán only effective either random data points, which seems to aggravate when the dimension D grows. Rmizug. on the other hand, is effective on both sets of points, and continues work well for growing D. Furthermore, covariate-shift signifi- cantly affects Urain for higher-dimensional cases, while 9 Smizup performs well in such a case.
innotonicity-enforcii
MPD
20/100
#0/200
80/400
1005500
Valid. RMSE
Test RMSE
Valid. RMSE
Test RMSE
Volid. RMSE
Tea RMSE
Valid, RMSE
Test RMSE
Non-mon. 0.007
0.107
0.006
0.082
0.007
0.087
0.011
0.146
Reendom 0.008
0.117
0.006
0.081
0.007
0.093
0.012
0.125
Deeaco 0.008
0.115
0.006
0.086
0.007
0.089
0.012
0.134
Omiizup 0.008
0.114
0.007
0.084
0.012
Table 5: Prediction performance of models trained on generated data in spaces of growing dimen-
sion (D) and number of monotonic dimensions (M) )). Different regularization strategies do not affect prediction performance. The performance gap consistently observed across the evaluation sets highlights the shift between the two sets of points. The lower the values of RMSE the better.
13

---

Under review as a conference paper at ICLR 2022
MI/D
20/100
40/200
80/400
100/500
Prandom
Ptest
Prandom
Ptest
Prandom
Ptest
Prandom
Ptest
Non-mon. 99.90%
99.99%
97.92%
94.96%
98.47%
96.56%
93.98%
90.01%
2 Rrandom 0.00%
3.49%
0.00%
4.62%
0.01%
11.36%
0.02%
19.90%
Atrain 1.30%
0.36%
4.00%
0.58%
9.67%
0.25%
9.25%
5.57%
Amizap 0.00%
0.35%
0.00%
0.44%
0.00%
0.26%
0.00%
0,42%
Table 6: Fraction of monotonic points for models trained on generated data in spaces growing dimension (D) and number of monotonic dimensions (lM)) Different regularization strategies is effective on only one Prondom or Ptest, while mizzup seems effective throughout conditions. The lower the values of p the better.
C DATASETS, MODELS, AND TRAINING DETAILS FOR EXPERIMENTS REPORTED IN SECTION 3.2
Algorithm T describes procedure used to compute the proposed regularization Qemirup-
Algorithm Procedure to compute 9miszup.
Input mini-batch X [[xxd]- model h, monotonic dimensions M
Xn O} Initialize set of points used to compute regularizer.
X [[Nx Uniform(X**) Sample random mini-batch with size N.
X concat(X, X) Concatenate data and random batches.
repeat
i ~ Uniform 11.2,..,2N7?? # Sample random pair of points.
X Uniform((0 1])
x = - (1- A) Mix random pair.
Xg.add(x) # Add to set of regulari zation points.
until Maximum number of pairs reached
a Qminmpp (h, M) = Xan Exex, Eiem max 0, -0,- 0h(z)
dz,
return Qmizup
In Table 7 we list details on the three datasets used to evaluate our proposals reported in Section
Dataset Compa Loan Lending CluF
Dim X
M
Train
# Test
Task
13
4937
1235
Classification Regression
33
11
8500
1500
Blog Feedback
280
8
47287
6904
Regression
Table 7: Description of datasets used for empirical evaluation.
Models follow the architecture in (Liu et al. 2020) using dense layers whose weights are kept sep- arate in early layers for the input dimensions with respect which monotonicity is be enforced. We set the depth of all networks to 3, and use a bottleneck of size 10 for two datasets (Compas and Loan Lending Club), and 100 for the case of the Blog Feedback dataset and the experiments on generated data. Training is carried out with the Adam optimizer (Kingma & Ba 2014) with global learning rate of 5e-3, and y is set to 1e4. The training batch size is set to 256 throughout experiments.
https://wwer http:///www..ooottooooooaaaa/aaa
full schema
14

---

Under review as a conference paper at ICLR 2022
Model
arg maXkey (())
arg maxkey (z)
10%
WideResNet
85.68%
16.35%
MonoWideeesNet
85.77%
82.21% %
30%
WideResNet
92.12%
14.51
MonoWideRRsNe 92.42%
88.88%
60%
WideResNet 94.51%
10.08%
Wideeeeee 94.86%
93.81% %
Table 8: Top-1 accuracy obtained by both standard and group monotonic models sub-samples of CIFAR-10. 10. Predicition performance obtained by classifiers defined by the total activations is upper bounded by the performance obtained at the output layer for monotonic models.
D MODELS AND TRAINING DETAILS FOR EXPERIMENTS REPORTED IN SECTION 4
For the case of 100 WideResNets (agoruyko & Komodakis 2016 used. The models initialized randomly and trained both with and without the monotonicity penalty. Standard stochastic gradient descent (SGD) implements the parameters update rule with learning rate starting 0.1, being decreased by factor of 10 on epochs 10, I 150, 250, and 350. Training is carried out for total of 600 epochs with batch size of 64. For ImageNet, on the other, training consists of fine tuning a pre-trained ResNet-50, where the fine-tum fine-tuning phase included the monotonicity penalty. We do so by training the model for 30 epochs on the full ImageNet training partition. In this case, given that the label set y is relatively large, using the standard ResNet-50 50 would result in small slices Se. To avoid that, we add an extra final convolution layer with W = 15K. Training is once more carried out with SGD using learning rate set to 0.001 in this case, and reduced by a factor of 5 at epoch 20. In both cases, the group monotonicity property is enforced at the last convolutional layer. Other hyperparameters such as the strength y of the monotonicity penalty temperature y used to compute Pgroup are set to and 50 for the case of CIFAR-10, and to 5 and 10 for the of ImageNet. Both momentum and weight decay are further employed and their corresponding parameters are set to 0.9 and 0.0001. For MNIST classifiers, training is performed for 20 epochs using a batch size of 64 and the Adadelta optimizer ZZeiler 2012) with learning rate of 1.
well as the inverse
E ENFORCING GROUP MONOTONICITY UNDER SMALL SAMPLES
Using CIFAR-10, we further evaluate how the proposed group monotonicity penalty behaves in data- constrained settings, i.e., we check whether or not the property can be enforced under small sample regimes. We do so by sub-sampling the original training data by randomly selecting a fraction of the training images uniformly across classes. We then train the same WideResNet for the same computation budget in terms of number of iterations as the models trained in the complete set of images. The learning rate schedule also matches that of the training on the full dataset in that the learning rate is reduced at exactly the same iterations across all training cases. Results are reported in Table
8 for sub-samples corresponding to 10%, 30%, and 60% of CIFAR-10. Results are consistent across the three sets of results showing that predictions obtained from the total activation of feature slices approximate the prediction performance of the underlying model for the case of group monotonic predictors, i.e., the extent to which the underlying model is able to accurately predict correct classes upper bound the resulting "level of monotonicity". In simple terms, the better the classifier, the more group monotonic can be made.
F SELECTING FEATURE MAPS TO COMPUTE VISUAL EXPLANATIONS
Approaches based on Class Activation Maps (CAM) such as Grad-CAM and its variations Selvaraju al. 2017 Chattopadhay al. 2018) seek extract explanations from convolutional models. By
15

---

Under review as a conference paper at ICLR 2022
Figure 5: Example of explanation meat-map and corresponding occlusion obtained with Grad-CAM and a ResNet-50 trained on ImageNet. The example belongs to the validation set and corresponds to the class snowmobile.
explanation we mean to refer to indications of properties of the data implying the predictions of a given model. Under such framework, one can obtain so-called explanation heat-maps through the following steps: (1) Compute weighted sum of activations of feature maps in a chosen layer; (2) Upscale the results in order to match the dimensions of the input data; (3) Superimpose results onto the input data. Specifically for the case of applications to image data, following those steps results in highlighting the patches of the input that were deemed relevant to yield the observed predictions. Different approaches were then introduced in order to define the weights used in the first step. A very common choice is to use the total gradient of the output corresponding to the prediction with respect activations of each feature map.
For the case of group monotonic classifiers, we are interested in verifying whether one can define useful explanation heat-maps by considering only the feature slices corresponding the predicted class, i.e., for a given input pair (I, y). we compute explanation heat-maps considering only its cor- responding feature activation slice (z)). We thus design an experiment to evaluate the effectiveness of such an approach by using external auxiliary classifiers to perform predictions from test data that was occluded using explanation heat-maps obtained using different models and sets of representa- tions. In other words, we use the explanation maps to remove from the data the parts that were not indicated as relevant. We then assume that good explanation maps will be such that classifiers are able to correctly classify occluded data since relevant patches are conserved. In further details, oc- clusions are computed by first applying a CAM operator given a model h and data æ, which results in heat-map with entries in [0, 1]. We then use such heat-map multiplicative mask to get an occluded version of x, denoted x', i.e.:
CAM(2, ht)
(10)
where the operator indicates element-wise multiplication. An example of such procedure is shown in Figure 5 We apply the above procedure to all of the validation data, and use resulting points then assess the prediction performance of auxiliary classifiers.
Explanation maps are computed using the same models discussed in Section 4.2. for ImageNet. The CAM operator corresponds to a variation of Grad-CAM++ (Chattopadhay et al. 2018) where the model activations are directly employed for weighing feature maps rather than the gradients. We consider 4 auxiliary pre-trained classifiers corresponding to ResNext-50 (Xie et al. 2017). MobileNet-v3 (Howard et al., 2019)), VGG-16 (Simonyan & Zisserman, 2014), and SqueezeNet (landola et al., 2016). Results are reported in Table which also include the reference performance of the auxiliary classifiers on the standard validation set in order to provide an idea of the gap in per- formance resulting from removing parts of test images via occlusion. We highlight the performance reported in the last row of the Table. In that case, explanation maps for the group monotonic model are computed from only the features of the class slice, which is enough to match the performance of standard ResNet-50 with full access to the features. This suggests that representations learned
16

---

Under review as a conference paper at ICLR 2022
Aux. classifier
Model (h)
ResNext-50
MobileNet-v3
VGG-16
SqueezeNet
Reference perf.
77.62%
74.04%
71.59%
58.09%
ResNet-50
72.94%
68.31%
67.34%
49.95%
MonoResNet-50
72.88%
68.75%
66.99%
48.92%
MonoResNet-500 (Constrained)
72.44%
66.55%
66.92%
45.83%
Table 9: Top-1 accuracy of auxiliary classifiers evaluated on data created by occluding patches deemed irrelevant by explanation heat-maps given by different models. The performance of mono- tonic classifiers when constrained to consider only the feature maps within the slice corresponding to their prediction is further reported and shown to closely math the performance of cases where the full set features considered.
by group monotonic models such that all the information required explain given class is contained in the slice reserved for that class.
17

---

Under review as a conference paper at ICLR 2022
G
EXAMPLES OF EXPLANATION HEAT-MAPS AND OCCLUDED DATA
50
50
100
100
100
100
150
150
150
150
200
200
200
100
200
100
0-
0
0-
50
50
50
50
100
100
100
100
150
150
150
200
200
100
200
100
200
100
200
100
200
50
50
200
200
200
200
100
200
0
100
200
100
200
0
100
200
100
100
100
100
150
150
150
150
200
200
200
200
100
200
100
200
100
200
200
Figure 6: Examples explanation heat-maps superimposed onto images. From left to right we have the original image, results obtained from ResNet-50, monoResNet-50, and moneResNet- 50 where the CAM operator only access the slice corresponding to the underlying class. All are obtained with Grad-CAM.
18

---

Under review as a conference paper at ICLR 2022
100
100
100
100
150
150
150
150
200
200
200
200
100
200
100
200
100
200
200
50
50
100
100
100
100
150
150
150
150
200
200
200
200
200
200
50
50
50
50
100
100
100
100
150
150
150
100
200
100
200
100
200
100
200
50
50
200
200
200
200
100
200
100
200
100
200
0
100
200
Figure 7: Examples of occluded data using explanation heat-maps. From left to right we have the original image, results obtained from ResNet-50, a monoResNet-50, and menoResNet-50 where the CAM operator only access the slice corresponding to the underlying class. All are obtained with Grad-CAM.
19

---

Under review as a conference paper at ICLR 2022
Figure 8: HUE circle of RGB images. Original image from: httpps:/en..wkkipeed...o wiki/Hue
H ANALYSIS OF COLOR SEQUENCES FOR GENERATED DATA
We performed a set of experiments in order to evaluate whether some kind of ordering could be ob- served once we generate data for increasing values of z, specifically on dimensions that correspond to colors. To do that, we created an increasing sequence of values by defining uniform grid in [0, 1] with 50 steps. We then encoded a particular image, but decoded latent vectors after substituting the value in the dimension corresponding to floor color by the values in the sequence.
Generated sequences of images are shown FiguresPannd and T0 for the base and monotonic models, respectively. In each such case, we plot the images on the left, and bottom-left patches of size 10x10 so as to highlight the color sequences that we observe with such an approach. Surprisingly, observed that monotonic models tend to generate colors in sequence that matches the HUE circle for RGB images, represented Figure X for reference. Besides visually verifying that to be the case across number of generated examples, in Table in Section 4.33 we check the fraction of the dataset where such sequences of patches are sorted in terms of their HUE angles.
(a) Data for increasing values for the latent dimension associated to floor color.
(b) Bottom-left 10x10 patches of generated images.
Figure 9: Data generated by standard model for traversals of on the dimension corresponding to floor color
20

---

Under review as a conference paper at ICLR 2022
(a) Data for increasing values the latent dimension associated to floor color.
(b) Bottom-left 10x10 patches of generated images.
Figure 10: Data generated by monotonic model for traversals of a on the dimension corresponding to floor color
21

---

Under review as a conference paper at ICLR 2022
I EXAMPLES OF DATA GENERATED WITH STANDARD AND MONOTONIC MODELS
(a) Input pair.
(b) Data generated by standard model.
Data generated monotonic model.
Figure 11: Generating data by moving along the line passing over latent representation for inputs for which single factor is different. Generative factor changing: floor color.
22

---

Under review as a conference paper at ICLR 2022
(a) Input pair.
(b) Data generated by standard model.
(c) Data generated by monotonic model.
Figure 12:
Generating data by moving along the line passing over latent representation for inputs for which single factor is different. Generative factor changing: wall color.
23

---

Under review as a conference paper at ICLR 2022
(a) Input pair.
(b) Data generated by standard model.
(c) Data generated by monotonic model.
Figure 13:
Generating data by moving along the line passing over latent representation for inputs for which single factor is different. Generative factor changing: object Color.
24

---

Under review as a conference paper at ICLR 2022
(a) Input pair.
(b) Data generated by standard model.
(c) Data generated by monotonic model.
Figure 14: Generating data by moving along the line passing over latent representation for inputs for which single factor is different. Generative factor changing: scale.
25

---

Under review as a conference paper at ICLR 2022
(a) Input pair.
(b) Data generated by standard model.
(c) Data generated by monotonic model.
Figure 15: Generating data by moving along the line passing over latent representation for inputs for which single factor is different. Generative factor changing: shape.
26

---

Under review as a conference paper at ICLR 2022
(a) Input pair.
(b) Data generated by standard model.
(c) Data generated by monotonic model.
Figure 16:
Generating data by moving along the line passing over latent representation for inputs for which single factor is different. Generative factor changing: orientation.
27


Reviews:
[{"review_id": "kGwQXk1V--N", "reviewer": "Reviewer_Sj4Q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Clarity:  Overall, the work was clearly written with a few minor typos here and there.  I felt that the group monotonicity section was a bit tough to follow at the beginning -- I only really understood what you were trying to do after seeing the applications.  It could probably be better motivated.\n\nNovelty:  The hard constraint on monotonicity is a simple application of Mixup.  While the evaluation is definitely of general interest, the novelty is a bit low here.  The second approach is novel to me at least.  On first reading, I wasn't even certain what kinds of things that you could do with it.  I think it should really be explained a bit more in the introduction as it was, to me at least, a very different kind of requirement.\n\nSignificance:  The significance of this work, in my opinion is entirely in the new group monotonicity property.  While the proposed applications of this were interesting, I feel that there could possibly even more utility in certain domains -- though I can't say for certain.\n\nSpecific comments:\n\n- \"i.e.\" and \"e.g.\" must always be followed by a comma, e.g., like this.\n- As mentioned about, I'd really like to hear your high level motivation for the group monotonicity property.  It appears to be a bit different from the first type of monotonicity discussed, i.e., it's different than just soft enforcing a hard monotonicity constraint, right?\n- What was the configuration of the detector in Table 3?  I didn't see an explanation of the choice of tau?  How did you determine the perturbation budgets?  Thoughts about why performance on MNIST is so good?\n- Figure 2 was a bit small and too difficult for me to really get a good understanding of what it was trying to show.  I understand that some types of monotonicity result from the training, but it's less clear to me what advantages it has in this case (other than showing off the approach).", "summary_of_the_paper": "The paper investigates two different strategies for adding monotonicity into the training of NNs: one in which the monotonicity is required on some subset of the input and one in which monotonicity is encouraged, but not required, in a part of the model.  Experiments illustrate that these approaches do achieve different types of monotonicity in practice.", "summary_of_the_review": "An interesting paper that fills some gaps in existing work and posits some interesting applications of monotonicity.  The first contribution seems a little weak as it is built almost exclusively from existing methods with little deviation while the second contribution has more novelty.  I feel that the second approach needs some better motivation and possibly a better application  -- I would prefer that you drop the first approach to include it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a_OBmY9V7vb", "reviewer": "Reviewer_KKi2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1) This paper addresses an important problem.  The monotonicity constraint is often overlooked by the ML community, but it does play an important role in many real-world applications. Therefore, an improved method to encourage monotonicity can have sizable impact on these applications, and help ML practitioners better impose their priors on the hypothesis space.\n2) Writing is clear.\n\nWeaknesses\n1) The contribution is rather incremental.\n2) The authors didn't provide adequate explanations for certain important details in the paper itself.  For example, the authors propose to create synthetic examples by interpolating both features and vectors, but in the experiment involving binary classification data set, it's not clear how to interpolate the labels here. Did the authors simply interpolate between 0 and 1?  Why we need to use both interpolated examples as well as random examples (not clear from the paper)?", "summary_of_the_paper": "This paper proposes an incremental improvement to existing methods that encourage monotonicity through a regularization term. The contribution of the paper is about how to sample the data to compute this regularization term, which is an expectation w.r.t. a data distribution. So instead of purely sampling from existing training data or performing uniform sampling in potentially high-dimensional feature space, this paper proposes to use mixup, which essentially involves creating synthetic examples by interpolating between existing training examples. The authors also extend this regularization term to the case where the prediction function outputs a vector, e.g., multi-class classification), and the case of VAE. The authors provide some experiment evidence that the proposed change can reach better performance for some data sets.", "summary_of_the_review": "Given the incremental contributions, I consider this paper marginally below the acceptance threshold. It can be made stronger if the authors can provide more insights or make the experiments more illustrative or extend to the case of label's order on categorical features (treated as a partially ordered set).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZgdDEhRz4rf", "reviewer": "Reviewer_P7xG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Strengths}$\n\nThe paper is pretty straightforward and easy to understand. The content is supported with proper experimentation and detailed explanations. The authors demonstrate how to use monotonicity through penalty terms in models that are not constructed to be monotonic without the use of constraints.\n\n$\\textbf{Weaknesses}$\n\n1. Consistent grammar issues.\n\nExamples:\n\npage 2 \"such group of methods\" --> \"such a group of methods\"\n\npage 2 \"monotonicity has been also observed\" --> \"monotonicity has also been observed\"\n\npage 2 \"we define a new notion of monotonicity which is shown to be useful\" --> \"we define a new notion of monotonicity and show that it is useful\"\n\npage 2 \"such property\" --> \"such a property\"\n\npage 2 \"such class of models\" --> \"such a class of models\"\n\npage 8 \"For example, assume we are interested in generating images of simple geometric forms, and desire to control factors...\" --> \"For example, assume we are interested in generating images of simple geometric forms and desire to control factors…\"\n\nThere are also missing commas and other similar grammar issues.\n\n2. Claim that lattice models \"scale poorly with respect to the input space\" is only partially true. This is the case for standard lattice models; however, the cited paper Monotonic Kronecker-Factored Lattice is an extension of lattice models that has linear space/time complexity. Thus, the claim in this paper that they scale poorly is not correct and needs to be changed. This claim sets the stage for demonstrating why the approach in the paper is desirable, but the claim is not true and the reality of linear space/time complexity hurts the argument of the paper. I would suggest re-working this claim to better show the reader why we would seek to use the methods outlined instead of a model that is monotonic by construction (and thus guaranteed to be monotonic everywhere in the input space and hold true even under a distribution shift).\n\n3. Formatting issue section 3.1 the first sentence is weirdly formatted and is difficult to parse.\n\n4. I found it a bit odd that the formula for mixup was not included in section 3.1 like the other penalty terms. I think explicitly writing out the formula for p_test like p_train and p_random would make things more clear for the reader.\n\n5. Paper claims that performance is matched when imposing group monotonicity, but the results in Table 2 clearly show a significant decrease in performance. I would like the authors to more clearly explain their claim that performance is matched.\n\n", "summary_of_the_paper": "This paper proposes techniques for using monotonicity both as a requirement and as a regularizer. The primary contributions are the mixup penalty term and group monotonicity, both of which build upon previous work in the field.", "summary_of_the_review": "My primary concern with the paper is that Monotonic Kronecker-Factored Lattice provides a linear space/time complexity model that is monotonic by construction and thus monotonic everywhere in the input space and holds true even under a distribution shift. I believe that this model somewhat undermines the usefulness of the models described in the paper. I think the paper should make more clear why we would prefer to use the methods described over a model such as Monotonic Kronecker-Factored Lattice given that the model does scale well wrt the input space. This is not to say that the methods described are not useful; rather, this is to say that it should be made more clear why the methods described are as useful as claimed.\n\nAs for group monotonicity, I think this is a novel contribution that the authors have shown to work well and potentially be quite useful, particularly for the generative and adversarial cases.\n\nFor these reasons, I believe the paper is marginally below the acceptance threshold. Should the authors properly address my concerns, I see so reason not to bump up my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BWR0nmGNCm", "reviewer": "Reviewer_onzS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths.\n--------------\nI found the paper written clearly and fairly easy to understand. \nThe claims in Sections 1--3 are justified and the authors provide good context for their work. The results of Section 3 are interesting for applications were partial monotonicity is desired but is not a hard requirement (e.g. for robustness and interpretability).\nI found the application in Section 4.3 interesting. Showing how imposing monotonicity allows better controllability of a generative model.\n\nWeaknesses\n-----------------\n1. In Section 3. The paper uses a measure $\\rho$ that is essentially the fraction of examples at which *local* monotonicity (in any of the prescribed directions in $M$) is violated and then show that this measure decreases when using the paper's method over the baselines. However, I'm not certain that this measure corresponds to the global monotonicity requirment that is often desired in practice: namely, the one that appears in Definition 1. For example, consider a $1$-D function over $[1,99.99]$ whose graph is a piecewise linear curve connecting the points $(0,100), (0.99,100.99), (1,99), (1.99, 99.99), (2,98),  (2.99,98.99), ..., (99, 1), (99.99,1.99)$. This function has nonnegative derivative at about $99%$ of its domain, yet if one chooses two points $x_1, x_2$ uniformaly and independently from the domain, then there's at least a $97%$ chance that $f(min{x_1,x_2}) > f(max{x_1,x_2})$. I think, therefore, that it would be good to complement the local $\\rho$ with an estimate of the probability that Definition 1 would not hold over the distribution in question (training, test or random).\n\n2. Section 4.1\nThe authors introduce the notion of group monotonicity, but it's unclear how the regularizer introduced in equation 3 helps to encourage that property. Specifically, 1) Only the sum of the gradient is taken into account (so it could be that a component a_w_{i,j} has a very negative gradient, but still the sum will be positive), and 2) the softmax in equation 3 seems to encourage that the total gradient of $S_y$ is larger than the total gradient of all the other $S_k$'s, not that it's positive.\nPerhaps I'm missing something?\n\n3. Section 4.2\nThe paper claims that the fact that a good performance of the \"total activation classifier\" shows evidence that the original classifier satisfies group monotonicity.\nBut that claim is not clear to me. The total activation classifier does not depend on the part of the network that computes the output from the intermediate layer which is critical for the satisfaction of group monotonicity. \n\n4. Section 4.2.2\nThe paper doesn't compare their methods to other methods for detecting noisey/adverserial test examples. \n\n", "summary_of_the_paper": "The paper has two main contributions:\n\n1. It takes a known monotonicity regularizer and trains with it using a new distribution that is roughly a mixing of uniform and the training distribution. It shows empirically that using this method increases the \"size\" of the input region in which the model is monotonic. \n\n2. It defines several regularizers that are meant to encourage \"monotonic behavior\" of the model's output w.r.t the outputs of an intermediate layer or latent variable. It provides experiments that show how using these regularizers does not hurt the model's performance and two applications of the added structure: 1) detecting noisey/adverserial examples and 2) more controllable generation in generative models.\n", "summary_of_the_review": "See above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach to enforce monotonicity in the context of risk minimization, or to promote it as an inductive bias.  This improves upon existing point-wise gradient based methods by expanding the region where monotonicity is enforced.  Group monotonicity is found valuable as a regularization for convolutional models, and multiple applications were shown where the approach appears effective.

The paper is well written, and received detailed discussion. Despite the rebuttal, some major concerns remain, such as drop in accuracy, and empirical estimate of the probability that Definition 1 would not hold over the distribution in question.  Overall, revisions are needed to make the paper publishable.
Decision: Reject