Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper investigates two different strategies for adding monotonicity into the training of NNs: one in which the monotonicity is required on some subset of the input and one in which monotonicity is encouraged, but not required, in a part of the model.  Experiments illustrate that these approaches do achieve different types of monotonicity in practice.
main_review: Clarity:  Overall, the work was clearly written with a few minor typos here and there.  I felt that the group monotonicity section was a bit tough to follow at the beginning -- I only really understood what you were trying to do after seeing the applications.  It could probably be better motivated.

Novelty:  The hard constraint on monotonicity is a simple application of Mixup.  While the evaluation is definitely of general interest, the novelty is a bit low here.  The second approach is novel to me at least.  On first reading, I wasn't even certain what kinds of things that you could do with it.  I think it should really be explained a bit more in the introduction as it was, to me at least, a very different kind of requirement.

Significance:  The significance of this work, in my opinion is entirely in the new group monotonicity property.  While the proposed applications of this were interesting, I feel that there could possibly even more utility in certain domains -- though I can't say for certain.

Specific comments:

- "i.e." and "e.g." must always be followed by a comma, e.g., like this.
- As mentioned about, I'd really like to hear your high level motivation for the group monotonicity property.  It appears to be a bit different from the first type of monotonicity discussed, i.e., it's different than just soft enforcing a hard monotonicity constraint, right?
- What was the configuration of the detector in Table 3?  I didn't see an explanation of the choice of tau?  How did you determine the perturbation budgets?  Thoughts about why performance on MNIST is so good?
- Figure 2 was a bit small and too difficult for me to really get a good understanding of what it was trying to show.  I understand that some types of monotonicity result from the training, but it's less clear to me what advantages it has in this case (other than showing off the approach).
summary_of_the_review: An interesting paper that fills some gaps in existing work and posits some interesting applications of monotonicity.  The first contribution seems a little weak as it is built almost exclusively from existing methods with little deviation while the second contribution has more novelty.  I feel that the second approach needs some better motivation and possibly a better application  -- I would prefer that you drop the first approach to include it.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an incremental improvement to existing methods that encourage monotonicity through a regularization term. The contribution of the paper is about how to sample the data to compute this regularization term, which is an expectation w.r.t. a data distribution. So instead of purely sampling from existing training data or performing uniform sampling in potentially high-dimensional feature space, this paper proposes to use mixup, which essentially involves creating synthetic examples by interpolating between existing training examples. The authors also extend this regularization term to the case where the prediction function outputs a vector, e.g., multi-class classification), and the case of VAE. The authors provide some experiment evidence that the proposed change can reach better performance for some data sets.
main_review: Strengths
1) This paper addresses an important problem.  The monotonicity constraint is often overlooked by the ML community, but it does play an important role in many real-world applications. Therefore, an improved method to encourage monotonicity can have sizable impact on these applications, and help ML practitioners better impose their priors on the hypothesis space.
2) Writing is clear.

Weaknesses
1) The contribution is rather incremental.
2) The authors didn't provide adequate explanations for certain important details in the paper itself.  For example, the authors propose to create synthetic examples by interpolating both features and vectors, but in the experiment involving binary classification data set, it's not clear how to interpolate the labels here. Did the authors simply interpolate between 0 and 1?  Why we need to use both interpolated examples as well as random examples (not clear from the paper)?
summary_of_the_review: Given the incremental contributions, I consider this paper marginally below the acceptance threshold. It can be made stronger if the authors can provide more insights or make the experiments more illustrative or extend to the case of label's order on categorical features (treated as a partially ordered set).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper has two main contributions:

1. It takes a known monotonicity regularizer and trains with it using a new distribution that is roughly a mixing of uniform and the training distribution. It shows empirically that using this method increases the "size" of the input region in which the model is monotonic. 

2. It defines several regularizers that are meant to encourage "monotonic behavior" of the model's output w.r.t the outputs of an intermediate layer or latent variable. It provides experiments that show how using these regularizers does not hurt the model's performance and two applications of the added structure: 1) detecting noisey/adverserial examples and 2) more controllable generation in generative models.

main_review: Strengths.
--------------
I found the paper written clearly and fairly easy to understand. 
The claims in Sections 1--3 are justified and the authors provide good context for their work. The results of Section 3 are interesting for applications were partial monotonicity is desired but is not a hard requirement (e.g. for robustness and interpretability).
I found the application in Section 4.3 interesting. Showing how imposing monotonicity allows better controllability of a generative model.

Weaknesses
-----------------
1. In Section 3. The paper uses a measure $\rho$ that is essentially the fraction of examples at which *local* monotonicity (in any of the prescribed directions in $M$) is violated and then show that this measure decreases when using the paper's method over the baselines. However, I'm not certain that this measure corresponds to the global monotonicity requirment that is often desired in practice: namely, the one that appears in Definition 1. For example, consider a $1$-D function over $[1,99.99]$ whose graph is a piecewise linear curve connecting the points $(0,100), (0.99,100.99), (1,99), (1.99, 99.99), (2,98),  (2.99,98.99), ..., (99, 1), (99.99,1.99)$. This function has nonnegative derivative at about $99%$ of its domain, yet if one chooses two points $x_1, x_2$ uniformaly and independently from the domain, then there's at least a $97%$ chance that $f(min{x_1,x_2}) > f(max{x_1,x_2})$. I think, therefore, that it would be good to complement the local $\rho$ with an estimate of the probability that Definition 1 would not hold over the distribution in question (training, test or random).

2. Section 4.1
The authors introduce the notion of group monotonicity, but it's unclear how the regularizer introduced in equation 3 helps to encourage that property. Specifically, 1) Only the sum of the gradient is taken into account (so it could be that a component a_w_{i,j} has a very negative gradient, but still the sum will be positive), and 2) the softmax in equation 3 seems to encourage that the total gradient of $S_y$ is larger than the total gradient of all the other $S_k$'s, not that it's positive.
Perhaps I'm missing something?

3. Section 4.2
The paper claims that the fact that a good performance of the "total activation classifier" shows evidence that the original classifier satisfies group monotonicity.
But that claim is not clear to me. The total activation classifier does not depend on the part of the network that computes the output from the intermediate layer which is critical for the satisfaction of group monotonicity. 

4. Section 4.2.2
The paper doesn't compare their methods to other methods for detecting noisey/adverserial test examples. 


summary_of_the_review: See above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes techniques for using monotonicity both as a requirement and as a regularizer. The primary contributions are the mixup penalty term and group monotonicity, both of which build upon previous work in the field.
main_review: $\textbf{Strengths}$

The paper is pretty straightforward and easy to understand. The content is supported with proper experimentation and detailed explanations. The authors demonstrate how to use monotonicity through penalty terms in models that are not constructed to be monotonic without the use of constraints.

$\textbf{Weaknesses}$

1. Consistent grammar issues.

Examples:

page 2 "such group of methods" --> "such a group of methods"

page 2 "monotonicity has been also observed" --> "monotonicity has also been observed"

page 2 "we define a new notion of monotonicity which is shown to be useful" --> "we define a new notion of monotonicity and show that it is useful"

page 2 "such property" --> "such a property"

page 2 "such class of models" --> "such a class of models"

page 8 "For example, assume we are interested in generating images of simple geometric forms, and desire to control factors..." --> "For example, assume we are interested in generating images of simple geometric forms and desire to control factors…"

There are also missing commas and other similar grammar issues.

2. Claim that lattice models "scale poorly with respect to the input space" is only partially true. This is the case for standard lattice models; however, the cited paper Monotonic Kronecker-Factored Lattice is an extension of lattice models that has linear space/time complexity. Thus, the claim in this paper that they scale poorly is not correct and needs to be changed. This claim sets the stage for demonstrating why the approach in the paper is desirable, but the claim is not true and the reality of linear space/time complexity hurts the argument of the paper. I would suggest re-working this claim to better show the reader why we would seek to use the methods outlined instead of a model that is monotonic by construction (and thus guaranteed to be monotonic everywhere in the input space and hold true even under a distribution shift).

3. Formatting issue section 3.1 the first sentence is weirdly formatted and is difficult to parse.

4. I found it a bit odd that the formula for mixup was not included in section 3.1 like the other penalty terms. I think explicitly writing out the formula for p_test like p_train and p_random would make things more clear for the reader.

5. Paper claims that performance is matched when imposing group monotonicity, but the results in Table 2 clearly show a significant decrease in performance. I would like the authors to more clearly explain their claim that performance is matched.


summary_of_the_review: My primary concern with the paper is that Monotonic Kronecker-Factored Lattice provides a linear space/time complexity model that is monotonic by construction and thus monotonic everywhere in the input space and holds true even under a distribution shift. I believe that this model somewhat undermines the usefulness of the models described in the paper. I think the paper should make more clear why we would prefer to use the methods described over a model such as Monotonic Kronecker-Factored Lattice given that the model does scale well wrt the input space. This is not to say that the methods described are not useful; rather, this is to say that it should be made more clear why the methods described are as useful as claimed.

As for group monotonicity, I think this is a novel contribution that the authors have shown to work well and potentially be quite useful, particularly for the generative and adversarial cases.

For these reasons, I believe the paper is marginally below the acceptance threshold. Should the authors properly address my concerns, I see so reason not to bump up my score.
review_title: null

Meta-review: This paper proposes a new approach to enforce monotonicity in the context of risk minimization, or to promote it as an inductive bias.  This improves upon existing point-wise gradient based methods by expanding the region where monotonicity is enforced.  Group monotonicity is found valuable as a regularization for convolutional models, and multiple applications were shown where the approach appears effective.

The paper is well written, and received detailed discussion. Despite the rebuttal, some major concerns remain, such as drop in accuracy, and empirical estimate of the probability that Definition 1 would not hold over the distribution in question.  Overall, revisions are needed to make the paper publishable.
Decision: Reject