Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method to build neural networks where an output is provably monotonic with respect to certain inputs. This is intended for applications where domain knowledge exists for such monotonic relations. For such applications, the benefits are better robustness and better interpretability.
strength_and_weaknesses: This paper contains the following statements that breaks the double blind review policy. This may warrant rejection without review and I'll leave the decision to the Area Chair.

-- "Our algorithm has been adopted to classify the decays of subatomic particles produced at the CERN Large Hadron Collider in the real-time data-processing system of the LHCb experiment, which was our original motivation for developing this novel architecture."

-- "The algorithm described here has, in fact, been implemented by a high-energy particle physics experiment at the European Center for Nuclear Research (CERN), and is actively being used to collect data at the Large Hadron Collider (LHC) in 2022, where highenergy proton-proton collisions occur at 40 MHz."

-- "Due to the simplicity and practicality of our method, the LHCb experiment is now using the proposed architecture for real-time data selection at a data rate of about 40 Tbit/s."

-- "Those data will be made available in later years at the discretion of the LHCb collaboration."


Having said the above, I'll provide a brief review of the proposed technique anyway.

The motivation of this work is meaningful. The monotonicity guarantee is correct. 

However, whether the resulting neural networks are still expressive is very much in question.

The proposal is to first build a neural network g(x) that has a Lipschitz constant of lamda with respect to L1 norm, and then the overall model is
 f(x) = g(x) + lamda * sum_over_mono_set (x_i)
where x_i's are the subset of input features that we have the domain knowledge about monotonicity. This can easily be extended to when f(x) and g(x) are vectors, and we can have a different set of x_i's for each dimension in f(x), depending on the domain knowledge.

The main problem is in building g(x). For a linear layer, the L1 norm of the weight matrix is the max over sum of absolute values across each row. For g(x) which may have multiple layers, its Lipschitz bound from linear layers is the product of the said max-row-sum values across layers. This is very restrictive for larger networks and much more restrictive than past works that work with L2 Lipschitz bounds.
On pages 4-5, the authors argue that using GroupSort instead of ReLU would help. That is not true, and choosing non-linearities cannot solve the problem of restrictive linear layers.
If, for a particular application, it is sufficient to have g(x) as one linear layer plus some non-linearity, the proposed method should work fine. If two layers, this becomes questionable. For deeper networks, hidden dimensions need to be very small for this method to be viable. For general neural network architectures, I see little hope with the proposal. 

clarity,_quality,_novelty_and_reproducibility: Clarity is good.

Quality is fair.

Novelty is poor. Please see the comments above.

Reproducibility is good.

summary_of_the_review: This paper may get rejected for breaking blind review policy.

If not: this paper is working on a meaningful problem, however the proposed technique has significant limitations. It may work for very small neural networks but is not applicable to general architectures.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a novel method for learning monotonic neural networks, meaning that its output can be constrained and guaranteed to always increase (or decrease) with respect to any subset of its inputs. It accomplishes this by starting with a standard feedforward architecture whose weights and activation function are bounded so that the function has a Lipschitz constant of 1, and then they simply add (or subtract) a residual connection from the monotonic input directly to the network's output, guaranteeing that the partial derivative w.r.t. the monotonic input is always positive (or negative).

The authors show empirically that this architecture performs favorably compared to an unconstrained network, as well as compared to a variety of prior work on monotonic machine-learned function classes.
strength_and_weaknesses: Strengths:
* Achieves guaranteed monotonicity with a relatively simple function class that is easy and efficient to implement and train.
* Side benefit of a regularized function class, due to the Lip_1 constraint. But this may be a weakness when highly-flexible function classes are desirable without needing to resort to very deep models.

Weaknesses:
* Uses a somewhat unusual activation function, "GroupSort". I wonder if there are alternatives that can satisfy the conditions for monotonicity and are more similar to more commonly-used activation functions.
clarity,_quality,_novelty_and_reproducibility: Very clear overall. I could understand it with minimal difficulty. Seems novel/original, as far as I know.

Nits:
* Many of the citations are incorrectly formatted (need parentheses around them).
* "In this article" should be "In this paper"
* "Fig 3.1" should be "Fig 1"
summary_of_the_review: A very nice, clear, idea for training monotonic functions that seems relatively easy to implement and very effective.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper constructs a novel type of architecture for monotonic neural networks by adding a single residual connection to an expressive (weight-constrained) Lipschitz network. The resulting network architecture is simple in implementation and theory foundations, robust, and highly expressive. The algorithm is compared to state-of-the-art methods across various benchmarks.
strength_and_weaknesses: ### Strengths

- The paper relies on a simple (but seemingly novel) idea: by adding a residual connection to a weight-constrained Lipschitz network, one can obtain a network that inherits the expressiveness of the Lipschitz network, while the bound of the Lipschitz norm can be used to specify the strength of the residual connection to ensure monotonicity. This is simpler and is certainly more powerful than two existing ideas in the field (ensure positive weight across the nodes of a layer, or penalize the negative gradient in the samples).
- The main ideas of the work were developed based on well-founded rationales; the technical details (such as choice of the norm, or how datasets are obtained and analyzed) are carefully designed. The paper is well-written with a sufficient literature review. Visual information is intuitive.
- The problem studied in this paper is an important and of great interest to a broad audience in the field. Its impact on applied science is highlighted by the motivating example (real-time decision-making at high frequency at the Large Hadron Collider). As stated in the paper, the LHCb experiment is now using the proposed architecture for real-time data selection.
- The design of the experiments is very careful and rigorous. The results showcase that the new approach could outperform state-of-the-art methods across various benchmarks previously studied in the field.

### Weaknesses

- None noted. However, it could be argued that the main technical part of the work (technical structure, expressiveness, and universal approximation of Lipschitz networks) are all pre-existing results in the field and that the technical and theoretical contributions of the work are minimal.
clarity,_quality,_novelty_and_reproducibility: (see Strengths and Weaknesses)
summary_of_the_review: The paper addresses an important question. The approach is novel and of broad interest. The proposed structure is better designed and outperforms pre-existing approaches in constructing monotonic networks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a guaranteed monotonic architecture by correcting for the Lipschitz constant of a network. The models are very small but perform well on standard datasets. While there is a clear limitation in the lack of scalability of this approach, there is also value in having strong models for smaller problems, and therefore this paper can be accepted.

The paper talks a lot about being state-of-the-art but is missing a discussion of and empirical comparison with Sivaraman, Aishwarya, et al. "Counterexample-guided learning of monotonic neural networks." Advances in Neural Information Processing Systems 33 (2020): 11936-11948. This is something to be corrected for the final version of the paper.
justification_for_why_not_higher_score: Limited scalability
justification_for_why_not_lower_score: Is good at what it does.
Decision: Accept: poster