Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a novel neural network layer to model partial monotonicity in regression analysis. To enforce monotonicity w.r.t. a subset of input features, the authors define a hierarchical lattice structure over a unit hypercube spanning the space of monotonic features. The authors then employ an ordered set (based on the coordinates) of vertices of the lattice structure to define upper and lower bounds of the target regression function for a given set of input features. Given the upper and lower bounds corresponding to an input, the authors propose to approximate the target function by using interpolation between the bounds. They learn an auxiliary neural network with sigmoid head to output the interpolation weight given input features. The authors perform complexity analysis of the proposed method and in the numerical experiments, they demonstrate that their method is competitive or better than compared methodologies, which also includes a generic multi-layer perceptron model.
strengths_and_weaknesses: Strengths:
- Compared to previous approaches, the proposed method reduces the spatial complexity to the dimensionality of monotonic features
- The lattice layer can be trained using standard optimisation methods such as back propagation.
- Numerical results are competitive or better than compared approaches

Weaknesses:
- The approach requires prior knowledge of monotonic features and the layer cannot yield optimal results if monotonicity assumptions do not hold.
- Time and space complexity of the proposed method increases exponentially w.r.t. the lattice granularity and the number of monotonic features, hence it's application to a large number of monotonic features can become infeasible and the performance may suffer due to a coarse granularity of the lattice structure.
- Although the method is competitive to generic methods like MLP, it can be simply down to the choice of architecture used in the comparison.
questions: Have you analysed if for instance in your experiments a generic model like an MLP also learns (partially) monotonic relationships between input and target variables, even though it is not enforced by design?
limitations: n/a
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a new model architecture to represent partially monotonic functions that better scale to a larger number of inputs compared to current baselines. Monotonic functions are ubiquitous in real-world ML applications, with many examples included in the paper. The new technique is based on a reparametrization of lattice models (interpolated lookup tables). Instead of explicit parametrization of Gupta et al. [2016], the proposed method uses an auxiliary (non-monotonic) neutral network to linearly interpolate in the space of upper and lower bounds of each lattice site/vertex. The auxiliary model is thus implicitly defining the value of the vertices on the lattice while satisfying monotonicity constraints.

Note: I reviewed an earlier version of this work. The new submission is improved in several aspects, including the presentation of the method and empirical studies.

strengths_and_weaknesses: Strength:
* The lattice reparametrization idea presented in the paper is very interesting, as it makes it possible to model monotonic functions via a generic non-monotonic auxiliary function. This makes it possible to use generic off-the-shelf models on non-monotonic features (e.g. a CNN applied to an input image), while making sure the model is monotonic w.r.t. other features (e.g. quantile value in a quantile regression task).
* The proposed implicit representation of the lattice parameters forces monotonicity by construction. This avoids the need for the expensive parameter projection step during training, which is needed for explicitly parameterized lattice models.
* The method should be easy to implement in any standard ML framework.


Weaknesses:
* The calculation of vertex values in the proposed method adds extra compute compared to explicit representations when the number of features in each lattice is small. If simplex interpolation is used (Gupta et al. [2016]), the extra computation needed for the proposed method will change the overall evaluation time from O(D logD) to O(K^D) for D features used in a lattice with K vertices along each dimension. This is not a big concern since the main use case argued in the paper is when only a small number of features are monotonic, and the proposed method only needs to apply lattice interpolation to the monotonic features. However, I believe simplex interpolation could have been studied in the empirical studies at least for the baseline lattice models.
* Unlike the studies of baseline lattice models (Gupta et al. [2016], Fard et al. [2016], You et al. [2017], Wang and Gupta [2020]), empirical studies in this paper do not use a trainable calibration layer before passing the input to the lattice. Since the vertices of the lattices are fixed at equidistance locations, standard linear normalization is likely suboptimal for features with non-uniform distributions. Increasing the lattice resolution (as the paper concurs with previous work) will not solve the issue since it is adding parameters uniformly across the input domain where they might not be needed.

questions: Questions:
* The specific parametrization of the lattice described in the work results in a compound influence of the first output of the auxiliary model through the hierarchy of calculations, whereas the last output of the auxiliary model is only used in the calculation of the value of the last vertex. One would expect the gradients to flow very differently in the backward path for different outputs of the auxiliary model (and possibly issues vanishing gradient if D is large). Did the authors notice any significant effects related to this asymmetry? 

Notes and suggestions:
* The citations in the paper have incorrect formatting. The authors should choose the correct citation latex macros that would include parentheses when inlined in text.
* IMO, “Explicitly Parameterized Lattice” (EP Lattice) might be a better name for the baseline.
* Projected gradient descent used for baseline lattice models is very much a standard training algorithm, with projections supported as first class citizens in several ML libraries (e.g. TF/Keras, PyTorch). The calculation of the projection is of course non-trivial and specific to the modeling. It would be better if the authors are more explicit about this distinction in their description of the baseline training algorithms.
* Line 170: … hold for any ordering sigma: … any ordering sigma consistent with the partial ordering defined by the monotonicity constraints.
* Please check plots in a black and white print. Some line formattings are hard to tell apart at the current resolution.

limitations: IMO the authors have sufficiently discussed the limitations of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel neural net layer that improves on the TensorFlow lattice layer by being able to take advantage of standard neural net training procedures (the existing lattice layer requires a special training procedure) and also being able to handle high-dimensional inputs (whereas the existing lattice layer would run into memory issues).
strengths_and_weaknesses: I would say that overall, the technical advance is very incremental. For example, much of the same ideas are already used in survival analysis for estimating the monotonically decreasing survival probability function over time (e.g., Kvamme and Borgan 2021, although the same ideas in parameterizing the monotonic survival function appear in earlier papers, some of which are referenced within Kvamme and Borgan), where standard neural net training procedures suffice. One could perhaps view the proposed HLL method as a generalization of what is already done in survival analysis that does not need a specialized training procedure (although really I'd be surprised if the same ideas used by survival analysis researchers haven't already been used by researchers in other fields that need to enforce monotonicity). That said, I think a paper that neatly shows that there's a general idea that subsumes existing approaches is valuable.

Strengths:
- The key technical ideas are actually not that complicated although the presentation perhaps make them sound more complicated than they actually are.
- The author(s) provide sufficient empirical evidence that the proposed hierarchical lattice layer (HLL) is better than the existing TensorFlow lattice layer in the case of partially monotone regression.

Weaknesses:
- I found the presentation at times to be more complicated than it needs to be. I would suggest adding a simple running example (could be very low-dimensional) throughout the paper that already clearly shows why the proposed method clearly works and we really don't need a specialized training procedure.
- It would be helpful relating the proposed method to possibly special cases of how people have previously enforced monotonicity without resorting to specialized training procedures (I mentioned already that this is done in survival analysis but I'd assume that it's also done in other fields too).
- Minor: Please appropriately switch between using "citep" and "citet" for references (currently, the paper basically uses the equivalent of "citet" too often so that the text suddenly switches to author names in a manner that is not grammatically correct and detracts from reading the paper). As a few examples:
    - In line 19-20, I'd suggest you use "citep" so that the text says "... if the other features are equal [Potharst and Feelders, 2002]."
    - In lines 25-28, if you use "citep", you would get: "For example, they have a better regularization capability [Dugas et al., 2000; Fard et al., 2016; You et al., 2017] and better interpretability [Gupta et al., 2016], and they can be used for fair machine learning [Wang and Gupta, 2020]."

Reference:

Kvamme H, Borgan Ø. Continuous and discrete-time survival prediction with neural networks. Lifetime Data Analysis. 2021 Oct;27(4):710-36.
questions: Please address the points that I listed as weaknesses above.
limitations: Yes, the author(s) do briefly address the limitation of their approach (i.e., it doesn't handle large $m$), and I found their response to the question on potential negative societal impact in the checklist to be adequate.
review_title: null

Meta-review: metareview: Overall, the reviews about this paper are very positive. The authors spent great effort engaging in discussions and improving the paper with clarifications and additional experiments. We recommend accepting the paper.
confidence: Certain
Decision: Accept