Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work develops an algorithm for GP inference with constraints through a novel analytical update rule for an additive variant of GP inference. An active set technique for which model points to retain is developed based on Euclidean subspace projections. Numerical results illuminate the merits of the proposed approach.


strengths_and_weaknesses: Strengths:

The algorithm appears novel to my knowledge, and addresses a fundamental and open problem in additive reparameterizations of Bayesian linear regression with Gaussian processes.

The 2d example, as well as the extended numerical evaluations, provide substantive evidence that the proposed technique works well In practice for imposing constraints in this setting.


Weaknesses:

The authors should contrast their approach to imposing constraints through Bayesian regression with frequentist analogues. In particular, through a well-known link to kernel ridge regression, one can impose convex or linear constraints, which are very much related to the setting considered here. See, for instance:

A. Koppel, K. Zhang, H. Zhu, and T. M. Basar. ”Projected Stochastic Primal-Dual Method
for Constrained Online Learning with Kernels” in IEEE Trans. Signal Processing, May. 2019.

Marteau-Ferey, U., Bach, F., & Rudi, A. (2020). Non-parametric models for non-negative functions. Advances in neural information processing systems, 33, 12816-12826.

And follow-on works.

The authors also consider a way to deal with complexity bottleneck of GP posterior inference through a sequential dimensionality reduction approach with the squared-norm. Substantial complexity reduction of computing an inference is achieved through this active set approach. What I wonder is how is this related or different from a large history on active set techniques for complexity reduction of GP inference. In particular, recent works have  established specific convergence guarantees as a function of model complexity. How does the proposed technique contrast with these performance certificates? Is the squared-norm approach sufficient due to the additive linear nature, but beyond this situation one should consider metrics in distribution space (such as Hellinger or Wasserstein?)

McAllester DA (1999) "Pac-bayesian model averaging." In: Proceedings of the twelfth annual conference on Computational learning theory, pp 164–170

Burt D, Rasmussen CE, Van Der Wilk M (2019) "Rates of convergence for sparse variational gaussian process regression." In: International Conference on Machine Learning, pp 862–871

A. Koppel, H. Pradhan, and K. Rajawat. “Consistent Online Gaussian Process Regression
Without the Sample Complexity Bottleneck,” in Statistics and Computing, Springer, Sept.
2021

Such a contrast is not carefully addressed in this manuscript. Specifically, see Table 1 in the last of the preceding list of references.

The main theoretical results are of a nature of providing analytical expressions for the update rules. While this is credible and interesting, additional justification in the context of what would/would not guarantee constraint satisfaction is missing.

The numerical results do not really consider strong benchmarks for evaluating the tradeoffs in the constraint satisfaction/model fitness achieved by this method as compared with some alternatives. Moreover, it is unclear how does the active set approach appropriately capture the right statistical properties of the fully dense GP. 



Overall opinion: 

The algorithm is novel, and its update expressions combined with the active set approach to dimensionality reduction, ensures its efficiency. However, these aspects are only addressed heuristically, and no performance certificates are provided. Moreover, numerically no strong baseline is considered to determine whether the proposed technique actually achieves state of the art performance. For these reasons, the work is below the bar.


questions: See response to previous field.
limitations: See response to previous field.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an additive Gaussian process with support for monotonicity constraints and scalable to high dimensions. In particular, a sequential dimension reduction algorithm is presented which can identify the active input dimensions. The author also provided illustrative results on real-life data.


strengths_and_weaknesses: [Strengths]
1. The paper is well-written. The proposed GP framework is an interesting generalization of additive GPs.
2. The technical derivation looks solid, and the kernel of the proposed GP has a simple analytical form.
3. The authors also validated the method on a Vienne river flood data where the proposed MaxMod algorithm correctly activated the relevant input variables.

[Weaknesses]
1. It's not clear how the kernel parameters are estimated. In Eq. (11), the objective is given by a constrained quadratic optimization. However, this objective could be nonconvex in terms of the kernel parameters. Are there efficient solvers?
2. The proposed GP framework seems to rely on particular basis functions (Eq. (4)) which have domain [0, 1]. Does this domain impose limitations on the applicability of the GP framework? What are the desiderata for these basis functions?

questions: 1. Are the kernel parameters inferred jointly with other parameters in Eq. (11)?
2. What are the desiderata for the basis functions Eq. (4)?

limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors consider the problem of regression tasks with an additive Gaussian process prior and linear inequality constraint. The authors propose a finite dimensional approximation to the Gaussian process as a linear combination of triangular basis functions with Gaussian weights. The weights are then estimated via solving a quadratic program (for MAP estimation) or approximately sampled with HMC to handle the inequality constraints. Additionally, the authors consider the problem of variable selection in this model, and propose a forward selection method based on the difference between the posterior mode with and without the inclusion of a particular variable (or knot). The authors demonstrate their method for inference and variable selection on synthetic data, that is generated to be additive and monotonic along certain dimensions. Finally, the authors apply the method to a flood dataset generated by a utility company via simulation.
strengths_and_weaknesses: ### Strengths
- The scope of the paper is well-defined and of interest to those modeling with Gaussian processes. Incorporating prior knowledge such as inequality constraints into these models seems to be an important task. 
- The forward selection algorithm (MaxMod) for selecting which dimensions to include seems appealing for high-dimensional datasets, and the experiments seem to suggest that it can be effective. 
- The writing is generally quite clear.

### Weaknesses
- I am concerned that the contribution regarding extension of existing work to handle high-dimensional, additive models is over-claimed. It looks to me like the method considered is very similar to the references [5,6], applying the prior works approach to each dimension (separately).If there is a major distinction that I have missed between the approaches, I would appreciate if the authors could highlight this more explicitly. Otherwise, I think this contribution should be contextualized a bit more clearly relative to existing work, particularly in section 3 where it isn't entirely clear what is new and what is a based directly on existing methods.
questions: 1. (This is a comment) I think it would be useful to include a figure showing a.) the hat basis functions, b.) a sample from a 1-dimensional Gaussian process prior with a given kernel (e.g. Matérn 5/2) as well as the sample (using the same realization of Gaussian noise) modified to the finite dimensional model (eqn 5) with a handful of knots.

2. (Lines 96-97) You refer to a function as being monotonic on $[0,1]^d$. I don’t think this is a well-defined notion (since $[0,1]^d$ isn’t totally ordered). What do you mean by this? I assume you mean that restricted to each component it is monotonic, in which case the first equivalence you state is really just a definition. This should be made clear. 

3. Equation 8 seems more restrictive than the original formulation in which $\mathcal{C}_i$ need only be convex. Are they somehow equivalent? If not, it would be best to initially introduce $\mathcal{C}_i$ as a set defined by finitely many linear inequality to avoid confusion.

4. Notationally, the first equality in (eqn 10) seems to suggest that the MAP estimate can be computed by computing a MAP estimate along each dimension separately and combining these. Is this what is intended? I don’t see why this would be true (and in equation 11 this does not seem to be the case). If it is, could you please give some explanation as to why? If not, consider changing the notation slightly.

5. (Minor, line 116) “et” -> “and”

6. (Minor) Is the information benefit directly tied to ideas from information theory? If not, it may be worth considering renaming.

7. (Minor, Algorithm 1, 5) “D” -> “d”

8. In the flood dataset experiments, you selected your training set via finding a subset as close as possible to a LHD (250-251). Would findings have differed if you had instead selected training data uniformly at random? Similarly, what if $n$ were somewhat larger than $2d$? In particular, is it important to the practical application of the method that the covariate design be something like a LHD? I didn’t see where an assumption like this was made in the discussion of the method, but it seems all of the experiments used this. 

9. How robust is the proposed MaxMod algorithm in the presence of model mis-specification? For example, if the function being modelled has non-additive components, can this lead to issues with variable selection?
limitations: There is some discussion of limitations in the paper. I think it would be nice to see a synthetic experiment where the assumptions of the model are not precisely true (e.g. the function used to generate the data has some, perhaps small, additive component). I expect most real-world data is not exactly additive, and it would be nice to see how robust the proposed method is to small mis-specifications.
review_title: null

Meta-review: metareview: This paper deals with the problem of regression with an additive Gaussian process prior and a linear inequality constraint. A finite-dimensional approximation is proposed to the Gaussian process in terms of a linear combination of triangular basis functions with Gaussian weights. The weights are then estimated by solving a quadratic program or approximately sampled to handle the inequality constraints. Additionally, the authors consider the problem of variable selection and propose a forward selection method based on the difference between the posterior mode with and without the inclusion of a particular variable. The reviews were mixed, but are leaning towards acceptance.
confidence: Less certain
Decision: Accept