Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary
--------
The paper addresses Gaussian process regression (GPR) over 'positive' and 'negative' data points, such that the regression model fit the former and avoid the latter. The 'negative' data points in this work are each represented by individually distributed Gaussian random variables, with known parameters (mean vector and variance).
The authors propose a novel GPR loss (6) for optimizing GP model parameters $\theta$, striking a compromise between the regular GP loss and the KL divergence between the predictive GP distribution and the set of 'negative' data points. The approach is novel and an interesting take on the problem, and it has several beneficial properties which are investigated empirically.

Strong points
-------------
1. Compared to previous work on GPR with 'positive' and 'negative' data points, the proposed solution is elegant in its simplicity, and scalable for predictions (since the 'negative' data is only used during kernel parameter optimization). I especially like the scalability aspect, since it would allow the use of a otherwise prohibitively large amount of 'negative' examples. This is useful in navigation, for example in contextes where the boundaries of static obstacles are commonly represented by potential fields.
2. The experiments are fairly extensive and demonstrate several beneficial properties of the proposed method, such as fast learning convergence and compatibility with sparse GP methods.

Weak points
-------------
1. The problem of GPR with both 'positive' and 'negative' data points (and in the context of robot navigation) has been investigated before [1], where a non-stationary kernel function is proposed. The approach in [1] should be addressed, and compared with, at least qualitatively.
2. The experiments are not representing any situation where the distinction of 'positive' and 'negative' data points naturally occur. This makes it hard to assess how well the proposed method works for real problems of the kind the paper intends to address.
3. The technical clarity is varied:
     1. Equation (5) is incorrect? The two multivariate Gaussian distributions have different dimension (n vs m), and they do not share the same support ($X$ and $\bar X$ can differ). Maybe it should be "$p(y|\theta, \bar X)$". But is this then entirely compatible with the prior sentence: $\theta$ is in essence the "GP regression model learned from the positive data pairs" and yet it is stated that you maximize KL divergence between the GP regression model and the positive data pairs with respect to $\theta$? If $\theta$ is changed, then the first distribution in the KL divergence is no longer the GP model learned from the positive data points. I suggest that this section is made clearer and more precise. Equation (6) on the other hand corresponds to the stated proposal.
   2. It is not obvious to me how you make use of the re-parameterization trick proposed by (Kingma & Welling, 2013).
   3. Why do you not minimize the loss (6) directly, instead of first minimizing NLL and then maximizing KL at each iteration (Algorithm 1)?
   4. What value does $\lambda$ and $\sigma_{neg}$ have in the experiments?
4. The 'Random shuffling technique' is not at all clear to me. 
   1. How can you get a set of *valid* data pairs (i.e. $x_i$ and $y_i$) by shuffling the labels ("$y_i$")? 
   2. If randomly create pseudo-negative data points, especially using existing data points (that are either designated 'negative' or shuffled in some way to produce new input-output pairs as 'negative'), then isn't it likely given real data sets that these data points will be in conflict (e.g. overlap) with other 'positive' data points? If so, what do you mean with that these are *valid* and what does this mean for the experiment (more details on the limitations of the experiment and interpretations of the results should be present in the paper)?

Reason for score
----------------
Although an interesting take on the problem, the paper in its current form is lacking technical clarity, comparisons to directly related works, and the evaluation is weak in that it does not consider actual instances of the addressed problem in the empirical evaluation.

Minor comments
--------------
"to model the uncertainty by providing the predictive variance" -> "to model [target] uncertainty by providing predictive variance"

"in both the positive and negative data pair." -> "in both the positive and negative data pair set."

Page 4: "$P(y|\theta, X)$" -> "$p(y|\theta, X)$"

"loglikelihood" -> "log likelihood"

--------------

[1] S. Choi, E. Kim, K. Lee and S. Oh, "Leveraged non-stationary Gaussian process regression for autonomous robot navigation," 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, 2015, pp. 473-478, doi: 10.1109/ICRA.2015.7139222.

Edit: Markdown problem with nested lists

Edit: Upgraded rating from 4 to 5

-----
Thank you for addressing most of my concerns.

I have increased my rating, but the paper still needs some work in my opinion. More specifically,
* a more thorough comparison with [1], e.g. by including the quantitative comparison you were working on.
* an improved representative experiment from the problem domain. The newly added one is a bit simplistic and not motivated enough. Although stated to be 2D, it is (as far as I can tell) indistinguishable from a 1D regression task? How do you handle the 2D output in GPR? How are the data points generated that are used as training data, and what could motivate such a generation of data in a plausibly real setting?
review_title: Novel approach, but not a new problem in GP regression.

final_score: rating: 3: Clear rejection
review_content: Summary
--

This paper in concerned with Gaussian process regression under constraints that aim to discourage the model from learning certain values (negative constraints). These are called negative data pairs, and the authors propose an extension to the standard GP methodology to incorporate these constraints in the model. This is done by iteratively training a standard GP and maximising the KL between the GP and blobs of the negative data pairs.

The problem tackled in the paper is of relevance in fields such as spatial statistics and robotics, with possible applications also in more general ML tasks. The paper could be of interest to a narrow audience at ICLR. The presentation is rather clear and the main idea is easy to follow.


Concerns
--

1. My main concern is originality and novelty of the approach presented in the paper. The authors tackle an important practical problem (or limitation), but the presented contribution alone is small. The proposed approach for including negative data pairs is straightforward, and something that one might find in an application paper (where the main interest is in solving a task related to an application). 

2. Limitations. The limitations of the approach should have been discussed in detail. Questions related to uni- vs. multimodality (splitting), bias influenced by the parameter \lambda, behaviour in multi-output (2d, like Fig. 1), would perhaps best have been illustrated with suitable simulated test cases and included as figures (including random samples from the GP posterior). Some of the issues are visible in Fig. 2, but these are not discussed in detail.

3. Experiments. The experiments do not appear convincing. Rather than artificially changing standard test data sets to fit your task setting, it would have been more interesting to see actual problems, where the model class would have been beneficial.

4. Presentation. Even if the method itself is easy to understand from how it is presented, the paper would deserve improvements in the presentation. As a practical suggestion: Background material could be presented more concisely, the methods section refined to be more concise, and the additional space used for improving experiments and discussion.

5. From how Alg. 1 now reads, I would interpret \lambda not to have any effect on the training. This should not be the case. Alg. 1 does not agree with the objective in Eq. (6).

6. Minor: Typo in Eq. (4).
review_title: Learning what not to model

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper incorporates information of obstacles to avoid (e.g robot navigation trajectory in the room where the robot has to avoid items such as furniture) into Gaussian process regression fit. They call the obstacles, negative datapairs and the rest of data, positive datapairs. The aim is to have a GP where the probability of passing through the negative datapairs is low. The proposed method is called the Gaussian process with negative constraints (GP-NC). 

To be able to fit the GP regression to positive datapairs and avoid negative datapairs, they maximize the KL-divergence between the distributions of GP learned from positive datpairs $(p(y|\theta. X))$ and negative datapiars $(q(\hat{y}|\hat{X}))$ which will have a bound between $[0, \inf]$.
For being able to maximize the KL-divergence with the marginal log-likelihood they change the scale of KL-divergence to the log scale and a parameter $\lambda$. $\lambda$ is a tradeoff between curve fitting and avoidance of the negative datapoints. KL term has an analytical solution since both distributions are Gaussians.
They compare their method against SVGP (Hensman et al., 2013) and PPGPR (Jankowiak et al., 2019) and the exact GP.

Comments and questions: 
The paper is well-written and easy to follow. This paper is also technically sound and to the best of my knowledge is novel and relevant to the community.

In figure two you have mentioned you sampled the inducing inputs randomly from the whole range of training inputs. Did you choose the same inducing inputs for the SVGP and SVGP-NC? 

In the error calculation, standardized mean squared error (SMSE) could be a better choice than RMSR since the former incorporates variance information as well.

In the experiment section, all negative datapairs are synthetically made. I was wondering if you could apply your method to a dataset with real negative datapiars (like a robot in the room avoiding obstacles)? 

In figure 3 (3DRoad) the approximate results are better than the exact GP. Is this because of the scale of the data and not being able to converge?

Miscellaneous comments:
In Table 1 row 2 should be wine quality - white

################### After the rebuttal ################

I thank the authors for addressing the issues that were raised. The paper is indeed addressing a very practical issue in the ML community.
After reading other reviewers' comments and concerns I decreased my score. 

review_title: Incorporating negative constraints into Gaussian process regression

final_score: rating: 3: Clear rejection
review_content: SUMMARY 

This paper introduces GP-NC, a new methodology that allows Gaussian processes to stay far from certain data points (referred to as negative datapairs). These negative datapairs complement the standard training points to be fitted. The authors show enhanced performance when using this technique with standard GPs, SVGP and PPGPR.

###################################################

PROS

1) I am not aware of previous works in the GP literature dealing with this scenario, i.e. data pairs to be avoided during training.

####################################################

CONS

1) I think this work is not sufficiently motivated. The authors claim that the brand-new negative datapairs can be useful for modelling navigation problems with GPs. However, this type of problem is not addressed in the experimental section. I would expect to see how the proposed method compares to the previous approaches that have modelled the navigation problem with GPs. 

2) In fact, due to the lack of real problems that require the presented technique, in the experimental section they just simulate some negative datapairs (following the so-called random shuffling technique). Moreover, this simulation technique is pretty poor in my view, since different inputs could have very similar outputs, which may lead to a simulated datapair almost identical to a standard training datapair. 

3) Some arguments are too vague/ambiguous. For instance, the equivalence between the scale of magnitude of both terms in eq. (5). It is stated that, by using the so-called logarith trick, both terms have the same "scale of magnitude". How is that defined? and how can the equivalence be proved? Or the sentence that says "A Gaussian process is a Bayesian non-parametric approach that fits Gaussian distributions to functions". 

4) Low quality writing in general. I have come across many typos and sentences that are not well written. See e.g.:
* caption of Fig. 1 "is been given".
* after eq. (5) "minimize",
* "Hewing et al. (2020) are some of the works using sampling based techniques for trajectory prediction"
* "We trained a sparse SVGP model to regress a curve on the using the classical GP framework and the one with negative constraints GP-NC"
* Caption of Figure 4: it says five datasets, but there are six
* Many references are missing the journal/conference/arxiv reference. Format not homogeneous across references.


############################################

AFTER THE REBUTTAL

I have decided to keep my score unchanged. The authors include a new "road navigation experiment" in Section 5.2. However, I have several concerns about it:
* This is a synthetic experiment that the authors have created themselves. Therefore, it does not address my concern on seeing actual real-world problems that can be solved with this approach.
* The results of GP-NC (the proposed approach) and GP are very similar. I don't really see that the predictive mean of GP-NC avoids the "negative samples" more than the standard GP.
* In fact, the design of this experiment is not clear to me. They say at the beginning that they use the present location ($x,y$) to predict the next location ($\hat x, \hat y)$. How do they model these two outputs? How do they obtain the plotted error bars for this model? This is not sufficiently explained. My impression is that they are just fitting a standard 1-dimensional GP being the input the x-dimension and the output the y-dimension. 
review_title: Not sufficiently motivated

Meta-review: This paper is very pleasant to read. The reviewers also like the key idea discussed and find the targeted application interesting and practical. However, after reading the indeed interesting motivation, all four reviewers expected to see more from the evaluation section, including more challenging and realistic set-ups and clearer gains over standard methods. The reviewers also discuss how both the navigation problem as well as the GP constraint problem have been tackled in the past, often in combination (e.g. reference [1] by R1). Therefore, it would be needed to see additional experimental evaluation in line with those previous works.
Decision: Reject