Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary:    The manuscripts proposes a method to accelerate GP inference in situations where derivative observations are available and the number of data points N and the data dimension D tends to be large. Extending previous work on stochastic variational approximations, the central idea of the paper is to use a number p of directional derivatives as variational parameters rather than full derivatives which greatly reduces the computational footprint.

   Strengths
   + Computational methods to tackle high D, high N GP inference problems with derivative information available are not fully scalable and hence a relevant research subject.
   + The proposed approach is clear and builds on established techniques.
   + The computational benefits demonstrated in an extensive set of experiments are substantial and convincing.


   Weaknesses
   - Experimental results are without error bars.
main_review:    1) Clarity

   The paper is well written. The notation is concise and the technical content is easily accessible. The experiments are well described.

   2) Originality

   The proposal to use directional derivatives as variational parameters to render GP inference with derivatives more scalable seems novel.

   3) Significance

   Although the setting of GP regression with high N, high D and derivative information available sound like a niche application, the presented experiments show that these settings do exist and could not be tackled before. Beyond this, the scope of the method seems rather limited.

   4) Reproducibility

   The data and the code to run the experiments will be released upon acceptance. Although not available during the review process, it should be possible to reproduce the results.

   5) Empirical analysis
   
   The manuscript contains a wide range of experiments illustrating various aspects of the method on a number of data sets along with comparative results relative to the relevant baseline methods.

   6) Minor Issues and Typos

   line 197: DSKIP rather than DKIP
   line 206: learning tasks, we perform
   Table 1, caption: Hartmann rather than Hatrmann
   Figure 1, caption: ability to incorporates
   line 256: In this experiment, we show
   line 283: in Figure 4
   line 312: reasonable assumption in very high
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose a scalable inference method for GP with derivatives. Previous methods could only work for large data in low-dimensions or small data in high-dimensions. They solve the issues by using the inducing points and directional derivatives. They compare with other alternatives on multiple datasets and show that scalable GP with directional derivatives has the best performance. 


main_review: The directional derivative idea to map high-dimensional derivative kernels down to a scalar is novel and seems to work practically well. However, I'm still a bit skeptical about the choice of directions which could have a potential effect I assume. But this is not analyzed in the paper at all. The paper needs to provide better intuitions, especially in the experiment part to convince readers about that. The paper needs more edits to make it clear and easy to follow.

limitations_and_societal_impact: 1. mu_f(x_i) is not defined before or after the use in eq. 4.

2. How to construct kernel matrices in eq. 12 from eq. 11?

3. How to select the directions for v? Especially with large D. I assume mapping high-dimensional derivatives down to a scalar with v and p=1,2 would be quite tricky. 

4. I don't understand this sentence very much "as this would confound the performance improvements achieved by higher fidelity modeling by incorporating derivative information." Could you explain it? I'm confused that if the derivative information is not considered in BO, what is \nabla f in this example? 

5. Why DSKI have N/A in table 1? 

6. In sec 5.2, how many inducing points, i.e., M, are used with DSVGP? DSKI on 11606 observations of 34818 locations. What are 11606 and 348181?

7. Since DSKI claims that it could scale to high dimensions, it would still be helpful to include DSKI for applications with large D, like sec 5.3 and 5.4.

8. Flip the order of the two figures in figure 4 since stellarator is introduced before rover.

9. It's a bit confusing to arrange the experiments with the current order: 5.1 regression, 5.2 surface reconstruction, 5.3 BO, 5.4 regression, 5.5 BO, 5.6 regression without derivative information. It feels like the comparable methods switch between different groups. This makes the logic hard to follow. It might be helpful to have a table summarizing what application the experiment is and whether it's large N or large D or both.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a new method to enable Gaussian processes with derivatives to deal with large and high-dimensional datasets. Several synthetic and real-world experiments are conducted in evaluating the proposed method. 
main_review: Originality and Significance: To be honest, I am a little bit disappointed when reading the technical part and experiments. Since two key ingredients discussed in the introduction:
1. heteroscedastic noise
2. principal to select $p$

I am expecting the authors can address 1. what is the advantages of heteroscedastic noise rather than i.i.d noise? 2. how will the model perform with different $p$. 

It is natural that selecting a small number of $p$ can reduce the time complexity, however, the authors did not really mention how to determine $p$ for different tasks? PCA, t-sne, or even gplvm can do this dimensionality reduction right? I am expecting more discussion relating to derivative and dimensionality reduction. This should be the golden apple of this paper.

Moreover, Sec 4.1 seems to be simple adaption of VSGP to VSGP with derivatives, is there any difficulty in adopting VI? Any other benifits in adopting VI other than reducing time complexity?

The two aspects above are not extensively discussed and evaluated in the paper which makes the contribution incremental. 

Quality & Clarity: The writing of this paper is really smooth and easy to follow. I really appreciate the efforts the authors put in polishing this paper. 
limitations_and_societal_impact: refer to the detailed comments above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In a spirit similar to the SVGPs which sparsify the training data, the authors propose to augment the SVGP framework with derivative information by sparsifying the derivative data with *inducing directional derivatives*. By virtue of these modeling choices, the resulting algorithm scales favorably independent of data size as in SVGPs, and now additionaly also w.r.t. input dimension. Empirically, the authors show much better scalability in both high-N and high-D than existing work.
main_review: Overall, the paper checks all the boxes in terms of writing and narrative framing, presenting a clean linear story. The contribution is important, given that it brings a general approach to augment variational GP models with derivative information via PPGPR to allow heteroscedasticity. The choice of using directional derivatives is natural in hindsight, making this contribution even more important.

One setting which seems to be missing is the choice of using a fixed set of inducing directions for all inducing inducing points, instead of picking unique directions for every inducing point. Intuitively, is this choice the natural first step? Would the flexibility of the model suffer if we started with directions shared across all inducing points?

Section 5.6 on using DPPGPR even when no derivative information is available is interesting, but not entirely unclear. How does the model in Eq. $(6)$ change if we do not have derivative information. This does not appear to have been spelled out clearly, or isn't obvious.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper addresses the problem of scalable inference in Gaussian process (GP) regression with derivative information for the general case where both the number of observations (N) and the dimensionality (D) are large. While previous work has addressed the large-N-low-D and low-N-large-D regimes independently, the paper proposes the use of inducing directional derivatives (which play a similar role to that of the inducing variables in standard variationally sparse GP models) rather than full derivatives to derive a scalable variational inference algorithm. A comprehensive set of experiments (on synthetic data, implicit surface reconstruction, training graph convolutional networks with Bayesian optimization, large-scale regression with derivative information, Rover trajectory planning and standard GP regression on UCI datasets) show the benefits of the proposed approach with respect to previous methods. 

I believe the contribution of this paper to be significant to the NeurIPS/GP community as it develops a new method for settings where previous work cannot be applied. The reviewers raised several concerns regarding the choice of inducing directions and the use of different directions per inducing point. I believe the authors have addressed these successfully. A limitation of this work is the very small number of directional derivatives (p) used in the experiments (p=1, 2). As the complexity of the proposed algorithm scales as (M^3 p^3), exploring larger p values will necessarily lead to fewer inducing variables (M). 

Decision: Accept (Poster)