Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a computationally efficient way to train the GP-LVM based on a connection between marginal likelihood and cross-validation in Fong and Holmes (2020). To make the estimation practical, stochastic estimation is used for sampling the size of the hold-out set and also the size of the active set. Amortization is also used for approximating $p(z|x)$. In the experiments, the performance of the method is compared with GP-LVM and VAE. 
strengths_and_weaknesses: Strengths:

- Applying the idea of estimating MLL with cross-validation scores to GP-LVM is interesting.
- Several ablation studies on different active set sizes.

Weaknesses:

- It is not clear how different hold-out set sizes (R) will affect the proposed estimation. Also, I have difficulty understanding what “by first uniformly sampling R” means. Does it mean first pick R and then uniformly sample $r \in [1:R]$? I could not infer this from the pseudo-algorithm or experiments as well. If the sampling is by uniformly sampling $r \in [1:R]$, it seems necessary to have more discussion about whether the sampling is of high variance and if there are ways to reduce the variance.
- The presentation of the experimental results can be improved. Some important experiment details are deferred to the appendix, such as what kernel, NN architecture is used. They serve as very important information to understand the results. Also, not sure why RMSE/MAE is used as a metric as the MNIST or CIFAR10 are classification tasks. For evaluating the quality of the representation learned, it is not clear why 2 is selected as the number of dimensions. It might be better to consider trying a range of different dimensions.
- There are two major components in the proposed method: the SAS approximation and amortization. It would be better if there is some ablation study on both SAS and amortization. For example, by comparing GP-LVM-SAS-Amortization with GP-LVM-InducingPoint-Amortization and GP-LVM-SAS with GP-LVM-InudcingPoint without amortization.
- It seems that the baseline methods are only compared with the proposed method in terms of representation quality. There is no comparison on NLPD/RMSE/MAE. And it would be better to include some regression tasks as well.
questions: 1. How is R picked? Also, will the uniform sampling of r results in high variance of the estimate?
2. Why does Bayesian GP-LVM perform much worse than the proposed method or VAE? Some explanation and ablation studies would be helpful.
limitations: N.A.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a new approach to train Gaussian process latent variable models (GP-LVMs). The idea relies on the (previously shown) connection between cross-validation (CV) and marginal likelihood (ML). More specifically, the authors interpret the leave-R-out CV objective from an optimization perspective, which allows for stochastic and scalable optimization of the ML objective of vanilla and Bayesian GP-LVM formulations. The experiments demonstrate that the framework leads to more compact latent representations compared to previous formulations. On MNIST, the deterministic variant of the framework is shown to match the exact ML with a certain hyperparameter choice (batch and active set size).
strengths_and_weaknesses: Overall, I liked the paper as it nicely brings theoretical and practical aspects together. My major comments concern the derivation/notation and experiments, which strangely both escalate and diminish my score.

Strengths:
- The framework has solid theoretical foundations. I find the connection with CV important as it distinguishes the proposed framework from heuristics-based optimization objectives.
- The results are impressive. The improvement over existing deterministic and Bayesian GP-LVMs is very clear. Looking at the latent nearest neighbor classification experiment, one can even argue that the learned latent space is more structured than that of VAE.

Weaknesses:
- The presentation can be improved. I list several points in the subsequent "Question" section. Most importantly, I do not understand
  - how to obtain (5) from (4), and
  - how $S_{PCV}$ in (5) has $x_A$ on the left hand side of the conditioning while (3) has $x_A$ on the right.
- The claims in lines 29-34 are not well-supported. It would be much better to list the problems of training with inducing points (with references) instead of quotations.
- The experiments are slightly worrying (please see the next "Question" section again)

I would be glad to increase my score if my concerns are addressed (even verbally).

**UPDATE:** After the author response, I increase my score to 5.


questions: Presentation/clarity/notation:
- What is $K_{NN}$ in (1)?
- Shouldn't $S_{CV}(x|R)$ in (3) be also conditioned on $z$?
- How about including the formal definitions of $S_{PCV}$ and $S_{CCV}$ in the main paper instead of supplements?
- $r$ appears on the middle term in (4) but only $R$ appears on the right hand side. 
- The first equation in line 114 involves $R$ on the left but not on the right. If I understood correctly, $|\mathcal{A}|=N-R$ (hence $R$ implicitly appears on the right. In that case, I would suggest using modifying the notation a bit to reflect this.
- Are $\mathcal{A}$ and $\mathcal{A}_p$ used interchangeably?

Experiments:
- The loss curves in Figure 2 oscillate a lot.
- The MNIST latent space with Bayesian SAS is missing.
- Only $2D$ latent spaces are considered, which significantly restricts the generalizability of the findings.
- NLPD consistently increases with active set size (for which I would use the shorthand $|\mathcal{A}|$ instead of $A$). Is this just the opposite of what we would expect?

Misc:
- Although the proposed framework seems suitable for GP models in general, the authors particularly focus on GP-LVMs. I wonder why authors restrict themselves to GP-LVM. It would also be nice to see how the framework performs on a standard GP regression task with simple noisy inputs and outputs. 
limitations: Everything sounds good.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors investigate a novel stochastic gradient ascent method for fitting Gaussian Process Latent Variable Models. In practice, if I understood correctly, the method samples a "minibatch" from the full data to build a GP, and measures posterior likelihood on the remaining hold out data. This is justified by recent mathematical results that show the full log marginal likelihood ($O(n^3)$ cost) is equivalent to the marginal likelihood over subsets ($O(b^3)$ with batchsize $b$) of all subset sizes and ( $O(n-b)$, linear cost) summations over terms excluded from the subset. This alternative marginal likelihood evaluation is a a sum and an expectation and is amenable to stochastic Monte-Carlo approximation, and therefore unbiased stochastic gradient ascent.

The authors apply this trick to both standard and Bayesian GP-LVMs implemented both with and without amortization over a range of datasets and demonstrate the benefits.
strengths_and_weaknesses: **Strengths**
- I felt the paper was very well written, providing a clear story of GP-LVMs and their milestones over the years and with clear interpretation
- the result of Fong and Holmes 2020 appears to show the marginal likelihood can be (stochastically) approximated using just partitions of data, which is very impactful for GPs that require cubic cost effectively reducing cost to smaller cubes and linear costs.
- I felt the application to GP-LVMs and Bayesian prior GP-LVMs with and without amortization covered a wide extensive range of empirical use cases and provides value to the community.

**Weaknesses**
- I feel that evaluating marginal likelihood on a stochastically subsampled data to reduce the cost of GP training is a fairly common trick, but obviously biased. Admittedly I cannot seem to find papers using it exactly, a google search for "Gaussian Process minibatch" returned some results [1, 2].
- Hence the only _practical_ novelty (over naive subsampling) in this work is 
    - (1) the first term of Equations 6 and Equation 8 and
    - (2) the batch size is random (but it doesn't need to be? See nitpick)
- The rest of the paper seems to be technical background on GP-LVMS and Fong and Holmes (2020).
- Can the authors include a naive random subsampling baseline? Simply SAS methods without the First terms of equation 6? This would serve as an ablation to show the impact of the theoretical result.

**Nit Picks**
- Equation 4: $R$ is said to be averaged over $R$ yet it is a constant on the right term but not in the centre or left terms, this is explained in the Appendix "Additionally, Eq. (3) holds for every size of the hold-out data R ∈ [1, 2, · · · , N]" and would make Eqn 4 easier less confusing to me.
- Equation 3: for me, seeing the bold $x_n$ automatically made me think it was a set of correlated points requiring whose probability would have led to $n^3$ matrix inversion.
- Appendix Equation 4: I was very curious to know the intermediate steps to deriving $S_{PCV}(x, R)$ and I couldn't find the answer during a (very) quick skim through Fong and Holmes (2020).


[1] https://arxiv.org/abs/2111.10461

[2] https://nipunbatra.github.io/blog/ml/2021/09/03/param-learning-sgd.html
questions: - in the Standard GP-LVM case, can the authors confirm that the only practical novelty (over naïve subsampling data) is the random batch size and additional term in Equation (6) and (8)? (If so, this can be portrayed as a strength)
- Can the authors include such a naive subsampling data baseline forming an ablation study?
limitations: The authors clearly discuss the Gaussian likelihood assumption. No more unaccounted limitations come to my mind.
review_title: null

Meta-review: metareview: This paper proposes a stochastic algorithm to make inference in Gaussian process latent variables models more efficient. The key idea is to exploit an equivalence between the marginal likelihood and the leave-R-out cross validation score (due to Fong and Holmes) to reduce the original cubic complexity in the full dimensional space to cubic complexity on subsampled data plus the (linear) complexity of combining these estimates. Some reviewers felt the fundamental idea of the paper might be extendable to other GP models, and there were concerns about the clarity in several parts. (Though all agreed this was improved in the revision, further improvement remains possible.) There were also some concerns that the proposed idea presented a relatively small methodological leap. However, in the end the consensus was that the paper provides a valuable contribution linking intuitions about subsampling to formal guarantees and for this reason makes a valuable contribution.
confidence: Certain
Decision: Accept