Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a novel GP-based model for few-shot regression. The model makes GPs more flexible by adding Continuous Normalizing Flow to the output, specifically an ODE-based mapping. Moreover, this mapping is contextualized by making it input-dependent, which adds more expressivity to the predictive posterior. This flexibility makes the model better suited for modelling complex distributions (e.g. multimodality cases) and, as shown experimentally, is beneficial for few-shot regression, where tasks might differ (domain shift).

The proposed model is experimentally shown to perform better than the main competitor, Deep Kernel Transfer (DKT), on a variety of few-shot regression datasets.

Overall, this is a nicely written paper with an interesting novel method and good experimental results. Some parts of the model, however, are not given enough explanation and testing. 
main_review: ### Strengths:

The prosed methodology is solid. Bayesian non-parametrics, and GP specifically, is very suitable for tackling the task of few-shot regression. Its extension with CNF for more flexibility in the predictive posterior is a very sensible thing to do. To the best of my knowledge, the math is correct.

The method is well-tested for the given task of few-shot regression on 5 datasets.

The paper is well written and clear.

### Weaknesses:

The paper attempts to do two things at once: propose a new model and apply it for a specific task of few-shot regression. Because of this, the model is not described in enough detail. Invertible ODE-based mapping is not described at all. I would like to see some synthetic experiments comparing the proposed model to a GP on a standard regression task, before moving on to few-shot regression experiments.

The two main problems of standard GPs wrt the few-shot learning task, that this work addresses, are limited flexibility of the posterior distribution and the assumption of high similarity between tasks. While the benefit of NGGP in terms of modelling more complex distributions is clearly explained and shown (e.g. the multimodal Power Dataset experiment), the other main claim is not well supported. The authors point that "GPs assume similarity between subsequent tasks" and claim that the proposed method (NGGP) is "adaptive to dissimilarities and domain shifts". However, they do not explain how exactly the proposed method is supposed to adapt to domain shifts. Explaining this as "contextualization provided by the ODE-based mapping" is way too vague, especially given that ODE based mapping is not discussed much at all. Even though the experiments show good performance, it could be attributed to the more flexible posterior. This adaptation claim definitely should be supported by a detailed explanation.



### Other comments and typos:

A flow transformation in the model is applied to the noisy GP outputs. I would like to see some more discussion about this design choice and its potential limitations.

66: "overweights" -> "weights" ?

120: in eq 5 sum is over n, but in the det you put k.

144: ODE-based mapping should be described.

Fig 4: why do you plot different kernels? is it the best-performing ones?

Fig 4 (a) the due to the vertical scaling, we pretty much can not see the predictive mean (red) in the first two plots on the bottom.
limitations_and_societal_impact: Limitations of the proposed method are described in the paper. Negative societal impact is discussed appropriately given the general theoretical nature of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose to make the output distributions of Gaussian processes more flexible by combining them with continuous normalizing flows. They then show that in the application of few-shot learning, this can have benefits over standard GPs without the flows.

UPDATE: Score raised due to clarifications in the author response.
main_review: Strengths:
- The idea seems to be novel and it is intuitive that NFs could help improve the flexibility of the distribution.
- The empirical results are promising.

Weaknesses:
- The description of the method is somewhat unclear and it is hard to understand all the design choices.
- Some natural baselines and important related work seem to be missing.

Major comments:
- The lack of flexibility of standard GPs is not a new observation as has been approached in the past, possibly most famously by the deep GP [1]. These models have recently become a mainstream tool with easily usable frameworks [e.g., 2], so that it would seems like a natural baseline to compare against.
- Generally, a lot of related work seems to be missed by this paper. For instance, meta-learning kernels for GPs for few-shot tasks has already been done by [3] and then later also by [4,5,6]. These should probably be mentioned and it should be discussed how the proposed method compares against them.
- The paper proposes to use CNFs, but these require solving a complex-looking integral (e.g., Eq. 9). It should be discussed how tractable this integral is or how it is approximated in practice. Moreover, it seems like an easier choice would be standard NFs, so it should be discussed why CNFs are assumed to be better here. Possibly, one should also directly compare against a model with a standard NF as an ablation study.
- In l. 257ff it is claimed that the proposed GP methods are less prone to memorization. How does this compare to the results in [4], where DKT seems to memorize as well? Could the regularization proposed in [4] be combined with the proposed model?

Minor comments:
- In l. 104 it is said that every kernel can be described by a feature space parameterized by a neural network, but this is trivially not true. For instance, for RBF kernels, the RKHS is famously infinite-dimensional, such that one would need an NN with infinite width to represent it. So at most, NNs can represent finite-dimensional RKHSs in practice. This limitation should be made more clear.
- l. 151 with GP -> with a GP
- l. 152 use invertible mapping -> use an invertible mapping
- l. 161 the "marginal log-probability" is more commonly called "log marginal likelihood" or "log evidence"
- Eq. (8): should it be $\mathbf{z}$ instead of $\mathbf{y}$?
- In the tables, it would be more helpful to also bolden the fonts of the entries where the error bars overlap with the best entry.

[1] Damianou & Lawrence 2012, https://arxiv.org/abs/1211.0358

[2] Dutordoir et al. 2021, https://arxiv.org/abs/2104.05674

[3] Fortuin et al. 2019, https://arxiv.org/abs/1901.08098

[4] Rothfuss et al. 2020, https://arxiv.org/abs/2002.05551

[5] Venkitaraman et al. 2020, https://arxiv.org/abs/2006.07212

[6] Titsias et al. 2020, https://arxiv.org/abs/2009.03228
limitations_and_societal_impact: The limitations of the method are hard to assess, mostly because the choice of CNFs over NFs or any other flexible distribution family is not well motivated and because (theoretical and empirical) comparisons to many relevant related methods are missing. This should be addressed.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Authors propose a method for incorporating continuous normalising flows (CNFs) into the learning mechanism of Gaussian processes (GP). The work is motivated by the scenario of few-shot learning, where they need more flexibility to model subsequent tasks that might differ between them. Experiments show that training such models is possible.
main_review: **Originality and significance:** 

- I find that the authors do not cite or refer the work of Maroñas et al. in AISTATS 2021 “Transforming Gaussian Processes with Normalizing Flows”, which is extremely similar to the proposed approach or at least relevant for the analysis or comparison. The AISTATS proceedings were published on the 29th March, so it is not the case of overlapping dates with the submission. Additionally, if one googles “Gaussian Processes + Normalising Flows”, 4/5 results point that work… So I see it very difficult to ignore.
- The proposition of CNFs is interesting to leverage the flexibility of the GP model, I like that. However I do not see clearly the contribution. I understand that it might be to use the ODE-based mechanism of Grathwohl et al. (ICLR 2019) instead of the usual chain of transformations in NFs, but without having cited “Maroñas et al. in AISTATS 2021”, it is difficult to claim.
- Overall, I think that the paper is in the good direction, but I think that its significance is limited as long as some key references are omitted and it is not very well placed in the context of GP models (I see that references to Deep Kernel Learning for few-shot learning are preferred).
- The few-shot learning scenario is not fully motivated, for instance, it is difficult to understand the limitations that it produces in the GP approach; at least to me.

**Quality and clarity:** 

- Section 4 introduces a lot of decisions, particularly, on how the proposed model differs from the usual GP learning mechanism. Some of these are not motivated or explained. I consider very important to explain very well each one of them. For instance, why the feature extractor is introduce in the kernel? Is the dimensionality of X super huge? A question arise to me on this point, does this feature extractor preserves the Euclidean distance between data points? If not, what would be the effect on the learning process?
- The notation could be a bit more rigorous, I see that the definition of different variables is equal (for example, the function of the GP and the functions of the NFs). Also in L164 when you refer to Eq. (8) using the transformed variable z, the other equation is written using y. This slightly reduce the clarity and technical quality of the paper..
- The experiments are large enough and I see the effort of authors for demonstrating the performance of the model. However I feel that the comparison with the DKT model is a bit “unfair” or at least reduced. Can be the NLL used to compare different models? My intuition is that it is not allowed.
- The experiment with the sines does not look as a critical situation where a GP could fail, and Figure 5(b) seems a bit overfitted to the data. Would this model generalise once trained? It seems that the DKT preserved better the transition of probability between the examples.

**Questions and other comments:**

- If one looks to ODEs or Grathwohl et al. (ICLR 2019), it is well-known that they are used for modelling dynamical systems. That’s why the time “t” appears and it is integrated in Equation (6). The apparition of the time in the context of the paper is not clear to me, why an ODE is needed? I guess that the “t” variable/dynamics substitute the chain of functional transformations of NFs. Is that right?

**References:**

- J. Maroñas, O. Hamelijnck, J. Knoblauch and T. Damoulas. Transforming Gaussian Processes with Normalizing Flows. AISTATS 2021
- W. Grathwohl, R. T. Q. Chen, J. Bettencourt, I. Sutskever and D. Duvenaud. FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models. ICLR 2019

limitations_and_societal_impact: Both limitations and societal impact considerations are well addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes to improve GP for few shot learning by using continuous normalising flows to give non-Gaussian responses from the Gaussian responses given by GP. The approach is illustrated over a range of data sets.
main_review: The paper uses continuous normalising flows (CNF) to to warp the responses of Gaussian processes. This is a significant generalisation of Warped GP [Snelson, 2003], riding on current default use of GP that gives vector responses. The paper is clearly written.

# Detail comments
1) The paper claims that "GP assume a high similarity between subsequent tasks" (lines 30, 136). Some support for this claim, either by way of illustration or references, will be very helpful to motivate this work for few-shot learning.
2) Cite and discuss relation to [Snelson, 2003] and  [Lázaro-Gredilla, 2012].
3) Cite a paper for spectral mixture kernel (line 214).
4) "Continuous normalising flows" should be in the title.
5) In table 3, NGGP+RBF does better than NGGP+Spectral. An analysis of the results will be useful. In particular, for this case, I would like to know if the use of NGGP does away with the need for covariance functions more complicated then RBF/squared-exponential.
6) In limitations, can you also discuss in relation to sparse GP?
7) In addition to lines 300 to 301, it will be useful to give concrete numbers and timing for the experiments that are illustrated in section 5.
8) Lastly, this paper is set within the context of few-shot regression. However, NGGP could be more useful than just for this. Have the authors tried to use NGGP for, say, just normal regression tasks, or quantile regression? How about experiments tried in [Snelson, 2003] and  [Lázaro-Gredilla, 2012]?

Edward Snelson, Zoubin Ghahramani, Carl Rasmussen. Warped Gaussian Processes.  Advances in Neural Information Processing Systems 16 (NIPS 2003)

Miguel Lázaro-Gredilla. Bayesian Warped Gaussian Processes. Advances in Neural Information Processing Systems 25 (NIPS 2012)




limitations_and_societal_impact: See Q6 above.
review_title: null

Meta-review: The reviewers ultimately agree that the key idea proposed in this paper, namely the use of continuous normalizing flows (CNFs) to achieve highly flexible non-Gaussian predictive posteriors, applicable to few-shot regression and achieving strong results in that domain. This core idea is both natural and novel, and the empirical validation on the particular task of few-shot regression is strong.

While I am ultimately recommending acceptance, I want to echo the comments of a few of the reviewers that the somewhat narrow focus on few-shot learning seems to arbitrarily limit the potential breadth of impact of this paper. This is particularly true since the core methodological contribution -- namely the use of a CNF to achieve a non-Gaussian predictive posterior -- is largely separate from the adaptation to few-shot regression, which involves utilizing simple weight sharing in a deep kernel, alternating optimization over tasks. 

Finally, I'll note that because of the relative decoupling of the authors' core contribution and the mechanism they use to approach few shot learning, the argument made in the author feedback that they don't compare to deep GPs etc because these methods are not intended for few-shot regression is somewhat unconvincing. A number of papers have explored the combination of deep kernels with deep GP models, so the adaptation to DKT is quite straightforward. Because deep GPs result in (continuous) mixture predictive distributions, a number of deep GP papers presenting toy results very similar to your own Figure 1 as motivation. I would urge the authors to consider the above points, and how their papers impact may be broadened by including these comparisons.
Decision: Accept (Poster)