Non-Gaussian Gaussian Processes for Few-Shot Regression
Authors: Marcin Sendera
Jacek Tabor
Aleksandra Nowak
Andrzej Bedychaj
Massimiliano Patacchiola
Tomasz Trzcinski
Przemysław Spurek
Maciej Zieba
Venue: NeurIPS 2021 Poster (2021)
Keywords: Meta-Learning
Few-Shot Learning
Few-Shot Regression
Normalizing Flows
Gaussian Processes

Abstract: Gaussian Processes (GPs) have been widely used in machine learning to model distributions over functions, with applications including multi-modal regression, time-series prediction, and few-shot learning. GPs are particularly useful in the last application since they rely on Normal distributions and enable closed-form computation of the posterior probability function. Unfortunately, because the resulting posterior is not flexible enough to capture complex distributions, GPs assume high similarity between subsequent tasks - a requirement rarely met in real-world conditions. In this work, we address this limitation by leveraging the flexibility of Normalizing Flows to modulate the posterior predictive distribution of the GP. This makes the GP posterior locally non-Gaussian, therefore we name our method Non-Gaussian Gaussian Processes (NGGPs). More precisely, we propose an invertible ODE-based mapping that operates on each component of the random variable vectors and shares the parameters across all of them. We empirically tested the flexibility of NGGPs on various few-shot learning regression datasets, showing that the mapping can incorporate context embedding information to model different noise levels for periodic functions. As a result, our method shares the structure of the problem between subsequent tasks, but the contextualization allows for adaptation to dissimilarities. NGGPs outperform the competing state-of-the-art approaches on a diversified set of benchmarks and applications.

Non-Gaussian Gaussian Processes for Few-Shot Regression
Marcin Sendera Jagiellonian University
Jacek Tabor Jagiellonian University
Aleksandra Nowak Jagiellonian University
Andrzej Bedychaj Jagiellonian University
Massimiliano Patacchiola University of Cambridge
Tomasz Trzcinski
Przemyslaw Spurek Jagiellonian University
Maciej Zieba Wroclaw University of Science and Technology, Tooploox
Jagiellonian University,
Warsaw University of Technology, Tooploox
Abstract
Gaussian Processes (GPs) have been widely used in machine learning model distributions over functions, with applications including multi-modal regression, time-series prediction, and few-shot learning. GPs particularly useful in the last application since they rely on Normal distributions and enable closed-form computation of the posterior probability function. Unfortunately, because the resulting posterior is not flexible enough to capture complex distributions, GPs assume high similarity between subsequent tasks requirement rarely met in real-world conditions. In this work, we address this limitation by leveraging the flexibility of Normalizing Flows to modulate the posterior predictive distribution of the GP. This makes the GP posterior locally non non-Gaussian, therefore we name our method Non-Gaussian Gaussian Processes (NGGPs). ODE-based mapping that operates on each component of the random variable vectors and shares the parameters across all of them. We empirically tested the flexibility of NGGPs various few-shot learning regression datasets, showing that the mapping can incorporate context embedding information to model different noise levels for periodic functions. As result, method shares the structure the problem between subsequent tasks, but the contextualization allows for adaptation to dissimilarities. NGGPs outperform the competing caate-ofthh--a approaches on a diversified set benchmarks and applications.
propose invertible
Introduction
are of the most important probabilistic methods, and they have been widely used to model distributions over functions in a variety applications such as multi-modal regression (56). time-series prediction [ 27 and meta-learning
Gaussian Processes (GPs)
3
46
29 45). Recent works 39 49 where the model is trained to solve supervised task with only few labeled samples available. This particular application is well-fitted to GPs since they can determine the posterior distribution in closed-form from small set of data samples 24
propose to use GPs in the few-shot learning scenario
Corresponding author: marcin, sendera@kootoorr.............a.. uj edu ppl
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Classical Gaussian Processes
Geussian Gaussian Processes
Figure I: Results of Deep Kernels with classical GP (left) and NGGP (right). The one-dimensional samples were generated randomly from sin(x) and sin(x) functions with additional noise. NGGP, compared to GP, does not have an assumption of Gaussian prior, which allows for modeling a multi-modal distribution.
However, the generalization capabilities of GPs come at the price of reduced flexibility when the modeled distributions are complex, egg., they have high skewness or heavy tails. Furthermore, GPs assume a high similarity between subsequent tasks. This condition is rarely met in real- world applications where tasks can vary during time, as the case heteroscedastic regression. These limitations of GPs also extend to multi-modal learning more generally, multi-label regression 56
In this work, we address those drawbacks by modeling the GPs posterior predictive distributions with a local non- Gau-Gaasiinn approximation. We do so by introducing new method that we have named
Gaussian Processes (NGGPs). In NGGPs. we leverage the flexibilityy of Continuous Normalizing Flows (CNF) [T6] model arbitrary probability distributions. In particular, we propose an invertible ODE-based mapping that operates on each component of the random variable vectors. This way, we can compute set of CNFs parameters shared across all vectors, with the resulting mapping incorporating the information of the context to model different noise for periodic functions. Figure I shows how NGGPs are able to capture the overall structure of problem, whereas standard GPs fail. NGGPs are able to reconstruct multi-modal sine function while adapting to local dissimilarities thanks to the contextualization provided by the ODE-based mapping. We provide empirical evidence that NGGPs outperform competitive state-of-the the approaches diversified set benchmarks and applications open-source license
few-shot learning scenario; the code released with an
The contributions of our work can be summarized as follows:
We introduce Non-Gaussian Gaussian Processes (NGGPs), a new probabilistic method for modeling complex distributions through locally Gaussaan posteriors.
We show how invertible ODE-based mappings can be coupled with GPs to process the marginals of multivariate random variables resulting in more flexible models.
We extensively test NGGPs on variety of few-shot learning benchmarks, achieving state- of-the-art art performances in most conditions.
2 Related Work
The related work section is divided into three parts. First, we present general Few-Shot Learning problem. Then, we discuss GPs, focusing on models, which use flow architectures. Finally, in the third paragraph, we describe existing approaches to Few-Shot Learning, which use Gaussian Processes.
Few-Shot Learning Few-Shot Learning aims at solving problems in which the number of obser- vations is limited. Some of the early methods in this domain have applied two-phase approach by pre-training on the base set of training tasks and then fine-tuning the parameters to the test tasks 28 An alternative approach given by paramerrttt metric-learning algorithms, which aim at optimizing metric, that is then used to calculate the distance between the target observations
https://githu/ http://ggttuuub.omm/oooooee gwwp//ommm/oooooommeoooooo gwwssian--gnnnee

---

and the support set items 48 38 42 Another popular approach to few-shot learning is Model Ag-
6. MAML aims finding a set of joined task parameters that can be easily fine-tuned to new test tasks via few gradient descent updates. MAMI. can also be treated as a Bayesian hierarchical model 100 15 18 Bayesian MAML.
nostic Meta-Learning (MAML) 9
and its variants [2
62 S
52
55 combines efficient gradient-based meta-learning with oon-paamettric variational inference in principled probabilistic framework. A few algorithms have been focusing exclusively on regression tasks. An example is given by ALPaCA 11. which uses dataset of sample functions to learn domain-specific encoding and prior over weights.
Gaussian Processes GPs have been applied to numerous machine learning problems, such as spatio-temporal density estimation 7. robotic control [53]. or dynamics modeling in transcriptional processes in the human cell 221. The drawback of GP lies in the computational cost of the training step, which is (nn)) (where n. denotes the number of observations in the training sample).
In [. the authors extend the flexibility of GPs by processing the targets with a learnable monotonic mapping (the warping function). This idea is further extended in [22]. which shows that is possible to place the prior of another GP on the warping function itself. Our method is different from these approaches, since the likelihood transformation is obtained by the use of a learnable CNF mapping.
In 26], the authors present the Transformed Gaussian Processes (TGP), a new flexible family of function priors that use GPs and flow models. TGPs exploit Bayesian Neural Networks (BNNs) as input-dependent parametric transformations. The method can match the performance of Deep GPs at fraction of the computational cost.
The methods discussed above are trained on single dataset, that kept unchanged. Therefore, is not trivial to adapt such methods to the the few-shot setting.
Few-Shot Learning with Gaussian Processes When the number of observations is relatively small, GPs represent an interesting alternative to other regression approaches. This makes GPs a good candidate for meta-learning and few-shot lea-sholearniig, as shown by recent publications that have explored this research direction. For instance, Adaptive Deep Kernel Learning (ADKL) [5 proposes a variant of kernel learning for GPs, which aims at finding appropriate kernels for each task during inference by using a meta-learning approach. A similar approach can be used to learn the mean function [.. 37. the authors presented theoretically principled PAC-Bayesinn meta-learning It can be used with different base learners (e.g., GPs or BNNs) Topics related to kernel tricks and meta-llerning have been explored 47 The authors propose to use nonparametric kernel regression for the inner loop update. In [R], the authors introduce an ceermation-thhoreef framework for meta-learning by using a variational approximation to the information bottleneck. In their GP-based approach, to account for likelihoods other than Gaussians, they propose approximating the non- non-Gaussian terms in the posterior with Gaussian distributions (by using amortized functions), while we use CNFs to increase the flexibility of the GPs.
framework for
In 29 the authors present Deep Kernel Transfer (DKT): a Bayesian treatment for the meta-learning inner loop through the use of deep kernels, which has achieved state-of-the-art results. In DKT, the deep kernel and the parameters of the GP are shared across all tasks and adjusted to maximize the marginal log-likelihood, which is equivalent to Maximum-Likellben is particularly effective in the regression case since it is able to capture prior knowledge about the data through the GP kernel. However, in many settings, prior assumptions could be detrimental if they are not met during the evaluation phase. This is the case in few-shot regression, where there can be significant difference between the tasks seen at training time and the tasks seen at evaluation time. For instance, if we are given few-shot tasks consisting of samples from periodic functions but periodicity is violated at evaluation time, then methods like DKT may suffer in terms of predictive accuracy under this domain shift. In this work, we tackle this problem by exploiting the ficxibility of CNFs.
type II (ML-II) learning. DKT
3 Background
Gaussian Processes. The method proposed in this paper strongly relies on Gaussian Processes (GPs) and their applications in regression problems. GPs are a well-established framework for principled uncertainty guantification and automatic selection of hyperparameters through a marginal likelihood objective 35)) More formally, a GP a collection of random variables such that the joint distribution of every finite subset of random variables from this collection is a multivariate Gaussian (11. We
3

---

denote Gaussian Process as f()) ~ (P(,()),k(..)) where v(x) and ((x,x' are the mean and covariance functions. When prior information is not available, a common choice for μ is the zero constant function. The covariance function must impose a valid covariance matrix. This is achieved by restricting & to be a kernel function. Examples of such kernels include the Linear kernel, Radial Basis Function (RBF) kernel, Spectral Mixture (Spectral) kernel [50. or Cosine-Similario kernel [[%] Kernel functions can also be directly modeled as inner products defined the feature space imposed by a feature mapping X V:
Mz,z') (v)())) (aa")
An advantage of the formulation above is that it can be easily implemented by modeling to through a neural network. Throughout this work, we call this technique the NN Linear kernel (sometimes called Deep Kernel 291). Since every kernel can be described in terms of Equation ① such an approach may be desired if no prior information about the structure of the kernel function is available.
Gaussian Processes provide a method for modcling probability distributions over functions. Consider regression problem:
y; = f(,) + Es, for i == I....,m. where 6, are i..dd noise variables with independent NN a²) distributions. Let be the matrix composed of all samples x, and let y be the vector composed of all target values y.. Assuming that f()) ~ GP (0, 1s (-x-)), we obtain:
(2)
y/X ~ (O, K + oTl,,
(3)
where j.j k(x,, xj). Analogously, inference the unknown during the training samples is obtained by conditioning over the normal distribution. Let (y, x) be the train data and let (y., X.) be the test data. Then the distribution of y. given y, X, X, also a Gaussian distribution 64
y+ ly, X, X. ~ N( K.):
where:
1. = K X. K. = K( X.)
X X + 12 y X K (X, X) + o (X, (X,,))
+ o - K (X,,
Continuous Normalizing Flows. Normalizing Flows (NF)
36 are gaining popularity among
generative models thanks their flexibility and the case of training via direct negative log-likelihood (NLL) optimization. Flexibility is given by the change-of-variiae variable with know prior p(z) to from some observed space with unknown distribution. This mapping is performed through a series of (parametric) invertible functions: y = f, ....0 (z). Assuming known prior p(z) for z, the ligeliiolooodd for y is given by:
technique that maps a latent
N
Of,
log p(y) = p(z)
Σ log det ===11
(5)
dzn-1
where = f.-1 o f- (y) is result of the invertible mapping. The biggest challenge in This due to the fact that they need to be expressive while guarantecing an efficient calculation of the Jacobian determinant, which usually has a cubic cost. An alternative approach is given by CNF models [6]. CNTs use continuous, time-dependent transformations instead of sequence of discrete functions f.. ......... fi. Formally, we Oz(()
normalizing flows the choice of the invertible functions f,....,ff, f,.
introduce a function gz(z(t), t) that models the dynamics of ztt), øt by B. In the CNF setting, we aim at finding a solution y == z(ti) for the differential equation, assuming the given initial state z == zttu) with a known prior. As a consequence, the transformation function fg is defined as:
g (z()), t). parametrized
y == fz(z) z I/
gz(z(t),
(6)
The inverted form of the transformation can be easily computed using the formula:
S(a ds
fg (y) =
gz(z(t),t)dt
(7)
The log- bog-rrbbability y can be computed by:
Oga dt where (y) z. dz((t)
log p(y) logg(ff (y))
Tr
(8)
10

---

Input
p(z x)
f(z, h(x))
p(y xx)
data
Continuous
X
Feature extractor
Gaussian Process
Invertible
Mapping
Figure 3:
The general architecture of our approach. The input data are embedded by the feature extractor hn-) and then used create kernel for the GP. Next, the output of the GP adjusted using an invertible mapping f(-) which is conditioned on the output of the feature extractor. This allows us model complex distributions the target values y.
4 Non-Gaussian Gaussian Processes
In this work, we introduce Non-Gaussian Gaus- sian Processes (NGGPs) to cope with the signifi- cant bottlenecks of Gaussian Processes for Few- Shot regression tasks: reduced flexibility and assumption about the high similarity between the structure of subsequent tasks. We propose to model the posterior predictive distribution non-Gaie Gaussian on each datapoint. We are do- ing so by incorporating the flexibility of CNFs. However, we do not stack the CNF on GP to model the multidimensional distribution over y. Instead, we attack the problem with an invert- ible ODE-based mapping that utilize each component of the random variable vector and create the specific mapping for each datapoint (see Figure 2
Real data distribution P(Y)))
Continuous Invertible transformation
Input
Process approximation
O000
Figure General idea of NGGP, A complex multi- modal distribution can be modelled by exploiting a continuous invertible transformation to fit the Normal distribution used by the GP. Image inspired by Figure in [[]]
The general overview of method is presented in Figure Consider the data matrix X, which stores the observations x, for a given task. Each element is processed by a feature extractor h(-) to create the latent embeddings. Next, model the distribution of the latent variable with GP. Further, we use an invertible mapping f(.) in order to model more complex data distributions. Note that the transformation is also conditioned the output of the feature extractor h(.) to include additional information about the input.
The rest of this section is organized follows. In Section we demonstrate how the marginal can be calculated during training. In Section 4.2 we demonstrate how to perform an inference stage with the model. Finally, in Section 4.3
we show how the model is applied to the few-shot setting.
4.1 Training objective
Consider the GP with feature extractor h+(-) parametrized by & and any kernel function ko(::) parametrized by 0. Assuming the given input data and corresponding output values z, we can define the marginal log log-probability for the GP:
log p(z X, Φb 0)
+ +
2 log K + 0²I| log(2Tr,
(9)
2
where D is the dimension of K is the kernel matrix, and ki = ko (ha (x;). (x;)). Taking into account Equation
express the log marginal likelihood as follows:
Ög Oz((1)
logp(y X, Φ 0, 3) logglo/
Tr
dl,
(10)
where Fgg (y) z, p(z Φ, is the marginal defined by Equation 2 and fg (-) is the trans- formation given by Equation (6). In the next stage of the pipeline, propose to apply the flow transformation [ (-) independently to each one of the marginal elements in y. that is fz (y) = [fg (y11...., (yD)]T, with (.) sharing its parameters across all components. In other words, while the GP captures the dependency across the variables, the flow operates indepen- dently on the marginal components of Additionally, the flow is conditioned the information
5

---

Algorithm NGGP in the few-shot setting, train and test functions.
Require: [Tm. train dataset and = (S., Q test task.
Parameters: kernel hyperparameters, & feature extractor parameters, ,3 flow transformation parameters. Hyperparameters: T/, 7: step size hyperparameters for the optimizers.
I: function TRAIN(D, a. 7) 7,. 0, Φ. ()
2: while not done do
3:
Sample task 7 (X. y)
4: C. logr(y X, Φ. (3)
See Equation
Update
0
VVoL,
b Updating kernel hyperparameters
6:
Φ - Φ nV
b Updating feature extractor parameters
7:
B VVLL
& Updating flow transformation parameters
8:
end while
9: return d, ,3
10:
end function
11:
function TEST(T., 0, Φb. 33
12:
Assign support S. (X. ,, y..) and query Q. (X..g: Vo.a)
13: return ply..g (x, Y. X. 6,
D See Equation
E
end function
encoded by the feature extractor, such that it can account for the context information h (xa) from the corresponding input value X.
Ya = (zd, (xa))) = 2d za+ 1.'h ga(za(t), 1, (xa))dt
(11)
The inverse transformation can be easily calculated with the following formula:
fg (ya) yd 1/ go g=(z((t), ho
(12)
The final marginal log- ligeeiidl can be expressed as:
D Σ
Ogg dl, @za(t)
logp(y X, cb, 0,
log log db 0)
(13)
where 2h CG' h. (X)) is the vector of inverse functions ff( (za, (xa))) given by Equation
12
The transformation described above can be paired with popular CNF models. Here we choose Ffjord [16], which has showed perform better low-dimensional data when compared against discrete flows like RealNVP (5) or Glow [T) Note that, the CNF is applied independently on the components of the GP outputs and shared across them. Therefore, do not have any issue with the estimation of the Jacobian, since this corresponds to the first-order derivative of the output w.r.t. the scalar input.
4.2 Inference with the model
At inference time, estimate the posterior predictive distribution p(y.|X., X, Φ, 0, where we have access to training data (y. X) and model the probability of D, test outputs y, given the inputs X,. The posterior has a closed expression (see Section B Since the transformation given by Equation operates independently on the outputs, we are still able to model the posterior in closed form:
Oge di, Dza(t)
Dgg(yy|X..,, X, do 0,3) = logg(zz
x, Φ
(14)
where 2 - 1 (y (X.) 2 -f (y h4 (X) are the inverted transformations for test and train data, and p(z Xx, 2 X, Φb, 0) is the GP posterior described Equation

---

200
03
855
(a) NGGP NN Linear
DKT Spectral
Figure 4: The results for the sines dataset with mixed-noise for the best performing kernels for NGGP (NN Linear) and DKT (Spectral). The top plot each figure represents the estimated density (blue hue) and predicted curve (red line), well as the true test samples (navy blue dots). For three selected input points (denoted by black vertical lines), we plot the obtained marginal densities the bottom images (red color). In addition, for the NGGP method, we also plot the marginal priors (in green) for each these three points. It may be observed that NGGP more successful in modeling the marginal for varying noise levels.
4.3 Adaptation for few-shot regression
In few-shot learning, we are given a meta-dataset of tasks D = [T,, J where each task T, contains support set Sn and a query set Qn At training time, both support and query contain input-output pairs y), and the model is trained to predict the target in the query set given the support. At evaluation time, we are given previously unseen task T. = (S., Q.). and the model used to predict the target values of the unlabeled query points. We are interested in few-shot regression, where inputs are vectors and outputs are sealars.
We follow the paradigm of Deep Kernel Transfer (DKT) introduced in 24 and propose the following training and testing procedures (see Algorithm Db. During the training stage, we randomly sample the task, calculate the loss defined by Equation 31 and update all the parameters using gradient-buu bbsed optimization. During testing, we simply identify the query and support sets and calculate the posterior given by Equation 14
Experiments
In this section, we provide an extensive evaluation our approach (NGGP) on a set of challenging few-shot regression tasks. We compare the results with other baseline methods used in this domain. As quantitative measures, we use the standard mean squared error (MSE) and, when applicable, the negative log- likelhodddo (NLL).
Sines dataset We start by comparing NGGP other few-shot learning algorithms regression task defined on sines functions. To this end, we adapt the dataset from [9] in which every task composed of points sampled from sine wave with amplitude in the range [0.1, 5.0), phase in the range [0, x). and Gaussian noise N (0, 0.1). The input points are drawn uniformly at random from the range [-5, 5). We consider support and query points during the training and 5 support and 200 query points during inference. In addition, following 29 we also consider an out-of- nat-of-range scenario, in which the range during the inference is extended to [-5, 10]. We also perform a variation of sines experiment in which we inject input-dependent noise. The target values in this setting modeled by A sin (rr :p) + Ja sple, where the amplitude, phase, input, and noise points are drawn from the same distributions as in the standard setup described before. We refer to this dataset ablation as mixed-noise sines. For more information about the training regime and architecture, refer to Supplementary Materials A Table presents the results of the experiments. We use the DKT method as reference since it provides state-of- of-te-of-the-ar with more baseline methods, please refer to Supplementary Materials B
simple
the-art art results for the few-shot sines dataset 29
For a report
Both DKT and our NGGP perform very well when paired with the Spectral Mixture Kernel, achieving the same performance in-range data. However, approach gives superior results in the out-of-
7

---

Table I: The MSE and NLL results for the inference tasks on sines datasets in the in- in-range and out out-range settings, Lowest results in bold (the lower the better).
sines
mixed-noks sines
Method
in-range
out-offrange runge
in-range
out-of-range
MSE
MSE
NLL
MSE
MSF
NTL
DKT + RBF
1.36F1.64
0076::0.06
2.44:2.70
0.6910.06
L60
0.48 + 0.22
2.99
2.37
2.01 :: 0.59
DKT Spectral
0.0210.01
-0.8310.003
-0.7.10.14
0.18 0.12
0.37.0.16
1.33
1.30
0.40
DKT NN Lincar
0.02:00022
-0.7320.11
6.61431.63
38.38:40.16
0.18:00111
0
5.85
12,10
8.64 6.55
NGGP + RBF
1.2++1.4 1.40
-0.74:0.07
3.0242533
0065+0008
1.30+ 1.36
0.33
0.366
3.0
183 0:53
NGGP + Spectal
0.02+0.01
-0.83+00.88
0.03++0.02
-0.80+0.007
0.22 0.14
0.44
0.19
1.14 1.14+00.90 0.90
1.38 0.38
NGGP + NN Linear
0,04+0003
-0.73+0.10
1.34
29.86=27.97
0,20 0.12
0.17
0.15
4.74 6.29
2.92 1.93
range scenario, confirming that NGGP is able to provide a better estimate of the predictive posterior for the unseen portions of the task. It is also worth noting that in all settings, NGGP consistently achieves the best NLL results. This is particularly evident for the in-range mixed-noise sines dataset. We analyze this result in Figure 4. where NGGP successfully models the distribution of the targets, predicting narrow marginals for the more centralized points and using wider distributions for the points with larger noise magnitude. This is in contrast with DKT, which fails capture different noise levels within the data. These observations confirm our claim that the NGGP is able to provide a good estimate in the case of heteroscedastic data.
Head-pose trajectory In this experiment, we use the Queen Mary University of London multiview face dataset [13]. This dataset is composed of grayscale face images of 37 people (32 train, 5 test). There are 133 facial images per person, covering a viewsphere of +-900 in yaw and +:300 in tilt at 10º increment. We follow the evaluation procedure provided in (2] Each task consists of randomly sampled trajectories taken from this discrete manifold. The in-range scenario includes the full manifold, while the out-of-range scenario includes only the leftmost 10 angles. At evaluation time, the inference is performed over the full manifold with the goal of predicting the tilt. The results are provided in Table In terms of MSE, our NGGP method competitive with other approaches. but it achieves significantly better NLL results, especially in the out-of out-of-range setting. This suggests that NGGPs are indeed able to adapt to the differences between the tasks seen at training time and tasks seen at evaluation time by providing a probability distribution that accurately captures the true underlying data.
Table 2: Quantitative results for Queen Mary University of London for in-range and out-of-range settings, taking into account NLL and MSE measures.
Method
In-range
out-of-range
MSE
NLL
MSE
NLL
Feature Transter/1
0,23+:0.04
0.20:20.01
Feature Transfer/100
0.22:t-003
0.18:1:0:01
MAML (1 step)
0,21:10.01
0.18:1.002
DKT RBF
0.124:0.04
0.13::0.4
0.14::0.03
0.71
DKT Spectral
0.10:1:0001
0.03:.0.13
0.07.10.05
0.00:10.09
DKT NN Linear
0.04-10.03
.0.12:10.12
0,12:10.05
0.30110.11
NGGP NN Lineur
0.02:10.02
-0.47:1..12
0.06:10.05
0.24
NGGP + Spectral
0.03:1.0.03
.0.68.10.23
0.03-1.0.03
.0.62.10.24
Object pose prediction We also study the behavior of NGGP in a pose prediction dataset introduced in [54]. Each task in this dataset consists of 30 gray-scale images with resolution 128 x 128, divided evenly into support and query. The tasks are created by selecting an object from the Pascal 3D [51] dataset, rendering it in 100 random orientations, and sampling out of it 30 representations. The goal to predict the orientation relative to a fixed canonical pose. Note that 50 randomly selected objects are used to create the meta-training dataset, while the remaining 15 are utilized to create distinct meta-test set. Since the number of objects in meta-training is small, model could memorize the canonical pose of each object and then use it to predict the target value, completely disregarding the support points during the inference. This would lead to poor performance on the unscen objects in the meta-test tasks. This special case of overfitting is known as the memorization problem [54]
We analyze the performance of GP-based models this setting by evaluating the performance of We compare them against the methods used in [54], namely MAML [9]
DKT and NGGP model [
Information about architecture and training regime is given in Supplementary Materials
8

---

DKT RBBF NGGP--REP
NIL value
4
DKT(10)
DKT(100)
10
100
Number of Support examples
NGGP(10)
NGGP(100)
(a) NLL results DKT NGGP.
Single day comparison DKT vs. NGGP.
Figure 5: The results for the Power dataset experiment: (a) The quantitative comparison between DKT and NGGP considering different numbers of support examples. (b) The power consumption for single day randomly selected from the test data. We compare DKT vs. NGGP (with RBF kernel) considering 10 and 100 support points. NGGP captures multi-modality and thus better adjusts the data distribution.
Conditional Neural Processes (CNP) 12 and their meta-regularized versions devised to address the memorization problem MR-MAML and MR-CNP (54). In addition, we also include the fine-tuning (FT) baseline and CNP versions with standard regularization techniques such as Bayes-by- by-Backprop (BbB) and Weight Decay [Z The results presented in Table
Both GP-related approaches: NGGP and DKT are similar or usually outperform the standard and meta- regularized methods, which indicates that they are less prone memorization and therefore benefit from bet- ter generalization. The NLL is significantly lower for NGGP than for DKT, confirming that NGGP is better at inferring complex data distributions.
Table 3: Quantitative results for the object pose prediction task. We report the mean and standard deviation over 5 trials. The lower the better. Asterisks (*) denote values reported in (54
Method
MSE
NLL
Power Dataset In this series of experiments, we use Power [ dataset and define an experimental setting for the few-shot setting. We treat each time series com- posed of 1440 values (60 minutes 24 hours) rep- resents the daily power consumption (sub metering 3) as a single task. We train the model using the tasks from the first 50 days, randomly sampling 10 points per task, while validation tasks are generated by randomly selecting from the following 50 days.
MAML X MR MAML
5,39 1.31
2.26 0.09
CNPs
8.48 0.12
MR-CNP*
2,89 + 0.18
0.18
FT
FT Weight Decay
6.16 1.6:1:0 0.12
CNP Weight Decay
6.86 0.27
CNP + BbB*
7.73 1733-211 0.82
DKT RBF DKT Spectral
1.82+:0.11
1.35 0.10
1.79 0.15
1.30 +: 0.06
NGGP RBF
1.98 + 0.27
0.22 + 0.08
Quantitative and qualitative analysis are provided in Figure 5 We use only NLL to assess the results due to the multi-modal nature of the data and analyze the value of the criterion for different numbers of support examples. NGGP better adjusts to the true data distribution, even in the presence of very few support examples during inference. This experiment supports the claim that NGGPs are well-suited for modeling multi-modal distributions and step functions.
Spectral
0.45
NASDAQ and EEG datasets In order to test the performance of our methods for real-world time series prediction, we used two datasets NASDAQ100 B0 and EEG [8. For an extensive description of the datasets and evaluation regime of this experiment, see Supplementary Materials Quantitative results are presented in Table Our experiments show that NGGP outperforms the baseline DKT method across all datasets. The improvement is especially visible for the oul-of-range NASDAQ100 when both methods use the RBF kernel. The results suggest that NGGPs can successfully used to model real-world datasets, when the data does not follow Gaussian distribution.

---

Table 4: Quantitative results for NASDAQ and EEG datasets.
(a) NASDAQ100
(h) EEG
in-range
in-range
Method
MSE 100
Method
MSE
NLL
NGGP-+ + RBF
0.012 1: 0.014
.3,092 1: 0.255
NGGP + RBF
0.222 0.181
-1.715
0.282
NGGP NN Linear
0.023 1: 0.044
-2.567 1: 1.235
NGGP NN Linear
0.361
0.223
-13387 +: 0.273
DKT + NN Linear
0.027 + 0.032
-2.429
+ 0.271
DKT NN Linear
0.288
0.169
-1.443 - 0.188
DKT RBF
0.022 0.042
-2,878
0.706
DKT RBF
-1.640
0.237
out-of-range
out-of-range
Method
100
Method
MSE
NLL
NGGP- + RBF
0.016 1: 0.034
-2,978 1: 0.571
NGGP + RBF
0.463 0.415
-1.47
0.221
NGGP NN Linear
0.003 1: 0.004
-2.998 1: 0.260
NGGP NN Linear
0.452
0.578
-1,046 0.624
DKT + NN Linear
0.005 + 0.006
-2.612 + 0.059
DKT + NN Linear
0.528 0.642
-1.270 - 0.622
DKT RBF
0.181 0.089
1,049 + 2.028
DKT RBF
0.941
-1.242 0.685
6 Conclusions
In this work, introduced NGGP generalized probabilistic framework that addresses the main limitations of Gaussian Processes, namely its rigidity in modeling complex distributions. NGGP leverages the flexibility of Normalizing Flows to modulate the posterior predictive distribution of GPs. Our approach offers robust solution for few-shot regression since it finds a shared set parameters between consecutive tasks while being adaptable dissimilarities and domain shifts. We have provided an extensive empirical validation of our method, verifying that it can obtain state-of-the-an performance wide range of challenging datasets. In future work, we will focus applications of few-shot regression problems needing the estimation of exact probability distribution (e.g., continuous object-tracking) and settings where there is potential discontinuity in similarity for subsequent tasks (e.g., continual learning).
Limitations The main limitation of NGGP is the costs of learning flow-based models, that could be more expensive than using a standard DKT when the data come from a simple distribution. In such a case, other methods like DKT could be more efficient. Moreover, GPs are expensive for tasks with a large number of observations, making NGGP better fit for few-shot learning rather than bigger settings. Finally, in some cases, it can be more challenging to train and fine-tune NGGP than DKT because the number of parameters and hyper- paper-parameters is overall larger (e.g. the parameters of the flow).
Broader Impact Gaussian Processes for regression already have had huge impact on various 21
NGGPs make it possible to apply priori knowledge and expertise to even more complex real-world systems, providing fair and human-conscious solutions, i.e., in neuroscience social studies (see experiments on individual power consumption, EEG, and NASDAQ datasets from section 5). The proposed method is efficient and represents a great tool for better uncertainty quantification. Careful consideration of possible applications of our method must be taken into account to minimize any possible societal impact. For instance, the use of NGGP in bbject-tracking could be harmful if deployed with malevolent and unethical intents in applications involving mass surveillance.
real-world applications 7 5
25
Acknowledgments
This research was funded by Foundation for Polish Science (grant no POIR. POIR.04.04. 14.04400-00-110 14.0000-4D/18-000 carried out within the Team-Net program co-financed by the European Union under the European Regional Development Fund) and National Science Centre, Poland (grant no 2020/39/B/ST6/015L1
The work of M. Zieba was supported by the National Centre of Science (Poland) Grant No. 2020/37/B/ST6/03463 The work of P. Spurek was supported by the National Centre of Science (Poland) Grant No. 2019/33/B/ST6/0088 Area Digiworld under the program Excellence Initiative Research University at the Jagiellonian University in Kraków. The authors have applied CC BY license to any Author Accepted Manuscript (AAM) version arising from this submission, accordance with the grants' open access conditions.
This research was funded by the Priority Research
10

---

References
[1] Individual household electric power consumption data set. luttps:/arrciie ics uci edu/ml/ Accessed: 2021-05-25.
[2] Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty neural network. In International Conference on Machine Learning, pages 1613-1622. PMLR, 2015.
[3] Sofiane Brahim-Belhouari and Amine Bermak, Gaussian process for nonstationary time series prediction. Computational Statistics & Data Analysis, 47(4):705-712, 2004.
[4] Wei- Yu Chen, Ben-Cheng Liu, Zsolt Kira, Yu- Vu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv:1904.04222 2019.
[5] Laurent Dinh, Jascha Sohl- Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint urXiv:1605.0880 2016.
[6] Yingiun Du, Haoliang Sun, Xiantong Zhen, Jun Xu, Yilong Yin, Ling Shao, and Cees G. M. Snock. Metakernel: Learning variational random features with limited labels, 2021.
[7] Vincent Dutordoir, Hugh Salimbeni, Mare Deisenroth, and James Hensman. Gaussian process conditional density estimation, 2018.
[8] SM Fernandez-Fraan MA experiments for eeg signals based ssvep brain computer interface. International Journal of Advanced Research, 6(2):1718-1732 2018. Accessed: 2021-05-25.
JC Pedraza-Oree and JM Ramos-Arreguit Screen task
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pages 1126 1135, PMLR, 2017.
[10] Chelsea Finn, Kelvin Xu. and Sergey Levine. Probabilistic model-agnostic meta-learning. arXiv preprint arXiv:1806.000 028.0028777.2 2018.
[11] Vincent Fortuin, Heiko Strathmann, and Gunnar Rätsch. Meta-leaming mean functions for gaussian processes, arXiv preprint arXiv:1901.0809, 2019.
[12] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint 1877v:180,.01622, 2018.
[13] Shaogang Gong. Stephen McKenna, and John Collins. An investigation into face pose distributions. In Proceedings of the Second International Conference on Automatic Face and Gesture Recognition, pages 265-270. IEEE, 1996.
[14] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta-learning probabilistic inference for prediction. arXiv preprint 1rXiv:/805.0092 2018.
[15] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-basee meta-learning as hierarchical bayes. arXiv preprint arXiv: 1801108930 2018.
[16] Will Grathwohl, Ricky TQ Chen, Jesse Betterncourt Ilya Sutskever, and David Duvenaud. Ffjord: Free- form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367 2018.
[17] James Harrison, Apoorva Sharma, and Marco Pavone. Meta-learning priors for efficient online bayesian regression. In International Workshop on the Algorithmic Foundations of Robotics, pages 318-337, Springer, 2018
[18] Ghassen Jerfel, Erin Grant, Thomas Griffiths, and Katherine Heller. Reconciling meta-learning and continual learning with online mixtures tasks. arXiv preprint arXiv:1812.06089, 2018.
[19] Diederik Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions, arXiv preprint arxiv: 807.03039, 2018.
[20] Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In Advances in neural information processing systems, pages 950-957, 1992.
[21] Neil Lawrence, Guido Sanguinetti, and Magnus Rattray. Modelling transcriptional regulation using gaussian processes. In B. Schölkopf, J. Platt, and T. Hoffmman, editors, Advances in Neural Information Processing Systems, volume 19. MIT Press, 2007.

---

[22] Miguel Lázaro-Gredilla. Bayesian warped gaussian processes. Advances in Neural Information Processing Systems, 25:1619-1627,201 2012.
[23] Miguel Lázzro-Grediill. Bayesian warped gaussian processes. In E. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
[24] Yan Li, Ethan X Fang, Huan Xu, and Tuo Zhao. International conference learning representations 2020. In International Conference on Learning Representations 2020, 2020.
[25] Zhu Li, Adrian Perez-Suay. Gustau Camps-Valle Valls, and Dino Sejdinovic. Kernel dependence regularizers and gaussian processes with applications to algorithmic fairness, 2019.
[26] Juan Maroñas, Oliver Hamelijnck, Jeremias Knoblauch, and Theodoros Damoulas. Transforming gaussian processes with normalizing flows. International Conference on Artificial Intelligence and Statistics, pages 1081-1089, PMLR, 2021.
[27] Fatemeh Najibi, Dimitra Apostolopoulou. and Eduardo Alonso. Enhanced performance gaussian process regression for probabilistic short-term solar output forecast. International Journal of Electrical Power & Energy Systems, 130:10691 16, 2021.
[28] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and datu engineering,
2009.
[29] Massimiliano Patacchiola, Jack Turner, Elliot J Crowley, Michael O'Boyle, and Amos Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. Advances in Neural Information Processing Systems, 33, 2020.
[30] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual-stage attention-based recurrent neural network for time series prediction, 2017. Accessed: 2021-05-25.
[31] Joaquin Cainonero-Candell and Carl Edward Rasmussen. A unifying view of sparse approximate gaussian process regression. The Journal of Machine Learning Research, 6:1939-1959 2005.
[32] Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. arXiv preprint arXiv:1909.04630 2019.
[33] Carl Edward Rasmussen, Gaussian processes in machine learning. In Summer school on machine learning, pages 63-71. Springer, 2003.
[34] Carl Edward Rasmussen and C Williams. Gaussian processes for machine learning the mit press. Cam- bridge, MA, 2006.
[35] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.
[36] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pages 1530-1538. PMIR, 2015.
[37] Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. Pacoh: Bayes-optimal meta- learning with pac-guarantees. International Conference on Machine Learning, pages 9116 9126. PMLR, 2021.
[38] Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. arXiv preprint 77 2017.
[39] Jake Snell and Richard Zemel. Bayesian few-shot classification with one-vs-each p\ olya-gamma aug- mented gaussian processes. arXiv preprint arXiv:2007. 2020.
[40] Edward Snelson, Zoubin Ghahramani, and Carl Rasmussen. Warped gaussian processes. S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems, volume 16. MIT Press, 2004.
[41] Edward Snelson. Carl Edward Rasmussen, and Zoubin Ghahramani. Warped gaussian processes. Advances in neural information processing systems, 16:337-344, 2004.
[42] Flood Sung. Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208. 2018
12

---

[43] Michalis K Titsias, Sotirios Nikoloutsopoulos, and Alexandre Galashov. Information theoretic meta learning with gaussian processes. arXiv preprint arXiv:2009.05228, 2020.
[44] Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033 IEEE,
2012.
[45] Prudencio Tossou, Basile Dura, Francois Laviolette, Mario Marchand, and Alexandre Lacoste. Adaptive deep kernel learning. arXiv preprint 1905. 12131, 2019.
[46] Martin Trapp. Robert Peharz, Franz Pernkopf. and Carl Edward Rasmussen. Deep structured mixtures of gaussian processes. International Conference on Artificial Intelligence and Stafistics, pages 2251-2261. PMLR, 2020.
[47] Arun Venkitaraman, Anders Hansson, and Wahlberg. Task similarity aware meta-learning through nonparametric kernel regression. arXiv preprint arXiv:2006.07212, 2020.
[48] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukeuoglu, and Daan Wierstra. Matching networks for one shot learning. arXiv preprint 16Xiv:660..040000 2016.
[49] Yaqing Wang, Quanming Yao. James Kwok, and Lionel Ni. Generalizing from a few examples: survey on few-shot learning. ACM Computing Surveys (CSUR), 53(3):1-34, 2020.
[50] Andrew Wilson and Ryan Adams. Gaussian process kernels for pattern discovery and extrapolation. In International conference on machine learming, pages 1067-1075. PMLR, 2013.
[51] Xiang, Roozbeh Mottaghi, Silvio Savarese. Beyond pascal: benchmark for 3d object detection in the wild. In IEEE winter conference on applications of computer vision, pages 75-82. IEEE, 2014.
[52] Xu, Sean-Francois Ton, Hyunjik Kim, Adam Kosiorek, and Whye Teh. Metafun: Meta-leaming with iterative functional updates. In International Conference on Machine Learning, pages 10617 (0617-10627. PMLR, 2020.
[53] Yan Yeung and Yu Zhang. Learning inverse dynamics by gaussian process begrression under the multi-task learning framework. In The Path to Autonomous Robots, pages 1-12. Springer, 2009.
[54] Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. arXiv preprint arXiv:1912.038222 2019.
[55] Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages
2018.
[56] Maciej Zicba, Marcin Przewiežlikoski, Marek Smieja, Jacek Tabor, Tomasz Trzcinski, and Prze- mystaw Spurek. Regflow: Probabilistic flow-based regression for future prediction. arXiv preprint 1xXiv:2011.1462,, 2020.
13

---

1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section f
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
See
Section x
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Section S and Supplementary Materials A
(b) Did you specify all the training details (e.g., data splits, byperparameters, how they
were chosen)? [Yes] See Section Jnd Supplementary Materials A
(c) Did you report error bars (c.g., with respect the random seed after running experi-
ments multiple times)? [Yes] See Section S and Supplementary Materials A
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [No]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 5 and Supplementary Materials A
(b) Did you mention the license of the assets? [Yes] See Supplementary Materials A
(c) Did you include any new assets either in the supplemental material or as a URL? [N/]]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See Supplementary Materials A
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "o8Jhq5L9ukZ", "reviewer": "Reviewer_3o7L", "summary": "Authors propose a method for incorporating continuous normalising flows (CNFs) into the learning mechanism of Gaussian processes (GP). The work is motivated by the scenario of few-shot learning, where they need more flexibility to model subsequent tasks that might differ between them. Experiments show that training such models is possible.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality and significance:** \n\n- I find that the authors do not cite or refer the work of Maroñas et al. in AISTATS 2021 “Transforming Gaussian Processes with Normalizing Flows”, which is extremely similar to the proposed approach or at least relevant for the analysis or comparison. The AISTATS proceedings were published on the 29th March, so it is not the case of overlapping dates with the submission. Additionally, if one googles “Gaussian Processes + Normalising Flows”, 4/5 results point that work… So I see it very difficult to ignore.\n- The proposition of CNFs is interesting to leverage the flexibility of the GP model, I like that. However I do not see clearly the contribution. I understand that it might be to use the ODE-based mechanism of Grathwohl et al. (ICLR 2019) instead of the usual chain of transformations in NFs, but without having cited “Maroñas et al. in AISTATS 2021”, it is difficult to claim.\n- Overall, I think that the paper is in the good direction, but I think that its significance is limited as long as some key references are omitted and it is not very well placed in the context of GP models (I see that references to Deep Kernel Learning for few-shot learning are preferred).\n- The few-shot learning scenario is not fully motivated, for instance, it is difficult to understand the limitations that it produces in the GP approach; at least to me.\n\n**Quality and clarity:** \n\n- Section 4 introduces a lot of decisions, particularly, on how the proposed model differs from the usual GP learning mechanism. Some of these are not motivated or explained. I consider very important to explain very well each one of them. For instance, why the feature extractor is introduce in the kernel? Is the dimensionality of X super huge? A question arise to me on this point, does this feature extractor preserves the Euclidean distance between data points? If not, what would be the effect on the learning process?\n- The notation could be a bit more rigorous, I see that the definition of different variables is equal (for example, the function of the GP and the functions of the NFs). Also in L164 when you refer to Eq. (8) using the transformed variable z, the other equation is written using y. This slightly reduce the clarity and technical quality of the paper..\n- The experiments are large enough and I see the effort of authors for demonstrating the performance of the model. However I feel that the comparison with the DKT model is a bit “unfair” or at least reduced. Can be the NLL used to compare different models? My intuition is that it is not allowed.\n- The experiment with the sines does not look as a critical situation where a GP could fail, and Figure 5(b) seems a bit overfitted to the data. Would this model generalise once trained? It seems that the DKT preserved better the transition of probability between the examples.\n\n**Questions and other comments:**\n\n- If one looks to ODEs or Grathwohl et al. (ICLR 2019), it is well-known that they are used for modelling dynamical systems. That’s why the time “t” appears and it is integrated in Equation (6). The apparition of the time in the context of the paper is not clear to me, why an ODE is needed? I guess that the “t” variable/dynamics substitute the chain of functional transformations of NFs. Is that right?\n\n**References:**\n\n- J. Maroñas, O. Hamelijnck, J. Knoblauch and T. Damoulas. Transforming Gaussian Processes with Normalizing Flows. AISTATS 2021\n- W. Grathwohl, R. T. Q. Chen, J. Bettencourt, I. Sutskever and D. Duvenaud. FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models. ICLR 2019\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Both limitations and societal impact considerations are well addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3,5"}, "extra_text": {}}, {"review_id": "jbWHj-R0df", "reviewer": "Reviewer_AvKx", "summary": "The authors propose to make the output distributions of Gaussian processes more flexible by combining them with continuous normalizing flows. They then show that in the application of few-shot learning, this can have benefits over standard GPs without the flows.\n\nUPDATE: Score raised due to clarifications in the author response.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The idea seems to be novel and it is intuitive that NFs could help improve the flexibility of the distribution.\n- The empirical results are promising.\n\nWeaknesses:\n- The description of the method is somewhat unclear and it is hard to understand all the design choices.\n- Some natural baselines and important related work seem to be missing.\n\nMajor comments:\n- The lack of flexibility of standard GPs is not a new observation as has been approached in the past, possibly most famously by the deep GP [1]. These models have recently become a mainstream tool with easily usable frameworks [e.g., 2], so that it would seems like a natural baseline to compare against.\n- Generally, a lot of related work seems to be missed by this paper. For instance, meta-learning kernels for GPs for few-shot tasks has already been done by [3] and then later also by [4,5,6]. These should probably be mentioned and it should be discussed how the proposed method compares against them.\n- The paper proposes to use CNFs, but these require solving a complex-looking integral (e.g., Eq. 9). It should be discussed how tractable this integral is or how it is approximated in practice. Moreover, it seems like an easier choice would be standard NFs, so it should be discussed why CNFs are assumed to be better here. Possibly, one should also directly compare against a model with a standard NF as an ablation study.\n- In l. 257ff it is claimed that the proposed GP methods are less prone to memorization. How does this compare to the results in [4], where DKT seems to memorize as well? Could the regularization proposed in [4] be combined with the proposed model?\n\nMinor comments:\n- In l. 104 it is said that every kernel can be described by a feature space parameterized by a neural network, but this is trivially not true. For instance, for RBF kernels, the RKHS is famously infinite-dimensional, such that one would need an NN with infinite width to represent it. So at most, NNs can represent finite-dimensional RKHSs in practice. This limitation should be made more clear.\n- l. 151 with GP -> with a GP\n- l. 152 use invertible mapping -> use an invertible mapping\n- l. 161 the \"marginal log-probability\" is more commonly called \"log marginal likelihood\" or \"log evidence\"\n- Eq. (8): should it be $\\mathbf{z}$ instead of $\\mathbf{y}$?\n- In the tables, it would be more helpful to also bolden the fonts of the entries where the error bars overlap with the best entry.\n\n[1] Damianou & Lawrence 2012, https://arxiv.org/abs/1211.0358\n\n[2] Dutordoir et al. 2021, https://arxiv.org/abs/2104.05674\n\n[3] Fortuin et al. 2019, https://arxiv.org/abs/1901.08098\n\n[4] Rothfuss et al. 2020, https://arxiv.org/abs/2002.05551\n\n[5] Venkitaraman et al. 2020, https://arxiv.org/abs/2006.07212\n\n[6] Titsias et al. 2020, https://arxiv.org/abs/2009.03228", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the method are hard to assess, mostly because the choice of CNFs over NFs or any other flexible distribution family is not well motivated and because (theoretical and empirical) comparisons to many relevant related methods are missing. This should be addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Xd7ay3Tl7O", "reviewer": "Reviewer_apVw", "summary": "The paper proposes to improve GP for few shot learning by using continuous normalising flows to give non-Gaussian responses from the Gaussian responses given by GP. The approach is illustrated over a range of data sets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper uses continuous normalising flows (CNF) to to warp the responses of Gaussian processes. This is a significant generalisation of Warped GP [Snelson, 2003], riding on current default use of GP that gives vector responses. The paper is clearly written.\n\n# Detail comments\n1) The paper claims that \"GP assume a high similarity between subsequent tasks\" (lines 30, 136). Some support for this claim, either by way of illustration or references, will be very helpful to motivate this work for few-shot learning.\n2) Cite and discuss relation to [Snelson, 2003] and  [Lázaro-Gredilla, 2012].\n3) Cite a paper for spectral mixture kernel (line 214).\n4) \"Continuous normalising flows\" should be in the title.\n5) In table 3, NGGP+RBF does better than NGGP+Spectral. An analysis of the results will be useful. In particular, for this case, I would like to know if the use of NGGP does away with the need for covariance functions more complicated then RBF/squared-exponential.\n6) In limitations, can you also discuss in relation to sparse GP?\n7) In addition to lines 300 to 301, it will be useful to give concrete numbers and timing for the experiments that are illustrated in section 5.\n8) Lastly, this paper is set within the context of few-shot regression. However, NGGP could be more useful than just for this. Have the authors tried to use NGGP for, say, just normal regression tasks, or quantile regression? How about experiments tried in [Snelson, 2003] and  [Lázaro-Gredilla, 2012]?\n\nEdward Snelson, Zoubin Ghahramani, Carl Rasmussen. Warped Gaussian Processes.  Advances in Neural Information Processing Systems 16 (NIPS 2003)\n\nMiguel Lázaro-Gredilla. Bayesian Warped Gaussian Processes. Advances in Neural Information Processing Systems 25 (NIPS 2012)\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See Q6 above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "EE-xy2djxUw", "reviewer": "Reviewer_SDgP", "summary": "The authors propose a novel GP-based model for few-shot regression. The model makes GPs more flexible by adding Continuous Normalizing Flow to the output, specifically an ODE-based mapping. Moreover, this mapping is contextualized by making it input-dependent, which adds more expressivity to the predictive posterior. This flexibility makes the model better suited for modelling complex distributions (e.g. multimodality cases) and, as shown experimentally, is beneficial for few-shot regression, where tasks might differ (domain shift).\n\nThe proposed model is experimentally shown to perform better than the main competitor, Deep Kernel Transfer (DKT), on a variety of few-shot regression datasets.\n\nOverall, this is a nicely written paper with an interesting novel method and good experimental results. Some parts of the model, however, are not given enough explanation and testing. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n\nThe prosed methodology is solid. Bayesian non-parametrics, and GP specifically, is very suitable for tackling the task of few-shot regression. Its extension with CNF for more flexibility in the predictive posterior is a very sensible thing to do. To the best of my knowledge, the math is correct.\n\nThe method is well-tested for the given task of few-shot regression on 5 datasets.\n\nThe paper is well written and clear.\n\n### Weaknesses:\n\nThe paper attempts to do two things at once: propose a new model and apply it for a specific task of few-shot regression. Because of this, the model is not described in enough detail. Invertible ODE-based mapping is not described at all. I would like to see some synthetic experiments comparing the proposed model to a GP on a standard regression task, before moving on to few-shot regression experiments.\n\nThe two main problems of standard GPs wrt the few-shot learning task, that this work addresses, are limited flexibility of the posterior distribution and the assumption of high similarity between tasks. While the benefit of NGGP in terms of modelling more complex distributions is clearly explained and shown (e.g. the multimodal Power Dataset experiment), the other main claim is not well supported. The authors point that \"GPs assume similarity between subsequent tasks\" and claim that the proposed method (NGGP) is \"adaptive to dissimilarities and domain shifts\". However, they do not explain how exactly the proposed method is supposed to adapt to domain shifts. Explaining this as \"contextualization provided by the ODE-based mapping\" is way too vague, especially given that ODE based mapping is not discussed much at all. Even though the experiments show good performance, it could be attributed to the more flexible posterior. This adaptation claim definitely should be supported by a detailed explanation.\n\n\n\n### Other comments and typos:\n\nA flow transformation in the model is applied to the noisy GP outputs. I would like to see some more discussion about this design choice and its potential limitations.\n\n66: \"overweights\" -> \"weights\" ?\n\n120: in eq 5 sum is over n, but in the det you put k.\n\n144: ODE-based mapping should be described.\n\nFig 4: why do you plot different kernels? is it the best-performing ones?\n\nFig 4 (a) the due to the vertical scaling, we pretty much can not see the predictive mean (red) in the first two plots on the bottom.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the proposed method are described in the paper. Negative societal impact is discussed appropriately given the general theoretical nature of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers ultimately agree that the key idea proposed in this paper, namely the use of continuous normalizing flows (CNFs) to achieve highly flexible non-Gaussian predictive posteriors, applicable to few-shot regression and achieving strong results in that domain. This core idea is both natural and novel, and the empirical validation on the particular task of few-shot regression is strong.

While I am ultimately recommending acceptance, I want to echo the comments of a few of the reviewers that the somewhat narrow focus on few-shot learning seems to arbitrarily limit the potential breadth of impact of this paper. This is particularly true since the core methodological contribution -- namely the use of a CNF to achieve a non-Gaussian predictive posterior -- is largely separate from the adaptation to few-shot regression, which involves utilizing simple weight sharing in a deep kernel, alternating optimization over tasks. 

Finally, I'll note that because of the relative decoupling of the authors' core contribution and the mechanism they use to approach few shot learning, the argument made in the author feedback that they don't compare to deep GPs etc because these methods are not intended for few-shot regression is somewhat unconvincing. A number of papers have explored the combination of deep kernels with deep GP models, so the adaptation to DKT is quite straightforward. Because deep GPs result in (continuous) mixture predictive distributions, a number of deep GP papers presenting toy results very similar to your own Figure 1 as motivation. I would urge the authors to consider the above points, and how their papers impact may be broadened by including these comparisons.
Decision: Accept (Poster)