Inducing Gaussian Process Networks
Authors: Alessandro Tibo
Thomas Dyhre Nielsen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Gaussian processes
Kernel Methods
Classification
Regression

Abstract: Gaussian processes (GPs) are powerful but computationally expensive machine learning models, requiring an estimate of the kernel covariance matrix for every prediction. In large and complex domains, such as graphs, sets, or images, the choice of suitable kernel can also be non-trivial to determine, providing an additional obstacle to the learning task. Over the last decade, these challenges have resulted in significant advances being made in terms of scalability and expressivity, exemplified by, e.g., the use of inducing points and neural network kernel approximations.

In this paper, we propose inducing Gaussian process networks (IGN), a simple framework for simultaneously learning the feature space as well as the inducing points. The inducing points, in particular, are learned directly in the feature space, enabling a seamless representation of complex structured domains while also facilitating scalable gradient-based learning methods. 

We consider both regression and (binary) classification tasks and report on experimental results for real-world data sets showing that IGNs provide significant advances over state-of-the-art methods. We also demonstrate how IGNs can be used to effectively model complex domains using neural network architectures.

INDUCING GAUSSIAN PROCESS NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Gaussian processes (GPs) are powerful but computationally expensive machine learning models, requiring an estimate of the kernel covariance matrix for every prediction. In large and complex domains, such graphs, sets, images, the choice of suitable kernel can also be non-trivial to determine, providing an addi- tional obstacle the learning task. Over the last decade, these challenges have resulted in significant advances being made in terms of scalability and expres- sivity, exemplified by, c.g., the use of inducing points and neural network kernel approximations. In this paper, we propose inducing Gaussian process networks (IGN), a simple framework for simultancously learning the feature space as well the inducing points. The inducing points, in particular, are learned directly in the feature space, enabling seamless representation of complex structured domains while also facilitating scalable gradient-based learning methods. We consider both regression and (binary) classification tasks and report on experimental results for real- world data sets showing that IGNs provide significant advances over state-of- the-art art methods. We also demonstrate how IGNs can be used to effectively model complex domains using neural network architectures.
INTRODUCTION
Gaussian processes are powerful and attractive machine learning models, in particular in situations where uncertainty estimation is critical for performance, such as for medical diagnosis @Dusenberry al.] (020)). Whereas the original Gaussian process formulation was limited in terms of scalability. there has been significant progress in scalable solutions with Quinionero-Candlla & Rasmussen (2005) providing an early unified framework based on inducing points as a representative proxy of the training data. The framework by Sortiionnr--Candee
& Rasmussen (2005) has also been extended to variational settings (Titsias 2009: Wilson et al. 2016b] Bauer 2016), further enabling a probabilistic basis for reasoning about the number of inducing points (Uhrenholt et al. 2021). In
terms of computational scalability, methods for leveraging the available computational resources have recently been considered (Nguyen et al., 2019b] Wang al. 2019), with Chen et al. (2020)
also providing insights into the theoretical underpinnings for gradient descent-based solutions in correlated settings (as in the case for GPs).
Common for most of the inducing points-based approaches to scalability is that the inducing points live in the same space as the training points (see e.g. (Snelson & Ghahramani 2006 Titsias 2009
Hensman et al. 20013) Damianou & Lawrence 2013)). ). However, learning inducing points in the input space can be challenging for complex domains (e.g. over graphs), domains with high dimensionality
(e.g. images), or domains with varying cardinality (e.g. text or point clouds) (Lee et al.) 2019 Aitchison et al. 202T). More recently, methods for reasoning about the inducing points directly in embedding space have also been considered, but these methods often constrain the positions of the inducing points (Wilson et | (216a) or the structure of the embedding space (Lázaro-Grediila & Figueiras- Vidall 2009 Bradshaw et al. 2017), they rely complex inference/lerrning procedures (Aitchison et al. 2021)
In this paper, propose inducing Gaussian process networks (IGN) as simple and scalable framework for jointly learning the inducing points and the (deep) kernel (Wilson et al. (016a). Key to the framework is that the inducing points live in unconstrained feature space rather than in the input space. By defining the inducing points in the feature space together with an amortized pseudo-label function, able represent the data distribution with simple base kernel (such as

---

6
5
22
7122
T22
g(x3)
22
T T(233
23
9(.)
g x
6
5
9 (x4
21 1(
T3
44
To the left four MNIST digits are embedded in the feature space by the neural network g. To the right, the feature space where both features and inducing points exist. The observations associated with the inducing points 21, 22, and 2 given by the pseudo-label function while the predictions associated with g g(x1), ), g (22)), g (axs), and (xxa) are estimated using the GP posterior.
Figure 1:
the RBF and dot-product kernel), relying the expressiveness of the learned features for capturing complex interactions.
For learning IGNs, rely maximum likelihood-basee learning objective that is optimized using mini-batch gradient descent (Chen et al. 2020). This setup allows the method to scale to large data sets as demonstrated in the experimental results. Furthermore, by only having the inducing points defined in feature space, we can seamlessly employ standard gradient- gradient-based techniques for learning the inducing points (even when the inputs space is defined over complex discrete/hybrid objects) without the practical difficulties sometimes encountered when learning deep neural network structures.
We evaluate the performance of the proposed framework on several well-known data sets and show significant improvements compared state of the art methods. We provide a qualitative analysis of the framework using a two-class version of the MNIST dataset. This is complemented by a more detailed quantitative analysis using the full MNIST and CIFARIO data sets. Lastly, demonstrate the versatility of the framework, we also provide sentiment analysis results for both text- based and a graph- dataset derived from the IMDB movie review dataset.
THE INDUCING GAUSSIAN PROCESS NETWORKS (IGN) FRAMEWORK
We start by considering regression problems, defined over an input space X and label space of observations E, modeled by a Gaussian process:
f ~ GP(O,,((..)) y f()+ x YEX,VER,
(I)
where k(;;)) X'XX' IR denotes the kernel describing the prior covariance, and //(0, o.) is the
noise associated with the observations. We assume access to a set of data points D = (xx; y/: } generated from the model in Equation [ and we seek to learn the parameters that define A and a.
in order to predict outputs for new points æ, in X. In what follows we shall use X and to denote (......, )T and (y1,... .. yr)",, respectively.
Firstly, we propose to embed the input points using neural network ge, : + Rd parameterized by 0y. Secondly, for modeling the kernel function k we introduce set of m inducing points Z (zi...., Zm) T 2; E Ed together with a (linear) pseudo-label function re, Rd R parameterized by 0,. We will use r = (ro, (z)),... To, (zm to denote the evaluation of r on Z, where r will play a rôle similar to that of inducing variables (Quiñonero-Canded 2005). In the remainder of this paper, we will sometimes drop the parameter subscripts O, and 0, from g and r for ease of representation. An illustration of the model and the relationship between the training data and the inducing points can be see in Figure II
Rasmussen
We finally define Rdd x Idd + R. as the kernel between pairs of vectors in Idd, In particular, we denote with
(Kxx)y = h(g(z:), (Kxz)i k(g(m;); 2,)
(K2x); = k(z,, g(x,)) (Kzz)ij k(2,, 2;),
2

---

the four matrices corresponding the kernel evaluations for all pairs of observed inputs and inducing points. The joint distribution over the function values at the observed inputs and the inducing points now given as
y r ~ N {o.| Kxx + oz21 Kzx
Kxz Kzz
(2)
where we slightly overload notation by using r to denote the function values associated with the inducing points, though they do not correspond to real observations. For given data set D, goal is to jointly learn {Z, gg. ,,, o.} by considering the marginal likelihood:
plyl x 0) No. Kx12);
where ÿ is the predictive mean:
ÿ = Kxz
(4)
and K x|2 is the posterior kernel given the inducing points, i.c.
Kx12 = (Kxx + 1) KxzK22 K2x.
(5)
Note that the kernel K z is implicitly parameterized by Og through the embedding function ge.- Furthermore, by jointly optimizing 0, together with the other model parameters we establish the connection between f and r.
As in (Chen et al. 2020), we define our objective function in terms of the log-likelihood of the posterior of Equation
(0:
yy 9) x2 ÿ) log xxzz
n 1og(2
(6)
2
which we can minimize wrt. the parameters using mini-natch gradient descent with appropriate scaling factors for the gradients (Chen el al. 2020). The gradient of E(0; wrt. 0 can be expressed
as:
Va ((0:D) K
K Xiiz (y Voù tr(K xiz VoKx1z)
+ k x 2
(y 8) D) K x VoKxzz
where (rA)) represents the trace of a matrix A. Note that in Equation 7 both ý and K Kxiz depend on O and that the joint optimization of the model parameters, in particular 0,,, establishes the connection between the Gaussian process and the pseudo-label function.
Based on the chain rule of differentiation, we see that the gradient of Equation 7 with respect the inducing points Z and the parameters 0, of the pscudo-label function do not depend on Vge,, that is the gradients of go, have no impact on the updates of Z and 0,.
Proposition The gradients V Z ((; D) and Vo, ((: D) do not depend Vgo,-
This is a key advantage of IGNs as learning the inducing points is therefore not influenced by the gradient of potentially complex embedding function g and the entailed optimization difficulties. Additionally, as the inducing points (only) exists in IRd, the underlying learning framework for the inducing points is indifferent the structure of the input space X' and whether it is discrete or continuous or defined over, e.g., graphs or images.
Once the IGN parameters = {Z, 0g. O,, a.} have been learned, we can find the predictive distribu- tion for input points X. and inducing points Z, by first considering the joint distribution over the associated function values
Kzz Kx.z
.. Kx.X.)
Kzx.
~ NONN0,
0,
which in turn gives the predictive distribution
p(f.Ix.,0) = (f.,
(8)
where
f. -xx,zKzfrr
Kx. Kx.x. Kx.zKzZKxx.
3

---

Finally, IGNs can straightforwardly be extended to solve classification tasks. For ease of exposition, we only consider binary classification problems, but the general method can straightforwardly be extended to a multi-valued setting _ That is, we assume that the label space is given by ) {0, 1} and that we have access to a data set D = {(x y:) R== where the underlying data generating process is defined by a latent function ((x with GP prior
f~ GP (()))
and y/æ (f(x)), where (--) is the cumulative Gaussian function; see (Rasmussen & Williams 2006). In order to perform maximum likelihood estimation of the model parameters, we approximate the likelihood using Laplace approximation. For the sake of compactness, defer the full derivation to Appendix B
2.1 COMPUTATIONAL COMPLEXITY
The main aspect related to the computational complexity of the IGN framework concerns the kernel computation Kz x- If n, is the number of inducing points and is the mini-batch size (recall that for training we use mini-batch gradient descent), the complexity of computing the kernel is cubic wrt 72z, i.e. On However, at inference time, we can omit the term connected K Kzz. since it only needs to be efficiently computed once (Sharma et al. (013). If we at this phase further assume that n, > b, the complexity reduces to the computation of K zx and Kxx, i.c. O( b).
RELATED WORKS
The combination of kernels and neural networks has previously been explored, most notably in the context of deep kernel learning (Wilson et al. 2016a). Using deep neural network architecture, (Wilson et al., 20166) transform the input vectors into feature space based on which a base kernel applied (here the RBF kernel and the spectral mixture base kemel (Wilson & Adams 2013) are used). The kernel parameters and the neural network weights are jointly learned by maximizing the marginal likelihood of the Gaussian process, but relying on pre-training of the underlying deep neural network architecture. This work has been subsequently extended into a variational setting (Wilson et al., 2016b), also providing support for multi-class classification but again relying on a somewhat cumbersome pre-training.
A key difference between our approach and Wilssnn et al. 201 b) is in our choice of inducing points. For instance, in (Wilson et al.) 201 6a b). the inducing points are placed on regular multi- dimensional lattice based on which the deep kernel is evaluated (van Amersfoort et al.) 2022). In IGNs the inducing points are (only) defined in feature space, where they are treated as unconstrained parameters and learned jointly together with the neural network weights and (any) kernel parameters. The pscudo-label function of IGNs plays role similar to that of inducing variables (Wilson et al.) 2016b), but where each inducing variable is defined by its own set of variational parameters, the pseudo-label function offers an amortized approach by having the parameters of the pseudo-label function being shared among all inducing points. Furthermore, as shown in Section 4 our learning scheme is end-to-end and does not require any pre-training of networks in (Wilson al. 2016a).
Finally, IGNS rely on mini-batch maximum likelihood-based gradient descent (Chen et al.. 2020).
thus avoiding the GP-KISS kernel approximation of (Wilson et al., 201 (0) and the grid layout of the inducing points together with the variational-baat based sampling setup of (Wilson et al. 2016b).
Several other related works exploit inducing points. For instance, TTitsias 2009 Hensman al. 2013 Damianou & Lawrence 2013) propose to maximize a lower bound of the exact marginal
likelinood to learn the inducing points in the input space, in contrast to our IGN framework where the inducing points are defined in (transformed) feature space. Using inducing point in a transformed is also explored in, e.g., L.rdill-Gredillaa & Figueiras- Vidal 2009), but here the transformed space is fixed, e.g., in the form of the frequency domain of the input space. In our IGN framework, we instead use transformation that is jointly learned together with (deep) kernel; moreover, our inducing points can (in principle) also live in space different from the embedding space. Closely related to IGNs is (Snelson & Ghahramani 2006), where inducing points are jointly learned with kernel
For the experiment results, (see Section classification tasks.
we have used one-vs-all approach to solve the multi-class
4

---

Table 1: Comparison of root mean square enror (RMSE) and negative log-likelihood (NI.L) of different GPs on the benchmark datasets. The best results are in bold (lower is better).
IGN (OURS)
SGGP
DKL
DIWP
DATASET
RMSE
NLL
RMSE
NLL
RMSE
NLL
RMSE
NLL
LEVY
0.17
0.9810.08
0.27
4.59
0.26+0.03
1.2640.06
0.60
1.26.0.02
GRIEWANK
0.05
0.76+0.02
0.07
0.87
0.15+0.07
1.44+0.77
0.08
0.93
BOREHOLE
0.00=0.00
0.41+0.03
0.17
1.84
0.12+0.03
1.14
0.04
0.93
PROTEIN 0.64
1.19_0.04
0.66
0.91:1.42
0.99
1.13:10.05
0.88
1.39
PM2.5
0.31
1.1610.03
0.29
7.1013.24
0.63
2.32
0.82.1..05
1.35
ENERGY
0.74
1.02+0.01
0.79
1.24
0.71
1.21
0.68g0..1
1.21
BIKE-HOUR
0.01
0.28+0.00
0.22+0.00
0.51
0.13+0.03
0.9640.0s
0.60+a.06
1.21
QUERY 0.07
0.77+0.02
0.05
1.97
0.07+0.03
1.1640.36
0.41
1.34
parameters using gradient descent. However, contrast to IGNs, the continuous optimization of Z proposed by |Snelson & Ghahramani (2006) is considerably more simplified as the inducing points are learned in the input space. Similarly to the proposed framework, Aitchison et al. (202I) exploit inducing Gram matrix. The Gram matrix are used to sample inducing points in the feature space, and subsequent GP predictions are conditionally sampled on the inducing points in the features space. This process, in contrast to IGNs, relies complex doubly-stochastic variational inference process.
The papers cited above are representatives of GP methods that are methodologically related to the proposed IGN framework. Not all of the cited methods are, however, in line with state of the art in terms of, e.g., accuracy results, hence the experimental section below also includes descriptions and comparisons of other GP methods, which forms the basis for the empirical evaluation and analysis.
4 EXPERIMENTS
In this section, we present and discuss experimental results showing the potential of the IGN framework on both regression and classification tasks. Both settings are aimed at showing the ability of IGNs to jointly learn the inducing points and the model parameters. In the experiments, we also compare IGNs against other state of the art methods that either learn the kernel parameters or the inducing points. The variety of the experiments aim to provide insight into the properties of the framework and show the potential of IGNs for diverse real-world datasets, outperforming state-of-the-aat methods in the majority of the cases.
4.1 REGRESSION TASKS
Following the setup described in (Chen et al. 2020), we compared our approach to deep kernel learning (DKL) (Wilson et al. (0T6a)), deep kernel processes (DIWP) (Aitchison et al. 2021). and sgGP (Chen et al. (020) on several simulated and real regression benchmark datasets. Briefly, DKL exploits inducing points (in the embedding space) and an approximation of the kernel to scale to large datasets, DIWP defines deep kernel processes where positive definite Gram matrices are progressively transformed by nonlinear kernel functions and by sampling from (inverse) Wishart distributions.
The sizes and feature dimensions for each dataset are reported in Appendix (see Table 3) The simulated datasets (Levy. Griewank and Borehole) are from the Virtual Library of Simulation Experiments while the real datasets (Protein, PM2.5, Energy, Bike-hour, and Query) are from the UCI repository
For each dataset, we repeated the experiments 10 times. In each experiment, the dataset was randomly split into 60% training set and 40% test set. Furthermore, the training set was normalized have 0 mean and standard deviation, and the test set was scaled accordingly. For all the datasets, we considered simple neural network with stacked dense layers with 128 units and ReLU activations.
http: eduuml
5

---

Levy
Griewank
0.7
DKL
25
30
0.7
DKL
20.0
0.6
15.0
RMSE
IGN
0.5
IGN
12.5
0.4
10.0
0.4
10
7.5
0.3
022
-5
0.1
2.5
2
8
18
32
64
128
258
512
2
6
16
32
128
258
512
Grid size / Inducing points
Grid size / Inducing points
Figure 2: RMSE of DKL (circles) and IGN (triangles) for Levy and Griewank datasets in function of the grid size (for DKL) and number of inducing points (for IGN). The colors encode the time (in seconds) that each epoch takes to be completed. DKL RMSE improves with the grid size but the computational time drastically increases. On the other hand, IGN requires more inducing points but the results overall better and the computational time remains stable. All the models were run CPU for having a fair comparison.
On top of those we added final feature layer with 64 units. We used 512 inducing Dointshhavinn the same dimension as the feature layer (i.e., 64 dimensions somewhat arbitrarily chosen). an RBF kernel with y = 1.0 for all dimensions, and linear pseudo-label function. The model weights are updated with the Adam optimizer for 500 epochs on
of size 128.
Table T summarizes the results, where for each dataset we report the average and standard deviation of the root mean squared error and negative log-likelihood on the test set over the 10 experimental repetitions. For Levy. Borehole, and Bike-hour datasets, approach outperforms by a large margin the other methods. While for PM2.5, Energy, and Query datasets our approach performed slightly worse than the others, we still obtained comparable results. Overall, our method remained stable as the standard deviations over 10 experiments are small and comparable to sgGP. Additional results using other baseline methods can be found in Appendix C
Finally, for illustrating the effect of imposing grid structure over the inducing points wrt. computation time and accuracy, we have made a comparison of IGN and DKL (Wilson al.) 2016a). Recall that DKL requires a re-dimensional grid whose cost grows exponentially with the amount of data. In this experiments, we use the exact same architecture in terms of layers and kernel as described in Section 4.1 for both IGN and DKL. Figure shows the root mean square error for Levy and Griewank datasets. Our method, although requiring more inducing points, outperforms DKL in terms of computational time and root mean square errors (evaluated on the test sets). It not possible to conduct experiments for DKL with more than 8 inducing points as we hit the memory limit.
4.2 TOY-MNIST
Moy-MNIST is the same as MNIST, but limited digits belonging to class or The aim of this experiments is to show the ability of IGNs to classify relatively ambiguous digits, and provide insights into the inducing points and the uncertainty associated with the predictions. In this case, the training and test sets consist of 1 11,339 and 1, 1,850 images, respectively. The image labels associated with classes and are replaced with and -1, respectively, and we assume the exact same model specification as in Section 4.1 but now only using 64 inducing points.
By repeating the experiment 10 times we obtain a test set accuracy of 99.25 + 0.09%. For each of the 64 inducing points, we retrieve the closest image in the training set based on the chosen kernel in feature space, after which we refine each of the images (relative to the distance to the closest inducing point) using gradient descent with respect to the input image. The 64 inducing points (in the feature space) and digit images corresponding to the inducing points are shown in Figure For this experiment, the intuitive interpretation of the images associated with the inducing point also provides a degree of explainabilty, even though the inducing points are learned in feature space. As with standard Gaussian processes, IGNs retains the feature of uncertainty estimation. This illustrated in Figure which depicts a scatter plot of the images in the test set, given in terms of the expected
"We randomly initialized the inducing points with same initializer as the final feature layer, i.c. Xavier uniform initializer.

---

5
5
5
5
5
5
5
5
6
6
6
6
5
5
6
5
5
6
5
5
6
5
G
5
6
5
5
5
6
(
5
5
5
5
5
5
5
6
5
5
5
6
6
6
5
5
6
5
6
5
5
G
5
6
5
5
b 5
5
5
5
6
5
5
Figure 3: To the left the 64 inducing points projected into two-dimensional space. To the right the 64 pseudo images corresponding the 64 inducing points for the Toy-MNIST experiment.
0.70
0.50 55
0.60
Varlance
0.66
0.60
0.40
0.35
5 0.63
6
...06.0.88
0.30
Mean
Figure 4: Mean (X-axis) against variance (y-axis) plot for the Toy-MNIST dataset. Orange and blue dots correspond to images labeled with and 5, respectively. Images associated with low variances and high mean (in absolute value) correspond to clear digits. On the other hand, images with high variance and mean close to the decision boundary, i.e. 0.5, correspond to ambiguous digits.
value of the predictions (X-axis) and the variance of the predictions (y-axis). The two distinct orange and blue clusters contain the digits corresponding to classes 6 and 5. The figure also illustrates some of the extreme predictions: the two digits with means far from 0.5 and low variance correspond to clearly distinguishable digits, while the digit with a mean close to 0.5 and relatively high variance corresponds to a more ambiguous image.
4.3 MNIST AND CIFAR10
In this section evaluate IGN computer vision classification tasks. Specifically, here we focus MNIST and CIFARIO. 10. We compare our approach against deep kernel processes (DIWP) (Aitchison al. 2021) as well as Deep Gaussian Processes (DGP) (Damianou & Lawrencel 2013). NNGP (Blun- dell et al. 2015), and stochastic variational deep kernel learning (SVI-DKL) Wilson et al. 2016b). Briefly, DGP uses graphical models nest layers of Gaussian processes, and NNGP adopts back- compaiiiileee algorithm for learning probability distribution on the weights of neural network to estimate the uncertainty of the model.
For the sake of simplicity we decompose with a one-vs-all approach each multiclass task into 10 binary tasks. We again used the same model as Section 4.1 (this model comparable to the other competitor models in terms of number of parameters), but with 32 and 16 inducing points for MNIST and CIFARIO, respectively (the same number of inducing points was used by all methods). For each of the datasets, repeated the experiments 10 times and report the average accuracy and negative logeliiel in Table We observe that our method outperforms all the others in terms of accuracy, while keeping the standard deviation comparably small. The negative log- likelihood is better for CIFAR- 10, but worse for MNIST, suggesting that training a full multi caalticcassiiiiattion model could be beneficial, (Murphy 2022).
7

---

Table 2: Test accuracy (ACC) and negative log-likelihood (NLL) comparison of MNIST and CIFARIO for IGN approach against DGP, NNGP, DIWP, and SVI-DKL.
IGN (OURS)
DGP
NNGP
DIWP
SVI-DKL
DATASET
Acc
NLL
AcC
NLL
Acc
NLL
Acc
NLL
AcC
NLL
MNIST 98.0.o.u 0.6+0.0 96.540.1 0.1 96.5+0.0 0.1+0.0 97.7+0.0
0.1 97.4+0.1 1.3+0.0
CIFARIO 51.020.2 1.120.0 46.820.1 1.5.0.0
47.420.1 1.520.0
50.520.1
1.420.0 48.1 1.620.0
4.4 STRUCTURED DATASETS
In this section, we investigate the potential of our model applied to datasets with more complex structure. Specifically, consider classification tasks on text and graphs, respectively. The datasets for both tasks are derived from IMDB:
MIDB-TEX" (Maas al. 2011) consists of 25,000 training reviews and 25,000 test reviews. Positive and negative labels are balanced within the training and test sets. We preprocess the data by keeping the top 20,000 words and limiting the length of words in a review to 200.
IMDB GRAPH is social network dataset contained in the collection described in (Yanardag & Vishwanathan 2015). It consists of constructed genre-specific collaboration networks where nodes represent actresses/actors who are connected by edge if they have appeared together in a movie of a given genre. Collaboration networks are generated for the genres Action and Romance for this dataset. The data then consists of the ego-graphs for all actresses/actors in all genre networks, and the task is to identify the genre from which an ego-graph has been extracted. It contains 1,000 graphs with labels balanced among the two classes.
For both experiments, we aim to show that our approach can jointly handle complex data and complex models for extracting features, hence the aim is not to compare against state-of-the-ar methods. In particular, the two IGN instantiations for the two datsets differ only in the embedding functions/feature extractors being used. For both datasets, we also provide baseline accuracies obtained by two classifiers sharing the architectures of the feature extractors in the IGNs.
For IMDB-TEXT, we used 128 inducing points and trained a transformer-based model (Vaswani et al. 2017) to learn features from the reviews. We repeated the experiments 10 times for the baseline and our approach obtaining accuracies of 84.35% + 0.10 and 84.43% + 0.96, respectively. Similarly to Section 4.2) we can also inspect the learned inducing points. For example, included below are the two training set examples (positive and negative) that are closets to two randomly chosen learned inducing points 5
was literally preparing to hate this movie so believe me when say this film is worth seeing my score 7 out of 10.
can't believe that so many are comparing this movie to Argento's 's ... you''s re looking for good horror movie look elsewhere.
Similar to Section 4.2 we also here see that by associating the learned inducing points with training instances provides an additional element of explainability the inducing points.
For the GRAPH, we follow the setup described by Tibo et al. (020)) and represent each graph as set of neighborhoods. We performed a 10 times 10 fold cross- validatio for both the baseline and IGN. We ran 100 epochs of the Adam optimizer with learning rate 0.001 on mini-batches of size 32. We obtained accuracies equal to 72.49% 0.60 and 73.40% 0.63, for the baseline and IGN, respectively, Without any particular effort, in terms of fine tuning model structure and parameters, our method outperforms most of the graph neural networks reported in (Nguyen et al. 2019a).
"More correspondences between training set reviews and inducing points reported in the supplementary material.
8

---

0.50
Average RMSE
0.80
0.80
0.25
0.18
0.70
0.25
0.00
Average
Varisane
0.97
0.58
0.85
0.97
0.81
0.94
0.88 0.96
0.74
0.95
Energy
Query
Figure 5: Average RMSE (first row) and variance (second row) estimated on 10 runs for PROTEIN, PM2.5, ENERGY, BIKE-HOUR, QUERY, and IMBD-TEXT. For IMDB we report the classification error. The variance constantly decreases with the number of inducing points for all the datasets.
ABLATION ON NUMBER OF INDUCING POINTS
Here study the impact of the number of inducing points for the regression and classification tasks defined in the previous sections. The variance of Gaussian processes estimations decrease with the number of training points (Rasmussen & Williams 2006). In case, the training points are replaced by inducing points but we retain similar result. For the sake of completeness, we slightly reformulate the proposition and provide proof in the supplementary material.
Proposition 2. The variance of test point (æ., f.) given set of inducing points Z can never increase by the inclusion of an additional inducing point z & Z.
When training IGNs with different number of inducing points, there are no guarantees that the same subset of inducing points will be learned during training. However, we still observe consistent decrease in variance when the number inducing points increases. For the regression tasks, we report here the results for all the real datasets described in Section 4.1 For the classification task we focus on IMDB- TEXT described in Section 4.4 In all the cases, we repeated the experiments 10 times varying the number of inducing points (4,8,16.32,66 (2.32,64,128,2566..6 1282256.11)) Figure depicts the root mean square error (on the top), and the average variance with error bars (on the bottom) for the different datasets. From the figure number of inducing points for all datasets. For some of the datasets with only a few inducing points, the error bars lowe, but the root mean square error is always higher. Finally, it worth noticing that even when the root mean square error stabilizes, the variance keeps decreasing with the number of inducing points.
that the average variance consistently decreases with the
CONCLUSIONS AND FUTURE WORK
While Gaussian processes are powerful machine learning methods that can estimate uncertainty, they remain intractable on large datasets. Several works have addressed this problem using inducing points but, to the best our knowledge, these methods remain limited to comparatively simple datasets. In this paper, we introduce framework for learning Gaussian processes for large and complex datasets by combining inducing points in feature space with neural network kernel approximations. We empirically showed the ability of our method standard machine learning benchmarks well as structured (graph and text) datasets, with the proposed method outperforming other state-of-the-art approaches.
The flexibility our IGN framework enables the combination Gaussian process training with complex deep learning architectures. We believe that method provides basis for further research calibration uncertainty estimates deep neural network models. Furthermore, as part of future work, also plan to investigate how position IGN within fully variational setting (Titsias 2009). exploring the hierarchical structure of the model, possibly in the context of deep kernel processes (Aitchison et al. 2021).

---

ETHICS STATEMENT
The work reported in this paper not related to human subjects, practices to data set releases, discriminatiio/iiss/aaaan concerns, and research integrity issues. The work is also not related to any particular application domain. The paper proposes a simple but general framework for learning inducing points sparse Gaussian processes, also contributing with insights/explainabbiity into the model (through the learned inducing points) and potentially improved accuracy and uncertainty esti- mates. Depending on the application domain, the significance and interpretation of these insights and estimates may require careful scrutiny by domain experts, in particular for safety critical applications.
REPRODUCIBILITY STATEMENT
An open-source implementation of our IGN framework has been uploaded supplementary material together with the paper. Upon acceptance of the paper, the code will be placed in a public GitHub repository. The implementation also contains detailed information about the data sets, parameters, and random seeds used in the experiments.
REFERENCES
Laurence Aitchison, Adam Yang. and Sebastian Ober. Deep kernel processes. In International Conference Machine Learning, pp. 130-140. PMLR, 2021.
Matthias Bauer. Understanding probabilistic sparse Gaussian process approximations, In In proc. of NIPS, 2016.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight Uncertainty in Neural Networks. In In proc. of ICML, 2015. ULL/: 05424
John Bradshaw, Alexander G. de G. Matthews, and Zoubin Ghahramani. Adversarial examples, uncertainty, and transfer testing robustness in Gaussian process hybrid deep networks. 2017. doi: 10. URL
02476 LrXiiv1707.024751
[stat].
Hao Chen, Lili Zheng, Raed Al Kontar, and Garvesh Raskutti. ent descent in correlated settings: A study on Gaussian processes. In In proc. of
Stochastic gradi-
NeurIPS, 2020. URL httpps: .ererieee..ppp..p oc [ob22455aFF3000e4442354303333333
httpr/iissccpppppe///22222e httppapee//22//heeee. .ttml
Andreas Damianou and Neil D Lawrence. Deep Gaussian processes. In Artificial intelligence and statistics, pp. 207-215. PMLR, 2013.
Michael W. Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M. Dai. Analyzing the role model uncertainty for electronic health records. In Proc. of the ACM Conference Health, Inference, and Learning, CHIL, New York, NY, USA, 2020. doi: 10. 00.115/3366855.33844577
3368555 3384457
James Hensman, Nicolò Fusi, and Neil Lawrence. Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, pp. 282-290, 2013.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Tch. Set transformer: framework for attention-based commutaaion-invariaan neural networks. In JCMI, pp. PMLR, 2019.
Miguel Lázaro-Gredill and Aníbal Figueiras- Higuel Vidal. Inter-domain Gaussian processes for sparse infer- ence using inducing features. Advances in Neural Information Processing Systems, volume 22, 2009.
Andrew Maas. Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In proc. of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
10

---

Kevin P. Murphy. probml
Probabilistic Machine Learning: An introduction. MIT Press, 2022. URL
Dai Quoe Nguyen, Tu Dinh Nguyen, and Dinh Phung. Universal graph transformer self-attention networks. arXiv preprini arXiv: arXiv: 19.9. 18555, 2019a.
Duc-Trung Nguyen, Maurizio Filippone, and Pietro Michiardi. Exact Gaussian process regres- sion with distributed computations. In Proc. of the 34th ACM/SIGAPP Symposium on Applied Computing, SAC, pp. 1286-1299 1295, 2019b. doi: 10.114453297288032974444
Joaquin Quiñonero--anme Candela and Carl Edward Rasmussen. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research, ((65):1939-19555 2005. ISSN 1533-7928.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, 2006.
Girish Sharma, Abhishek Agarwala, and Baidurya Bhattacharya. A fast parallel gauss jordan algorithm for matrix inversion using cuda. Computers & Structures, 128:31-37, 2013.
Edward Snelson and Zoubin Ghahramani. Sparse Gaussian processes using pseudo-inputs. Advances in neural information processing systems, 18: 1257, 2006.
Alessandro Tibo, Manfred Jaeger, and Paolo Frasconi. Learning and interpreting multi-multi-instance learning networks. J. Mach. Learn. Res., 21:193-1, 2020.
Michalis Titsias. Variational Learning of Inducing Variables in Sparse Gaussian Processes. proc. of the Twelth International Conference on Artificial Intelligence and Statistics, pp. 567-574. PMLR, April 2009, URL URRLitttps:///ooeeddiiin 1938-7228.
mlr. wtt..r/.ppeeee//tttttttissssee.e html ISSN:
Anders Kirk Uhrenholt, Valentin Charvet, and Björn Sand Jensen. Probabilistic of induc- ing points in sparse Gaussian processes. Jul 2021. URL http:::/aaaxxii..og//a///////// arXiv: 2010.09370.
09370
Joost van Amersfoort, Lewis Smith, Andrew Jesson. Oscar Key, and Yarin Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. 1.XX:::2022.14009, Mar 2022. URL
http:///wwwwvvvv.aaaa/// abs///////// 2102 11409
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 5998-600s 2017.
Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q Weinberger, and Andrew Gordon Wilson. Exact Gaussian processes on a million data points. Advances in Neural Information Processing Systems, 32: 14:44448--4555 2019.
Andrew Wilson and Ryan Adams. Gaussian process kernels for pattern discovery and extrapolation. In proc. of the 30th ICML, pp. 1067-1075. PMLR, May 2013. URL https:://prcceee.....
ml r. www.pessss.. / wi sonn3nn3333 3. html
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In proc. of the 19th International Conference Artificial Intelligence and Statistics, pp. 370-378. PMLR, May 2016a. URL httttt::///ooceeeee
mlr www.ppressssseee www.ee//www/wwwsssoeir..
html
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Stochastic Variational Deep Kernel Learning. proc. of the 29th conference on Neural information Processing (NIPS), 2016b. URL
Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In proc. of the 2 ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1365-1374, 2015.
11

---

A ABOUT THE VARIANCE OF INDUCING POINTS
Proposition 2. The variance of test point f.) given a set of inducing points Z can never increase by the inclusion of an additional inducing point z & Z.
Proof. Let S and R be two sets containing inducing points. Let K fu be the variance associated with fa conditioned only on the inducing point f- conditioned only on the inducing point in R U S. If xxxx, x. is the top-left scalar in K faar representing the variance of f., then we prove that f,,r K x.
Let Ke be the variance associated with
Let K be the covariance
matrix of the Gaussian process having the following block structure:
Kx.x.
Kx_R
Kx.s]
K =
KRX.
KRR
KRS
Ksx.
KSR
Kss
where the subscripts represents the two sets (among the observation {a. and inducing points in R and S) used to evaluate the kernel matrix. We compute now K f., as
Kx.x.
Kx.s Kss
Xx.R KSR
K K,,,e/==
Knn KRx
Ksx.
Kxx,x. Ksx.
K'X.s
(10)
K'ss
Note that Xx,x. S Kx.x. Finally,
Kf.l.r- = K',x. - K' K'sx. S 8K',,,
which concludes the proof.
B IGNS FOR CLASSIFICATION
Here provide additional details about the full derivation (see Section 2.1 in the main paper) of IGNs for classification. For ease of exposition, we only consider binary classification problems, but the general method can straightforwardly be extended to multi-valued setting. For the experiment results, (see Section 4.3 in the main paper) we therefore use one-vs-all approach to solve multiclass classification tasks. We assume that the label space is given by {0. 1} and that we have access to data set D = {(xi, }:) where the underlying data generating process is defined by a latent function fax with GP prior
~ TP(P k(;)))
and y x ~ ((f())), where ((-) is the cumulative Gaussian function; see (Rasmussen & Williams 2006
For inference, we calculate the posterior distribution X., 0) given by
Ply|X.. 0) / plulf-))(f. 0)df.
Assuming that f. ~ N(,, 2²) for some l and a we get (Rasmussen Williams 2006, Section 3.9):
p(y 2x 0) (aa with
V1+0*
where μ and o² can be found from the predictive distribution in Equation 8 (in the main paper).
For learning the IGN parameters 0 we perform maximum likelihood estimation, but using the Laplace approximation to derive approximation of the marginal likelihood. Firstly, the likelihood is defined
as
ply X, (
/lulf))(f1x,0011
(11)
Denoting by
f = arg max log p(gg((y))) log (f
12

---

and employing the Laplace approximation in Equation 11 goal is to maximize the following
log- log-likelihood approximation
00 D) = logl(y) + log Logg(ff], 0) log([A[) +
(12)
2
where represent the accumulated constant terms and
(V(logr(ylf)) + logp(f) ())
(13)
The form of p(flx.0) 0) is given in Equation 3 (in the main paper) and using the shorthand notation
a = KxzKzzro.
Kxx Kxz K 22 K2x,
the log-likelihood in Equation
12 can be expressed as
() ss logp(ylf) log(1A|)
log (K)(
2 1 a)- (f a) +c.
2
(14)
The term plylf fff in Equation 13
factorizes, hence W VV log((y)f)) is diagonal (Rasmussen &
Williams 2006 Page 43) with
Wa
VV (oggp(u)).)) = VV fi) V: fi (fi)
(fi)²
④ fi)
D(y: fi)
From Equation 14 VV lfg 0) K-1, therefore A W K-1, We finally approximating
the log-likelihood (omitting constant terms) as
((0 D) log pggp(y) f) log(1 - W -K
2
1 [ a a)-a) T (f- -a)
2
Lastly, we find f using Newton's approximation for T steps. Starting with f 0, for StST. we have
+ Vlogply)
where V Vlogp(uslft) = v.N(fs)/
C ADDITIONAL EXPERIMENTS ON REGRESSION TASKS
Following the setup described CChn al. 2020), we compared our approach PCG-based exact GP (EGP) (Wang et al.) 2019), sparse GP regression (SGPR) (Titsias] 2009). stochastic variational GP (SVGP) (Hensman et al. 2013), and sgGP (Chen et al.) 2020) on several simulated and real regression benchmark datasets. Briefly, EGP leverages GPU parallelization and conjugate gradients to compute the exact covariance matrix on large datasets, SGPR uses a variational formulation for sparse approximations that jointly infers the inducing inputs and the kernel hyper-parametee by maximizing a lower bound of the true log marginal likelihood, SVGP variationally decomposes Gaussian processes to depend stochastic gradient descent on the Gaussian process likelihood to learn the kernel hyper-parameters.
of globally relevant inducing variables, and sgGP exploits
D IMDB-TEXT INDUCING POINTS
We report the training set reviews close to the inducing points for the IMBD-TexI experiment presented in Section 4.4 for one of the ten runs. Recall that we use 128 inducing points. In this case, we obtained 60 unique inducing points corresponding to 32 positive and 28 negative training set reviews. The inducing points are all different each other but there are collisions when choosing the closests points in the training set. Note that the inducing points are balanced among the two classes. For the sake of space, for each review report the initial and last 25 words.
13

---

Table 3: Comparison of root mean square error (RMSE) of different GPs on the benchmark datasets. We report the mean and standard eπrоr of RMSE. The best results are in bold (lower is better). For query and borehole datasets, EGP is not able to fit due memory constraints.
DATASET
SIZE
D
IGN (OURS)
SGGP
EGP
SGPR
SVGP
LEVY 10,000
4
0.18
00.00
0.31 0.000
0.564..01
0.58
GRIEWANK
10,000
6
0.05
0.07
0. 0.1920.07
0.1310.00
0.09
BOREHOLE
1,000,000
0.00
0.17
0.18
0.1720.00
PROTEIN
45,730
9
0.64
0.66
0.69
0.72+6.00
0.68
PM2.5
41,757
15
0.31
0.29 +0.00
0.29 +.29+...0
0.64
0.54+t0.01
ENERGY
19.735
27
0. 0.74:0..2
20000
0.80 07
0.84
0.80 01
BIKE-HOUR
17,379
15
0.01
0.22
0.23 00
0.28 00
0.25
QUERY 100,000
4
0.08 00
0.05
0.06
0.06
1. positive- there have been several films about zorro some even made in europe e g alain delon this role has also been played by outstanding actors good performance of hadley as zorro he was quick smart used well his whip and sword and his voice was the best for any zorto
2. negative- i love the frequently masters of horror series horror fans live in a constant lack of projects like this and the similar project with gave ... up has to have payoff that exceeds build up not the other way around storytelling math 101 br br end of spoilers big oops
3. positive- i rented the film don't t think it got a theatrical release here out expecting the worse the previews made the film look awful i.. be proclaimed 'the worst film ever i recommend this film for anybody interested in the show a flawed but innovative and interesting piece of film
4. positive- one of the best of the fred astaire and rogers films great music by irving berlin solid support from randolph scott harriet nelson lucille ball ... jazzy and it's a great song br br fun all the way although i got tired of we joined the navy after the third time
5. positive forbidden planet rates as landmark in science fiction carefully staying within hard aspects of the genre science not fantasy nerds will love while still ... the edge destroying its creator just as it did thousands of centuries earlier to the krell br br maybe the krell had teenage daughters too
6. negative- the bad news is it's still really dreadful gave it because occasionally some of this slapstick parody actually seems funny br br this one and get this dvd back to the video store on time you'll 'Il really hate yourself if you have to pay a late fee
7. positive- nice character development in a pretty cool milieu being male i' 'm probably not qualified to totally understand it but they do a nice job but within this world it needed to happen good acting all around with something positive taking place in the lives of some pretty good people
8. negative- mercifully there's s no video of this wannabe western that a stay afloat vehicle for big frank at time when his career was floundering the ... you up late and having bout of insomnia but you can sit through it you've more than most of my movie buff friends
9. negative- bela lugosi is an evil who sends brides poisoned on their wedding day steals the body in his fake ambulance hearse and takes it home in discount store 2 for fl which i think a pretty accurate anyone paying more for this would be out their mind
10. positive its no surprise that busey later developed a in his this film is also poor decision but one enjoyed fully the first 5 wet myself some of best work by far rent or buy it today my vote a perfect 10 on the poo meter that
11. positive- before sky i saw diane tender performance in this otherwise of movie campers are invited to the camp of their youth and experience it comic acting turn by noted director sam raimi makes this movie you can pull out again and again like looking up an old friend
12. positive- the line of course is from the lord's prayer thy will be done earth as it is in heaven sweden especially its far north the ending is what you make of it i guess but it's not spoiling it to say daniel achieves what he set out to do
14

---

13. positive my certainly is fair looking woman this film is a lost gem a dead satire mockumentary of the early 90's hip hop scene this regard regard this movie like the 1000 islands of upstate new york it's wonderful little secret you want to keep to yourself 14. positive- i rented this film from netflix for two reasons was in the mood for what thought would be silly '50s sci fi also generally very good and the acting is much better than one might expect was particularly impressed with reeves jeff corey and walter reed
15. negative- brilliant book with wonderful characterizations and insights into human nature particularly the nature of addiction which still resonate strongly today br br for the ... normally excellent but inappropriately cast actors all in all a weak adaptation your three hours would be better spent reading or reading the book
16. negative- it's boggles the mind how this movie was nominated for seven oscars and won one not because it's abysmal or because given the collective credentials director hungry to be recognized it could've been morphed something better but what's left looks like film nobody was really interested in making
17. positive- comedy of funny proportions from the guys that brought you south park and most of the guys from this movie has utterly disgusting and ... turn the sport sour and its up ta coop ta fix it and along the way you will laugh alot that's 's all there is enjoy
18. negative- there was bugs bunny cartoon titled baby buggy bunny that was exactly this plot baby faced robbed a bank and the money in the ... the bugs bunny dvd it's was much more original the first time 1954 plus you'll 'lll get lot more classic bugs bunny cartoons boot
19. negative- first off really loved henry fool which puts me in a very small pool of movie goers parker posey is one of best actresses ... ride i' be happy to spoil this movie for you but it's been done it's rotten the fool franchise is dead long live henry fool
20. positive- imagine victorian literature slowly sinking into the of the increasingly distant past pulled down by the weight of its under skirts along comes television.... ... coarse have been made to modern tastes and without having felt preached to another bbc classic highly recommended this is how romantic literature should be
21. negative- i knew this movie wasn't going be amazing but thought i would give it a chance am a fan of luke wilson the movie without people getting annoyed the movie had its moments but i'm glad i didn't 't spend 50 see in the theater
22. negative- - have read several good reviews that have defended and the various aspects of this film one thing i see over and over is annoyance... ... of good and terrible acting i would recommend for a cheap thrill but hardly a diamond in the rough that micro budget horror
23. negative- this movic is like the thousand cat and mouse movies that preceded it the following may look like spoiler but it really just describes ... exiting the theater from holly movie and if you have ever felt that way too heed my warning stay miles away from this movie
24. negative- wonder who how and more importantly why the decision to call richard attenborough to direct the most singular sensation to hit broadway in many ... well michael douglas was in true forgot m absolutely wrong and you are absolutely right nothing like richard attenborough michael douglas musical
25. positive- this movie will go down down in history as of the greats right along side of citizen kane casablanca and on the waterfront someone ... do yourself and your family a favor and buy it immediately i' i'm still holding out hope for a special edition dvd one of these days
26. negative- this is high grade cheese fare of b movie kung fu flicks bruce wannabe lee is played by bruce think of course let's ... flashback for a scene just shown 3 minutes ago they must've ve thought that only one with attention disorder could fully understand this film br br
27. negative- the only previous gordon film i had watched was the kiddie adventure the magic sword 1962 though followed this soon after with empire o.... them then again this particular version is further sunk by the tacked on electronic score which is wholly inappropriate and cheesy in the extreme
15

---

28. positive- hilarious evocative confusing brilliant film reminds of or holy mountain lots strange characters about and looking for what is it laughed almost watch on screen or at his big slide show smart funny quirky and outrageously hot make more films write more books keep the nightmare alive
29. positive- the villian in this movie is one mean sob and he seems to enjoy what he is doing that is what guess makes him ... guess you can make up your own mind about the true ending i'm left feeling that only one character should have survived at the end
30. negative- okay what the hell kind of trash have i been watching now the mountain has got to be one of the most incoherent and insane good heroine this is the type of european horror film that could have been legendary if only someone had bothered to write structured screenplay
31. positive- i found this movie to be very good in all areas the acting was brilliant from all characters especially ms stone and character just gets audience which was misled by some faulty terrible reviews about the movie before it even started you won't regret it if you go see it
32. negative- a lot of horror fans seem to love scarecrows so won't be very popular in saying that i found to be rather boring involving killer scarecrows to my knowledge apart from dark night of the scarecrow which is much better i would recommend that over scarecrows any day
33. negative- david mamet's film debut has been hailed by many as a real thinking man's 's movie a movie that makes you question everybody and everything i.. unfulfilled and if you like me predicted ahead of time that margaret was going to be conned you will find this revelation just as unsatisfying
34. positive while this was a better movie than 101 dalmations live action not animated version think it still fell a little short of what disney ... as many disney films are here's hoping the third will be even better still because you know they probably want to make one
35. positive- - think i read this someplace joe johnston director of the film and also one of the guys who founded industrial light and magic for... ... first homer jr did not like the idea but he warmed up to it after the movie poster paperback novel came out and took off
36. positive- stephen king movies are a funny thing with me either really love them or loathe them some of the productions such as desperation ... very watchable and enjoyable adaption br for uk readers this production has most recently been shown on sci fi and sky thriller horror channels
37. positive- if you' re researching ufo facts then this video is very important the of the video is the comments made by buzz he is without a in details should not detour your from viewing this video if nothing else it is interesting and i recommend you watch with an open mind
38. positive- rupert friend gives performance prince albert that lifts the young victoria to unexpected levels he is superb as we know queen victoria fell ... believe for minute she victoria no real sense of period it may no have been her fault but her prince deserved the crown
39. positive- seeing moonstruck after so many years is a reminder of how sweet and funny this film was when it first appeared who knew that cher ... used to be its best entertainment with no social significance whatsoever if they'd d only lost that's along the way it would have been perfect
40. negative- -i first learned of the wendigo many years ago in one of alvin scary stories books
according to that story the wendigo after calling your count br br anyway avoid patricia clarkson and erik per sullivan dewey on malcolm in the middle have done far better than this
41. positive well maybe not immediately before the rodney king riots but even a few months before timely enough my parents said that they saw it but either way grand canyon is a great movie it kevin kline as my favorite actor also starring mary mary louise parker and alfre woodard
16

---

42. positive finally movie where the audience is kept guessing until the end what will happen well we all kind of know that the lives of ... his drug and sex addictions and a father who finally discovers exactly what happened the day of the robbery this movie will get you thinking
43. negative- wow don't even really remember that much about this movie except that it stunk br br the plot's basically a girl's parents neglect her ... you do see it don't I expect much 1 out of 10 br br seriously if you want a pokemon movie rent pokemon the first movie
44. positive- the group of people are travelling to in an awful bus led by a drunk conductor and his dumb son who likes to drive with... bigger than him in the end the movie takes one turn and the trip becomes nothing but a swan s song of a dying country
45. negative- yes this movie is real thief it stole some shiny oscars from just because politicians wanted another war hero movie to boost the acceptance if we consider this title reasonable piece of the s wars are cool genre you surely have much better movies to choose from
46. positive- was going through a list of oscar winners and was surprised to see that this film beat butch cassidy and the sundance kid for ... by hoffman to take this role otherwise he may have been typecast after the graduate anyway this considered an all time great for a reason
47. positive- inspirational tales about triumph of the human spirit are usually big turn offs for me the most surprising thing about men of honor is how doesn't 't disappoint he creates a darkly funny portrait director george jr set out to make an old style flick and comes up with a winner
48. negative- this movie was god awful from conception to execution the us needs to set up star wars site in this remote country this ... gymnast star in real life i would probably kick him in the face after double with 2 2 twists in the layout position
49. positive wouldn't call we're back a story simply a kiddie version of jurassic park found it more interesting than that like the former it ... kind would actually say that john goodman doing voice here is sort of a precursor to his voice work in monsters inc worth seeing
50. negative- chinese ghost story ii is a totally superfluous sequel to two excellent fantasy films the film delivers the spell casting special effects that one can ... a little extra money out a successful formula they won't 't be able to do it again the cash cow is now dead as a
51. negative- the direction had clearly stated that this film's 's idea and plot is totally original however as to those who have read comic we can clearly- ... watching this thus making this movie getting what shouldn't have has became one of the best budget films in china for this year
52. negative- let me start by saying that i understand that invasion of the star creatures was meant to be a parody of the sei fi films ... the double feature dvd with invasion of the bee girls that movie is academy award winning stuff in comparison with invasion of the star creatures
53. positive knowing when to end movie is just as important as casting directing and acting and it's nice to see when a director script get a mansion br br this is a great independent production and one that wastes little time getting going and it won't waste your time either
54. negative italian born has inherited from her deceased lover karl an ultra modern and isolated house in the middle of the woods it's winter and she ... around for it i think i'll give it the benefit of the doubt it's definitely not what i was expecting from this indie film
55. negative- is a horror comedy that doesn't 't really have enough horror or comedy to qualify as one or the other it has one scene that is ... the movie that is weaker in general plot and spine because of production values that just shows you how uninteresting found the look of
56. positive- - there were a lot of 50's sci movies they were big draws for the drive in theaters lot of them were crappy even ... worried about the invisible monster forbidden planet is movie a sci fi fan can watch several times and find something new with each viewing
57. negative- curiously it is rene eyes and mouth not buddy the that emerge as the focal point of buddy a jim henson pictures production through francis thompson needs a good pick up shot she gives rene another extreme close up wonder what the lipstick budget was on this picture from
17

---

58. positive now this is what i'd call a good horror with occult superatural undertones this nice low budget french movie caught my attention from the very very confusing towards the end but redeems itself by the time it's over br br thought his was very good movie 10
59. negative and ethel buff's too will love her loud vocals as the wicked witch but this cartoon sequel to the wizard of oz is bereft of. sure baby boomers will get charge from it since it has been out of for so long as a curiosity item just fair from
60.
negative this is another one of those vs insects features a theme that was popular in the late 70's only you can't really call it horror after having seen ants lacking suspense action thrills shocks and creepiness the only thing you'll Il be left with after seeing ants an annoying itch
18


Reviews:
[{"review_id": "zKE7hCLwNu", "reviewer": "Reviewer_r62e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Inducing points have traditionally been heavily relied on to alleviate the cubic computational complexity associated with Gaussian process training and inference. There have been various different ways in which inducing points have been used in the literature, but they have nearly always been limited to either the input space of the data, or else were constrained by either the positions (e.g., to form a grid structure) or choice of embedding (such as trigonometric representations of the input space). In this work, the authors propose an approach for learning the inducing points along with the deep kernel function itself, which no longer has to be pre-trained. This allows for greater flexibility in the adaptation of inducing points since they are not constrained by the complexity or structure of the input space. The authors demonstrate how the proposed Gaussian process network models (IGN) achieve comparable performance to existing techniques on standard tabular and image data, while also being easily extendable to datasets having more specialized structures, such as text and graphs.", "summary_of_the_review": "I think there are some solid ideas in this paper that are backed by a fairly extensive experimental evaluation. Nevertheless, I feel quite ambivalent about the paper’s novelty and contributions overall, and would like to see more depth and insight included in a future revision of the paper. I am presently borderline on this work, tending towards rejection in the belief that there is still considerable room for improvement on the current submission.", "strength_and_weaknesses": "- The paper identifies an interesting limitation of existing inducing point approaches, and sets up a clear problem statement that is well framed in the context of related work. Showing how the proposed model can be extended to classification tasks via the Laplace approximation also makes the paper come across as more complete.\n- I appreciated how the paper considered an extensive selection of varied datasets in the experimental evaluation, which effectively demonstrates how the proposed IGNs can be used by practitioners across different use cases. The ablation studies assessing how the variance changes as more inducing points are added is also interesting, if not entirely surprising.\n\n------\n\n- I believe that some of the experiments would benefit from a more thorough deep dive into why certain results were obtained. For example, IGNs appear to perform quite inconsistently on the tabular regression datasets where the method varies from over-performing on some datasets while comparing less favorably to competing techniques on others.\n- I expect there to be a substantial risk of overfitting using this scheme, especially compared to methods using variational inference to optimize the location of inducing points. I believe this to be a particularly important consideration for practitioners, however I didn’t find much discussion in the paper.\n- I would have liked to see the method be framed in a similar context to the discussion featured in _“Understanding probabilistic sparse gaussian process approximations”_ by Bauer et al (2016) which presents a more principled discussion of how different inducing point GP approximations compare against each other. Such insight is currently missing from this work.\n- Although the paper stresses improved computational complexity as being one the primary benefits of using IGN models, this comparison is currently relegated to the supplementary material, whereas I believe this would also have been interesting to feature in the main paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "uiCDa3FCzV", "reviewer": "Reviewer_dTCT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new Gaussian process model that combines inducing point technique and deep kernel learning. The proposed method can be used for regression and classification. Experimental results verify the effectiveness of the proposed method on various types of real data.", "summary_of_the_review": "I agree that the proposed method is simple but useful. But the technical novelity seems moderate.", "strength_and_weaknesses": "S1. The authors proposes a scalable learning method for Gaussian processes with deep kernel. The proposed method is simple but useful.\n\nS2. Experiments demonstrate the effectiveness of the proposed method on various datasets.\n\nW1. Although the proposed method is practically useful, it seems a straightforward combination of inducing points and deep kernel learning.\n\nW2. There is no discussion of predictive variance. It would be good to find out how accurately it can be estimated by the proposed method.\n\nW3. There is still some concern about the number of inducing points, although an ablation study on it has been done in Section 5. This hyperparameter may be quite difficult to determine, since the appropriate value may vary depending on the difficulty of the problem and the dimension of z.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6IWOvz7DQXT", "reviewer": "Reviewer_MpyF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a deep kernel learning method for Gaussian processes. Deep kernel learning parametrises the covariance function using a neural network, and the proposed method further optimises a set of inducing points in the transformed space as well as employs stochastic gradient descent, allowing for faster inference. The authors demonstrate the expressiveness of their method through experiments on tabular data for regression, image classification, and classification of text and graphs.\n", "summary_of_the_review": "The proposed method is a variation of deep kernel learning for Gaussian processes. The novelty seems to be that the authors use mini-batch stochastic gradient descent to optimise the inducing point log-marginal likelihood from Snelson & Ghahramani (2005) instead of a variational objective used in previous work. This is an incremental change, however, and the experimental section is a little off in my opinion; instead of only showing that the method performs well on a range of tasks, there should have been a focus on comparing it to the other strongly related methods, trying to understand the strengths and weaknesses of each approach.\n\nDeep kernel learning as a topic is interesting to the GP community broadly, and the proposed method does achieve good empirical results, which should make it useful for practitioners as well as a strong baseline for future papers. However, given its current focus (and, in my opinion, incorrectly claimed novelty), I think it's not strong enough for acceptance.", "strength_and_weaknesses": "### Strengths\nDeep kernel learning is an interesting field for the GP community, so the paper should be interesting to that part of the ICLR community. The paper also builds on recent advances in mini-batch optimisation for GPs, which is an exciting approach, and the empirical results are quite impressive.\n\n### Weaknesses\nThe main weakness, in my view, is the novelty, which I consider limited. Deep kernel learning with learnable inducing points in the learnt feature space has been explored several times (for instance by Bradshaw et al. (2017), which the authors cite, and Ober et al. (2021)), and the fact that the authors highlight this as a novelty is somewhat misleading, I think. The main thing, in my view, that distinguishes this paper from others is that the authors use mini-batch optimisation of the inducing point log-marginal likelihood from Snelson & Ghahramani (2005), whereas other papers (at least the ones I could find) seem to use a variational sparse approximation in the learnt feature space. This is interesting since the convergence of SGD for GP inference was only recently established (Chen et al., 2020), but it is still an incremental change. This is also reflected in section 2, which is simply a summary of sparse GP regression and classification.\n\nWith that in mind, I think the experimental section is a little uninteresting. DKL has been shown to be a powerful approach many times, so the good performance of the proposed method is not that surprising. Instead, given the incremental nature, I would have been more interested in seeing a comparison between this method and other ways of using inducing inputs in feature space. For instance, Bradshaw et al. (2017) use a variational sparse approximation and optimise the resulting ELBO. Is this approximation better or faster than the one proposed here? Ober et al. (2021) show that DKL is prone to overfitting; is that also the case here? Similarly, the reason for computing the inducing outputs is not clear to me (see the questions in the next section), and it would have been interesting to see a comparison of ways to handle the outputs (e.g., representing them with a function as proposed vs optimising their values directly vs integrating them out).\n\n\n### References\n- Bradshaw et al. (2017): https://arxiv.org/abs/1707.02476\n- Chen et al. (2020): https://arxiv.org/abs/2111.10461\n- Ober et al. (2021): https://arxiv.org/abs/2102.12108", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject