Deep Ensemble Kernel Learning
Authors: Devanshu Agrawal
Jacob D Hinkle
Venue:  (2021)
Keywords: kernel-learning
gaussian-process
Bayesian
ensemble

Abstract: Gaussian processes (GPs) are nonparametric Bayesian models that are both flexible and robust to overfitting. One of the main challenges of GP methods is selecting the kernel. In the deep kernel learning (DKL) paradigm, a deep neural network or ``feature network'' is used to map inputs into a latent feature space, where a GP with a ``base kernel'' acts; the resulting model is then trained in an end-to-end fashion. In this work, we introduce the ``deep ensemble kernel learning'' (DEKL) model, which is a special case of DKL. In DEKL, a linear base kernel is used, enabling exact optimization of the base kernel hyperparameters and a scalable inference method that does not require approximation by inducing points. We also represent the feature network as a concatenation of an ensemble of learner networks with a common architecture, allowing for easy model parallelism. We show that DEKL is able to approximate any kernel if the number of learners in the ensemble is arbitrarily large. Comparing the DEKL model to DKL and deep ensemble (DE) baselines on both synthetic and real-world regression tasks, we find that DEKL often outperforms both baselines in terms of predictive performance and that the DEKL learners tend to be more diverse (i.e., less correlated with one another) compared to the DE learners.

Under review as a conference paper at ICLR 2021
DEEP ENSEMBLE KERNEL LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Gaussian processes (GPs) are nomparametric Bayesian models that are both flexi- ble and robust to overfitting. One of the main challenges of GP methods is select- ing the kernel. In the deep kernel learning (DKL. paradigm, deep neural network or "feature network" is used to map inputs into a latent feature space, where a GP with "base kernel" acts; the resulting model is then trained in an end-to-end fashion. In this work, we introduce the "deep ensemble kernel learning" (DEKL) model, which is a special case of DKL. In DEKL, a linear base kernel is used, enabling exact optimization of the base kernel hyperparameters and scalable inference method that does not require approximation by inducing points. We also represent the feature network as concatenation of an ensemble of learner networks with a common architecture, allowing for easy model parallelism. We show that DEKL is able approximate any kernel if the number of learners in the ensemble is arbitrarily large. Comparing the DEKL, model to DKL, and deep en- semble (DE) baselines both synthetic and real-world regression tasks, we find that DEKL often outperforms both baselines in terms of predictive performance and that the DEKL learners tend to be more diverse (i.e., less correlated with one another) compared to the DE learners.
INTRODUCTION
In recent years, there has been growing interest in Bayesian deep learning (DL), where the point predictions of traditional deep neural network (DNN) models are replaced with full predictive dis- tributions using Bayes' Rule (Neal, 2012) Wilson 2020). The advantages of Bayesian DL over traditional DI. are numerous and include greater robustness to overfitting and better calibrated uncer- tainty quantification (Guo al., 2017 Kendall & Gal 2017). Furthermore, the success of traditional DL. already rests on number of prohabilistic elements such as stochastic gradient descent (SGD), dropout, and weight initialization- all of which have been given Bayesian interpretations (Smith & Le 2018 Gal & Ghahramani 2016 Kingma al. 2015: Schoenholz et al. 2016 Jacot al. 2018).
80 that insights into Bayesian DL may help to advance DL as a whole.
Gaussian processes (GPs) nonparametric Bayesian models with appealing properties, they admit exact inference for regression and allow for a natural functional perspective suitable for pre- dictive modeling (Rasmussen & Williams 2005). While first glance GPs appear unrelated to DL models, a number of interesting connections between GPs and DNNs exist in the literature, suggesting that GPs can constitute a valid approach to Bayesian DL (Neal 1996 Lee et al. 2018 de Matthews et al. 22018) Jacot et al. 20188: [Damianou & Lawrence 2013) Salimbeni & Deisenroth
2017 Agrawal et al. 2020). A GP prior typically characterized by its covariance function or "kerne"" which determines the class of functions that the GP can model, as well as its generaliza- tion properties outside training data. Kernel selection is the primary problem in GP modeling, and unfortunately traditional kernels such as the radial basis function (RBF) kernel are not sufficiently expressive for complex problems where more flexible models such as DNNs generally perform well. This is the key motivation for kernel learning, which refers to the selection of optimal kernel out of a family of kernels in a data-driven way.
A number of approaches to kernel learning exist in the literature, including some that parameterize kernels using DNNs (Zhou et al. 2019 Li et al. 2019 Bullins et al., 2018 Sinha & Duchi 2016). As these approaches involve learning feature representations, they are fundamentally different from condom-feature methods for efficient kernel representation (Rahimi & Recht 2008). However, these approaches are not specific GPs and do not take advantage of robust Bayesian framework.

---

Under review as a conference paper at ICLR 2021
In contrast, the deep kernel learning (DKL) paradigm does exactly this; In DKL, DNN is used as a feature extractor that maps data inputs into a latent feature space, where GP inference with some "base kernel" is then performed (Wilson et al.) 201 a Jean et al. 2016 Al-Shedivat et al. 2017 Bradshaw et al. 2017 Izmailov et al., 2018 Xuan et al. 2018). The resulting model is then trained end-to-end using standard gradient-based optimization, usually in variational framework. We note that the DKL model is just GP with a highly flexible kernel parameterized by a DNN. By optimizing all hyperparameters (including the DNN weights) with type II maximum likelihood estimation, the DKL model is able to learn an optimal kernel in manner directly informed by the data, while also taking advantage of the robustness granted by the Bayesian framework. A special case of DKL that is worthy of note was considered in Dasgupta et al. (2018), who use a kernel.
linear base kernel and impose a soft orthogonality constraint to learn the eigenfunctions Although similar in spirit to the approach in this paper, their method does not make use of an efficient variational method, nor is distributed training made possible since all of the basis functions are derived from the same feature network.
In this work, we introduce the "deep ensemble kemee learning" (DEKL) model- a simpler and more efficient special case of DKL with specifications- the base kernel is linear, and the fea- ture network is partitioned into an "ensemble" of "learners" with common network architecture. In contrast to nonlinear kernels, the linear kernel allows us derive efficient training and inference method for DEKL that circumvents the inducing points approximation commonly used in traditional DKL. The hyperparameters of the linear kernel can also be optimized in closed form, allowing us to simplify the loss function considerably. Convenience aside, we show that DEKL remains highly expressive, proving that it universal in the sense that can approximate any continuous kernel so long as its feature network is arbitrarily wide. In other words, we may keep the base kernel simple if we are willing to let the feature network be more complex. The second specification of DEKL lets us handle the complexity of the feature network; because the feature network is partitioned, it admits easy model parallelism, where the learners in the ensemble are distributed. Moreover, our universal- ity result only requires the number of learners to be arbitrarily large; the learners themselves need not grow (meaning fixed-capacee capacity learners are sufficient), avoiding additional model parallelism.
From different perspective, DEKL may be regarded as an extension of traditional ensembling methods for DNNs and in particular the deep ensemble (DE) model of Lakshminarayaaaa et al. (2177), which is also highly parallelizable. In DE, each DNN learner parameterizes a distribution over the variates (e.g., the mean and variance of a Gaussian in regression, or the logits of softmax vector in classification). Each learner is trained independently with maximum likelihood estimation, and the final predictive distribution of the DE is then defined to be a uniform mixture of the individ- ual learner predictive distributions. Although not Bayesian itself, the DE model boasts impressive predictive performance and was shown to outperform Bayesian methods such as probabilistic back propagation (Hernández-Looanr
& Adams 2015) and MC-dropout (Gal & Ghahramani (016). In contrast, in DEKL., the learners are trained jointly via shared linear GP layer. We surmise that this may help to promote diversity (i.e., low correlation) among the learners by facilitating coordination, which we verify experimentally. Unlike non-Bayesian joint ensemble training methods such that of Webb et al. (2019), we hypothesize that the DEKL learners might learn to diversify in order to better approximate the posterior covariance- DEKL to be more efficient than DKL and more robust than DE, by drawing on the strengths of both (see Fig. for a comparison of model architectures).
inherently Bayesian feature. We therefore expect
DEEP ENSEMBLE KERNEL LEARNING
A DKL model is GP whose kernel encapsulates a DNN for feature extraction (Wilson et al.) 2016b a). A deep kernel is defined as
K Keeep
= Krase (x 0), ()x;(();)))))))))))
(1)
where 0) is a DNN with weight parameters 0 and Kisse (77) is any chosen kernel-calle the "base kernel"- witth hyperparameters 7. Note that the kernel hyperparameters of Kawep include all hyperparameters 7 of the base kernel Kruse well as the DNN weight parameters 0; Given the
expressive power of DNNs, the deep kernel is also highly expressive and may be viewed as a method to automatically select a GP model.
2

---

Under review as a conference paper at ICLR 2021
Data
Leamer
NEL.Loo loss
Learer
Data
Leamer 2
NELL Joss
Data
Feature network
GP
VFF loss
Dara
Leamer
Linear GP
VFF loss
Data
Leamer 3
NELL loss
Leamer
(a) Deep Ensembles (DE)
(b) Deep Kernel Learning (DKL)
(c) Deep Ensemble Kernel Learning (DEKL)
Training various neural architectures considered in this work. Deep ensembles aa) use identical prediction networks with independent initialization and training. Deep kernel learning (b) uses neural feature extraction network with expressive kernel such an RBF. Our model, deep ensemble kernel learning (c) uses an ensemble feature networks trained jointly through a linear kernel that enables exact inference via minimization of variational free energy for regression.
Figure 1:
Our proposed method, DEKL, is a special case DKL. Whereas RBF and Matern kernels are typically used base kernels in DKL, in DEKL take the base kernel be the linear kernel:
(in 22; V) = x Va2.
where V is a symmetric positive-semidefinit matrix. In order to enable parallel computation, in DEKL, we use feature network (o:;;)) 0) that is partitioned into an "ensemble" of subnetworks (;;(; called "learner"", having identical network architectures; i.e., (p=; the concatenation of the outputs of the (+),; DEKL offers number of advantages over general DKL:
1. Unlike the hyperparameters of the RBF kernel, in DEKL, we can optimize the hyperpa- rameters of the linear base kernel in closed form.
2. The linear base kernel allows us think of DEKL model not just as GP but as a finite- dimensional Bayesian linear model (BLM), conditional on the feature networks; this lets us derive an efficient inference method that is much simpler than the inducing points method used in general DKI..
3. Finally, the partitioned architecture of the feature network makes it much more amenable to model parallelism.
Note that DEKL fundamentally diffferent from random-feature methods such as that Rahimi & Recht|(008)) where the learners pp-; 0.) are random features that are not optimized during training.
potential drawback partitioning the feature network we do in DEKL is that, compared gen- eral DKL, the network is less expressive. However, the following universal approximation theorem for DKL implies that this effect can be compensated by adding parallel learners:
Theorem (Universal kernel approximation theorem). Let X' C RR.. be some compact Euclidean domain, and let a R + R be a non-polynomial activation function (Pinkus) 1999). Then, given a continuous, symmetric. positive-dete definite kernel : X x X R and any >> O. there exist
finite number H of affine functions 35 off- Th and symmetric positive semi-definite matrix V E BHH such that for any xı, D2 E X,
H
- 02) <E
The proof, found in Appendix A contains straightforward combination Mercer's Theorem (Mercer 1909) and the Universal Approximation Theorem, attributed to Cybenko. Hornik. Leshno, and Pinkus (Cybenko) 1989 Homik et al., 1989 Leshno et al.) 1993 Pinkus] (199).. Note that
Thm. lets represent non-stationary continuous kernels, in contrast methods such random Fourier feature expansion (Rahimi & Recht 2007).
The approximation in Thm. requires possibly large number H of affine functions 8,. However, in DEKL we replace the functions x - (z)())) with an ensemble of strictly more flexible DNN learners (o(:;))). which can help reduce the number H of learners required; in the proof of Thm.
3

---

Under review as a conference paper at ICLR 2021
approximate each eigenfunction of the target kernel with linear combination of the learners; the learners are sufficiently expressive, then it may take only one learner per eigenfunction to approximate the target kernel. We also allow each learner (o(:;,) to have multiple outputs M. In the case of the simple learners x - o (3))))), a learner with M outputs is simply a concatenation of M single-output learners, suggesting that multi-output learners may help to further reduce the number H of required learners.
The summation in Eq. should be understood as a deep kernel as in Eq. [ where the DNN yp in Eq. Dis is the concatenation of all learners and the base kernel is given by the linear kernel in Eq, 2 Theorem is thus a statement about the universality of DEKL with a linear base kernel. Given that other kernels such as the RBF are more popular choices of base kernel in the DKL literature, it is natural to wonder if DEKI. remains universal kernel approximator we change the base kernel. turns out that not all choices of base kernel give universality, as is implied by the following remark.
Remark 2. For a deep kernel (Eq. D with base kernel Kiose: : R RH + IR be universal kernel approximator, the base kernel must be unbounded both above and below.
We give more details in Appendix B but intuitively, since the base kernel is the outermost function in the deep kernel, any bound on its range will prevent the deep kernel from approximating kernels with unbounded range, such the dot product kernel. The class of base kernels with bounded (or half-bounded) range, and thus the base kernels that do not give universality, is large and includes many popular kernels such as the RBF kernel and periodic kernel. The linear base kernel in Eq. 2 is therefore special, as it is not only convenient but also grants us universality.
We note the converse of Remark 2 is not true; unbounded base kernel does not guarantee that DEKL model is a universal kernel approximator: For example, restricting the matrix V in the linear base kernel (Eq. 2) diagonal matrix breaks universality; this is because Thm. T must hold for even the simplest learners I + ((,(z)),, which fails to happen when we restrict V (see Appendix B for details). Classifying all base kernels for which DEKL universal remains an important open problem for future work.
2.1 EXACT INFERENCE
Consider dataset {(,, y.) E ER., x RR In DEKL, we model this data with likelihood p(y I If) and GP prior GP KDE(,, V)), where the kemel DE :; 0, V) RRD x RRD - IR is defined as (D 0, V) (<:;; V (x2;; 0),
(4) where V is an H x H symmetric positive semidefinite matrix and ((;;)) = (,)); is the concatenation of H DNN learners cp:; 0.) R² + R with common network architecture. The DEKL model is thus a standard GP but with special kernel, and thus one may proceed following standard practices for GP inference. However, in contrast to other popular GP kernels such as the RBF kernel, the kernel in Eq. has rank most H, and we can leverage this property to derive a more efficient GP inference method by regarding it as a Bayesian linear model (BLM) where the learners act as the BLM basis functions. More precisely, our GP prior is equivalent to taking the latent function f : IR." + E to be generative model of the form
(x) (xx; 0) a, a~ (O
Now formulated as BLM, we may perform Bayesian inference by inferring the posterior on a; when the number of training points N is large, this approach is significantly more efficient than direct GP inference, which requires Cholesky decomposition of complexity O(N²).
In the case of regression with a Gaussian likelihood, the posterior on a admits a closed form. More- over, the evidence of the above model given the training data also admits a closed form, allowing us to obtain maximum evidence estimates of the DEKL hyperparameters V and 0 as is standard practice in GP methods. However, the dependence of the evidence on 0 in general complex, so that the maximum evidence estimates must be approximated. Moreover, the evidence is not separa- ble across the training data, so that optimization is not amenable to SGD with minibatching and is thus not scalable to large datasets. We therefore perform approximate inference and maximum evi- dence estimation in a variational framework, discussed in the next section. We reiterate that DEKL admits exact inference small datasets and that maximum evidence estimation can be performed through non-stochastic gradient- based optimization methods; however, we consider the variational framework as it can handle larger class of problems.
4

---

Under review as a conference paper at ICLR 2021
2.2 VARIATIONAL INFERENCE
In variational inference, we approximate the posterior distribution on the vector a with a variational distribution by solving an optimization problem. In the GP literature, it is standard take the vari- ational distribution to itself be a GP. In our DEKL model formulated as a BLM, this corresponds to a normal variational distribution (M, 2) on a, where μ E R2 and Σ E IR "x* We now minimize the KL divergence between the variational and posterior distributions on which is equivalent to minimizing the variational free energy loss function:
LVFE 2,0,)
Σ (H,L, + KT,H,, V.
(5)
1=1
NELL; (1, 2, = - -B (w. [log((s) (1 0) Ta:]]
(6)
(L(4, V) = DRL. (N (1, m) [N)
1
H
2" V 4 tr(EV-)) log det(EV
2
Observe that we optimize both the variational parameters µ and as well the kernel hyperparam- eters 0 and V as is standard in GP inference. Note also that this loss function is separable over the training data and is thus amenable to optimization with SGD and minibatching; this paper, we focus on regression tasks with Gaussian likelihood plv If) = [y; with precision т. The corresponding NELL admits a closed form and is given by
(p. Σ, 0) = [1 O) μ 11 + _ (zx 0)
() EE 0) 2 log + log(2 (8) 2
Note that practice, we parameterize E with its positive-denie definite lower-triangular Cholesky factor in order guarantee the symmetric sesitive-defnittnees. of Σ.
For scalability to large training sets, DKL inference is usually performed using the inducing points approximation in sparse variational framework (Titsias) 2009 Hensman et al. 2013). In the case of DEKI, however, due to using linear base kernel, the inducing points approximation is not necessary, seen above. Instead, our parameters fe and Σ take the place of the inducing parameters in the sparse variational GP framework, and we are able to maintain an exact model for the posterior GP.
2.3
OPTIMAL PRIOR COVARIANCE
The variational free energy depends on the prior covariance V only through the KL term, which has a tractable form. This allows us optimize V in closed form; we find the optimal prior covariance
to be
(9) Next, we substitute this optimal prior covariance back into the KL term, thereby eliminating the prior covariance from the variational free energy altogether. After some simplification, the KI. term takes the following form:
Ve 1 = > + up
KLo E) KL E, V. (1. E)) = log(1 + TT)
(10)
(See Appendix C for derivations of Eqs. 1 During training, this term drives down the signal-to- noise ratio (SNR) -y=-1m encouraging the model to learn a more robust fit to the data.
We contrast the form of Eq. T0 with the KL term obtained by fixing the prior covariance to the identity V =: I:
(L, Σ) (L 1) - 1111
1 tr(E) log det(E)
1
H
(11)
2 We note that while Eq. 11 includes a quadratic penalty on μ., Eq. TO includes only logarithmic penalty: optimizing the prior covariance therefore results in weaker regularization of p. The same is true with respect to Σ; both Eqs. 10 incllde roughly logarithmic penalties on Σ-1 that encourage larger Σ, but Eq. [ additionally includes linear penalty on E. resulting in stronger regularization of Σ. We note that regression, the NELL term already favors small Σ (see Eq. [8. that the KL term is the only thing preventing Σ from degenerating. By weakening the regularizatin on E, we suspect the KI. term with optimal prior covariance be beneficial in practice.
5

---

Under review as a conference paper at ICLR 2021
Deep Ensemble (H-1)
DEKL-I (H-1)
DEKL-I (H-55)
DRL-1
200
200
200
200
0-
220
2200
-200
-200
-5
-5
-5
-5
Deep (Hm5)
DEKL-O (H-1)
(H=5)
DKL-O
100
100
100
100
-100
-100
-100
-100
Figure 2: Synthetic cubic dataset. The first column shows a single DE learner and a DE model of five learners. The second and third columns show one DEKL learner and a DEKL model of five learners, both with and without optimal prior covariance. The fourth column shows a DKL model of the same width as the five-learner DEKL, both with and without optimal prior covariance. The predictive mean function (blue) with three units of standard deviation is shown for each model trained on the cubic dataset (red), with ground truth (black). Although the five-lcarner DEKL has far fewer parameters than DKL. due partitioning of the feature network, it approximates the full DKL well on this dataset, and improves over the equivalent DE model.
EXPERIMENTS
In experiments, evaluate DEKL on various regression datasets, comparing it to two key baselines- DE and DKL (with linear base kernel). For both DKL and DEKL, we also compare the effect of the KL term with optimal prior covariance, KLO (see Eq. I0). to that of an identity prior covariance, KT,, (see Eq. 11).
3.1 SYNTHETIC DATA
For our first experiment, we consider a simple regression problem with one-dimensional inputs and outputs, as it allows for easy visualization of the posterior predictive distribution. For this purpose, we generate data via = x² + where (0, 3²) as in Lakshminarayanan et al. (017)) We sample twenty (Ir y) pairs with to the DKL and DE baselines. For DE, we follow the implementation of Lakshminarar
Unif(-4,44 4). As already stated, compare the DEKL model
et al.
(2117). except that we do use adversarial samples, which the authors say is optional. For the DEKL and DE models, we use either one learner (H = 1) or five learners (H) = 5), where each learner is a shallow MLP with rectified linear unit (ReLU) activation and hidden neurons. For the DKL model, we use a shallow MLP feature network with ReLU activation and 250 hidden and five output neurons, so that has the same total hidden width as the five-learner DE and DEKL models. Note that the only difference between the leae-rr DEKL and DKL models is compaeepared to the DKL model the DEKL model is missing some connections in its second layer.
We train all models the training set using the Adam optimizer with learning rate 10 and full batches for 1000 epochs. For the DEKL and DKL models, we set the noise precision to true value
The predictive distributions of all models on their input domain are visualized in Fig. 2 We make a number of observations: First, the DKL and DEKL models with fixed prior display better uncertainty quantification than DE, having lower variance within the training domain (interpolation) and higher variance outside the training domain (extrapolation). Second, DE and DEKL. with learner have

---

Under review as a conference paper at ICLR 2021
less flexible means compared to their five-learner counterparts, which we expect due to reduced expressivity. However, DE is more confident with one leamnrr while DEKL is less confident. This highlights the difference in the uncertainty quantification mechanisms of the two models; while DE relies on multiple learners for its predictive variance, DEKL, offers some predictive uncertainty even with one learner thanks to Bayesian inference. Third, using the optimal prior covariance results in overconfidence. We suspect this is because theKL term in Eq. [0 imposes weaker regularization on u than does the KL term in Eq. [I which may be critical on dataset of only 20 training points. Finally, the five-leamer DEKL predictive distribution is very similar to that of DKL, indicating that the partitioning of the feature network in DEKL grants efficiency without hurting predictive performance.
3.2 WOOLD DATA
We test DEKL model on set of regression tasks from the UCI repository that has become popular benchmark in the Bayesian DL. literature, starting with the work of Tieenández--Lbbta & Adams (201)). We again compare DEKL, to the DE and DKL baselines as in Sec. [3.1 and the same experimental setup as in Lakshminarayanan et al. (2017). We only consider DE and DEKL with five learners, with each learner having 50 hidden neurons, except on the Protein dataset, where we use 100 hidden neurons (and correspondingly, 500 hidden neurons in the DKL models).
We consider 20 random train-test splits of each dataset except Protein, where we use only 5 train- test splits. For each split, we use 20% of the training set for validation. We train all models on the training sets using the Adam optimizer with learning rate 10-3 and minibatch size 128 for 1000 epochs. For the DEKL. and DKI. models, we set the noise precision by performing grid search and selecting the value that results in the lowest negative log-likelihooe (NLL.) on the validation set. For all models, we find the epoch where NLL is minimized on the validation set, then retrain all models on the full dataset up to that number of epochs, and finally evaluate the models on the test sets.
We report the test root mean square error (RMSE) and test NLL averaged over all train-test splits (Tables 1 for brevity, we list Table 2 in Appendix D where we also include the definitions of RMSE and NLL). We see that DEKL often outperforms DE, especially in terms of RMSE, suggest- ing that joint ensemble Bayesian training can often be beneficial. A further comparison of DE and DEKL may be found in Appendix H. where we examine predictive performance as a function of number of learners and find that the performance gap between DE and DEKL is largely independent of the number of learners, at least up to five learners. We note that DEKL also attains either com- parable or superior performance other GP methods, both shallow and deep, with RBF kernel in terms of NLL (GP scores are listed in Salimbeni & Deisenroth (2017), who use the same experimen- tal setup as we do). More surprisingly, DEKL often outperforms DKL., indicating that the benefit of DEKL over DKL. goes beyond simple efficiency or ease of parallelization; partitioning the feature network can often lead to a significant boost predictive performance as well. We also observe that KL, vs. KLo has little impact performance, suggesting that either the data is rich enough that the non-diagonal terms of the prior covariance matrix have an insignificant effect on the model evidence, that the feature network is expressive enough to include the Cholesky factor of the prior covariance matrix in the case of KLJ.
3.3 DIVERSITY OF LEARNERS
To see if joint ensemble training in a Bayesian framework promotes diversity among learners, we compare the diversity of learners in DEKL that in DE on the UCI datasets. Given an ensemble of learners, we measure its diversity in terms of the functional correlation matrix of its learners, and we say that the ensemble more diverse if the learners have lower correlation. We base this notion of diversity the work of Brown et al. (2005), who showed that extending the classic bias- variinee decomposition of the mean squared error to an ensemble of learners requires an additional term in the decomposition that quantifies the average covariance between distinct learners.
For each UCI dataset and each train-test split, we load the DE and DEKL models (with and without optimal prior covariance) already trained following Sec. 3.2 and evaluate their feature matrices on the entire dataset (including both training and test points). Each row of feature matrix corresponds to observation, and each column corresponds to the output of one of five learners. For DE,
7

---

Under review as a conference paper at ICLR 2021
Table 1: Test-RMSE (mean and std. dev. across 20 train-test splits) UCI datasets, using deep ensembles (DE), deep kernel learning (DKL) and deep ensemble kernel learning (DEKL). Suffices and -O indicate using KL, (Eq. (T) and KLo (Eq. 10 respectively. The numbers N and D
the size and dimension of each dataset respectively.
Dataset
N
D
DE
DKL.II
DKL-O
DEKL-I
DEKL-O
Boston housing
506
13
3.29 1.01
0.77
3.43 1.68
3,03 = 0.87
3.03 0.86
Concrete
1030
8
5.79
4.93
4.79
4.52
Energy
768
8
2.04
+ 0.31
461+0.22
0.94
0.48
0.47 0.05
Kin8nm
8192
8
0.08 + 0.00
0.000
0.08 +.01
0.07 0.00
Naval Propulsion
11934
26
0.00
0.00
0.00
0.00 0.00
Power plant
9568
0.16
0.25
0.23
0.20
3.11
Protein
45730
9
4.06 0.12
0.11 0.07
3.95
Wine
1599
22
0.63 - 0.03
0.68 1.14
0.67 - 0.06
0.63 0.05
0.04
Yacht
308
7
0.72 = 0.24
2.92 + 3.06
2.52 + 2.78
0.61 0.22
0.62 0.22
DEKU
BEKL.O
1.00
1.00
1.00
609
10n50m
012
0.0
-0.2
Figure 3: Correlation matrices of learners averaged over all train-test splits on the Kin8nm dataset using deep ensembles (DE) and deep ensemble kernel learning (DEKL). Suffices -I and -O indicate using KT. (Eq. IT) and KT. (Eq, TOp, respectively.
use the mean output of each learner as one of the features. For DEKL, we multiply each column of the feature matrix ($(X; 0) in the notation of Sec. 2.2 with X the dataset) by the corresponding element of the parameter mean of the final BLM layer, and we additionally scale all columns by the number of learners (five); doing this, the DE and DEKL. feature matrices comparable in the sense that the predictive means of both models can be obtained as the average of the feature matrix columns. Finally, for each dataset, train-test split, and model, we calculate the correlation matrix of the feature matrix and then average all correlation matrices across the train-test splits.
We visualize the correlation matrices Fig. 3 We show only one dataset here for brevity; results for all datasets may be found in Appendix F Overall, DEKL results in lower correlation between learn- ers thus greater diversity, as hypothesized, and using the optimal prior covariance often boosts diversity further. However, more diversity does not necessarily imply better predictive performance; elucidating the relationship between diversity and predictive performance future work. We also observe extreme cases where the learners in DE are strongly correlated, while the learners DEKL are almost entirely uncorrelated or orthogonal. This ability for the learners to approximately orthogonalize even in DEKL-I suggests that the individual learners and that the restriction the prior covariance to the identity matrix not a significant hindrance expressive power; suspect this may be why KL, KLo has little impact on performance on the UCI datasets (Table
topic we leave for
quite flexible
4 CONCLUSION
We have introduced the DEKL model: special case of DKL that is more easily parallelizable and thus more efficient than other DKL models of comparable network complexity. We have seen that DEKL often achieves better predictive performance than deep ensembles, which also highly
8

---

Under review as a conference paper at ICLR 2021
parallelizable. Even with linear base kerne,, we have shown that DEKL is universal kernel ap- proximator if the feature network is allowed to be arbitrarily wide, and we handle the complexity of the feature network by partitioning it into an ensemble of leamner. In our experiments, we found that DEKL, often outperforms DE and promotes more diversity, suggesting that jointly training learners in Bayesian framework can be beneficial. Interestingly,
also observed that DEKL. often out- performs DKL of the same width, suggesting that partitioning the feature network not only grants computational efficiency through model parallelism but may also boost performance by reducing the parameter count compared to fully-connected DKL.
We are considering several avenues for future work. We plan to study the scalability of DEKL to a very large number of learners in distributed setting, which may be necessary to approximate a kernel of very high or infinite rank. A key challenge in this endeavor is that the size of the variational covariance Σ grows quadraticly with the number of learners; to deal with this, we will consider asymptotic regimes where efficient approximations of are justified, namely the many-simple- learners regime and the complex-leareers regime. In the first of these regimes, the dimension R of the span of I/ learners is much less than H. which allows us to assume that Σ is of low rank, specifically R. In the second regime, the learners may be sufficiently flexible to be orthogonal to one another, allowing us to assume E is diagonal. This could lead to an approach similar to that of Dasgupta al.] (2018), but with distributed computation and using our variational framework. More generally, we may consider to approximate Σ as a sum of a low-rank matrix and a diagonal matrix, possibly striking balance between the two asymptotic regimes.
Another interesting topic of investigation is the mechanism by which DEKL achieves performance often superior to that of DE. In DE, the learners are trained independently, and thus any diversity among the learners is solely due to random initialization and the nonconvexity of the loss function. In contrast, in DEKL, the learners are able "communicate" with one another through the common final GP layer, and thus we hypothesize that the learners may "coordinate" with one another to ensure diversity. Indeed, in our experiments, we found that DEKL. does lead to greater diversity, but the precise mechanism by which this diversity emerges and how it impacts predictive performance are less clear. We believe that diversity in DEKL may be linked to the posterior covariance, which is generally nonzero in this Bayesian setting.
is also tempting to extend DEKL to deep Gaussian processes (DGPs), which are models defined as compositions of GPs (Damianou & Lawrence 2013 Cutajar et al. 2017 Salimbeni & Deisenroth 2017). This will result in a DGP inference method that again does not require an inducing points approximation, similar to the random features expansion approach of Cutajar al. (2017), except that the features are trainable DNNs that need not be orthogonal. Given that DEKL. is a universal kernel approximator, we conjecture that DGP with DEKL kernels can approximate any DGP with continuous kernels, under some distance metric on stochastic processes. Such a result would sug- gest universal model for stochastic processes (or some suitably well-behaved subspace of them), arguably the most general universality theorem we can imagine in predictive modeling.
REFERENCES
Devanshu Agrawal, Theodore Papamarkou, and Jacob Hinkle. Wide neural networks with bottle- necks are deep Gaussian processes. Journal of Machine Learning Research, 21(175);1-66, 2020.
URL
wwwpee///w/////0/0//0 h1t
Maruan Al-Shedivat, Andrew Gordon Wilson, Yunus Saatchi, Zhiting Hu, and Eric P. Xing. Learn- ing scalable deep kernels with recurrent structure. Journal of Machine Learning Research, 18 498. html
(82):1-37, 2017. http::/tt://///// wttp://.rrgpppe//////////////
John Bradshaw, Alexander G de G Matthews, and Zoubin Ghahramani. Adversarial examples, uncertainty, and transfer testing robustness in Gaussian process hybrid deep networks. arXiv preprint
2017.
Gavin Brown, Jeremy L Wyatt, and Peter Tiño. Managing diversity in regression ensembles. Journal of machine learning research, (6Sep):16211111 2005.
Brian Bullins, Cyril Zhang, and Yi Zhang. Not-so-random features. In International Confer- ence on Learning Representations, 2018. URL ptttp:://openreview. net/ wwweet/forummiimm
Hk@XMWgRb)

---

Under review as a conference paper at ICLR 2021
Kurt Cutajar, Edwin V Bonilla, Pietro Michiardi, and Maurizio Filippone. Random feature pansions for deep Gaussian processes. In International Conference on Machine Learning, pp- 884-893, 2017.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Con- trol, Signals and Systems, 2(4):303-314, 1989.
Andreas Damianou and Neil Lawrence. Deep Gaussian processes. In Artificial Intelligence and Statistics, pp. 207-215, 2013.
Sambarta Dasgupta, Kumar Sricharan, and Ashok Srivastava. Finite rank deep kemnll learning. In Third Workshop Bayesian Deep Learning, NeurIPS, 2018.
Alexander G. de Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. International Conference on Learning Representations, 2018. accepted as poster.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp. 1050-1059, 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330, 2017.
James Hensman, Nicold Fusi, and Neil D Lawrence. Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, pp. 282-290, 2013.
José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learn- ing of Bayesian neural networks. International Conference on Machine Learning, pp. 1861- 1869, 2015.
Kurt Hornik, Maxwell Stinchcombe, Halbert White, et al. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359-366) 1989.
Pavel Izmailov, Alexander Novikov, and Dmitry Kropotov. Scalable Gaussian processes with bil- lions of inducing inputs via tensor train decomposition. In International Conference Artificial Intelligence and Statistics, pp. 726-735, 2018.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: convergence and gen- eralization in neural networks. In Advances in neural information processing systems, pp. 8571- - 8580, 2018
Neal Jean, Michael Xie, and Stefano Ermon. Semi-supervised deep kernel learning. In NIPS Bayesian Deep Learning Workshop. 2016.
Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, pp. 5574-5584, 2017.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. Advances in Neural Information Processing Systems, pp. 2575-2583, 2015.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. Advances in neural information processing systems, pp. 6402-6413, 2017.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as Gaussian processes. In International Conference Learning Representations, 2018
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward net- works with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861-867, 1993.
10

---

Under review as a conference paper at ICLR 2021
Chun-Liang Li, Wei-Cheng Chang, Youssef Mroueh, Yiming Yang, and Barnabas Poczos. Implicit kernel learning. The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2007-2016, 2019.
James Mercer. Functions of positive and negative type, and their connection with the theory of inte- gral equations. Philosophical Transactions the Royal Society of London. Series A. Containing Papers of Mathematical or Physical Character, 209:415-446, 1909.
Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Nenworks, pp. 29-53. Springer, 1996.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica, 8: 143-195, 1999. doi: 10.001750662299000291.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20:1177-11888 2007.
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advances in neural information processing systems, 21:1313- 1320, 2008
Carl Edward Rasmussen and Christopher Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005. ISBN 026218253X.
Hugh Salimbeni and Marc Deisenroth. Doubly stochastic variational inference for deep Gaussian processes. In Advances in Neural Information Processing Systems. pp. 4588 188894499,.0 2017.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information
propagation. In International Conference Learning Representations, 2016.
Aman Sinha and John C Duchi. Learning kernels with random features. In Advances in Neural Information Processing Systems, pp. 1298-1306, 2016.
Samuel L Smith and Quoc V Le. Bayesian perspective on generalization and stochastic gradient descent. In International Conference on Leurning Representations, 2018.
Michalis Titsias. Variational learning of inducing variables in sparse Gaussian processes. In Artificial Intelligence and Statistics, pp. 567-574, 2009.
Andrew M. Webb, Charles Reynolds, Wenlin Chen, Henry Reeve, Dan-Andrei Iliescu, Mikel Lujan, and Gavin Brown. To Ensemble Not Ensemble: When does End-To-End Training Fail? arXiv preprint arXiv: XiXv:1902.00422 04422, February 2019.
Andrew G Wilson, Zhiting Hu, Ruslan R Salakhutdinov, and Eric P Xing. Stochastic variational deep kernel learning. Advances in Neural Information Processing Systems, pp. 2586-2594, 2016a.
Andrew Gordon Wilson. The case for Bayesian deep learning. arXiv preprint arXiv:2001.10995. 2020.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning- In Artificial Intelligence and Statistics, pp. 370-378, 2016b.
Junyu Xuan, Jie Lu, Zheng Yan, and Guangquan Zhang, Bayesian deep reinforcement learning via deep kernel learning. International Journal of Computational Intelligence Systems, 12(1): 161, 2018.
Yufan Zhou, Changyou Chen, and Jinhui Xu. Kernelnet: A data-dependent kernel parameterization for deep generative modeling. arXiv preprint arXiv: 1912,00979, 2019.

---

Under review as a conference paper at ICLR 2021
A PROOF OF UNIVERSAL KERNEL APPROXIMATION THEOREM
Proof of Thm. 7 By Mercer's Theorem Mercer 1909 Sec. 29), there exist non-negative scalars
{Ar Jr=1 and an orthonormal basis of continuous functions fe, A R called "feature maps"
such that the target kernel x admits the representation
6( 12)
Σ (xa)er)))
(12)
where the sum on the righthand side converges absolutely and uniformly on the compact set A. Selecting 0 < 8 e, uniform convergence guarantees the existence of an integer such that for all I1, E X,
R Σ (x)((())))()))) - x((1, x2)
< 6.
(13)
By continuity of the feature maps e, and compactness of X, the feature maps are bounded, so there exists finite positive B max,=1....fF V supzex [e, By the classic Universal Approxima- tion Theorem (Pinkus 1999). there exists shallow MLP & X RR of width H and parameters
Urr and 3 X + for I...., H such that for all x E X.
(x(x) = Σ u (Bi(x)
(14)
B, c-8
(x(()) - (x)e(())
<> < min
(15)
3NB
(1)))) (x2)
An (222)
< <> 2B) <3y
(16)
R Σ (1))
(z1)) (x22)
< <NBY <c
(17)
R
Σ
2h (z (x2) x2)1,12)
<) -8= =c
(18)
More explicitly, we have the uniform bound
Σ
((11))))(((())))((((()) x(21, <C.
(19)
Letting miy E Ur. the bound takes the form claimed in the statement of the theorem. Since this matrix V of the form UUT, is clearly symmetric and positive semi-definite.
B NON-UNIVERSAL DEKL
The kernel in Eq. 4 may be written as
Koe(f1, Kin (p(1)))) ((12; 0); V),
(20)
where the linear base kernel Kim is given by Eq. 2 Theorem [ then states that kernel of this form can approximate any continuous target kernel arbitrary accuracy, given some 0 and (and sufficiently many learners); i.e., DEKL with linear base kernel is universal. However, universality fails to hold if we replace the linear base kernel with certain other popular kernels.
First suppose we replace the linear base kernel in Eq. 20 with a base kernel K R x R H R whose range is bounded either above or below by M. Suppose we wish to use Eq, 20 to approximate the dot product kernel
Kan (Z1, x22 az 22,
O,
where IT, 22 are taken to lie in the closed unit ball. This dot product kernel then has range i-a, a). select the coefficient of the dot product kernel to be sufficiently large so that either M
12

---

Under review as a conference paper at ICLR 2021
(in case Kiuse is bounded above) or M a+c (in case Kuase is bounded below) for some 0, then the deep kernel is incapable of approximating the constructed target dot product kernel, regardless of the feature network or learners used.
Next consider the linear kernel in Eq. 2 but where V is restricted diagonal positive semidefi- nite matrix, and suppose we replace Kin in Eq. 20 with this new isotropic linear kernel. In case of arbitrarily complex learners, the resulting DEKL model is of course universal, as is able to repre- sent any target kernel as arbitrary continuous feature embedding (thanks to the complexity of the learners) followed by an inner product. However, universality fails to hold if we take the learners be simple. Consider learners of the form p(a; B,) (z)( where 3 is an affine map and is the ReLU activation function. Since the diagonal elements of V are non-negative to guarantee positive semidefiniteness, then the resulting DEKI. kernel KE a sum-produce of Rel.U-activate functions with non-negative coefficients and is therefore itself non-negative. This kernel is therefore incapable of approximating target kernels that can return negative values, such as the dot product kernel.
C DERIVATION OF THE KL TERM WITH OPTIMAL PRIOR COVARIANCE
Here we derive the optimal prior covariance V, with respect to the variational free energy in Eq. and use eliminate the prior covariance from the loss function entirely. We note that V appears only in the KI. term (Eg. and only through its inverse V-1, Minimizing the KI. term with respect V-1, we obtain Eq. as follows:
vV KL V) | =0
141 + - =0
V. 1p E.
13

---

Under review as a conference paper at ICLR 2021
Substituting this back into Eq. 7 and applying the Matrix Inversion Lemma, obtain Eq. 10 as follows:
(L, (,, V.)
1
2 M (mM T +2) + ++ tr tr( T +
log det [ep T + E) -
2
2
2
1
Σー Σー Σ-1
zd
1+ 1 11-E-111 11
11
Eー1 - "1 --1
tr
+ u E-11
deet(u + Σ)
+ log
2
det E
2
1
1 14 2 uep
2
2"
11 2 + p E-11
E uu
1r
1
1 J Σ-1
det 21 + TT Σ 11)
H
log
del E
1 Σ 1 T 4 (1 1- 1+ 2
2"
tr (11²)
+ 2 21 + 15-1
log(1 n¹ Σ H
1 n -
21 / E
11 I-1
21+ E--1
log(1 + +1² - k))
2 log(1 + + TΣ-1I p).
D NEGATIVE LOG-LIKELIHOOD ON UCI DATASETS
Here list the test NLL scores of all models tested on all UCI datasets (Table Given a test set with N, data points, if y4 is the true output of the ith point and μę and of are a model's 's predictive mean and variance (including the noise), then the RMSE and NLL of the model are given by
(y) 14)2
RMSE
Σ M - m,),
NLL
log o² + log(2r),
N.
N.
20
where we note that the NLL is normalized by the number of data points. DEKL achieves lower NLL than does DE on some of the datasets, suggesting that DEKL is method that is sometimes worth trying DE fails to give satisfactory performance. We note, however, that DEKL achieves lower RMSE (Table I than does DE more often than it does lower NLL; RMSE is perhaps the more relevant metric to measure mean predictive performance, we conclude that DEKL is certainly method worth keeping in the practitioner's toolbox.
14

---

Under review as a conference paper at ICLR 2021
Table 2: Test-NLL (mean and std. dev. across 20 train-test splits) UCI datasets, using deep ensembles (DE), deep kernel learning (DKL) and deep ensemble kernel learning (DEKL). Suffices and -O indicate using KL, (Eq. [T] and KLo (Eq. 10). respectively.
Dataset
DKL-I
DKL-O
DEKL-I
DEKL-O
Boston housing
2.51 0.23
2.59 + 0.19
2.66 + 0.40
2.54 0.21
2.54 +.2
Concrete
3.08 0.28
1.01-0.11
2.99
2.94
2.95 0.12
Energy
1.66 1 1.38
1.13 0.55
1.64
1.08
0.67 0.07
Kin8nm
-1.26
1.14
-.05
-1.16
-.02
Naval Propulsion
6.69:10.11
-5.65
6.28
0.92
0.68
Power plant
0.05
2.00
0.06
2.78
2.78
Protein
2.75
2.82 0.03
2.79
2.03
2.79
Wine
0.96 + 0.10
1.03 1.3 0.23
1.01 0.09
0.96 0.07
0.97 + 0.07
Yacht
0.21 + 0.17
2.06 + 1.33
1.07 + 0.06
1.08 0.07
E REGRESSION RESULTS FOR VARIOUS NUMBERS OF LEARNERS
On the UCI datasets, we evaluate DE and DEKL (both with and without optimal prior covariance) for varying number of learners maintains superior performance over DE in terms of RMSE as the number of learners increased, although any trends in the performance gap itself are unclear. We believe in-depth study more complex problems and a much greater number of learners is required to see clearer trends, which leave for future work.
its effect test performance (Figs. 455. DEKL often
DIVERSITY OF LEARNERS
Here we present the results of the experiment described Sec. on all UCI datasets (Figs. B). Overall, DEKL tends to have more diversity than DE, sometimes by very large margin; for example, on Kin8nm, the DE learners are almost perfectly correlated while the DEKL learners are almost orthogonal. However, the relationship between diversity and predictive performance is more nebulous. On Boston housing, Concrete, Power plant, and Protein, DEKL has both more diversity and lower RMSE (Table TT than DE. However, on Energy and Yacht, both DE and DEKL have almost perfect correlation among their respective learners, but DEKL achieves lower RMSE; on the other hand, Kin8nm, Naval propulsion, and Wine, DEKL has greater diversity but not lower RMSE. Understanding when diversity is beneficial for predictive performance is topic we leave for future work.
15

---

Under review as a conference paper at ICLR 2021
Boston housing
Concrete
Energy
3.4
6.5
2.0
6.0
RASE
3.2
5.5
5.0
3.0
0.5
3
1
2
2
Learners
Learners
Learners
Kin8nm
Naval propulsion
Power plant
3.98
3.96
RMBE
0.080
3.94
0.0004
3.92
0.075
3.90
2
Leamers
Learners
Learners
Protein
Wine
Yacht
1.4.
4.4
0.6400
1.2-
RMSE
4.2
0.6375
RMSE
0.6350
0.8
4.0
0.6325
0.6
Learners
Learners
Learners
Mean test RMSE (across all train-test splits) of the DE (red, solid), DEKL-I (green, dotted), and DEKL-O (blue, dashed) models nine UCI datasets function of the number of leamnrrs.
Figure 4:
16

---

Under review as a conference paper at ICLR 2021
Boston housing
Concrete
Energy
2.65
3.2
2.60
3.1
2.55
2.50
Learners
Learners
Learners
Kin8nm
Naval propulsion
Power plant
2.80
-6.4
-1.15
2.79
-6.6
-1.20
2.78
2.77
1
2
Learners
Protein
Wine
Yacht
2.85
0.97
1.0
0.96
2.75
0.95
5
2
4
Figure 5: Mean NLL (across all train-test splits) of the DE (red, solid), DEKL-I (green, dotted), and DEKL-O (blue, dashed) models on nine UCI datasets as function of the number of learners.
17

---

Under review as a conference paper at ICLR 2021
1.00
1.00
D1K--
106
0.75
0.75
0.75
0.0
0.38
0.50
023
0.23
0.23
1003
100
0.00
-0.25
-0.32
-0.23
-3.03
1.00
Concele
-0.55
0.75
0.75
0.35
-1.00
-1.c0
032
435
0.00
0.00
8.00
-0.35
-025
-0.5
-0.0
-010
-030
Figure 6: Correlation matrices of learners averaged over all train-test splits the Boston hous- ing, Concrete, and Energy datasets using deep ensembles (DE) and deep ensemble kernel learning (DEKL). Suffices -I and -O indicate using KLr (Bq. II and KLo (Eq. TO), respectively.
18

---

Under review as a conference paper at ICLR 2021
1.00
1.00
DIK-0
108
0.75
0.75
0.75
0.00
0.38
0.50
023
0.23
0.23
1003
100
1000
-0.25
-0.32
-0.23
-3.03
-3.03
1.00
Noval Propud
-0.55
0.75
0.75
0.75
-1.00
-1.00
032
Pawer plant
835
0.00
0.00
3.00
-0.25
-025
-0.5
-0.5
-010
-030
-035
Figure 7: Correlation matrices of learners averaged over all train-test splits the Kin8nm, Naval propulsion, and Power plant datasets using deep ensembles (DE) and deep ensemble kernel learning (DEKL). Suffices -I and -O indicate using KLr (Bq. II and KLo (Eq. [0), respectively.
19

---

Under review as a conference paper at ICLR 2021
1.0
1.00
DIK-0
108
0.75
0.75
0.75
0.0
0.38
0.50
023
0.23
0.23
Freein
1003
100
1000
-0.25
-0.23
-0.32
-0.2
-3.03
-3.03
1.00
Whe
-0.25
0.75
0.75
0.35
-1.00
-1.00
032
435
10oht
0.00
0.00
8.00
-0.35
-025
-0.5
-0.0
-0.0
-030
Figure 8: Correlation matrices of learners averaged all train-test splits the Protein, Wine, and Yacht datasets using deep ensembles (DE) and deep ensemble kernel learning (DEKL). Suffices and -O indicate using KL, (Eq. [1] and KLo (Eq. 10), respectively.
20


Reviews:
[{"review_id": "xKuMFfzeJ2p", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a variant of the Deep Kernel Learning model (DKL) [1] where multiple independent networks are trained for the features instead of a single network. In addition, the paper proposes to use a linear kernel as a base kernel which allows for universal approximation of any arbitrary kernel, as well as allowing for exact inference of the kernel hyperparameters. The use of stochastic variational inference is proposed for inferring the neural networks weights. The model is compared against Deep Ensemble (DE) and DKL on a synthetic dataset and the UCI dataset.\n\nThe paper is well written, easy to follow and is technically correct. The main weaknesses of the paper is that by using a linear kernel as the base kernel, it is not clear how the proposed model is different from a Bayesian Neural Network with an extra linear layer. Furthermore, the experiments do not necessarily showcase the strengths of the proposed model, namely, the universal approximation property and the regularization that an ensemble provides. I think the paper could increase its impact by effectively implementing distributed training as well as improving the experiments.\n\n[1] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning.\nIn Artificial Intelligence and Statistics, pp. 370–378, 2016b.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nx_AOaC4wme", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nIn this paper, an extension to deep kernel learning is proposed. A linear kernel is used as the base kernel, which enables exact optimization of the kernel hyperparameters. There is a universal approximator theorem stating that the deep neural network with linear kernel could approximate any kernel function, which is quite obvious from the perspective of random Fourier features as well. Also, multiple neural networks are used to produce features and the features are concatenated.  I am not sure why the word ensemble is used, but it is really a concatenation of features instead of ensembling predictions. Besides the exact inference, standard variational inference is also proposed for the linear base kernel. Experiments are conducted on synthetic data and UCI datasets, with comparison to DKL with linear kernel and deep ensembles. \nIn general, I could not find very interesting contributions from the paper. The framework follows closely from DKL with a linear kernel. But I have a question about the proof of the universal approximator theorem, to approximate the different eigenfunctions, I would assume the hidden layer of the neural network to be at least of O(B) where B is the number of eigenfunctions to be approximated. So potentially, the neural network needs to be very wide which might not be practical. Also, I am confused why the “concatenation” of features is referred to as “ensemble” in the paper. It is super confusing to me unless I am not understanding how the features are used jointly. Also, the authors mention that “a learner with M output is simply a concatenation of M single-output learners, suggesting that multi-output learner may help to further reduce the number of H of required learners.” With the same number of nodes in the hidden layers, I don’t think a multi-output learner is equivalent to M different single-output learners, therefore the argument made here might not be valid.\nIn terms of experiments, it is weird that for DKL linear kernel is used instead of the original spectral mixture kernel with random Fourier features. It makes the most sense to compare the linear kernel with something like RBF or spectral mixture kernel. Also, UCI regression tasks seem to be rather easy and do not necessarily need a deep neural network to do feature engineering. It would be necessary to conduct experiments on more complicated tasks such as images to validate the effectiveness of the proposed linear kernel approach. \n\n------------After author's response----------------\n\nMy major concern is about the connection between the universal approximation theorem and the proposed architecture. In the paper, the authors mentioned that \" the following universal approximation theorem for DKL implies that this effect can be compensated by adding parallel learners\". However, the fact that a multi-output learner is not equivalent to M different single-output learners makes it hard to justify the proposed architecture theoretically from the universal approximation theorem. \nI think this is something that is crucial to be justified, otherwise, the theory does not really match with the proposed method.", "extra_scores": {}, "extra_text": {}}, {"review_id": "A-pyFNZxUB2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce a deep ensemble kernel learning approach as a linear-based learning combination, from a deep learning scheme, to approximate kernel functions under a Bayesian (GP) framework. Namely, a universal kernel approximation strategy is proposed from eigen-based decomposition and deep learning-based function composition. Then, a variational inference strategy is used to solve the optimization from kernel-based mappings. Two regularization strategies are studied: optimal prior covariance and isotropic covariance. Results demonstrate the benefits of the proposal.\n\nTwo comments:\nCould you include the GP and sparse GP results to compare the predictions? Your approach can be scalable from the stochastic variational variance; however, the studied databases are small in terms of the number of samples so that the well-known GP algorithms could be compared.\nIn most of the cases, isotropic and optimal covariance-based methods seem to achieve the same results; why? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-v1nwGDZYA8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a special case of deep kernel learning (DKL) using a linear base kernel, where the inputs to the kernel are the outputs of neural nets with identical architectures (which could be thought of as learners that get ensembled); the resulting approach is called deep ensemble kernel learning (DEKL). The paper provides a universal kernel approximating result for DEKL (Theorem 1), explains how to train DEKL efficiently for large datasets using variational inference, and demonstrates that DEKL can outperform deep ensembles and DKL on various datasets.\n\nOverall, this paper is well-written and easy to follow, and appears to combine some standard ensembling ideas with DKL. I think the paper would benefit from more extensive literature review and a more thorough experimental results section. See the \"weaknesses\" listed below for details. In terms of significance, I find this paper to constitute more of an incremental advance.\n\nStrengths:\n- very well-presented, intuitive method\n- theory and experiments are easy to follow\n\nWeaknesses:\n- Theorem 1 seems straightforward and rather unsurprising given the Universal Approximation Theorem; moreover, approximating the kernel to be of the form phi(x)^T phi(x) (so taking V to be the identity matrix) is already known to be able to approximate a variety of kernels with arbitrary accuracy when phi is chosen with appropriate randomness (see Rahimi and Recht’s \"Random Features for Large-Scale Kernel Machines\" (2007) and also the follow-up paper \"Weighted Sums of Random Kitchen Sinks: Replacing minimization with randomization in learning\" (2008)) -- perhaps it would be helpful to discuss a bit how your approach relates to these existing methods (there seem to be some similar ideas being used); naturally, a related question I have is how much performance degrades if V is just taken to be identity rather than using the choice given in equation (7)\n- for the experiments, providing some basic characteristics of the UCI datasets used would be helpful (e.g., dataset size and dimensionality), to get a sense of how high-dimensional the datasets are\n- I didn't really get intuition for when/why DEKL should outperform DKL (especially in terms of dataset characteristics), and am left wondering how much simpler methods would fare on the UCI datasets (e.g., linear regression, random forest regression, or other classical methods); for example, is DKL optimizing over too rich a model class or something?\n\nTypo right after equation (2): $\\varphi$ is missing a close parenthesis", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a DKL variant with a linear kernel. Representations from several networks is combined through concatenation, making it not quite an ensemble. It's shown that the model is a universal kernel approximator. Experiments are conducted on a large number of UCI datasets.

Following the discussions, the paper still has the following shortcomings:
- some lack of clarity in the presentation (for instance, explaining the equivalence between a multi-output learner and M different single-output learners)
- lack of experiments on data where deep learning is typically used (images); the UCI datasets have structured data and other ensembles like XGBoost may outperform the baselines presented in this paper
- difference in performance between DKL and DEKL, especially since DKL benefits from a larger model space, theoretically. maybe DEKL has better sample complexity, but does this advantage hold in the case of the large datasets that deep learning is used for?
Decision: Reject