Structural Kernel Search via Bayesian Optimization and Symbolical Optimal Transport
Authors: Matthias Bitzer
Mona Meister
Christoph Zimmer
Venue: NeurIPS 2022 Accept (2022)
Keywords: Bayesian Optimization
Gaussian Process
Kernel Search
Kernel

Abstract: Despite recent advances in automated machine learning, model selection is still a complex and computationally intensive process. For Gaussian processes (GPs), selecting the kernel is a crucial task, often done manually by the expert. Additionally, evaluating the model selection criteria for Gaussian processes typically scales cubically in the sample size, rendering kernel search particularly computationally expensive. We propose a novel, efficient search method through a general, structured kernel space. Previous methods solved this task via Bayesian optimization and relied on measuring the distance between GP's directly in function space to construct a kernel-kernel. We present an alternative approach by defining a kernel-kernel over the symbolic representation of the statistical hypothesis that is associated with a kernel. We empirically show that this leads to a computationally more efficient way of searching through a discrete kernel space.

Structural Kernel Search via Bayesian Optimization and Symbolical Optimal Transport
Matthias Bitzer Mona Meister Christoph Zimmer Bosch Center for Artificial Intelligence, Renningen, Germany (matthias, bitzer3, mona. meister, christoph. zimmer)@de. bosch com
Abstract
Despite recent advances in automated machine learning, model selection is still a complex and computationally intensive process. For Gaussian processes (GPs), selecting the kernel is crucial task, often done manually by the expert. Addition- ally, evaluating the model selection criteria for Gaussian processes typically scales cubically in the sample size, rendering kernel search particularly computationally expensive. We propose novel, efficient search method through a general, struc- tured kernel space. Previous methods solved this task via Bayesian optimization and relied on measuring the distance between GP's directly in function space to construct kernel-kernel. We present an alternative approach by defining a kernel-kernel over the symbolic representation of the statistical hypothesis that is associated with a kernel. We empirically show that this leads to computationally more efficient way of searching through discrete kernel space.
Introduction
In many real-work applications of machine learning, tuning the hyperparameters or selecting the machine learning method itself is crucial part of the workflow. It is often done by experts and data-scientist. However, the number of possible methods and models is constantly growing, and it is becoming increasingly important to automatically select the right model for the task at hand. Bayesian optimization (BO) is prominent method that can be used for model selection and hyperpa- rameter tuning. It can handle black-box oracles with expensive function evaluations, which are two characteristics often encountered when doing model selection [17]. Important applications
this context choosing the hyperparameters in the training process of deep neural networks (DNN) 117.
or dealing with discrete and structured problems like choosing the architecture of DNN's 5 1T.
Gaussian processes (GP) are another important model-class. They are often utilized as surrogate models BO [77, for time-series and statistical modeling 77 3] or in active learning loops [23 [5]
The properties of GP's mainly governed by its kernel that specifies the assumptions made on the underlying function. Choosing the right kernel is therefore a crucial part of applying GP's and is often done by the expert. Recent work [9] treated the kernel selection problem and used Bayesian optimization solve it. This allowed searching over a highly structured, discrete space of kernels. However, their proposed kernel-kernn measures the distance between two GP's directly in function space, which a computationally expensive task itself. This makes the method difficult to apply for the frequent scenarios where the evaluation of the model selection criteria requires only a medium amount of time.
black-box optimization
We propose measuring the distance between two kernels via their symbolical representation of their associated statistical hypothesis. We utilize the highly general kernel- -gramarr presented in [3), as underlying kernel space, where each kernel is build from base kernels and operators, like e.g.
LIN + ((SE x PER) + SE)
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

forming effectively a description of the statistical hypothesis that is modeled by the GP. Our main idea build a distance over these symbolical descriptions, rather than measuring the distance between two GP's directly in function space. We employ optimal transport principles, known from neural- architecture search (see TT [,, to build a pseudo-distance between two hypotheses descriptions and use it to construct kernel -kenel, which is subsequently utilized in the BO loop. We will show that the induced kernel search method is more efficient, in terms of number of function evaluations and computational time, compared to alternative kernel search methods over discrete search spaces.
The main challenge we encountered is the quantification of dissimilarity between two symbolical representations of kernels. We use the tree representation of each symbolical description and apply optimal transport distances over tree features. Subsequently, we empirically show that the deduced GP over GP's s provides a well-behaved meta-model and show its advantages for kernel search. In summary, our contributions are:
1. We construct pseudo- distance over GP's that acts over the symbolical representations of the underlying statistical hypothesis.
2. We use the pseudo-distance to construct a novel "kernel-kernel" and build it in a BO loop to do model selection for GP's.
3. We empirically show that our meta-GP model is well-behaved and that we outperform previous methods in kernel search over discrete kernel spaces.
There is consideralle existing work on constructing flexible kernels and learning their hyperparameters 20 21 18 22). In these kinds of works, the structural form of the kernel is predefined and the free parameters of the kernel are optimized, often via marginal likelihood maximization. While being able to efficiently finding the hyperparameter due to the differentiability of the marginal likelihood, one still needs to predefine the structural form of the kernel in the first place. This is hard task one need to decide if long-range correlation or nonstationarity should be considered or if dimensions are ignored or not. Our method is build to automatically select the structure of the kernel. Some of the mentioned methods 20 21 are able approximate any stationary kernel via Bochner's Theorem and therefore consider a broad kernel space themselves. However, even elementary statistical hypotheses require nonstationary kernels, such linear trends modeled by the linear kernel. Our search space is not restricted to stationary kernels.
Related Work:
We consider highly general, discrete kernel space that is induced by the kernel grammar 3 Recent work [9] used BO to search through this space via a kernel that measures similarity in function space. We also utilize BO but employ fundamentally different principle of measuring the distance, which is computationally more efficient. We dedicate Section 3.2 to a more precise comparison to the method of [9. Additionally, the original kernel grammar paper 3 suggested greedy search for searching through the kernel grammar. Furthermore, [4] employed a genetic algorithm based on cross-over mutations. We empirically compare against all approaches.
In the area of BO over structured spaces, our method most similar to the neural-architecuree search (NAS) procedures presented in [ who use optimal transport distances over features extracted from the graph-representation of neural networks. Compared to these methods we use OT principles to do model selection for GP's which is a fundamentally different task.
2 Background and Set-up
Our main task is efficient model selection for Gaussian processes. In order to provide background information, we give a small introduction to GP's and model selection for GP's. Subsequently, we present review of the kernel-grammar 3 and show how we use it for our approach. We show how a kernel can be represented via a symbolical description. In Section 3 we will present how we use the symbolical description construct kernel over kernels and how use it in the BO loop.
2.1 Gaussian Processes
A Gaussian process is distribution over functions ʃ: - RR over given input space X' which is fully characterized via the covariance/kernel function k( = Cov(s(s), [(x') and the mean function u(x) ;= [z)) We therefore can write as shorthand notation f (P(u()),k(;)))) The

---

kernel can be interpreted as a similarity measure between two elements of the input space, with the GP assigning higher correlations function values whose inputs are more similar according to the kernel. Furthermore, the kernel governs the main assumptions on the modeled function such as smoothness, periodicity or long-range correlations and therefore provides the inductive-bias of the GP. An important property of GP's is that they are not restricted to euclidean input spaces X' C Rd. but can also be defined on highly structured spaces like trees and graphs, a property we will later use to define a GP over GP's.
While our method might also be used for model selection in classification, we consider from now on Gaussian processes regression. For regression, a dataset D = (X. y) with X = {I1. (I1...... IN C X and y (y1, ................................ NN E RN is given, where we suppose that (P(u()), k(.))) and y4
f(a + e, with e; GP with mean and covariance functions
i.id (0, o²). Given the observed data DD the posterior distribution S/D again
(x(x) = m(x)) + k(x) 02 ((X)),
kp(a, y) = k(,, v) (K +
with K = [R= mn)) and k(x) [M<<r, (1), ..),...,, k(x,, ((ar,,00 (((x,NN)) xN)) ) (see 13) Probabilistic predic- tions can be done via the resulting predictive distribution |= D) =N(uD(x*) kD(w" (,""="))) ,)))).
2.2 Model selection for GP's
Typically, the kernel kg comes with set of parameters 0 that can be learned via maximization of the marginal likelihood p(y f, f. (MAP) estimation, in case the parameters O are equipped with prior p(0). This procedure is sometimes also called model selection, as one selects the hyperparameters of the kernel given the data (see (3).. However, we consider selecting the structural form of the kernel itself. The structural form of the kernel determines the statistical hypothesis that is assumed to be true for the data-generating process. Intuitively, the kernel is similar to the architecture in deep neural networks, which induces an inductive bias.
N(y (X), ko (X,X) + I²I) or via maximum a posterioni
Our goal is to do model selection over discrete, possibly infinite space of kernels K :- (ki,, k..... As each kernel comes with its own parameters, we are actually dealing with a space kernel families. Thus, when mentioning a kernel it we associate it with its whole family over parameters {kg 10 E O). Once a kernel is selected, predictions are done with learned kernel parameters (that usually are a by-product of calculating the model selection criteria). The parameters 9 are potentially equipped with a prior p(0) depending on the selection criteria. As mean function, we always utilize the zero mean function yι(=)= O in case y is centered and a constant mean function otherwise. This is common choice GP regression. Given some model selection criteria g :. - R our task is solving specific model selection criteria,
k- = arg max g (D). While our method is not restricted we focus on the model evidence p(y X, k), which is a well-known selection criteria for probabilistic models (see 14 3) Given prior on the kernel parameters p(0) and the likelihood variance p(o²). the logelodel evidence of the marginalized GP is given as
g(k(D) = log x(y|X, k) = log plylx, 0, ()(p(o))((0k)))ppl))
This quantity can be approximated via Laplace approximation of p(0, (D (see Appendix A for details). Computing this approximation includes performing a MAP estimation of the GP parameters. Thus, once the log evidence has been computed, learned kernel hyperparameters OMAP are automatically provided. Performing the MAP estimation scales cubically in the data set size N, which renders model selection for GP's computationally intense.
2.3 Kernel Grammar
The kernel grammar introduced by 3 specifies highly general search space over kernels. The grammar based on the observation that kernels are closed under addition and multiplication, meaning, for kernels kn (z. x²) and (a also ki (xr,x') x²) + k2(2,, x²) and kj (x, x') x ha(ar, x') are kernels. Given some base kernels, such as the squared exponential kernel SE, the linear kernel LIN, the periodic kernel PER or the rational quadratic kernel RQ, different statistical hypotheses can be
3

---

generated via addition and multiplication. For example,
LIN + PER x SE
describes linear trend with locally periodic component, such that it might be a useful hypoth- esis for time-series applications. For multidimensional data the base kernels are applied single dimensions, denoted e.g. with SE, for the squared exponential kernel defined on dimension i. The grammar contains many popular hypotheses such as the ARD-RBF kernel that can be expressed via multiplication over the dimensions II SE, as well as additive models Σ SE, or polynomials of order m with
LIN.
The specification of the complete search space is given grammar, where base kernel is denoted as B and a subexpression is denoted with For example, in the expression LIN + (PER x SE) the expression PER SE is subexpression. Starting with all base kernlls, the search space is defined as all kernels that can be reached via the following operations:
1. Add a base kernel to a subexpression: S - S +
2. Multiply subexpression with base kernel: +
3. Exchange a base kernel with another base kernel: B + B'
Starting from the set of base kernels, these operations can lead to all algebraic expressions, showing
expressiveness of the grammar [3]. The authors 3 suggest using greedy search to search
through the grammar, where the kernel with the highest value for the selection criteria is selected and expanded with all possible operations. After the selection criteria is calculated on the neighbors, the search progresses to the next stage by expanding the neighbors of the best kernel found.
We consider generalized notion of the kernel grammar, where we consider set of base kernels {B)...., B, J. E N and a set of operators " Ti}, I E N where T₃ K. x K. + K,j = 1, I....,/
are closed operators on the space all kernel functions K. Examples are addition and multiplication, but also the change-poin operator which was considered in 7 Thus, in general the grammar operations are
1. Apply operator T, onto a subexpression and base kerne _ S - T,,,
2. Exchange base kernel with another base kernel: B - B
The considered kernel space can be defined precisely in the following way: Definition For I, E N, let (KI..... } be set of (base-) kernels (symbollically represented as (B1....,B,)) and (Tı..... } a set of operators with T, : K. x K. + K.jj =1 =....., Let
Lo = {k1,...,ke .
L = (T, (k1, K2) k1, 22 E Li-1s5 = 1;
for = =...., M. We call i == Las the kernel- -ene--raammar generated kernel space with depth M.
2.4 Representation of Kernels
The base kernels and their combination via the operators describe the structural assumptions of the final kernel and imply the assumptions that are made function space. Our main hypothesis that the symbolical representation of the kernel, given by the subexpressions S and base kernels, already contains sufficient information for model selection. Therefore, we will define a kernel-kernel over the symbolical representations and utilize it for Bayesian optimization. Concretely, each kernel k K in our described space can be written as a tree T, for example:
ADD
LIN
MULT
LIN + ((SE + PER) X SE)
ADD
SE
SE
PER
In case the operator T', is not symmetric we also add the operation $ - (B, S). However, all operators considered in this work are symmetric.

---

Each operator T and each base kernel B; is represented by their respective name, where operators the nodes of the tree and base kernels are the leafs. The way how the operators and base kernels are connected is represented through the tree structure. For given expression tree T, we denote the multiset of all subexpressions/uubree as Subtrees(7) (T). Furthermore, we consider paths the leafs of the tree, for example, for the tree above one path to leaf would be:
ADD - MULT - ADD - PER
We denote the multiset of all paths in the tree T as Paths(T). Lastly, we also consider the multiset as Basc(T). For each deseribed multiset,
of all base kernels that exist in given expression denote the number of occurrences of element & in the multiset as (n)).. When building the multisets also account for two symmetries in the elements that can be applied if operator associative and commutative, which we elaborate further in Appendix A.
Depending on the operators that are used, several trees can describe the same kernel & E K. For technical reasons, we denote with f K + T T(K) a mapping that maps a given kernel E k to one tree T that induces this kemel, where T (K) denotes the set of all expression trees that can generate a kernel in IK (see details in Appendix A).
3 Kernel-Kernel
Our kernel-kernel will be defined via a pseudo-metric over the expression trees. Optimal transport (OT) principles have proven themselves to be effective in BO methods over structured spaces, as shown in (5] and [ for neural architecture search or [6] for BO over molecule structures. We only
follow this line of work and also rely optimal transport, although
simple ground metric to allow for closed-form computations. To allow OT metrics to be used, we summarize each expression tree T to discrete probability distributions of their building blocks Base(T), Paths(7) and Subtrees(T) via
Whsse
Σ CTBase(T)
wgdg
Upaths
wedg
Wabttrees
Σ
wede
SEPatha(T) T)
EESubtree(T)
where &g is the Dirac delta and we is the frequency of the element & in the respective multiset. For example, the frequency of expression SE x PER in the multiset Subtree(T) is calculated as
WSEXPER = n(SE x PER)
[Subtree
Each probability distribution represents a different modeling aspect that is induced by a kernel kt e İ and its corresponding expression tree T:
1. Wbase specifies which base kernels are present in T, thus, which base assumptions in function space are included such periodicity, linearity, local smoothness.
2. wpaths specifies how the base kernels in T are used, e.g. whether a periodic component is applied additively or multiplicatively.
3. Waubtrees specifies the interaction between the base kernels in T, for example if contains an addition of linear and periodic component or not.
Our pseudo-metric between kernels j and ko (and its associated trees Ti and T2) uses all three modeling aspects via the optimal transport distances between the respective discrete probability distributions Whase, Wpaths and Waubtrees
In general the OT distance with ground metric d between two discrete probability distribution W1 Exen di and we Σeen w2, over Ω is defined
W W w2) inf EER(
where æ E R(w,, w2) a combined probability distribution over with marginal distribution (A x ?) wi (A) and x(2 x B) (BB for Borel sets A, B. While for general ground metric
5

---

d x 92 + R the optimization problem in 1 can not be solved in closed-fore we use as ground metric d(E, E 1848 which has closed-form solution the total variation distance between w1 and w2 (see 199, p. 22), i.c.
1 Σ EER
W Wa(wi, W2)
W1,E w2,8
2
Utilizing this distance over the modeling assumptions Wbase; Wpaths and Waubtrees allows for fast computation of the final pscudo-metriy and for proper positive semi definite (p.s.d) kemel-kernel in the end [see Appendix CJ. This is not guarantueed for general OT distances (see [[I]). We define the final distance between two kernels i and 22 (and its associated trees Ti and T2) sum over the OT distances of their respective modeling components
d(T1, T2) =- W Wi(Wi, [(W1, base)
+02 W W2(Wi, W2.paths)
(2)
+ag W subrree + W2,subtrees),
where a; 2 0 and Σ; a; 1 are weighting parameters that will later be learned automatically via marginal likelihood maximization for GPs.
In case the kernel grammar contains base kernels that act on single dimensions such as SE,, we define for each dimension :..... D an individual distribution over base kernels
wg6g
base
(EBase(T.o)
where Base(T, i) is the multiset of all base kernels in T defined on dimension i. We also include the empty-expression ENULL for which WExuLa = whenever no base kernel of dimension contained in T. base thus summarizes which base kernels are present that act on dimension or if the dimension ignored. The distance in this case defined as
d(T1, T2) =0 Σ w Wa(w base W base
+02 W Wa(Wi, WaWWh, W2.pathss
+agWaale Wayga((1,suttrrrss w2.sbrresss)
The reason for this distinction is that two kernels kn and k2, which have different active dimensions, are considered to be farther apart with this distance, making it casier to allow variable selection, which is important aspect of model selection.
Our defined function d(T1. Tz) induces indeed pseudo metric in the kernel -rrammr generated kernel space, shown in the following proposition. In particular, it fulfills the triangle inequality.
Proposition 1
Let K. be the kernel space generated by kernel grammar. Let f 12 + T(K) be mapping that maps kernel IK one of its expression trees T. Then d(k), K2) == d(F (ki), (k2)) a preudo-metrie on K. where d given by (2) or 3 depending on the base kernels in IK.
Given the useudo-metric we are now able to define a kernel-kernel with
Ksor(k1, k2) = oeexp -d(kk 1z) 12
(4)
which we call Symbolical-Optiin Optimal-Transport (SOT) kernel-kernel Here, / denotes the lengthscale and o² the variance of the kernel-kernel. Both parameters are learned in combination with the distance weights via marginal likelihood maximization (see Appendix A).
3.1 Bayesian Optimization for Model Selection
We utilize the proposed kernel-kernel to do model selection for GP's via Bayesian optimization, which similar task to [9]. Compared to (9. we use our proposed kernel-kernel, which is fundamentally different and computationally more efficient way of measuring similarity in GP space.

---

Table 1: RMSE on test kemeellog-eviidence pairs (a 0.05)
Dataset
Hellinger
kNN
Mean
SOT (ours)
Airfoil
0.1773 (0.004)
0.2231 (0.013) 0.3769 (0.002)
0.0936 (0.003)
Airline
0.3569 (0.007)
0.3813 (0.011)
0.4013 (0.005)
0.3464 (0.011)
LGBB
0.3772 (0.021)
0.6795 (0.025)
0.8519 (0.016)
0.2783 (0.008)
Powerplant
0.1912 (0.012)
0.2236 (0.011)
0.2925 (0.009)
0.0137 (0.005)
Concrete
0.2489 (0.006)
0.1806 (0.010)
0.2912 (0.005)
0.0451 (0.002)
Given a model selection criteria gD : K + R for a given dataset D, we want solve k- arg max (k(D) via Bayesian optimization. We thus define surrogate GP model for (kk(D) via
G-GP((())) KsoT(::))
where (k( = is the constant mean function. As BO acquisition function a()) we appected-Improvemenn (EI). Here, D. denotes the of already queried kermel-selection-criteei pairs (k.g (k) for which the meta-GP posterior f(D, calculated. Starting with initial set of pairs Do we follow the standard BO iterations where in each iteration the acquisition function is maximized K2 arg ma a(k given the current set of already evaluated kernel-selection- criteria pairs D.. Then the selection criteria is queried at the chosen kernel [D) A complete description of the method can be found in Algorithm I in Appendix A.
A crucial part in the BO cycle is the optimization of the acquisition function max a (k))D)). While for Euclidean spaces gradient- this is not option for a structured space like the considered grammar-generated kernel space. We propose using an evolutionary algorithm to optimize the acquisition function, which can be seen in Algorithm 2 in Appendix A. While evolutionary algorithm seems to be computationally intense procedure that takes place in each BO iteration, we emphasize that the evaluation of the acquisition function is very efficient for our proposed method. The reason for the efficiency is that the evaluations of our kernel-kernel K Ksor(M, (k1, k2) at two kernels ki l2 K is very cheap, compared for example to the method in 9 (see Appendix for computational time comparision).
methods or grid-based methods could used to solve that task,
3.2 Comparision to Hellinger Kernel-Kernel
The work of [9] also uses BO for kernel selection but with different principle of measuring the distance in GP space. In particular, they propose measuring the distance of two GP's M and M' via the induced prior distributions p(f M) in function space evaluated on the design matrix X of the dataset. Conditioned on the parameters of the GP 6, the distributions p X, (, 0) are Gaussian and they the Hellinger distance over Gaussians base distance between GP models d[Me, M'g (0, 0') given the hyperparameters. They finally take the expectation over the hyperparameter priors to construct their final distance. Thus, for each kernel-kernel evaluation K (MM, M') the integral
d(M,M') / d(Mo, 0') ((0)p(((00))000))
needs to be computed, where the integrand scales cubically in [X]. Each kernel-kernne evaluation is therefore a computationally hard problem by itself. They side-pass this issue partly by selecting subset of X input locations and by using sample-based estimation of the integral. While this could make the method useful for oracles with very long run times, such as model selection for very large data sets, we see conceptual problems in the case of medium oracle run times. For our method, kernel-kerne) evaluations K ((K1, k2) are very efficient as they scale only in the size of the expression tree and no integrals need be computed.
Experiments
In the following section, we show experimental results for our novel meta-GP model and kernel search method. We evaluate our meta-model on a meta-regression task where we predict test kernel- log-evidence pairs based on training pairs. Secondly, we consider kernel search and compare it
7

---

LGBB (d-2,N-150) 50)
Powerplant (d=4,N=500)
Airfoil (d-5 N-500)
Airline (d=1.N-100)
Concrate (d-8,N-500)
X-ans: CPU-Time
Figure 1: Plots for the model selection task over number of evaluated models (top) and CPU-time (bottom) showing the normalized log-evidence of the best model found up to this iteration/tins point.
against the greedy method in [3] the evolutionary algorithm in 4 and against the BO method [9] The implementation of our method is available at htttp:://githu/
Selection Criteria and Datasets: In all our experiments, we consider the normalized log-model evidence (({[D) log p(y X /)) as model selection criteria, as also done in (9]. We always use the Laplace approximation calculate the log-model evidence, where we use repeats to do MAP estimation of the kernel parameters. Furthermore, we consider the following publicly available datasets:
Airline, LGBB, Airfoil, Powerplant, Concrete. Airline a one dimensional time-series dataset, LGBB is a two-dimensional dataset with low observation noise. Powerplant, Airfoil and Concrete are four, five, and eight dimensional, whereas Powerplant exhibits higher observation noise. We use medium-sized training sets 100 and 150 datapoints for Airline and LGBB and 500 datapoints for the other three datasets. All training sets are uniformly drawn from the full datasets. The outputs are normalized and the inputs are scaled to be in the unit interval. Further details can be found in Appendix D.
Search Spaces: We consider two search spaces. The first consists of the base kernels SE., LIN,, PER,, RQ, and the operators + and x. This is the space considered (3] for time series and low-dimensional datasets. The second search space uses base kernels SE, and RO, and also + and x as operators and was considered in (3) for higher dimensional base datasets. We consider the first space for Airline and LGBB and the second for Powerplant, Airfoil, and Concrete. The hyperparameter priors for the base kernels can be found in the Appendix A.
Prediction of Selection Criteria: We evaluate our meta-GP model on a meta meta-regression task by predicting test cernel-log-evidenee pairs based on training pairs. The quality of predictions in kernel space might also be an indicator for good performance in BO for kernel search. Given the base dataset D we create a training set D [k,, ((x::D))): E Ktrain C K, i 1,..., Merain } and test set Drest [(k,, (ik,gg(,, pairs. We generate the train and tests sets exactly in 19]. where we create first initializing it with all base kernels and iteratively pick one kernel of the current set, apply one random operation of the kernel grammar and add the resulting kernel to the current set and repeat this process until we have Titrain Attstt kernels in Rcomphete- We then divide the set uniformly into Ktrain and Keest. We compare our model to the mean preddctor as baseline, which just predicts the mean of the train set at each test point and a kNN predictor based on the kernel- grammar operations [see Appendix or [. Furthermore, we compare against the meta-GP model [9. We report root mean squared enror (RMSE) scores on the test sets in Table [ It can be observed that our method leads to more precise predictions on all four meta prediction tasks, indicating that the symbolical representations already contain sufficient information to predict log-evidence values.
g(k, [D); E Keest C K 1,... "ttstt) each containing 500 kernel-log-evidencc
set Keomplete by
kernel-grammar
8

---

Table 2: Predictive negative log-likelihood values on test set at final time stamp, Values are marked bold if they are not significantly different from the best value according to two-sample t-test (a = 0.05).
Dataset
Greedy
Hellinger
Tree-GEP
SOT (ours)
Airline
-0,4042 (0.615)
0.3368 (0.207)
-0.5594 (0.580)
-0.7015 (0.471)
LGBB
-0.7528 (0.661)
-0.8787 (0.854)
-1.0701 (0.532)
-0.9325 (0.492)
Powerplant
-0,0053 (0.054)
0.0580 (0.037)
-0.0241 (0.057)
-0.0661 (0.032)
Airfoil
0.0837 (0.026)
0,9080 (0.205)
0.1826 (0.118)
0.1006 (0.090)
Concrete
0.3254 (0.019)
0.6633 (0.207)
0.2812 (0.074)
0.2872 (0.044)
Concerning kernel search, we compare against the BO method that employs the Hellinger kernel-kernel [9], against greedy search [3] and against the evolutionary algorithm presented in 4 referred to as TreeGEP. Both BO methods run for 50 iterations and the kernel-kernel hyperparameters are updated in each iteration via marginal likelihood maximization. Our method applied the evolutionary Algorithm 2 (see Appendix A) to optimize its acquisition function, using a population size of 100. As it is computationally unfeasible to apply the same kind of acquisition function optimization for the Hellinger kernel-kernel we their method of optimizing the acquisition function where an active set of kernels is updated in each iteration. Both BO methods and TreeGEP start with the same set of initial kernels, for which apply two random grammar operations for each base kernel. Greedy search by design needs to start from the empty kernel. We give it head start by the number of initial datapoints (see Appendix D). For each dataset, we display results from 30 independent runs with different seeds, namely medians and quartiles of the log-evidence score over iterations and CPU-time in Figure [ The implementation of both BO methods is based on GPflow [10]. Further experimental details and parameter settings for all methods can be found in Appendix D.
Model Selection Setup:
As shown in Figure [ we outperform all methods in terms of perfor- mance over number of model evaluations. This is not surprising against the two heuristics, as they are not optimized towards keeping the number of model evaluations low. However, it is notable that we are more sample-efficient compared to [9, who solve much harder problem for comput- ing their kernel-kernel. In terms of performance over CPU-time, we outperform all methods on Airfoil, Powerplant, L.GBB and Concrete significantly. On Airline, the advantage over the heuris- tics is smaller the reason is the lower oracle time- which benefits the heuristics that do not need to optimize the acquisition function. The reason for the poor performance of the Hellinger kernel-kernel in terms of CPU-time is the high ratio of acquisition function optimization to oracle time, which was as high as 50 in our experiments (see detailed numbers in Appendix D).
Model Selection Results:
Test Performance: When optimizing model selection criteria, one expects that this also material- izes in a better test performance. In Table ( we therefore show the predictive negative log-likelihood (NLL) scores on held out test-set of the selected models at the final time stamp. We observe that the advantage in the model selection value gets transferred to the test performance.
Further Experiments: In Appendix E include further investigations the behavior of our method on simulated data coming from a ground-truth kernel. Furthermore, we include a comparison of the selected kernel to the standard RBF kernel and against Functional Kernel Learning (FKL) 22
5 Limitations
In case the dataset size is very small, greedy search or evolutionary algorithms might have an advantage terms of computational time as the optimization of the acquisition function outweighs the computation of the model selection criteria. Thus, in these instances our method might not show a strong benefit. Furthermore, the kernel grammar is often used for downstream applications that use the selected hypothesis for interpretation, such as 7 who build an automatic natural language description of the selected hypothesis. Depending on the dataset and how many steps are employed in the acquisition function optimization, relatively large hypotheses might be found as optimal. This

---

might render the interpretation difficult. However, we show in Appendix E example configurations of our algorithm that can be used get smaller, well interpretable hypothesis.
Conclusion
We presented a novel way of doing BO for model selection of GP's by measuring the distance between two GP's via the symbolical description of the underlying statistical hypothesis. The main contribution is the deduced pseudo-metric over kernels and the resulting SOT kernel-kernel We show that our approach leads to more efficient way of searching through discrete kernel space compared to other BO methods and search heuristics.
References
[1] F. Bach. Exploring large feature spaces with hierarchical multiple kernel learning. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008
[2] G. W. Benton, W. J. Maddox, J. P. Salkey, J. Albinati, and A. G. Wilson. Function-Space Distributions over Kernels. Curran Associates Inc., Red Hook, NY, USA, 2019.
[3] D. Duvenaud, J. Lloyd, R. Grosse, J. Tenenbaum, and G. Zoubin. Structure discovery in non- parametric regression through compositional kernel search. In S. Dasgupta and D. McAllester. editors, Proceedings of the 30th International Conference on Machine Learning. volume 28 of Proceedings of Machine Learning Research, pages 1166-1174, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[4] S.-S. Jin. Compositional kernel learning using tree-based genetic programming for gaussian process regression. Struct. Multidiscip. Optim., 62(3): sep 2020.
[5] K. Kandasamy, W. Neiswanger, J. Schneider, B. P6czos, and E. P. Xing. Neural architecture search with bayesian optimisation and optimal transport. In Proceedings of the 32nd Interna- tional Conference on Neural Information Processing Systems, NIPS'18, 18, page 2020-2029, Red Hook, NY, USA, 2018. Curran Associates Inc.
[6] K. Korovina, S. Xu, K. Kandasamy, W. Neiswanger, B. Poczos, J. Schneider, and E. Xing. Chembo: Bayesian optimization of small organic molecules with synthesizable recommenda- tions. In Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 3393-3403. PMLR, 26-28 Aug 2020.
[7] R. Lloyd, D. Duvenaud, R. Grosse, J. B. Tenenbaum, and Z. Ghahramani. Automatic con- struction and natural-language description of nonparametric regression models. In Proceedings of the Twenty- Eighth AAAI Conference on Artificial Intelligence, 14, page 1242-1250. AAAI Press, 2014.
[8] D. C. MacKay. Bayesian interpolation. Neural Compulation, 4(3):415-4477 05 1992.
[9] G. Malkomes, Schaff, and Garnett. Bayesian optimization for automated model selection. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[10] A. G. d. G. Matthews, M. van der Wilk, T. Nickson, K. Fujii, A. Boukouvalas, P. León- VillaVill,grá Z. Ghahramani, and J. Hensman. GPflow: A Gaussian process library using TensorFlow. Journal of Machine Learning Research, 18(40):1-6, apr 2017.
[11] V. Nguyen, T. Le, M. Yamada, and M. A. Osborne. Optimal transport kernels for sequential and parallel neural architecture search. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages
PMLR, 18-24 Jul 2021.
[12] W. Ober, C. Rasmussen, and M. van der Wilk. The promises and pitfalls of deep kernel learning. In C. de Campos and M. H. Maathuis, editors, Proceedings of the Thirty-Sevents Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 2206-1216 PMLR, 27-30 Jul 2021.
10

---

[13] Rasmussen and Williams. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, USA, Jan. 2006.
[14] C. Rasmussen and Z. Ghahramani. Occam's 's razor. In T. Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000.
[15] J. Schreiter, D. exploration for active learning with gaussian processes. In Proceedings of the 2015th European Conference on Machine Learning and Knowledge Discovery in Databases Volume Part III, ECMLPKDD'1S 15, page 133-149, Gewerbestrasse 11 CH-6330, Cham (ZG), CHE, 2015. Springer.
Tuong, M. Eberts, B. Bischoff, H. Markert, and M. Toussaint. Safe
[16] J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, 2004.
[17] J. Snoek, H. Larochelle, and R. P. Adams. Practical bayesian optimization of machine learning algorithms. In F. Pereira, C. Burges, Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
[18] S. Sun, G. Zhang. C. Wang, W. Zeng, J. Li, and R. Grosse. Differentiable compositional kernel learning for Gaussian processes. In J. Dy and A. Krause, editors. Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4828 4837. PMLR, 10-15 Jul 2018.
[19] C. Villani. Optimal Transport: Old and New. Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2008
[20] A. Wilson and R. Adams. Gaussian process kernels for pattern discovery and extrapolation. In Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1067-1075 Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[21] A. G. Wilson, E. Gilboa, A. Nehorai, and J. P. Cunningham. Fast kernel learning for multidi- mensional pattern extrapolation. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[22] A. G. Wilson, Hu, R. Salakhutdinov, and E. P. Xing. Deep kernel learning. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 370-378, Cadiz, Spain, 2016. PMLR.
[23] C. Zimmer, M. Meister, and D. Nguyen-Tuone Safe active learning for time-series modeling with gaussian processes. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the pa- per's contributions and scope? [Yes] We clarify main contributions in the end of the introduction and provide the mentioned method Section 3 and the mentioned experimental results in Section 4
(b) Did you describe the limitations of your work? [Yes] In Section
(c) Did you discuss any potential negative societal impacts of your work? [No] No obvious direct negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Given in Definition [ Proposition and in Section C
(b) Did you include complete proofs of all theoretical results? [Yes] In Section C
3. If you experiments...

---

(a) Did you include the code. data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as a URL)? [N/A] We will provide code as soon as clearance of code is completed.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Either in Section 4 or Appendix
(c) Did you report error bars (c.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We report either standard deviations or quartiles.
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We analyze the search performance over CPU-time.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Yes cite the GPflow related work, which the main python framework we are using.
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Code is available at wittts:://// //tpss://ii thub cwm hwom//bbcchrseerccbbooeese
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "tFipHSkQC-Q", "reviewer": "Reviewer_i6Db", "summary": "This paper considers the problem of selecting the kernel of a Gaussian process (i.e., the model selection problem), which is one of the hyperparameter optimization problems for Bayesian optimization. \nThe authors considered that a kernel function is consisted of a set of \"basis\" kernels in the space of kernel functions and a number of operations, following the previous (kernel grammer) work of Duvenaud et al.\nA kernel grammar can be mapped to a tree that represents it.\nFor the discrete probability distribution that is a summary of this representation tree, we can define the Wasserstein distance with the indicator function as the cost function. In this case, the Wasserstein distance coincides with the total variation distance. This distance can be used to define a meta kernel function (kernel-kernel) that measures the similarity of two kernel functions on the space of kernel functions. The proposed method uses a Gaussian process with this kernel-kernel as the covariance function, and Bayesian optimization is used to select the kernel model. \nIn the experiment, the authors compared and evaluated the model selection performance of the proposed method with existing methods on four types of benchmark data including time series.", "questions": "- The proposed method seems to cover only model selection for the kernel function itself, but if we want to further perform hyperparameter selection for the kernel, can we combine it with conventional methods (e.g., marginal likelihood maximization)?\n\n- Is it possible to approximate the Wasserstein distance and use the proposed method when a general distance that is not an indicator function is used as the base distance?\n\n- Are the hyperparameters alpha_1, 2, and 3 in distance (3) optimized at the same time as the other kernel-kernel hyperparameters? How does this affect performance of the proposed method?", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strength\n\n- Traditionally, model selection for the kernel function itself has been done manually depending on the problem or by multiple kernel modeling for several candidate kernels. On the other hand, this study performs Bayesian optimization of combinations of kernel components based on the similarity defined between the kernel functions. In this approach, only the base kernel and operation rules need to be prepared, and there is no need to prepare multiple candidate kernel functions as in multiple kernel learning. Bayesian optimization for model selection allows for the \"construction\" of good kernel functions without human intervention.\n\n- The mapping of kernels to representation trees allows the problem of finding the optimal kernel to be viewed as a problem of finding the optimal tree structure.　\n\n- The proposed method has higher scalability than existing methods (Malkomes et al.) that use Bayesian optimization based on Hellinger distance for the same problem. This is because the Hellinger distance-based method requires an integral calculation on the kernel parameters for kernel-kernel evaluation.\n\nweakness\n\n- In this paper, an indicator function is used for the base distance to describe the Wasserstein distance as a closed-form solution (total variation distance). However, this seems to be a discrete and extreme similarity evaluation that only looks at whether or not the elements that make up the two representation trees match.\n\n- The distance for kernel-kernel (3) itself contains the hyperparameters alpha_1, 2, and 3, so it seems that the hyperparameter tuning for kernel-kernel are going to be harder than for ordinary BO when performing BO on the kernel space.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_B8mX1iUmD-", "reviewer": "Reviewer_Cka7", "summary": "To find the best kernel for a GP, rather than conducting BO with a value function given by a GP comparison in function space, this paper develops a novel way to compare two kernels, based on optimal transport between feature representations of the tree representation of the kernel.", "questions": "Given that your method eschews functional distance like in [2], are we sure that it would do well in cases where the true kernel is known? I suppose this can be evaluated on synthetic data. Or is there some weakness here, hidden in the simplicity (both theoretical and computational) of the pairwise evaluation?\nHow can the OT distance be interpreted? Is there something of interest happening with the multiplicative factors in the total distance eq3 once they get learnt?", "limitations": "Yes, though section 5 could be made more explicit. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper comes up with a good idea for kernel search, inspired I believe from references [10,4] which conduct NAS by BO using OT over the computational graph. Thus the central idea is at hand (structure search by BO using OT over structure distance), though not immediate.\nThe paper is clearly written. Many details are left to the appendix, but this still seems adequate.\nThe experimental validation is adequate.\nThe formalization of the OT metric in sec 3 is justified thoroughly.\nFig 1 is not legible when printed on A4 format.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UmmfCtnMBv9", "reviewer": "Reviewer_PF7c", "summary": "The paper proposes a method for kernel selection (search) wherein kernels are built from base kernels using fundamental operations (eg addition, multiplication etc).  Kernels so constructed are represented as trees, allowing their similarity to be evaluated based on their grammar (tree structure and assumptions regarding basic operations, eg their commutativity or otherwise) rather than e.g. distance between GPs.", "questions": "1. When constructing base kernels, do you \"fix\" hyperparameters like length-scales (for the SE kernels) for each base, or do you tune such parameters using e.g. max-likelihood?  Further:\n1a. if the former, do you attempt to give some flexibility by e.g. having multiple SE kernels with different lengthscales as distinct base kernels?\n1b. if the latter, how do you measure the similarity between two examples of the \"same\" base kernels with different lengthscales?\n\n2. Similarly, when doing operations like ADD, do you considered allowing for weighted sums (a1.K1(x,y) + a2.K2(x,y), where a1,a2 are scaling factors) rather than simple sums (K1(x,y)+K2(x,y))?  This would allow more flexibility, but I'm not sure how well it would fit in your kernel similarity calculations.\n\n3. As a rough order-of-magnitude, how complicated do SOT kernels become?  I would imagine this would depend on how long you let the BO run and what heuristic limits you place on the your evolutionary algorithm, but I'm curious how complex the tree becomes.\n\n4. Does using an evolutionary algorithm rather than global optimisation when maximising your BO acquisition function adversely affect convergence guarantees that usually apply to BO?", "limitations": "As noted previously, more experimental comparisons (more datasets) and at least one non-parametric result (e.g. hyperkernels) would improve this paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The construction of the kernel-kernel is thorough and I am persuaded that, while heuristic, this is a good alternative to measuring the difference between GPs or kernels using e.g. L2-norm.  Further, the overall approach of building kernels from base kernels reflects the human (parametric) approach to the problem well.\n\nMy main problems with this paper are:\n\n1. Comparison with non-parametric approaches to kernel selection (such as hyperkernels) is missing.\n2. It could be argued that this approach is incremental, simply substituting one kernel-kernel [2] with another that has the advantage of being more readily evaluated.\n3. Experimental results are limited.\n\nI would be more comfortable in overlooking the second point if there was more experimental results (ie. more than 4 datasets evaluated), but it is difficult to draw strong conclusions from limited results, even though the improvements shown appear strong.  However I think it is important to include comparisons with non-parametric approaches for completeness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IqxXk-908uO", "reviewer": "Reviewer_iFD5", "summary": "This paper proposes a new way to select a kernel (covariance function) for Gaussian processes (GP). The authors borrow ideas from NN architecture search to propose a so called symbolical-optimal-transport (SOT) kernel over the architecture of the kernel. In contrast to prior work, they do not compare to different kernels in function space. Instead they compare the symbolic architecture of the kernel directly. They do this by capturing the structure of the kernel in a tree and then compare these trees by comparing them by some kind of optimal-transport metric. \nThis SOT kernel is then used for Bayesian optimization for model selection. This is fairly standard. A comparison to a hellinger kernel-kernel is given. The SOT kernel is computational advantageous because no integrals have to be computed.\nThe authors then demonstrate in a series of four experiments that the SOT kernel outperforms Greedy, Hellinger, Tree-GEP and SOT. This is true for both the training and the test set.", "questions": "--- How does this work together with hyper parameter tuning of GP? Can I first run your method and then tune the hyper parameters as usual?\n--- Can this be used to pick the kernel for RKHS regression or SVMs?\n--- On page 4: Are Rule 3 and Rule 2 ('exchanging a base kernel with another kernel') really necessary? It seems to that they assuming the other rules would be enough...\n\nMinor points:\n--- The plural of GP is GPs, not GP's.\n--- Please define g before it's used on page 3. ", "limitations": "The limitations are transparent and well-addressed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: While this idea has been executed for Neural Networks before, the presented idea is novel for Gaussian process model selection. The experiments are very encouraging and the idea seems natural. The computational advantages of this approach (over the known function space view) are well-explained. While the idea is simple, I do not consider this a weakness. The topic is very significant because model selection is extremely important for GPs. The quality of writing is good, it is quite easy to understand.\n\nWeakness: There is almost no theory in the paper, except for the trivial Proposition 1. I think the focus on \"Optimal Transport\" is misleading because the special case of the OT distance that is used is quite trivial, and looks more like an l1 norm of the weights. Nowhere else is optimal transport used. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept