Kernel Identification Through Transformers
Authors: Fergus Simpson
Ian Davies
Vidhi Lalchand
Alessandro Vullo
Nicolas Durrande
Carl Edward Rasmussen
Venue: NeurIPS 2021 Poster (2021)
Keywords: Gaussian Processes
kernel design
Bayesian model selection
deep learning

Abstract: Kernel selection plays a central role in determining the performance of Gaussian Process (GP) models, as the chosen kernel determines both the inductive biases and prior support of functions under the GP prior. This work addresses the challenge of constructing custom kernel functions for high-dimensional GP regression models. Drawing inspiration from recent progress in deep learning, we introduce a novel approach named KITT: Kernel Identification Through Transformers. KITT exploits a transformer-based architecture to generate kernel recommendations in under 0.1 seconds, which is several orders of magnitude faster than conventional kernel search algorithms. We train our model using synthetic data generated from priors over a vocabulary of known kernels. By exploiting the nature of the self-attention mechanism, KITT is able to process datasets with inputs of arbitrary dimension. We demonstrate that kernels chosen by KITT yield strong performance over a diverse collection of regression benchmarks. 

Kernel Identification Through Transformers
Fergus Simpson Secondmind Cambridge, UK fergus@secondmind..a
Ian Davies InstaDeep London, UK
Vidhi Lalchand University of Cambridge Cambridge, UK
Alessandro Vullo Secondmind Cambridge, UK
Nicolas Durrande Secondmind Cambridge, UK
Carl Rasmussen University of Cambridge Cambridge, UK
Abstract
Kernel selection plays central role in determining the performance of Gaussian Process (GP) models, the chosen kernel determines both the inductive biases and prior support of functions under the GP prior. This work addresses the chal- lenge of constructing custom kernel functions for high-dimensional GP regression models. Drawing inspiration from recent progress in deep learning, we introduce a novel approach named KITT: Kernel Identification Through Transformers. KITT exploits a transformer bassd urchitecture to generate kernel recommendations in under 0.1 seconds, which is several orders of magnitude faster than conventional kernel search algorithms. We train our model using synthetic data generated from priors over a vocabulary of known kernels. By exploiting the nature of the self- attention mechanism, KITT is able to process datasets with inputs of arbitrary di- mension. We demonstrate that kernels chosen by KITT yield strong performance over diverse collection of regression benchmarks.
Introduction
In recent years deep parametric models have become a prominent class of model for supervised learning and have delivered impressive empirical performance over wide range of tasks. An im- portant limitation, however, is that in their conventional form deep models do not provide prediction uncertainty. While their Bayesian counterparts try achieve this, they require significant modi- fications to the training procedure and are computationally expensive. Uncertainty quantification in deep models widely considered be an open problem, the large array of research proposing alternative Bayesian neural networks underscores this [10, 12, 17).
On the other hand, kernel driven methods within the Bayesian framework like Gaussian processes (GPs) account for prediction uncertainty by design. While GPs provide a flexible framework for inferring distributions over functions, the inductive biases are controlled by the kernel function A well chosen kernel will typically yield dramatically better performance than poorly chosen one.
How should we learn expressive kernels for high- digeesiimnnio tasks? This has frequently been high- lighted as a central question for the continued relevance of GP methods [11]. This work uses rep- resentations generated by deep neural network to identify suitably expressive kernels for high- dimensional GP regression tasks. Kernel recommendation is performed by decoder with access to large vocabulary of primitive kemnels and products of primitive kernels. The decoder maps
WWrk undertaken while Secondmind
also called covariance function or covariance kernel
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

encoded representation of dataset to kernel that can be used to model it. The representation is attained by encoding the dataset, treated as a sequence of (input, output) pairs D { 9/ / utilising the permutation-equivariann nature of self-attention networks.
By training KITT with sufficiently rich vocabulary of kernels, it can predict suitable kernels for diverse array of real datasets. This work presents the following novel contributions:
Inspired by the successes of image captioning networks, develop novel framework named KITT for amortised kernel search. KITT takes raw datasets for predictive modelling input and proposes kernels composed from large vocabulary of kernel functions.
KITT's architecture introduces two novel features: is entirely agnostic to the length and dimensionality of the data we wish to perform inference on, and it offers double permuta- tion invariance (this ensures its outputs are invariant to permutations in either input dimen- sions or data points).
We show that KITT can deliver kernel predictions in under 0.1 seconds.
We introduce a novel variant of the linear kernel which forms key component of KITT's vocabulary.
We demonstrate that the kernels identified by KITT offer strong performance against other baselines which deal with kernel engineering in the context of GPs.
2 Background
This section offers a brief review of the two topics which are central this work, namely Gaussian Processes and Transformers.
Gaussian Processes. GPs offer highly versatile framework for predictive modelling [23] with generalisation properties controlled by a kernel function parameterised by hyperparameters. The functional form of the kernel governs the global attributes of the supported functions, such as smoothness and periodicity. However, a suitable kernel function is unknown a priori and the choice of the kernel function is fundamental model selection problem. Once kernel has been chosen, training conventionally proceeds by learning a point estimate of the hyperparameters that maximise the GP log marginal likelihood l argmax, log p(y 0). The marginal likelihood available in plosed-fomm for models with Gaussian likelihoods. Below we briefly summarise the standard GP framework.
GPs are distributions over functions from which one can sample realised function values for given inputs. Concretely, for observations X (ax; [ and positive definite kernel function ko(:s)) )). with hyperparameters 0, f()) ~ GP(O,, kg). Typically, we observe noisy realisations of the latent function which are corrupted with Gaussian noise, 34 = f(x,) + E,, 6, (O, o.), and infer the kernel hyperparameters through maximising the likelihood of the model.
The GP marginal likelihood objective, p(y)0), is obtained by marginalising the likelihood ylf ((f, I) over the prior f| 0 ~ NO0, Ko),
(0) / plylf)p(f10) df NO Kg + +
(1)
where f = fX) denotes vector of realised function values, and K₂ denotes the N N covariance matrix corresponding to evaluations of the covariance function at the N training inputs, (Ko)sj =
ko( 2.;).
A long standing question is how best to select an appropriate kernel function for a given task. One approach is to search over discrete space of kernels, defined by combining selection of primitive kernels with a predefined grammar [2. 6]. Typically, a greedy search is performed to identify the kernel offering the best representation of the data. Ideally, the quality of the kernel is quantified by the Bayesian model evidence, which can be computed by marginalising the marginal likelihood over the hyperparameters. However, since the integral is challenging compute, each kernel's 's suitability is instead usually determined via a proxy for the model evidence, such as the Bayesian Information Criterion (BIC) [25].
For principled, Bayesian approach to kernel design, instead of selecting a single kernel, one ought to consider multiple candidates. In other words, it is desirable to marginalise over the space of

---

kernels, not only over the space of functions for single kernel.
p(ylD) - D
(2)
This yields rich posterior distribution comprised of a mixture of Gaussians. A conventional ker- nel search makes three key approximations. First, that contributions from all but the single chosen kernel K* can be neglected; second, that contributions from all but the maximum likelihood hy- perparameters 0" can be neglected; third, that the proxy for the model evidence is reliable one. There are several regimes, for example where the data is sparse or noisy, where all three of these assumptions do not hold. We shall aim to improve upon all three of these issues.
Transformers Transformers are a form of deep neural network which rely upon the attention mechanism [32] to capture global context. While they were originally proposed to tackle machine translation tasks [29], they have rapidly attained state-of-the-at performance in a number of other areas of machine learning [14, 21, 22]. Of particular relevance to this work, they have been success- fully applied to image captioning [5, 33], a task which involves summarising the key characteristics of rich data in grammatical form. This has a striking parallel to the challenge of selecting an ap- propriate kernel, especially since a form of grammar can be used to construct a broad selection of GP kernels.
The self-attention mechanism of transformers naturally lends itself to the permutation-invann ting. as demonstrated by Zaheer et al. [34]. This invariance to permutations in the ordering of inputs has been exploited create Set Transformers, introduced in Lee al. [18]. The AHGP model [19] uses variant of the Set Transformer to infer the hyperparametens of the spectral mixture kernel.
set-
Like all deep networks, transformers thrive when presented with abundance of training data. Fortunately, for the task we have at hand, the training set is unlimited in size as we may sample training data from GPs with known kernels.
As stressed by Liu et al. [19], selecting an architecture which reflects the appropriate invariances is vital. The AHGP model is designed to be invariant to permutations in the ordering of the datapoints. We go one step further, and introduce a model which doubly permutation invariant: its output is also invariant permutations in the input dimension (whereas AHGP is equivariant to the input dimensions).
KITT
In this section describe and motivate KITT, network which takes inputs set of datapoints {x,, y,), and outputs a kernel recommendation in the form a 'caption', by utilising a kernel grammar. The code is available at https://githi:: http:::://gghtbb...ooggggggapppp
wtt.c/oo///rree hoon//rggsimson/kittt.
Kernel Grammar and Vocabulary: Throughout the GP literature, the most commonly used ker- nels belong to limited set of primitive functions. In this work, utilise eight primitive kemels: the squared exponential; periodic; white noise; three variants of the Matern kernel (2. 3 2: 2); the
cosine kernel, and our novel variant of the linear kernel. This list comprises six stationary isotropic kernels, one stationary anisotropic kernel (cosine), and one non-stationary kernel (linear).
From this small set of primitive kernels, we wish construct a larger array of more expressive ker- nels. This can be achieved by leveraging the closure properties of kernel functions [27]. Permitted operations include addition, multiplication, convolution, composition, and affine transformations. For simplicity, this work shall only consider two operators: addition and multiplication.
Whilst this grammar of addition and multiplication appears superficially simple. it has some id- iosyncrasies that would make it challenging for a network learn. For example, if a network is unaware that multiplication is commutative, it would have to learn to recognise k1 * k2 and k k separately, for all combinations of i and lz. It would also need learn that multiplying a noise kernel with another stationary kernel yields another noise kernel. Encoding this information priori greatly facilitates the learning process. To achieve this, we enlarge the vocabulary by defining prod- uct kernels as single 'words', rather than incorporating products as part of the grammar (the full list provided in the Appendix). This allows us to exclude redundant combinations from consideration. For example, only single token is used to represent both k2 k1 and K1 f2. While this enlarges
3

---

Encoder
Decoder
Decoder
Decoder
Decoder
Shape (0
Shapertd
Maaam 1/2
Matern X22
RBF x Maler 122
<END>
PREDICTION: Nater 1/2 Natem a12 RBF x Matem 1/2
Figure 1: The architecture for KITT, partly motivated by image captioning networks, which also act to transform rich dataset into grammatical expression of its contents.
the vocabulary, we can be confident that the network will be able cope, since there will still be far fewer 'words' than can be found in natural language tasks where transformers are known to excel. We use products of two primitive kernels as part of the base vocabulary and found that incorporating higher order products do not significantly change performance.
A further advantage of defining product kernels the vocabulary level is that we need longer include operators inside the vocabulary. The multiplication is already baked into the expanded set of kernels, while the addition takes place implicitly, much like the white space between words in a natural language task. This precludes the construction of nested structures of operators, which would allow for even richer kernels. However, since this work represents a first attempt at perform- ing a kernel search with a neural network, we choose to keep the captioning task to be a relatively simple one, and leave more complex grammatical compositions as an opportunity for future explo- ration. Even with this relatively simple grammar, if we permit a caption of four words from a base vocabulary of 32, we are effectively searching across a space of around 36 000 kernels.
While the priors we impose upon the hyperparameters will have some impact during op- timisation, it is their influence upon the generation of KITT's training data that is of central im- portance to this work. Random samples are drawn from the hyperparameter priors p(0) and input locations x ~ (-2.5, 2.5) before each random sample of y generated. The variances and length- scale parameters of all kernels (including product kernels) are assigned lognormal priors, such that log 8 /O, 1).
Priors.
The cosine kernel is unique among KITT's vocabulary, in that it is inherently anisotropic, which can be important if a preferred direction exists within the data. Samples drawn from the kernel manifest as plane waves which propagate along a characteristic direction of the kernel. If we were to impose a lognormal prior (or any other positively constrained prior) on its lengthscales, this would restrict the direction of the kernel to a small fraction 21-D of its permitted parameter space. We therefore adopt a different approach in this case, assigning é Cauchy 5) for the lengthscales.
Training. KITT is trained entirely on synthetic data. Each training example is generated from randomly selected kernel, with a randomly drawn set of hyperparameters. This data could be generated on the fly, but since this can be computationally expensive process, we generated training set in advance which comprised of 200, 000 labeled examples. While the model can be constructed for an arbitrary number of inputs points and input dimensions, during training we restrict ourselves to the case of 4 input dimensions and 64 input points per sample. The loss function corresponds to - log (() the negative log probability the network assigned to the correct term in the vocabulary. The Adam optimiser was used with an initial learning rate of 10 and decay schedule with a decay rate of 0.1 every 50, 000 iterations. Due to the relatively noisy nature of the classification task, a relatively large batch size 128 was found to be beneficial. The vocabulary included product kernels of at most two terms in addition to the primitive kernels, yielding a final vocabulary of size 34.
Heteroscedastic noise. When taking the product between the white noise and any stationary ker- nel, we recover another noise kernel. These redundant expressions are omitted from KITT's vocab- ulary. However, the result of the product between the noise kernel and the (non-stationary) linear kernel merits special attention. The linear kernel is defined as
K (L, x²) a²(x c)(2' c),
(3)

---

30030
241
NUPD
0.2
0.0
Vitte
11.30
ciaan
Power
-9.9
-2.
-0.
Constant Noise
Linear Noise
Shifted Linear Noise (this work)
Figure 2: Negative log likelihood values for three different noise models when used alongside the RBF kernel. All of the datasets clearly benefit from the modelling of heterescedastic noise, while three benefit from the additional freedom offered by the shift parameter.
where o² denotes the vector variances for the linear kernel, and c represents the shift parameter. Unlike the other primitive kernels, the linear kernel possesses independent variance terms for each input dimension.
The product between the linear kernel and the noise kernel generates a form of noise whose variance changes linearly with respect the inputs. This presents opportunity model heteroscedastic noise, within the conventionally homoscedastic domain of GP regression models. When modelling real world tasks, this potentially offers a major advantage, since the noise variance often changes across the input space.
Note that if we simply set O, as is often assumed when working with the linear kemel,, then the linearly varying noise term is extremely limited. The noise variance could only ever increase as we move away from the origin. As with the cosine kernel, this reduces us small fraction (2' D) of the viable parameter space. In order to lift this restriction, we introduce a 'shift' vector in the linear kernel, such that the origin is free to what an appropriate prior for this shift vector would be. We consider it equally likely that the noise amplitude inereases decreases with æ. To reflect this belief, and accounting for our normalised inputs, we seek a prior of the form do² /dx ~ (O, 0.1). For large displacements, we note that the shift parameter can be approximately expressed as the ratio of two normally distributed variables: the gradient of the noise and the noise amplitude at the origin. This observation suggests a suitable prior on is given by the Cauchy distribution. We adopt e Cauchy (0, 5) throughout.
along each input dimension. This naturally leads to ask
Model Architecture. The kernel's likelihood ()) is invariant to permutations of the ordering of the datapoints, and to permutations of the ordering of the input dimensions. KITT's output of kernel recommendations should therefore exhibit these two important properties. Zaheer et al. [34] demonstrated that function f(X) which invariant to permutations in the elements of X can be expressed in the form p PE, (xx;))). where and & are differentiable functions. This was exploited by Lee al. [18] in constructing the Set Transformer. The scenario we encounter is slightly more complex in that there is two-ticred hierarchy of invariances. We seek a function over the training set D, which can be expressed as collection of high dimensional data vectors D,: f((2)) Here the function (D,, must be invariant over the permutations of the different input
dimensions. This can therefore be decomposed in a similar manner, <(D.) = p [Σ, (DD)))
Combining these two equations leaves final expression of the form
S((D)) = EV' (E
(4)
where and j can be either dimensions or datapoints. This can be interpreted as: encode over j: pool over j; encode over i; pool over i; decode. This formalism sets the foundations for our choice of architecture, as shown in Figure 1. At a lower level, KITT is comprised of the following components whose acronyms we define here:
5

---

rFF: Row-wise feed-forward layer with ReLU
MP: Mean pooling function
SAB: Set Attention Block [18]
LayerNorm: Layer Normalisation [1]
Multihead:
Attention Mechanism [29].
Encoder: The encoder has architecture of transformer with self attention blocks. Our goal is to encode datasets Dj { Y/ ;==1 of shape N (D+1) where æ, R and ye RR need to incorporate both invariance to ordering of the data points (row-wise shuffle) and equivariance to re-ordering of the dimensions (column-wist shuffle). In order to achieve this, encoder has two sub-components responsible for encoding along the sequence and dimension axcs respectively:
1. SEQLENC: ENC: IR (D(+1) + DxE A sequence encoding component which acts on input datasets and outputs dimension-level representations G = {gu)am Ia E R where E is the embedding dimension. The sequence encoder forward pass formulated as:
SEQLENC(D) = MP(SAB x6 (IFF(D))
Where mean pooling is applied over the sequence. Our implementation of the SAB component [18] differs slightly from that of Lee et al. [18]. SAB(Z) LayerNorm(C Z), where have defined C' = (rooouu(FFF[Muutiieee
(Z, Z, 2)1) [29].
ENLENC: Px PxF A dimension encoding component: DIM_ENC which acts on dimension level encodings to generate final representations (hald=1 of dimension E. The di- mension encoder forward pass formulated as:
DIM_ENC(G) = xFF X
The encoder forward pass entails passing each input data the sequence encoding component followed by the dimension encoding component.
ENCODER DIMLENC(SEE
Let Sy denote the set of all permutations of the row-wise indices (1,2...., N and D denote an input dataset with the ordering of indices given by π Sy The sequence encoding component SEQ ENC is invariant to a permutation of the indices within a dataset D.
Proposition 1:
SEQ ENC(D) SEQ ENC(D=) SN
Proposition 2: Let On denote the set of all permutations of the column-wise indices (dimensions) (1.2..... D} and DD denote an inpul dataset with the ordering of indices given by an arbitrury components are
E Qn. The sequence encoder JEC and dimension encoder equivariant to a permutation of the dimensions within a dataset 'D
ENCODER (D,)
V QD
We include proofs for propositions and the supplementary.
Decoder: KITT's decoder iteratively builds a caption from the encoded dataset representations, generated by the encoder described above, and a prompt which consists of the kernel expression thus far (see Fig. 1). Our decoder closely resembles the one proposed in Vaswani et al. [29] except that we remove the positional encodings and adjust the number of layers. The decoder uses self- attention blocks to first attend to the prompt and then to attend to the representations from the encoder using the processed dataset representations as query values. These two applications of attention are alternated in several layers until new component kernel is proposed from a distribution generated from the final representations. We note that this component kernel proposition invariant to the ordering of the dataset representations and thus to shuffling of the dimensions of the original dataset. Hence, the model is fully invariant. The end-to-end process is depicted in Figure and a detailed schematic of the decoder is included in the supplementary material.
Scalability. Training of KITT is only performed once. Once training of the network is completed, all inference procedures require only single forward pass through the network. As a result, instead of the O (3) cost commonly associated with explicit marginal likelihood evaluations, forward pass through KITT is O(DN22 + D²). Due to effective parallelisation of the attention mechanism on GPUs, noted by Vaswani et al. [29], and the modest size of the KITT network, we experience near constant wall clock time in practice.
cost of

---

10-
1.0
Test Accuracy
10
Snonder
Wall clock soc.
04-
0.4
lap
10:4
Tell,72
80
0.
250
500
750
1000
1250
1500
10
10 10*
Number of tast points
Number of tast dimensions
Figure 3: Left and centre: Classification performance for random samples drawn from primitive kernels across range of test sizes and dimensionality. The vertical dashed lines denote the condi- tions under which the network was trained. Right: The time taken to predict kernel for each of the UCI datasets, While KITT's overhead remains approximately constant, the tree search becomes impractical for larger inputs.
Inference. Given some new dataset D, inference proceeds below, closely mimicking the proce- dure used to generate caption for images. Note that the best caption not necessarily constructed by choosing the best kernel each step of the decoder.
1. Pass the data D into the encoder; pass the resulting encodings and an (initially empty) kernel expression & into the decoder.
2. Retrieve the output probabilities, selecting the kernel & with the highest probability and append the chosen kernel (or operator) to our full kernel expression E.
3. Repeat steps & 2 until either the < STOP> token is selected the maximum caption length is reached.
4. Repeat steps 1-3 several times but now select kernels stochastically, weighted by their probability, to construct set of high-ranking kernel expressions.
5. For each of the top three candidate kernels expressions, ranked by their total probability assigned by the network, we optimise the associated hyperparameters with BFGS [9].
6. Combine the posterior distribution of the forecasts based either the overall probability assigned by the network, or another proxy for the model evidence such as the Bayesian Information Criterion.
In summary, we select a kernel based upon the output of the pretrained KITT network, before opti- mising the hyperparameters in conventional manner.
4 Experimental Results
In this section we explore KITT's ability to predict kernels for synthetic data and standard regression benchmarks. We present four baselines assess the performance of KITT regression data. AHGP [19] is another deep network designed to assist GP inference, as outlined in 2. The neural kernel network [28], offers differentiable form of kernel composition. We also include a greedy kernel search algorithm based upon the Automatic Statistician procedure outlined in Duvenaud et al. [7]. These three algorithms span a wide range computational overheads, with AHGP being the fastest and the kernel search being the slowest. As a more familiar reference point, we also include the RBF-ARD baseline, which uses the same priors described previously.
Ground Truth Recovery. Identifying primitive structure is an important building block in being able produce sensible kernel recommendations for real-data in high dimensions. Capturing this structure is the aim of our encoder. In order to test the ability of the encoder to capture primitive structure, we form a classification transformer by taking the KITT encoder and appending dense layer followed by a softmax activation. The resulting model is trained on datasets of fixed size and dimensionality to predict kernels for synthetic datasets drawn from GPs with known kernels. We demonstrate test performance in terms of accuracy for varying test input sizes and dimensions. We draw 300 random samples from a selection of primitive kernels, for varying combinations of test size and dimensionality. The results shown in Figure 3 demonstrate that the classifier is able to reliably
7

---

Crangy
MUPD
0.0
Tudk
Powa
1.0
-6-
0.5
-0.5
10
2.8-
-0.-
2.6
RBF
Tree Search
NKN
AHGP
KITT (E+D)
KITT (E)
Figure 4: Negative predictive log likelihood values on UCI regression tasks for variety of kernel selection methods. KITT remains competitive with the most computationally intensive approach, the tree search, while offering the advantage of being several orders of magnitude faster. The 'E' denotes sole use of the encoder followed by a classification layer to select kernels, while 'E+D' generates captions with the decoder.
generalise its structure detection capabilities to higher dimensional tasks, which were unseen during training the network. As is expected, a moderately sized dataset least ~ 200 points needed to achieve reasonable level of prediction accuracy, and this continues to improve with increasingly large datasets.
UCI Regression. We evaluate KITT eight real-world UCI regression tasks spanning range of input sizes and input dimensions (from 4 to 14). We adopt the same benchmarking methodology as Liu al. [19]. which includes a 90/10 train/test split, and subsampling 2,000 datapoints for those cases where the dataset exceeds this number. For each dataset, we predict a kernel caption with maximum expression length of three terms. The caption is either constructed sequentially by the decoder, or in the case of the classifier, by summing the three highest scoring kernels.
The resulting NLPD values from KITT, and three other approaches kernel design, shown in Figure 4. Uncertainties are estimated from repeated experiments with ten different splits. KITT consistently outperforms the other transformer- slower tree search method (see Figure 3). We also perform an ablation study, details of which can be found in the supplementary material, demonstrating that KITT outperforms random selection from its vocabulary.
model, AHGP, and is competitive with the far
We note that the AHGP performance is weaker than that of the RBF. This is significant because the RBF a subset of the Spectral Mixture Product model, equivalent to a single component set to zero mean frequency. This suggests that the AHGP network perhaps focused on learning how identify the lengthscale of the primary component of the multi-component Spectral Mixture Product.
For deeper understanding of KITT's performance, Figure 5 compares the network's 's output against realised test performance on the Yacht dataset, across all 34 kernel classes. The three kernels KITT assigned high probability to, namely Linear x RBF, Linear x Matern32 and Linear x Matern52, correspond to the three strongest test performances.
Computational overhead. One of the most compelling features of KITT the speed which inference can be performed. Identifying a suitable kemel for previously unseen dataset only entails a single forward pass through the encoder, and small number through the decoder, each of which requires around two hundredths of a second. Furthermore, as illustrated in the right hand panel of Figure 3, the time-cost of prediction is robust increasing data-set sizes and dimensionality.
The KITT network was trained on Tesla V100 GPU for approximately eight hours, with Adam [16). This procedure occurred only once, and does not need to be repeated when performing infer- ence. To generate kernel prediction requires small fraction of a second, and is largely insensitive
2 recurring misconception in the literature is that predictive errors on Naval dataset are so small that they may be neglected, and these are sometimes listed as "0.00". We stress that they are small only because of the small variance of the raw data. This should have no bearing on its significance alongside the other datasets, and the RMSE should not be rounded down to

---

2.00
1.75
1,50
1.2
1,00
6.75
6550
625
0,0
5000
1.035
0.000
00mh
0:100
3.755
6.155
KITT ansigned probebiity
Figure 5: A comparison of KITT's kernel predictions against their test performance, the Yacht dataset. Each dot represents one of the 34 kernels in KITT's vocabulary. KITT successfully identifies the three top performing kernels, and assigns low probability to the 31 alternative options.
to the size of the input data, as seen in the right hand panel of Figure 3. Once kernel has been recommended, training typically required further ten seconds. It is possible this step could also be greatly accelerated in future, if KITT were used in tandem with hyperparameter optimisation network, similar to AHGP [19].
5
Related work
In this section review approaches that either directly, through kernel construction, indirectly target the issue of model selection in GPs.
Amortised Hyperparmeter Learning (AHGP): Liu et al. [19] also use self-attention based trans- formers, but with the goal amortising hyperparameter learning. They train on input-output regres- sion based datasets estimate the final set of GP hyperparameters that would otherwise be learnt as a result of maximizing the marginal likelihood. In order to circumvent kermel selection they choose the flexible spectral mixture (SM) kernel with fixed number of components per dimension, yielding a kernel with product structure over dimensions. The SM kernel arises from modelling the spectral density (Fourier dual) of kernel function as Gaussian mixture. Our work differs from AHGP as we focus on kernel design rather than optimising hyperparameter values.
Kernel Engineering:
There are several examples in the GP literature of kernels being handcrafted to model one- or two-dimensional data [6. 7, 24]. While this may be feasible in low-dimensional data with the aid of visual inspection, it is much less straightforward in a high-dimensional settings. Automated kernel engineering approaches search over finite space of kernel structures which are progressively built by adding and multiplying small number of base kernels. The focus is on devising an effective search algorithm over discrete structures where the end result is a composite kernel built from simpler known base kernels. This is largely the idea behind the Automatic Statis- tician project [15] where a greedy search procedure searches over all possible operators and sub- expressions select the highest scoring combination. Our work similarly operates universe of compositional kernel structures but with a distinctly different model where we regress kernel labels on data sets with end-to-end gradient based training yielding a fast and scalable method.
There are other methods that bring to bear both the benefits of deep architecture and the analytical flexibility of kernel methods for the problem of representation learning [4, 13, 31]. The methods work by transforming the inputs to a GP with a neural network (NN) and jointly learning the parameters of the NN and the GP. The contention is that simple base kernel (like squared exponential (SE) kernel) works better when applied to the representations learnt by the NN than when applied to the raw input. These works try to side-step the problem of learning a sophisticated kernel apt for the data by focusing instead on learning a transformation of imputs. However, these methods can suffer from overfitting due to the joint training of millions of parameters of the NN in conjunction with the GP hyperparameters [20].
Deep Kernels:
Novel Kernels: Other noted work includes the spectral mixture kernel which reparameterizes the kernel terms of its spectral density (see Bochner's Theorem [3]) and derives closed form kernels which can be used as drop-in replacements for any stationary kernel function [26, 30].

---

Discussion
This work proposes novel approach to addressing the kernel selection problem in GPs. By lever- aging the potential for unlimited training data, we train a transformer-base model to identify the likelihood of sample given a kernel class. Despite being trained solely on synthetic data, KITT is capable of selecting suitable kernels for previously unseen, real-world datasets. While we focus our efforts on the case of one-dimensional outputs, similar models could be developed for outpit regression, classification, latent variable modelling and time-series prediction tasks. A major advan- tage of a pre-trained model for kernel structure detection is the speed of inference. By being able to recommend kernel in fraction of second, KITT is dramatically faster than competing methods such as greedy search algorithmss or differentiable kernel networks. Furthermore, it offers superior scalability. Empirically, input dimensions and dataset sizes. It was found to predict competitive kernels for high-dimensional real valued regression tasks. The ground truth experiments demonstrate its generalisation ability where it is able to identify structure in high-dimensional datasets. This work presents a powerful hybrid approach where kernel selection is informed by representation learning, by inferring range of kernels compatible with the data. This achieves two aims expressivity and ensemble uncer- tainty while spurring new possibilities for informed model selection in Gaussian processes. Given the high degree of complementarity with AHGP, which offers near-instantaneous optimisation of hyperparamters, there appears promising prospects for transformers to enhance the development, flexibility and scalability of Gaussian Process models.
found that KITT is capable of pattern discovery across a broad range of
Acknowledgments and Disclosure of Funding
The authors would like thank the anonymous reviewers for their helpful feedback. VL acknowl- edges funding from the Alan Turing Institute and Qualcomm Innovation Fellowship (Furope).
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton, Layer normalization. arXiv preprint 1607.06500 2016.
[2] Francis Bach. High-dimensional non-linear variable selection through hierarchical kernel learning. arXiv preprint wwXhv:0909.0844, 2009.
[3] Salomon Bochner et al. Lectures on Fourier integrals, volume 42. Princeton University Press, 1959.
[4] Roberto Calandra, Jan Peters, Carl Edward Rasmussen, and Mare Peter Deisenroth. Manifold Gaussian Processes for regression. In 2016 International Joint Conference Neural Networks (IJCN), pages 3338-3345. IEEE, 2016.
[5] Marcella Cornia, Mattco Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578-10587, 2020.
[6] David Duvenaud. Automatic model construction with Gaussian Processes. PhD) thesis, Uni- versity of Cambridge, 2014.
[7] David Duvenaud, James Robert Lloyd, Roger Grosse, Joshua B Tenenbaum, and Zoubin Ghahramani. Structure discovery in nonparametric regression through compositional kernel search. arXiv preprint /rXiv:/302.449 2013.
[8] David Duvenaud, Hannes Nickisch, and Carl Rasmussen. Additive Gaussian Processes. In Advances in neural information processing systems, pages 226-234, 2011.
[9] Roger Fletcher. Practical methods of optimization. John Wiley & Sons, 2013.
[10] Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050- 1059, 2016.
10

---

[11] Arthur Gretton, Philipp Hennig, Carl Edward Rasmussen, and Bernhard Schölkopf. New direc- tions for learning with kernels and Gaussian Processes (dagstuhl seminar 16481). Dagstuhl Reports, volume 6. Schloss Dagstuhl-LeibmnzzZZennn fuer Informatik, 2017.
[12] José Miguel Lemandez-Loobto and Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In International Conference Machine Learning, pages 1861-1869 2015.
[13] Geoffrey E Hinton and Russ R Salakhutdinov. Using deep belief nets to learn covariance kernels for Gaussian Processes. In Advances neural information processing systems, pages 1249-1256, 2008.
[14] John Jumper, R Evans, A Pritzel, T Green, M Figurnov, Tunyasuvunakool, O Ronneberger, R Bates, A Zidek, A Bridgland, et al. High accuracy protein structure prediction using deep learning. Fourteenth Critical Assessment of Techniques for Protein Structure Prediction (Ab- stract Book), 22:24, 2020.
[15] Hyunjik Kim and Yee Whye Tch. Scaling up the Automatic Statistician: Scalable structure discovery using Gaussian Processes. In Amos Storkey and Fernando Perez-Cruz, editors, Pro- ceedings of the 21st International Conference on Artificial Intelligence and Statistics, vol- ume 84 of Proceedings of Machine Learning Research, pages 575-584. PLMR, 2018.
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1111v:141..69800 2014.
[17] Balaji Lakshminarayanaan Alexander Pritzel, and Charles Blundell. Simple and scalable pre- dictive uncertainty estimation using deep ensembles. In Advances in neural information pro- cessing systems, pages 6402-6413, 2017.
[18] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-iavarian neural networks. In International Conference on Machine Learning, pages 3754-3753. PMLR, 2019.
[19] Sulin Liu, Xingyuan Sun, Peter Ramadge, and Ryan P Adams. Task-agnostic amortized inference of Gaussian Process hyperparameters. Advances in Neural Information Processing Systems, 33, 2020.
[20] Sebastian W Ober, Carl Rasmussen, and Mark van der Wilk. The promises and pitfalls of deep kernel learning, arXiv preprint arXiv:2102.12108 2021.
[21] Emilio Parisotto and Russ Salakhutdinov. Efficient transformers in reinforcement learning us- ing actor-learner distillation. In International Conference on Learning Representations, 2021.
[22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.0002. 2021.
[23] Carl E Rasmussen and Christopher KI Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[24] Carl Edward Rasmussen and Christopher KI Williams. Gaussian Processes for machine leamn- ing, chapter Springer, 2006.
[25] Gideon Schwarz. et al. Estimating the dimension of model. Annals of statistics, 6(2):461-464 1978.
[26] Fergus Simpson, Alexis Boukouvalas, Vaclav Cadek, Elvijs Sarkans, and Nicolas Durrande. The minecraft kernel: Modelling correlated Gaussian Processes in the fourier domain. In International Conference on Artificial Intelligence and Statistics, pages 1945-1953. PMLR, 2021.
[27] Alex J Smola and Bernhard Schölkopf. Learning with kernels, volume 4. Citeseer, 1998.

---

[28] Shengyang Sun, Guodong Zhang, Chaoqi Wang. Wenyuan Zeng, Jiaman Li, and Roger Grosse Differentiable compositional kernel learning for gaussian processes. In International Confer- ence on Machine Learning. pages 4828 4837. PMLR, 2018.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural informa- lion processing systems, pages 5998-6008, 2017.
[30] Andrew Gordon Wilson. Covariance kernels for fast automatic pattern discovery and extrap- olation with Gaussian Processes. PhD thesis, University Cambridge, 2014.
[31] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial Intelligence and Statistics, pages 370-378, 2016.
[32] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pages 2015.
PMLR,
[33] Jun Yu, Jing Li, Zhou Yu, and Qingming Huang. Multimodal transformer with multi-view visual representation for image captioning. IEEE transactions circuits and systems for video technology, 30(12):4467-4480, 2019.
[34] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in neural information processing systems, pages 3391-3401, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We highlight the simplicity of
our grammar, and note that at present the optimisation of hyperparameters is currently still required as a separate step.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the
supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Included supple- mentary material
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [Yes] See supple- mentary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Detailed specifications given in Section 4.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See 'Computational overhead" in Section 4
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] Reference to GPflow MMatheww et al)
12

---

(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material URL? [Yes]
We include code and a pretrained KITT model the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "ogYmmwUVcDG", "reviewer": "Reviewer_VZVF", "summary": "The paper presents a new approach to select kernel functions via transformers. The proposed approach mimics image captioning tasks where the model takes the pair of data covariates and targets as image inputs,and then produces kernel grammars as captions. The paper designs a transformer architecture in the way that it is invariant to permutations across input indices and input dimensions. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Kernel selection is one of the important problems in learning Gaussian processes. And using the representation power of transformers is a very interesting idea. The method allows to train a transformer once and reuse many times to predict kernel structures  and therefore avoids exhausted search procedures like in the automatic statistician.\n\nHowever, the technical contribution of this paper is limited. The double permutation invariance is obtained via a function composition of deep sets. Although experiments show promising results, the paper may need more ablation study to demonstrate the efficacy of the proposed method (please see below).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While experiments show promising results in medium-dimensional datasets, the paper may need to verify that the method should work for 1D datasets. Because it is easier to justify the quality of found kernels from plotting and visualization. A plot of kernel structure found by KITT for Mauna or airline passengers might be sufficient. One suggestion here is that the paper may follow the setup of the signal-to-noise experiment in [Duvenaud et al. 2013] to check if the method is sensitive to noise. \n\nMinors:\n- The explanation of why AHGP performs worse than RBF is not clear. If RBF is the subset of Spectral Mixture (SM), SM is at least as good as RBF. And if AHGP really learned the lengthscale of the primary component, the paper should provide a way to demonstrate it.\n\n- Missing related work on stochastic grammar using probabilistic programming. For example, https://arxiv.org/pdf/1611.07051.pdf \n\nQuestions:\n\n- The output of KITT is a sum of product kernels in which components are also commutative and order does not matter. Transformers for natural language generates sentences where the order or words within a sentence matters. Does it cause any difficulty to identify the kernel in KITT?\n\n- What potential challenges may arise if data and kernel grammar includes change-point and change-window? \n\n- In Figure 3 (first), why does the increase in the number of test points result in better classification accuracy although the model trained on 64 points?\n\n- Is automatic relevant determination (ARD) considered during sampling from priors? Is there a way to extend this work where kernel structures vary across input dimensions?\n\n---- After rebuttal ----\nOverall, the proposed approach for GP kernel selection is interesting. So I raised the score to 6. However, the paper representation can be improved. I hope the authors could add the discussion of 1-dimensional case to the paper as in the rebuttal. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "XdLG7zMxkbx", "reviewer": "Reviewer_KsnS", "summary": "The paper proposes a transformer-based neural network that can recommend appropriate kernels for Gaussian process regression. In the first step, the encoder uses a permutation-invariant version of multi-head self-attention to generate a representation of an entire dataset. This is further used by the decoder to recursively select a sequence of primitive kernels, which are then additively combined. The entire model is trained on a synthetic dataset consisting of samples drawn from GP priors with different kernels. Finally, the method is evaluated on a selection of regression datasets from the UCI repository in terms of prediction fit and runtime.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides valuable and---to the best of my knowledge---novel contributions to the field of Gaussian process kernel selection. Compared to the closely related work by Liu et al., KITT can also recommend non-stationary kernels. Compared to search-based methods, the computational overhead during test time is significantly reduced. The paper is generally well-written and easy to understand; however, the structure of the presentation can be improved. Please see the detailed comments below.\n \nThe key strength of the paper is that the model is trained entirely on synthetic data while still being able to select good kernels for previously unseen, real-world datasets. In my opinion, the authors should emphasize this fact much more, as it is never really mentioned explicitly in the paper.\n \nIts main weakness is that the description of the method lacks vital details (for example, about the data generation process) or spreads them across several sections, making it hard to understand the procedure. Furthermore, the evaluation seems slightly unfinished and relatively weak compared to the literature. Please see the comments below for more details.\n \nOverall, I do not see any significant issues with the paper, and I deem that the idea is interesting and useful. For now, I would still rate it slightly below the acceptance threshold due to the issues mentioned in the comments. However, I am confident that the authors can address my comments/questions during the discussion phase, and I would be happy to raise my score if that is the case.\n\n\n# Questions/Detailed Comments \n* What training data was used for training the KITT network? That information is fragmented into several sections of the paper, the supplementary material and I even had to take a look at the code and related work (Liu et al.) to be completely sure. For example, the data generation process is only mentioned briefly in the paragraph about GP hyperparameter priors, the “training” paragraph states the dimensionality of the data, and the supplementary material contains the information that 200,000 samples are used in total. Why not compile this in a single paragraph about the training procedure? That would make the paper much easier to read and understand.\n* It seems a bit odd to me that the AHGP model performs consistently worse than the simple RBF-ARD baseline, because, as you mentioned, the RBF kernel should be included in their formulation of the spectral mixture kernel. Were you able to reproduce results from the AHGP paper?\n* How exactly does the “encoder-only” model mentioned in Fig. 4 work? I did not find this information anywhere in the paper. Is it perhaps the classification model from the ground truth recovery experiment and does it simply select a single primitive kernel expression from the vocabulary or a combination of the top-3 based on softmax values?\n* Based on the experimental results, it seems that the encoder-only model’s performance is on par with the full model including the decoder. Is there any specific reason why you would ever want to use the decoder if the encoder-only model works just as well?\n* The “Scalability” section (l. 232 ff.) feels a bit out of place. Its claims would need to be justified or you should at least provide references. In my opinion, the entire KITT pipeline also includes an $O(N^3)$ term as you optimize hyperparameters after the transformer suggests a suitable kernel, using the marginal likelihood gradient. I would suggest scrapping this section entirely and focusing on a more detailed empirical comparison of wall clock runtime instead, which is probably more relevant than asymptotic complexity for people who want to use your method anyway.\n* Evaluation in the AHGP paper includes experiments in the domain of Bayesian optimization and Bayesian quadrature as well. It would be interesting to see how your methods performs for those tasks, but it is not strictly necessary in my opinion. However, your main selling point over the search-based methods is computational efficiency, so you should certainly include runtime comparisons for the UCI experiments or at least summarize the results and refer to the supplementary material. To make the comparison fair, I think that the KITT’s runtime should also include the optimization of hyperparameters.\n* What is the effect of increasing/decreasing the maximal sequence length that the encoder is allowed to output? What happens if you allow products of more than two kernels. I only found the sentence “We use products of two primitive kernels as part of the base vocabulary and found that incorporating higher order products do not significantly change performance.” (l. 130 f.), but the supplementary material does not contain experimental results that support this statement.\n\n## Minor:\n* l. 57: “This quantity is available in closed-form for data modelled by Gaussian likelihoods.” You my want to explicitly state that “this” refers to the log marginal likelihood and not its global maximum to avoid confusion.\n* Please cite published conference/journal papers instead of the arxiv version. E.g., Vaswani et. al was accepted at NIPS’17\n\n## Typos:\n* l. 122: “multiplying noise kernel” -> “multiplying a noise kernel”\n* p. 5, fig. 2 caption: “heterescedastic” -> “heteroscedastic”\n* l. 191: There is a footnote mark for the Set Attention Block, but the actual footnote does not seem to exist.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "VOEdIGAAScU", "reviewer": "Reviewer_NPVS", "summary": "The authors propose KITT, a novel method exploiting a transformer-based architecture to generate kernel recommendations. KITT has two main features: (1) agnostic to the length and dimensionality of the data; (2) it offers double permutation invariance. They demonstrate the effect of KITT on synthetic data compared with other baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The kernel search problem is new for transformer-based architecture but it's not novel for kernel design. The authors just provide some prior kernel selections and select the result from them. I think one interesting part is Proposition 1 & 2. However, the main result is based on Zaheer et al. [33] and their Proposition can be seen as a direct extension.\n\nQuality: I think the claim is intuitive and heuristic. For example, after defining a set of prior kernels, the authors apply addition and multiplication operations to generate more kernel candidates. The advantage is that the double permutation invariance is supported by their proof.\n\nClarity: In my view, the paper is not well organized and difficult to follow. For example, the contribution part in lines 36-47 is too scattered. Also, I think Section 3 is disordered: they use bold titles for each unparallel part. For instance, Encoder and Decoder should be the sub-part of Model Architecture.\n\nSignificance: The authors verify their KITT synthetic data and show its improvement compared with baselines. I am not sure about the improvement on the real-world data (like NLP dataset). ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors don't provide limitations of their work in the paper.\n\nI mainly have the following concerns about the paper.\n1. In my view, KITT apply an ensemble of tricks or technique together. Thus, an ablation study is important to illustrate the effect of different parts. For example, authors use a lot of places to describe heteroscedastic noise. I wonder what if we remove the noise and keep other parts the same.\n\n2. As said in the Main Review part, all experiments are done on synthetic data, which is a simpler case in reality. Since Transformer is popular for NLP and CV tasks, can the authors show KITT on some small real-world data?\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new approach to select kernel functions via transformers. The proposed idea is interesting, and is a novel contribution to the field of Gaussian process and kernel selection with significantly reduced computational overhead during test time. Experimental results can be further strengthened by adding in more ablation studies. Moreover, organization and presentation can be improved.
Decision: Accept (Poster)