SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. As shown in the main text, the difference comparison function is context-preserving for  $\mathbf{Q}_{m:}^{\prime} = \mathbf{Q}_{mnn}|_{n=1}^{N} = (g(\mathbf{x}_n, \mathbf{x}_m^\star), \mathbf{R}_{nn})_{n=1}^{N}$ , for a given  $m$ . In other words, we can reconstruct  $(\mathbf{X}, \mathbf{Y})$  given only the diagonal of the  $\mathbf{R}$  matrix and the comparison between a single target input  $\mathbf{x}_m^\star$  and the elements of the context set,  $g(\mathbf{x}_n, \mathbf{x}_m^\star)$ . Following a construction similar to the one in Lemma A.2, we can map  $\rho_m$  back to  $\mathbf{Q}_{m:}^{\prime}$ . Then since  $g$  is context-preserving with respect to  $\mathbf{Q}_{m:}^{\prime}$ , we can reconstruct  $(\mathbf{X}, \mathbf{Y})$  up to translations.
A potentially surprising consequence of our context-preservation theorems is that in principle no information is lost about the entire context set. Thus, in principle an RCNP with a sufficiently large network is able to reconstruct any high-order interaction of the context set, even though the RCNP construction only builds upon interactions of pairs of points. However, higher-order interactions might be harder to encode in practice. Since the RCNP representation is built on two-point interactions, depending on the network size, it may be harder for the network to effectively encode the simultaneous interaction of many context points.
B.1 Training procedure, error bars and statistical significance
This section describes the training and evaluation procedures used in our regression experiments. The training procedure follows [3, Appendix G]. Stochastic gradient descent is performed via the Adam algorithm [21] with a learning rate specified in each experiment. After each epoch during model training, each model undergoes validation using a pre-generated validation set. The validation score is a confidence bound based on the log-likelihood values. Specifically, the mean  $(\mu_{\mathrm{val}})$  and standard deviation  $(\sigma_{\mathrm{val}})$  of the log-likelihood values over the entire validation set are used to calculate the validation score as  $\mu_{\mathrm{val}} - 1.96\sigma_{\mathrm{val}} / \sqrt{N_{\mathrm{val}}}$  where  $N_{\mathrm{val}}$  is the validation dataset size. The validation score is compared to the previous best score observed within the training run, and if the current validation score is higher, the current model parameters (e.g., weights and biases for neural networks) are saved. The models are trained for a set number of epochs in each experiment, and when the training is over, the model is returned with the parameters that resulted in the highest validation score.
To ensure reliable statistical analysis and fair comparisons, we repeat model training using the above procedure ten times in each experiment, with each run utilizing different seeds for training and validation dataset generation and model training. In practice, we generated ten seed sets so that each training run with the same model utilizes different seeds, but we maintain consistency by using the same seeds across different models in the same experiment. Each model is then represented with ten training outcomes in each experiment, and we evaluate all training outcomes with a fixed evaluation set or evaluation sets.
We evaluate the methods based on two performance measures: (i) the log-likelihood, normalized with respect to the number of target points, and (ii) the Kullback-Leibler (KL) divergence between the posterior prediction map  $\pi$  and a target  $\pi^{*}$ , defined as
$$
\operatorname {K L} (\pi | | \pi^ {*}) = \int \pi (x) \log \frac {\pi (x)}{\pi^ {*} (x)} \mathrm {d} x,
$$
assuming that  $\pi$  is absolutely continuous to  $\pi^{*}$  with respect to a measure  $\lambda$  (usually, the Lebesgue measure). We report the KL normalized with respect to the number of target points.
Throughout our experiments, we rely on the KL divergence to evaluate the experiments with Gaussian process (GP) generated functions ('Gaussian'), for which we can compute the ground-truth target  $\pi^{*}$  (the multivariate normal prediction of the posterior GP). We use the normalized log-likelihood for all the other ('non-Gaussian') cases. For each model and evaluation set, we report both the average score
calculated across training runs and the standard deviation between scores observed in each training run.
We compare models based on the average scores and also run pairwise statistical tests to identify models that resulted in comparable performance. The results from pairwise comparisons are used throughout the paper in the results tables to highlight in bold the models that are considered best in each experiment. Specifically, we use the following method to determine the best models:
- First, we highlight in bold the model (A) that has the best empirical mean metric.
- Then, for each alternative model (B), we run a one-sided Student's t-test for paired samples, with the alternative hypothesis that model (B) has a higher (better) mean than model (A), and the null hypothesis that the two models have the same mean. Samples are paired since they share a random seed (i.e., the same training data).
- The models that do not lead to statistically significant  $(p < 0.05)$  rejection of the null hypothesis are highlighted in bold as well.
B.2 Experimental details and reproducibility
The experiments carried out in this work used the open-source neural processes package released with previous work [3]. The package is distributed under the MIT license and available at https://github.com/wesselb/neuralprocesses [1]. We extended the package with the proposed relational model architectures and the new experiments considered in this work. Our implementation of RCNP is available at https://github.com/acerbilab/relational-neural-processes.
For the purpose of open science and reproducibility, all training and evaluation details are provided in the following sections of this Appendix as well as in the Github repository linked above.
B.3 Details on Figure 1
In Figure 1 in the main text, we compare CNP and RCNP models. The RCNP model used in this example utilizes a stationary kernel to encode translation equivariance. The encoding in the CNP model and the relational encoding in the RCNP model are 256-dimensional, and both models used an encoder network with three hidden layers and 256 hidden units per layer. In addition, both models used a decoder network with six hidden layers and 256 hidden units per layer. We used the CNP architecture from previous work [3] and modeled the RCNP architecture on the CNP architecture. The encoder and decoder networks use ReLU activation functions.
We trained the neural process models with datasets generated based on random functions sampled from a noiseless Gaussian process prior with a Matérn- $\frac{5}{2}$  covariance function with lengthscale  $\ell = 0.5$ . The datasets were generated by sampling 1-30 context points and 50 target points from interval  $[-2, 2]$ . The neural process models were trained for 20 epochs with  $2^{14}$  datasets in each epoch and learning rate  $3 \cdot 10^{-4}$ . The datasets used in training the CNP and RCNP models were generated with the same seed.
Figure 1 shows model predictions in two regression tasks. First, we visualized the predicted mean and credible bounds calculated as 1.96 times the predicted standard deviation in the range  $[-2, 2]$  given 10 context points. This corresponds to the standard interpolation (INT) task. Then we shifted the context points and visualized the predicted mean and credible bounds in the range [2, 6]. This corresponds to an out-of-input-distribution (OOID) task. The Gaussian process model used as a reference had Matérn- $\frac{5}{2}$  covariance function with lengthscale  $\ell = 0.5$ , the same process used to train the CNP and RCNP models.
C Computational cost analysis
C.1 Inference time analysis
In this section, we provide a quantitative analysis of the computational cost associated with running various models, reporting the inference time under different input dimensions  $d_{x}$  and varying context set sizes  $N$ . We compare our simple and full RCNP models with the CNP and ConvCNP families.
All results are calculated on an Intel Core i7-12700K CPU, under the assumption that these models can be deployed on devices or local machines without GPU access.
Table S1 shows the results with different input dimensions  $d_x$ . We set the size of both context and target sets to 20. Firstly, we can see the runtime of convolutional models increase significantly as they involve costly convolutional operations, and cannot be applied in practice to  $d_x > 2$ . Conversely, the runtime cost remains approximately constant across  $d_x$  for the other models. Secondly, our RCNP models have inference time close to that of CNP models, while FullRCNP models are slower (although still constant in terms of  $d_x$ ). Since context set size is the main factor affecting the inference speed of our RCNP models, we delve further into the computational costs associated with varying context set sizes  $N$ , keeping the input dimension  $d_x = 1$  and target set size  $M = 20$  constant (Table S2). Given that RCNP has a cost of  $O(NM)$ , we observe a small, steady increase in runtime. Conversely, the FullRCNP models, with their  $O(N^2 M)$  cost, show an approximately quadratic surge in runtime as  $N$  increases, as expected.
Table S1: Comparison of computational costs across different models under different input dimensions  $d_{x}$ . The size of both context sets ( $N$ ) and target sets ( $M$ ) is set to 20. We report the mean value and (standard deviation) derived from 50 forward passes executed on various randomly generated datasets.
Runtime (×10-3s)
dx=1	dx=2	dx=3	dx=5
RCNP	2.35 (0.39)	2.29 (0.28)	2.28 (0.32)	2.25 (0.27)
RGNP	2.31 (0.30)	2.41 (0.34)	2.31 (0.26)	2.50 (0.40)
FullRCNP	8.24 (0.46)	9.09 (1.78)	8.27 (0.56)	9.08 (1.72)
FullRGNP	8.40 (0.56)	9.25 (1.85)	8.37 (0.62)	9.20 (1.76)
ConvCNP	4.46 (0.38)	31.19 (2.43)	-	-
ConvGNP	4.51 (0.48)	38.04 (2.31)	-	-
CNP	1.86 (0.30)	2.10 (0.38)	2.14 (0.30)	2.09 (0.29)
GNP	2.01 (0.27)	2.09 (0.26)	2.20 (0.26)	2.10 (0.32)
Table S2: Comparison of computational costs across different models under different context set sizes  $N$ . We maintain the input dimension at  $d_x = 1$ , and a fixed target set size of  $M = 20$ . We report the mean value and (standard deviation) derived from 50 forward passes executed on various randomly generated datasets.
Runtime (×10-3s)
N=10	N=20	N=50	N=100
RCNP	1.90 (0.26)	2.34 (0.48)	2.40 (0.30)	2.90 (0.46)
RGNP	2.10 (0.26)	2.53 (0.34)	2.55 (0.22)	2.93 (0.37)
FullRCNP	2.83 (0.33)	8.80 (1.59)	38.03 (4.71)	154.17 (16.04)
FullRGNP	3.08 (0.34)	8.68 (1.02)	38.33 (5.88)	161.07 (19.54)
ConvCNP	4.47 (0.43)	4.47 (0.51)	5.10 (0.62)	5.83 (0.88)
ConvGNP	5.10 (0.54)	4.55 (0.35)	5.26 (0.50)	4.70 (0.46)
CNP	1.95 (0.27)	1.86 (0.28)	1.87 (0.21)	1.97 (0.19)
GNP	2.48 (0.70)	2.03 (0.19)	2.17 (0.48)	2.11 (0.23)
C.2 Overall training time
For the entire paper, we conducted all experiments, including baseline model computations and preliminary experiments not included in the paper, on a GPU cluster consisting of a mix of Tesla P100, Tesla V100, and Tesla A100 GPUs. We roughly estimate the total computational consumption to be around 25 000 GPU hours. A more detailed evaluation of computing for each set of experiments is reported when available in the following sections.
D Details of synthetic regression experiments
We report details and additional results for the synthetic regression experiments from Section 5.1 of the main text. The experiments compare neural process models trained on data sampled from both Gaussian and non-Gaussian synthetic functions, where 'Gaussian' refers to functions sampled from a
Gaussian process (GP). Our procedure follows closely the synthetic regression experiments presented in [3].
D.1 Models
We compare our RCNP models with the CNP and ConvCNP model families in regression tasks with input dimensions  $d_x = \{1, 2, 3, 5, 10\}$ . The CNP and GNP models used in this experiment encode the context sets as 256-dimensional vectors when  $d_x < 5$  and as 128-dimensional vectors when  $d_x \geq 5$ . Similarly, all relational models including RCNP, RGNP, FullRCNP, and FullRGNP produce relational encodings with dimension 256 when  $d_x < 5$  and dimension 128 when  $d_x \geq 5$ . The encoding network used in both model families to produce the encoding or relational encoding is a three-layer MLP, featuring 256 hidden units per layer for  $d_x < 5$  and 128 for  $d_x \geq 5$ . We also maintain the same setting across all CNP and RCNP models in terms of the decoder network architecture, using a six-layer MLP with 256 hidden units per layer for  $d_x < 5$  and 128 for  $d_x \geq 5$ . The encoder and decoder networks use ReLU activation functions. Finally, the convolutional models ConvCNP and ConvGNP, which are included in experiments where  $d_x = \{1, 2\}$ , are employed with the configuration detailed in [3, Appendix F], and GNP, RGNP, FullRGNP, and ConvGNP models all use linear covariance with 64 basis functions.
Neural process models are trained with datasets representing a regression task with context and target features. The datasets used in these experiments were generated based on random functions sampled from Gaussian and non-Gaussian stochastic processes. The models were trained for 100 epochs with  $2^{14}$  datasets in each epoch and learning rate  $3 \cdot 10^{-4}$ . The validation sets used in training included  $2^{12}$  datasets and the evaluation sets used to compare the models in interpolation (INT) and out-of-input-distribution (OOID) tasks included  $2^{12}$  datasets each.
D.2 Data
We used the following functions to generate synthetic Gaussian and non-Gaussian data:
- Exponentiated quadratic (EQ). We sample data from a GP with an EQ covariance function:
$$
f \sim \mathcal {G P} \left(\mathbf {0}, \exp \left(- \frac {\| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {2} ^ {2}}{2 \ell^ {2}}\right)\right), \tag {S3}
$$
where  $\ell$  is the lengthscale.
- Matérn- $\frac{5}{2}$ . We sample data from a GP with a Matérn- $\frac{5}{2}$  covariance function:
$$
f \sim \mathcal {G P} \left(\mathbf {0}, \left(1 + \sqrt {5} r + \frac {5}{3} r ^ {2}\right) \exp \left(- \sqrt {5} r\right)\right), \tag {S4}
$$
where  $r = \frac{||\mathbf{x} - \mathbf{x}'||_2}{\ell}$  and  $\ell$  is the lengthscale.
- Weakly periodic. We sample data from a GP with a weakly periodic covariance function:
$$
f \sim \mathcal {G P} \left(\mathbf {0}, \exp \left(- \frac {1}{2 \ell_ {d} ^ {2}} | | \mathbf {x} - \mathbf {x} ^ {\prime} | | _ {2} ^ {2} - \frac {2}{\ell_ {p} ^ {2}} \right.\left|\left| \sin \left(\frac {\pi}{p} (\mathbf {x} - \mathbf {x} ^ {\prime})\right)\right|\right| _ {2} ^ {2}\right)\left. \right), \tag {S5}
$$
where  $\ell_d$  is the lengthscale that decides how quickly the similarity between points in the output of the function decays as their inputs move apart,  $\ell_p$  determines the lengthscale of periodic variations, and  $p$  denotes the period.
- Sawtooth. We sample data from a sawtooth function characterized by a stochastic frequency, orientation, and phase as presented by:
$$
f \sim \omega \langle \mathbf {x}, \mathbf {u} \rangle_ {2} + \phi \mod 1, \tag {S6}
$$
where  $\omega \sim \mathcal{U}(\Omega)$  is the frequency of the sawtooth wave, the direction of the wave is given as  $\mathbf{u}\sim \mathcal{U}(\{\mathbf{x}\in \mathbb{R}^{d_x}:||\mathbf{x}||_2 = 1\})$ , and  $\phi \sim \mathcal{U}(0,1)$  determines the phase.
- Mixture. We sample data randomly from either one of the three GPs or the sawtooth process, with each having an equal probability to be chosen.
In this set of experiments, we evaluate the models using varying input dimensions  $d_x = \{1, 2, 3, 5, 10\}$ . To maintain a roughly equal level of difficulty across data with varying input dimensions  $d_x$ , the hyperparameters for the above data generation processes are selected in accordance with  $d_x$ :
$$
\ell = \sqrt {d _ {x}}, \quad \ell_ {d} = 2 \cdot \sqrt {d _ {x}}, \quad \ell_ {p} = 4 \cdot \sqrt {d _ {x}}, \quad p = \sqrt {d _ {x}}, \quad \Omega = \left[ \frac {1}{2 \sqrt {d _ {x}}}, \frac {1}{\sqrt {d _ {x}}} \right]. \tag {S7}
$$
For the EQ, Matérn- $\frac{5}{2}$ , and weakly periodic functions, we additionally add independent Gaussian noise with  $\sigma^2 = 0.05$ .
The datasets representing regression tasks were generated by sampling context and target points from the synthetic data as follows. The number of context points sampled from each EQ, Matérn- $\frac{5}{2}$ , or weakly periodic function varied uniformly between 1 and  $30 \cdot d_x$ , and the number of target points was fixed to  $50 \cdot d_x$ . Since the datasets sampled from the sawtooth and mixture functions represent more difficult regression problems, we sampled these functions for 1-30 context points when  $d_x = 1$  and  $1 - 50 \cdot d_x$  context points when  $d_x > 1$ , and we also fixed the number of target points to  $100 \cdot d_x$ .
All training and validation datasets were sampled from the range  $\mathcal{X} = [-2, 2]^{d_x}$  while evaluation sets were sampled in two ways. To evaluate the models in an interpolation (INT) task, we generated evaluation datasets by sampling context and target points from the same range that was used during training,  $\mathcal{X}_{\mathrm{test}} = [-2, 2]^{d_x}$ . To evaluate the models in an out-of-input-distribution (OOID) task, we generated evaluation datasets with context and target points sampled from a range that is outside the boundaries of the training range, specifically  $\mathcal{X}_{\mathrm{test}} = [2, 6]^{d_x}$ .
D.3 Full results
The results presented in the main text (Section 5.1) compared the proposed RCNP and RGNP models to baseline and convolutional CNP and GNP models in selected synthetic regression tasks. The full results encompassing a wider selection of tasks and an extended set of models are presented in Table S3, S4, S5, S6, S7. We constrained the experiments with EQ and Matérn-  $\frac{5}{2}$  tasks (Table S3 and S4) to input dimensions  $d_{x} = \{1,2,3,5\}$  owing to the prohibitive computational memory costs associated with the FullRCNP model included in these experiments. The other experiments (Table S5-S7) were repeated with input dimensions  $d_{x} = \{1,2,3,5,10\}$ . Our RCNP models demonstrate consistently competitive performance when compared with ConvCNP models in scenarios with low dimensional data, and significantly outperform the CNP family of models when dealing with data of higher dimensions.
Table S3: Synthetic (EQ). Comparison of the interpolation (INT) and out-of-input-distribution (OOID) performance of our RCNP models with different CNP baselines on EQ function with varying input dimensions. We use normalized Kullback-Leibler divergences as our metric and show mean and (standard deviation) obtained from 10 runs with different seeds. Trivial refers to a model that predicts a Gaussian distribution utilizing the empirical mean and standard deviation derived from the context outputs. "F" denotes failed attempts that yielded very bad results. Missing entries could not be run. Statistically significantly best results are bolded.
dx=1	dx=2	dx=3	dx=5
INT	OOID	INT	OOID	INT	OOID	INT	OOID
RCNP (sta)	0.22 (0.00)	0.22 (0.00)	0.26 (0.00)	0.26 (0.00)	0.40 (0.01)	0.40 (0.01)	0.45 (0.00)	0.45 (0.00)
RGNP (sta)	0.01 (0.00)	0.01 (0.00)	0.03 (0.00)	0.03 (0.00)	0.05 (0.00)	0.05 (0.00)	0.11 (0.00)	0.11 (0.00)
FullRCNP (iso)	0.22 (0.00)	0.22 (0.00)	0.26 (0.00)	0.26 (0.00)	0.31 (0.00)	0.31 (0.00)	0.35 (0.00)	0.35 (0.00)
FullRGNP (iso)	0.03 (0.00)	0.03 (0.00)	0.08 (0.00)	0.08 (0.00)	0.14 (0.00)	0.14 (0.00)	0.25 (0.00)	0.25 (0.00)
ConvCNP	0.21 (0.00)	0.21 (0.00)	0.22 (0.00)	0.22 (0.00)	-	-	-	-
ConvGNP	0.00 (0.00)	0.00 (0.00)	0.01 (0.00)	0.01 (0.00)	-	-	-	-
CNP	0.25 (0.00)	2.21 (0.70)	0.33 (0.00)	4.54 (1.76)	0.44 (0.00)	3.30 (1.55)	0.57 (0.00)	1.22 (0.09)
GNP	0.02 (0.00)	F	0.05 (0.00)	2.25 (0.61)	0.09 (0.01)	2.54 (1.44)	0.19 (0.00)	0.74 (0.02)
Trivial	1.03 (0.00)	1.03 (0.00)	1.13 (0.00)	1.13 (0.00)	1.12 (0.00)	1.12 (0.00)	1.03 (0.00)	1.03 (0.00)
We also conducted an additional experiment to explore all the possible combinations of comparison functions with the RCNP and FullRCNP models on EQ and Matérn- $\frac{5}{2}$  tasks. Table S8 provides a comprehensive view of the results for EQ tasks across three input dimensions. The empirical results support our previous demonstration that RCNP models are capable of incorporating translation-equivariance with no loss of information: the stationary versions of the RCNP model
Table S4: Synthetic (Matérn- $\frac{5}{2}$ ). Comparison of the interpolation (INT) and out-of-input-distribution (OOID) performance of our RCNP models with different CNP baselines on Matérn- $\frac{5}{2}$  function with varying input dimensions. We use normalized Kullback-Leibler divergences as our metric and show mean and (standard deviation) obtained from 10 runs with different seeds. Trivial refers to a model that predicts a Gaussian distribution utilizing the empirical mean and standard deviation derived from the context outputs. "F" denotes failed attempts that yielded very bad results. Missing entries could not be run. Statistically significantly best results are bolded.
dx=1	dx=2	dx=3	dx=5
INT	OOID	INT	OOID	INT	OOID	INT	OOID
RCNP (sta)	0.25 (0.00)	0.25 (0.00)	0.30 (0.00)	0.30 (0.00)	0.39 (0.00)	0.39 (0.00)	0.35 (0.00)	0.35 (0.00)
RGNP (sta)	0.01 (0.00)	0.01 (0.00)	0.03 (0.00)	0.03 (0.00)	0.05 (0.00)	0.05 (0.00)	0.11 (0.00)	0.11 (0.00)
FullRCNP (iso)	0.25 (0.00)	0.25 (0.00)	0.30 (0.00)	0.30 (0.00)	0.32 (0.00)	0.32 (0.00)	0.29 (0.00)	0.29 (0.00)
FullRGNP (iso)	0.03 (0.00)	0.03 (0.00)	0.09 (0.00)	0.09 (0.00)	0.16 (0.00)	0.16 (0.00)	0.21 (0.00)	0.21 (0.00)
ConvCNP	0.24 (0.00)	0.24 (0.00)	0.26 (0.00)	0.26 (0.00)	-	-	-	-
ConvGNP	0.00 (0.00)	0.00 (0.00)	0.01 (0.00)	0.01 (0.00)	-	-	-	-
CNP	0.29 (0.00)	F	0.39 (0.00)	6.75 (2.72)	0.46 (0.00)	1.75 (0.42)	0.47 (0.00)	0.93 (0.02)
GNP	0.02 (0.00)	2.96 (1.77)	0.07 (0.00)	1.86 (0.26)	0.11 (0.00)	1.23 (0.17)	0.19 (0.00)	0.62 (0.02)
Trivial	1.04 (0.00)	1.04 (0.00)	1.06 (0.00)	1.06 (0.00)	0.98 (0.00)	0.98 (0.00)	0.79 (0.00)	0.79 (0.00)
Table S5: Synthetic (weakly-periodic). Comparison of the interpolation (INT) and out-of-input-distribution (OOID) performance of our RCNP models with different CNP baselines on weakly-periodic function with varying input dimensions. We use normalized Kullback-Leibler divergences as our metric and show mean and (standard deviation) obtained from 10 runs with different seeds. Trivial refers to a model that predicts a Gaussian distribution utilizing the empirical mean and standard deviation derived from the context outputs. "F" denotes failed attempts that yielded very bad results. Missing entries could not be run. Statistically significantly best results are bolded.
dx=1	dx=2	dx=3	dx=5	dx=10
INT	OOID	INT	OOID	INT	OOID	INT	OOID	INT	OOID
RCNP (sta)	0.24 (0.00)	0.24 (0.00)	0.24 (0.00)	0.24 (0.00)	0.28 (0.00)	0.28 (0.00)	0.31 (0.00)	0.31 (0.00)	0.31 (0.00)	0.31 (0.00)
RGNP (sta)	0.03 (0.00)	0.03 (0.00)	0.05 (0.00)	0.05 (0.00)	0.05 (0.00)	0.05 (0.00)	0.08 (0.00)	0.08 (0.00)	0.11 (0.00)	0.11 (0.00)
ConvCNP	0.21 (0.00)	0.21 (0.00)	0.20 (0.00)	0.20 (0.00)	-	-	-	-	-	-
ConvGNP	0.01 (0.00)	0.01 (0.00)	0.02 (0.00)	0.02 (0.00)	-	-	-	-	-	-
CNP	0.31 (0.00)	2.88 (0.91)	0.39 (0.00)	1.81 (0.43)	0.39 (0.00)	1.58 (0.50)	0.42 (0.00)	2.20 (0.81)	0.75 (0.00)	1.03 (0.11)
GNP	0.06 (0.00)	F	0.07 (0.00)	2.57 (0.76)	0.08 (0.01)	1.47 (0.27)	0.11 (0.01)	0.62 (0.04)	0.22 (0.01)	0.49 (0.05)
Trivial	0.78 (0.00)	0.78 (0.00)	0.81 (0.00)	0.81 (0.00)	0.80 (0.00)	0.80 (0.00)	0.77 (0.00)	0.77 (0.00)	0.76 (0.00)	0.76 (0.00)
deliver performance comparable to the stationary FullRCNP models. Conversely, since the RCNP model is not context-preserving for rigid transformations, the isotropic versions of the RCNP model exhibit inferior performance in comparison to the isotropic FullRCNPs.
E Details of Bayesian optimization experiments