SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We evaluate GPP's ability to probe using the 3D Shapes dataset [Burgess and Kim, 2018], which contains images generated from 6 ground truth independent primitives: 10 floor colors, 10 wall colors, 10 object colors, 8 scales, 4 shapes and 15 orientations of the shapes, as shown below.
Datasets and models. We construct a 2-level concept ontology (see Appendix for visualization) consisting of a disjunctive level where colors are binarized to warm/cool and scale to small/large, and a conjunctive level that combines shapes, binarized colors and binarized scales.
We train three CNN models on labels generated from this ontology: 1)  $\mathcal{M}.1$  is trained on 64 labels (binarized scale, floor, wall object color x 4 shapes), 2)  $\mathcal{M}.2$  is trained on 8 labels (4 shapes x binarized scale) 3)  $\mathcal{M}.3$  is trained on 8 labels (binary floor, wall and object color).
We define two probing tasks to investigate each model: 1) task  $\mathcal{P}.1$ : binarized colors (floor, wall, object) and 2) task  $\mathcal{P}.1$ : binarized scales and 4 shapes.
Baseline descriptions. The baselines include 1) LP: linear probes [Alain and Bengio, 2016]; 2) SVM: SVM probes (a baseline from Kim et al. [2018]); 3) LPE: linear probes ensembled via bootstrap [Kim et al., 2018]. For each linear probe in LPE, we train a logistic regression classifier on a dataset sampled from the original set of observations with replacement. The size of the dataset is the same as the number of observations. Each ensemble has 100 linear probes. LPE can also describe a distribution over classifiers and be evaluated as probabilistic probes, making this a strong baseline. For GPP, we use  $\mathrm{Beta}(0.1,0.1)$  as the prior and set strength to be 5.
4.2 Validating GPP as a probe
To compare with baseline methods, we use AUROC to evaluate the deterministic classifier defined by the judged probability  $\mathbb{E}[g(a)],\forall a\in \pmb{a}_q$  (defined in §3.4). Queries are sampled disjointly from the training data and observations. Figure 4 shows that GPP performs competitively against LP and SVM. GPP and LP perform comparable with respect to AUROC and sample efficiency (needing as few as 10 observations for probing concepts in task  $\mathcal{P}$  .1).
One may first guess that  $\mathcal{M}.3$  is the best model that represents color, since it was explicitly tasked to distinguish color. However, our probing results reveal what is obvious in hindsight: while  $\mathcal{M}.2$  is only tasked to distinguish shapes and scales, separating where the shapes are requires detecting color (i.e., the only cue to separate shape from background is color). This is shown in  $\mathcal{M}.2$ 's high AUROC for  $\mathcal{P}.1$  in Figure 4.  $\mathcal{M}.3$  is only tasked to distinguish colors, so performance on  $\mathcal{P}.2$  remains low.
4.3 Estimating concept fuzziness
What happens when the concepts we are trying to probe are inherently fuzzy? This is commonly the case in the real world, where situations and the language used to describe them contain extensive ambiguity. We again focus on  $\mathcal{M}.1$  and concepts in task  $\mathcal{P}.1$ . We control the fuzziness of concepts in  $\mathcal{P}.1$  by randomly flipping positive labels to negative (but not vice versa) in the observations. We test ground truth label probabilities of 0.25, 0.5, 0.75 and 1, where 1 indicates no fuzziness. To measure GPP's alignment with ground truth label probabilities, we use the Pearson correlation coefficient between ground truth label probabilities and judged probabilities for all queries.
Figure 4: AUROC results for probing  $\mathcal{M}.1$ ,  $\mathcal{M}.2$  and  $\mathcal{M}.3$  with  $\mathcal{P}.1$  and  $\mathcal{P}.2$  tasks.  $\mathcal{M}.1$  and  $\mathcal{M}.3$  are trained on color-related tasks and all probes obtain better performance on color concepts  $\mathcal{P}.1$  than geometry concepts  $\mathcal{P}.2$ . The probing results reveal something obvious in hindsight; while  $\mathcal{M}.2$  is trained on geometry-related tasks, it needed color to separate the object from the background color in order to detect geometry, resulting in significant representation of color concepts,  $\mathcal{P}.1$ .
Figure 5: Left: Pearson correlation coefficient between ground truth label probabilities and judged probabilities. GPP obtains a much higher correlation than LPE, showing that GPP is better at detecting fuzzy concepts. Middle and Right: Relationship between judged probability and episteme of positive labels using LPE and GPP when ground truth label probability is 0.5. LPE fails to detect fuzziness and assigns extreme judged probabilities to queries for 8 or more observations. With 2 observations, LPE has higher episteme than with more observations. This behavior is irrational [Fox and Ülkümen, 2011]: fewer observations should indicate lower episteme, and with low episteme high judged probability indicates LPE is being confidently ignorant (no knowledge but believes something will definitely happen). On the contrary, GPP shows rational behavior: with high episteme, the mass of GPP's judged probability predictions centers at 0.5, which aligns with the ground truth.
Figure 6: Alea predictions as a function of episteme for LPE and GPP. When ground truth label probability is 0.5 (Left and Middle), LPE predicts low alea for observations larger than 2, which does not agree with the ground truth alea (0.69). GPP shows much more rational predictions. Comparing Middle and Right, GPP predicts higher alea with more observations for ground truth label probability 0.5 (the corresponding alea is 0.69), and its alea predictions converge to 0 (ground truth alea is 0) for ground truth label probability 1.
Figure 5 shows that GPP can accurately probe fuzzy concepts by predicting both judged probability (expected level of fuzziness) and episteme (how sure it is about the judged probability). GPP achieves a higher Pearson correlation coefficient than LPE across all sizes of observations. In particular, the relationship between judged probabilities and episteme coincides with the two-dimensional framework for characterizing uncertainty from Fox and Ülkümden [2011] (see Fig. 2), showing the validity of GPP as a rational method to probe fuzziness.
Figure 5 (Right) confirms that the judged probability alone cannot be used to distinguish whether concept is fuzzy or the stimuli are far away: in both cases, judged probabilities can be 0.5. Hence, it is important to remember that judged probability is only accurate with enough observations. We illustrate this point from another angle in Figure 6, which shows the relation between alea and episteme predicted by LPE, GPP with ground truth label probability 0.5 or 1. When ground truth label probability is 0.5, GPP can have low alea under low episteme (same effect as Figure 1), but with more episteme, GPP puts more mass of predictions for high alea because the ground truth alea is high. LPE does not show this kind of rational behaviors. When ground truth label probability is 1, more mass of alea predicted by GPP converges to 0 as episteme increases, since the ground truth alea is 0.
4.4 Experiment setups for OOD detection
When learning new concepts, it is natural for people to be able to say "I'm not sure since I haven't learned it yet". For probes, the corresponding ability is to perform out-of-distribution (OOD) detection, which identifies datapoints that are different from the observed training data. GPP naturally performs OOD detection using episteme predictions. With an uninformative prior for the labels, $^3$  high episteme indicates in-domain (ID), while low episteme indicates OOD.
Method descriptions. For GPP, we use the negative latent variance of posterior function  $f$  (Eq. 2) as a proxy of episteme for computational efficiency. It is easy to show that the entropy of classifier  $g$  is upper bounded by the entropy of latent function  $f$ , which is fully determined by its posterior variance (more details in Appendix). In practice, we found that using the negative latent variance as OOD detection scores for GPP is robust to the level of informativeness of the Beta prior, because the latent GPs are agnostic with respect to the logistic transformation in Eq. 2.
As for baselines, we compare against MSP (maximum predicted softmax probabilities using LP) [Hendrycks and Gimpel, 2016], Maha (negative Mahalanobis distance-based score) [Lee et al., 2018], NN (deep nearest neighbors) [Sun et al., 2022] and LPE. Recall that LPE uses bootstrapping to create an ensemble of linear probes. Each linear probe in LPE can be viewed as i.i.d. samples from an underlying distribution over classifiers (i.e.,  $g$  in Figure 1). Similar to approximating metrics with samples in §3.4, we can estimate the negative variance of predictions from the set of linear probes, and use it as the OOD detection score for LPE.
As shown in the extensive analyses of OOD detection methods and tasks in Appendix E of Tran et al. [2022], Maha and LPE (LPE is equivalent to their "Entropy" method for ensembles) achieved top performance, surpassing more recent proposals from Ren et al. [2021].
Datasets and models. We generate a set of queries from task  $\mathcal{P}$  1 of 3D Shapes that include 1024 ID images and 1024 OOD images (uniform random noise), and see if OOD queries can be detected using the methods above based on representations from model  $\mathcal{M}$  1.
We also evaluate the OOD detection performance of all methods on real world datasets and models. Queries and observations in the real world datasets are sampled disjointly from the validation split of the ImageNet dataset [Russakovsky et al., 2015]. We make 10 sets of  $D_{s}$  using 10 binary classification tasks defined by superset of ImageNet classes. Query points consist of 128 images from the classes the probe has observed and 128 OOD images (uniform random noise). The ID examples are images that come from the same distribution that the probe observes. For example, consider a probe trained to classify "all dogs" vs "all cats"; images of dogs and cats would be ID, whereas random noise would be OOD. The models we used are the pre-trained ResNet-50 [He et al., 2016] and CoCa-Base [Yu et al., 2022]. See more details in Appendix.
For example,  $\mathrm{Beta}(0.1, 0.1)$  is a more informative prior than  $\mathrm{Beta}(1, 1)$ , because the former means the classifier outputs either 0 or 1 with high probability, while the latter puts a uniform distribution over any real numbers in  $[0, 1]$ . See Figure 9 for visualization of the PDF of Beta distributions.
Figure 7: AUROC for ID/OOD detection with an increasing number of observations for task  $\mathcal{P}$ .1 of 3D Shapes (Left) and real-world datasets constructed from ImageNet (Middle and Right), showing GPP's comparable performance and data efficiency in OOD detection.
4.5 Results for OOD detection
Figure 7 shows the competitive performance of GPP as an OOD detector for all the OOD detection tasks we constructed. While GPP might not be the best OOD detection approach for each of the tasks, GPP performed consistently well and on par with the best benchmark method for each task. This means, while GPP is mainly a probing method, it is also competitive for OOD detection.
Note that the effective term in GPP's negative latent posterior variance,  $k(a, a_o) K^{-1} k(a_o, a)$  (here  $a_o$  is the concatenated observed activations and  $K = k(\pmb{a}_o, \pmb{a}_o) \in \mathbb{R}^{O \times O}$ ), looks very similar to Mahalanobis distance,  $(a - \hat{u}_c)^\top \hat{\Sigma}^{-1}(a - \hat{u}_c)$  (Eq. 2 from Lee et al. [2018], where  $\hat{\Sigma} \in \mathbb{R}^{d \times d}$ ,  $d$  is the dimension of activations). However, GPP's variance measures the similarity between function values but Mahalanobis distance measures the similarity between activations. Also note that the Gaussian estimated for Mahalanobis distance is degenerate for  $d > O$ , and so the distance is still computed on a  $O$ -dimensional manifold; this is possibly why the Maha method shows inferior performance.
We introduced GPP, an uncertainty-aware probing framework that uses Gaussian processes to measure both the aleatory uncertainty intrinsic to the concept represented by a model and the epistemic uncertainty reflecting the probe's confidence about its judgement. These uncertainty measurements also equip GPP with the ability to detect OOD stimuli. Empirically, we verified the strong performance of GPP for data-efficient probing that can correctly surface fuzziness in concepts and OOD detection which shows GPP can also surface epistemic uncertainty (i.e., knows that it does not know). These results illustrate how distinguishing between different forms of uncertainty can be beneficial both for deepening our theoretical understanding of models and for practical applications.
Limitations. We have validated our approach on both synthetic and real-world datasets, and demonstrated its use for OOD detection, but there are a much wider range of models, datasets, and applications that can be explored. Fully evaluating the potential of this approach and achieving the greatest impact will require pursuing this investigation, which we hope to do in future work. In particular, we plan to explore active few-shot learning and distributionally robust probing. Finally, GPP can only be used with models for which activations are made available, which may present an obstacle to use with certain commercial machine learning systems.
Broader impact. Being able to understand what models can and cannot represent has huge implications. Not only does it impact important downstream tasks, such as OOD detection, but it ultimately aids human-machine communication and collaboration. While a (potentially) superficial interaction (e.g., with a chatbot) is already possible, we also know models hallucinate [Ji et al., 2023], leaving people unsure about the validity/faithfulness of this interaction. Work such as ours ultimately aims to help humans to better use and develop models that aligns well with our concepts, as well as potentially learning new concepts from machines.
We thank Fei Sha for constructive discussions on aleatory uncertainty in classification. We also thank Geoff Pleiss, Alexander Terenin, Ben Adlam, Jasper Snoek and Zackary Nado for helpful conversations on Gaussian process classification. Images of "blicket" from https://www.iwawine.com/Images/riedel-vinum-martini-glasses_10.jpg and https://libreshot.com/wp-content/uploads/2017/12/green-olives-on-the-tree.jpg (by Martin Vorel).
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning (ICML), 2018.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022.
Michael E McCloskey and Sam Glucksberg. Natural categories: Well defined or fuzzy sets? Memory & Cognition, 6(4):462-472, 1978.
Fei Xu and Joshua B Tenenbaum. Word learning as Bayesian inference. Psychological Review, 114 (2):245, 2007.
Carl Edward Rasmussen and Christopher KI Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
Craig R Fox and Gülden Ülkümen. Distinguishing two dimensions of uncertainty. Fox, Craig R. and Gülden Ülkümen (2011), “Distinguishing Two Dimensions of Uncertainty,” in Essays in Judgment and Decision Making, Brun, W., Kirkebøen, G. and Montgomery, H., eds. Oslo: Universitetsforlaget, 2011.
William C Howell and Sarah A Burnett. Uncertainty measurement: A cognitive taxonomy. Organizational Behavior and Human Performance, 22(1):45-68, 1978.
E. T. Jaynes. Probability Theory: The Logic of Science. Cambridge university press, 2003.
Ian Hacking. The Emergence of Probability: A Philosophical Study of Early Ideas about Probability, Induction and Statistical Inference. Cambridge University Press, 2006.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Mate Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745, 2011.
Philipp Hennig and Christian J Schuler. Entropy search for information-efficient global optimization. Journal of Machine Learning Research (JMLR), 13:1809-1837, 2012.
Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient Bayesian optimization. In International Conference on Machine Learning (ICML), 2017.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In International Conference on Machine Learning (ICML), 2017.
Zi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-trained Gaussian processes for Bayesian optimization. arXiv preprint arXiv:2109.08215, 2023.
Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning (ICML), 2018.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Machine Learning (ICML), 2016.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning (ICML), 2022.
Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXiv:2207.07411, 2022.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
George A. Miller. WordNet: A lexical database for English. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994. URL https://aclanthology.org/H94-1111.
Table 2: Notations for GPP.
Terminology	Symbol	Meaning
Model	M	The model to probe.
Stimuli space	X	The space of stimuli.
Representation space	A ⊆ Rd	The space of vector representations of M.
Basis functions	φ	The function (contained in M) mapping from X to A.
Vector representation	φ(x)	The vector representation of a stimulus x ∈ X.
α	α : A → R+, mapping to the 1st parameter of a Beta distribution.
β	β : A → R+, mapping to the 2nd parameter of a Beta distribution.
tα	A random function such that tα(a) ~ Gamma(α(a), 1).
tβ	A random function such that tβ(a) ~ Gamma(β(a), 1).
Beta distribution	Beta(α(a), β(a))	The Beta distribution for g(a), ∀a ∈ A.
Mean function	μ	μ : A → R.
Kernel function	k	k : A × A → R.
GP	GP(μ, k)	A GP with mean function μ and kernel function k.
fα ~ GP(μ, k)	fα : A → R, the (latent) function for approximating tα.
fβ ~ GP(μ, k)	fβ : A → R, the (latent) function for approximating tβ.
θ	Parameter for the Beta GP in GPP, θ = (μ, k).
Beta GP	G(θ)	A distribution over functions mapping from A to [0, 1].
Classifier	g ~ G(θ)	g = 1/(1+e-γ) : A → [0, 1], a random function.
Observations	D	D = {(φ(xi), yi)}i=1|D|, xi ∈ X, yi ∈ {0, 1}.
Beta GP posterior	G(θ | D)	The Beta GP conditional on observations D.
Queries	aq	aq = [φ(xj)]j=1|aq| ∈ Rd×|aq|.
Predicted probabilities	g(aq)	g(aq) = [g(a)]a∈aq.
In Table 2, we include the main notation used in this paper.
B Details on the Beta GP in GPP
As shown in § 3.2 and §3.3, with observations  $D = \{(a_i, y_i)\}_{i=1}^{|D|}$ , the posterior of a Beta GP can be written as the transformed version of a GP, i.e.,
$$
g = \frac {1}{1 + e ^ {- f}} \sim \mathcal {G} (\theta \mid D), \text {w h e r e} f \sim \mathcal {G P} \left(\mu_ {D}, k _ {D}\right).
$$
We show how to obtain the mean and kernel functions  $\mu_{D}, k_{D}$  in §B.1, as well as the posterior of weights in §B.2 for the cosine kernel. The behavior of GPP relies on two hyperparameters,  $\epsilon$  and  $s$ , and we explain what they are and how to set them in §B.3. In §B.4, we analyze and bound the episteme of GPP.
As a reference, Figure 8 shows the graphical model of GPP.
Figure 8: Left: Graphical model of the Beta GP in GPP with  $N$  observations. Right: The stochastic process that a Beta GP approximates, where  $h = \frac{t_{\alpha}}{t_{\alpha} + t_{\beta}} \sim \mathrm{Beta}(\epsilon, \epsilon), t_{\alpha} \sim \mathrm{Gamma}(\epsilon, 1), t_{\beta} \sim \mathrm{Gamma}(\epsilon, 1)$ , and  $t_{\alpha} \perp t_{\beta}$ . In Beta GP,  $g = \frac{e^{f_{\alpha}}}{e^{f_{\alpha}} + e^{f_{\beta}}}$ , where  $f_{\alpha}$  approximates  $\log t_{\alpha}$ , and  $f_{\beta}$  approximates  $\log t_{\beta}$ . Hyperparameter  $s$  is the weight for each observation.
B.1 Posterior inference (extension of §3.3.2)
Without loss of generality, we write observations as a union of a dataset (of size  $n$ ) with the positive labels only and a dataset (of size  $|D| - n$ ) with negative labels only, i.e.,  $D = \{(a_i, y_i)\}_{i=1}^{|D|} = D^{+} \cup D^{-}$  where  $D^{+} = \{(a_i, y_i)\}_{i=1}^{n}$  and  $D^{-} = \{(a_i, y_i)\}_{i=n+1}^{|D|}$ .
For convenience, we use the following short-hand notation:
$$
v ^ {\prime} = \log (\frac {1}{\epsilon + s} + 1), \quad v ^ {\prime \prime} = \log (\frac {1}{\epsilon} + 1), \quad y ^ {\prime} = \log (\epsilon + s) - \frac {v ^ {\prime}}{2}, \quad y ^ {\prime \prime} = \log (\epsilon) - \frac {v ^ {\prime \prime}}{2}.
$$
Observing a positive example is equivalent to observing  $y'$  with noise  $\mathcal{N}(0, v')$  on  $f_{\alpha}$  and observing  $y''$  with noise  $\mathcal{N}(0, v'')$  on  $f_{\beta}$ . Vice versa for observing a negative example. We also denote  $1_m$  as a column vector of size  $m$  filled with 1s, and  $I_m$  as an identity matrix of size  $m$ .
Recall that  $f_{\alpha} \sim \mathcal{GP}(\mu, k)$ ,  $f_{\beta} \sim \mathcal{GP}(\mu, k)$  and  $f_{\alpha} \perp f_{\beta}$ . The posterior for  $f_{\alpha}$  is  $f_{\alpha} \mid D \sim \mathcal{GP}(\mu_{\alpha}, k_{\alpha})$ . For any  $a, a' \in \mathcal{A}$ ,
$$
\mu_ {\alpha} (a) = \mu (a) + k (a, \boldsymbol {a}) K _ {\alpha} ^ {- 1} (\boldsymbol {y} _ {\alpha} - \mu (\boldsymbol {a})), \quad k _ {\alpha} (a, a ^ {\prime}) = k (a, a ^ {\prime}) - k (a, \boldsymbol {a}) K _ {\alpha} ^ {- 1} k (\boldsymbol {a}, a ^ {\prime}), \tag {5}
$$
$$
k (a, \boldsymbol {a}) = [ k (a _ {i}, a) ] _ {i = 1} ^ {| D |} \in \mathbb {R} ^ {1 \times | D |}, \quad k (\boldsymbol {a}, a ^ {\prime}) = [ k (a _ {i}, a ^ {\prime}) ] _ {i = 1} ^ {| D |} \in \mathbb {R} ^ {| D | \times 1},
$$
$$
\mu (\pmb {a}) = [ \mu (a _ {i}) ] _ {i = 1} ^ {| D |} \in \mathbb {R} ^ {| D | \times 1}, \quad K = [ k (a _ {i}, a _ {j}) ] _ {i = 1, j = 1} ^ {| D |} \in \mathbb {R} ^ {| D | \times | D |},
$$
$$
\boldsymbol {y} _ {\alpha} = \left[ \begin{array}{c} y ^ {\prime} 1 _ {n} \\ y ^ {\prime \prime} 1 _ {| D | - n} \end{array} \right] \in \mathbb {R} ^ {| D | \times 1}, \quad K _ {\alpha} = K + \left[ \begin{array}{c c} v ^ {\prime} I _ {n} & 0 \\ 0 & v ^ {\prime \prime} I _ {| D | - n} \end{array} \right] \in \mathbb {R} ^ {| D | \times | D |}.
$$
Similarly for  $f_{\beta} \mid D \sim \mathcal{GP}(\mu_{\beta}, k_{\beta})$ , we have
$$
\mu_ {\beta} (a) = \mu (a) + k (a, \boldsymbol {a}) K _ {\beta} ^ {- 1} (\boldsymbol {y} _ {\beta} - \mu (\boldsymbol {a})), \quad k _ {\beta} (a, a ^ {\prime}) = k (a, a ^ {\prime}) - k (a, \boldsymbol {a}) K _ {\beta} ^ {- 1} k (\boldsymbol {a}, a ^ {\prime}), \tag {6}
$$