Gaussian Process Probes (GPP) for Uncertainty-Aware Probing
Authors: Zi Wang
Alexander Ku
Jason Michael Baldridge
Thomas L. Griffiths
Been Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: Interpretability
probing
Bayesian
Gaussian process
transparency
TL;DR: We introduce Gaussian process probes (GPP), a unified and simple framework for probing and measuring uncertainty about concepts represented by models.

Abstract: Understanding which concepts models can and cannot represent has been fundamental to many tasks: from effective and responsible use of models to detecting out of distribution data. We introduce Gaussian process probes (GPP), a unified and simple framework for probing and measuring uncertainty about concepts represented by models. As a Bayesian extension of linear probing methods, GPP asks what kind of distribution over classifiers (of concepts) is induced by the model. This distribution can be used to measure both what the model represents and how confident the probe is about what the model represents.  GPP can be applied to any pre-trained  model with vector representations of inputs (e.g., activations). It does not require access to training data, gradients, or the architecture. We validate GPP on datasets containing both synthetic and real images. Our experiments show it can (1) probe a model's representations of concepts even with a very small number of examples, (2) accurately measure both epistemic uncertainty (how confident the probe is) and aleatory uncertainty (how fuzzy the concepts are to the model), and (3) detect out of distribution data using those uncertainty measures as well as classic methods do. By using Gaussian processes to expand what probing can offer, GPP provides a data-efficient, versatile and uncertainty-aware tool for understanding and evaluating the capabilities of machine learning models.

Gaussian Process Probes (GPP) for Uncertainty-Aware Probing
Zi Wang" Google DeepMind
Alexander Ku" Google DeepMind
Jason Baldridge
Thomas L. Griffiths Princcton University
Been Kim Google DeepMind
Google DeepMind
Abstract
Understanding which concepts models can and cannot represent has been funda- mental to many tasks: from effective and responsible use of models to detecting out of distribution data. We introduce Gaussian process probes (GPP), a unified and simple framework for probing and measuring uncertainty about concepts rep- resented by models. As Bayesian extension linear probing methods, GPP asks what kind of distribution over classifiers (of concepts) is induced by the model. This distribution can be used to measure both what the model represents and how confident the probe is about what the model represents. GPP can be applied any pre-trained model with vector representations of inputs (e.g., activations). It does not require access to training data, gradients, the architecture. We validate GPP on datasets containing both synthetic and real images. Our experiments show it can (1) probe model's representations of concepts even with very small number of examples, (2) accurately measure both epistemic uncertainty (how confident the probe is) and aleatory uncertainty (how fuzzy the concepts are the model), and (3) detect out of distribution data using those uncertainty mcasures as well as classic methods do. By using Gaussian processes to expand what probing can offer, GPP provides a data-efficient, versatile and uncertainty- aware tool for understanding and evaluating the capabilities of machine learning models.
Introduction
Deep learning models have become pervasive in a wide range of fields, yet they remain largely uninterpretable. As result, gaining insight into which concepts deep learning models do or do not represent is of growing importance. This has led to the development of family of methods known as "probes" [Alain and Bengio, 2016), which aim to understand the representations learned by these models. One common approach to probe the representations of model is to train independent classifiers for each concept and measure their accuracy [Alain and Bengio, 2016, Kim al., 2018, Belinkov, 2022]. For example, if the classifier returns 0.5 probability (for the binary case) across range of examples, we would reasonably conclude that the model does not represent the concept.
However, simply examining the output of classifier may not adequately reflect the intricate nature of representing a concept. For example, we can think of two different scenarios that may result in the same classifier output. If you ask person if an olive is fruit, only about 60% will say yes (likewise 72% for avocado, 53% for pumpkin, and 40% for acorn) [McCloskey and Glucksberg, 1978]. This does not indicate that people have limited representation of the fruit concept, but that
'Equal contribution. (wangzi, alexku, jasobaldrdde, beenkim @google.com, cong@princeton.ede Our code can be found https: //tgt//ggttt//// com/ http://ooooooooeeeeeeeeeeee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

there is intrinsic fuzziness in the label. By contrast, if you show somebody an olive in martini glass and tell them it new concept called "blicket," they may be 60% sure judging whether an olive on a tree is a blicket (as opposed to "blicker" having specialized meaning that applies to cocktails). In this second case, the uncertainty because they do not have enough observations to be certain about what the new concept is common problem in word learning [Xu and Tenenbaum, 2007].
Fuzzy concept: fruit
New concept: blicket
Labels:
1
1 60% (Olive fruit) 100% (Olive is 60% fruit)
Probability:
100% (Image is olive)
Confidence level:
60% (Image is blicket)
We introduce a probabilistic approach based on Gaussian processes (GPs) [Rasmussen and Williams, 2006] that makes it possible to delineate different kinds of uncertainty in a model's concepts. Specifically, our goal to probe a pre-trained model by constructing a probability distribution over possible binary classifiers of concepts induced by the model. Extracting this distribution allows the probe to produce two kinds of uncertainty measures - aleatory and epistemic [Fox and Ülkümen, 2011]. This allows us to distinguish cases where 1) there is fundamental fuzziness of the label (aleatory uncertainty) or 2) it does not have enough observations (epistemic uncertainty), whereas a binary classifier would predict equal probability for both and provide no further explanatory power.
Our framework also provides natural building block for answering many downstream questions. such as detecting distriibititt (OOD) examples. Using the resulting uncertainty measures, we show that our method can correctly probe what a model represents, and that it achieves competitive performance in OOD detection. We validate this approach using both synthetic and real images.
2 Related work
Distinguishing aleatory and epistemic uncertainty has a long history in fields including cognitive science [Howell and Burnett, 1978, Fox and Ülkümen, 2011], probability theory [Jaynes, 2003], phi- losophy [Hacking, 2006] and more. Our work builds upon Fox and Ülkümen [2011], which introduced framework for distinguishing the two dimensions of uncertainty from human perspective.
In machine learning, Gaussian processes (GPs) and Bayesian neural nets (BNN) D have enabled principled ways of predicting epistemic uncertainty, which led to better information-be based decision making strategies [Houlsby al., 2011 1, Hennig and Schuler, 2012, Wang and Jegelka, 2017, Gal et al., 2017, Wang et al., 2023]. Motivated by regression problems in reinforcement learning, Depeweg al. [2018] decomposed uncertainty in BNN to conditional entropy term as aleatory uncertainty and the mutual information between model weights and predictions as epistemic uncertainty. Our decomposition simplifies these quantities' and specializes them for classification. To our knowledge, aleatory uncertainty (fuzziness of concepts) has not been studied in the GP classification literature, and no previous work in the GP literature has been done establish the correspondence between human perceptions of uncertainty and GP classification predictions.
Probing has been popular approach to understand which concepts model represents. There are mixed views of what probing should mean; either is about estimating mutual information between representations and labels [Pimentel et al., 2020] or using minimum description length (MDL) [Voita and Titov, 2020] or representing the distance as squared Euclidean distance under linear transformation [Hewitt and Manning. 2019], or using analysis with or without supervision [Saphra and Lopez, 2019, Wu et al., 2020]. Others considered diffferent types of probing: conditional probing that compares two concepts [Hewitt et al., 2021]. probing that asks where the task-relevant information "emerges" [Kunz and Kuhlmann, 2022] which concepts are "used" [Lasri et al., 2022], and finally, probing/localiznng that enables editing [Meng et al., 2022]. To the best of our knowledge. none of these probing methods offer measures of uncertainty.
We define episteme as negative entropy. which has the same form as the mutual information [ g) = H [y] H[y g] used by Depeweg et al. [2018], since equivalently, (yr 9) H[g H[g v).

---

3 GPP: Gaussian process probes
We introduce Gaussian process probes (GPP), simple probabilistic method for probing that naturally distinguishes aleatory and epistemic uncertainty. We formally define the problem of probabilistic probing $3.1, followed by review of the Beta Gaussian processes (GPs) that GPP builds upon in $3.2. In $3.3 and $3.4, we detail our method and how it measures uncertainty.
3.1 Problem formulation
Our goal is probe pre-trained model in a probabilistic manner by constructing probability distri- bution over binary classifiers based on the model's vector representation of inputs (e.g.. activations). Here, binary classifier refers to function mapping stimulus to the probability of a positive label.
Assumption 1. The model, M, contains basis functions that map from stimuli (e.g., s., images, texts) to a vector representation: :
C Rdd where X is the space of stimuli and the space of d-dimensional real-valued vector representations.
For example, basis functions & can map from images to last-layer activations of convolutional neural network (CNN). The probabilistic probe treats basis functions & as feed-forward black box, and does not require accessing the training data, gradients, or architecture used during model pre-training.
In order to define a stochastic process to describe distribution over classifiers, the data generating process is assumed as follows.
Assumption 2. There exists stochastic process G(0) parameterized by 0, and classifier h A + [o, 1] distributed uccording to 9(0), such that the label y E {0, 1} for any vector representation E A is independently distributed according to Bernoulli distribution with parameter h(a); iee. ply a, h) = (1a))((1-(())+
Now the posterior inference of the stochastic process (in our case, Beta GP) requires conditioning on observations, and the stochastic process measures uncertainty for a set queries. We denote a set of observations as D, which consists of representation-labee pairs, i.e., D = {((a;)) vn) here, label y is either 0 or 1. The queries, a [o(x;) )1,=] e [dx]|a,,, are vector representations set of stimuli whose labels are to be predicted together with uncertainty measures.
Note that there are no additional assumptions about this probing dataset, e.g., the distributions of stimuli in observations or queries. We use GPP for OOD detection, as shown later. Unlike existing probes (e.g.. [Hewitt et al., 2021, Xu et al., 2020]), we do not consider stimulus and its vector representation as random variables with some underlying distributions. Instead, viewing them as deterministic allows us to probe with any stimuli (with potentially any underlying distributions).
3.2 Background: Beta Gaussian processes
GPP uses Beta GP to define a prior distribution over classifiers. The Beta GP is a special case of Dirichlet-based GPs [Milios al., 2018], which have been shown to either outperform or achieve similar performance than classic GP classification approximations [Rasmussen and Williams, 2006]. For each representation a A, Beta GPs assume the binary label y generated by sampling Bemoulli(g(a)) where the Bernoulli parameter g(a) ~ Beta (a(a).((a) The Beta variable g(a) can be written as two independent Gamma variables:
(a) g(a) (a) + (a(aa' to (a) Gamma(a(a)) 1) and (œ(a) ~ Gamma(B(aa).. 1),
where (a) and (a) are independent. The Gamma distributions are then approximated with Log-normas distributions via moment matching. That means,
Va va), where UA = log( a(a) + 1); 2
fa (a) = log((t ~ ((loo(((aa)
(1)
and the same for fg. Beta GPs use two GPs to model the two latent functions fa and fa.
1
g = 1 T where f = Sa - So, ~ GP(u, k), GP(u, k), 11 fz.
(2)
3

---

Observations
Sampled classifiers
g ( D)
0),
PDF
0.5
01
0.4
0.0
0.5
1.0
Probe
g(a)
9(0 D)
g(a)
0.00
a1
0.3
0.53
a
(xx)
PDF
0.45
Model
0.0
00
M
0.5
1.0
g(a)
Figure 1: Illustrative example of probabilistic probing: The model of interest M transforms stimulus ar to its vector representation a ((xx) IRa. The probe, G(0 D), a distribution parameterized by 0 over possible classifiers (mapping from representations predicted probabilities) conditional on the observation set D.. Suppose D contains two datapoints. Sampled classifiers y from G D) are depicted in the middle box. The aggregation of predictions from the sampled classifiers produces the distribution for g g(a); g(a) is the predicted probability of any representation a. Notice that the two data points in D are located close each other, yet has opposite labels, indicating fuzziness in labels around this region. Imagine a new query data a. the top right-most, we see that close to D produces distribution that is centered around g(a) 0.5. If the representation is far away from D (bottom right-most), the probe can have higher epistemic uncertainty due to lack of information, and the predicted probability follows the prior distribution (in this case, the prior has modes, g(a) and one at g(a) = 1). Note that both distributions have an expectation at g(a) = 0.5, for drastically different reasons: fuzzy label v.s. lack of knowledge.
Note that the latent function f Eq. still GP given that is the sum of two independent GPs. We denote the Beta GP (0)), where (j,, k). While we focus probing as binary classification this work, is straightforward to extend GPP to multi-class classification by switching the Beta prior to Dirichlet prior [Milios et al., 2018] 8] (i.e., using more latent functions than just fa and fa),
3.3 Adapting Beta GPs for GPP
There are two important details in Beta GPs that require special attention: how to set the prior and how approximate the posterior of classifier y. This requires setting mean and kernel functions μe, in the prior GP(u, k) of the latent functions fa and fg. and computing their posterior Sax I's D.
We adapt these aspects of Beta GPs for probing settings, that also enables users incorporate the characteristics their end-tasks by setting intuitive hyperparameters. See more explanations and illustrations in the Appendix.
3.3.1 Setting the prior
First, we need to make sure the prior distribution approximated by GPs in our GPP matches with the Beta prior. Without loss of generality, we assume that for any E A, the prior distribution for g(a) Beta(e, for some probability of positive label might be bi-modal with density centered at both and (c.g., label is not fuzzy) (e < 1), or uniformly distribution in [0. (e 1), or centered at 0.5 (e > 1).
0. This hyperparameter reflects the characteristics of the task at hand; the
To match the Beta prior, the prior for both (a) and fa( has be (Vlo((c) - -,, v), where " log(? 1 + 1). To match this normal distribution, we set the mean function constant p(a) = log(e) 2, and constrain the kernel function satisfy k(a, a)

---

3.3.2 Posterior inference
This formulation offers closed-form solution for posterior inference of the GP.. Namely, any = in D {(a y: y results updating Beta parameters of g(a,) be Beta(e + s, e) and any yi results in Beta (E, E + s), for some s 2 1. This hyperparameter s describes how much "strength" (i.e., weight of the observation) we add to the Beta posterior. To match the GP posterior the Beta posterior, we synthetically add observations with heteroscedastic noise to functions fa and fa. If
y: 1, the observation for fa is (d,, log(e + 8) 5 with noise (0, v') where v' log( c+s 1), +1). And vice versa for y/s = O. The posterior of a GP remains a GP with closed-form updates to its mean and kernel function. Once we have the posterior of latent functions fa D and fa D, obtaining the posterior of classifier g is trivial by updating the GP priors Eq. 2 to the GP posteriors.
and the observation for fa is (a,, log(e) =") with noise (N e²)) where v" log(? ! + 1).
3.3.3 The cosine kernel
The choice of kernel function is important for GPP as it directly determines the characteristics of the distribution of latent functions, which translates the distribution of classifiers that GPP uses. We define a cosine kernel as:
a Ta' +1
k(a,a') (aaal + 1 [la'll+1]] where log( 1).
Note that multiplying the constant v is the constraint have from setting the prior for the Beta GP. Using the cosine kernel in the GP is equivalent to defining a distribution over linear latent functions in augmented space of the original representation space A, i.e.,
Vo
[a [
(a) = W.T (ba) + u(a), where (aa) = |a|
E RR and Wa ~ 10,
(4)
And similarly for fa with weights W3. Here μ is the mean function of the Beta GP. Note that we have the closed-form posterior distribution for the weights, which remains multivariate Gaussian distribution. See Appendix for more details.
Why is the kernel defined in this way? During the pre-training stage of deep learning models, a weighted sum of the activations and bias term are used construct neurons of the next layer. Following Wang et al. [2023], we can view each next layer neuron as independent samples of classifiers from the same distribution, described by Beta GP in this work. This means classifiers from the Beta GP should all consider a bias term. Hence we augment the activations by an additional dimension with value fixed at (Eq. 4). Additionally, we normalize the augmented activations, v(a). such that k(a, a) = e (the constraint from setting the prior in $3.2) holds for all a E A (Eq. 3).
We can interpret the cosine kernel as scaled cosine of the angle between two vectors (standard cosine similarity) in an augmented representation space described by activations and biases.
3.4 Uncertainty measures
The probabilistic probing formulation makes it possible to compute posterior distribution over labels for queries. Building on terms from Fox and Ülkümen [2011], we define uncertainty measures that identify the epistemic and aleatory uncertainty of the posterior prediction g(a) where T+e-7
1
G D), Va E A. Let the posterior of the latent function be f(a) D ~ ((uD(a))kn(a)))
Episteme : -a)], the negative entropy of the distribution of g(a), which describes the amount of knowledge the probe has about the label of stimulus a. In GPP, we have H[g(a)] = H[f(a)] - (p)()))))((() [log( + e -f(o))] and [a) = 1/ log(2nekkp(a))
Alea == Hly o(a)] El-g(a) log(g(a)) (1 g(a)) log(1 {a))), the expected entropy of the conditional distribution p(y g(a)), where is the label of stimulus a. Higher alea corresponds to more fuzziness in the label of a.
Judged probability := E[g(a)] = El ا which is the expected probability (judged by the probe) that the label of stimulus a is positive.
Figure shows the relationship of episteme, alea and judged probability for a rational agent (white region, based on [Fox and Ülkümen, 2011]).
5

---

Query
(8)
(c)
(d)
Epi: Prob: 0.96
Epistemm:: 1.3
Epistome:: 1.5
Epistume: 1,5
Judged probahilityy 0.6
Judged probability: 0.79 Alca: 0.44
Judged probability: 0.37 Alca: 0.56
Alex:
Alex:
Episteme (pciot); 0.042 Judged probability: Alear 0.44
Observations
Figure 3: How episteme, judgeed probability and alea of GPP changes as more observations are given the CoCa model [Yu et al., 2022]. As observations added, episteme increases while judged probability fluctuates. (a) With positive observation, the judged probability for the query is 0.96, showing that the query and an the observation are very close representation space. (b) With
positive (rabbit) and negative (dog) example, the judged probability lowers to 0.6, showing that the CoCa model would also put rabbits and dogs close representations, but the queried rabbit still closer to the observed rabbit. (c) With more positive example of rabbit, judged probability
increases together with episteme. (d) If we the 3rd observation (c) be negative, GPP gets more evidence that the query is probably negative and the judged probability decreases to 0.37.
Low episteme means episteme that we are "highly confident" about the underlying probability. Alea, on the other
"not sure" and high
certtiily yes"
Imationality
hand, increases with the intrinsic randomness
Alea
of the true label. The judged probability that the stimulus's 's label is positive reflects both epis- teme and alea. With low episteme, the judged probability should be closer to 0.5, meaning the probe does not have enough information to draw plausible conclusion. It would be irrational (grey areas in Figure 2) for person believe something will definitely happen even if there no evidence. Analogously, it is irrational for probe to produce probabilities close to 0 or while having low episteme. By accumulating evidence (increasing episteme), the judged prob- ability can move closer and closer to the ground truth (moving to the right in the figure). An accurate probe should produce judged probability that matches the ground truth fuzziness of label only if episteme is high.
Judged probabiiity
0.5
"purely mindon"
Alea
Irrationality
"cortainly
"not sure"
Episteme
"highly confideer"
Figure 2: Interaction of alea and episteme (recre- ation of Figure from Fox and Ülkümen [2011]).
Figure 3 is real-world example (using a CoCa model [Yu et al., 2022] and natural images, shown in the figure, as query and observations) how our measures of uncertainty change we increase small number of positive and negative observations.
Table presents the results of applying GPP to the concepts involving olives introduced in $1. The judged probabilities from GPP match human judgments [McCloskey and Glucksberg, 1978]: olives have 57% chance of being fruit with high level of confidence, and GPP is very uncertain about whether olives on tree are blickets.
Table 1: GPP predictions for olives in $1.
Concept
Judged prob.
Episteme
Alea
Fruit
57%
2.3
0.63
Blicket
91%
0.49
0.22
Validating GPP
In addition to probing concepts, GPP predicts aleatory and epistemic uncertainty, which allows us to determine the fuzziness of concept detect out-of-domain (OOD) stimuli.

---

We use 3D Shapes [Burgess and Kim, 2018] and construct datasets based on its concept ontology validate these uses of uncertainty measures. In these 3 settings, we can control the ground truth level of fuzziness by artificially injecting noise to labels ($4.2, $4.3). In $4.5, we evaluate the OOD performance of GPP and competitive baselines on 3D Shapes as well as 10 datasets defined by coarse-grained labels of the ImageNet dataset [Russakovsky et al., 2015].
4.1 Experiment setups for probing
We evaluate GPP's ability to probe using the 3D Shapes dataset [Burgess and Kim, 2018], which contains images generated from 6 ground truth independent primitives: 10 floor colors, 10 wall colors, 10 object colors, 8 scales, 4 shapes and 15 orientations of the shapes, as shown below.
Datasets and models. We construct 2-level concept ontology (see Appendix for visualization) consisting a disjunctive level where colors are binarized to warm/cool and scale to small/large, and conjunctive level that combines shapes, binarized colors and binarized scales.
We train three CNN models on labels generated from this ontology: 1) M.11 is trained on 64 labels (binarized scale, floor, wall object color x 4 shapes), 2) M.2 is trained on 8 labels (4 shapes binarized scale) 3) M.3 trained on labels (binary floor, wall and object color).
We define two probing tasks to investigate each model: 1) task P.1: binarized colors (floor, wall, object) and 2) task P. binarized scales and shapes.
Baseline descriptions. The baselines include 1) LP: linear probes [Alain and Bengio, 2016]; 2) SVM: SVM probes (a baseline from Kim et al. [2018]); 3) LPE: linear probes ensembled via bootstrap [Kim et al., 2018]. For each linear probe LPE, we train a logistic regression classifier on dataset sampled from the original set of observations with replacement. The size of the dataset is the same as the number of observations. Each ensemble has 100 linear probes. LPE can also describe distribution over classifiers and be evaluated probabilistic probes, making this strong baseline. For GPP, we use B0ta(0.1, 0.1) as the prior and set strength to 5.
4.2 Validating GPP probe
To compare with baseline methods, we use AUROC to evaluate the deterministic classifier defined by the judged probability E[g[a)], Va E a, (defined in $3.4). Queries are sampled disjointly from the training data and observations. Figure shows that GPP performs competitively against LP and SVM. GPP and LP perform comparable with respect to AUROC and sample efficiency (needing as few 10 observations for probing concepts in task P.1).
One may first guess that M.3 is the best model that represents color, since it was explicitly tasked to distinguish color. However, our probing results reveal what is obvious in hindsight: while M.2 is only tasked to distinguish shapes and scales, separating where the shapes are requires detecting color (i.e., the only cue to separate shape from background is color). This is shown in M.2's high AUROC for P.1 in Figure M.3 is only tasked distinguish colors, so performance on P.2 remains low.
4.3 Estimating concept fuzziness
What happens when the concepts we are trying to probe are inherently fuzzy? This is commonly the case in the real world, where situations and the language used describe them contain extensive ambiguity. We again focus on M.1 and concepts in task P.1. We control the fuzziness of concepts in P.1 by randomly flipping positive labels to negative (but not vice versa) ground truth label probabilities of 0.25, 0.5, 0.75 and 1, where I indicates GPP's alignment with ground truth label probabilities, we use the Pearson correlation coefficient between ground truth label probabilities and judged probabilities for all queries.
observations. We test fuzziness. To measure
7

---

Model M.1
Model M.2
Model M.3
LP
0.9
SVM
GPP
AUROC
0.8
0.8
0.8
Target taskss
0.7
0.7
0.6
0.5
0.5
50 100
50
100
50
100
Ohservations
Figure 4: AUROC results for probing M.., M.2 and M.3 with P.1 and P.2 tasks. M.1 and M.. are trained on color-related tasks and all probes obtain better performance on color concepts P.1 than geometry concepts P.2. The probing results reveal something obvious in hindsight; while M. is trained ecometry-relatee tasks, it needed color to separate the object from the background color in order to detect geometry, resulting in significant representation color concepts, P.1.
LPE
CPP
Pearzon Coef..
0.8
Tudged probesbii biiliy
Judged probebiliit
Observations
0.6
2
0.6
8
0.4-
0.4
0.4
Probe
0.2
128
GPP
0.0
50
100
10-3
10-1
10-1
100
Observalions
Epistemze
Epistemee
Figure 5: Left: Pearson correlation coefficient between ground truth label probabilities and judged probabilities. GPP obtains much higher correlation than LPE, showing that GPP is better at detecting fuzzy concepts. Middle and Right: Relationship between judged probability and episteme of positive labels using LPE and GPP when ground truth label probability is 0.5. LPE fails to detect fuzziness and assigns extreme judged probabilities observations, LPE has higher episteme than with more observations. This behavior is irrational [Fox and Ülkümen, 2011]: 1]: fewer observations should indicate lower episteme, and with low episteme high judged probability indicates LPE is being confidently ignorant (no knowledge but believes something will definitely happen). On the contrary, GPP shows rational behavior: with high episteme, the mass of GPP's judged probability predictions centers at 0.5, which aligns with the ground truth.
queries for or more observations. With
Ground truth prob,
GPP
Ground truth prob,
Ground truth prob.
0.6
0.6
Observations
0.5
Alea
0.4
0.4
16
0.3
32
0.2-
0.2
0.0
10-2
10-1
10
10-2
10F
10-1
10F
Episteme
Episteme
Episteme
Figure 6: Alea predictions as a function of episteme for LPE and GPP. When ground truth label probability is 0.5 (Left and Middle), LPE predicts low alea for observations larger than 2, which does not agree with the ground truth alea (0.69). GPP shows much more rational predictions. Comparing Middle and Right, GPP predicts higher alea with more observations for ground truth label probability 0.5 (the corresponding alea is 0.69), and its alea predictions converge to 0 (ground truth alea is 0) for ground truth label probability 1.

---

Figure 5 shows that GPP can accurately probe fuzzy concepts by predicting both judged probability (expected level of fuzziness) and episteme (how sure it is about the judged probability). GPP achieves higher Pearson correlation coefficient than LPE across all sizes of observations. In particular, the relationship between judged probabilities and episteme coincides with the two-dimensional framework for characterizing uncertainty from Fox and Ülkümen [2011] (see Fig. 2), showing the validity of GPP as a rational method to probe fuzziness.
Figure 5 (Right) confirms that the judged probability alone cannot be used to distinguish whether concept fuzzy or the stimuli are far away: in both cases, judged probabilities can be 0.5. Hence, it is important to remember that judged probability is only accurate with enough observations. We illustrate this point from another angle in Figure 6, which shows the relation between alea and episteme predicted by LPE, GPP with ground truth label probability 0.5 or 1. When ground truth label probability is 0.5, GPP can have low alea under low episteme (same effect as Figure 1), but with more episteme, GPP puts more mass of predictions for high alea because the ground truth alea is high. LPE does not show this kind of rational behaviors. When ground truth label probability 1, more mass of alea predicted by GPP converges to 0 as episteme increases, since the ground truth alea is O.
4.4 Experiment setups for OOD detection
When learning new concepts, it is natural for people to be able to say "I'm not sure since haven't 't learned it yet". For probes, the corresponding ability to perform out-of-distribution (OOD) detection, which identifies datapoints that are different from the observed training data. GPP naturally performs OOD detection using episteme predictions. With an uninformative prior for the labels, high episteme indicates in-domain (ID), while low episteme indicates OOD.
Method descriptions. For GPP, we use the negative latent variance of posterior function f (Eq. 2) as a proxy of episteme for computational efficiency. It is easy to show that the entropy of classifier g is upper bounded by the entropy of latent function f. which is fully determined by its posterior variance (more details in Appendix). In practice, we found that using the negative latent variance as OOD detection scores for GPP is robust to the level of informativeness of the Beta prior, because the latent GPs are agnostic with respect to the logistic transformation in Eq. 2.
As for baselines, we compare against MSP (maximum predicted softmax probabilities using LP) [Hendrycks and Gimpel, 2016], Maha (negative Mahalanobis distance-based score) [Lee et al., 2018], NN (deep nearest neighbors) [Sun et al., 2022] and LPE. Recall that LPE uses bootstrapping to create an ensemble of linear probes. Each linear probe in LPE can be viewed as i.i.d. samples from an underlying distribution over classifiers (i.e., 9 in Figure 1). Similar to approximating metries with samples in $3.4, we can estimate the negative variance of predictions from the set of linear probes, and use it as the OOD detection score for LPE.
As shown in the extensive analyses of OOD detection methods and tasks in Appendix E of Tran et al. [2022], Maha and LPE (LPE is equivalent to their "Entropy" method for ensembles) achieved top performance, surpassing more recent proposals from Ren et al. [2021].
Datasets and models. We generate a set of queries from task P.1 1 of 3D Shapes that include 1024 ID images and 1024 OOD images (uniform random noise), and see if OOD queries can be detected using the methods above based on representations from model M.1.
We also evaluate the OOD detection performance of all methods on real world datasets and models. Queries and observations in the real world datasets are sampled disjointly from the validation split of the ImageNet dataset [Russakovsky et al., 2015]. We make 10 sets of Ds using 10 binary classification tasks defined by supersets of ImageNet classes. Query points consist of 128 images from the classes the probe has observed and 128 OOD images (uniform random noise). The ID examples are images that come from the same distribution that the probe observes. For example, consider a probe trained to classify "all dogs" vs "all cats": images of dogs and cats would be ID, whereas random noise would be OOD. The models we used are the pre-trained ResNet-50 [He et al., 2016] and CoCa-Base (Yu et al., 2022]. See more details in Appendix.
'For example, 10,1, 0.1) is a more informative prior than Beta(1, 1). because the former means the classifier outputs either 0 or I with high probability, while the latter puts uniform distribution over any real numbers in [0, 1]. See Figure 9 for visualization of the PDF of Beta distributions.

---

M.1
ResNet-S0 50
CoCa Base
1.0
1.0
1.0
MSP
AUROC (D)/ODD)
0.8
AUROC (IDOOOD)
0.8
NN
LPE
0.4
04
0.4
0.2
0.2
0.2
0.0
32
Observations
Observations
Observations
Figure 7: AUROC for ID/OOD detection with an increasing number of observations for task P. of 3D Shapes (Left) and real- world datasets constructed from ImageNet (Middle and Right), showing GPP's comparable performance and data efficiency in OOD detection.
4.5 Results for OOD detection
Figure shows the competitive performance of GPP an OOD detector for all the OOD detection tasks we constructed. While GPP might not be the best OOD detection approach for each of the tasks, GPP performed consistently well and par with the best benchmark method for each task. This means, while GPP is mainly probing method, it is also competitive for OOD detection.
Note that the effective term in GPP's negative latent posterior variance, k(a, a.) K-)))- a) (here a, is the concatenated observed activations and K k( a.) E RR Oxxx)), looks very similar to Mahalanobis distance, (a ú.)) 1-1 (a- -e) (Eq. 2 from Lee et al. [2018]. where E Rdxd, is the dimension of activations). However, GPP's variance measures the similarity between function values but Mahalanobis distance measures the similarity between activations. Also note that the Gaussian estimated for Mahalanobis distance is degenerate for d > O, and the distance is still computed on a O-dimensional manifold; this is possibly why the Maha method shows inferior performance.
Conclusion
We introduced GPP, an uncertainty-ave -aware probing framework that uses Gaussian processes to measure both the aleatory uncertainty intrinsic to the concept represented by model and the epistemic uncertainty reflecting the probe's 's confidence about its judgement. These uncertainty measurements also equip GPP with the ability to detect OOD stimuli. Empirically, we verified the strong performance of GPP for data-efficient probing that can correctly surface fuzziness in concepts and OOD detection which shows GPP can also surface epistemic uncertainty (i.e., knows that it does not know). These results illustrate how distinguishing between different forms of uncertainty can be beneficial both for deepening theoretical understanding of models and for practical applications.
Limitations. We have validated our approach on both synthetic and real-world datasets, and demonstrated its use for OOD detection, but there are much wider range of models, datasets, and applications that can be explored. Fully evaluating the potential of this approach and achieving the greatest impact will require pursuing this investigation, which we hope to particular, we plan to explore active few-shot learning and distributionally robust probing. Finally, GPP can only used with models for which activations are made available, which may present an obstacle to use with certain commercial machine learning systems.
future work. In
Broader impact. Being able to understand what models can and cannot represent has huge implica- tions. Not only does it impact important downstream tasks, such as OOD detection, but it ultimately aids human-machin communication and collaboration. While (potentially) superficial interaction (e.g., with a chatbot) is already possible, we also know models hallucinate [Ji et al., 2023], leaving people unsure about the validity/faithfulness of this interaction. Work such as ours ultimately aims to help humans to better use and develop models that aligns well with our concepts, as well as potentially learning new concepts from machines.
10

---

Acknowledgments and Disclosure of Funding
We thank Fei Sha for constructive discussions on aleatory uncertainty in classification. We also thank Geoff Pleiss, Alexander Terenin, Ben Adlam, Jasper Snoek and Zackary Nado for helpful conversations on Gaussian process classification. Images of "blicket" from http:::/www.wwwwtt/
com/ httpe////mmm/e///eeeeeeeeeeeeeee - 10. and https: /ttbbr:s/libbbshttt com http:/p-oooooot http://oooottttttppaaaa//////111 green-olives olives-an
jpg (by Martin Vorel).
References
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprini arXiv: M.K::v6610.144: 2016.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning (ICML), 2018.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational
Linguiittcss, 48(1):207-219, 2022.
Michael McCloskey and Sam Glucksberg. Natural categories: Well defined or fuzzy sets? Memory & Cognition, 6(4):462-472, 1978.
Fei Xu and Joshua B Tenenbaum. Word learning as Bayesian inference. Psychological Review, 114 (2):245, 2007.
Carl Edward Rasmussen and Christopher KI Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
Craig R Fox and Gülden Ülkümen. Distinguishing two dimensions of uncertainty. Fox, Craig R. and Güilden Ülkümen (2 "Distinguishing Two Dimensions of Uncertainty," in Essays in Judgment and Decision Making, Brun, W. Kirkebøen, G. and Montgomery, H., eds. Oslo: Universitesforlaaet, 2011.
William Howell and Sarah A Burnett. Uncertainty measurement: cognitive taxonomy. Organiza- tional Behavior and Human Performance, 22(1):45-68, 1978.
E. T. Jaynes. Probability Theory: The Logic of Science. Cambridge university press, 2003.
Ian Hacking. The Emergence of Probability: A Philosophical Study of Early Ideas about Probability. Induction and Statistical Inference. Cambridge University Press, 2006.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 111.5744, 2011.
Philipp Hennig and Christian J Schuler. Entropy search for information-efficciet global optimization. Journal of Machine Learning Research (JMLR), 13:1809-1888 18399 2012.
Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient Bayesian optimization. In International Conference on Machine Learning (ICML), 2017.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In International Conference on Machine Learning (ICMI), 2017.
Zi Wang, George F. Dahl, Kevin Swersky, Chansoo Lee, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-trained Gaussian processes for Bayesian optimization. arXiv preprint arXiv:2109.08211 2023.
Stefan Depeweg, Mose-Miguel Hernandez-Lobato, Finale Doshi- and Steffen Udluft. Decom- position of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In International Conference Machine Learning (ICML), 2018.

---

Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell.
probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609-4622 Online, July 2020. Association for Computational Linguistics. doi: 10. 1..11866 10.18665/112002.aclmmillll 420. URL. https: / aalaaccanttt anncllnnhhllgyy wwtpooggg........o acl-0m...mmmaiii. 420.
Elena Voita and Ivan Titov. Information-theorer probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1 183-196, Online, November 2020. Association for Computational Linguis- tics. doi: 10.18653/v 1/ wttp:e0.emmlpmmaai 14. URL https: httppss//aaaaattt http:///caaantootooo/oo org/2020. emnlp-ma i in. 14.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Humun Language Technologies, Volume / (Long and Short Papers), pages 1129-4138, Minneapolis, Minnesota, June 2019, Association for Computational Linguisties. doi: 18.186 1419, URL https: / acl aclanthology org/ M1N-N19111
Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Humun Language Technologies, Volume (Long and Short Papers), pages 3257-3267, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. URL https:/ / ac antho wow..ooooo.....o... org/N1 N1 9329.
doi: 10. 10.18653/11/N19-133
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. Perturbed masking: Parameter-free free probing for analyzing and interpreting BERT. In Proceedings of the 58th Annual Meeting the Association for Computational Linguistics. pages 3166-41776 Online, July 2020. Association for Computational Linguistics. doi: 18.18653/v 1/2020. acl-main.383. URL https: http:::s//aaaaaatooooooooo org/ 1020.acl--o 383.
John Hewitt, Kawin Ethayarajh, Percy Liang, and Christopher D Manning. Conditional probing: measuring usable information beyond baseline. In International Conference Learning Representations (ICLR), 2021.
Jenny Kunz and Marco Kuhlmann. Where does linguistic information emerge in neural language models? measuring gains and contributions across layers. In International Conference on Compu- tational Linguistics, 2022.
Karim Lasri, Tiago Pimentel, Alessandro Lenci, T. Poibeau, and Ryan Cotterell. Probing for the usage of grammatical number. In Annual Meeting of the Association for Computational Linguistics, 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable infor- mation under computational constraints. In International Conference on Learning Representations (ICLR), 2020.
Dimitrios Milios, Raffaello Camoriano, Pietro Michiardi, Lorenzo Rosasco, and Maurizio Filippone. Dirichlet- based Gaussian processes for large-scale calibrated classification. In Advances in Neural Information Processing Systems (NeurIPS). 2018.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. arXiv preprint arX1v:2205.019117 2022.
Chris Burgess and Hyunjik Kim. 3D Shapes dataset. https//giiibbbccocccccmm//////ss 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015.
12

---

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of- examples in neural networks. In International Conference on Machine Learning (ICML), 2016.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference Machine Learning (ICML), 2022.
Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang. Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXiv:2207.0744 2022.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshmi- narayanan, A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2/06.09022, 2021.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
George A. Miller. WordNet: A lexical database for English. In Human Language Technology: Proceedings of Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994. URL www.oogggooo.....o.............. org//94 11111994-1111...
pettss//aclaar wog.olooo.....oo
13

---

Table Notations for GPP.
Terminology
Symbol
Meaning
Model
M
The model probe.
Stimuli space
The space stimuli.
Representation space
A
C Rd
The space vector representations of M. The function (contained in .M) mapping from X' to A.
Basis functions
Vector representation
((x)
The vector representation of a stimulus E X.
a
a : + R mapping to the 1st parameter Beta distribution. A + R+ mapping to the 2nd parameter of Beta distribution.
8
ta
A random function such that t, (a) Gamma (a(a), 1).
to
A random function such that ta((a)
(aamma(3(a)) 1).
Beta distribution
(eta(a(a),B(a))
The Beta distribution for g(a), Va E A.
Mean function
#: A R.
Kernel function
k A x A + IR
GP
GP(M,)) fo GP(u,k) f3 ~ GP(M,))
A GP with mean function μ and kernel function k.
fo A+ the (latent) function for approximating ta.
f& -> IR, the (latent) function for approximating tg. Parameter for the Beta GP GPP, (μ, k).
Beta GP
9(0)
distribution over functions mapping from A to [0, 1].
Classifier
~ 9(0)
, A + [0, 1J. a random function.
Observations
D {(((x;); x)) ))) x, X, y E (0, 1).
Beta GP posterior
(( D)
The Beta GP conditional on observations D.
Queries
ag
ag = (6(x;)),,= Rddlaal,,
Predicted probabilities
g(ag)
g(a,) [n(a))asa,
A Notation
In Table 2, we include the main notation used in this paper.
B Details on the Beta GP in GPP
As shown in $ 3.2 and $3.3, with observations D ((a 11) the posterior of a Beta GP can be written as the transformed version of a GP, i.c.,
g = - ~ 9(0 D), where f gP(up,kp).
We show how to obtain the mean and kernel functions μpp kD in 1,.1, as well as the posterior of weights in $B.2 for the cosine kernel. The behavior of GPP relies on two hyperparameters, e and and explain what they are and how them in $B.3. In $B.4, analyze and bound the episteme of GPP.
As reference, Figure shows the graphical model of GPP.
14

---

Sa
SB
y
1 h
N
Figure 8: Left: Graphical model of the Beta GP in GPP with observations. Right: The stochastic process that a Beta GP approximates, where h
to +tp
Beta(e, e), to Gamma(e, 1),
Gamma(e, 1), and ta 4g- Beta GP, g etu + approximates log tg. Hyperparameter is the weight for each observation.
where Sa approximates log la. and fB
B.1 Posterior inference (extension of $3.3.2)
Without loss of generality, write observations union of dataset (of size n) with the positive labels only and a dataset (of size [D] n) with negative labels only, i..., D = {(a,, y:) = D+ U D where D {(a;, y/ ) and D {ar, y:)
For convenience, we use the following short-hand notation:
1 1 6+8
1), v" oo=1 +1) logl + 2 5/ =og(e v"
=' logg
Observing a positive example is equivalent to observing y' with noise (0, v') on fa and observing y" with noise (0, v") on fg. Vice versa for observing negative example. We also denote 1m column vector of size m filled with 1s, and In as an identity matrix of size m.
Recall that So GP(u, k), Sa GP(u, k) and Sa IL Sg. The posterior for sa is Sa D GP(Ha,ka) ka). For any a, a' A.
Ha(a) u(a) + k(a, a) KT (Ya p(a)), ka(a,a') = k(a, a') - k(a, a)KK "(a,a') (5)
where
k(a, = [h(a, 121 E EiIID! k(a.a') = [a(aa, a 11) E RlDIi
u(a)= = [u(a;)][21 Dix1 K [M(a; az) ( R DIi|D]
and
ya
y'
E ERIDII
K. = K +
N'In
E [aliiaaa
1 1 -n
Similarly for fg D ~ (P(usikp) we have
Ms(a) u(a) k(a, K5 ( p(a)), kola,d') k(a, k(a, ala k(a,a') (6)
where
I' / p|-
In K3= K
E RDIx
E R Dillaa
U3 =
0
v' J'In -n_
Since f = Sa - fs, by combining Eq. and Eq. 6, we have ʃ D ~ GP(μD, kD); and
pn (a) = (a) ms(a)) = k(a, a) [K (y, ((aa)) - Kg 2 (ya - u(a))).
kp(a, a') = ki (a, a') + ka(a, a') 2k(a, a') - k(a, a) (K= + KT¹) k(a,a') Thus, we obtain the closed-form posterior for function f-
(7)
For classifier g(a) Fia- we can then get its PDF as follows,
Pa(a) (v) 2(1 4) /VTkkp(aaa) exp (log(y) log(1 v) =DD())))) 2kp(a,
(8)
15

---

B.2 Posterior inference for weights (extension of $3.3.3)
we use the cosine kernel in $3.3.3, the posterior of fa can be written as fa(a) W. (b + u(a), where Wa D Nu Ea), Wa R2+11 This means (a) N (u., (a) + pla), (aa) (a)(.
Because of Eq. 5 and Eq. 4, we can also write the posterior of fa as
(a(a) = u(a) + v(a) (a) (Ya - p(a)): (a = (aa) ((a)
((a) (b)))
By comparing the above two ways of writing the posterior of far obtain
No Ka (yo u(a)), E. lat1 (ba) (())",
Similarly, for fa(a) W3 (0a)) m(a), Ws D Nw,, Eg), have
ug = ((la)55100 (y3 m(a)); Σg = Ta+1 "(a) ((a)
Then, for f = Ja ʃ3 W (a)) ()) W D (M, E), we have
vbaa (ya m(a)) 1 (ys m{aa)) Σ 2Ta+1 va(a) (Ka¹ + k5z) (a)
This means we can directly sample classifiers from a Beta GP with a cosine kernel by sampling weights W from multivariate Gaussian distribution defined above.
B.3 How to set hyperparameters
There are two hyperparameters in GPP with the cosine kernel: c, which determines the prior, and s, which determines the posterior.
For any a E A. the prior on the probability that the label positive is Beta(r, c). As noted in $3.3.1, < 1 reflects belief that g(a) should be close to either 0 or 1; gives uniform distribution over (0, 1]; and > reflects a belief that g(a) is centered 0.5. In the Beta GP, the Beta prior is approximated as
(log(t) log(1 - y))²
PV(n)(U)=
exp
(9)
y(1 - v) 4T log( +1
4 4oog( + 11
Eq. 9 is obtained using the prior of i.e., (a) 0, kn 2k(a, a) 2 log( Users can choose Beta(e, é) for moment matching in Eq. to get a better approximation of (e)
_ + + in Eq. 8.
Figure 9 shows both the PDF of Beta priors and the approximations.
For setting the hyperparameter s, users can also directly inspect the behaviors of different values of and choose an appropriate value. Figure 10 and Figure 11 show how the posterior changes with one negative or two opposite- labell observations. Larger s leads more concentrated posterior.
Since all of these distributions are easily computable and can be visualized clearly, users can directly inspect the behaviors of these different hyperparameters and choose suitable option.
B.4 Analyses of episteme (extension of $3.4)
For each a E A, episteme the negative of H[g(a)]. By Eq. 8, we have
H[[((a)]
Po(a) (3) log log))(()))) (v) dy
-Ejlog ()(a)())
Eflog (y(1 - y))l H[/(a)
< H[).
16

---

-= 0.1,
= 1e-06
E=
0.5
E=1.5,
8=1
1.25
1.5-
Approx with
1.00
Apprex with
1.0-
PDP
0.75
0.50
Beta
0.5-
Beta
0.25
Approx with
Approx with E
Approx with
Approx with 8
0.00
0.0-
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0,00
0.25
0.50 g(a)
0.75
1.00
g(a)
g(a)
Figure 9: PDF of Betaaee, e) and the approximations that either for moment matching é. Because both the PDF Beta distributions and the approximations in Eq. 9 are easily computable, users can inspect the distributions directly and choose the right é match with their own beliefs.
8 1, 2=
5=1 1
E 1,
0.5,
s 5 E = 1, 0.5, = 100
12.5
Beta
Beta
200
10.0
Approx with
Approx with
7.5
Approx with ė
150
Approx with E
PDF
100
1-
Beta
Approx with e
2.5
50
Approx with
0.0
10-3
10-1
10-3
10-1
10-3
10-1
g(a)
gla)
gla)
Figure 10: PDF of Beta(e, + 8) and the approximates that either uses for moment matching or These distributions are the (approximated) posteriors of g(a) after observing 1 negative example. Hyperparameter are 1 (Left), 5 (Middle) or 100 (Right), and with a larger the approximate becomes more concentrated at a lower value of g(a).
E= 0.5. 5=1
E= 8 5=5
E
8=
100
10.0
Beta
Beta
PDF
1.0
Approx with
Approx with Approx with
5.0
Approx with
Beta
Approx with
2.5
Approx with
0.0
0.0
0.00
0.25
0.50 g(a)
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
g(a)
g(a)
Figure 11: The same setup as Figure 10, except that the observations include 1 positive and negative examples at the same representation a. A larger results in PDF that more concentrated at g(a) 0.5.
17

---

small warm cube, warm wall, warm floor
large sphere, warm wall, cool floor
large cool
Conjunctive level
cylinder, cool wall, warm floor
Disjunctive level
object is
object color is warm
wall color is warm
wall color is warm
small
Primitives
object is capsule
object is cube
object is XXL
object is XS
object is XXS
object is magenta
wall magenta object is red object is orange
wall red wall orange
floor magenta
floor red floor orange
Figure 12: Ontology of training labels for the 3D Shapes dataset [Burgess and Kim, 2018).
The last inequality is because y(1 v) 1, i.e., log (y(1 v))
In 13.3.1, we set constraint on the kernel Is such that k:a,, a) = =log(? + 1). By Eq. 7, the entropy of f(a) can be bounded as follows.
1 H[)(a)] log((oeek a)) (og(4Telog(= 1))-
1
Hence there exists a lower bound on episteme for GPP. However, H[((a)] can approach -oo because (1) variable can infinitely close or and (2) kip (a, a) can also infinitely close which means episteme has finite upper bound.
In natural language, our analyses of episteme show that ignorance has limit, but knowledge has no limit. This is widely recognized idea, and it is also reflected in the words of Zhuangzi, a Chinese philosopher from the 4th century BCE: "Your life has a limit, but knowledge has none."
C Experiment details
In this section, we include details on experiment setups and additional results.
C.1 3D Shapes ontology
The ontology of training labels for the 3D Shapes dataset [Burgess and Kim, 2018] is illustrated in Figure 12. Images are generated from 6 ground truth independent primitives: 10 floor colors, 10 wall colors, 10 object colors, 8 scales, 4 shapes and 15 orientations of the shapes (orientation is excluded from the ontology since it's only distinguishable for cubes). The disjunctive level of the ontology groups together ranges of color and shape primitives into binary concepts: warm/cool and small/large, respectively. Concepts in the conjunctive level are the Cartesian product of concepts in the disjunctive level and the shape primitives.
C.2 Real-world OOD detection
In-distribution (ID) queries are sampled disjointly from the validation split of the ImageNet dataset [Russakovsky et al., 2015], where the probe observes 10 sets of Ds using 10 binary classification
18

---

tasks defined by ImageNet superclasses. These superclasses are defined by building a tree using the WordNet hierarchy [Miller, 1994] where the leaves of this tree are ImageNet classes (e.g.. the superclass "dog" contains Chihuahua, Japanese Spaniel, Maltese, etc.). The ten classification tasks we use are: (1) dog vs snake, (2) fish vs lizard, (3) bird vs snake, (4) dog vs bird, (5) cat vs bird, (6) fish vs snake, (7) bird vs fish, (8) snake vs lizard, (9) cat vs dog, (10) bird vs lizard.
cut-of-distribution (OOD) images are generated with pixel-wise uniform random noise. This noise is passed through the basis function & to construct the OOD query.
C.3 Relations between judged probability, episteme and alea
In this section, we present more empirical results. The experiment setting is the same as $4.3.
First, we evaluate how judged probability and alea change as episteme increases, and how judged probability changes as alea increases. Figure 13 and Figure 14 show the results for GPP and LPE respectively. Please use a higher quality PDF viewer the points don't show up in the figures. Each row corresponds to a different ground truth probability, which means the probability that an originally positive stimulus remains to have positive labels in observations, when we manually inject fuzziness to concepts in the 3D Shapes dataset [Burgess and Kim, 2018]. So in the ideal case, judged probability predictions should converge to the ground truth probability for stimuli that are originally positive. stimulus that is originally positive.
Each scattered point corresponds to the predictions
GPP consistently produces rational uncertainty measures. There are no extreme judged probability predictions with low episteme. Alea converges to low values for 1.0 ground truth probability and higher values when the ground truth probability 0.25 or 0.75, and alea converges to the highest values when the ground truth probability is 0.5. These are all expected since with 0.5 ground truth probability, the level fuzziness reaches the highest.
On the contrary, LPE tends to have more extreme predictions on judged probability no matter what the ground truth probability is. However, the average judged probability LPE does get affected by the ground truth probability. For example, when the ground truth probability is 0.25, more masses of judged probability accumulate between 0 to 0.2. This means the average judged probability can be close to 0.25. Similarly, when the ground truth probability is 0.5, about half of the predictions of judged probability are between 0.8 to 1 1.0, and the other half are between 0.0 to 0.2. While this ensures the judged probability is close to ground truth probability if we average over all stimuli, the individual predictions cannot be used to evaluate the fuzziness of concepts.
Figure 15 shows AUROC, AUPRC and accuracy of GPP and LPE for different numbers of observa- tions. These metrics averaged over both positive and negative queries. Interestingly, even when the ground truth probability is 0.25 (only 1/4 of the positive examples remain positive), GPP can still achieve almost 1.0 AUROC and AUPRC. As expected, the accuracy of GPP is about 0.5 for 0.25 ground truth probability (since the judged probability predictions are mostly lower than 0.5 for the positive examples, and the negative examples are almost all correct). But because LPE does not always have low judged probability predictions even if ground truth probability is 0.25, its accuracy higher than 0.5.
These results confirm the rationality and good performance of GPP as a valid probing method.
19

---

16
32
128
GPP Ground truth prob. = 1.0
GPP Ground truth proh 1.0
GPP/ Ground truth prob. 1.0
0.6
Jadged gimbabniity
0.5
probabillty
044
0.4
02
10
10-1
10-2
Hpisteme
Episteme
GPP Ground Iruth prob. 0.75
GPP Ground truth prob. 0.75
CPP Ground Iruth prob. 0.75
1.0
0.7
1.0
0.8
0.8
Julged prohabitiny
Jadged probabiltt
0.6-
0.6
0.4
0.4
10
10a
10
10"
2X 10
10 10
6x 10-
lpisteme
Episteme
Alea
GPP Ground truth prob.
0.5
GPP Ground truth prob. 0,5
GPP Ground truth prob. 0.5
0.8
0.6
0,8
juitged grobibiiity
0.6
0.5
Judgeddprii probabiliit
0.6
0.4
0.4
0.2
0.2
0.1
0.0
0.0
Epistame
GPP Ground truth prob. 0.25
GPP Ground truth prob. 0.25
GPP Ground Iruth prob. = 0.25
0.7
0.6
jodged probubiity
0.6-
0.5
jodged probabiitty
0.6
0.4
04
0.3
0.
0.2
0.2
02
0.1
109
10
10
Eplateme
Epistame
Alsa
Figure 13: Relationships between judged probability, episteme and alea using GPP.
20

---

Obsorvations
LPE Ground truth
LPE Ground truth prob.
LPE Ground truth
1.0
1.0
0.7
1.0
Jolged prshabiitt
0.5
Judged probbiiityy
0.6-
0.4-
0.6
0.3
0.2
0.2
0.2
0.0
10
10"H
10
10
107
10-3
10
10-1
10P
10
10
10
10
10
10
Epistens
Epistame
Alea
truth prob.
Ground truth
LPE Ground prob.
0.74
0.8-
0.8-
Judgee probabiliy
0.5
Judgee grobabiliy
0.6
0.4
0.6
0.2
0.2
0.2
0.1
0.0
10
10
10
10a
107
10
10
10
10"
10
10
10
10
10
10
10a
Opisteme
Epistome
Alea
Ground prob.
Ground prob.
Graund prob.
0.7
0.8
0.8
julged prshahiitt
0.5
Judged prbbaiiity
0.6
0.4
0.6
0.2
0.2
0.2
0.1
0.0
10-4
10-3
10-2
16
10c
10"
10-4
10-3
10-2
10-1
10F
10"
10-4
10
10
10-1
16'
Epistems
Episteme
Alea
LPE Ground truth prob. 0.25
Ground truth prob.
0.25
LPE Ground truth prob. 0.25
0.7
0.8
0.8
0.8
Judgeed pribebiiity
Judged grobebllttty
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.1
10.22
10
10P
10
10
10
10:7
10"
10
10
10
10"
10
100
Egistecse
Epistame
Alea
Figure 14: Relationships between judged probability, episteme and alea using LPE.
21

---

Ground truth prob. 1.0
Ground truth prob. 1.0
Ground truth prob. 1.0
0.975
0.950
0.950
0.923
0.325
0.900
0.850
Probe
0.850
Probe
Probe
0.825
QPF
022
GPP
0.800
0.825
LPE
80
100
100
Ground teath prob. 0.75
Ground Irath prob. 0.75
Ground Irulh probh 0.75
Prode
0.975
GPP
0.950
0.950
0.9 0.925
0.900
0.900
0.875
0.800
Probe
Probe
GPP
639
0.800
LPE
LPE
12o
Obsorvations
Observations
Ground prob.
Ground truth prob.
Ground truth prob.
0.5
1.00
0.85
Probe
02F
LP2
0.95
0.55
0.75
0.90
ALPRC
0.30
Araurayy
0.85
0.65
0.80
0.80
GPP
0.75
LPE
LPE
0.55
20
40
00
80
100
1200
20
40
100
120
20
40
600
80
100
120
Observations
Observatins
Observatinss
Ground prob.
Ground
Ground prob.
0.25
1.00
1.00
0.95
0.50
0.80
LPE
0.75
0.90
AU1922
0.70
0.65
Probe
0.55
0.50
20
40
60
BD
100
120
0
20
40
60
80
100
120
20
40
60
80
300
120
Observattons
Observatioss
Obsecvations
Figure 15: AUROC, AUPRC and accuracy of GPP and LPE using different numbers of observations.
22


Reviews:
[{"review_id": "pwY1KZyExh", "reviewer": "Reviewer_5Gmc", "summary": "The authors introduce Gaussian process probes, a probabilistic probing method. They use this method to obtain additional insights into the internals of deep learning models, using the concepts of aleatoric and epistemic uncertainty.", "questions": "* What other methods and datasets could serve as benchmarks and how would the method perform there", "limitations": "No concerns about addressing limitations", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This is an interesting and useful addition to the literature on probing methods. It enables obtaining insights into the function of a deep learning model that could not have been obtained before.\n* The method is novel to the best of my knowledge.\n* The method is well-presented. The methods section is well-structured and clear.", "weaknesses": "* Other parts of the paper could use from better writing. In particular, I thought the intro and the experiments section could use additional polishing.\n* The paper could benefit from more extensive experiments. At the very least, it would be standard to test the method on more than one dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback. \n\n\n> “writing” \n\n\nThank you for these suggestions. We will polish both the intro and experiment sections.\n\n\n\n\n> “test the method on more than one dataset” \n\n\nIn the paper, we conducted experiments on 2 standard datasets and 1 set of photographic images for demo: (1) 3D Shapes [Burgess and Kim, 2018], (2) ImageNet [Russakovsky et al., 2015], and (3) online or proprietary real world images. For (1), we constructed **3 datasets based on concept ontology of the 3D Shapes dataset** to set up experiments where we can train several CNN models to get representations for specific tasks. The 3D Shapes dataset helps us to validate the method with a full control over the ground truth labels (with different ontologies) and levels of label noise. The results are shown in Figure 4-7. For (2), we constructed **10 datasets defined by coarse-grained labels of the ImageNet dataset** (Section C.2) to verify the usefulness of epistemic uncertainty predictions from GPP, and performed experiments on OOD detection tasks. The results can be found in Figure 7. For (3), we collected **online and proprietary images** (that the models have never been trained on) to demonstrate the predictions from GPP. The demos can be found in the introduction, Figure 3 and Table 1. Hence in total, we have 14 different settings in which we evaluated the approach, using several different datasets. We will clarify this point in the paper.\n\n\n\n\n> “What other methods and datasets could serve as benchmarks and how would the method perform there”\n\n\nAs mentioned above, we used 3 types of datasets (14 settings in total) and we will make this clearer in the paper. For baseline methods, we included established SOTA or near-SOTA methods, including LP (linear probes), SVM probes, LPE (linear probe ensembles) [Kim et al., 2018], MSP (maximum predicted softmax probabilities using LP) [Hendrycksand Gimpel, 2016], and Maha (negative Mahalanobis distance-based score) [Lee et al., 2018]. As shown in the extensive analyses of OOD detection methods and tasks in Appendix E of Tran et al., 2022, Maha and LPE (LPE is equivalent to their “Entropy” method for ensembles) achieved top performance, surpassing more recent proposals from Ren et al., 2021. We will include more OOD detection baselines such as “deep nearest neighbors” [Sun et al., 2022] in the revised version of the paper. Preliminary results on “deep nearest neighbors” can be found in the PDF from the “global” reply.\n\n\n\n\nWe thank the reviewer again for the helpful suggestions and we will address them in our paper accordingly.\n\n\n*References*\n\n\nChris Burgess and Hyunjik Kim. 3D Shapes dataset. https://github.com/deepmind/3dshapes-dataset/,2018.\n\n\nOlga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang,Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNetLarge Scale Visual Recognition Challenge.International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.\n\n\nBeen Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.Interpretability beyond feature attribution:  Quantitative testing with concept activation vectors(TCAV). InInternational Conference on Machine Learning (ICML), 2018.\n\n\nDustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han,Zi Wang, Zelda Mariet, Huiyi Hu, et al.  Plex: Towards reliability using pretrained large modelextensions.arXiv preprint arXiv:2207.07411, 2022.\n\n\nJie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022, 2021.\n\n\nYiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. InInternational Conference on Machine Learning (ICML), 2022.\n"}, {"review_id": "GF8tEaiEHJ", "reviewer": "Reviewer_iFt6", "summary": "- The authors propose a probabilistic probing method to understand a given pre-trained classifier. \n- The authors describe how looking at classifier predicted class probabilities is not enough since \"0.5\" in a binary task can happen for several reasons spanning the aleatoric/epistemic uncertainty spectrum\n- On the other hand, the proposed probabilistic method allows for posterior estimates of classifier's predicted probabilities, which allows for example to produce variance/entropy/etc\n- The particular method used is to compute functions g() of a classifier's representation a(x) for inputs x and to study the distribution of g()\n- The distribution of g() is defined through a hierarchical GP called the Beta GP\n- After giving an exposition of the model the authors introduce two sets of experiments: whether the probe can correctly identify that there is true label uncertainty in some synthetic experiments, and whether the model can correctly identify that data is OOD for a given classifier.\n\nWhile proposed method is well motivated and properly defined, I have some major questions regarding evaluation (proper definition of all tasks and baselines). For now, I think the paper needs some clarification before acceptance, but would be glad to raise my score, given clarification from authors.", "questions": "The results for the first experimental section (uncertainty quantification) seem convincing relative to the baselines but one major issue\n- LPE is not cited, and the authors do not mention that they propose it. There is only a high-level description given. I was trying to look for a precise definition of the method since it is taken as one of the main baselines, but cannot find it anywhere.\n- Please give a citation for LPE, or precisely define it and state that this is a baseline proposed by the authors.\n- This is moreover important since one cannot arbitrarily compare model entropies/variances with each other for continuous models without more assumptions (same base measure, etc). And for this, precise definition of all distributions is important.\n\nFor OOD detection, two main groups of concerns, task definition and baselines:\n\nTask issue 1:\n    - there seem two be two datasets, a synthetic one and ImageNet\n    - this section doesn't explicitly name which synthetic dataset. I only assume by continuity that it is the shapes dataset from the previous subsection. Please name all datasets in all sections/captions/figures they are used in rather than just refer to \"synthetic data\".\n    - The authors say \"we generate another set of queries that include 1024 ID images and 1024 OOD images (uniform random\n    noise)\" but don't say from where is the in-distribution.\n\nTask issue 2:\n    - For Imagenet there are also some missing details that stop me from understanding the experiment.\n    - I understand this: \"Queries and observations are sampled disjointly from the validation split of the ImageNet dataset\"\n    - I understand this + I read the appendix section: \"We make 10 set of Ds using 10 binary classification tasks defined by supersets of ImageNet classes\"\n    - How are the pre-trained classifiers use? Did you specifically pull checkpoints that were also trained on the binarized problems? Or did you somehow pool together a multiclass classifier's probabilities for the underlying model + renormalize? What happened to the probabilities assigned to the other classes?\n    - Please let us know and then revise the paper to clarify the exact use of the pre-trained models versus the data and what exactly was computed, including any equations if there were any transformation steps from multiclass checkpoints to the binary task, or clarify that binary classification model checkpoints were used that correspond to the same binarization that you applied to the data\n    - Also, is the out-distribution in both datasets just uniform noise, or uniform noise added to ID data?\n\nBaselines: The baselines seem a little out of date, with the two non-LPE baselines (again, where is LPE from?) being from 2016 and 2018. It's okay to include older baselines as part of a broader evaluation, but why not include also recent methods from 2020-2023 such as any of the below. I might be missing context on this sub-area of ML, but at least in others I review actively, it's fairly rare to find a paper that only evaluates to methods from <=2018. Several well-cited recent methods are documented below. Maybe not all are applicable as baselines for your particular setup/model assumptions/method assumptions (e.g. black box vs having access to something), but please clarify.\n\n    - 2018, ODIN, Enhancing the reliability of out-of-distribution image\n    detection in neural networks. https://arxiv.org/abs/1706.02690\n\n    - 2018, A simple unified framework for detecting out-of-distribution samples and adversarial attacks.\n    https://papers.nips.cc/paper_files/paper/2018/hash/abdeb6f575ac5c6676b747bca8d09cc2-Abstract.html\n\n    - 2019 Likelihood Ratios for Out-of-Distribution Detection https://proceedings.neurips.cc/paper_files/paper/2019/file/1e79596878b2320cac26dd792a6c51c9-Paper.pdf\n\n    - 2020, Energy-based out-of-distribution detection. https://proceedings.neurips.cc/paper/2020/hash/f5496252609c43eb8a3d147ab9b9c006-Abstract.html\n\n    - 2021, REACT, React: Out-of-distribution detection with rectified activations.\n    https://proceedings.neurips.cc/paper/2021/hash/01894d6f048493d2cacde3c579c315a3-Abstract.html\n\n    - 2022, Dice: Leveraging sparsification for out-of-distribution detection. In ECCV,\n    https://arxiv.org/abs/2111.09805\n\n    - 2022, Out-of-distribution detection with deep nearest neighbors. https://arxiv.org/abs/2204.06507\n\n    - 2022, Scaling out-of-distribution detection for real-world settings, https://arxiv.org/abs/1911.11132\n\n    - 2022, Vim: Out-of-distribution with virtual-logit matching, https://arxiv.org/abs/2203.10807\n\n    - 2022, a review: https://arxiv.org/pdf/2110.11334.pdf\n\n\nIn short, it seems like good work, but it seems like authors know more than the readers (about task definitions, datasets, why newer baselines were not used). Please clarify. Glad to increase score if sufficient answers given to these concerns. Most concerns can be answered without experiments except the \"old baselines\" concern, which requires either experiments or a commitment to include newer baselines.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The papers strength's are:\n- Beta-GP-based probing method is very clearly defined\n- All of the computations e.g. posterior computations are precisely stated in the appendix\n- The overall motivation of the work (probabilistic probing) is solid\n- The introduced entropy/variance based metrics that pull apart some aspects of aleatoric versus epistemic uncertainty, adapted from previous literature, seem like a nice contribution to uncertainty/probing evaluation metrics", "weaknesses": "There are two downstream uses of the method                                                                                                                                                                                                                                                              \n- checking correlation of probes' reported uncertainty versus the true label uncertainty for synthetic/semi-synthetic datasets\n- OOD detection\n\nThe precise definitions for baselines in uncertainty estimation experiments were not given (namely LPE) and more definitions are necessary to correctly interpret the variance/entropy-based metrics. See \"Questions\".\n\nFor OOD detection, the superiority over baselines is exemplified but the baselines are out of date. The synthetic dataset is not named, and some details are missing from the Imagenet result to understand exactly how to the pretrained model checkpoints were run on the binarized data. See \"questions\"\n\n\nMinor writing style suggestion that does not affect my review: In sentences like\n\n\"There are important details in Beta GPs that require special attention: how to set the prior and how to approximate the posterior.\"\n\nSince there are a few distributions floating around that were recently introduced to the reader, it could be helpful to include the symbols in mid-sentence: \"There are two important details in Beta GPs that requires special attention: how to set the prior (mu and k(,) in GP(mu,k))\nand how to approximate the posterior p(f_alpha,f_beta | D, mu k ).\"\n\nAnd include a sentence after\n\n\"With samples from the posterior (falpha, fbeta) we can then approximate the distribution of g()~G with the log normals parameterized by f_alpha, f_beta.\" \n\nThis could help the reader piece everything together.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and constructive feedback.\n\n\nWe would like to first clarify that our primary goal is not to perform OOD detection but to understand which concepts a model can and cannot represent (i.e., probing). We designed GPP (Gaussian process probes) to measure uncertainty of “concepts” by using activations/representations as a medium -- a new way to \"probe\". Probing and delineating different kinds of uncertainty for deep neural net representations has not been well studied, but both fuzziness in concepts and unsureness are ubiquitous cognitive phenomena. When learning new concepts, it is natural for people to perform OOD detection (i.e., more colloquially, being able to say “I’m not sure since I haven’t learned it yet”). GPP fills in the gap between existing probing literature and a person’s intuitive understanding of concepts. This is an important step to advance the new capabilities of explainable and interpretable AI.\n\n\n> “precise definition” and “citation for LPE” and “distributions” in LPE\n\n\nThank you for pointing out this omission. Linear probe ensembling (LPE) adopts a standard bootstrap aggregating method to ensemble linear classifiers. The same method was used as a component in Kim et al., 2018. Tran et al., 2022 used ensembling and an entropy score for OOD detection. We will include both citations. \n\n\nPrecise description of LPE: LPE is an ensemble of linear probes. For each linear probe, we train a logistic regression classifier on a dataset sampled from the original set of observations with replacement. The size of the dataset is the same as the number of observations. Each ensemble has 100 linear probes.\n\n\nBy bootstrapping, each linear probe in LPE can be viewed as i.i.d. samples from an underlying distribution over classifiers (denoted as $g$ in the paper). Since variance/entropy-based metrics etc are computed with classifier samples, we can use those linear classifiers in LPE to compute those metrics in the same way as GPP (Section 2.4).\n\n\n\n\n> “writing style suggestion”\n\n\nThank you. We will polish the writing accordingly.\n\n\n\n\n> “Task issue 1” on names of datasets and what is in distribution\n\n\nWe thank the reviewer for their comment, and will clarify these points in the paper. The synthetic dataset is the 3D Shapes dataset [Burgess and Kim, 2018]. The ID examples for ImageNet are images that come from the same distribution that the probe observes. For example, consider a probe trained to classify \"dogs\" vs \"cats\"; images of dogs and cats would be ID, whereas random noise would be OOD.\n\n\n\n\n> “Task issue 2”: “How are the pre-trained classifiers use?”\n\n\nThe pre-trained models are off-the-shelf ImageNet classifiers trained on all 1000 fine-grained classes (e.g., \"hummingbird\"). The probe takes as input the model's intermediate activations (features), and is trained to classify between two coarse-grained classes (e.g., \"bird\" vs \"cat\"). These coarse-grained classes are constructed using the WordNet hierarchy (https://wordnet.princeton.edu/). \n\n\n\n\n> “Did you specifically pull checkpoints that were also trained on the binarized problems?”\n\n\nNo, as mentioned above, we used off-the-shelf ImageNet classifiers trained on all 1000 fine-grained classes. These classifiers are the pre-trained models used for probing.\n\n\n\n\n> “did you somehow pool together a multiclass classifier's probabilities for the underlying model + renormalize? What happened to the probabilities assigned to the other classes?”\n\n\nNo, the probe used the activations of the pre-trained model. Our goal was to see if the probe can distinguish between ID/OOD data so that it is a reliable probe. We did not need the probability outputs of the original pre-trained model for our purpose.\n\n\n\n\n> “is the out-distribution in both datasets just uniform noise, or uniform noise added to ID data?”\n\n\nThe OOD data in both datasets is pixel-wise uniform noise, not a noisy version of the ID data.\n\n\n\n\n> “OOD detection baselines”\n\n\nFor OOD detection baselines, we included MSP (maximum predicted softmax probabilities using LP) [Hendrycksand Gimpel, 2016], Maha (negative Mahalanobis distance-based score) [Lee et al., 2018] (this is the 2nd paper pointed out by the reviewer) and LPE (negative predicted variance from linear probe ensembles) [Tran et al., 2022, Kim et al., 2018]. As shown in the extensive analyses of OOD detection methods and tasks in Appendix E of Tran et al., 2022, Maha and LPE (LPE is equivalent to their “Entropy” method for ensembles) achieved top performance, surpassing more recent Ren et al., 2021.\n\n\n We included preliminary results on “deep nearest neighbors” [Sun et al., 2022] pointed out by the reviewer in the PDF, and we will add it to the revised version of the paper. \n\n\nWe thank the reviewer again for the helpful suggestions and we will address those in detail in our revised version.\n\n\n\n\n*References*\n\n\nBeen Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.Interpretability beyond feature attribution:  Quantitative testing with concept activation vectors(TCAV). InInternational Conference on Machine Learning (ICML), 2018.\n\n\nDustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han,Zi Wang, Zelda Mariet, Huiyi Hu, et al.  Plex: Towards reliability using pretrained large modelextensions.arXiv preprint arXiv:2207.07411, 2022.\n\n\nJie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022, 2021.\n\n\nKimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems (NeurIPS), 2018.\n\n\nYiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. InInternational Conference on Machine Learning (ICML), 2022.\n"}, {"review_id": "idF1nicGOP", "reviewer": "Reviewer_3D6e", "summary": "The manuscript proposes a Gaussian process-based probing (monitoring a layer using only the layer’s activations without influencing the model itself) method that can estimate uncertainty in prediction. In the experimental result section, the proposed method is applied to several example datasets that can check whether the proposed method can correctly estimate fuzziness in concepts and to out of distribution (OOD) detection.   ", "questions": "In eq. (4), what is \\mu(a)? I could not find its definition in the text. \n\nRegarding eq. (4) (and eq. (2)), it appears that we don’t need to consider two linear functions, W_alpha and W_beta, because they can be reduced a single linear function f = f_alpha - f_beta = W^T \\psi(x), where W=W_alpha - W_beta. What did I miss here?\n", "limitations": "Yes. No potential negative societal impact of their work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "I think that the text is easy to read and understand. This work can be significant because we need more tools to understand deep neural networks, which are black-box models. Uncertainty quantification is also a major selling point of the proposed method. However, I hope that the manuscript would have provided monitoring and diagnostic tools to understand layers’ behaviors using estimated uncertainty.  ", "weaknesses": "1. I think that the novelty of the proposed method is incremental. In fact, a number of works have been proposed based on GP or Deep GP models for uncertainty estimation (including decomposing uncertainty into Aleatoric and Epistemic uncertainty) and out-of-distribution (OOD) detection. The authors put together several ideas (from existing work) to create a new method that can be a good tool for certain problems. However, I did not find any new innovations or improvements from the methodologies of the proposed method.  \n\n2. I think that the manuscript could be further improved in terms of presentation. My main concern is that the motivation for the use of Beta (Dirichlet) Gaussian processes is not clearly stated in the main text. Why not just use the original Gaussian process formulation for classification? This approach requires approximation for inference, but there already have been accurate and practical tools, e.g., expectation and propagation (EP). If the main motivation is about computational complexities (as in the reference paper for Beta GP [ref Milios et al., 2018]), then the examples included in the experiments do not seem to be well designed to get the readers appreciate this motivation from them (as the numbers of the training datasets appear to be small).     \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and constructive feedback.\n\n\nWe would like to emphasize the novelty of this work comes from the application of kinds of uncertainty and GPs in the context of probing. . Our goal is not simply to measure the uncertainty of predictions from a neural net, where we agree GPs have previously been used. Rather, we designed GPP (Gaussian process probes) to measure uncertainty related to  any \"concepts\" represented by a neural net from its activations -- a new way to \"probe\" neural nets. To the best of our knowledge, delineating different kinds of uncertainty for deep neural net representations via probing has not been studied in the past. Both intrinsic fuzziness in concepts and uncertainty as to which concept applies are ubiquitous cognitive phenomena, making it natural to ask how these quantities can be extracted from neural nets. When learning new concepts, it is natural for people to perform OOD detection (i.e., more colloquially, being able to say “I’m not sure since I haven’t learned it yet”). GPP fills in the gap between the existing probing literature, which has measured the presence or absence of concepts but not uncertainty about those concepts, and people’s intuitive representation of concepts (where uncertainty of different kinds can easily be articulated). This is an important step for advancing the capabilities of explainable and interpretable AI.\n\n\n\n\n> “monitoring and diagnostic tools to understand layers’ behaviors using estimated uncertainty”\n\n\nWe agree with the reviewer that deeper analyses and comparisons between layers are exciting directions for follow-on work. Since GPP can be used for any layers of activations in a deep neural network, our work builds a strong foundation from which to  pursue these directions in the future. \n\n\n\n\n> “novelty” from a GP perspective\n\n\nIt is straightforward to distinguish the two kinds of uncertainty in the **GP regression** framework. However, to the best of our knowledge, aleatory uncertainty (fuzziness of concepts) has not been studied in the **GP classification** literature. There also has not been previous work in the GP literature to establish the correspondence between human perceptions of uncertainty [Fox and Ülkümen,2011] and GP classification predictions. \n\n\n\n\n> “the motivation for the use of Beta (Dirichlet) Gaussian processes”, “Why not just use the original Gaussian process formulation for classification?”\n\n\nThank you  for raising this point. Beta (or Dirichlet) Gaussian process classifiers have been shown to either outperform or achieve similar performance than classic GP classification approximations [Milios et al., 2018]. While our work is not limited to Beta GPs, we used the Beta (Dirichlet) GPs mainly because to the best of our knowledge, the Beta (Dirichlet) GPs is the state-of-the-art GP classification method. The other reason we decided to use the Beta GP is because the GP posterior inference can be done in closed form, and we can easily inspect the observations and posteriors to make sure the Beta GP is doing what we intended to do. We will make this motivation clearer in the paper.\n\n\n\n\n> “In eq. (4), what is \\mu(a)?”\n\n\n$\\mu$ is the mean function of the Beta GP first mentioned in the last paragraph of Section 2.2, and defined in Section 2.3.1. We will make this clearer for Eq. (4).\n\n\n\n\n> “Regarding eq. (4) (and eq. (2)), it appears that we don’t need to consider two linear functions, W_alpha and W_beta, because they can be reduced a single linear function f = f_alpha - f_beta = W^T \\psi(x), where W=W_alpha - W_beta. What did I miss here?”\n\n\nThat’s a very nice observation. We simplified the expression to a single linear function, but we have to use two different functions in the Beta GP, because each observed datapoint  transforms to two pseudo datapoints with heteroscedastic noises, one for $f_\\alpha$ and the other for $f_\\beta$ (Section 2.3.2). During inference, the posteriors of $f_\\alpha$ and $f_\\beta$ are updated accordingly. If we only use one function $f$, we cannot directly obtain the posterior of $f$ without updating $f_\\alpha$ and $f_\\beta$. Please see Section B.1 in the appendix for more details.\n\n\n\n\n*References*\n\n\nCraig R Fox and Gülden Ülkümen. Distinguishing two dimensions of uncertainty.Fox, CraigR. and Gülden Ülkümen (2011),“Distinguishing Two Dimensions of Uncertainty,” in Essaysin Judgment and Decision Making, Brun, W., Kirkebøen, G. and Montgomery, H., eds. Oslo:Universitetsforlaget, 2011.\n\n\nDimitrios Milios, Raffaello Camoriano, Pietro Michiardi, Lorenzo Rosasco, and Maurizio Filippone.Dirichlet-based Gaussian processes for large-scale calibrated classification.Advances in NeuralInformation Processing Systems, 31, 2018.\n"}, {"review_id": "W9NFXcZafG", "reviewer": "Reviewer_VzJz", "summary": "This work introduces a unified framework called Gaussian process probes (GPP) for probing and quantifying uncertainty in models' representations of concepts. GPP extends linear probing methods and uses a Bayesian approach to estimate the distribution of classifiers induced by the model. This distribution measures the model's ability to represent concepts and provides a measure of confidence in the representation. GPP is a simple procedure applicable to any pre-trained models with vector representations, requiring no access to training data, gradients, or architecture details. The validation experiments on synthetic and real-world datasets demonstrate that GPP can effectively probe concept representations with a small number of examples, accurately measure both epistemic (confidence) and aleatory (fuzziness) uncertainties, and detect out-of-distribution data.", "questions": "Can you explain in more detail the choice of prior as outlined in Section 2.3.1. I'm not sure I understand what you mean by matching the Beta prior and matching the normal distribution in lines 125-126. I think this must impose important assumptions on the model, affecting the outcomes so it would be good to communicate it clearly.\n\nAre there any practical implications to considering the two types of uncertainty separately? Are there any cases where it might give counterintuitive results?\n\nDo you foresee any practical issues with using higher dimensional Dirichlet-based GPs? Can the inference still be performed in closed form? ", "limitations": "The authors outlined limitations in the main part of the paper and provided a broader impact statement. From a practitioner's point of view, it would be important to outline the exact assumptions that the GP model imposes (I assume those will depend on the specific dataset/model). I believe this is done to some extent in the paper but I have further questions (as mentioned above).\n\nThe quality of the writing could be improved, it requires a proof-read.\n\nMinor: \nLine 69 - best repeated\nLine 124 - uniformly distributed\n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important problem of understanding the inner representations of complex models. It does so by providing some estimates over two different types of uncertainty. It gives clear explanations for the epistemic and aleatoric uncertainties. The use of GPs and the study of two types of uncertainty in this particular context seems novel.\n\n", "weaknesses": "As a non-expert in the area of probing, I found some parts of the paper hard to follow (see Questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback. \n\n\n> “Can you explain in more detail the choice of prior as outlined in Section 2.3.1. I'm not sure I understand what you mean by matching the Beta prior and matching the normal distribution in lines 125-126.”\n\n\nThe goal is to use a Log-normal distribution to approximate a Gamma distribution. Because a Beta variable can be written as two independent Gamma variables, we can approximate the Beta variable with two Log-normal variables. The logarithms of Log-normal variables are distributed according to normal distributions. That is why we can approximate a Beta distribution with two normal distributions. Please see more details on how close the PDF approximations are in Section B.3 (especially Figure 9) in the appendix included in the supplementary material.\n\n\n\n\n> “Are there any practical implications to considering the two types of uncertainty separately? Are there any cases where it might give counterintuitive results?”\n\n\nThe practical implications are that we can now distinguish between fuzziness in concepts (i.e., aleatory) and not having enough information to reveal what the concept is (i.e., epistemic). In the past literature, if a linear probe predicts 0.5 probability for the binary case, we would conclude that the model does not represent the concept. However, in this work, we showed that predicting 0.5 does not mean that the model cannot represent the concept, since some concepts can be intrinsically fuzzy. For example, some people think tomatoes are fruits but others don’t, but that does not mean people don’t have representations for fruits. Another situation is that the probe does not have enough observations to tell what the concept is. Please see the figure and 2nd paragraph in the introduction for more insights. In short, the predictions from GPP can reveal much richer information than previous methods, and the rich information provides practical insights on what the model is able to represent. We have not seen cases where GPP gives counterintuitive results.\n\n\n\n\n> “Do you foresee any practical issues with using higher dimensional Dirichlet-based GPs? Can the inference still be performed in closed form?”\n\n\nFor multiclass problems using Dirichlet-based GPs, the number of latent functions is the same as the number of classes. Hence the computation for inference grows linearly with the number of classes. The inference can still be performed in closed form, since those latent functions are independently distributed according to Gaussian processes, and the inference for each GP can be done in closed form. For higher dimensional activations, we don’t expect any issues since the GP computations rely on the kernel and the kernel only requires computing norms or inner products of activations.\n\n\n\n\n> “writing” and typos\n\n\nThanks for pointing this out. We will further polish the paper with more rounds of proof-read.\n"}, {"review_id": "C9yIWDZz53", "reviewer": "Reviewer_zzXk", "summary": "This paper provides Gaussian process probes (GGP), a probabilistic method to evaluate uncertainty for a binary classification task over a (pre-trained) feature extractor. The core idea is to use GP instead of a linear probe on the feature extractor. The use of GP provides a natural way to estimate two uncertainty measures: aleatory and epistemic. To examine the performance of GPP, the authors conduct two experiments using synthetic and real images, showing that GPP can preferably estimate uncertainty. ", "questions": "1. In 154, it is said the kernel k is defined as \"k(a, a) = v\", but the definition (3) tells us that  \"k(a, a) = v (||a||^2 + 1) / (||a|| + 1)^2\", which is not v unless a = 0. Why does this inconsistency happen?\n2. Line 144 says that GP with the kernel (3) is \"equivalent to defining a distribution over linear latent functions\", but then why does Fig 1 show some nonlinearity? \n3. What is the training procedure of LPE in Fig 5? Did you confirm that LPE was properly trained?", "limitations": "The authors fairly address the limitations. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Novel formulation to evaluate uncertainty for linear probe problems. \n2. The paper is well-written and easy to follow. A lot of illustrative figures help to understand the key concepts. \n3. Solid derivation of the GP model and uncertainty estimation.", "weaknesses": "1. The experiment results (Fig 5) look somewhat strange. For me, the LPE result doesn't make sense. The setting here is that only positive labels are noisy (flipped to negative with 50% probability), and the negative labels do not contain noise. In such a situation, the well-trained classifier will output 0.5 as the judged probability for positive and 0 for negative examples. This should occur when the classification problem is linearly separable on the feature space, and the number of observations is large enough. I assume both conditions are satisfied when the number of observations is 128, but I cannot see such a tendency from LPE. Also, due to the imbalanced noise, I expect to see a vertically asymmetric behavior for LPE (e.g. the upper limit of judged probability would be close to 0.5), but it is not. \n2. This paper doesn't explicitly explain multiclass cases. Line 111 says extending GPP for multiclass problems is straightforward, but I don't feel it is trivial. Also there are no experiments for multiclass cases. \n3. Some technical descriptions are unclear. See \"Questions\" below. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback. \n\n\n> LPE results in Figure 5, “the well-trained classifier will output 0.5 as the judged probability for positive and 0 for negative examples. This should occur when the classification problem is linearly separable on the feature space, and the number of observations is large enough. I assume both conditions are satisfied when the number of observations is 128, but I cannot see such a tendency from LPE.”\n\n\nIn our experiments, we randomly flipped the labels for the positive class, but it is possible for the linear classifier to find features that linearly separate the activations with new labels. This is especially true if the dimension of the activations is larger than the number of observations. For example, a hyperplane in 3D almost always exists to separate 2 points. Since the number of dimensions for the activations is 256, and the number of observations is 128, it is very likely that there exists a hyperplane that separates the activations (unless the activations with positive labels are almost overlapping). Also note that points in high dimensions are naturally distant from each other. Roughly speaking, for logistic regression, it only requires the distance to be larger than 3 to predict the judged probability to be larger than 95% or lower than 5%. So it is not surprising that judged probability predictions of LPE are close to 0 or 1.\n\n\n\n\n> “due to the imbalanced noise, I expect to see a vertically asymmetric behavior for LPE (e.g. the upper limit of judged probability would be close to 0.5), but it is not.”\n\n\nAs mentioned above, it is very likely for LPE to find a hyperplane that separates activations with the new labels. And since the queries are sampled from the same distribution as the observations with original labels, LPE is still going to output probabilities close to 0 or 1.\n\n\n\n\n> “ Line 111 says extending GPP for multiclass problems is straightforward, but I don't feel it is trivial. Also there are no experiments for multiclass cases”\n\n\nExtending the Beta GP component of GPP to multiclass only requires switching the Beta distribution to the Dirichlet distribution (Beta distribution is a special case of Dirichlet), and we can achieve this by using k latent functions instead of 2, where k is the number of classes. That is what we meant by being straightforward, and we will make this clearer in the paper. \n\n\nWe aim to detect whether a model is able to represent a concept or not (probing as binary classification), so multiclass classification was not the best problem setup for us. However, we agree that the multiclass problem is a very important topic, and we need to formulate the problem carefully and spend more effort to understand and build tools for multiclass probing (in addition to the GP classification component). For example, some problems may require multiclass multilabel outputs, where the binary version of GPP can be directly plugged in for each label. But for multiclass single-label problems, we might need to redefine episteme for each label and understand the new relationship of episteme, alea and judged probability. We hope to explore the multiclass probing problem more in our future work, and we will add discussions about these considerations in the next version of this paper.\n\n\n\n\n> “1. In 154, it is said the kernel k is defined as \"k(a, a) = v\", but the definition (3) tells us that \"k(a, a) = v (||a||^2 + 1) / (||a|| + 1)^2\", which is not v unless a = 0. Why does this inconsistency happen?”\n\n\nThank you for pointing this out. We made a typo in Equation (3). There should be a square root in the denominator to properly normalize, i.e., $k(a, a') = v\\frac{a^T a' + 1}{(\\lVert a\\rVert^2+1)^{\\frac12}(\\lVert a'\\rVert^2+1)^{\\frac12}}$, which ensures that $k(a,a)=v$. This was implemented properly in our code (line 191 of code/gp.py, which is used by line 211 in the cosine_kernel function). We will correct this in the paper.\n\n\n\n\n> “2. Line 144 says that GP with the kernel (3) is \"equivalent to defining a distribution over linear latent functions\", but then why does Fig 1 show some nonlinearity?”\n\n\nThis is because the linear latent functions ($f_\\alpha$ in Equation 4) are only linear to the *normalized activations with bias terms*. The latent functions are nonlinear in the space of activations.\n\n\n\n\n> “3. What is the training procedure of LPE in Fig 5? Did you confirm that LPE was properly trained?”\n\n\nLPE is a bootstrap ensemble of linear probes. For each member of the ensemble, we trained a logistic regression classifier on a dataset sampled from the original set of observations with replacement. The size of the dataset was the same as the number of observations. Each ensemble has 100 members. We also made sure to include at least one positive and one negative example so that logistic regression is possible. We will make these clear in the new version of the paper. \n\n\nTo validate the correctness of our LPE implementation, we performed sanity checks using LPE for model M.1 and task P.1 with no fuzzy labels. In this sanity check, LPE was able to achieve AUROC=1 with 40 or more observations. We include this result in the PDF attached to the “global response”, and the full figure can be found in Figure 15 in the appendix. Figure 14 of the appendix also illustrates that judged probabilities predicted by LPE are close to 1 for the majority of positive examples when ground truth probability is 1; if the ground truth probability is 0.25, the judged probabilities predicted by LPE are closer to 0. Collectively, these pieces of evidence support that LPE was properly trained in our experiments.\n"}]

Author rebuttal:
We are encouraged that the reviewers found our work novel (**Reviewers 5Gmc, VzJz, zzXk**), significant (**3D6e**), well-motivated (**iFt6, VzJz**), well-written and easy to follow (**zzXk, 3D6e**). Moreover, **Reviewer 5Gmc** acknowledged that our measures of uncertainty for probing are interesting and useful, and enable “obtaining insights into the function of a deep learning model that could not have been obtained before”. We are also pleased that reviewers recognized that our method was well-presented (**5Gmc**), clear (**iFt6, VzJz**) and solid (**zzXk**). 

In the attached PDF, we included the following figures: 
1. Figure 1 (for Reviewers iFt6, 5Gmc) presents additional results on OOD detection with a recent baseline method [Sun et al., ICML 2022] suggested in iFt6.
2. Figure 2 (for Reviewer zzXk) shows that LPE is a valid probing method and supports the fact that it was properly trained.

We really appreciate the suggestions and questions from the reviewers, and we reply to them individually below. We will incorporate all feedback in the new version of the paper.


Meta-review: The consensus after the discussion period is this submission should be accepted because of its technical contributions, potential practical impact and presentation clarity. Some reviewers requested changes to writing styles and additional OOD baselines. Would be great if the authors could address these for the camera-ready version.
Decision: Accept (poster)