Title: Gaussian Process Probes (GPP) for Uncertainty-Aware Probing
Venue: NeurIPS 2023
Accepted: yes
Keywords: Interpretability

probing

Bayesian

Gaussian process

transparency
Abstract: Understanding which concepts models can and cannot represent has been fundamental to many tasks: from effective and responsible use of models to detecting out of distribution data. We introduce Gaussian process probes (GPP), a unified and simple framework for probing and measuring uncertainty about concepts represented by models. As a Bayesian extension of linear probing methods, GPP asks what kind of distribution over classifiers (of concepts) is induced by the model. This distribution can be used to measure both what the model represents and how confident the probe is about what the model represents.  GPP can be applied to any pre-trained  model with vector representations of inputs (e.g., activations). It does not require access to training data, gradients, or the architecture. We validate GPP on datasets containing both synthetic and real images. Our experiments show it can (1) probe a model's representations of concepts even with a very small number of examples, (2) accurately measure both epistemic uncertainty (how confident the probe is) and aleatory uncertainty (how fuzzy the concepts are to the model), and (3) detect out of distribution data using those uncertainty measures as well as classic methods do. By using Gaussian processes to expand what probing can offer, GPP provides a data-efficient, versatile and uncertainty-aware tool for understanding and evaluating the capabilities of machine learning models.

Gaussian Process Probes (GPP) for Uncertainty-Aware Probing

Abstract

Understanding which concepts models can and cannot represent has been fundamental to many tasks: from effective and responsible use of models to detecting out of distribution data. We introduce Gaussian process probes (GPP), a unified and simple framework for probing and measuring uncertainty about concepts represented by models. As a Bayesian extension of linear probing methods, GPP asks what kind of distribution over classifiers (of concepts) is induced by the model. This distribution can be used to measure both what the model represents and how confident the probe is about what the model represents. GPP can be applied to any pre-trained model with vector representations of inputs (e.g., activations). It does not require access to training data, gradients, or the architecture. We validate GPP on datasets containing both synthetic and real images. Our experiments show it can (1) probe a model's representations of concepts even with a very small number of examples, (2) accurately measure both epistemic uncertainty (how confident the probe is) and aleatory uncertainty (how fuzzy the concepts are to the model), and (3) detect out of distribution data using those uncertainty measures as well as classic methods do. By using Gaussian processes to expand what probing can offer, GPP provides a data-efficient, versatile and uncertainty-aware tool for understanding and evaluating the capabilities of machine learning models.

1 Introduction

Deep learning models have become pervasive in a wide range of fields, yet they remain largely uninterpretable. As a result, gaining insight into which concepts deep learning models do or do not represent is of growing importance. This has led to the development of a family of methods known as 'probes' [Alain and Bengio, 2016], which aim to understand the representations learned by these models. One common approach to probe the representations of a model is to train independent classifiers for each concept and measure their accuracy [Alain and Bengio, 2016, Kim et al., 2018, Belinkov, 2022]. For example, if the classifier returns 0 . 5 probability (for the binary case) across a range of examples, we would reasonably conclude that the model does not represent the concept.

However, simply examining the output of a classifier may not adequately reflect the intricate nature of representing a concept. For example, we can think of two different scenarios that may result in the same classifier output. If you ask a person if an olive is a fruit, only about 60% will say yes (likewise 72% for avocado, 53% for pumpkin, and 40% for acorn) [McCloskey and Glucksberg, 1978]. This does not indicate that people have a limited representation of the fruit concept, but that

there is intrinsic fuzziness in the label. By contrast, if you show somebody an olive in a martini glass and tell them it is a new concept called 'blicket,' they may be 60% sure in judging whether an olive on a tree is a blicket (as opposed to 'blicket' having a specialized meaning that applies to cocktails). In this second case, the uncertainty is because they do not have enough observations to be certain about what the new concept is - a common problem in word learning [Xu and Tenenbaum, 2007].

Fuzzy concept: fruit

New concept: blicket

Labels:          1               0                    ?

1                         ?

Probability:                                               60% (Olive is fruit)                      100% (Image is olive)

Confidence level:                                             100% (Olive is 60% fruit )             60% (Image is blicket)

We introduce a probabilistic approach based on Gaussian processes (GPs) [Rasmussen and Williams, 2006] that makes it possible to delineate different kinds of uncertainty in a model's concepts. Specifically, our goal is to probe a pre-trained model by constructing a probability distribution over possible binary classifiers of concepts induced by the model. Extracting this distribution allows the probe to produce two kinds of uncertainty measures - aleatory and epistemic [Fox and Ülkümen, 2011]. This allows us to distinguish cases where 1) there is fundamental fuzziness of the label (aleatory uncertainty) or 2) it does not have enough observations (epistemic uncertainty), whereas a binary classifier would predict equal probability for both and provide no further explanatory power.

Our framework also provides a natural building block for answering many downstream questions, such as detecting out-of-distribution (OOD) examples. Using the resulting uncertainty measures, we show that our method can correctly probe what a model represents, and that it achieves competitive performance in OOD detection. We validate this approach using both synthetic and real images.

2 Related work

Distinguishing aleatory and epistemic uncertainty has a long history in fields including cognitive science [Howell and Burnett, 1978, Fox and Ülkümen, 2011], probability theory [Jaynes, 2003], philosophy [Hacking, 2006] and more. Our work builds upon Fox and Ülkümen [2011], which introduced a framework for distinguishing the two dimensions of uncertainty from a human perspective.

In machine learning, Gaussian processes (GPs) and Bayesian neural nets (BNN) [] have enabled principled ways of predicting epistemic uncertainty, which led to better information-based decision making strategies [Houlsby et al., 2011, Hennig and Schuler, 2012, Wang and Jegelka, 2017, Gal et al., 2017, Wang et al., 2023]. Motivated by regression problems in reinforcement learning, Depeweg et al. [2018] decomposed uncertainty in BNN to a conditional entropy term as aleatory uncertainty and the mutual information between model weights and predictions as epistemic uncertainty. Our decomposition simplifies these quantities 2 and specializes them for classification. To our knowledge, aleatory uncertainty (fuzziness of concepts) has not been studied in the GP classification literature, and no previous work in the GP literature has been done to establish the correspondence between human perceptions of uncertainty and GP classification predictions.

Probing has been a popular approach to understand which concepts a model represents. There are mixed views of what probing should mean; either it is about estimating mutual information between representations and labels [Pimentel et al., 2020] or using minimum description length (MDL) [Voita and Titov, 2020] or representing the distance as squared Euclidean distance under a linear transformation [Hewitt and Manning, 2019], or using analysis with or without supervision [Saphra and Lopez, 2019, Wu et al., 2020]. Others considered different types of probing: conditional probing that compares two concepts [Hewitt et al., 2021], probing that asks where the task-relevant information 'emerges' [Kunz and Kuhlmann, 2022] or which concepts are 'used' [Lasri et al., 2022], and finally, probing/localizing that enables editing [Meng et al., 2022]. To the best of our knowledge, none of these probing methods offer measures of uncertainty.

3 GPP: Gaussian process probes

We introduce Gaussian process probes (GPP), a simple probabilistic method for probing that naturally distinguishes aleatory and epistemic uncertainty. We formally define the problem of probabilistic probing in §3.1, followed by a review of the Beta Gaussian processes (GPs) that GPP builds upon in §3.2. In §3.3 and §3.4, we detail our method and how it measures uncertainty.

3.1 Problem formulation

Our goal is to probe a pre-trained model in a probabilistic manner by constructing a probability distribution over binary classifiers based on the model's vector representation of inputs (e.g., activations). Here, binary classifier refers to a function mapping a stimulus to the probability of a positive label.

Assumption 1. The model, M , contains basis functions that map from stimuli (e.g., images, texts) to a vector representation: φ : X ↦→ A ⊂ R d , where X is the space of stimuli and A the space of d -dimensional real-valued vector representations.

For example, basis functions φ can map from images to last-layer activations of a convolutional neural network (CNN). The probabilistic probe treats basis functions φ as a feed-forward black box, and does not require accessing the training data, gradients, or architecture used during model pre-training.

In order to define a stochastic process to describe a distribution over classifiers, the data generating process is assumed as follows.

Assumption 2. There exists a stochastic process G ( θ ) parameterized by θ , and a classifier h : A ↦→ [0 , 1] distributed according to G ( θ ) , such that the label y ∈ { 0 , 1 } for any vector representation a ∈ A is independently distributed according to a Bernoulli distribution with parameter h ( a ) ; i.e., p ( y | a, h ) = h ( a ) y (1 -h ( a )) 1 -y .

Now the posterior inference of the stochastic process (in our case, Beta GP) requires conditioning on observations, and the stochastic process measures uncertainty for a set of queries. We denote a set of observations as D , which consists of representation-label pairs, i.e., D = { ( φ ( x i ) , y i ) } | D | i =1 ; here, label y i is either 0 or 1 . The queries, a q = [ φ ( x j )] | a q | j =1 ∈ R d ×| a q | , are vector representations of a set of stimuli whose labels are to be predicted together with uncertainty measures.

Note that there are no additional assumptions about this probing dataset, e.g., the distributions of stimuli in observations or queries. We use GPP for OOD detection, as shown later. Unlike existing probes (e.g., [Hewitt et al., 2021, Xu et al., 2020]), we do not consider a stimulus and its vector representation as random variables with some underlying distributions. Instead, viewing them as deterministic allows us to probe with any stimuli (with potentially any underlying distributions).

3.2 Background: Beta Gaussian processes

GPP uses a Beta GP to define a prior distribution over classifiers. The Beta GP is a special case of Dirichlet-based GPs [Milios et al., 2018], which have been shown to either outperform or achieve similar performance than classic GP classification approximations [Rasmussen and Williams, 2006]. For each representation a ∈ A , Beta GPs assume the binary label y is generated by sampling y ∼ Bernoulli ( g ( a )) where the Bernoulli parameter g ( a ) ∼ Beta ( α ( a ) , β ( a )) . The Beta variable g ( a ) can be written as two independent Gamma variables:

where t α ( a ) and t β ( a ) are independent. The Gamma distributions are then approximated with Log-normal distributions via moment matching. That means,

and the same for f β . Beta GPs use two GPs to model the two latent functions f α and f β ,

/gid00014/gid00042/gid00031/gid00032/gid00039

/gid00017/gid00045/gid00042/gid00029/gid00032

/gid00473/gid00473/gid00473/gid00473/gid00473/gid00473

/gid00016/gid00029/gid00046/gid00032/gid00045/gid00049/gid00028/gid00047/gid00036/gid00042/gid00041/gid00046

/gid00020/gid00028/gid00040/gid00043/gid00039/gid00032/gid00031/gid00001/gid00030/gid00039/gid00028/gid00046/gid00046/gid00036/gid00414/gid00032/gid00045/gid00046

g~ g(θ | D)

{（,0),(°,1)}

0.6

0.6

0.5

a1

0.4

0.4

0.2

g(θ | D)

g(a)

0.4

0.6

a1

0.3

0.5

=

Φ(x)

0.2

0.5

0.4

Op

ao

D

{（

0),(

1)}

0.5

g(a)

C-

M

0.5

g(a)

Note that the latent function f in Eq. 2 is still a GP given that it is the sum of two independent GPs. We denote the Beta GP as G ( θ ) , where θ = ( µ, k ) . While we focus on probing as binary classification in this work, it is straightforward to extend GPP to multi-class classification by switching the Beta prior to a Dirichlet prior [Milios et al., 2018] (i.e., using more latent functions than just f α and f β ).

3.3 Adapting Beta GPs for GPP

There are two important details in Beta GPs that require special attention: how to set the prior and how to approximate the posterior of classifier g . This requires setting mean and kernel functions µ, k in the prior GP ( µ, k ) of the latent functions f α and f β , and computing their posterior f α , f β | D .

We adapt these aspects of Beta GPs for probing settings, that also enables users to incorporate the characteristics of their end-tasks by setting intuitive hyperparameters. See more explanations and illustrations in the Appendix.

3.3.1 Setting the prior

First, we need to make sure the prior distribution approximated by GPs in our GPP matches with the Beta prior. Without loss of generality, we assume that for any a ∈ A , the prior distribution for g ( a ) is Beta ( glyph[epsilon1], glyph[epsilon1] ) for some glyph[epsilon1] > 0 . This hyperparameter glyph[epsilon1] reflects the characteristics of the task at hand; the probability of a positive label might be bi-modal with density centered at both 0 and 1 (e.g., label is not fuzzy) ( glyph[epsilon1] < 1 ), or uniformly distribution in [0 , 1] ( glyph[epsilon1] = 1 ), or centered at 0 . 5 ( glyph[epsilon1] > 1 ).

To match the Beta prior, the prior for both f α ( a ) and f β ( a ) has to be N (log( glyph[epsilon1] ) -v 2 , v ) , where v = log( 1 glyph[epsilon1] +1) . To match this normal distribution, we set the mean function to a constant µ ( a ) = log( glyph[epsilon1] ) -v 2 , and constrain the kernel function to satisfy k ( a, a ) = v .

3.3.2 Posterior inference

This formulation offers a closed-form solution for posterior inference of the GP. Namely, any y i = 1 in D = { ( a i , y i ) } | D | i =1 results in updating Beta parameters of g ( a i ) to be Beta ( glyph[epsilon1] + s, glyph[epsilon1] ) and any y i = 0 results in Beta ( glyph[epsilon1], glyph[epsilon1] + s ) , for some s ≥ 1 . This hyperparameter s describes how much 'strength' (i.e., weight of the observation) we add to the Beta posterior. To match the GP posterior to the Beta posterior, we synthetically add observations with heteroscedastic noise to functions f α and f β . If y i = 1 , the observation for f α is ( a i , log( glyph[epsilon1] + s ) -v ′ 2 ) with noise N (0 , v ′ ) where v ′ = log( 1 glyph[epsilon1] + s +1) , and the observation for f β is ( a i , log( glyph[epsilon1] ) -v ′′ 2 ) with noise N (0 , v ′′ ) where v ′′ = log( 1 glyph[epsilon1] +1) . And vice versa for y i = 0 . The posterior of a GP remains a GP with closed-form updates to its mean and kernel function. Once we have the posterior of latent functions f α | D and f β | D , obtaining the posterior of classifier g is trivial by updating the GP priors in Eq. 2 to the GP posteriors.

3.3.3 The cosine kernel

The choice of kernel function is important for GPP as it directly determines the characteristics of the distribution of latent functions, which translates to the distribution of classifiers that GPP uses. We define a cosine kernel as:

Note that multiplying the constant v is the constraint we have from setting the prior for the Beta GP. Using the cosine kernel in the GP is equivalent to defining a distribution over linear latent functions in an augmented space of the original representation space A , i.e.,

And similarly for f β with weights W β . Here µ is the mean function of the Beta GP. Note that we have the closed-form posterior distribution for the weights, which remains a multivariate Gaussian distribution. See Appendix for more details.

Why is the kernel defined in this way? During the pre-training stage of deep learning models, a weighted sum of the activations and a bias term are used to construct neurons of the next layer. Following Wang et al. [2023], we can view each next layer neuron as independent samples of classifiers from the same distribution, described by a Beta GP in this work. This means classifiers from the Beta GP should all consider a bias term. Hence we augment the activations by an additional dimension with value fixed at 1 (Eq. 4). Additionally, we normalize the augmented activations, ψ ( a ) , such that k ( a, a ) = v (the constraint from setting the prior in §3.2) holds for all a ∈ A (Eq. 3).

We can interpret the cosine kernel as a scaled cosine of the angle between two vectors (standard cosine similarity) in an augmented representation space described by activations and biases.

3.4 Uncertainty measures

The probabilistic probing formulation makes it possible to compute a posterior distribution over labels for queries. Building on terms from Fox and Ülkümen [2011], we define uncertainty measures that identify the epistemic and aleatory uncertainty of the posterior prediction g ( a ) where g = 1 1+ e -f ∼ G ( θ | D ) , ∀ a ∈ A . Let the posterior of the latent function be f ( a ) | D ∼ N ( µ D ( a ) , k D ( a )) .

Episteme := -H [ g ( a )] , the negative entropy of the distribution of g ( a ) , which describes the amount of knowledge the probe has about the label of stimulus a . In GPP, we have H [ g ( a )] = H [ f ( a )] -µ D ( a ) -2 E [log(1 + e -f ( a ) )] and H [ f ( a )] = 1 2 log(2 πek D ( a )) .

Alea := H [ y | g ( a )] = E [ -g ( a ) log( g ( a )) -(1 -g ( a )) log(1 -g ( a ))] , the expected entropy of the conditional distribution p ( y | g ( a )) , where y is the label of stimulus a . Higher alea corresponds to more fuzziness in the label of a .

Judged probability := E [ g ( a )] = E [ 1 1 -e -f ( a ) ] , which is the expected probability (judged by the probe) that the label of stimulus a is positive.

Figure 2 shows the relationship of episteme, alea and judged probability for a rational agent (white region, based on [Fox and Ülkümen, 2011]).

(a)

(b)

(c)

(d)

1

0

1

1

0

1

Query

Observations

0

0

1

Episteme (prior): 0.042

Judged probability: 0.5

Epi: 0.88

Prob: 0.96

Episteme: 1.3

Judged probability: 0.6

Episteme: 1.5

Judged probability: 0.79

Episteme: 1.5

Judged probability: 0.37

Low episteme means we are 'not sure' and high episteme that we are 'highly confident' about the underlying probability. Alea, on the other hand, increases with the intrinsic randomness of the true label. The judged probability that the stimulus's label is positive reflects both episteme and alea. With low episteme, the judged probability should be closer to 0.5, meaning the probe does not have enough information to draw a plausible conclusion. It would be irrational (grey areas in Figure 2) for a person to believe something will definitely happen even if there is no evidence. Analogously, it is irrational for a probe to produce probabilities close to 0 or 1 while having low episteme. By accumulating evidence (increasing episteme), the judged prob-

0.5

1

0

Episteme

Irrationality

Judged probability

Irrationality

'purely random'

'certainly yes'

'certainly no'

'not sure'

'highly confident'

ability can move closer and closer to the ground truth (moving to the right in the figure). An accurate probe should produce a judged probability that matches the ground truth fuzziness of a label only if episteme is high.

Figure 3 is a real-world example (using a CoCa model [Yu et al., 2022] and natural images, shown in the figure, as query and observations) of how our measures of uncertainty change as we increase a small number of positive and negative observations.

Table 1 presents the results of applying GPP to the concepts involving olives introduced in §1. The judged probabilities from GPP match human judgments [McCloskey and Glucksberg, 1978]: olives have a 57% chance of being fruit with a high level of confidence, and GPP is very uncertain about whether olives on a tree are blickets.

4 Validating GPP

In addition to probing concepts, GPP predicts aleatory and epistemic uncertainty, which allows us to determine the fuzziness of a concept or detect out-of-domain (OOD) stimuli.

We use 3D Shapes [Burgess and Kim, 2018] and construct 3 datasets based on its concept ontology to validate these uses of uncertainty measures. In these 3 settings, we can control the ground truth level of fuzziness by artificially injecting noise to labels (§4.2, §4.3). In §4.5, we evaluate the OOD performance of GPP and competitive baselines on 3D Shapes as well as 10 datasets defined by coarse-grained labels of the ImageNet dataset [Russakovsky et al., 2015].

4.1 Experiment setups for probing

We evaluate GPP's ability to probe using the 3D Shapes dataset [Burgess and Kim, 2018], which contains images generated from 6 ground truth independent primitives: 10 floor colors, 10 wall colors, 10 object colors, 8 scales, 4 shapes and 15 orientations of the shapes, as shown below.

Datasets and models. We construct a 2-level concept ontology (see Appendix for visualization) consisting of a disjunctive level where colors are binarized to warm/cool and scale to small/large, and a conjunctive level that combines shapes, binarized colors and binarized scales.

We train three CNN models on labels generated from this ontology: 1) M . 1 is trained on 64 labels (binarized scale, floor, wall object color x 4 shapes), 2) M . 2 is trained on 8 labels (4 shapes x binarized scale) 3) M . 3 is trained on 8 labels (binary floor, wall and object color).

We define two probing tasks to investigate each model: 1) task P . 1 : binarized colors (floor, wall, object) and 2) task P . 1 : binarized scales and 4 shapes.

Baseline descriptions. The baselines include 1) LP: linear probes [Alain and Bengio, 2016]; 2) SVM: SVM probes (a baseline from Kim et al. [2018]); 3) LPE: linear probes ensembled via bootstrap [Kim et al., 2018]. For each linear probe in LPE, we train a logistic regression classifier on a dataset sampled from the original set of observations with replacement. The size of the dataset is the same as the number of observations. Each ensemble has 100 linear probes. LPE can also describe a distribution over classifiers and be evaluated as probabilistic probes, making this a strong baseline. For GPP, we use Beta (0 . 1 , 0 . 1) as the prior and set strength to be 5.

4.2 Validating GPP as a probe

To compare with baseline methods, we use AUROC to evaluate the deterministic classifier defined by the judged probability E [ g ( a )] , ∀ a ∈ a q (defined in §3.4). Queries are sampled disjointly from the training data and observations. Figure 4 shows that GPP performs competitively against LP and SVM. GPP and LP perform comparable with respect to AUROC and sample efficiency (needing as few as 10 observations for probing concepts in task P . 1 ).

One may first guess that M . 3 is the best model that represents color, since it was explicitly tasked to distinguish color. However, our probing results reveal what is obvious in hindsight: while M . 2 is only tasked to distinguish shapes and scales, separating where the shapes are requires detecting color (i.e., the only cue to separate shape from background is color). This is shown in M . 2 's high AUROC for P . 1 in Figure 4. M . 3 is only tasked to distinguish colors, so performance on P . 2 remains low.

4.3 Estimating concept fuzziness

What happens when the concepts we are trying to probe are inherently fuzzy? This is commonly the case in the real world, where situations and the language used to describe them contain extensive ambiguity. We again focus on M . 1 and concepts in task P . 1 . We control the fuzziness of concepts in P . 1 by randomly flipping positive labels to negative (but not vice versa) in the observations. We test ground truth label probabilities of 0 . 25 , 0 . 5 , 0 . 75 and 1 , where 1 indicates no fuzziness. To measure GPP's alignment with ground truth label probabilities, we use the Pearson correlation coefficient between ground truth label probabilities and judged probabilities for all queries.

0

50

100

Observations

0.5

0.6

0.7

0.8

0.9

1.0

AUROC

0

50

100

Observations

0.5

0.6

0.7

0.8

0.9

1.0

Model M.2

0

50

100

Observations

0.5

0.6

0.7

0.8

0.9

1.0

Model M.3

Probe

LP

SVM

GPP

Target tasks

P.1

P.2

0

50

100

Observations

0.0

0.2

0.4

0.6

0.8

Probe

LPE

GPP

10

3

10

1

10 1

Episteme

0.0

0.2

0.4

0.6

0.8

1.0

Judged probability

LPE

10

1

10 0

Episteme

0.0

0.2

0.4

0.6

0.8

Judged probability

GPP

Observations

2

8

16

32

128

10

3

10

1

10 1

Episteme

0.0

0.2

0.4

0.6

LPE | Ground truth prob. = 0.5

10

1

10 0

Episteme

0.1

0.2

0.3

0.4

0.5

0.6

0.7

GPP | Ground truth prob. = 0.5

10

1

10 0

Episteme

0.0

0.2

0.4

0.6

GPP | Ground truth prob. = 1

Observations

2

8

16

32

128

Figure 5 shows that GPP can accurately probe fuzzy concepts by predicting both judged probability (expected level of fuzziness) and episteme (how sure it is about the judged probability). GPP achieves a higher Pearson correlation coefficient than LPE across all sizes of observations. In particular, the relationship between judged probabilities and episteme coincides with the two-dimensional framework for characterizing uncertainty from Fox and Ülkümen [2011] (see Fig. 2), showing the validity of GPP as a rational method to probe fuzziness.

Figure 5 (Right) confirms that the judged probability alone cannot be used to distinguish whether concept is fuzzy or the stimuli are far away: in both cases, judged probabilities can be 0 . 5 . Hence, it is important to remember that judged probability is only accurate with enough observations. We illustrate this point from another angle in Figure 6, which shows the relation between alea and episteme predicted by LPE, GPP with ground truth label probability 0 . 5 or 1 . When ground truth label probability is 0 . 5 , GPP can have low alea under low episteme (same effect as Figure 1), but with more episteme, GPP puts more mass of predictions for high alea because the ground truth alea is high. LPE does not show this kind of rational behaviors. When ground truth label probability is 1 , more mass of alea predicted by GPP converges to 0 as episteme increases, since the ground truth alea is 0.

4.4 Experiment setups for OOD detection

When learning new concepts, it is natural for people to be able to say 'I'm not sure since I haven't learned it yet'. For probes, the corresponding ability is to perform out-of-distribution (OOD) detection, which identifies datapoints that are different from the observed training data. GPP naturally performs OODdetection using episteme predictions. With an uninformative prior for the labels, 3 high episteme indicates in-domain (ID), while low episteme indicates OOD.

Method descriptions. For GPP, we use the negative latent variance of posterior function f (Eq. 2) as a proxy of episteme for computational efficiency. It is easy to show that the entropy of classifier g is upper bounded by the entropy of latent function f , which is fully determined by its posterior variance (more details in Appendix). In practice, we found that using the negative latent variance as OOD detection scores for GPP is robust to the level of informativeness of the Beta prior, because the latent GPs are agnostic with respect to the logistic transformation in Eq. 2.

As for baselines, we compare against MSP (maximum predicted softmax probabilities using LP) [Hendrycks and Gimpel, 2016], Maha (negative Mahalanobis distance-based score) [Lee et al., 2018], NN (deep nearest neighbors) [Sun et al., 2022] and LPE. Recall that LPE uses bootstrapping to create an ensemble of linear probes. Each linear probe in LPE can be viewed as i.i.d. samples from an underlying distribution over classifiers (i.e., g in Figure 1). Similar to approximating metrics with samples in §3.4, we can estimate the negative variance of predictions from the set of linear probes, and use it as the OOD detection score for LPE.

As shown in the extensive analyses of OOD detection methods and tasks in Appendix E of Tran et al. [2022], Maha and LPE (LPE is equivalent to their 'Entropy' method for ensembles) achieved top performance, surpassing more recent proposals from Ren et al. [2021].

Datasets and models. We generate a set of queries from task P . 1 of 3D Shapes that include 1024 ID images and 1024 OOD images (uniform random noise), and see if OOD queries can be detected using the methods above based on representations from model M . 1 .

We also evaluate the OOD detection performance of all methods on real world datasets and models. Queries and observations in the real world datasets are sampled disjointly from the validation split of the ImageNet dataset [Russakovsky et al., 2015]. We make 10 sets of D s using 10 binary classification tasks defined by supersets of ImageNet classes. Query points consist of 128 images from the classes the probe has observed and 128 OOD images (uniform random noise). The ID examples are images that come from the same distribution that the probe observes. For example, consider a probe trained to classify "all dogs" vs "all cats"; images of dogs and cats would be ID, whereas random noise would be OOD. The models we used are the pre-trained ResNet-50 [He et al., 2016] and CoCa-Base [Yu et al., 2022]. See more details in Appendix.

2 16 32

64

128

Observations

0.0

0.2

0.4

0.6

0.8

1.0

AUROC (ID/OOD)

M.1

2

4

6

8

10 12 14 16 18

Observations

0.0

0.2

0.4

0.6

0.8

1.0

AUROC (ID/OOD)

ResNet-50

2

4

6

8

10 12 14 16 18

Observations

0.0

0.2

0.4

0.6

0.8

1.0

CoCa-Base

MSP

Maha

NN

LPE

GPP

4.5 Results for OOD detection

Figure 7 shows the competitive performance of GPP as an OOD detector for all the OOD detection tasks we constructed. While GPP might not be the best OOD detection approach for each of the tasks, GPP performed consistently well and on par with the best benchmark method for each task. This means, while GPP is mainly a probing method, it is also competitive for OOD detection.

Note that the effective term in GPP's negative latent posterior variance, k ( a, a o ) K -1 k ( a o , a ) (here a o is the concatenated observed activations and K = k ( a o , a o ) ∈ R O × O ), looks very similar to Mahalanobis distance, ( a -ˆ u c ) glyph[latticetop] ˆ Σ -1 ( a -ˆ u c ) (Eq. 2 from Lee et al. [2018], where ˆ Σ ∈ R d × d ; d is the dimension of activations). However, GPP's variance measures the similarity between function values but Mahalanobis distance measures the similarity between activations. Also note that the Gaussian estimated for Mahalanobis distance is degenerate for d > O , and so the distance is still computed on a O -dimensional manifold; this is possibly why the Maha method shows inferior performance.

5 Conclusion

We introduced GPP, an uncertainty-aware probing framework that uses Gaussian processes to measure both the aleatory uncertainty intrinsic to the concept represented by a model and the epistemic uncertainty reflecting the probe's confidence about its judgement. These uncertainty measurements also equip GPP with the ability to detect OOD stimuli. Empirically, we verified the strong performance of GPP for data-efficient probing that can correctly surface fuzziness in concepts and OOD detection which shows GPP can also surface epistemic uncertainty (i.e., knows that it does not know). These results illustrate how distinguishing between different forms of uncertainty can be beneficial both for deepening our theoretical understanding of models and for practical applications.

Limitations. We have validated our approach on both synthetic and real-world datasets, and demonstrated its use for OOD detection, but there are a much wider range of models, datasets, and applications that can be explored. Fully evaluating the potential of this approach and achieving the greatest impact will require pursuing this investigation, which we hope to do in future work. In particular, we plan to explore active few-shot learning and distributionally robust probing. Finally, GPP can only be used with models for which activations are made available, which may present an obstacle to use with certain commercial machine learning systems.

Broader impact. Being able to understand what models can and cannot represent has huge implications. Not only does it impact important downstream tasks, such as OOD detection, but it ultimately aids human-machine communication and collaboration. While a (potentially) superficial interaction (e.g., with a chatbot) is already possible, we also know models hallucinate [Ji et al., 2023], leaving people unsure about the validity/faithfulness of this interaction. Work such as ours ultimately aims to help humans to better use and develop models that aligns well with our concepts, as well as potentially learning new concepts from machines.

Acknowledgments and Disclosure of Funding

We thank Fei Sha for constructive discussions on aleatory uncertainty in classification. We also thank Geoff Pleiss, Alexander Terenin, Ben Adlam, Jasper Snoek and Zackary Nado for helpful conversations on Gaussian process classification. Images of 'blicket' from https://www.iwawine.com/Images/riedel-vinum-martini-glasses_ 10.jpg and https://libreshot.com/wp-content/uploads/2017/12/ green-olives-on-the-tree.jpg (by Martin Vorel).

References

Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 , 2016.

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning (ICML) , 2018.

Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics , 48(1):207-219, 2022.

Michael E McCloskey and Sam Glucksberg. Natural categories: Well defined or fuzzy sets? Memory &Cognition , 6(4):462-472, 1978.

Fei Xu and Joshua B Tenenbaum. Word learning as Bayesian inference. Psychological Review , 114 (2):245, 2007.

Carl Edward Rasmussen and Christopher KI Williams. Gaussian Processes for Machine Learning . The MIT Press, 2006.

Craig R Fox and Gülden Ülkümen. Distinguishing two dimensions of uncertainty. Fox, Craig R. and Gülden Ülkümen (2011),'Distinguishing Two Dimensions of Uncertainty,' in Essays in Judgment and Decision Making, Brun, W., Kirkebøen, G. and Montgomery, H., eds. Oslo: Universitetsforlaget , 2011.

William C Howell and Sarah A Burnett. Uncertainty measurement: A cognitive taxonomy. Organizational Behavior and Human Performance , 22(1):45-68, 1978.

T. Jaynes. Probability Theory: The Logic of Science . Cambridge university press, 2003.

Ian Hacking. The Emergence of Probability: A Philosophical Study of Early Ideas about Probability, Induction and Statistical Inference . Cambridge University Press, 2006.

Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745 , 2011.

Philipp Hennig and Christian J Schuler. Entropy search for information-efficient global optimization. Journal of Machine Learning Research (JMLR) , 13:1809-1837, 2012.

Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient Bayesian optimization. In International Conference on Machine Learning (ICML) , 2017.

Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In International Conference on Machine Learning (ICML) , 2017.

Zi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-trained Gaussian processes for Bayesian optimization. arXiv preprint arXiv:2109.08215 , 2023.

Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning (ICML) , 2018.

Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. Information-theoretic probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4609-4622, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.420. URL https://aclanthology.org/2020.acl-main.420 .

Elena Voita and Ivan Titov. Information-theoretic probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 183-196, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.14. URL https://aclanthology.org/2020. emnlp-main.14 .

John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4129-4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1419. URL https://aclanthology.org/N19-1419 .

Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 3257-3267, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1329. URL https://aclanthology.org/N19-1329 .

Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. Perturbed masking: Parameter-free probing for analyzing and interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4166-4176, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.383. URL https://aclanthology.org/ 2020.acl-main.383 .

John Hewitt, Kawin Ethayarajh, Percy Liang, and Christopher D Manning. Conditional probing: measuring usable information beyond a baseline. In International Conference on Learning Representations (ICLR) , 2021.

Jenny Kunz and Marco Kuhlmann. Where does linguistic information emerge in neural language models? measuring gains and contributions across layers. In International Conference on Computational Linguistics , 2022.

Karim Lasri, Tiago Pimentel, Alessandro Lenci, T. Poibeau, and Ryan Cotterell. Probing for the usage of grammatical number. In Annual Meeting of the Association for Computational Linguistics , 2022.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In International Conference on Learning Representations (ICLR) , 2020.

Dimitrios Milios, Raffaello Camoriano, Pietro Michiardi, Lorenzo Rosasco, and Maurizio Filippone. Dirichlet-based Gaussian processes for large-scale calibrated classification. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022.

Chris Burgess and Hyunjik Kim. 3D Shapes dataset. https://github.com/deepmind/3dshapes-dataset/, 2018.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115 (3):211-252, 2015.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Machine Learning (ICML) , 2016.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning (ICML) , 2022.

Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXiv:2207.07411 , 2022.

Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022 , 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys , 55(12):1-38, 2023.

George A. Miller. WordNet: A lexical database for English. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994 , 1994. URL https://aclanthology.org/H94-1111 .

A Notation

In Table 2, we include the main notation used in this paper.

B Details on the Beta GP in GPP

As shown in § 3.2 and §3.3, with observations D = { ( a i , y i ) } | D | i =1 , the posterior of a Beta GP can be written as the transformed version of a GP, i.e.,

We show how to obtain the mean and kernel functions µ D , k D in §B.1, as well as the posterior of weights in §B.2 for the cosine kernel. The behavior of GPP relies on two hyperparameters, glyph[epsilon1] and s , and we explain what they are and how to set them in §B.3. In §B.4, we analyze and bound the episteme of GPP.

As a reference, Figure 8 shows the graphical model of GPP.

θ

f

α

f

β

g

y

x

s

N

glyph[epsilon1]

t

α

t

β

h

y

x

s

N

B.1 Posterior inference (extension of §3.3.2)

Without loss of generality, we write observations as a union of a dataset (of size n ) with the positive labels only and a dataset (of size | D | -n ) with negative labels only, i.e., D = { ( a i , y i ) } | D | i =1 = D + ∪ D -where D + = { ( a i , y i ) } n i =1 and D -= { ( a i , y i ) } | D | i = n +1 .

For convenience, we use the following short-hand notation:

Observing a positive example is equivalent to observing y ′ with noise N (0 , v ′ ) on f α and observing y ′′ with noise N (0 , v ′′ ) on f β . Vice versa for observing a negative example. We also denote 1 m as a column vector of size m filled with 1 s, and I m as an identity matrix of size m .

Recall that f α ∼ GP ( µ, k ) , f β ∼ GP ( µ, k ) and f α ⊥ ⊥ f β . The posterior for f α is f α | D ∼ GP ( µ α , k α ) . For any a, a ′ ∈ A ,

µ α ( a ) = µ ( a ) + k ( a, a ) K -1 α ( y α -µ ( a )) , k α ( a, a ′ ) = k ( a, a ′ ) -k ( a, a ) K -1 α k ( a , a ′ ) , (5) where

and

Similarly for f β | D ∼ GP ( µ β , k β ) , we have

µ β ( a ) = µ ( a ) + k ( a, a ) K -1 β ( y β -µ ( a )) , k β ( a, a ′ ) = k ( a, a ′ ) -k ( a, a ) K -1 β k ( a , a ′ ) , (6) where

Since f = f α -f β , by combining Eq. 5 and Eq. 6, we have f | D ∼ GP ( µ D , k D ) , and

Thus, we obtain the closed-form posterior for function f .

For classifier g ( a ) = 1 1+ e -f ( a ) , we can then get its PDF as follows,

B.2 Posterior inference for weights (extension of §3.3.3)

If we use the cosine kernel in §3.3.3, the posterior of f α can be written as

This means f α ( a ) | D ∼ N ( u glyph[latticetop] α ψ ( a ) + µ ( a ) , ψ ( a ) glyph[latticetop] Σ α ψ ( a )) .

Because of Eq. 5 and Eq. 4, we can also write the posterior of f α as

By comparing the above two ways of writing the posterior of f α , we obtain

Similarly, for f β ( a ) = W glyph[latticetop] β ψ ( a ) + µ ( a ) , W β | D ∼ N ( u β , Σ β ) , we have

Then, for f = f α -f β = W glyph[latticetop] ψ ( a ) , W | D ∼ N ( µ, Σ) , we have

This means we can directly sample classifiers from a Beta GP with a cosine kernel by sampling weights W from a multivariate Gaussian distribution defined above.

B.3 How to set hyperparameters

There are two hyperparameters in GPP with the cosine kernel: glyph[epsilon1] , which determines the prior, and s , which determines the posterior.

For any a ∈ A , the prior on the probability that the label is positive is Beta ( glyph[epsilon1], glyph[epsilon1] ) . As noted in §3.3.1, glyph[epsilon1] < 1 reflects a belief that g ( a ) should be close to either 0 or 1 ; glyph[epsilon1] = 1 gives a uniform distribution over [0 , 1] ; and glyph[epsilon1] > 1 reflects a belief that g ( a ) is centered at 0 . 5 . In the Beta GP, the Beta prior is approximated as

Eq. 9 is obtained using the prior of f , i.e., µ D ( a ) = 0 , k D ( a, a ) = 2 k ( a, a ) = 2 log( 1 ˆ glyph[epsilon1] +1) , in Eq. 8. Users can choose Beta (ˆ glyph[epsilon1], ˆ glyph[epsilon1] ) for moment matching in Eq. 9 to get a better approximation of Beta ( glyph[epsilon1], glyph[epsilon1] ) .

Figure 9 shows both the PDF of Beta priors and the approximations.

For setting the hyperparameter s , users can also directly inspect the behaviors of different values of s and choose an appropriate value. Figure 10 and Figure 11 show how the posterior changes with one negative or two opposite-label observations. Larger s leads to more concentrated posterior.

Since all of these distributions are easily computable and can be visualized clearly, users can directly inspect the behaviors of these different hyperparameters and choose a suitable option.

B.4 Analyses of episteme (extension of §3.4)

For each a ∈ A , episteme is the negative of H [ g ( a )] . By Eq. 8, we have

small warm

large warm

large cool

Conjunctive level

cube, warm wall,

sphere, warm

cylinder, cool

warm floor

wall, cool floor

wall, warm floor

Disjunctive level

object is

object color

wall color is

wall color is

small

is warm

warm

warm

Primitives

object is capsule

object is cube

object is XXL

object is XS

object is XXS

object is magenta

object is orange

object is red

wall is orange

wall is red

floor is magenta

.··

floor is orange

floor is red

The last inequality is because y (1 -y ) < 1 , i.e., log ( y (1 -y )) < 0 .

In §3.3.1, we set a constraint on the kernel k such that k ( a, a ) = log( 1 glyph[epsilon1] +1) . By Eq. 7, the entropy of f ( a ) can be bounded as follows.

Hence there exists a lower bound on episteme for GPP. However, H [ g ( a )] can approach -∞ because (1) variable y can be infinitely close to 0 or 1 , and (2) k D ( a, a ) can also be infinitely close to 0 , which means episteme has no finite upper bound.

In natural language, our analyses of episteme show that ignorance has a limit, but knowledge has no limit. This is a widely recognized idea, and it is also reflected in the words of Zhuangzi, a Chinese philosopher from the 4th century BCE: 'Your life has a limit, but knowledge has none.'

C Experiment details

In this section, we include details on experiment setups and additional results.

C.1 3D Shapes ontology

The ontology of training labels for the 3D Shapes dataset [Burgess and Kim, 2018] is illustrated in Figure 12. Images are generated from 6 ground truth independent primitives: 10 floor colors, 10 wall colors, 10 object colors, 8 scales, 4 shapes and 15 orientations of the shapes (orientation is excluded from the ontology since it's only distinguishable for cubes). The disjunctive level of the ontology groups together ranges of color and shape primitives into binary concepts: warm/cool and small/large, respectively. Concepts in the conjunctive level are the Cartesian product of concepts in the disjunctive level and the shape primitives.

C.2 Real-world OOD detection

In-distribution (ID) queries are sampled disjointly from the validation split of the ImageNet dataset [Russakovsky et al., 2015], where the probe observes 10 sets of D s using 10 binary classification

tasks defined by ImageNet superclasses. These superclasses are defined by building a tree using the WordNet hierarchy [Miller, 1994] where the leaves of this tree are ImageNet classes (e.g., the superclass "dog" contains Chihuahua, Japanese Spaniel, Maltese, etc.). The ten classification tasks we use are: (1) dog vs snake, (2) fish vs lizard, (3) bird vs snake, (4) dog vs bird, (5) cat vs bird, (6) fish vs snake, (7) bird vs fish, (8) snake vs lizard, (9) cat vs dog, (10) bird vs lizard.

Out-of-distribution (OOD) images are generated with pixel-wise uniform random noise. This noise is passed through the basis function φ to construct the OOD query.

C.3 Relations between judged probability, episteme and alea

In this section, we present more empirical results. The experiment setting is the same as §4.3.

First, we evaluate how judged probability and alea change as episteme increases, and how judged probability changes as alea increases. Figure 13 and Figure 14 show the results for GPP and LPE respectively. Please use a higher quality PDF viewer if the points don't show up in the figures. Each row corresponds to a different ground truth probability, which means the probability that an originally positive stimulus remains to have positive labels in observations, when we manually inject fuzziness to concepts in the 3D Shapes dataset [Burgess and Kim, 2018]. So in the ideal case, judged probability predictions should converge to the ground truth probability for stimuli that are originally positive. Each scattered point corresponds to the predictions on a stimulus that is originally positive.

GPP consistently produces rational uncertainty measures. There are no extreme judged probability predictions with low episteme. Alea converges to low values for 1.0 ground truth probability and higher values when the ground truth probability is 0.25 or 0.75, and alea converges to the highest values when the ground truth probability is 0.5. These are all expected since with 0.5 ground truth probability, the level fuzziness reaches the highest.

On the contrary, LPE tends to have more extreme predictions on judged probability no matter what the ground truth probability is. However, the average judged probability of LPE does get affected by the ground truth probability. For example, when the ground truth probability is 0.25, more masses of judged probability accumulate between 0 to 0.2. This means the average judged probability can be close to 0.25. Similarly, when the ground truth probability is 0.5, about half of the predictions of judged probability are between 0.8 to 1.0, and the other half are between 0.0 to 0.2. While this ensures the judged probability is close to ground truth probability if we average over all stimuli, the individual predictions cannot be used to evaluate the fuzziness of concepts.

Figure 15 shows AUROC, AUPRC and accuracy of GPP and LPE for different numbers of observations. These metrics are averaged over both positive and negative queries. Interestingly, even when the ground truth probability is 0.25 (only 1/4 of the positive examples remain positive), GPP can still achieve almost 1.0 AUROC and AUPRC. As expected, the accuracy of GPP is about 0.5 for 0.25 ground truth probability (since the judged probability predictions are mostly lower than 0.5 for the positive examples, and the negative examples are almost all correct). But because LPE does not always have low judged probability predictions even if ground truth probability is 0.25, its accuracy is higher than 0.5.

These results confirm the rationality and good performance of GPP as a valid probing method.

10

1

10 0

Episteme

0.2

0.4

0.6

0.8

1.0

Judged probability

GPP | Ground truth prob. = 1.0

Observations

2

8

16

32

128

10

1

10 0

Episteme

0.0

0.1

0.2

0.3

0.4

0.5

0.6

GPP | Ground truth prob. = 1.0

10

2

10

1

0.2

0.4

0.6

0.8

1.0

Judged probability

GPP | Ground truth prob. = 1.0

10

1

10 0

Episteme

0.0

0.2

0.4

0.6

0.8

1.0

Judged probability

GPP | Ground truth prob. = 0.75

10

1

10 0

Episteme

0.2

0.3

0.4

0.5

0.6

0.7

GPP | Ground truth prob. = 0.75

2×10 1

3×10 1

4×10 1

6×10 1

0.0

0.2

0.4

0.6

0.8

1.0

Judged probability

GPP | Ground truth prob. = 0.75

10

1

10 0

Episteme

0.0

0.2

0.4

0.6

0.8

Judged probability

GPP | Ground truth prob. = 0.5

10

1

10 0

Episteme

0.1

0.2

0.3

0.4

0.5

0.6

0.7

GPP | Ground truth prob. = 0.5

10

1

0.0

0.2

0.4

0.6

0.8

Judged probability

GPP | Ground truth prob. = 0.5

10

1

10 0

Episteme

0.0

0.2

0.4

0.6

0.8

Judged probability

GPP | Ground truth prob. = 0.25

10

1

10 0

Episteme

0.1

0.2

0.3

0.4

0.5

0.6

0.7

GPP | Ground truth prob. = 0.25

10

1

0.0

0.2

0.4

0.6

0.8

Judged probability

GPP | Ground truth prob. = 0.25

10

3

10

2

10

1

10 0

10 1

Episteme

0.0

0.2

0.4

0.6

0.8

1.0

Judged probability

LPE | Ground truth prob. = 1.0

Observations

2

8

16

32

128

10

3

10

2

10

1

10 0

10 1

Episteme

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

LPE | Ground truth prob. = 1.0

10

10

10

8

10

6

10

4

10

2

10 0

0.0

0.2

0.4

0.6

0.8

1.0

Judged probability

LPE | Ground truth prob. = 1.0

10

3

10

2

10

1

10 0

10 1

Episteme

0.0

0.2

0.4

0.6

0.8

1.0

Judged probability

LPE | Ground truth prob. = 0.75

10

3

10

2

10

1

10 0

10 1

Episteme

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

LPE | Ground truth prob. = 0.75

10

5

10

4

10

3

10

2

10

1

10 0

0.0

0.2

0.4

0.6

0.8

1.0

Judged probability

LPE | Ground truth prob. = 0.75

10

4

10

3

10

2

10

1

10 0

10 1

Episteme

0.0

0.2

0.4

0.6

0.8

1.0

Judged probability

LPE | Ground truth prob. = 0.5

10

4

10

3

10

2

10

1

10 0

10 1

Episteme

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

LPE | Ground truth prob. = 0.5

10

4

10

3

10

2

10

1

10 0

0.0

0.2

0.4

0.6

0.8

1.0

Judged probability

LPE | Ground truth prob. = 0.5

10

3

10

2

10

1

10 0

10 1

Episteme

0.0

0.2

0.4

0.6

0.8

1.0

Judged probability

LPE | Ground truth prob. = 0.25

10

3

10

2

10

1

10 0

10 1

Episteme

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

LPE | Ground truth prob. = 0.25

10

8

10

6

10

4

10

2

10 0

0.0

0.2

0.4

0.6

0.8

1.0

Judged probability

LPE | Ground truth prob. = 0.25

0

20

40

60

80

100

120

Observations

0.800

0.825

0.850

0.875

0.900

0.925

0.950

0.975

1.000

AUROC

Ground truth prob. = 1.0

Probe

GPP

LPE

0

20

40

60

80

100

120

Observations

0.825

0.850

0.875

0.900

0.925

0.950

0.975

1.000

AUPRC

Ground truth prob. = 1.0

Probe

GPP

LPE

0

20

40

60

80

100

120

Observations

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Ground truth prob. = 1.0

Probe

GPP

LPE

0

20

40

60

80

100

120

Observations

0.800

0.825

0.850

0.875

0.900

0.925

0.950

0.975

1.000

AUROC

Ground truth prob. = 0.75

Probe

GPP

LPE

0

20

40

60

80

100

120

Observations

0.825

0.850

0.875

0.900

0.925

0.950

0.975

1.000

AUPRC

Ground truth prob. = 0.75

Probe

GPP

LPE

0

20

40

60

80

100

120

Observations

0.70

0.75

0.80

0.85

0.90

0.95

Ground truth prob. = 0.75

Probe

GPP

LPE

0

20

40

60

80

100

120

Observations

0.75

0.80

0.85

0.90

0.95

1.00

AUROC

Ground truth prob. = 0.5

Probe

GPP

LPE

0

20

40

60

80

100

120

Observations

0.75

0.80

0.85

0.90

0.95

1.00

AUPRC

Ground truth prob. = 0.5

Probe

GPP

LPE

0

20

40

60

80

100

120

Observations

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Ground truth prob. = 0.5

Probe

GPP

LPE

0

20

40

60

80

100

120

Observations

0.70

0.75

0.80

0.85

0.90

0.95

1.00

AUROC

Ground truth prob. = 0.25

Probe

GPP

LPE

0

20

40

60

80

100

120

Observations

0.75

0.80

0.85

0.90

0.95

1.00

AUPRC

Ground truth prob. = 0.25

Probe

GPP

LPE

0

20

40

60

80

100

120

Observations

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Ground truth prob. = 0.25

Probe

GPP

LPE

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides Gaussian process probes (GGP), a probabilistic method to evaluate uncertainty for a binary classification task over a (pre-trained) feature extractor. The core idea is to use GP instead of a linear probe on the feature extractor. The use of GP provides a natural way to estimate two uncertainty measures: aleatory and epistemic. To examine the performance of GPP, the authors conduct two experiments using synthetic and real images, showing that GPP can preferably estimate uncertainty.\n\nSTRENGTHS:\n1. Novel formulation to evaluate uncertainty for linear probe problems. \n2. The paper is well-written and easy to follow. A lot of illustrative figures help to understand the key concepts. \n3. Solid derivation of the GP model and uncertainty estimation.\n\nWEAKNESSES:\n1. The experiment results (Fig 5) look somewhat strange. For me, the LPE result doesn't make sense. The setting here is that only positive labels are noisy (flipped to negative with 50% probability), and the negative labels do not contain noise. In such a situation, the well-trained classifier will output 0.5 as the judged probability for positive and 0 for negative examples. This should occur when the classification problem is linearly separable on the feature space, and the number of observations is large enough. I assume both conditions are satisfied when the number of observations is 128, but I cannot see such a tendency from LPE. Also, due to the imbalanced noise, I expect to see a vertically asymmetric behavior for LPE (e.g. the upper limit of judged probability would be close to 0.5), but it is not. \n2. This paper doesn't explicitly explain multiclass cases. Line 111 says extending GPP for multiclass problems is straightforward, but I don't feel it is trivial. Also there are no experiments for multiclass cases. \n3. Some technical descriptions are unclear. See \"Questions\" below.\n\nQUESTIONS:\n1. In 154, it is said the kernel k is defined as \"k(a, a) = v\", but the definition (3) tells us that  \"k(a, a) = v (||a||^2 + 1) / (||a|| + 1)^2\", which is not v unless a = 0. Why does this inconsistency happen?\n2. Line 144 says that GP with the kernel (3) is \"equivalent to defining a distribution over linear latent functions\", but then why does Fig 1 show some nonlinearity? \n3. What is the training procedure of LPE in Fig 5? Did you confirm that LPE was properly trained?\n\nLIMITATIONS:\nThe authors fairly address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a unified framework called Gaussian process probes (GPP) for probing and quantifying uncertainty in models' representations of concepts. GPP extends linear probing methods and uses a Bayesian approach to estimate the distribution of classifiers induced by the model. This distribution measures the model's ability to represent concepts and provides a measure of confidence in the representation. GPP is a simple procedure applicable to any pre-trained models with vector representations, requiring no access to training data, gradients, or architecture details. The validation experiments on synthetic and real-world datasets demonstrate that GPP can effectively probe concept representations with a small number of examples, accurately measure both epistemic (confidence) and aleatory (fuzziness) uncertainties, and detect out-of-distribution data.\n\nSTRENGTHS:\nThe paper addresses an important problem of understanding the inner representations of complex models. It does so by providing some estimates over two different types of uncertainty. It gives clear explanations for the epistemic and aleatoric uncertainties. The use of GPs and the study of two types of uncertainty in this particular context seems novel.\n\nWEAKNESSES:\nAs a non-expert in the area of probing, I found some parts of the paper hard to follow (see Questions below).\n\nQUESTIONS:\nCan you explain in more detail the choice of prior as outlined in Section 2.3.1. I'm not sure I understand what you mean by matching the Beta prior and matching the normal distribution in lines 125-126. I think this must impose important assumptions on the model, affecting the outcomes so it would be good to communicate it clearly.\n\nAre there any practical implications to considering the two types of uncertainty separately? Are there any cases where it might give counterintuitive results?\n\nDo you foresee any practical issues with using higher dimensional Dirichlet-based GPs? Can the inference still be performed in closed form?\n\nLIMITATIONS:\nThe authors outlined limitations in the main part of the paper and provided a broader impact statement. From a practitioner's point of view, it would be important to outline the exact assumptions that the GP model imposes (I assume those will depend on the specific dataset/model). I believe this is done to some extent in the paper but I have further questions (as mentioned above).\n\nThe quality of the writing could be improved, it requires a proof-read.\n\nMinor: \nLine 69 - best repeated\nLine 124 - uniformly distributed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes a Gaussian process-based probing (monitoring a layer using only the layer’s activations without influencing the model itself) method that can estimate uncertainty in prediction. In the experimental result section, the proposed method is applied to several example datasets that can check whether the proposed method can correctly estimate fuzziness in concepts and to out of distribution (OOD) detection.\n\nSTRENGTHS:\nI think that the text is easy to read and understand. This work can be significant because we need more tools to understand deep neural networks, which are black-box models. Uncertainty quantification is also a major selling point of the proposed method. However, I hope that the manuscript would have provided monitoring and diagnostic tools to understand layers’ behaviors using estimated uncertainty.\n\nWEAKNESSES:\n1. I think that the novelty of the proposed method is incremental. In fact, a number of works have been proposed based on GP or Deep GP models for uncertainty estimation (including decomposing uncertainty into Aleatoric and Epistemic uncertainty) and out-of-distribution (OOD) detection. The authors put together several ideas (from existing work) to create a new method that can be a good tool for certain problems. However, I did not find any new innovations or improvements from the methodologies of the proposed method.  \n\n2. I think that the manuscript could be further improved in terms of presentation. My main concern is that the motivation for the use of Beta (Dirichlet) Gaussian processes is not clearly stated in the main text. Why not just use the original Gaussian process formulation for classification? This approach requires approximation for inference, but there already have been accurate and practical tools, e.g., expectation and propagation (EP). If the main motivation is about computational complexities (as in the reference paper for Beta GP [ref Milios et al., 2018]), then the examples included in the experiments do not seem to be well designed to get the readers appreciate this motivation from them (as the numbers of the training datasets appear to be small).\n\nQUESTIONS:\nIn eq. (4), what is \\mu(a)? I could not find its definition in the text. \n\nRegarding eq. (4) (and eq. (2)), it appears that we don’t need to consider two linear functions, W_alpha and W_beta, because they can be reduced a single linear function f = f_alpha - f_beta = W^T \\psi(x), where W=W_alpha - W_beta. What did I miss here?\n\nLIMITATIONS:\nYes. No potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The authors propose a probabilistic probing method to understand a given pre-trained classifier. \n- The authors describe how looking at classifier predicted class probabilities is not enough since \"0.5\" in a binary task can happen for several reasons spanning the aleatoric/epistemic uncertainty spectrum\n- On the other hand, the proposed probabilistic method allows for posterior estimates of classifier's predicted probabilities, which allows for example to produce variance/entropy/etc\n- The particular method used is to compute functions g() of a classifier's representation a(x) for inputs x and to study the distribution of g()\n- The distribution of g() is defined through a hierarchical GP called the Beta GP\n- After giving an exposition of the model the authors introduce two sets of experiments: whether the probe can correctly identify that there is true label uncertainty in some synthetic experiments, and whether the model can correctly identify that data is OOD for a given classifier.\n\nWhile proposed method is well motivated and properly defined, I have some major questions regarding evaluation (proper definition of all tasks and baselines). For now, I think the paper needs some clarification before acceptance, but would be glad to raise my score, given clarification from authors.\n\nSTRENGTHS:\nThe papers strength's are:\n- Beta-GP-based probing method is very clearly defined\n- All of the computations e.g. posterior computations are precisely stated in the appendix\n- The overall motivation of the work (probabilistic probing) is solid\n- The introduced entropy/variance based metrics that pull apart some aspects of aleatoric versus epistemic uncertainty, adapted from previous literature, seem like a nice contribution to uncertainty/probing evaluation metrics\n\nWEAKNESSES:\nThere are two downstream uses of the method                                                                                                                                                                                                                                                              \n- checking correlation of probes' reported uncertainty versus the true label uncertainty for synthetic/semi-synthetic datasets\n- OOD detection\n\nThe precise definitions for baselines in uncertainty estimation experiments were not given (namely LPE) and more definitions are necessary to correctly interpret the variance/entropy-based metrics. See \"Questions\".\n\nFor OOD detection, the superiority over baselines is exemplified but the baselines are out of date. The synthetic dataset is not named, and some details are missing from the Imagenet result to understand exactly how to the pretrained model checkpoints were run on the binarized data. See \"questions\"\n\n\nMinor writing style suggestion that does not affect my review: In sentences like\n\n\"There are important details in Beta GPs that require special attention: how to set the prior and how to approximate the posterior.\"\n\nSince there are a few distributions floating around that were recently introduced to the reader, it could be helpful to include the symbols in mid-sentence: \"There are two important details in Beta GPs that requires special attention: how to set the prior (mu and k(,) in GP(mu,k))\nand how to approximate the posterior p(f_alpha,f_beta | D, mu k ).\"\n\nAnd include a sentence after\n\n\"With samples from the posterior (falpha, fbeta) we can then approximate the distribution of g()~G with the log normals parameterized by f_alpha, f_beta.\" \n\nThis could help the reader piece everything together.\n\nQUESTIONS:\nThe results for the first experimental section (uncertainty quantification) seem convincing relative to the baselines but one major issue\n- LPE is not cited, and the authors do not mention that they propose it. There is only a high-level description given. I was trying to look for a precise definition of the method since it is taken as one of the main baselines, but cannot find it anywhere.\n- Please give a citation for LPE, or precisely define it and state that this is a baseline proposed by the authors.\n- This is moreover important since one cannot arbitrarily compare model entropies/variances with each other for continuous models without more assumptions (same base measure, etc). And for this, precise definition of all distributions is important.\n\nFor OOD detection, two main groups of concerns, task definition and baselines:\n\nTask issue 1:\n    - there seem two be two datasets, a synthetic one and ImageNet\n    - this section doesn't explicitly name which synthetic dataset. I only assume by continuity that it is the shapes dataset from the previous subsection. Please name all datasets in all sections/captions/figures they are used in rather than just refer to \"synthetic data\".\n    - The authors say \"we generate another set of queries that include 1024 ID images and 1024 OOD images (uniform random\n    noise)\" but don't say from where is the in-distribution.\n\nTask issue 2:\n    - For Imagenet there are also some missing details that stop me from understanding the experiment.\n    - I understand this: \"Queries and observations are sampled disjointly from the validation split of the ImageNet dataset\"\n    - I understand this + I read the appendix section: \"We make 10 set of Ds using 10 binary classification tasks defined by supersets of ImageNet classes\"\n    - How are the pre-trained classifiers use? Did you specifically pull checkpoints that were also trained on the binarized problems? Or did you somehow pool together a multiclass classifier's probabilities for the underlying model + renormalize? What happened to the probabilities assigned to the other classes?\n    - Please let us know and then revise the paper to clarify the exact use of the pre-trained models versus the data and what exactly was computed, including any equations if there were any transformation steps from multiclass checkpoints to the binary task, or clarify that binary classification model checkpoints were used that correspond to the same binarization that you applied to the data\n    - Also, is the out-distribution in both datasets just uniform noise, or uniform noise added to ID data?\n\nBaselines: The baselines seem a little out of date, with the two non-LPE baselines (again, where is LPE from?) being from 2016 and 2018. It's okay to include older baselines as part of a broader evaluation, but why not include also recent methods from 2020-2023 such as any of the below. I might be missing context on this sub-area of ML, but at least in others I review actively, it's fairly rare to find a paper that only evaluates to methods from <=2018. Several well-cited recent methods are documented below. Maybe not all are applicable as baselines for your particular setup/model assumptions/method assumptions (e.g. black box vs having access to something), but please clarify.\n\n    - 2018, ODIN, Enhancing the reliability of out-of-distribution image\n    detection in neural networks. https://arxiv.org/abs/1706.02690\n\n    - 2018, A simple unified framework for detecting out-of-distribution samples and adversarial attacks.\n    https://papers.nips.cc/paper_files/paper/2018/hash/abdeb6f575ac5c6676b747bca8d09cc2-Abstract.html\n\n    - 2019 Likelihood Ratios for Out-of-Distribution Detection https://proceedings.neurips.cc/paper_files/paper/2019/file/1e79596878b2320cac26dd792a6c51c9-Paper.pdf\n\n    - 2020, Energy-based out-of-distribution detection. https://proceedings.neurips.cc/paper/2020/hash/f5496252609c43eb8a3d147ab9b9c006-Abstract.html\n\n    - 2021, REACT, React: Out-of-distribution detection with rectified activations.\n    https://proceedings.neurips.cc/paper/2021/hash/01894d6f048493d2cacde3c579c315a3-Abstract.html\n\n    - 2022, Dice: Leveraging sparsification for out-of-distribution detection. In ECCV,\n    https://arxiv.org/abs/2111.09805\n\n    - 2022, Out-of-distribution detection with deep nearest neighbors. https://arxiv.org/abs/2204.06507\n\n    - 2022, Scaling out-of-distribution detection for real-world settings, https://arxiv.org/abs/1911.11132\n\n    - 2022, Vim: Out-of-distribution with virtual-logit matching, https://arxiv.org/abs/2203.10807\n\n    - 2022, a review: https://arxiv.org/pdf/2110.11334.pdf\n\n\nIn short, it seems like good work, but it seems like authors know more than the readers (about task definitions, datasets, why newer baselines were not used). Please clarify. Glad to increase score if sufficient answers given to these concerns. Most concerns can be answered without experiments except the \"old baselines\" concern, which requires either experiments or a commitment to include newer baselines.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-16"}

comments: SUMMARY:
The authors introduce Gaussian process probes, a probabilistic probing method. They use this method to obtain additional insights into the internals of deep learning models, using the concepts of aleatoric and epistemic uncertainty.

STRENGTHS:
* This is an interesting and useful addition to the literature on probing methods. It enables obtaining insights into the function of a deep learning model that could not have been obtained before.
* The method is novel to the best of my knowledge.
* The method is well-presented. The methods section is well-structured and clear.

WEAKNESSES:
* Other parts of the paper could use from better writing. In particular, I thought the intro and the experiments section could use additional polishing.
* The paper could benefit from more extensive experiments. At the very least, it would be standard to test the method on more than one dataset.

QUESTIONS:
* What other methods and datasets could serve as benchmarks and how would the method perform there

LIMITATIONS:
No concerns about addressing limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27