SlowGuess/NAACL_2022_OCR huggingface dataset:
Jiaoda Li Ryan Cotterell Mrinmaya Sachan
{jiaoda.li, ryan.cotterell, mrinmaya.sachan}@inf.ethz.ch
Probing is a popular method to discern what linguistic information is contained in the representations of pre-trained language models. However, the mechanism of selecting the probe model has recently been subject to intense debate, as it is not clear if the probes are merely extracting information or modeling the linguistic property themselves. To address this challenge, this paper introduces a novel model-free approach to probing, by formulating probing as a prompting task. We conduct experiments on five probing tasks and show that our approach is comparable or better at extracting information than diagnostic probes while learning much less on its own. We further combine the probing via prompting approach with attention head pruning to analyze where the model stores the linguistic information in its architecture. We then examine the usefulness of a specific linguistic property for pre-training by removing the heads that are essential to that property and evaluating the resulting model's performance on language modeling.
https://github.com/rycolab/probing- via-prompting
Pre-trained language models such as BERT (Devlin et al., 2019), GPT-2 (Radford et al., 2019), and GPT-3 (Brown et al., 2020) have increased the performance of data-driven natural language processing (NLP) models on a wide variety of tasks. Due to their strong performance on many language-based tasks that require some linguistic understanding, it is natural to hypothesize that the models must implicitly encode some linguistic knowledge. One avenue of research that attempts to uncover the linguistic knowledge encoded in these models is called probing (Conneau et al., 2018; Alain and Bengio, 2018; Tenney et al.,
2019b; Saleh et al., 2020). A common form of probing is diagnostic probing. Under this approach a classifier is trained on top of a pre-trained language model to perform a target linguistic task, which is closely related to the linguistic property in question. The predictive accuracy of the classifier is then taken as an indicator of how much knowledge about the target linguistic property is encoded in the language model representations.
However, diagnostic probing has its limitations. An inherent challenge in the endeavor is discerning what is encoded in the pre-trained representations from what is learned by the probe itself (Zhang and Bowman, 2018; Hewitt and Liang, 2019; Pimentel et al., 2020a; Cao et al., 2021). The probe could, in principle, learn the task on top of random representations. While the probe is trained to extract linguistic properties from the representations, there is no simple way to restrain the probe from learning the task on its own during training. Previous research tackles the challenge using random model baselines (Zhang and Bowman, 2018) and control tasks (Hewitt and Liang, 2019; Pimentel et al., 2020a). Cao et al. (2021) try to create a more selective probe using pruning.
In this work, we address the above limitation with a novel probing framework that we call probing via prompting (PP). Drawing inspiration from recent work on prompting (Brown et al., 2020; Liu et al., 2021), we reformat a suite of probing tasks into question-answer pairs and instruct the model to answer the questions with a prefix (Li and Liang, 2021). In effect, prompting acts as a model-free probe. Thus, the use of prompting instead of a diagnostic probe allows us to work around the dilemma of teasing apart what the representations contain versus what the probe learns.
In the empirical part of the paper, we conduct experiments on five linguistic tasks and show that all these properties are indeed encoded in the popular pre-trained language model, GPT-2.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1144 - 1157 July 10-15, 2022 ©2022 Association for Computational Linguistics
Figure 1: Illustration of different probing paradigms. Here, we show an example of named entity labeling: classifying a given entity span into pre-defined categories.
(a) Diagnostic probing (DP). It trains a probe on top of the contextual representations of the entity span ("a day") to predict the label ("DATE").
Furthermore, we show that probing via prompting appears to lead to insightful probing results. In the language of Hewitt and Liang (2019), PP obtains high selectivity, i.e., the results show high accuracy on the target task, but, as expected, PP does not work well with random representations. These results suggest that PP learns little information on its own and the extracted linguistic properties are indeed encoded in the model.
At a high level, this work takes the position that model-free probing methods such as PP are useful for accurately and faithfully locating and identifying the linguistic properties embedded in these representations and helping us understand how neural language models process text. In contrast to diagnostic probes, which require designing random model baselines and control tasks to control for the learning ability of the probe model, model-free probes like PP are less capable of learning the linguistic task themselves, and thus can naturally be more selective than diagnostic probes.
2 Probing via Prompting
We now introduce our probing via prompting framework (PP), illustrated in Fig. 1b.
2.1 Language Models
Let  $p$  be a language model with vocabulary  $\Sigma$ . In the case of an autoregressive language model,  $^1 p$  is defined as a distribution over  $\Sigma^{*}$  that is locally normalized, i.e., for any  $\boldsymbol{w} \in \Sigma^{*}$  we decompose
$p(\boldsymbol{w})$  according to the chain rule as follows:
$$
p (\boldsymbol {w}) = p \left(w _ {1}\right) \cdot \prod_ {i = 2} ^ {| \boldsymbol {w} |} p \left(w _ {i} \mid \boldsymbol {w} _ {<   i}\right) \tag {1}
$$
Each "local" distribution  $p(w_{i} \mid w_{<i})$  is defined over  $\Sigma$ . Traditionally, language models include an EOS symbol; this means they produce a distribution over (an infinite number of) finite strings.
In this work, we focus on the case when  $p$  is a Transformer-based language model (Vaswani et al., 2017)—specifically, we take  $p$  to be an instance of GPT-2 (Radford et al., 2019). In contrast to most language models, GPT-2 dispenses with the EOS symbol and therefore yields a distribution over infinite strings. As it will be necessary for later discussion, we further introduce notation to discuss the internal workings of the Transformer. We denote the layer activations  $A^{(0)},\ldots ,A^{(L)}$ , where  $L$  is the total number of layers; the  $0^{\text{th}}$  layer corresponds to the embedding layer. Here,  $A^{(\ell)} = \left[\mathbf{a}_0^{(\ell)},\dots,\mathbf{a}_{|\boldsymbol {w}|}^{(\ell)}\right]$  denotes the activation matrix of the  $\ell^{\text{th}}$  layer and  $\mathbf{a}_i^{(\ell)}$  is the activation vector for the token at position  $i$ . The activation at the last layer is used to compute the distribution for the next token:
$$
p (w _ {i + 1} \mid \boldsymbol {w} _ {\le i}) = \operatorname {s o f t m a x} (W \mathbf {a} _ {i} ^ {(L)}) \quad (2)
$$
where  $W$  is a matrix that maps activations to logits over the vocabulary.
2.2 Edge Probing via Prompting
The edge probing framework (Tenney et al., 2019a,b) decomposes many common structured-
2This yields a distribution over the  $\omega$  language  $\Sigma^{\omega}$
<sup>1</sup>Such language models are often called causal language models to differentiate them from cloze language models.
Task	Context	Target Label
POS	x SEP brand EOS	NN
Const.	x SEP is a global brand EOS	VP
Entity	x SEP Disney EOS	ORG
SRL	x SEP is SEP The important thing about Disney EOS	ARG1
Coref.	x SEP Disney SEP it EOS	True
Table 1: Example prompt and target label for each task.  $\mathbf{x} =$  "The important thing about Disney is that it is a global brand." The continuous prefix is neglected for simplicity.
prediction tasks into a common format of multiclass classification. In an edge probing task, a sentence  $\pmb{x} \in \Sigma^{*}$  and  $\text{spans}^3 s_1, s_2$  in  $\pmb{x}$  are provided as inputs, and the goal is to select a correct label  $y$  among a set of candidate labels  $\mathcal{V}$ . We have intentionally kept the definition of  $\mathcal{V}$  abstract because it is task-specific. E.g., in named entity labeling,  $\mathcal{V}$  will be a set of entity types, whereas in semantic role labeling  $\mathcal{V}$  will be a set of semantic roles. $^4$
Now we introduce how to perform edge probing via prompting. We follow the naming convention of Schick and Schütze (2021a) and begin describing our prompting approach by introducing a pattern-verbalizer pair.
Pattern. We convert  $x, s_1, s_2$  into a string, called the pattern, as follows
$$
\boldsymbol {p} = \boldsymbol {x} \circ \operatorname {S E P} \circ \boldsymbol {s} _ {1} \circ \operatorname {S E P} \circ \boldsymbol {s} _ {2} \circ \operatorname {E O S} \tag {3}
$$
where  $\circ$  is string concatenation. Note that now  $\pmb{p} \in (\Sigma \cup \{\mathrm{SEP}, \mathrm{EOS}\})^*$ .
Verbalizer. Next, we define a verbalizer function  $\mathsf{vb}:\mathcal{Y}\to \Sigma$  that maps each label to a token. In our implementation, we introduce a distinguished symbol  $\mathrm{CLS}[y]$  for each label  $y$ . Thus, our verbalizer becomes  $\mathsf{vb}:\mathcal{Y}\rightarrow \{\mathrm{CLS}[1],\ldots ,\mathrm{CLS}[|\mathcal{Y}|]\}$ , where  $|\mathcal{Y}|$  is the number of candidate classes.
Inference. Now we augment the language model  $p$  such that every conditional probability is over  $\Sigma \cup \{\mathrm{SEP},\mathrm{EOS},\mathrm{CLS}[1],\ldots ,\mathrm{CLS}[\lvert\mathcal{V}\rvert]\}$  (instead of just  $\Sigma$ ), so we expand  $W$  correspondingly to incorporate the enlarged vocabulary. The newly added
rows in  $W$  and the embeddings of the newly added symbols are both randomly initialized from a normal distribution with a mean of 0 and a variance of 0.02 and not updated during training. To make a prediction, we select the class whose verbalizer has the highest next-token probability:
$$
\widehat {y} = \underset {y ^ {\prime} \in \mathcal {Y}} {\operatorname {a r g m a x}} p \left(w _ {| \boldsymbol {p} | + 1} = \mathrm {v b} \left(y ^ {\prime}\right) \mid \boldsymbol {w} _ {\leq | \boldsymbol {p} |}\right) \tag {4}
$$
This completes our formalization of edge probing as prompting.
2.3 Prefix Tuning
To better instruct the language model to perform the target task, we prepend the pattern  $\pmb{p}$  with a prefix that is task-specific and independent of  $\pmb{x}$ ,  $s_1$  and  $s_2$ . Intuitively, we aim to steer a pre-trained language model to generate predictions using an instructive prefix. For instance, a prefix for named entity labeling could be an additional string in  $\Sigma^*$ , e.g., "classify the named entity into the following categories: person, organization ..." However, in preliminary experiments, we found that discrete prefixes perform poorly on GPT-2—the prime object of our study in this paper. Thus, we resort to a continuous prefix (Li and Liang, 2021). The technical details of performing continuous prefix tuning in the case of a Transformer language model are given in App. A.
We empirically benchmark our pruning method against several previously proposed works.
We experiment on five tasks derived from OntoNotes 5.0 (Weischedel et al., 2013): part-of-speech tagging (POS), constituent labeling
<sup>3</sup>Spans are contiguous substrings.
4The span  $s_2$  is to be omitted for single-span tasks such as entity labeling.
While one might think that an easy solution is to directly use category names as verbalizers, category names in edge probing tasks are often out-of-vocabulary words (e.g., ARG0) that have to be decomposed into multiple sub-tokens. Thus, it is easier to simply introduce new symbols into the vocabulary for each class label.
This replicates the findings of Li and Liang (2021), who also found that discrete prefixes performed poorly when applied to GPT-2 and BART; in their report, GPT-3 was the only exception.
(const.), named entity labeling (entity), semantic role labeling (SRL), and coreference (Coref.). They are simplified versions of the original tasks in OntoNotes that are made compatible with the edge probing framework. An example for each task is shown in Tab. 1.
3.2 Diagnostic Probing
We compare PP with diagnostic probing (DP), and consider two types of diagnostic probes: a logistic regression probe (LR) and a multilayer perceptron probe (MLP). The idea behind diagnostic probing is illustrated in Fig. 1a. In this paper, we train a DP to predict a label  $y \in \mathcal{V}$  for the given span(s)  $s_1$  (and optionally  $s_2$ ) of a sentence  $x$ , taking the contextual representations of the span(s) produced by the pre-trained model as inputs.
3.2.1 Contextual Representations.
DP takes a single vector as input, which represents the span(s) of interest under the context of the sentence. In this section, we introduce how to obtain such a vector from a pre-trained model. For an input sentence  $\pmb{x}$  of length  $|x|$ , we again denote the layer activations produced by the language model as  $A^{(0)},\ldots ,A^{(L)}$  and  $A^{(\ell)} = \left[\mathbf{a}_0^{(\ell)},\dots,\mathbf{a}_{|\pmb{x}|}^{(\ell)}\right]$ . Following Peters et al. (2018a), we pool the representations of the different layers into a scalar-mixed representation as follows. We define the matrix  $A = \left[\mathbf{a}_0,\dots,\mathbf{a}_{|\pmb{x}|}\right]$  with  $\mathbf{a}_i$  computed thusly:
$$
\mathbf {a} _ {i} = \sum_ {\ell = 1} ^ {L} n _ {\mathrm {M L P}} (\ell) \cdot \mathbf {a} _ {i} ^ {(\ell)} \tag {5}
$$
where  $n_{\mathrm{MLP}}$  is a distribution over the layers  $\{1, \dots, L\}$  that is learned during training. In case a span consists of multiple tokens, the per-token vectors (either the scalar-mixed vector  $\mathbf{a}_i$  or the last layer activation  $\mathbf{a}_i^{(L)}$ ) are combined into a span representation using self-attention pooling (Lee et al., 2017). If more than one span exists, the span representations are concatenated into one.
3.2.2 Baselines
DP (LR). The first diagnostic probe we consider is a multinomial logistic regression probe that resembles the classification head in Cao et al. (2021). Following Cao et al. (2021), we compute the span representations using the activations
Task	Method	Pre-trained	Random
POS	PP	94.28	13.14
DP (MLP)	94.01	47.89
DP (LR)	89.56	38.84
Majority	12.58
Chance	2.08
Const.	PP	86.66	35.98
DP (MLP)	82.09	45.24
DP (LR)	71.32	42.67
Majority	35.66
Chance	3.33
Entity	PP	93.81	15.91
DP (MLP)	88.43	35.87
DP (LR)	87.81	29.81
Majority	15.91
Chance	5.56
SRL	PP	85.46	33.36
DP (MLP)	84.13	53.05
DP (LR)	77.43	47.99
Majority	33.36
Chance	1.52
Coref.	PP	90.54	78.33
DP (MLP)	87.05	78.33
DP (LR)	81.21	78.33
Majority	78.33
Chance	50.00
Table 2: Accuracy (%) for pre-trained GPT-2 (Pre-trained) and random GPT-2 (Random).
$A^{(L)}$  from the last layer. The span representations are directly fed into a linear layer followed by a softmax output layer.
DP (MLP). The second diagnostic probe we consider is the MLP probe introduced by Tenney et al. (2019b). Here, we use the scalar-mixed representations of  $A$  to compute span representations, which are then fed into an MLP followed by a softmax output layer.
Majority. Some tasks are highly imbalanced. For instance, over one third of the constituents (Const.) in the training set are adjective phrases (ADJP). Therefore, for reference, we implement a majority baseline that always predicts the most frequent class.
<sup>7</sup>Note that we ignore the  $0^{\text{th}}$  layer (embedding layer) for easier comparison in § 4.1.
We investigate GPT2<sub>SMALL</sub> (which we refer to as simply GPT-2 in this paper), a Transformer model with 12 layers and 117M parameters pre-trained on a dataset of 8 million web pages (Radford et al., 2019). We also examine the probes on a random model with the same architecture as pretrained GPT-2, but the parameters are randomly reset. Since the goal of probing is to inspect the information acquired during pre-training, an ideal probe should have low accuracy on the random model.
For PP, we set the prefix length to 200 virtual tokens for tasks with unary edges (POS, Const., and entity) and 20 for those with binary edges (SRL, Coref.). For DP (MLP), we use a two-layer MLP with 512 hidden units. Following Tenney et al. (2019b), we linearly project the per-token representations  $\mathbf{a}_i$  into 256 dimensions before self-attentional pooling to improve performance. All models are trained for one epoch using the Adam optimizer (Kingma and Ba, 2015). Our implementation is based on Hugging Face Transformers (Wolf et al., 2020). Experiments are conducted on 8 Titan RTX GPUs.
In our experiments, we only study English. Results may vary for other languages. The English split contains 116K/16K/12K examples in the train/development/test sets, respectively. We train on the train set, experiment on the development set, and report final results on the test set.
We compute the classification accuracy on each task and present the results in Tab. 2. We observe that when GPT-2's parameters are randomly reset, PP performs substantially worse than the two diagnostic probes. Remarkably, the accuracy of PP only exceeds a majority-class classifier by a negligible amount on POS and Const., and is even identical to a majority-class classifier on entity, SRL and Coref. On the other hand, both DP (MLP) and DP (LR) outperform the majority-class baseline on all the tasks except for Coref., where the majority-class baseline already performed exceptionally well already. This result suggests that PP learns much less about the task on its own than DP, which makes it a better probe in terms of selectivity (Hewitt and Liang, 2019).
Meanwhile, when we consider pre-trained GPT-2, PP has higher accuracy on all the five probing tasks than DP (MLP) and DP (LR). We take these
Model	Probe	POS	POSC	Δ
Pre-trained	PP	94.28	74.48	19.80
DP (MLP)	94.01	69.58	24.43
DP (LR)	89.56	48.75	40.81
Random	PP	13.14	7.66	5.48
DP (MLP)	47.89	45.71	2.18
DP (LR)	38.84	35.76	3.08
Majority	12.58	6.58
Chance	2.08	2.08
Table 3: POS and POSC accuracy of various methods on pre-trained GPT-2.
results to indicate that prompting works quite well at extracting linguistic knowledge. The fact that our more selective probe still performs well on linguistic tasks confirms that the considered linguistic information is indeed encoded in the pre-trained model.
3.5 Control Tasks
Hewitt and Liang (2019) propose control tasks to estimate a probe's selectivity in complement with random model baselines. A control task associates the inputs of a given linguistic task with random outputs. The key idea is that the control task can only be learned by the probe itself, so a selective probe should have high linguistic task accuracy but low control task accuracy. They further measure selectivity using a metric  $\Delta$  as the difference between linguistic task accuracy and control task accuracy.
In our experiments, we also create a control task, abbreviated POSC, for POS, where we randomly assign a POS tag for each distinct word. The results are shown in the first three rows in Tab. 3. To our surprise, we find that PP performs quite well on POSC, having an accuracy of  $74.48\%$ , which is only  $19.80\%$  lower than its accuracy on POS. In contrast, the  $\Delta$  metric for DP (MLP) and DP (LR) are  $24.43\%$  and  $40.81\%$  respectively. Therefore, if we were to use control tasks to measure selectivity, this result would suggest that PP is the least selective probe, which would be contradictory to our results in § 3.4, where we show the opposite.
To resolve the contradiction, we re-examine the implicit assumption behind control tasks: Randomness excludes the possibility of representations encoding the information of a control task, so that the control task accuracy can be solely attributed to the probe itself. If this were true, then the probe
Figure 2: Layer distribution  $(n)$  for PP, DP (MLP), and DP (LR).
should be able to learn the task regardless of the representations it probes. To test that, we run the same experiments on the random model. The results are shown in rows 4-6 of Tab. 3. It is clear that accuracy on POSC under all three methods drops substantially when switching to the random model, which means the accuracy of a control task depends not only on the expressivity of the probe but essentially also on the representations.
Since we found manually crafted prompts do not work well in our preliminary experiments, we resorted to prefix tuning. As a result, our prompting approach is not fully parameter-free. PP still involves learning parameters, and thus, we still run the risk of the prefix learning the target task on its own. Even though PP's performance on randomly initialized GPT-2 is barely better than that of the majority-class baseline, it is still much higher than chance, which indicates that PP still learns from the training set—for instance, it appears to learn the majority-class label. This is in line with the findings of Zhong et al. (2021) that an automatically optimized prompt can identify the majority-class label.
Further study is needed to determine why PP performs worse on the random model but equally well or even better on the pre-trained model. PP is not simply less expressive because a less expressive model should perform worse on both pretrained and random models, e.g., DP (LR) is less expressive than DP (MLP), but PP is the best on
the pre-trained model and the worst on the random model. He et al. (2022) offer an interesting insight that continuous prompts and, in particular, prefix tuning can be regarded as adapters. Therefore, a possible explanation is that the adapter modules that are interlocked with the Transformer layer are more convoluted with the information encoded in the model than an external probe on top. When the model is pre-trained, they are able to apply modifications to the latent representations and steer the model on the fly to perform various tasks (Rebuffi et al., 2017), but if the model is randomly initialized, the noise hinders the learning of the task.
Now that we have demonstrated the basic utility of PP, we attempt to use our methodology to determine where in the representations the information is encoded. Thus, following Cao et al. (2021), we search for a subnetwork that optimizes the performance on the task of interest and analyze the resulting subnetwork. Since it has been shown that different attention heads in the Transformer capture different linguistic phenomena (Clark et al., 2019; Voita et al., 2019), we prune attention heads instead of individual weights. Concretely, we use differentiable subset pruning (DSP) proposed by Li et al. (2021), which allows us to directly control the number of heads to keep. Pruning is performed jointly with prefix tuning.
Essential and Non-essential Heads. We call the  $K$  heads that survive pruning essential heads for the task, and those that are removed non-essential
Figure 3: Center of gravity for PP, DP (MLP), and DP (LR).
heads. With  $n_{\mathrm{PP}}(\ell)$ , we denote the distribution that is proportional to the number of essential heads in each layer  $\ell$  under the PP scheme. We define  $n_{\mathrm{LR}}(\ell)$  analogously.
4.1 Subnetwork Analysis
We now investigate how the essential heads of different tasks are distributed in the model. To do so, we make use of the center of gravity as a summary statistic introduced in (Tenney et al., 2019a). For any given layer distribution  $n$ , we compute the expected layer:
$$
\mathbb {E} _ {n} [ \ell ] = \sum_ {\ell = 1} ^ {L} n (\ell) \cdot \ell \tag {6}
$$
A higher center of gravity means the information for the task is encoded in higher layers. In our experiments, we keep  $K = 96$  (out of  $12 \times 12 = 144$ ) heads in GPT-2 and we report the average of 5 runs with different random seeds. Fig. 2 depicts the layer distributions and Fig. 3 reports the centers of gravity.