SlowGuess/ACL_2020_OCR huggingface dataset:
Tiago Pimentel $^{\text{d}}$  Josef Valvoda $^{\text{d}}$  Rowan Hall Maudslay $^{\text{d}}$  Ran Zmigrod $^{\text{d}}$  Adina Williams $^{\text{e}}$  Ryan Cotterell $^{\text{d},\text{f}}$
$^{\mathrm{b}}$ University of Cambridge  $^{\mathrm{c}}$ Facebook AI Research  $^{\mathrm{d}}$ ETH Zürich
tp472@cam.ac.uk, jv406@cam.ac.uk, rh635@cam.ac.uk, rz279@cam.ac.uk, adinawilliams@fb.com, rdc42@cl.cam.ac.uk
The success of neural networks on a diverse set of NLP tasks has led researchers to question how much these networks actually "know" about natural language. Probes are a natural way of assessing this. When probing, a researcher chooses a linguistic task and trains a supervised model to predict annotations in that linguistic task from the network's learned representations. If the probe does well, the researcher may conclude that the representations encode knowledge related to the task. A commonly held belief is that using simpler models as probes is better; the logic is that simpler models will identify linguistic structure, but not learn the task itself. We propose an information-theoretic operationalization of probing as estimating mutual information that contradicts this received wisdom: one should always select the highest performing probe one can, even if it is more complex, since it will result in a tighter estimate, and thus reveal more of the linguistic information inherent in the representation. The experimental portion of our paper focuses on empirically estimating the mutual information between a linguistic property and BERT, comparing these estimates to several baselines. We evaluate on a set of ten typologically diverse languages often underrepresented in NLP research—plus English—totalling eleven languages. Our implementation is available in https://github.com/rycolab/info-theoretic-probing.
Neural networks are the backbone of modern state-of-the-art natural language processing (NLP) systems. One inherent by-product of training a neural network is the production of real-valued representations. Many speculate that these representations encode a continuous analogue of discrete linguistic properties, e.g., part-of-speech tags, due to the networks' impressive performance on many NLP tasks (Belinkov et al., 2017). As a result of this
speculation, one common thread of research focuses on the construction of probes, i.e., supervised models that are trained to extract the linguistic properties directly (Belinkov et al., 2017; Conneau et al., 2018; Peters et al., 2018b; Zhang and Bowman, 2018; Naik et al., 2018; Tenney et al., 2019). A syntactic probe, then, is a model for extracting syntactic properties, such as part of speech, from the representations (Hewitt and Liang, 2019).
In this work, we question what the goal of probing for linguistic properties ought to be. Informally, probing is often described as an attempt to discern how much information representations encode about a specific linguistic property. We make this statement more formal: We assert that the natural operationalization of probing is estimating the mutual information (Cover and Thomas, 2012) between a representation-valued random variable and a linguistic property-valued random variable. This operationalization gives probing a clean, information-theoretic foundation, and allows us to consider what "probing" actually means.
Our analysis also provides insight into how to choose a probe family: We show that choosing the highest-performing probe, independent of its complexity, is optimal for achieving the best estimate of mutual information (MI). This contradicts the received wisdom that one should always select simple probes over more complex ones (Alain and Bengio, 2017; Liu et al., 2019; Hewitt and Manning, 2019). In this context, we also discuss the recent work of Hewitt and Liang (2019) who proposes selectivity as a criterion for choosing families of probes. Hewitt and Liang (2019) defines selectivity as the performance difference between a probe on the target task and a control task, writing "[t]he selectivity of a probe puts linguistic task accuracy in context with the probe's capacity to memorize from word types." They further ponder: "when a probe achieves high accuracy on a linguistic task using a representation, can we conclude that the representation encodes linguistic structure, or has the probe
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609-4622 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
just learned the task?" Information-theoretically, there is no difference between learning the task and probing for linguistic structure, as we will show; thus, it follows that one should always employ the best possible probe for the task without resorting to artificial constraints.
In the experimental portion of the paper, we empirically analyze word-level part-of-speech labeling, a common syntactic probing task (Hewitt and Liang, 2019; Sahin et al., 2019), within our MI operationalization. Working on a typologically diverse set of languages (Basque, Czech, English, Finnish, Indonesian, Korean, Marathi, Tamil, Telugu, Turkish and Urdu), we show that only in five of these eleven languages do we recover higher estimates of mutual information between part-of-speech tags and BERT (Devlin et al., 2019), a common contextualized embedder, than from a control. These modest improvements suggest that most of the information needed to tag part-of-speech well is encoded at the lexical level, and does not require sentential context. Put more simply, words are not very ambiguous with respect to part of speech, a result known to practitioners of NLP (Garrette et al., 2013). We interpret this to mean that part-of-speech labeling is not a very informative probing task. We further investigate how BERT fares in dependency labeling, as analysed by Tenney et al. (2019). In this task, estimates based on BERT return more information than a type-level embedding in all analysed languages. However, our MI estimates still only show that BERT contains at most  $12\%$  more information than the control.
We also remark that operationalizing probing information-theoretically gives us a simple, but stunning result: contextual word embeddings, e.g., BERT (Devlin et al., 2019) and ELMo (Peters et al., 2018a), contain the same amount of information about the linguistic property of interest as the original sentence. This follows from the data-processing inequality under a very mild assumption. What this suggests is that, in a certain sense, probing for linguistic properties in representations may not be a well grounded enterprise at all. It also highlights the need to more formally define ease of extraction.
2 Word-Level Syntactic Probes for Contextual Embeddings
Following Hewitt and Liang (2019), we consider probes that examine syntactic knowledge in contextualized embeddings. These probes only consider a
token's embedding in isolation, and try to perform the task using only that information. Specifically, in this work, we consider part-of-speech (POS) and dependency labeling: determining a word's part of speech in a given sentence and the dependency relation for a pair of tokens joined by a dependency arc. Say we wish to determine whether the word love is a NOUN or a VERB. This task requires the sentential context for success. As an example, consider the utterance "love is blind" where, only with the context, is it clear that love is a NOUN. Thus, to do well on this task, the contextualized embeddings need to encode enough about the surrounding context to correctly guess the POS. Analogously, we need the whole sentence to know that love is the NOMINAL SUBJECT. Whereas in the sentence "greed can blind love", love is the DIRECT OBJECT.
Let  $S$  be a random variable ranging over all possible sequences of words. For the sake of this paper, we assume the vocabulary  $\mathcal{V}$  is finite and, thus, the values  $S$  can take are in  $\mathcal{V}^*$ . We write  $\mathbf{s} \in S$  as  $\mathbf{s} = s_1 \cdots s_{|\mathbf{s}|}$  for a specific sentence, where each  $s_i \in \mathcal{V}$  is a specific token in the sentence at the position  $i \in \mathbb{Z}_+$ . We also define the random variable  $W$  that ranges over the vocabulary  $\mathcal{V}$ . We define both a sentence-level random variable  $S$  and a word type-level random variable  $W$  since each will be useful in different contexts during our exposition.
Next, let  $T$  be a random variable whose possible values are the analyses  $t$  that we want to consider for token  $s_i$  in its sentential context,  $\mathbf{s} = s_1 \cdots s_i \cdots s_{|\mathbf{s}|}$ . In the discussion, we focus on predicting the part-of-speech tag of the  $i^{\text{th}}$  word  $s_i$ , but the same results apply to the dependency label of an edge between two words. We denote the set of values  $T$  can take as the set  $\mathcal{T}$ . Finally, let  $R$  be a representation-valued random variable for a token  $s_i$  derived from the entire sentence  $\mathbf{s}$ . We write  $\mathbf{r} \in \mathbb{R}^d$  for a value of  $R$ . While any given value  $\mathbf{r}$  is a continuous vector, there are only a countable number of values  $R$  can take. To see this, note there are only a countable number of sentences in  $\mathcal{V}^*$ .
Next, we assume there exists a true distribution  $p(t,\mathbf{s},i)$  over analyses  $t$  (elements of  $\mathcal{T}$ ), sentences  $\mathbf{s}$  (elements of  $\mathcal{V}^*$ ), and positions  $i$  (elements of  $\mathbb{Z}_{+}$ ). Note that the conditional distribution  $p(t\mid \mathbf{s},i)$  gives us the true distribution over analyses  $t$
In this work, we ignore the fact that the floating points have precision constraints in practice.
for the  $i^{\mathrm{th}}$  word token in the sentence s. We will augment this distribution such that  $p$  is additionally a distribution over  $\mathbf{r}$ , i.e.,
$$
p (\mathbf {r}, t, \mathbf {s}, i) = \delta (\mathbf {r} \mid \mathbf {s}, i) p (t, \mathbf {s}, i) \tag {1}
$$
where we define the augmentation as:
$$
\delta (\mathbf {r} \mid \mathbf {s}, i) = \mathbb {1} \left\{\mathbf {r} = \operatorname {B E R T} (\mathbf {s}) _ {i} \right\} \tag {2}
$$
Since contextual embeddings are a deterministic function of a sentence s, the augmented distribution in eq. (1) has no more randomness than the original—its entropy is the same. We assume the values of the random variables defined above are distributed according to this (unknown)  $p$ . While we do not have access to  $p$ , we assume the data in our corpus were drawn according to it. Note that  $W$ —the random variable over possible word types—is distributed according to
$$
p (w) = \sum_ {\mathbf {s} \in \mathcal {V} ^ {*}} \sum_ {i = 1} ^ {| \mathbf {s} |} \delta (w \mid \mathbf {s}, i) p (\mathbf {s}, i) \tag {3}
$$
where we define the deterministic distribution
$$
\delta (w \mid \mathbf {s}, i) = \mathbb {1} \left\{\mathbf {s} _ {i} = w \right\} \tag {4}
$$
2.2 Probing as Mutual Information
The task of supervised probing is an attempt to ascertain how much information a specific representation  $\mathbf{r}$  tells us about the value of  $t$ . This is naturally operationalized as the mutual information, a quantity from information theory:
$$
\mathrm {I} (T; R) = \mathrm {H} (T) - \mathrm {H} (T \mid R) \tag {5}
$$
where we define the entropy, which is constant with respect to the representations, as
$$
\mathrm {H} (T) = - \sum_ {t \in \mathcal {T}} p (t) \log p (t) \tag {6}
$$
and we define the conditional entropy as
$$
\begin{array}{l} \mathrm {H} (T \mid R) = \int p (\mathbf {r}) \mathrm {H} (T \mid R = \mathbf {r}) \mathrm {d} \mathbf {r} \tag {7} \\ = \sum_ {\mathbf {s} \in \mathcal {V} ^ {*}} \sum_ {i = 1} ^ {| \mathbf {s} |} p (\mathbf {s}, i) \mathrm {H} (T \mid R = \operatorname {B E R T} (\mathbf {s}) _ {i}) \\ \end{array}
$$
where the point-wise conditional entropy inside the sum is defined as
$$
\mathrm {H} (T \mid R = \mathbf {r}) = - \sum_ {t \in \mathcal {T}} p (t \mid \mathbf {r}) \log p (t \mid \mathbf {r}) \tag {8}
$$
Again, we will not know any of the distributions required to compute these quantities; the distributions in the formulae are marginals and conditionals of the true distribution discussed in eq. (1).
2.3 Bounding Mutual Information
The desired conditional entropy,  $\mathrm{H}(T\mid R)$  is not readily available, but with a model  $q_{\theta}(\mathbf{t}\mid \mathbf{r})$  in hand, we can upper-bound it by measuring their empirical cross entropy:
$$
\begin{array}{l} \mathrm {H} (T \mid R) := - _ {(t, \mathbf {r}) \sim p (\cdot , \cdot)} \left[ \log p (t \mid \mathbf {r}) \right] \tag {9} \\ = - \underset {(t, \mathbf {r}) \sim p (\cdot , \cdot)} {\mathbb {E}} \left[ \log \frac {p (t \mid \mathbf {r}) q _ {\boldsymbol {\theta}} (t \mid \mathbf {r})}{q _ {\boldsymbol {\theta}} (t \mid \mathbf {r})} \right] \\ = - \underset {(t, \mathbf {r}) \sim p (\cdot , \cdot)} {\mathbb {E}} \left[ \log q _ {\boldsymbol {\theta}} (t \mid \mathbf {r}) + \log \frac {p (t \mid \mathbf {r})}{q _ {\boldsymbol {\theta}} (t \mid \mathbf {r})} \right] \\ = \underbrace {\mathrm {H} _ {q _ {\boldsymbol {\theta}}} (T \mid R)} _ {\text {e s t i m a t e}} - \underbrace {\underset {\mathbf {r} \sim p (\cdot)} {\mathbb {E}} \mathrm {K L} (p (\cdot \mid \mathbf {r}) \mid | q _ {\boldsymbol {\theta}} (\cdot \mid \mathbf {r}))} _ {\text {e x p e c t e d e s t i m a t i o n e r r o r}} \\ \end{array}
$$
where  $\mathrm{H}_{q_\theta}(T\mid R)$  is the cross-entropy we obtain by using  $q_{\theta}$  to get this estimate. Since the KL divergence is always positive, we may lower-bound the desired mutual information
$$
\begin{array}{l} \mathrm {I} (T; R) := \mathrm {H} (T) - \mathrm {H} (T \mid R) \\ \geq \mathrm {H} (T) - \mathrm {H} _ {q _ {\theta}} (T \mid R) \quad (1 0) \\ \end{array}
$$
This bound gets tighter, the more similar—in the sense of the KL divergence— $q_{\theta}(\cdot \mid \mathbf{r})$  is to the true distribution  $p(\cdot \mid \mathbf{r})$ .
Bigger Probes are Better. If we accept mutual information as a natural operationalization for how much representations encode a target linguistic task (§2.2), the best estimate of that mutual information is the one where the probe  $q_{\theta}(t \mid \mathbf{r})$  is best at the target task. In other words, we want the best probe  $q_{\theta}(t \mid \mathbf{r})$  such that we get the tightest bound to the actual distribution  $p(t \mid \mathbf{r})$ . This paints the question posed in Hewitt and Liang (2019), who write
"when a probe achieves high accuracy on a linguistic task using a representation, can we conclude that the representation encodes linguistic structure, or has the probe just learned the task?"
as a false dichotomy.2 From an informationtheoretic view, we will always prefer the probe that does better at the target task, since there is no difference between learning a task and the representations encoding the linguistic structure.
2Assuming that the authors intended 'or' here as strictly non-inclusive. See Levinson (2000, 91) and Chevallier et al. (2008, 1743) on conversational implicatures from 'or'.
4611
3 Control Functions
To place the performance of a probe in perspective, Hewitt and Liang (2019) develops the notion of a control task. Inspired by this, we develop an analogue we term control functions, which are functions of the representation-valued random variable  $R$ . Similar to Hewitt and Liang (2019)'s control tasks, the goal of a control function  $\mathbf{c}(\cdot)$  is to place the mutual information  $\mathrm{I}(T;R)$  in the context of a baseline that the control function encodes. Control functions have their root in the data-processing inequality (Cover and Thomas, 2012), which states that, for any function  $\mathbf{c}(\cdot)$ , we have
$$
\operatorname {I} (T; R) \geq \operatorname {I} (T; \mathbf {c} (R)) \tag {11}
$$
In other words, information can only be lost by processing data. A common adage associated with this inequality is "garbage in, garbage out."
3.1 Type-Level Control Functions
We focus on type-level control functions in this paper. These functions have the effect of decontextualizing the embeddings, being related to the common trend of analyzing probe results in comparison to input layer embeddings (Belinkov and Glass, 2017; Liu et al., 2019; Hewitt and Manning, 2019; Tenney et al., 2019). Such functions allow us to inquire how much the contextual aspect of the contextual embeddings help the probe perform the target task. To show that we may map from contextual embeddings to the identity of the word type, we need the following assumption.
Assumption 1. Every contextualized embedding is unique, i.e., for any pair of sentences  $\mathbf{s},\mathbf{s}'\in \mathcal{V}^*$  we have  $(\mathbf{s}\neq \mathbf{s}')||(i\neq j)\Rightarrow \mathrm{BERT}(\mathbf{s})_i\neq$ $\mathrm{BERT}(\mathbf{s}')_j$  for all  $i\in \{1,\ldots |\mathbf{s}|\}$  and  $j\in$ $\{1,\dots ,|\mathbf{s}'|\}$
We note that Assumption 1 is mild. Contextualized word embeddings map words (in their context) to  $\mathbb{R}^d$ , which is an uncountably infinite space. However, there are only a countable number of sentences, which implies only a countable number of sequences of real vectors in  $\mathbb{R}^d$  that a contextualized embedder may produce. The event that any two embeddings would be the same across two distinct sentences is infinitesimally small. Assumption 1 yields the following corollary.
Corollary 1. There exists a function  $\operatorname{id} : \mathbb{R}^d \to V$  that maps a contextualized embedding to its word type. The function  $\operatorname{id}$  is not a bijection since multiple embeddings will map to the same type.
Using Corollary 1, we can show that any non-contextualized word embedding will contain no more information than a contextualized word embedding. More formally, we do this by constructing a look-up function  $\mathbf{e}: V \to \mathbb{R}^d$  that maps a word to a word embedding. This embedding may be one-hot, randomly generated ahead of time, or the output of a data-driven embedding method, e.g. fast-Text (Bojanowski et al., 2017). We can then construct a control function as the composition of the look-up function  $\mathbf{e}$  and the id function  $\mathbf{id}$ . Using the data-processing inequality, we can prove that in a word-level prediction task, any non-contextual (type level) word-embedding will contain no more information than a contextualized (token level) one, such as BERT and ELMo. Specifically, we have
$$
\begin{array}{l} \mathrm {I} (T; R) \geq \tag {12} \\ \operatorname {I} (T; \mathfrak {i d} (R)) = \operatorname {I} (T; W) \geq \operatorname {I} (T; \mathbf {e} (W)) \\ \end{array}
$$
This result<sup>4</sup> is intuitive and, perhaps, trivial—context matters information-theoretically. However, it gives us a principled foundation by which to measure the effectiveness of probes as we will show in §3.2.
3.2 How Much Information Did We Gain?
We will now quantify how much a contextualized word embedding knows about a task with respect to a specific control function  $\mathbf{c}(\cdot)$ . We term how much more information the contextualized embeddings have about a task than a control variable the gain,  $\mathcal{G}$ , which we define as
$$
\begin{array}{l} \mathcal {G} (T, R, \mathbf {c}) = \mathrm {I} (T; R) - \mathrm {I} (T; \mathbf {c} (R)) \tag {13} \\ = \mathrm {H} (T \mid \mathbf {c} (R)) - \mathrm {H} (T \mid R) \geq 0 \\ \end{array}
$$
The gain function will be our method for measuring how much more information contextualized representations have over a controlled baseline, encoded as the function  $\mathbf{c}$ . We will empirically estimate this value in §6. Interestingly enough, the gain has a straightforward interpretation.
Proposition 1. The gain function is equal to the following conditional mutual information
$$
\mathrm {I} (T; R \mid \mathbf {c} (R)) = \mathcal {G} (T, R, \mathbf {c}) \tag {14}
$$
4Note that although this result holds in theory, in practice the functions  $\mathbf{id}$  and  $\mathbf{e}(\cdot)$  might be arbitrarily hard to estimate. This is discussed in length in  $\S 4.3$
$^3$ Indeed, even if we sampled every embedding randomly from a  $d$ -dimensional Gaussian, the probability that we would ever sample the same real vector is zero.
$$
\begin{array}{l} \operatorname {I} (T; R \mid \mathbf {c} (R)) := \operatorname {I} (T; R) - \operatorname {I} (T; R; \mathbf {c} (R)) \\ = \mathrm {I} (T; R) - \mathrm {I} (T; \mathbf {c} (R)) \\ = \mathcal {G} (T, R, \mathbf {c}) \\ \end{array}
$$
The jump from the first to the second equality follows since  $R$  encodes, by construction, all the information about  $T$  provided by  $\mathbf{c}(R)$ .
Proposition 1 gives us a clear understanding of the quantity we wish to estimate: It is how much information about a task is encoded in the representations, given some control knowledge. If properly designed, this control transformation will remove information from the probed representations.
3.3 Approximating the Gain
The gain, as defined in eq. (13), is intractable to compute. In this section we derive a pair of variational bounds on  $\mathcal{G}(T,R,\mathbf{e})$  one upper and one lower. To approximate the gain, we will simultaneously minimize an upper and maximize a lowerbound on eq. (13). We begin by approximating the gain in the following manner
$$
\begin{array}{l} \mathcal {G} (T, R, \mathbf {e}) \approx \tag {15} \\ \underbrace {\mathrm {H} _ {q _ {\boldsymbol {\theta} 2}} (T \mid \mathbf {c} (R)) - \mathrm {H} _ {q _ {\boldsymbol {\theta} 1}} (T \mid R)} _ {\text {e s t i m a t e d} \mathcal {G} _ {q _ {\boldsymbol {\theta}}} (T, R, \mathbf {e})} \\ \end{array}
$$
these cross-entropies can be empirically estimated. We will assume access to a corpus  $\{(t_i,\mathbf{r}_i)\}_{i = 1}^N$  that is human-annotated for the target linguistic property; we further assume that these are samples  $(t_i,\mathbf{r}_i)\sim p(\cdot ,\cdot)$  from the true distribution. This yields a second approximation that is tractable:
$$
\mathrm {H} _ {q _ {\boldsymbol {\theta}}} (T; R) \approx - \frac {1}{N} \sum_ {i = 1} ^ {N} \log q _ {\boldsymbol {\theta}} \left(t _ {i} \mid \mathbf {r} _ {i}\right) \tag {16}
$$
This approximation is exact in the limit  $N\to \infty$  by the law of large numbers.
We note the approximation given in eq. (15) may be either positive or negative and its estimation error follows from eq. (9):
$$
\begin{array}{l} \Delta = \underset {\mathbf {r} \sim p (\cdot)} {\mathbb {E}} \operatorname {K L} (p (\cdot \mid \mathbf {r}) | | q _ {\theta 1} (\cdot \mid \mathbf {r})) \tag {17} \\ - \underset {\mathbf {r} \sim p (\cdot)} {\mathbb {E}} \operatorname {K L} (p (\cdot | \mathbf {c} (\mathbf {r})) | | q _ {\theta 2} (\cdot | \mathbf {c} (\mathbf {r}))) \\ = \mathrm {K L} _ {q _ {\boldsymbol {\theta} 1}} (T, R) - \mathrm {K L} _ {q _ {\boldsymbol {\theta} 2}} (T, \mathbf {c} (R)) \\ \end{array}
$$
where we abuse the KL notation to simplify the equation. This is an undesired behavior since
we know the gain itself is non-negative by the data-processing inequality, but we have yet to devise a remedy.
We justify the approximation in eq. (15) with a pair of variational bounds. The following two corollaries are a result of Theorem 2 in App. A.
Corollary 2. We have the following upper-bound on the gain
$$
\begin{array}{l} \mathcal {G} (T, R, \mathbf {e}) \tag {18} \\ \leq \mathcal {G} _ {q _ {\theta}} (T, R, \mathbf {e}) + \mathrm {K L} _ {q _ {\theta 1}} (T, R) \\ \end{array}
$$
Corollary 3. We have the following lower-bound on the gain
$$
\begin{array}{l} \mathcal {G} (T, R, \mathbf {e}) \tag {19} \\ \geq \mathcal {G} _ {q _ {\theta}} (T, R, \mathbf {e}) - \mathrm {K L} _ {q _ {\theta 2}} (T, \mathbf {c} (R)) \\ \end{array}
$$
The conjunction of Corollary 2 and Corollary 3 suggest a simple procedure for finding a good approximation: We choose  $q_{\theta 1}(\cdot \mid r)$  and  $q_{\theta 2}(\cdot \mid r)$  so as to minimize eq. (18) and maximize eq. (19), respectively. These distributions contain no overlapping parameters, by construction, so these two optimization routines may be performed independently. We will optimize both with a gradient-based procedure, discussed in §6.
4 Understanding Probing Information-Theoretically