SlowGuess/ACL_2020_OCR huggingface dataset:
In §3, we developed an information-theoretic framework for thinking about probing contextual word embeddings for linguistic structure. However, we now cast doubt on whether probing makes sense as a scientific endeavour. We prove in §4.1 that contextualized word embeddings, by construction, contain no more information about a word-level syntactic task than the original sentence itself. Nevertheless, we do find a meaningful scientific interpretation of control functions. We expound upon this in §4.2, arguing that control functions are useful, not for understanding representations, but rather for understanding the influence of sentential context on word-level syntactic tasks, e.g., labeling words with their part of speech.
4.1 You Know Nothing, BERT
To start, we note the following corollary
Corollary 4. It directly follows from Assumption 1 that BERT is a bijection between sentences  $\mathbf{s}$  and sequences of embeddings  $\langle \mathbf{r}_1, \dots, \mathbf{r}_{|\mathbf{s}|} \rangle$ . As BERT is a bijection, it has an inverse, which we will denote as BERT $^{-1}$ .
Theorem 1. BERT(S) cannot provide more information about  $T$  than the sentence  $S$  itself.
$$
\begin{array}{l} \mathrm {I} (T; S) \geq \mathrm {I} (T; \operatorname {B E R T} (S)) \tag {20} \\ \geq \mathrm {I} (T; \operatorname {B E R T} ^ {- 1} (\operatorname {B E R T} (S))) \\ = \operatorname {I} (T; S) \\ \end{array}
$$
This implies  $\mathrm{I}(T;S) = \mathrm{I}(T;\mathrm{BERT}(S))$ . This is not a BERT-specific result—it rests on the fact that the data-processing inequality is tight for bijections.
While Theorem 1 is a straightforward application of the data-processing inequality, it has deeper ramifications for probing. It means that if we search for syntax in the contextualized word embeddings of a sentence, we should not expect to find any more syntax than is present in the original sentence. In a sense, Theorem 1 is a cynical statement: under our operationalization, the endeavour of finding syntax in contextualized embeddings sentences is nonsensical. This is because, under Assumption 1, we know the answer a priori—the contextualized word embeddings of a sentence contain exactly the same amount of information about syntax as does the sentence itself.
4.2 What Do Control Functions Mean?
Information-theoretically, the interpretation of control functions is also interesting. As previously noted, our interpretation of control functions in this work does not provide information about the representations themselves. Indeed, the same reasoning used in Corollary 1 can be used to devise a function  $\mathrm{id}_s(\mathbf{r})$  which maps a contextual representation of a token back to its sentence. For a type-level control function  $\mathbf{c}$ , by the data-processing inequality, we have that  $\mathrm{I}(T;W)\geq \mathrm{I}(T;\mathbf{c}(R))$ . Consequently, we can get an upper-bound on how much information we can get out of a decontextualized representation. If we assume we have perfect probes, then we get that the true gain function is  $\mathrm{I}(T;S) - \mathrm{I}(T;W) = \mathrm{I}(T;S\mid W)$ . This quantity is interpreted as the amount of knowledge we gain about the word-level task  $T$  by knowing  $S$  (i.e., the sentence) in addition to  $W$  (i.e., the word type). Therefore, a perfect probe provides insights about language and not about the actual representations.
4.3 Discussion: Ease of Extraction
We do acknowledge another interpretation of the work of Hewitt and Liang (2019) inter alia; BERT makes the syntactic information present in an ordered sequence of words more easily extractable. However, ease of extraction is not a trivial notion to operationalize, and indeed, we know of no attempt to do so; it is certainly more complex to determine than the number of layers in a multi-layer perceptron (MLP). Indeed, a MLP with a single hidden layer can represent any function over the unit cube, with the caveat that we may need a very large number of hidden units (Cybenko, 1989).
Although for perfect probes the above results should hold, in practice  $\mathsf{id}(\cdot)$  and  $\mathbf{c}(\cdot)$  may be hard to approximate. Furthermore, if these functions were to be learned, they might require an unreasonably large dataset. Learning a random embedding control function, for example, would require a dataset containing all words in the vocabulary  $V$  —in an open vocabulary setting an infinite dataset would be required! "Better" representations should make their respective probes easily learnable—and consequently their encoded information is more accessible (Voita and Titov, 2020).
We suggest that future work on probing should focus on operationalizing ease of extraction more rigorously—even though we do not attempt this ourselves. As previously argued by Saphra and Lopez (2019, §5), the advantage of simple probes is that they may reveal something about the structure of the encoded information—i.e., is it structured in such a way that it can be easily taken advantage of by downstream consumers of the contextualized embeddings? Many researchers who are interested in less complex probes have, either implicitly or explicitly, had this in mind.
5 A Critique of Control Tasks
We agree with Hewitt and Liang (2019)—and with both Zhang and Bowman (2018) and Tenney et al. (2019)—that we should have controlled baselines when probing for linguistic properties. However, we disagree with parts of their methodology for constructing control tasks. We present these disagreements here.
5.1 Structure and Randomness
Hewitt and Liang (2019) introduces control tasks to evaluate the effectiveness of probes. We draw
$^{6}$ Xu et al. (2020) is a possible exception.
5 Actually, Hewitt and Liang likely had an intuition about this in mind when they wrote "[a] sufficiently expressive probe with enough training data could learn any task on top of it" (Hewitt and Liang, 2019).
inspiration from this technique as evidenced by our introduction of control functions. However, we take issue with the suggestion that controls should have structure and randomness, to use the terminology from Hewitt and Liang (2019). They define structure as "the output for a word token is a deterministic function of the word type." This means that they are stripping the language of ambiguity with respect to the target task. In the case of part-of-speech labeling, love would either be a NOUN or a VERB in a control task, never both: this is a problem. The second feature of control tasks is randomness, i.e., "the output for each word type is sampled independently at random." In conjunction, structure and randomness may yield a relatively trivial task that does not look like natural language.
What is more, there is a closed-form solution for an optimal, retrieval-based "probe" that has zero learned parameters: If a word type appears in the training set, return the label with which it was annotated there, otherwise return the most frequently occurring label across all words in the training set. This probe will achieve an accuracy that is 1 minus the out-of-vocabulary rate (the number of tokens in the test set that correspond to novel types divided by the number of tokens) times the percentage of tags in the test set that do not correspond to the most frequent tag (the error rate of the guess-the-most-frequent-tag classifier). In short, the best model for a control task is a pure memorizer that guesses the most frequent tag for out-of-vocabulary words.
5.2 What's Wrong with Memorization?
Hewitt and Liang (2019) proposes that probes should be optimized to maximize accuracy and selectivity. Recall selectivity is given by the distance between the accuracy on the original task and the accuracy on the control task using the same architecture. Given their characterization of control tasks, maximising selectivity leads to a selection of a model that is bad at memorization. But why should we punish memorization? Much of linguistic competence is about generalization, however memorization also plays a key role (Fodor et al., 1974; Nooteboom et al., 2002; Fromkin et al., 2018), with word learning (Carey, 1978) being an obvious example. Indeed, maximizing selectivity as a criterion for creating probes seems to artificially disfavor this property.
5.3 What Low-Selectivity Means
Hewitt and Liang (2019) acknowledges that for
the more complex task of dependency edge prediction, a MLP probe is more accurate and, therefore, preferable despite its low selectivity. However, they offer two counter-examples where the less selective neural probe exhibits drawbacks when compared to its more selective, linear counterpart. We believe both examples are a symptom of using a simple probe rather than of selectivity being a useful metric for probe selection.
First, Hewitt and Liang (2019, §3.6) point out that, in their experiments, the MLP-1 model frequently mislabels the word with suffix -s as NNPS on the POS labeling task. They present this finding as a possible example of a less selective probe being less faithful in representing what linguistic information has the model learned. Our analysis leads us to believe that, on contrary, this shows that one should be using the best possible probe to minimize the chance of misinterpreting its encoded information. Since more complex probes achieve higher accuracy on the task, as evidence by the findings of Hewitt and Liang (2019), we believe that the overall trend of misinterpretation is higher for the probes with higher selectivity. The same applies for the second example in Hewitt and Liang 2019, §4.2 where a less selective probe appears to be less faithful. The paper shows that the representations on ELMo's second layer fail to outperform its word type ones (layer zero) on the POS labeling task when using the MLP-1 probe. While the paper argues this is evidence for selectivity being a useful metric in choosing appropriate probes, we argue that this demonstrates, yet again, that one needs to use a more complex probe to minimize the chances of misinterpreting what the model has learned. The fact that the linear probe shows a difference only demonstrates that the information is perhaps more accessible with ELMo, not that it is not present.
Despite our discussion in §4, we still wish to empirically vet our estimation technique for the gain and we use this section to highlight the need to formally define ease of extraction (as argued in §4.3). We consider the tasks of POS and dependency labeling, using the universal POS tag (Petrov et al., 2012) and dependency label information from the Universal Dependencies 2.5 (Zeman et al., 2019). We probe the multilingual release of BERT<sup>7</sup> on eleven typologically diverse languages: Basque, Czech,
<sup>7</sup>We used Wolf et al. (2019)'s implementation.
4615
Language	# Tokens	# POS	H(T)	BERT	fastText	one-hot
Train	Test	H(T | R)	H(T | c(R))	G(T, R, c)	H(T | c(R))	G(T, R, c)
Basque	72,869	24,335	15	3.17	0.36	0.29	-0.06 (-2.0%)	0.80	0.44 (14.0%)
Czech	1,173,281	173,906	16	3.33	0.10	0.11	0.02 ( 0.5%)	0.35	0.25 ( 7.6%)
English	203,762	24,958	16	3.61	0.23	0.39	0.16 ( 4.4%)	0.64	0.41 (11.4%)
Finnish	162,584	21,078	14	3.17	0.25	0.19	-0.06 (-2.0%)	0.80	0.54 (17.1%)
Indonesian	97,495	11,779	15	3.24	0.38	0.35	-0.03 (-0.8%)	0.64	0.26 ( 8.0%)
Korean	295,899	28,234	16	3.04	0.33	0.60	0.27 ( 8.8%)	1.15	0.82 (27.0%)
Marathi	2,997	412	15	3.17	0.76	0.90	0.14 ( 4.4%)	1.49	0.74 (23.2%)
Tamil	6,329	1,988	13	3.15	0.58	0.47	-0.11 (-3.5%)	1.57	0.99 (31.4%)
Telugu	5,082	721	14	2.73	0.42	0.42	-0.00 (-0.1%)	0.93	0.51 (18.6%)
Turkish	37,769	10,023	13	3.03	0.36	0.23	-0.13 (-4.2%)	0.88	0.52 (17.1%)
Urdu	108,674	14,806	15	3.23	0.32	0.41	0.09 ( 2.8%)	0.54	0.22 ( 6.9%)
Table 1: Amount of information BERT, fastText or one-hot embeddings share with a POS probing task.  $\mathrm{H}\left( T\right)$  is estimated with a plug-in estimator from same treebanks we use to train the POS labelers.
English, Finnish, Indonesian, Korean, Marathi, Tamil, Telugu, Turkish and Urdu; and we compute the contextual representations of each sentence by feeding it into BERT and averaging the output word piece representations for each word, as tokenized in the treebank.
6.1 Control Functions
We will consider two different control functions. Each is defined as the composition  $\mathbf{c} = \mathbf{e}\circ \mathrm{i}\mathsf{d}$  with a different look-up function:
-  $\mathbf{e}_{\text{fastText}}$  returns a language specific fastText embedding (Bojanowski et al., 2017);
-  $\mathbf{e}_{\text {onehot }}$  returns a one-hot embedding.
These functions can be considered type level, as they remove the influence of context on the word.
6.2 Probe Architecture
As expounded upon above, our purpose is to achieve the best bound on mutual information we can. To this end, we employ a deep MLP as our probe. We define the probe as
$$
q _ {\boldsymbol {\theta}} (t \mid \mathbf {r}) = \tag {21}
$$
$$
\operatorname {s o f t m a x} \left(W ^ {(m)} \sigma \left(W ^ {(m - 1)} \dots \sigma \left(W ^ {(1)} \mathbf {r}\right)\right)\right)
$$
an  $m$ -layer neural network with the non-linearity  $\sigma(\cdot) = \mathrm{ReLU}(\cdot)$ . The initial projection matrix is  $W^{(1)} \in \mathbb{R}^{r_1 \times d}$  and the final projection matrix is  $W^{(m)} \in \mathbb{R}^{|T| \times r_{m-1}}$ , where  $r_i = \frac{r}{2^{i-1}}$ . The remaining matrices are  $W^{(i)} \in \mathbb{R}^{r_i \times r_{i-1}}$ , so we halve the number of hidden states in each layer. We optimize
over the hyperparameters—number of layers, hidden size, one-hot embedding size, and dropout—by using random search. For each estimate, we train 50 models and choose the one with the best validation cross-entropy. The cross-entropy in the test set is then used as our entropy estimate. For dependency labeling, we follow Tenney et al. (2019) and concatenate the embeddings for both a token and its head—i.e.  $r = [r_i; r_{\mathrm{head}(i)}]$ —as such, the initial projection matrix is actually  $W^{(1)} \in \mathbb{R}^{r_1 \times 2d}$ .
We know BERT can generate text in many languages. Here we assess how much it actually "knows" about syntax in those languages—or at least how much we can extract from it given as powerful probes as we can train. We further evaluate how much it knows above and beyond simple type-level baselines.
POS tags Table 1 presents these results, showing how much information BERT, fastText, and one-hot embeddings encode about POS tagging. We see that—in all analysed languages—type level embeddings can already capture most of the uncertainty in POS tagging. We also see that BERT only shares a small amount of extra information with the task, having small gains in all languages—BERT even presents negative gains in some of them. Although this may seem to contradict the information processing inequality, it is actually caused by the difficulty of approximating i d and  $\mathbf{c}(\cdot)$  with a finite training set—causing  $\mathrm{KL}_{q_{\theta 1}}(T\mid R)$  to be larger than  $\mathrm{KL}_{q_{\theta 2}}(T\mid \mathbf{c}(R))$ . This highlights the need to formalize ease of extraction, as discussed in §4.3.
Dependency labels As shown in Table 2, BERT improves over type-level embeddings in all lan
We initialize random embeddings at the type level, and let them train during the model's optimization. We also experiment with fixed random embeddings—results for this control are in the Appendix.
Language	# Tokens	# Classes	H(T)	BERT	fastText	one-hot
Train	Test	H(T | R)	H(T | c(R))	G(T, R, c)	H(T | c(R))	G(T, R, c)
Basque	67,578	22,575	29	4.03	0.62	0.75	0.13 ( 3.1%)	1.39	0.77 (19.0%)
Czech	1,104,787	163,770	42	4.24	0.42	0.59	0.17 ( 4.1%)	0.97	0.55 (13.1%)
English	192,042	23,019	48	4.48	0.45	1.00	0.55 (12.2%)	1.35	0.89 (19.9%)
Finnish	150,362	19,515	44	4.42	0.62	0.72	0.10 ( 2.2%)	1.77	1.15 (26.0%)
Indonesian	93,054	11,223	30	4.16	0.77	1.13	0.36 ( 8.6%)	1.52	0.75 (18.0%)
Korean	273,436	26,079	30	4.17	0.40	0.76	0.36 ( 8.7%)	1.50	1.10 (26.4%)
Marathi	2,624	365	39	4.01	1.39	1.65	0.26 ( 6.5%)	2.26	0.87 (21.6%)
Tamil	5,929	1,869	28	3.78	1.17	1.23	0.06 ( 1.6%)	2.44	1.27 (33.7%)
Telugu	4,031	575	41	3.64	1.09	1.31	0.23 ( 6.2%)	1.85	0.76 (20.9%)
Turkish	34,120	9,046	31	3.95	1.12	1.17	0.05 ( 1.2%)	2.01	0.89 (22.4%)
Urdu	104,647	14,271	24	3.83	0.63	0.93	0.30 ( 8.0%)	1.08	0.46 (11.9%)
Table 2: Amount of information BERT, fastText or one-hot embeddings share with a dependency arc labeling task.  $\mathrm{H}\left( T\right)$  is again estimated with a plug-in estimator from same treebanks we use to train our models.
guages on this task. Nonetheless, although this is a much more context-dependent task, we see BERT-based estimates reveal at most  $12\%$  more information than fastText in English, the highest resource language in our set. If we look at the lower-resource languages, in five of them the gains are of less than  $5\%$ .
Discussion When put into perspective, multilingual BERT's representations do not seem to encode much more information about syntax than a simple baseline. On POS labeling, BERT only improves upon fastText in five of the eleven analysed languages—and by small amounts (less than  $9\%$ ) when it does. Even at dependency labelling, a task considered to require more contextual knowledge, we could only decode from BERT at most (in English)  $12\%$  additional information—which again highlights the need to formalize ease of extraction.
We propose an information-theoretic operationalization of probing that defines it as the task of estimating conditional mutual information. We introduce control functions, which put in context our mutual information estimates—how much more informative are contextual representations than some knowledge judged to be trivial? We further explored our operationalization and showed that, given perfect probes, probing can only yield insights into the language itself and cannot tell us anything about the representations under investigation. Keeping this in mind, we suggest a change of focus—instead of concentrating on probe size or information, we should pursue ease of extraction going forward.
On a final note, we apply our formalization to evaluate multilingual BERT's syntactic knowledge
on a set of eleven typologically diverse languages. Although it does encode a large amount of information about syntax—more than  $76\%$  and  $65\%$ , respectively, about POS and dependency labels in all languages<sup>9</sup>—BERT only encodes at most  $12\%$  more information than a simple baseline (a type-level representation). On POS labeling, more specifically, our MI estimates based on BERT are higher than the control in less than half of the analyzed languages. This indicates that word-level POS labeling may not be ideal for contemplating the syntax contained in contextual word embeddings.
The authors would like to thank Adam Poliak and John Hewitt for several helpful suggestions.
Guillaume Alain and Yoshua Bengio. 2017. Understanding intermediate layers using linear classifier probes. In 5th International Conference on Learning Representations.
Evan Archer, II Memming Park, and Jonathan W. Pillow. 2014. Bayesian entropy estimation for countable discrete distributions. The Journal of Machine Learning Research, 15(1):2833-2868.
Yonatan Belinkov and James Glass. 2017. Analyzing hidden representations in end-to-end automatic speech recognition systems. In Advances in Neural Information Processing Systems, pages 2441-2451.
Yonatan Belinkov, Lluis Marquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017. Evaluating layers of representation in neural machine translation on part-of-speech and semantic
9This is measured as the relative difference between  $\mathrm{H}(T)$  and  $\mathrm{H}(T\mid R)$ . On average, this value is  $88\%$  and  $80\%$  on POS and dependency labels, respectively.
tagging tasks. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1-10, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135-146.
Susan Carey. 1978. The child as word learner. In Linguistic Theory and Psychological Reality. MIT Press.
Coralie Chevallier, Ira A. Noveck, Tatjana Nazir, Lewis Bott, Valentina Lanzetti, and Dan Sperber. 2008. Making disjunctions exclusive. The Quarterly Journal of Experimental Psychology, 61(11):1741-1760.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single  $\$ \& !\#\*$ vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Australia. Association for Computational Linguistics.
Thomas M. Cover and Joy A. Thomas. 2012. Elements of Information Theory. John Wiley & Sons.
George Cybenko. 1989. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4):303-314.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jerry A. Fodor, Thomas G. Bever, and Merrill F. Garrett. 1974. The Psychology of Language: An Introduction to Psycholinguistics and Generative Grammar. McGraw-Hill.
Victoria Fromkin, Robert Rodman, and Nina Hyams. 2018. An Introduction to Language. Cengage Learning.
Dan Garrette, Jason Mielens, and Jason Baldridge. 2013. Real-world semi-supervised learning of POS-taggers for low-resource languages. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 583-592, Sofia, Bulgaria. Association for Computational Linguistics.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods