SlowGuess/ACL_2020_OCR huggingface dataset:
ure 3. Overall, encoding of verb information is weaker and somewhat more uniform across the sentence than encoding of noun information. BERT and ELMo both strongly encode the causative-inchoative alternation across all tokens of the sentence. For GPT this is also the most strongly encoded feature, and as with subject animacy, it is more strongly encoded on the later words than on the verb itself. For ELMo, the dynamic-stative property is the most weakly encoded property across the sentence (except on the subject noun). For BERT the verb's tense is the most weakly encoded, consistently lagging behind ELMo's encoding of verb tense. Among ELMo embeddings, the subject determiner shows surprisingly high performance in encoding of all verb properties.
Interim summary GPT shows uniform strong encoding of subject information and solid encoding of verb information on the target and subsequent words—but weak encoding of object information on the object noun. BERT and ELMo show more nuance in their distribution of the information types, with BERT heavily deprioritizing gender information, but strongly encoding animacy and maintaining rich number information for both nouns across all words. ELMo too deprioritizes object gender across tokens, but it shows strong encoding of subject gender after the subject noun, mostly strong encoding of animacy (apart from object animacy on subject words), and consistently rich encoding of number for both nouns. Encoding of verb features is generally weaker than noun features, with BERT weakest on tense, ELMo weakest on dynamic-stative, and all contextual models strongest on the causative-inchoative distinction.
7 Distance manipulation tasks
Setup Because our sentences follow a fixed structure for category-specific probing, it is possible that differences in encoding from word to word are an effect of linear distance rather than the syntactic/semantic relationships between the words. We perform a follow-up analysis inspired by a task in Zhang and Bowman (2018), in which the authors investigate the effect of distance from the target word as a factor in how richly recurrent neural networks encode syntactic information. For all of our tasks, we introduce a manipulation to change linear distances between our target and probed words, by splicing relative clauses after the subject and adjectives before the object. For example:
The lawyer found the judge.
The lawyer who was hungry found the angry and competent judge.
For reasons of space, we display only subject task results, in Figure 4. All results may be found in our GitHub repository linked in Footnote 1.
Results When we increase linear distances between words, the patterns remain similar to those observed in the five-word sentences. GPT still consistently encodes subject information on each of the tokens after the subject noun is encountered, with the exception of animacy encoding on the subject noun itself. BERT and ELMo still show strong encoding of subject number and animacy across tokens, with BERT dispreferring gender information across tokens and ELMo dispreferring gender only on subject determiner and noun. The main difference is that ELMo shows a marked drop in subject number information (and a bit of a drop in gender and animacy) on the object noun.
These results suggest that the observed strong encoding of context information is not simply a function of the proximity of the words in our five-word sentences, given that the strong encoding patterns persist over the longer distances (with the slight exception of ELMo losing some encoding on the object noun). This may indicate syntactic awareness in the models, which would be consistent with the findings of, e.g., Hewitt and Manning (2019) and Tenney et al. (2019b). The results further suggest that the contextual encoders tag information as relevant to specific categories of target words in their contexts, operating flexibly across varying linear distances with different structures.
8 Word identity tasks
Setup We aim to show whether the encoders incorporate only more coarse-grained linguistic information in their embeddings, or if encoding is fine-grained enough to memorize the embedding patterns for specific context word identities. We use a variation of the word content task from Conneau et al. (2018) and Adi et al. (2016). The goal of the original word content task is to determine whether a sentence vector representation contains a given word. We adapt this task to test the extent to which contextual embeddings can identify a neighboring word at a given position. We formulate our word identity tasks as "What is the identity of the subject" or "What is the identity of the verb",
Figure 4: Distance manipulation probing task results with subject as target word. Vertical ranges show  $95\%$  confidence intervals computed with non-parametric bootstrap. Each cluster of adjacent bars of the same shade represents the three different tested information types—from left to right: number, gender, animacy
Figure 5: Word identity task: labeling identity of subject noun. Vertical ranges show  $95\%$  confidence intervals computed with non-parametric bootstrap.
etc. As in Section 6, we probe each word position independently, using our fixed five-word sentences.
For identity classification, we use a softmax  $k$ -way classification task, similar to the word content task in Conneau et al. (2018). The classifier for this task must choose which of the  $k$  output words is in the target position of the sentence. In pre-testing, we found best overall performance with a 30-way classification, for which we present the results here. Smaller and larger  $k$  produce similar patterns of results, but performance overall decreases.
Results We display results for probing subject noun identity ("what is the identity of the subject noun") in Figure 5.
This proves to be a challenging task, but we see clear trends suggesting that our encoders pick up on word identity signals. As before, GloVe embeddings are at chance on all but the target subject noun, and GPT embeddings are at chance for tokens to the left of the subject noun, satisfying our sanity checks. On the subject noun itself, encoders show comparably high recoverability of word identity, with BERT standing out as the strongest. GPT
and ELMo see a slight boost in recoverability of subject identity on the verb, and GPT surprisingly shows the most subject identity information on the object determiner. BERT representations retain consistently strong subject identity encoding throughout the sentence, as do GPT embeddings starting with the subject noun itself—but ELMo encoding of subject identity drops off sharply on the determiners and object noun. This suggests that information about surrounding word identities is distributed fairly evenly across sentence tokens for BERT and GPT, but ELMo keeps word identity information fairly local to the word position itself.
Probing for the identity of the verb and of the object produces analogous patterns of results. In particular, GLoVe embeddings are at chance on all words but the target word, while GPT embeddings are at chance before the target word, and pattern similarly to BERT afterwards in the object identity task. BERT is strong throughout, while ELMo shows more effect of distance from the target word.
While identity classification performance here is far above chance, it is also well below  $100\%$ . It is possible that performance will increase with a
stronger classifier, but it is also likely that encoding of context information at the granularity of word identity is not practical or necessary for contextual embeddings, such that they more strongly encode relevant context word features rather than word identities themselves, as these results suggest.
The results presented here shed light on how different contextual encoders distribute information across token embeddings of a sentence. While we cannot draw strong conclusions about causal relations between model properties and the observed patterns, we can make broad connections between the two to inform future investigations.
Overall, the deeper, transformer-based architectures of BERT and GPT do not produce dramatic differences in distribution of information relative to the shallower ELMo model—the main difference observed with ELMo's shallower recurrent architecture is a bit of a drop in information (particularly number and word identity) over longer distances, where BERT and GPT retain strong encoding. This is not necessarily surprising, given the potential of the self-attention mechanism to capture long-distance connections—it is perhaps more surprising that ELMo shows so little difference overall. These patterns suggest that deeper transformer models may not be critical for encoding and distributing these types of context information, except perhaps over substantial distances.
BERT and ELMo, the models that use bidirectional context, generally pattern more similarly to each other than to GPT, particularly in strongly encoding number and animacy over gender, and encoding number strongest overall for nouns; GPT shows more uniformity in encoding noun information (at least from the subject noun). This pattern suggests that using bidirectional versus unidirectional context has more impact on distribution of context information than does depth or architecture type. GPT's poor encoding of object information relative to subject and verb information further suggests that the left-to-right architecture may prioritize earlier information over later information.
As for the two bidirectional models, what BERT's particular properties seem to give it over ELMo, beyond more robustness to distance, is slightly different selectivity—dropping subject gender information earlier than ELMo does, while keeping object animacy information at a longer
distance, and dropping verb tense information a bit more. Given BERT's generally stronger performance on downstream tasks, this suggests that BERT's masked language modeling setup, in tandem with its greater capacity to handle longer distances, allows for a more nuanced picture of how bidirectional context information should be distributed across tokens for optimal predictive power.
In this paper we have begun to tackle a key question in our understanding of the contextual embeddings on which most current state-of-the-art NLP models are founded: what is it that contextual embeddings pick up about the words in their contexts? We have introduced a novel probing approach and a suite of tasks through which we have performed systematic, fine-grained probing of contextual token embeddings for information about features of their surrounding words. We apply these tests to examine the distribution of contextual information across sentence tokens for popular contextual encoders BERT, ELMo, and GPT.
We find that each of the tested word features can be encoded in contextual embeddings for other words of the sentence, often with perfect or near-perfect recoverability. However, we see substantial variation across encoders in how robustly each information type is distributed to which tokens. Distance manipulations indicate that the observed rich contextual encoding is not an artifact of proximity between words, and probing for information about context word identities suggests a weaker encoding of identity information than of more abstract word feature information. Bidirectional context appears to impact distribution patterns more than depth or architecture, though the transformer models show more robustness to distance. Overall, these results help to clarify the patterns of distribution of context information within contextual embeddings—future work can further clarify the impact of more diverse syntactic relations between words, and of additional types of word features. We make all datasets and code available for additional testing.
We would like to thank Itamar Francez and Sam Wiseman for helpful discussion, and anonymous reviewers for their valuable feedback. This material is based upon work supported by the National Science Foundation under Award No. 1941160.
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. Proceedings of the International Conference on Learning Representations (ICLR).
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Shammur Absar Chowdhury and Roberto Zamparelli. 2018. RNN simulations of grammaticality judgments on long-distance dependencies. In Proceedings of the 27th international conference on computational linguistics, pages 133-144.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48.
Allyson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing composition in sentence vector representations. In Proceedings of the 27th International Conference on Computational Linguistics.
Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 134-139.
Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 32-42.
Yoav Goldberg. 2019. Assessing BERT's syntactic abilities. arXiv preprint arXiv:1901.05287.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GLoVe: Global vectors for word representation. In Proceedings of the 2014 conference on Empirical Methods in Natural Language Processing (EMNLP).
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018a. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018b. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227-2237.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. Proceedings of the International Conference on Learning Representations (ICLR).
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3261-3275.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler-gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211-221.
Kelly Zhang and Samuel Bowman. 2018. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.