SlowGuess/NAACL_2022_OCR huggingface dataset:
evolve in the network. Figure 4 shows results for selected models. $^{8}$  The y-axis is the normalized number of aligned concepts across layers.
Overall Trend We observed mostly consistent patterns across models except for ALBERT, which we will discuss later in this section. We found that the shallow concepts (such as ngram and suffixes) and the linguistic ontologies (LIWC and WORDNET) are better represented in the initial layers and exhibit a downward trend in the higher layers of the network. On the contrary the core linguistic concepts (POS, Chunking, etc.) are better repre
sented in the higher layers (layer 8-10). The last layers do not show any consistently dominating human-defined concepts considered in this work. We can generalize on these trends and hypothesize on how encoded concepts evolve in the network: the initial layers of the pretrained models, group words based on their lexical and semantic similarities where the former is an artifact of subword segmentation. With the inclusion of context and abstraction in the higher layers, these groups evolve into linguistic manifolds. The encoded concepts in the last layers are influenced by the objective function and learn concepts relevant to the task. Durrani et al. (2021) also made similar observation
See Figure 10 in the Appendix for complete results.
when analyzing linguistic concepts in pre-trained models that are fine-tuned towards different GLUE tasks.
Concept-wise Trend In the following, we discuss different concepts in detail. As we mentioned earlier, the high presence of ngram and suffix concepts in the lower layers is due to subword segmentation. At the higher layers, the models start encoding abstract concepts, therefore get better alignment with the core linguistic concepts. Casing shows an exception to other lexical concepts and has similar trend to POS and SEM. Upon investigating we observed that the words appearing in these clusters have a hybrid connotation. For example, more than  $98\%$  of the encoded concepts that match with Casing are named entities, which explains the trend. The syntactic concepts observe peak in the higher-middle layers and a downward trend towards the end. These findings resonate with the earlier work on interpreting neural network representations for BERT. For example Liu et al. (2019a) also showed that probes trained with layers 7-8 give the highest accuracy when trained towards predicting the tasks of Chunking and CCG tagging. Although here, we are targeting a slightly different question i.e. how the latent concepts are encoded within the representations and how they evolve from input to output layers of the network.
We observed a downward trend in linguistic ontologies (WordNet, LIWC) as we go from lower layers to higher layers as opposed to the core linguistic concepts (POS, CCG, etc.). This is because of the context independent nature of these concepts as opposed to the core-linguistic concepts which are annotated based on the context. The embedding layer is non-contextualized, thus shows a high match with linguistic ontologies. With the availability of context in contextualized layers, the encoded concepts evolve into context-aware groups, resulting in higher matches with core-linguistic concepts.
Comparing Models While the overall trend is consistent among BERT-uc, mBERT and XLNet (and other studied models – Figure 10 in Appendix), the models somewhat differ in the last layers: see the large drop in core-linguistic concepts such as POS and Chunking for XLNet and mBERT in comparison to BERT. This suggests that BERT retains much of the core-linguistic information at the last layers. Durrani et al. (2020) observed a similar pattern in their study, where they showed BERT to
rein retain linguistic information deeper in the model as opposed to XLNet where it was more localized and predominantly preserved earlier in the network.
While the overall layer-wise trends of multilingual models look similar to some monolingual models (mBERT vs. XLNet in Fig 4b,c), the former's absolute layer-wise matches (numbers inside the brackets in Figure 4 e.g. Casing (166)) are generally substantially higher than the monolingual counterparts. For example, the POS and SEM matches of mBERT are  $38.9\%$  and  $30\%$  respectively which are  $18\%$  and  $15\%$  higher than BERT-uc. On the contrary, the number of matches with linguistic ontologies is often lower for multilingual models (mBERT LIWC alignment of 65 vs. BERT-uc alignment of 186). We hypothesize that the variety of training languages in terms of their morphological and syntactic structure has caused the multilingual models to learn more core-linguistic concepts in order to optimize the training task. Although, the knowledge captured within linguistic ontologies is essential, it may not be as critical to the training of the model as the linguistic concepts.
ALBERT showed a very different trend from the other models. Note that ALBERT shares parameters across layers while the other models have separate parameters for every layer. This explains the ALBERT results where we see relatively less variation across layers. More interestingly, the encoded concepts in the last layers of ALBERT showed presence of all human-defined concepts considered here (see the relatively smaller drop of ALBERT alignment curves in Figure 4).
5.3 Unaligned Concepts
In Table 1 we observed that at least  $27.6\%$  (in XLM-R) and up to  $56.4\%$  (in XLNet) encoded concepts did not align with the human-defined concepts. What concepts do these unaligned clusters contain? In an effort to answer this question, we analyzed these clusters and observed that many of them were compositional concepts that involves more than one fine-grained categories of the human defined concepts. Figure 5a shows an example of the unaligned concept which partly aligns with a semantic category (SEM:geopolitical entity) and a morphological category (POS:adjective). Similarly, Figure 5b is a verbs related to cognitive processes and Figure 5c shows an unaligned cluster that is composed of different verb forms (past, present and gerunds). The alignment with multiple human-
(a) SEM:GPE+POS:JJ
(b) POS:VB\*+LIWC:cogmech
Figure 5: Examples of unaligned encoded concepts: (a) combination of geopolitical entities and their related adjectives, (b,c) different forms of verb with specific semantics
(c) POS:VB*
defined concepts can be used to generate explanations for these unaligned concepts. For example, Figure 5a can be aligned as a mix of geopolitical entities and adjectives. We also quantitatively verified the number of unaligned encoded concepts that can be explained using composition of different concepts (See Appendix E: Table 9) and found that a majority of the clusters can be explained using a combination of three pre-defined concepts..
Moreover, note that encoded concepts are often multifacet i.e., they represent more than one relationship. For example, the encoded concept in Figure 5c consists of different forms of verbs but at the same time, these verbs are semantically similar. The semantic relationship present here is not adequately captured using the human-defined concepts used in this work. These are the novel concepts that require richer annotations or human-in-the-loop setup to generate adequate explanations.
5.4 Generalization of Results
Do the results generalize over different dataset selection and using different number of clusters? We ran experiments using different split of the news dataset for several models, and also performed alignment using different values of  $K$ , the number of clusters. The results are consistent across the board. Please see Appendix F for details.
We presented ConceptX, a novel framework for analyzing the encoded concepts within deep NLP models. Our method uses unsupervised clustering to discover latent concepts within the contextualized representations and then aligned these concepts with a suite of human-defined concepts to generate explanations for them. Our results illuminate how DNNs structure language information. A few notable findings are: i) lower layers capture
shallow linguistic concepts, ii) whereas the abstract linguistic concepts such as morphology and semantics are preserved higher in the network, iii) the extent of alignment varies across different models and different human-defined concepts, iv) we found that novel explanations and an improved coverage of concepts can be achieved via compositionality.
Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hessel Haagsma, Rik van Noord, Pierre Ludmann, DucDuy Nguyen, and Johan Bos. 2017. The parallel meaning bank: Towards a multilingual corpus of translations annotated with compositional meaning representations. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL '17, pages 242-247, Valencia, Spain.
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. arXiv preprint arXiv:1608.04207.
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations.
Yonatan Belinkov. 2021. Probing classifiers: Promises, shortcomings, and alternatives. CoRR, abs/2102.12452.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017a. What do Neural Machine Translation Models Learn about Morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver. Association for Computational Linguistics.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2020. On the linguistic representational power of neural machine translation models. Computational Linguistics, 45(1):1-57.
Yonatan Belinkov, Lluís Márquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017b. Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks. In Proceedings of the 8th International Joint Conference on Natural Language Processing (IJCNLP).
Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs encode soft hierarchical syntax. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 14-19, Melbourne, Australia. Association for Computational Linguistics.
Zhi Chen, Yijie Bei, and Cynthia Rudin. 2020. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451. Association for Computational Linguistics.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL).
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, D. Anthony Bau, and James Glass. 2019a. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI, Oral presentation).
Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durani, Jia Xu, and Hassan Sajjad. 2022. Discovering latent concepts learned in BERT. In International Conference on Learning Representations.
Fahim Dalvi, Avery Nortonsmith, D. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, and James Glass. 2019b. Neurox: A toolkit for analyzing individual neurons in neural networks. In AAAI Conference on Artificial Intelligence (AAAI).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of
the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics.
Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Yonatan Belinkov, and Preslav Nakov. 2019. One size does not fit all: Comparing NMT representations of different granularities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1504-1516, Minneapolis, Minnesota. Association for Computational Linguistics.
Nadir Durrani, Hassan Sajjad, and Fahim Dalvi. 2021. How transfer learning impacts linguistic knowledge in deep NLP models? In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 4947-4957, Online. Association for Computational Linguistics.
Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. 2020. Analyzing individual neurons in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4865-4880, Online. Association for Computational Linguistics.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. 2019. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems, 32:9277-9286.
K Chidananda Gowda and G Krishna. 1978. Agglomerative clustering using the concept of mutual nearest neighbourhood. Pattern recognition, 10(2):105-112.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1195-1205, New Orleans, Louisiana. Association for Computational Linguistics.
Julia Hockenmaier. 2006. Creating a CCGbank and a wide-coverage CCG lexicon for German. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, ACL '06, pages 505-512, Sydney, Australia.
Diewuwe Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure.
Akos Kádár, Grzegorz Chrupa, and Afra Alishahi. 2017. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics, 43(4):761-780.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668-2677. PMLR.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. ArXiv:1909.11942.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 681-691, San Diego, California. Association for Computational Linguistics.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv:1907.11692.
Jonathan Mamou, Hang Le, Miguel Del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, and Sueyeon Chung. 2020. Emergence of separable manifolds in deep language representations. In International Conference on Machine Learning, pages 6713-6723. PMLR.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Brussels, Belgium. Association for Computational Linguistics.
Julian Michael, Jan A. Botha, and Ian Tenney. 2020. Asking without telling: Exploring latent ontologies in contextual representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6792-6812, Online. Association for Computational Linguistics.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41.
Jesse Mu and Jacob Andreas. 2020. Compositional explanations of neurons. CoRR, abs/2006.14032.
James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. *Mahway: Lawrence Erlbaum Associates*, 71(2001):2001.
Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016. Investigating Language Universal and Specific Properties in Word Embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1478-1488, Berlin, Germany. Association for Computational Linguistics.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Peter Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math., 20(1):53-65.
Hassan Sajjad, Narine Kokhlikyan, Fahim Dalvi, and Nadir Durrani. 2021. Fine-grained interpretation and causation analysis in deep NLP models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials, pages 5-10, Online. Association for Computational Linguistics.
Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149-5152. IEEE.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Wolfgang G Stock. 2010. Concepts and semantic relations in information science. Journal of the American Society for Information Science and Technology, 61(10):1951-1969.
Xavier Suau, Luca Zappella, and Nicholas Apostoloff. 2020. Finding experts in transformer models. CoRR, abs/2005.07647.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Robert L. Thorndike. 1953. Who belongs in the family. Psychometrika, pages 267-276.
Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task chunking. In Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop.
Jesse Vig. 2019. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37-42, Florence, Italy. Association for Computational Linguistics.
Ekaterina Vylomova, Trevor Cohn, Xuanli He, and Gholamreza Haffari. 2016. Word Representation Models for Morphologically Rich Languages in Neural Machine Translation. arXiv preprint arXiv:1606.04217.
John Wu, Hassan Belinkov, Yonatan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2020. Similarity Analysis of Contextual Word Representation Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), Seattle. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32.
A Human-defined concept labels
A.1 Lexical Concepts:
Ngrams, Affixes, Casing, First and the Last Word.
A.2 Morphology and Semantics:
POS tags: We used the Penn Treebank POS tags discussed in (Marcus et al., 1993), which consists of 36 POS tags and 12 other tags (i.e., punctuation and currency symbols). In Table 2, we provide POS tags and their description.
SEM tags: (Abzianidze et al., 2017) consists of 73 sem-tags grouped into 13 meta-tags. In Table 3, we provide a detailed information of the tagset, and in Table 5, we provide fine and coarse tags mapping.
A.3 Syntactic:
Chunking tags: For Chunking we used the tagset discussed in (Tjong Kim Sang and Buchholz, 2000), which consists of 11 tags as follows: NP (Noun phrase), VP (Verb phrase), PP (Prepositional phrase), ADVP (Adverb phrase), SBAR (Subordinate phrase), ADJP (Adjective phrase), PRT (Particles), CONJP (Conjunction), INTJ (Interjection), LST (List marker), UCP (Unlike coordinate phrase). For the annotation, chunks are represented using IOB format, which results in 22 tags in the dataset as reported in Table 4.
CCG super-tags Hockenmaier (2006) developed, CCGbank, a dataset with Combinatory Categorical Grammar (CCG) derivations and dependency structures from the Penn Treebank. CCG is a lexicalized grammar formalism, which is expressive and efficiently parseable. It consists of 1272 tags.
A.4 Linguistic Ontologies:
WordNet: (Miller, 1995) consists of 26 lexicographic senses for nouns, 2 for adjectives, and 1 for adverbs. Each of them represents a supersense and a hierarchy can be formed from hyphen to hyponym.
LIWC: Over the past few decades, Pennebaker et al. (Pennebaker et al., 2001) have designed psycholinguistic concepts using high frequency words. These word categories are mostly used to study gender, age, personality, and health to estimate the