Venue: NAACL 2022

# Analyzing Encoded Concepts in Transformer Language Models

 Hassan Sajjad\({}^{\diamond}\) Nadir Durrani\({}^{\diamond}\) Fahim Dalvi\({}^{\diamond}\) Firoj Alam\({}^{\diamond}\) Abdul Rafae Khan\({}^{\dagger}\) Jia Xu\({}^{\dagger}\)

{hsajjad,ndurrani,faimaduddin,fialam}@hbku.edu.qa

\({}^{\diamond}\)Qatar Computing Research Institute, HBKU Research Complex, Qatar {akhan4,jku70}@stevens.edu

\({}^{\dagger}\)School of Engineering and Science, Steven Institute of Technology, USA

###### Abstract

We propose a novel framework ConceptX, to analyze how latent concepts are encoded in representations learned within pre-trained language models. It uses clustering to discover the encoded concepts and explains them by aligning with a large set of human-defined concepts. Our analysis on seven transformer language models reveal interesting insights: i) the latent space within the learned representations overlap with different linguistic concepts to a varying degree, ii) the lower layers in the model are dominated by lexical concepts (e.g., affixation), whereas the core-linguistic concepts (e.g., morphological or syntactic relations) are better represented in the middle and higher layers, iii) some encoded concepts are multi-faceted and cannot be adequately explained using the existing human-defined concepts.1

Footnote 1: The code is available at [https://github.com/hsajjad/ConceptX](https://github.com/hsajjad/ConceptX).

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3082 - 3101

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Contextualized word representations learned in deep neural network models (DDNs) capture rich concepts making them ubiquitous for transfer learning towards downstream NLP. Despite their revolution, the blackbox nature of the deep NLP models is a major bottle-neck for their large scale adaptability. Understanding the inner dynamics of these models is important to ensure fairness, robustness, reliability and control.

A plethora of research has been carried out to probe DNNs for the linguistic knowledge (e.g. morphology, syntactic and semantic roles) captured within the learned representations. A commonly used framework to gauge how well linguistic information can be extracted from these models is the _Probing Framework_Hupkes et al. (2018), where they train an auxiliary classifier using representations as features to predict the property of interest. The performance of the classifier reflects the amount of knowledge learned within representations. To this end, the researchers have analyzed what knowledge is learned within the representations through relevant extrinsic phenomenon varying from word morphology Vylomova et al. (2016); Belinkov et al. (2017) to high level concepts such as syntactic structure Blevins et al. (2018); Marvin and Linzen (2018) and semantics Qian et al. (2016); Reif et al. (2019); Belinkov et al. (2017) or more generic properties Adi et al. (2016); Rogers et al. (2020).

In this work, we approach the representation analysis from a different angle and present a novel framework ConceptX. In contrast to relying on the prediction capacity of the representations, we analyze the latent concepts learned within these representations and how knowledge is structured, using an unsupervised method. More specifically, we question: i) do the representations encode knowledge inline with linguistic properties such as word morphology and semantics? ii) which properties dominate the overall structure in these representations? iii) does the model learn any novel concepts beyond linguistic properties? Answers to these questions reveal how deep neural network models structure language information to learn a task.

Our inspiration to use the term _concept_ comes from _"concept based explanation"_ in computer vision Kim et al. (2018); Ghorbani et al. (2019); Chen et al. (2020). Stock (2010) defined a concept as "a class containing certain objects as elements, where the objects have certain properties". We define an _encoded concept as a cluster of context-aware latent representations of words, where the representations are encoder layer outputs._

Our framework clusters contextualized representations using agglomerative hierarchical clustering Gowda and Krishna (1978). The resulting clusters represent _encoded concepts_, captured within the learned representations (Please see Figure 1 for illustration). We then use a novel align

ment function that measures the amount of overlap between _encoded concepts_ and a range of predefined categories (that we call as _human-defined concepts_ in this paper). We experimented with affixes, casing, morphological, syntactic, semantic, WordNet Miller (1995), and psycholinguistic concepts LIWC Pennebaker et al. (2001). The use of such a diverse set of human-defined concepts enables us to cover various abstractions of language. In Figure 3 we present a few examples of human-defined concepts that were aligned with the encoded concepts.

We carry out our study on seven pre-trained transformer models such as BERT Devlin et al. (2019) and XLM-RoBERTa Conneau et al. (2020), with varying optimization functions, architectural details and training data. Some notable findings emerging from our analysis are as follows:

* Shallow concepts such as lexical ngrams or suffixes are predominantly captured in the lower layers of the network.
* WordNet and psycholinguistic-based concepts LIWC are also learned in the lower layers.
* Middle and higher layers encode concepts that capture core linguistic properties such as morphology, semantics and syntax.
* Roughly 50% of the encoded concepts adhere to our suite of human-defined linguistic concepts.
* The models learn novel concepts that are multi-faceted and cannot be adequately explained using the existing human-defined concepts.

Our contributions in this paper are as follow: i) We present ConceptX, a framework that interprets encoded concepts in the learned representation by measuring their alignment to the human-defined concepts. ii) We provide a qualitative and quantitative evidence of how knowledge is structured within deep NLP models with respect to a large suite of human-defined concepts.

## 2 Related Work

Most of the work done on interpretability in deep NLP addresses two questions in particular: _(i)_ what linguistic (and non-linguistic) knowledge is learned within contextualized representations, _Concept Analysis_ and _(ii)_ how this information is utilized in the decision making process, _Attribution Analysis_Sajjad et al. (2021). The former thrives on post-hoc decomposability, where we analyze representations to uncover linguistic phenomenon that are captured as the network is trained towards any NLP task Adi et al. (2016); Conneau et al. (2018); Liu et al. (2019); Tenney et al. (2019); Belinkov et al. (2020) and the latter characterize the role of model components and input features towards a specific prediction Linzen et al. (2016); Gulordava et al. (2018); Marvin and Linzen (2018). Our work falls into the former category.

Previous studies have explored visualization methods to analyze the learned representations Karpathy et al. (2015); Kadar et al. (2017), attention heads Clark et al. (2019); Vig (2019), language compositionality Li et al. (2016) etc. A more commonly used framework analyzes representations by correlating parts of the neural network with linguistic properties, by training a classifier to predict a

Figure 1: **ConceptX: i) Extract representations from trained model, ii) Cluster the representations to obtain encoded concepts, iii) Align the concepts to human-defined concepts



feature of interest (Adi et al., 2016; Belinkov et al., 2017a; Conneau et al., 2018). Several researchers used probing classifiers for investigating the contextualized representations learned from a variety of neural language models on a variety of character-Durrani et al. (2019), word- Liu et al. (2019) or sub-sentence level Tenney et al. (2019) linguistic tasks. Rather than analyzing the representations as a whole, several researchers also explored identifying salient neurons within the model that capture different properties Dalvi et al. (2019); Durrani et al. (2020); Suau et al. (2020); Mu and Andreas (2020) or are salient for the model irrespective of the property Bau et al. (2019); Wu et al. (2020).

Our work is inline with Michael et al. (2020); Dalvi et al. (2022), who analyzed latent concepts learned in pre-trained models. Michael et al. (2020) used a binary classification task to induce latent concepts relevant to a task and showed the presence of linguistically motivated and novel concepts in the representation. However, different from them, we analyze representations in an unsupervised fashion. Dalvi et al. (2022) used human-in-the-loop to analyze latent spaces in BERT. Our framework uses human-defined concepts to automatically generate explanations for the latent concepts. This enabled us to scale our study to many transformer models.

In a similar work, Mamou et al. (2020) applied manifold analysis technique to understand the amount of information stored about object categories per unit. Our approach does away from the methodological limitations of probing framework such as complexity of the probes, effect of randomness etc Belinkov (2021). However, it is important to mention that the two frameworks are orthogonal and complement each other.

## 3 Methodology

A vector representation in the neural network model is composed of feature attributes of the input words. We group the encoded vector representations using a clustering approach discussed below. The underlying clusters, that we term as the _encoded concepts_, are then matched with the human-defined concepts using an alignment function. Formally, consider a Neural Network (NN) model \(\mathbb{M}\) with \(L\) encoder layers \(\{l_{1},l_{2},...l_{l},...,l_{L}\}\), with \(H\) hidden nodes per layer. An input sentence consisting of \(M\) words \(w_{1},w_{2},...w_{i},...,w_{M}\) is fed into a NN. For each input word \(i\), we compute the node output (after applying the activation functions) \(y^{l}_{h}(w_{i})\) of every hidden node \(h\in\{1,...,H\}\) in each layer \(l\), where \(\overrightarrow{y}^{l}(w_{i})\) is the vector representation composing the outputs of all hidden nodes in layer \(l\) for \(w_{i}\). Our goal is to cluster representations \(\overrightarrow{y}^{l}\), from a large training data to obtain _encoded concepts_. We then align these with various human-defined concepts to obtain an explanation of them to build an understanding of how these concepts are represented across the network.

### Clustering

We use agglomerative hierarchical clustering Gowda and Krishna (1978), which we found to be effective for this task. It assigns each word to a separate cluster and then iteratively combines them based on Ward's minimum variance criterion that minimizes intra-cluster variance. Distance between two representations is calculated with the squared Euclidean distance. The algorithm terminates when the required \(K\) clusters (aka encoded concepts) are formed, where \(K\) is a hyperparameter. Each encoded concept represents a latent relationship between the words present in the cluster. Appendix C presents the algorithm.

### Alignment

Now we define the alignment function between the encoded and human-defined concepts. Consider a human-defined concept as \(z\), where a function \(z(w)=z\) denotes that \(z\) is the human-defined concept of word \(w\). For example, parts-of-speech is a human-defined concept and each tag such as noun, verb etc. represents a class/label within the concept, e.g. \(z(sea)=noun\). Similarly, suffix is a human-defined concept with various suffixes representing a class, e.g. \(z(bigger)=er\). A reverse function of z is a one-to-many function that outputs a set of unique words with the given human-defined concept, i.e., \(z^{-1}(z)=\{w_{1},w_{2},\dots,w_{J}\}\), like \(z^{-1}(noun)=\{sea,tree,\dots\}\), where \(J\) is the total number of words with the human-defined concept of \(z\). Following this notation, an encoded concept is indicated as \(c\), where \(c(w)=c\) is a function of applying encoded concept on \(w\), and its reverse function outputs a set of unique words with the encoded concept of \(c\), i.e., \(c^{-1}(c)=\{w_{1},w_{2},\dots,w_{I}\}\), where \(I\) is the set size.

To align the encoded concepts with the human-defined concepts, we auto-annotate the input data that we used to get the clusters, with the human-defined concepts. We call our encoded concept ( 

to be \(\theta\)-aligned (\(\Lambda_{\theta}\)) with a human-defined concept (\(z\)) as follows:

\[\Lambda_{\theta}(z,c)=\begin{cases}1,&\text{if $\frac{\sum_{w^{\prime} c_{k}-1}\sum_{w\in c^{-1}}\delta(w,w^{\prime})}{J}\geq\theta$}\\ 0,&\text{otherwise},\end{cases}\]

where Kronecker function \(\delta(w,w^{\prime})\) is defined as

\[\delta(w,w^{\prime})=\begin{cases}1,&\text{if $w=w^{\prime}$}\\ 0,&\text{otherwise}\end{cases}\]

We compute \(c\) and \(\Lambda_{\theta}(z,c)\) for the encoder output from each layer \(l\) of a neural network. To compute a network-wise alignment, we simply average \(\theta\)-agreement over layers.

## 4 Experimental Setup

### Dataset

We used a subset of WMT News 20182 (359M tokens) dataset. We randomly selected 250k sentences from the dataset (\(\approx\)5M tokens) to train our clustering model. We discarded words with a frequency of less than 10 and selected maximum 10 occurrences of a word type.3 The final dataset consists of 25k word types with 10 contexts per word.

Footnote 2: [http://data.statmt.org/news-](http://data.statmt.org/news-) crawl/en/

Footnote 3: Our motivation to select a small subset of data and limiting the number of tokens is as follows: clustering a large number of high-dimensional vectors is computationally and memory intensive, for example 200k vectors of size 768 each) require around 400GB of CPU memory. Applying transformations (e.g., PCA) to reduce dimensionality may result in loss of information and therefore undesirable. We wanted to stay true to the original embedddding space.

### Pre-trained Models

We carried out our analysis on various 12-layered transformer models such as BERT-cased (BERT-c, Devlin et al., 2019), BERT-uncased (BERT-uc), RoBERTa Liu et al. (2019), XLNet Yang et al. (2019) and ALBERT Lan et al. (2019). We also analyzed multilingual models such as multilingual-bert-cased (mBERT) and XLM-RoBERTa (XLM-R, Conneau et al., 2020) where the embedding space is shared across many languages. This choice of models is motivated from interesting differences in their architectural designs, training data settings (cased vs. un-cased) and multilinguality.

### Clustering and Alignment

We extract contextualized representation of words by performing a forward pass over the network using the NeuroX toolkit Dalvi et al. (2019). We cluster representations in every layer into \(K\) groups. To find an optimum value of \(K\), we experimented with the ELbow Thorndike (1953) and Silhouette Rousseeuw (1987) methods. However, we did not observe reliable results (see Appendix C). Therefore, we empirically selected \(K=1000\) based on finding a decent balance between many small clusters (over-clustering) and a few large clusters (under-clustering). We found that our results are not sensitive to this parameter and generalize for different cluster settings (See Section 5.4). For the alignment between encoded and human-defined concepts, we use \(\theta=90\%\) i.e., we consider an encoded concept and a human-defined concept to be aligned, if they have at least 90% match.

### Human-defined concepts

We experiment with the various **Human-defined concepts**, which we categorize into four groups:

* _Lexical Concepts:_ Ngrams, Affixes, Casing, First and the Last Word (in a sentence)
* _Morphology and Semantics:_ POS tags Marcus et al. (1993) and SEM tags Abziantize et al. (2017)
* _Syntactic:_ Chunking tags Tjong Kim Sang and Buchholz (2000) and CCG super-tags Hockennmaier (2006)
* _Linguistic Ontologies:_ WordNet Miller (1995) and LIWC Pennebaker et al. (2001)

At various places in this paper, we also refer to Morphology, Semantics and Syntactic concepts as core-linguistic concepts. We trained BERT-based classifiers using gold-annotated training data and standard splits for each core-linguistic concepts and auto-labelled the selected news dataset using these.4

Footnote 4: Please see Appendix B for details.

## 5 Analysis

In this section, we analyze the encoded concepts by aligning them with the human-defined concepts.

### Overall Alignment

First we present to what extent the encoded concepts in the entire network align with the human-defined concepts. We compute the overall score as the percentage of the aligned encoded concepts to the human-defined concepts across layers using the function described in Section 3.2.

 

found an overall match of at least 43.6% in XLNet and at most 72.4% in XLM-R (See Table 1). Interestingly, the multilingual models (mBERT and XLM-R) found substantially higher match than the monolingual models. The inclusion of multiple languages during training causes the model to learn more linguistic properties. Note that the extent of alignment with the human-defined concept may not necessarily correlate with its overall performance. For example XLNet performs outperforms BERT on the GLUE tasks, but aligns less with the human-defined concepts compared to BERT in our results. A similar observation was made by Belinkov et al. (2020) who also found that the translation quality of an NMT model may not correlate with the amount of linguistic knowledge learned in the representation. Various factors such as: architectural design, training data, objective function, initialization, etc, play a role in training a pre-trained model. More controlled experiments are needed to understand the relationship of each factor on the performance of the model and on the linguistic learning of the model.

We further investigated per concept5 alignment to understand which human-defined concepts are better represented within the encoded concepts. Figure 2 presents the results.

Footnote 5: The first word, last word and prefix concepts showed less less than 1% alignment with the encoded concepts. We do not present their results in the interest of space.

Lexical ConceptsPre-trained models encode varying amount of lexical concepts such as casing, ngrams and suffixes. We found between 7-11% encoded concepts that align with the casing concept (title case or upper case). We observed that most of these encoded concepts consist of named entities, which were grouped together based on semantics.

Comparing suffixes and ngramsWhile affixes often have linguistic connotation (e.g., the prefix _anti_ negates the meaning of the stem and the suffix \(ies\) is used for pluralization), the ngram units that become part of the vocabulary as an artifact of statistical segmentation (e.g., using BPE Sennrich et al. (2016) or Word-piece Schuster and Nakajima (2012)) often lack any linguistic meaning. However, models learn to encode such information. We found a match ranging from 1% (BERT-cased) up to 25% (XLM-R) when comparing encoded concepts with the suffix concept. A similar pattern is observed in the case of the ngram concept (which is a superset of the suffix concept) where a staggering 48% matches were found. Figure 5 shows an ngram cluster found in layer 2 of BERT-c.6

Footnote 6: Appendix A shows more examples of the ngram, suffix, LIWC and WordNet clusters.

Morphology and SemanticsWe found that the encoded concepts based on word morphology (POS) consistently showed a higher match across all models in comparison to the other abstract concepts, aligning a quarter of the encoded concepts in the case of mBERT. The alignment with semantic concepts is relatively lower, with at most 16% match across models. This reflects that while the models learn both linguistic properties, morphological ontology is relatively preferred compared to the semantic hierarchy.

SyntacticThese concepts capture grammatical orientation of a word, for example Chunking:B-NP is a syntactic concept describing words in the beginning of a noun phrase. CCG:PP/NP is a concept

Figure 2: Average Alignment (%) between encoded concepts and human-defined concepts



in CCG super tagging, describing words that takes a noun phrase on the right and outputs a preposition phrase for example "[in[the US]]". We found relatively fewer matches, a maximum of 7% and 14% matching encoded concepts for Chunking and CCG concepts respectively. The low matches for syntactic concepts suggest that the models do not encode the same syntactic hierarchy suggested by these human-defined syntactic tasks.

Linguistic OntologiesComparing the encoded concepts with static linguistic ontologies, we found WordNet concepts to be the second most aligned concept (11-21%) with the human-defined concepts. LIWC also shows a relatively higher alignment compared to the other human-defined concepts in a few models (e.g., BERT-c). However, this observation is not consistent across models and we found a range between 5-16% matches. These results present an interesting case where several models prefer the distinction of lexical ontology over abstract linguistic concepts such as morphology. Figure 3 shows examples of encoded concepts aligned with WordNet and LIWC. We see that these concepts are built based on a semantic relationship e.g., the clusters in Figure 2(b), 2(c) and 2(d) group words based on religious, facial anatomy, and specific motion-related vocabulary respectively.

Comparing ModelsThe results of multilingual models (mBERT, XLM-R) are intriguing given that their encoded concepts are dominated by ngram-based concepts and POS concepts, and their relatively lesser alignment with the linguistic ontologies. On the contrary, several monolingual models (BERT-c, ALBERT) showed a better match with linguistic ontologies specially WordNet.

The higher number of matches to the ngram (and suffix) concepts in the multilingual models is due to the difference in subword segmentation. The subword models in XLM-R and mBERT are optimized for multiple languages, resulting in a vocabulary consisting of a large number of small ngram units. This causes the multilingual models to aggressively segment the input sequence, compared to the monolingual models7 and resulted in highly dominated ngram-based encoded concepts, especially in the lower layers. This may also explain the relatively lower match that multilingual models exhibit to the linguistic ontologies. We discuss this further in the context of layer-wise analysis in Section 5.2.

Footnote 7: In our dataset, mBERT has 13% more words after subword segmentation compared to BERT-c.

Comparing BERT cased vs. uncased, interestingly BERT-uc consistently showed higher matches for the core-linguistic concepts (See Figure 2). We speculate that in the absence of casing information, BERT-uc is forced to learn more linguistic concepts, whereas BERT-c leverages the explicit casing information to capture more semantically motivated concepts based on linguistic ontologies.

The higher matches in multilingual models in comparison to the monolingual models, and BERT-uncased in comparison to BERT-cased suggest that the training complexity is one factor that plays a role in a model's ability to learn linguistic nuances. For example, multilingual models need to optimize many languages, which is a harder task compared to learning one language. Similarly, the absence of capitalization in training data makes the learning task relatively harder for BERT-uc compared to BERT-c models, thus resulting in higher matches for BERT-uc. We speculate that the harder the training task, the more language nuances are learned by a model. Belinkov et al.Belinkov et al. (2020) made a similar observation, where they showed that the linguistic knowledge learned within the encoder-decoder representations in NMT models correlates with complexity of a language-pair involved in the task.

### Layer-wise Alignment

Now we study the alignment of human-defined concepts across layers to understand how concepts

Figure 3: Examples of BERT-c encoded concepts aligned with the human-defined concepts



evolve in the network. Figure 4 shows results for selected models.8 The y-axis is the normalized number of aligned concepts across layers.

Footnote 8: See Figure 10 in the Appendix for complete results.

Overall TrendWe observed mostly consistent patterns across models except for ALBERT, which we will discuss later in this section. We found that the shallow concepts (such as ngram and suffixes) and the linguistic ontologies (LIWC and WORDNET) are better represented in the initial layers and exhibit a downward trend in the higher layers of the network. On the contrary the core linguistic concepts (POS, Chunking, etc.) are better represented in the higher layers (layer 8-10). The last layers do not show any consistently dominating human-defined concepts considered in this work. We can generalize on these trends and hypothesize on how encoded concepts evolve in the network: the initial layers of the pretrained models, group words based on their lexical and semantic similarities where the former is an artifact of subword segmentation. With the inclusion of context and abstraction in the higher layers, these groups evolve into linguistic manifolds. The encoded concepts in the last layers are influenced by the objective function and learn concepts relevant to the task. Durrani et al. (2021) also made similar observation

Figure 4: Layer-wise concept alignment. Y-axis is the normalized number of aligned concepts. The number within brackets of each human-defined concept, e.g. Casing (166), shows the maximum layer-wise match



when analyzing linguistic concepts in pre-trained models that are fine-tuned towards different GLUE tasks.

Concept-wise TrendIn the following, we discuss different concepts in detail. As we mentioned earlier, the high presence of ngram and suffix concepts in the lower layers is due to subword segmentation. At the higher layers, the models start encoding abstract concepts, therefore get better alignment with the core linguistic concepts. Casing shows an exception to other lexical concepts and has similar trend to POS and SEM. Upon investigating we observed that the words appearing in these clusters have a hybrid connotation. For example, more than 98% of the encoded concepts that match with Casing are named entities, which explains the trend. The syntactic concepts observe peak in the higher-middle layers and a downward trend towards the end. These findings resonate with the earlier work on interpreting neural network representations for BERT. For example Liu et al. (2019) also showed that probes trained with layers 7-8 give the highest accuracy when trained towards predicting the tasks of Chunking and CCG tagging. Although here, we are targeting a slightly different question i.e. how the latent concepts are encoded within the representations and how they evolve from input to output layers of the network.

We observed a downward trend in linguistic ontologies (WordNet, LIWC) as we go from lower layers to higher layers as opposed to the core linguistic concepts (POS, CCG, etc.). This is because of the context independent nature of these concepts as opposed to the core-linguistic concepts which are annotated based on the context. The embedding layer is non-contextualized, thus shows a high match with linguistic ontologies. With the availability of context in contextualized layers, the encoded concepts evolve into context-aware groups, resulting in higher matches with core-linguistic concepts.

Comparing ModelsWhile the overall trend is consistent among BERT-uc, mBERT and XLNet (and other studied models - Figure 10 in Appendix), the models somewhat differ in the last layers: see the large drop in core-linguistic concepts such as POS and Chunking for XLNet and mBERT in comparison to BERT. This suggests that BERT retains much of the core-linguistic information at the last layers. Durrani et al. (2020) observed a similar pattern in their study, where they showed BERT to retain linguistic information deeper in the model as opposed to XLNet where it was more localized and predominantly preserved earlier in the network.

While the overall layer-wise trends of multilingual models look similar to some monolingual models (mBERT vs. XLNet in Fig 4b,c), the former's absolute layer-wise matches (numbers inside the brackets in Figure 4 e.g. Casing (166)) are generally substantially higher than the monolingual counterparts. For example, the POS and SEM matches of mBERT are 38.9% and 30% respectively which are 18% and 15% higher than BERT-uc. On the contrary, the number of matches with linguistic ontologies is often lower for multilingual models (mBERT LIWC alignment of 65 vs. BERT-uc alignment of 186). We hypothesize that the variety of training languages in terms of their morphological and syntactic structure has caused the multilingual models to learn more core-linguistic concepts in order to optimize the training task. Although, the knowledge captured within linguistic ontologies is essential, it may not be as critical to the training of the model as the linguistic concepts.

**ALBERT** showed a very different trend from the other models. Note that ALBERT shares parameters across layers while the other models have separate parameters for every layer. This explains the ALBERT results where we see relatively less variation across layers. More interestingly, the encoded concepts in the last layers of ALBERT showed presence of all human-defined concepts considered here (see the relatively smaller drop of ALBERT alignment curves in Figure 4).

### Unaligned Concepts

In Table 1 we observed that at least 27.6% (in XLM-R) and up to 56.4% (in XLNet) encoded concepts did not align with the human-defined concepts. _What concepts do these unaligned clusters contain?_ In an effort to answer this question, we analyzed these clusters and observed that many of them were **compositional concepts** that involves more than one fine-grained categories of the human defined concepts. Figure 5a shows an example of the unaligned concept which partly aligns with a semantic category (SEM:geopolitical entity) and a morphological category (POS:adjective). Similarly, Figure 5b is a verbs related to cognitive processes and Figure 5c shows an unaligned cluster that is composed of different verb forms (past, present and gerunds). The alignment with multiple human 

defined concepts can be used to generate explanations for these unaligned concepts. For example, Figure 4(a) can be aligned as a mix of geopolitical entities and adjectives. We also quantitatively verified the number of unaligned encoded concepts that can be explained using composition of different concepts (See Appendix E: Table 9) and found that a majority of the clusters can be explained using a combination of three pre-defined concepts..

Moreover, note that encoded concepts are often multifacet i.e., they represent more than one relationship. For example, the encoded concept in Figure 4(c) consists of different forms of verbs but at the same time, these verbs are semantically similar. The semantic relationship present here is not adequately captured using the human-defined concepts used in this work. These are the _novel concepts_ that require richer annotations or human-in-the-loop setup to generate adequate explanations.

### Generalization of Results

_Do the results generalize over different dataset selection and using different number of clusters?_ We ran experiments using different split of the news dataset for several models, and also performed alignment using different values of \(K\), the number of clusters. The results are consistent across the board. Please see Appendix F for details.

## 6 Conclusion

We presented ConceptX, a novel framework for analyzing the encoded concepts within deep NLP models. Our method uses unsupervised clustering to discover latent concepts within the contextualized representations and then aligned these concepts with a suite of human-defined concepts to generate explanations for them. Our results illuminate how DNNs structure language information. A few notable findings are: i) lower layers capture shallow linguistic concepts, ii) whereas the abstract linguistic concepts such as morphology and semantics are preserved higher in the network, iii) the extent of alignment varies across different models and different human-defined concepts, iv) we found that novel explanations and an improved coverage of concepts can be achieved via compositionality.

## References

* Abzianidze et al. (2017) Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hessel Haagsma, Rik van Noord, Pierre Ludmann, Duc-Duy Nguyen, and Johan Bos. 2017. The parallel meaning bank: Towards a multilingual corpus of translations annotated with compositional meaning representations. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics_, EACL '17, Valencia, Spain.
* Adi et al. (2016) Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. _arXiv preprint arXiv:1608.04207_.
* Bau et al. (2019) Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. Identifying and controlling important neurons in neural machine translation. In _International Conference on Learning Representations_.
* Belinkov (2021) Yonatan Belinkov. 2021. Probing classifiers: Promises, shortcomings, and alternatives. _CoRR_, abs/2102.12452.
* Belinkov et al. (2017a) Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017a. What do Neural Machine Translation Models Learn about Morphology? In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL)_, Vancouver. Association for Computational Linguistics.
* Belinkov et al. (2020) Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2020. On the linguistic representational power of neural machine translation models. _Computational Linguistics_, 45(1):1-57.

Figure 5: Examples of unaligned encoded concepts: (a) combination of geopolitical entities and their related adjectives, (b,c) different forms of verb with specific semantics



Yonatan Belinkov, Lluis Marquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017b Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks. In _Proceedings of the 8th International Joint Conference on Natural Language Processing (IJCNLP)_.
* Blevins et al. [2018] Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs encode soft hierarchical syntax. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 14-19, Melbourne, Australia. Association for Computational Linguistics.
* Chen et al. [2020] Zhi Chen, Yijie Bei, and Cynthia Rudin. 2020. Concept whitening for interpretable image recognition. _Nature Machine Intelligence_, 2(12):772-782.
* Clark et al. [2019] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 276-286, Florence, Italy. Association for Computational Linguistics.
* Conneau et al. [2020] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451. Association for Computational Linguistics.
* Conneau et al. [2018] Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Dalvi et al. [2019a] Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, D. Anthony Bau, and James Glass. 2019a. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI, Oral presentation)_.
* Dalvi et al. [2022] Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. 2022. Discovering latent concepts learned in BERT. In _International Conference on Learning Representations_.
* Dalvi et al. [2019b] Fahim Dalvi, Avery Nortonsmith, D. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, and James Glass. 2019b. Neurox: A toolkit for analyzing individual neurons in neural networks. In _AAAI Conference on Artificial Intelligence (AAAI)_.
* Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, Minneapolis, Minnesota. Association for Computational Linguistics.
* Durrani et al. [2019c] Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Yonatan Belinkov, and Preslav Nakov. 2019c. One size does not fit all: Comparing NMT representations of different granularities. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1504-1516, Minneapolis, Minnesota. Association for Computational Linguistics.
* Durrani et al. [2021] Nadir Durrani, Hassan Sajjad, and Fahim Dalvi. 2021. How transfer learning impacts linguistic knowledge in deep NLP models? In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 4947-4957, Online. Association for Computational Linguistics.
* Durrani et al. [2020] Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. 2020. Analyzing individual neurons in pre-trained language models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4865-4880, Online. Association for Computational Linguistics.
* Ghorbani et al. [2019] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. 2019. Towards automatic concept-based explanations. _Advances in Neural Information Processing Systems_, 32:9277-9286.
* Gowda and Krishna [1978] K Chidananda Gowda and G Krishna. 1978. Agglomerative clustering using the concept of mutual nearest neighbourhood. _Pattern recognition_, 10(2):105-112.
* Gulordava et al. [2018] Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1195-1205, New Orleans, Louisiana. Association for Computational Linguistics.
* Hockenmaier [2006] Julia Hockenmaier. 2006. Creating a CCGbank and a wide-coverage CCG lexicon for German. In _Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics_, ACL '06, pages 505-512, Sydney, Australia.
* Hupkes et al. [2018] Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure.
* Kadar et al. [2017] Akos Kadar, Grzegorz Chrupala, and Afra Alishahi. 2017. Representation of linguistic form and function in recurrent neural networks. _Computational Linguistics_, 43(4):761-780 

Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. _arXiv preprint arXiv:1506.02078_.
* Kim et al. (2018) Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In _International conference on machine learning_, pages 2668-2677. PMLR.
* Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. _ArXiv:1909.11942_.
* Li et al. (2016) Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 681-691, San Diego, California. Association for Computational Linguistics.
* Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. _Transactions of the Association for Computational Linguistics_, 4:521- 535.
* Liu et al. (2019a) Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguistics.
* Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danoji Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. _ArXiv:1907.11692_.
* Mamou et al. (2020) Jonathan Mamou, Hang Le, Miguel Del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, and Sueyeon Chung. 2020. Emergence of separable manifolds in deep language representations. In _International Conference on Machine Learning_, pages 6713-6723. PMLR.
* Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. _Computational Linguistics_, 19(2):313-330.
* Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1192-1202, Brussels, Belgium. Association for Computational Linguistics.
* Michael et al. (2020) Julian Michael, Jan A. Botha, and Ian Tenney. 2020. Asking without telling: Exploring latent ontologies in contextual representations. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6792-6812, Online. Association for Computational Linguistics.
* Miller (1995) George A Miller. 1995. Wordnet: a lexical database for english. _Communications of the ACM_, 38(11):39-41.
* Mu and Andreas (2020) Jesse Mu and Jacob Andreas. 2020. Compositional explanations of neurons. _CoRR_, abs/2006.14032.
* Pennebaker et al. (2001) James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. _Mahway: Lawrence Erlbaum Associates_, 71(2001):2001.
* Qian et al. (2016) Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016. Investigating Language Universal and Specific Properties in Word Embeddings. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1478-1488, Berlin, Germany. Association for Computational Linguistics.
* Reif et al. (2019) Emily Reif, Anna Yuan, Martin Wattenberg, Fernando B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry ofbert. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. _Transactions of the Association for Computational Linguistics_, 8:842-866.
* Rousseeuw (1987) Peter Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. _J. Comput. Appl. Math._, 20(1):53-65.
* Sajjad et al. (2021) Hassan Sajjad, Narine Kokhlikyan, Fahim Dalvi, and Nadir Durrani. 2021. Fine-grained interpretation and causation analysis in deep NLP models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials_, pages 5-10, Online. Association for Computational Linguistics.
* Schuster and Nakajima (2012) Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 5149-5152. IEEE.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
* Stock (2010) Wolfgang G Stock. 2010. Concepts and semantic relations in information science. _Journal of the American Society for Information Science and Technology_, 61(10):1951-1969 

Xavier Suau, Luca Zappella, and Nicholas Apostoloff. 2020. Finding experts in transformer models. _CoRR_, abs/2005.07647.
* Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
* Thorndike (1953) Robert L. Thorndike. 1953. Who belongs in the family. _Psychometrika_, pages 267-276.
* Sang and Buchholz (2000) Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task chunking. In _Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop_.
* Vig (2019) Jesse Vig. 2019. A multiscale visualization of attention in the transformer model. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 37-42, Florence, Italy. Association for Computational Linguistics.
* Vylomova et al. (2016) Ekaterina Vylomova, Trevor Cohn, Xuanli He, and Gholamreza Haffari. 2016. Word Representation Models for Morphologically Rich Languages in Neural Machine Translation. _arXiv preprint arXiv:1606.04217_.
* Wu et al. (2020) John Wu, Hassan Belinkov, Yonatan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2020. Similarity Analysis of Contextual Word Representation Models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_, Seattle. Association for Computational Linguistics.
* Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLnet: Generalized autoregressive pretraining for language understanding. _Advances in neural information processing systems_, 32.

 



## Appendix A Human-defined concept labels

### Lexical Concepts:

Ngrams, Affixes, Casing, First and the Last Word.

### Morphology and Semantics:

POS tags:We used the Penn Treebank POS tags discussed in Marcus et al. (1993), which consists of 36 POS tags and 12 other tags (i.e., punctuation and currency symbols). In Table 2, we provide POS tags and their description.

SEM tags:Abzianidze et al. (2017) consists of 73 sem-tags grouped into 13 meta-tags. In Table 3, we provide a detailed information of the tagset, and in Table 5, we provide fine and coarse tags mapping.

### Syntactic:

Chunking tags:For Chunking we used the tagset discussed in Tjong Kim Sang and Buchholz (2000), which consists of 11 tags as follows: NP (Noun phrase), VP (Verb phrase), PP (Prepositional phrase), ADVP (Adverb phrase), SBAR (Subordinate phrase), ADJP (Advective phrase), PRT (Particles), CONJP (Conjunction), INTJ (Interjection), LST (List marker), UCP (Unlike coordinate phrase). For the annotation, chunks are represented using IOB format, which results in 22 tags in the dataset as reported in Table 4.

CCG super-tagsHockenmaier (2006) developed, CCGbank, a dataset with Combinatory Categorial Grammar (CCG) derivations and dependency structures from the Penn Treebank. CCG is a lexicalized grammar formalism, which is expressive and efficiently parseable. It consists of 1272 tags.

### Linguistic Ontologies:

WordNet:Miller (1995) consists of 26 lexicographic senses for nouns, 2 for adjectives, and 1 for adverbs. Each of them represent a supersense and a hierarchy can be formed from hypernym to hyponym.

LIWC:Over the past few decades, Pennebaker et al. Pennebaker et al. (2001) have designed psycholinguistic concepts using high frequency words. These word categories are mostly used to study gender, age, personality, and health to estimate the correlation between these attributes and word usage. It is a knowledge-based system where words are mapped different high level concepts.

## Appendix B BERT-based Sequence Tagger

We trained a BERT-based sequence tagger to auto-annotate our training data. We used standard splits for training, development and test data for the 4 linguistic tasks (POS, SEM, Chunking and CCG super tagging) that we used to carry out our analysis on. The splits to preprocess the data are avail

\begin{table}
\begin{tabular}{l l l} \# & **Tag** & **Description** \\ \hline
1 & CC & Coordinating conjunction \\
2 & CD & Cardinal number \\
3 & DT & Determiner \\
4 & EX & Existential there \\
5 & FW & Foreign word \\
6 & IN & Preposition or subordinating conjunction \\
7 & JJ & Adjective \\
8 & JJR & Advjective, comparative \\
9 & JS & Adjective, superlative \\
10 & LS & List item marker \\
11 & MD & Modal \\
12 & NN & Noun, singular or mass \\
13 & NNS & Noun, plural \\
14 & NNP & Proper noun, singular \\
15 & NUPS & Proper noun, plural \\
16 PDT & Predeterminer \\
17 POS & Possessive ending \\
18 PRP & Personal pronoun \\
19 PRPS & Possessive pronoun \\
20 RB & Adverb, comparative \\
21 RBR & Adverb, superlative \\
22 RBS & Adverb, superlative \\
23 RP & Particle \\
24 SYM & Symbol \\
25 TO & to \\
26 UH & Interjection \\
27 VB & Verb, base form \\
28 VBD & Verb, yeast tense \\
29 VBG & Verb, gerunt or present participle \\
30 VBN & Verb, past participle \\
31 VBP & Verb, non-3rd person singular present \\
32 VBZ & Verb, 3rd person singular present \\
33 WDT & Wh-determiner \\
34 WP & Wh-pronoun \\
35 WPS & Possessive wh-pronoun \\
36 WRB & Wh-adverb \\
37 \# & Found sign \\
38 S & Dollar sign \\
39 & Sentence-final punctuation \\
40 , & Comma \\
41 : & Colon, semi-colon \\
42 ( & Left bracket character \\
43 ) & Right bracket character \\
44 * & Straight double quote \\
45 ’ & Left open single quote \\
46 * & Left open double quote \\
47 ’ & Right close single quote \\
48 * & Right close double quote \\ \multicolumn{3}{c}{} \\ \end{tabular}
\end{table}
Table 2: Penn Treebank POS tags.



able through git repository9 released with Liu et al. (2019). See Table 4 for statistics and classifier accuracy.

Footnote 8: [https://github.com/nelson-liu/contextual-repr-analysis](https://github.com/nelson-liu/contextual-repr-analysis)

\begin{table}
\begin{tabular}{l|c c c c|c} \hline \hline Task & Train & Dev & Test & Tags & F1 \\ \hline POS & 36557 & 1802 & 1963 & 48 & 96.69 \\ SEM & 36928 & 5301 & 10600 & 73 & 96.22 \\ Chunking & 8881 & 1843 & 2011 & 22 & 96.91 \\ CCG & 39101 & 1908 & 2404 & 1272 & 94.90 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Data statistics (number of sentences) on training, development and test sets using in the experiments and the number of tags to be predicted

\begin{table}
\begin{tabular}{l l} \hline \hline
**ANA (anaphoric)** & **MOD (modality)** \\ \hline PRO & anaphoric \& detic pronouns: he, she, I, him \\ DEF & definite: the, IoTT, derDE \\ HAS & possessive pronoun: my, her \\ REF & reflexive \& reciprocal pron.. herself, each other \\ EMP & emphasizing pronouns: himself \\
**ACT (speech act)** & \\ GRE & greeting \& parting: hi, bye \\ ITJ & interjections, exclamations: alas, ah \\ HES & hesitation: err \\ OUE & interrogative: who, which, ? \\
**ATT (attribute)** & \\ QUC & concrete quantity: two, six million, twice \\ QUV & vague quauritics: millions, many, enough \\ COL & colour: red, crimson, light blue, chestnut brown \\ LST & intersective: open, vegetarian, quickly \\ SST & subsective: skillful surgeon, tall kid \\ PRI & privative: former, fake \\ DEG & degree: 2 meters tall, 20 years old \\ INT & intensifier: very, much, too, rather \\ REL & relation: in, on, ‘s, of, after \\ SCO & score: 3-0, grade A \\
**COM (comparative)** & \\ EQU & equative: as tall as John, whales are mammals \\ MOR & comparative positive: better, more \\ LES & comparative negative: less, worse \\ TOP & superulative positive: most, mostly \\ BOT & superrelative negative: worst, least \\ ORD & ordinal: 1st, 3rd, third \\
**UNE (unnamed entity)** & \\ CON & concept: dog, person \\ ROL & role: student, brother, prof.., victim \\ GRP & group: John \{,!\} Mary and Sam gathered, a group of people \\
**DXS (deixis)** & \\ DXP & place deixis: here, this, above \\ DXT & temporal deixis: just, later, tomorrow \\ DXD & discourse deixis: latter, former, above \\
**LOG (logical)** & \\ ALT & alternative \& repetitions: another, different, again \\ XCL & exclusive: only, just \\ NIL & empty semantics: \{.\}, to, of \\ DIS & disjunction \& exist. quantif:: a, some, any, or \\ IMP & implication: if, when, unless \\ AND & conjunction \& univ. quantified:: every, and, who, any \\ \hline \hline \end{tabular}
\end{table}
Table 3: Semantic tags.

 

Clustering details

Algorithm 1 assigns each word to a separate cluster and then iteratively combines them based on Ward's minimum variance criterion that minimizes intra-cluster variance. Distance between two vector representations is calculated with the squared Euclidean distance.

```
0:\(\overrightarrow{y}^{l}\): word representation of words Parameter:\(K\): the total number of clusters
1:for each word \(w_{i}\)do
2: assign \(w_{i}\) to cluster \(c_{i}\)
3:endfor
4:while number of clusters \(\neq K\)do
5: for each cluster pair \(c_{i}\),\(c_{i^{\prime}}\)do
6:\(d_{i,i^{\prime}}\) = inner-cluster difference of combined cluster \(c_{i}\) and \(c_{i^{\prime}}\)
7: endfor
8:\(c_{j}\),\(c_{j^{\prime}}\) = cluster pair with minimum value of \(d\)
9: merge clusters \(c_{j}\) and \(c_{j^{\prime}}\)
10:endwhile
```

**Algorithm 1** Clustering Procedure

### Selection of the number of Clusters

The Elbow curve did not show any optimum clustering point, with the increase in number of clusters the distortion score kept decreasing, resulting in over-clustering (a large number of clusters consisted of less than 5 words). The over-clustering resulted in high but wrong alignment scores e.g. consider a two word cluster having words "good" and "great". The cluster will have a successful match with "adjective" since more than 90% of the words in the cluster are adjectives. In this way, a lot of small clusters will have a successful match with many human-defined concepts and the resulting alignment scores will be high. On the other hand, Silhouette resulted in under-clustering, giving the best score at number of clusters = 10. We handled this empirically by trying several values for the number of clusters i.e., 200 to 1600 with step size 200. We selected 1000 to find a good balance with over and under clustering. We understand that this may not be the best optimal point. We presented the results of 600 and 1000 clusters to show that our findings are not sensitive to the number of clusters parameter.

## Appendix D Coarse vs. Fine-grained Categories

### Coarse vs. Fine-grained Categories

Our analysis of compositional concepts showed that several fine-grained concepts could be combined to explain an unaligned concept. For example, by combining verb categories of POS to one coarse verb category, we can align the encoded concept present in Figure 5c. To probe this more formally, we collapsed POS and SEM fine-grained concepts into coarser categories (27 POS tags and 15 SEM tags). We then recomputed the alignment with the encoded concepts. For most of the models, the alignment doubled compared to the fine-grained categorizes with at least 39% and at most 53% percent match for POS. This reflects that in several cases, models learn the coarse language hierarchy. We further questioned _how many encoded concepts can be explained using coarse human-defined concepts_. Compared to Table 1, the matches increased by at most 17 points in the case of BERT-uc. The XLM-R showed the highest matching percentage of 81%. The higher alignment suggests that most of the encoded concepts learned by pre-trained models can be explained using human-defined concepts. (See Appendix D for detailed results).

### Corase POS and SEM labels

Tables 5 and 6 present results for our mapping of fine-grained SEM and POS tags into coarser categories.

 

the alignment doubles in most of the cases which reflects that in several cases, models learn the coarse language hierarchy. However, they do not strictly adhere to fine-grained categories existed in human-defined concepts. We further extend the alignment of coarse POS and SEM categories to the overall alignment with the human-defined concepts. Table 8 presents the results. We see a match of up to 81% in the case of XLM-R. The high alignment suggests that many of the encoded concepts can be explained using coarse human-defined concepts.

## Appendix E Compositional Coverage

Table 9 shows the amount of coverage we obtain when aligning with the morphological concepts when allowing 90% of the words in the cluster to be from \(N\) concepts.

## Appendix F Robustness of Methodology across Datasets and Settings

Figure 8 shows the layer-wise patterns using 600 clusters instead of 1000 as used in the main paper. We observe that the overall trends largely remain the same.

To further demonstrate the robustness of our method with respect to dataset, we sub-sampled another dataset from the News corpus with a different vocabulary by selecting words that appear between 2 to 10 times in the corpus. Note that the selection of vocabulary is due to the memory and computation limitations. Figure 9 shows the results using this selection of data. Compared to Figure 4, we can see that the overall patterns are largely similar and confirms the robustness of our findings. The slight difference in the patterns of WordNet and LIWC are due to the large selection of proper nouns in the second set of the data.

## Appendix G Layer-wise results

Figure 10 present layer-wise results for all the understudied models.

 



\begin{table}
\begin{tabular}{c|c c c c c c c} \hline \hline Concepts & BERT-c & BERT-uc & mBERT & XLM-R & RoBERTa & ALBERT & XLNet \\ \hline
1 & 13\% & 16\% & 26\% & 24\% & 18\% & 17\% & 17\% \\
2 & 11\% & 12\% & 20\% & 23\% & 13\% & 13\% & 12\% \\
3 & 14\% & 13\% & 14\% & 18\% & 11\% & 15\% & 9\% \\
4 & 6\% & 6\% & 4\% & 4\% & 5\% & 5\% & 3\% \\
5 & 2\% & 1\% & 1\% & 1\% & 2\% & 1\% & 1\% \\
6 & 1\% & 0\% & 0\% & 0\% & 1\% & 1\% & 0\% \\ \hline \hline  & & & & & & & \\ \end{tabular}
\end{table}
Table 9: Percentage of alignment when an encoded concept is composed of morphological concepts. As can be seen, most concepts are composed of either 1, 2 or 3 morphological concepts, showing that several concepts learned by these models are indeed compositional in nature.

Figure 6: Example clusters: (a) ngram:ace, (b) POS:CD, (c) Chunking:B-VP + Sufffix:ed

Figure 7: Example clusters: (a) LIWC:cause, (b) WORDNET:verb.cognition, (c) WORDNET:noun.artifact

* [16] M. C. B. K. Wilson, J. R. L. Swanson, and A. S. Zweig. _The New York: The Case of the Study of the Performance of a Large Scale Model_. PhD thesis, University of California, Berkeley, CA (2009).
* [17] D. W. Anderson, P. G. Smith, and T. H. Lee. _A new approach to the design of the large scale model_. PhD thesis, University of California, Berkeley, CA (2008).

[MISSING_PAGE_POST]

 

Figure 9: Layer-wise results on a separately sampled dataset.

 

Figure 10: Layer view results.