SlowGuess/NAACL_2022_OCR huggingface dataset:
Hassan Sajjad $^{\diamond}$  Nadir Durrani $^{\diamond}$  Fahim Dalvi $^{\diamond}$  Firoj Alam $^{\diamond}$  Abdul Rafae Khan $^{\dagger}$  Jia Xu $^{\dagger}$
{hsajjad,ndurrani,faimaduddin,fialam}@hbku.edu.qa
$^{\diamond}$ Qatar Computing Research Institute, HBKU Research Complex, Qatar
{akhan4, jxu70}@stevens.edu
†School of Engineering and Science, Steven Institute of Technology, USA
We propose a novel framework ConceptX, to analyze how latent concepts are encoded in representations learned within pre-trained language models. It uses clustering to discover the encoded concepts and explains them by aligning with a large set of human-defined concepts. Our analysis on seven transformer language models reveal interesting insights: i) the latent space within the learned representations overlap with different linguistic concepts to a varying degree, ii) the lower layers in the model are dominated by lexical concepts (e.g., affixation), whereas the core-linguistic concepts (e.g., morphological or syntactic relations) are better represented in the middle and higher layers, iii) some encoded concepts are multi-faceted and cannot be adequately explained using the existing human-defined concepts.
Contextualized word representations learned in deep neural network models (DDNs) capture rich concepts making them ubiquitous for transfer learning towards downstream NLP. Despite their revolution, the blackbox nature of the deep NLP models is a major bottle-neck for their large scale adaptability. Understanding the inner dynamics of these models is important to ensure fairness, robustness, reliability and control.
A plethora of research has been carried out to probe DNNs for the linguistic knowledge (e.g. morphology, syntactic and semantic roles) captured within the learned representations. A commonly used framework to gauge how well linguistic information can be extracted from these models is the Probing Framework (Hupkes et al., 2018), where they train an auxiliary classifier using representations as features to predict the property of interest. The performance of the classifier reflects the
amount of knowledge learned within representations. To this end, the researchers have analyzed what knowledge is learned within the representations through relevant extrinsic phenomenon varying from word morphology (Vylomova et al., 2016; Belinkov et al., 2017a) to high level concepts such as syntactic structure (Blevins et al., 2018; Marvin and Linzen, 2018) and semantics (Qian et al., 2016; Reif et al., 2019; Belinkov et al., 2017b) or more generic properties (Adi et al., 2016; Rogers et al., 2020).
In this work, we approach the representation analysis from a different angle and present a novel framework ConceptX. In contrast to relying on the prediction capacity of the representations, we analyze the latent concepts learned within these representations and how knowledge is structured, using an unsupervised method. More specifically, we question: i) do the representations encode knowledge inline with linguistic properties such as word morphology and semantics? ii) which properties dominate the overall structure in these representations? iii) does the model learn any novel concepts beyond linguistic properties? Answers to these questions reveal how deep neural network models structure language information to learn a task.
Our inspiration to use the term concept comes from "concept based explanation" in computer vision (Kim et al., 2018; Ghorbani et al., 2019; Chen et al., 2020). Stock (2010) defined a concept as "a class containing certain objects as elements, where the objects have certain properties". We define an encoded concept as a cluster of context-aware latent representations of words, where the representations are encoder layer outputs.
Our framework clusters contextualized representations using agglomerative hierarchical clustering (Gowda and Krishna, 1978). The resulting clusters represent encoded concepts, captured within the learned representations (Please see Figure 1 for illustration). We then use a novel align-
The code is available at https://github.com/hsajjad/ConceptX.
Human Language Technologies, pages 3082 - 3101
Figure 1: ConceptX: i) Extract representations from trained model, ii) Cluster the representations to obtain encoded concepts, iii) Align the concepts to human-defined concepts
ment function that measures the amount of overlap between encoded concepts and a range of predefined categories (that we call as human-defined concepts in this paper). We experimented with affixes, casing, morphological, syntactic, semantic, WordNet (Miller, 1995), and psycholinguistic concepts (LIWC Pennebaker et al. (2001)). The use of such a diverse set of human-defined concepts enables us to cover various abstractions of language. In Figure 3 we present a few examples of human-defined concepts that were aligned with the encoded concepts.
We carry out our study on seven pre-trained transformer models such as BERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020), with varying optimization functions, architectural details and training data. Some notable findings emerging from our analysis are as follows:
- Shallow concepts such as lexical ngrams or suffixes are predominantly captured in the lower layers of the network.
- WordNet and psycholinguistic-based concepts (LIWC) are also learned in the lower layers.
- Middle and higher layers encode concepts that capture core linguistic properties such as morphology, semantics and syntax.
- Roughly  $50\%$  of the encoded concepts adhere to our suite of human-defined linguistic concepts.
- The models learn novel concepts that are multi-faceted and cannot be adequately explained using the existing human-defined concepts.
Our contributions in this paper are as follow: i) We present ConceptX, a framework that interprets encoded concepts in the learned representation by measuring their alignment to the human-defined concepts. ii) We provide a qualitative and quantitative evidence of how knowledge is structured within deep NLP models with respect to a large suite of human-defined concepts.
Most of the work done on interpretability in deep NLP addresses two questions in particular: (i) what linguistic (and non-linguistic) knowledge is learned within contextualized representations, Concept Analysis and (ii) how this information is utilized in the decision making process, Attribution Analysis (Sajjad et al., 2021). The former thrives on post-hoc decomposability, where we analyze representations to uncover linguistic phenomenon that are captured as the network is trained towards any NLP task (Adi et al., 2016; Conneau et al., 2018; Liu et al., 2019a; Tenney et al., 2019; Belinkov et al., 2020) and the latter characterize the role of model components and input features towards a specific prediction (Linzen et al., 2016; Gulordava et al., 2018; Marvin and Linzen, 2018). Our work falls into the former category.
Previous studies have explored visualization methods to analyze the learned representations (Karpathy et al., 2015; Kádár et al., 2017), attention heads (Clark et al., 2019; Vig, 2019), language compositionality (Li et al., 2016) etc. A more commonly used framework analyzes representations by correlating parts of the neural network with linguistic properties, by training a classifier to predict a
feature of interest (Adi et al., 2016; Belinkov et al., 2017a; Conneau et al., 2018). Several researchers used probing classifiers for investigating the contextualized representations learned from a variety of neural language models on a variety of character-Durrani et al., 2019), word- (Liu et al., 2019a) or sub-sentence level (Tenney et al., 2019) linguistic tasks. Rather than analyzing the representations as a whole, several researchers also explored identifying salient neurons within the model that capture different properties (Dalvi et al., 2019a; Durrani et al., 2020; Suau et al., 2020; Mu and Andreas, 2020) or are salient for the model irrespective of the property (Bau et al., 2019; Wu et al., 2020).
Our work is inline with (Michael et al., 2020; Dalvi et al., 2022), who analyzed latent concepts learned in pre-trained models. Michael et al. (2020) used a binary classification task to induce latent concepts relevant to a task and showed the presence of linguistically motivated and novel concepts in the representation. However, different from them, we analyze representations in an unsupervised fashion. Dalvi et al. (2022) used human-in-the-loop to analyze latent spaces in BERT. Our framework uses human-defined concepts to automatically generate explanations for the latent concepts. This enabled us to scale our study to many transformer models.
In a similar work, Mamou et al. (2020) applied manifold analysis technique to understand the amount of information stored about object categories per unit. Our approach does away from the methodological limitations of probing framework such as complexity of the probes, effect of randomness etc (Belinkov, 2021). However, it is important to mention that the two frameworks are orthogonal and complement each other.
A vector representation in the neural network model is composed of feature attributes of the input words. We group the encoded vector representations using a clustering approach discussed below. The underlying clusters, that we term as the encoded concepts, are then matched with the human-defined concepts using an alignment function. Formally, consider a Neural Network (NN) model  $\mathbb{M}$  with  $L$  encoder layers  $\{l_1,l_2,\dots l_l,\dots ,l_L\}$ , with  $H$  hidden nodes per layer. An input sentence consisting of  $M$  words  $w_{1},w_{2},\ldots w_{i},\ldots ,w_{M}$  is fed into a NN. For each input word  $i$ , we compute the node output (after applying the activation func
tions)  $y_{h}^{l}(w_{i})$  of every hidden node  $h\in \{1,\dots ,H\}$  in each layer  $l$ , where  $\vec{y}^l (w_i)$  is the vector representation composing the outputs of all hidden nodes in layer  $l$  for  $w_{i}$ . Our goal is to cluster representations  $\vec{y}^l$ , from a large training data to obtain encoded concepts. We then align these with various human-defined concepts to obtain an explanation of them to build an understanding of how these concepts are represented across the network.
3.1 Clustering
We use agglomerative hierarchical clustering (Gowda and Krishna, 1978), which we found to be effective for this task. It assigns each word to a separate cluster and then iteratively combines them based on Ward's minimum variance criterion that minimizes intra-cluster variance. Distance between two representations is calculated with the squared Euclidean distance. The algorithm terminates when the required  $K$  clusters (aka encoded concepts) are formed, where  $K$  is a hyperparameter. Each encoded concept represents a latent relationship between the words present in the cluster. Appendix C presents the algorithm.
3.2 Alignment
Now we define the alignment function between the encoded and human-defined concepts. Consider a human-defined concept as  $z$ , where a function  $z(w) = z$  denotes that  $z$  is the human-defined concept of word  $w$ . For example, parts-of-speech is a human-defined concept and each tag such as noun, verb etc. represents a class/label within the concept, e.g.  $z(sea) = noun$ . Similarly, suffix is a human-defined concept with various suffixes representing a class, e.g.  $z(bigger) = er$ . A reverse function of  $z$  is a one-to-many function that outputs a set of unique words with the given human-defined concept, i.e.,  $z^{-1}(z) = \{w_1, w_2, \ldots, w_J\}$ , like  $z^{-1}(noun) = \{sea, tree, \ldots\}$ , where  $J$  is the total number of words with the human-defined concept of  $z$ . Following this notation, an encoded concept is indicated as  $c$ , where  $c(w) = c$  is a function of applying encoded concept on  $w$ , and its reverse function outputs a set of unique words with the encoded concept of  $c$ , i.e.,  $c^{-1}(c) = \{w_1, w_2, \ldots, w_I\}$ , where  $I$  is the set size.
To align the encoded concepts with the human-defined concepts, we auto- annotate the input data that we used to get the clusters, with the human-defined concepts. We call our encoded concept  $(c)$
to be  $\theta$ -aligned  $(\Lambda_{\theta})$  with a human-defined concept  $(z)$  as follows:
$$
\Lambda_ {\theta} (z, c) = \left\{ \begin{array}{l l} 1, & \text {i f} \frac {\sum_ {w ^ {\prime} \in z ^ {- 1}} \sum_ {w \in c ^ {- 1}} \delta (w , w ^ {\prime})}{J} \geq \theta \\ 0, & \text {o t h e r w i s e}, \end{array} \right.
$$
where Kronecker function  $\delta (w,w^{\prime})$  is defined as
$$
\delta (w, w ^ {\prime}) = \left\{ \begin{array}{l l} 1, & \text {i f} w = w ^ {\prime} \\ 0, & \text {o t h e r w i s e} \end{array} \right.
$$
We compute  $c$  and  $\Lambda_{\theta}(z,c)$  for the encoder output from each layer  $l$  of a neural network. To compute a network-wise alignment, we simply average  $\theta$ -agreement over layers.
We used a subset of WMT News  $2018^{2}$  (359M tokens) dataset. We randomly selected 250k sentences from the dataset  $(\approx 5\mathrm{M}$  tokens) to train our clustering model. We discarded words with a frequency of less than 10 and selected maximum 10 occurrences of a word type. The final dataset consists of 25k word types with 10 contexts per word.
We carried out our analysis on various 12-layered transformer models such as BERT-cased (BERTc, Devlin et al., 2019), BERT-uncased (BERT-uc), RoBERTa (Liu et al., 2019b), XLNet (Yang et al., 2019) and ALBERT (Lan et al., 2019). We also analyzed multilingual models such as multilingualbert-cased (mBERT) and XLM-RoBERTa (XLMR, Conneau et al., 2020) where the embedding space is shared across many languages. This choice of models is motivated from interesting differences in their architectural designs, training data settings (cased vs. un-cased) and multilinguality.
4.3 Clustering and Alignment
We extract contextualized representation of words by performing a forward pass over the network using the NeuroX toolkit (Dalvi et al., 2019b). We
cluster representations in every layer into  $K$  groups. To find an optimum value of  $K$ , we experimented with the ELbow (Thorndike, 1953) and Silhouette (Rousseeuw, 1987) methods. However, we did not observe reliable results (see Appendix C). Therefore, we empirically selected  $K = 1000$  based on finding a decent balance between many small clusters (over-clustering) and a few large clusters (under-clustering). We found that our results are not sensitive to this parameter and generalize for different cluster settings (See Section 5.4). For the alignment between encoded and human-defined concepts, we use  $\theta = 90\%$  i.e., we consider an encoded concept and a human-defined concept to be aligned, if they have at least  $90\%$  match.
4.4 Human-defined concepts
We experiment with the various Human-defined concepts, which we categorize into four groups:
- Lexical Concepts: Ngrams, Affixes, Casing, First and the Last Word (in a sentence)
- Morphology and Semantics: POS tags (Marcus et al., 1993) and SEM tags (Abzianidze et al., 2017)
- Syntactic: Chunking tags (Tjong Kim Sang and Buchholz, 2000) and CCG super-tags (Hockenmaier, 2006)
- Linguistic Ontologies: WordNet (Miller, 1995) and LIWC (Pennebaker et al., 2001)
At various places in this paper, we also refer to Morphology, Semantics and Syntactic concepts as core-linguistic concepts. We trained BERT-based classifiers using gold-annotated training data and standard splits for each core-linguistic concepts and auto-labelled the selected news dataset using these.
In this section, we analyze the encoded concepts by aligning them with the human-defined concepts.
5.1 Overall Alignment
First we present to what extent the encoded concepts in the entire network align with the human-defined concepts. We compute the overall score as the percentage of the aligned encoded concepts to the human-defined concepts across layers using the function described in Section 3.2. We
2http://data.statmt.org/news-crawl/en/
<sup>3</sup>Our motivation to select a small subset of data and limiting the number of tokens is as follows: clustering a large number of high-dimensional vectors is computationally and memory intensive, for example 200k vectors (of size 768 each) require around 400GB of CPU memory. Applying transformations (e.g., PCA) to reduce dimensionality may result in loss of information and therefore undesirable. We wanted to stay true to the original embedding space.
4Please see Appendix B for details.
BERT-c	BERT-uc	mBERT	XLM-R	RoBERTa	ALBERT	XLNet
Overall alignment	47.2%	50.4%	66.0%	72.4%	50.1%	51.6%	43.6%
Table 1: Coverage of human-defined concepts across all clusters of a given model
Figure 2: Average Alignment  $(\%)$  between encoded concepts and human-defined concepts
found an overall match of at least  $43.6\%$  in XLNet and at most  $72.4\%$  in XLM-R (See Table 1). Interestingly, the multilingual models (mBERT and XLM-R) found substantially higher match than the monolingual models. The inclusion of multiple languages during training causes the model to learn more linguistic properties. Note that the extent of alignment with the human-defined concept may not necessarily correlate with its overall performance. For example XLNet performs outperforms BERT on the GLUE tasks, but aligns less with the human-defined concepts compared to BERT in our results. A similar observation was made by Belinkov et al. (2020) who also found that the translation quality of an NMT model may not correlate with the amount of linguistic knowledge learned in the representation. Various factors such as: architectural design, training data, objective function, initialization, etc, play a role in training a pre-trained model. More controlled experiments are needed to understand the relationship of each factor on the performance of the model and on the linguistic learning of the model.
We further investigated per concept<sup>5</sup> alignment
to understand which human-defined concepts are better represented within the encoded concepts. Figure 2 presents the results.
**Lexical Concepts** Pre-trained models encode varying amount of lexical concepts such as casing, ngrams and suffixes. We found between  $7 - 11\%$  encoded concepts that align with the casing concept (title case or upper case). We observed that most of these encoded concepts consist of named entities, which were grouped together based on semantics.
Comparing suffixes and ngrams While affixes often have linguistic connotation (e.g., the prefix anti negates the meaning of the stem and the suffix ies is used for pluralization), the ngram units that become part of the vocabulary as an artifact of statistical segmentation (e.g., using BPE (Sennrich et al., 2016) or Word-piece (Schuster and Nakajima, 2012)) often lack any linguistic meaning. However, models learn to encode such information. We found a match ranging from  $1\%$  (BERT-cased) up to  $25\%$  (XLM-R) when comparing encoded concepts with the suffix concept. A similar pattern is observed in the case of the ngram concept (which is a super-set of the suffix concept) where a staggering  $48\%$  matches were found. Figure 6a shows an ngram cluster found in layer 2 of BERT-c.
Morphology and Semantics We found that the encoded concepts based on word morphology (POS) consistently showed a higher match across all models in comparison to the other abstract concepts, aligning a quarter of the encoded concepts in the case of mBERT. The alignment with semantic concepts is relatively lower, with at most  $16\%$  match across models. This reflects that while the models learn both linguistic properties, morphological ontology is relatively preferred compared to the semantic hierarchy.
Syntactic These concepts capture grammatical orientation of a word, for example Chunking:B-NP is a syntactic concept describing words in the beginning of a noun phrase. CCG:PP/NP is a concept
present their results in the interest of space.
$^{6}$ Appendix A shows more examples of the ngram, suffix, LIWC and WordNet clusters.
5The first word, last word and prefix concepts showed less less than  $1\%$  alignment with the encoded concepts. We do not
Figure 3: Examples of BERT-c encoded concepts aligned with the human-defined concepts
in CCG super tagging, describing words that take a noun phrase on the right and outputs a preposition phrase for example "[in[the US]]". We found relatively fewer matches, a maximum of  $7\%$  and  $14\%$  matching encoded concepts for Chunking and CCG concepts respectively. The low matches for syntactic concepts suggest that the models do not encode the same syntactic hierarchy suggested by these human-defined syntactic tasks.
Linguistic Ontologies Comparing the encoded concepts with static linguistic ontologies, we found WordNet concepts to be the second most aligned concept (11-21%) with the human-defined concepts. LIWC also shows a relatively higher alignment compared to the other human-defined concepts in a few models (e.g., BERT-c). However, this observation is not consistent across models and we found a range between 5-16% matches. These results present an interesting case where several models prefer the distinction of lexical ontology over abstract linguistic concepts such as morphology. Figure 3 shows examples of encoded concepts aligned with WordNet and LIWC. We see that these concepts are built based on a semantic relationship e.g., the clusters in Figure 3b, 3c and 3d group words based on religious, facial anatomy, and specific motion-related vocabulary respectively.
Comparing Models The results of multilingual models (mBERT, XLM-R) are intriguing given that their encoded concepts are dominated by ngram-based concepts and POS concepts, and their relatively lesser alignment with the linguistic ontologies. On the contrary, several monolingual models (BERT-c, ALBERT) showed a better match with linguistic ontologies specially WordNet.
The higher number of matches to the ngram (and suffix) concepts in the multilingual models is due to the difference in subword segmentation. The subword models in XLM-R and mBERT are optimized for multiple languages, resulting in a vocabulary
consisting of a large number of small ngram units. This causes the multilingual models to aggressively segment the input sequence, compared to the monolingual models<sup>7</sup> and resulted in highly dominated ngram-based encoded concepts, especially in the lower layers. This may also explain the relatively lower match that multilingual models exhibit to the linguistic ontologies. We discuss this further in the context of layer-wise analysis in Section 5.2.
Comparing BERT cased vs. uncased, interestingly BERT-uc consistently showed higher matches for the core-linguistic concepts (See Figure 2). We speculate that in the absence of casing information, BERT-uc is forced to learn more linguistic concepts, whereas BERT-c leverages the explicit casing information to capture more semantically motivated concepts based on linguistic ontologies.
The higher matches in multilingual models in comparison to the monolingual models, and BERT-uncased in comparison to BERT-cased suggest that the training complexity is one factor that plays a role in a model's ability to learn linguistic nuances. For example, multilingual models need to optimize many languages, which is a harder task compared to learning one language. Similarly, the absence of capitalization in training data makes the learning task relatively harder for BERT-uc compared to BERT-c models, thus resulting in higher matches for BERT-uc. We speculate that the harder the training task, the more language nuances are learned by a model. Belinkov et al. (2020) made a similar observation, where they showed that the linguistic knowledge learned within the encoder-decoder representations in NMT models correlates with complexity of a language-pair involved in the task.
5.2 Layer-wise Alignment
Now we study the alignment of human-defined concepts across layers to understand how concepts
7In our dataset, mBERT has  $13\%$  more words after subword segmentation compared to BERT-c.
Figure 4: Layer-wise concept alignment. Y-axis is the normalized number of aligned concepts. The number within brackets of each human-defined concept, e.g. Casing (166), shows the maximum layer-wise match