Discovering Latent Concepts Learned in BERT

A large number of studies analyze deep neural network models and their ability to encode various linguistic and non-linguistic concepts , and subsequently provide an interpretation of the inner mechanics of these models . A major caveat to these studies , however , is that their scope of interpretation is limited to pre-defined concepts that mainly reinforce the traditional linguistic knowledge . Consequently , the interpretation does not reflect on novel concepts learned by a model . We address this limitation by discovering and analyzing latent concepts learned in a model in an unsupervised fashion , thus providing interpretation from the model ’ s perspective . In this work , we study i ) what latent concepts exist in the pre-trained BERT model , ii ) how the discovered latent concepts align or diverge from the classical linguistic hierarchy and iii ) how the concepts evolve across layers . Some notable findings of our analysis include : i ) the model learns novel concepts such as animal categories and demographic groups , which do not strictly adhere to any predefined categorization ( e.g. , POS , semantic tags ) , ii ) several latent concepts are based on multiple properties which may include semantics , syntax or morphology , iii ) lower layers dominate in learning shallow lexical concepts while higher layers learn semantic relations and iv ) the discovered latent concepts highlight potential biases learned in the model . We also release a novel BERT concept dataset ( BCD ) consisting of 183 concept labels and 1M annotated instances . 1 INTRODUCTION . Interpreting deep neural networks ( DNNs ) is essential for understanding their inner working and for their successful deployment to real world scenarios , especially for applications where fairness , trust , accountability , reliability and ethical decision-making are considered critically important metrics . A large number of studies have been devoted towards interpreting DNNs . A major line of work in interpreting deep NLP models reflects on their ability to learn various language concepts ( Tenney et al. , 2019b ; Liu et al. , 2019a ) . More specifically , they rely on pre-defined linguistic concepts such as : parts-of-speech tags and semantic tags , and probe whether the specific linguistic knowledge is learned in various parts of the network . For example , Belinkov et al . ( 2020 ) found that lower layers in the NMT model capture word morphology and the higher layers learn long-range dependencies . A pitfall to this line of research is its sole reliance on pre-defined concepts which may only provide a narrow view of what the model knows , largely ignoring any latent concepts learned within these models . Another drawback of using user-defined concepts is the involvement of human bias in choosing a concept which may result in a misleading interpretation . In this work , we target this issue and approach interpretability from a model ’ s perspective by discovering latent concepts in pre-trained models . Mikolov et al . ( 2013 ) ; Reif et al . ( 2019 ) showed that words group together in the high dimensional space based on syntactic and semantic relationships . We hypothesize that these groupings represent latent concepts , a model learns about the language . To this end , we cluster contextualized representations in high dimensional space and study the relation behind each group . More specifically , we use hierarchical clustering to identify groups of related words . We manually annotate the groups and assemble a concept dataset . This is the first concept dataset that enables model-centric interpretation and will serve as a benchmark for analyzing such models . We base our study on BERT ( Devlin et al. , 2019b ) . However , our method can be generically applied to other models . We study i ) what latent concepts exist in BERT , ii ) how the discovered latent concepts align with pre-defined concepts , and iii ) how the concepts evolve across layers . Our analysis reveals interesting findings such as : i ) the model learns novel concepts like the hierarchy in animal kingdom , demographic hierarchy etc . which do not strictly adhere to any pre-defined concepts , ii ) the lower layers focus on shallow lexical concepts while the higher layers learn semantic relations , iii ) the concepts learned at higher layers are more aligned with the linguistically motivated concepts compared to the lower layers , and iv ) the discovered latent concepts highlight potential biases learned in the model . There are two main contributions of our work : i ) we interpret BERT by analyzing latent concepts learned in the network , ii ) we provide a first multi-facet hierarchical concept dataset ( BCD ) that addresses a major limitation of a prominent class of interpretation studies . Our proposed concept pool facilitates interpretation from the perspective of the model and will serve as a common benchmark for future research . Moreover , it enriches existing linguistic and pre-defined concepts and has the potential to serve as a new classification dataset for NLP in general . BCD consists of 183 fine-grained concept labels with a total of 1M annotated instances . 2 RELATED WORK . The interpretation of DNNs in NLP has been largely dominated by the post-hoc model analysis ( Liu et al. , 2019a ; Shi et al. , 2016 ; Belinkov et al. , 2020 ; Tenney et al. , 2019a ) where the primary question is what linguistic concepts are encoded in the representation ( Radford et al. , 2018 ; Lakretz et al. , 2019 ; Dalvi et al. , 2019 ; Durrani et al. , 2020 ) . The studies have analyzed a range of concepts varying from low-level shallow concepts such as word position and sentence length to linguistically motivated concepts such as parts of speech ( POS ) , syntax and semantic tags . They provide interesting insights into what concepts are learned and encoded in the representation . One of the shortcomings of these studies is their reliance on pre-defined concepts for interpretation . Consequently , novel concepts that models learn about the language are largely undiscovered . Recently Michael et al . ( 2020 ) analyzed latent concepts learned in pre-trained models . They used a binary classification task to induce latent concepts relevant to that task and showed the presence of linguistically motivated and novel concepts in the representation . Mamou et al . ( 2020 ) analyzed representations of pre-trained models using mean-field theoretic manifold analysis and showed the emergence of linguistic structure across layer depth . Similar to Michael et al . ( 2020 ) ; Mamou et al . ( 2020 ) , we aim to analyze latent concepts learned in the representation . However different from them , we intend to analyze raw representations independent of a supervised task . The reliance on a supervision task effects the type of latent concepts found and may not fully reflect the latent concepts encoded in the raw representation . Moreover , the post-hoc classifiers require a careful evaluation to mitigate the concerns regarding the power of the probe ( Hewitt & Liang , 2019 ; Zhang & Bowman , 2018 ) . In this work , we address these limitations by analyzing latent concepts learned in BERT in an unsupervised fashion . In addition , we provide a novel multi-facet hierarchical concept dataset that significantly enrich already existing pre-defined concepts . 3 WHAT IS A CONCEPT ? . A concept represents a notion and can be viewed as a coherent fragment of knowledge . Stock ( 2010 ) defines concept as “ a class containing certain objects as elements , where the objects have certain properties ” . Deveaud et al . ( 2014 ) considers latent concepts as words that convey the most information or that are the most relevant to the initial query . A concept in language can be anything ranging from a shallow lexical property such as character bigram , to a complex syntactic phenomenon such as a adjectival phrase or a semantic property such as an event . Here , we loosely define concept as a group of words that are meaningful where meaningful can be a lexical , semantic , syntactic , morphological and/or any other relationship . For example , names of ice-hockey teams form a semantic cluster , words that occur in a certain position of a sentence represent a syntactic concept , words that are first person singular verbs form a morphological cluster , words that begin with ” anti ” form a lexical cluster . In our implementation , we consider clusters of word contextual representations in BERT as hidden concept candidates and the human-annotated candidates with linguistic meanings as our discovered hidden concepts . However , if a human is unable to identify a relation behind a group of words i.e . not-meaningful , we consider it as uninterpretable . Since , a relation may exist that humans are unable to comprehend . 4 METHODOLOGY . Consider a Neural Network ( NN ) model M with L layers { l1 , l2 , ... ll , ... , lL } , where each layer contains H hidden nodes . An input sentence consisting of M words w1 , w2 , ... wi , ... , wM is fed into NN . For the i-th word input , we can compute the node output ( after applying the activation functions ) ylh ( i ) of every hidden node h ∈ { 1 , ... , H } in each layer l , where −→y l ( i ) is the vector representation of all hidden node outputs in layer l for wi . Our goal is to cluster −→y l , the contextual representation , among all i-th input words . We call this clustering as latent concepts that map words into meaningful groups . We introduce an annotation schema and manually assign labels to each cluster following our definition of a concept . In the following , we discuss each step in detail . 4.1 CLUSTERING . Algorithm 1 ( Appendix A ) presents the clustering procedure . For each layer , we cluster the hidden nodes into K groups using agglomerative hierarchical clustering ( Gowda & Krishna , 1978 ) trained on the contextualized representation . We apply Ward ’ s minimum variance criterion that minimizes the total within-cluster variance . The distance between two vector representations is calculated with the squared Euclidean distance . The number of clustersK is a hyperparameter . We found an optimal value ofK using the Elbow method . Figure 2 shows the Elbow scores over a range of cluster values . We set K = 1000 since after that point , the inertia starts decreasing linearly . 4.2 DATA PREPARATION . The choice of a dataset is an important factor to be considered when generating latent concepts . Ideally , one would like to form latent clusters over a large collection with diverse domains . However , clustering a large number of points is computationally and memory intensive . In the case of agglomerative hierarchical clustering using Ward linkage , the memory complexity is O ( n2 ) , which translates to about 400GB of runtime memory for 200k input vectors . A potential workaround is to use a different linkage algorithm like single or average linkage , but each of these come with their own caveats regarding assumptions around how the space is structured and how noisy the inputs are . We address these problems by using a subset of a large dataset of News 2018 WMT1 ( 359M tokens ) . We randomly select 250k sentences from the dataset ( ≈5 million tokens ) . We further limit the number of tokens to ≈210k by discarding singletons , closed-class words , and tokens with a frequency higher than 10 . For every token , we extract its contextualized representations and embedding layer representation using the 12-layered BERT-base-cased model ( Devlin et al. , 2019a ) in the following way : First , we tokenize sentences using the Moses tokenizer2 and pass them through the standard pipeline of BERT as implemented in HuggingFace3 . We extract layer-wise representations of every token . If the BERT tokenizer splits an input token into subwords or multiple tokens , we mean pool over them to create an embedding of the input token . For every layer , we cluster tokens using their contextualized representations .

Summary: This paper studies the latent concept learned in BERT representations. The authors use a simple and effective clustering method to discover latent concepts in an unsupervised way. The contribution is primarily on the empirical side where the learned concepts are labeled and compared with pre-defined concepts. Then a layer-wise comparison is conducted where the results are similar to previous work but is done in an unsupervised way. The authors summarise this paper by organizing the discovered concepts into a BCD dataset. The writing is also clear. While there are many aspects deserving further pursuit, I think this is a solid paper that makes concrete contributions to the understanding of BERT.