Discovering Latent Concepts Learned in BERT

A large number of studies analyze deep neural network models and their ability to encode various linguistic and non-linguistic concepts , and subsequently provide an interpretation of the inner mechanics of these models . A major caveat to these studies , however , is that their scope of interpretation is limited to pre-defined concepts that mainly reinforce the traditional linguistic knowledge . Consequently , the interpretation does not reflect on novel concepts learned by a model . We address this limitation by discovering and analyzing latent concepts learned in a model in an unsupervised fashion , thus providing interpretation from the model ’ s perspective . In this work , we study i ) what latent concepts exist in the pre-trained BERT model , ii ) how the discovered latent concepts align or diverge from the classical linguistic hierarchy and iii ) how the concepts evolve across layers . Some notable findings of our analysis include : i ) the model learns novel concepts such as animal categories and demographic groups , which do not strictly adhere to any predefined categorization ( e.g. , POS , semantic tags ) , ii ) several latent concepts are based on multiple properties which may include semantics , syntax or morphology , iii ) lower layers dominate in learning shallow lexical concepts while higher layers learn semantic relations and iv ) the discovered latent concepts highlight potential biases learned in the model . We also release a novel BERT concept dataset ( BCD ) consisting of 183 concept labels and 1M annotated instances . 1 INTRODUCTION . Interpreting deep neural networks ( DNNs ) is essential for understanding their inner working and for their successful deployment to real world scenarios , especially for applications where fairness , trust , accountability , reliability and ethical decision-making are considered critically important metrics . A large number of studies have been devoted towards interpreting DNNs . A major line of work in interpreting deep NLP models reflects on their ability to learn various language concepts ( Tenney et al. , 2019b ; Liu et al. , 2019a ) . More specifically , they rely on pre-defined linguistic concepts such as : parts-of-speech tags and semantic tags , and probe whether the specific linguistic knowledge is learned in various parts of the network . For example , Belinkov et al . ( 2020 ) found that lower layers in the NMT model capture word morphology and the higher layers learn long-range dependencies . A pitfall to this line of research is its sole reliance on pre-defined concepts which may only provide a narrow view of what the model knows , largely ignoring any latent concepts learned within these models . Another drawback of using user-defined concepts is the involvement of human bias in choosing a concept which may result in a misleading interpretation . In this work , we target this issue and approach interpretability from a model ’ s perspective by discovering latent concepts in pre-trained models . Mikolov et al . ( 2013 ) ; Reif et al . ( 2019 ) showed that words group together in the high dimensional space based on syntactic and semantic relationships . We hypothesize that these groupings represent latent concepts , a model learns about the language . To this end , we cluster contextualized representations in high dimensional space and study the relation behind each group . More specifically , we use hierarchical clustering to identify groups of related words . We manually annotate the groups and assemble a concept dataset . This is the first concept dataset that enables model-centric interpretation and will serve as a benchmark for analyzing such models . We base our study on BERT ( Devlin et al. , 2019b ) . However , our method can be generically applied to other models . We study i ) what latent concepts exist in BERT , ii ) how the discovered latent concepts align with pre-defined concepts , and iii ) how the concepts evolve across layers . Our analysis reveals interesting findings such as : i ) the model learns novel concepts like the hierarchy in animal kingdom , demographic hierarchy etc . which do not strictly adhere to any pre-defined concepts , ii ) the lower layers focus on shallow lexical concepts while the higher layers learn semantic relations , iii ) the concepts learned at higher layers are more aligned with the linguistically motivated concepts compared to the lower layers , and iv ) the discovered latent concepts highlight potential biases learned in the model . There are two main contributions of our work : i ) we interpret BERT by analyzing latent concepts learned in the network , ii ) we provide a first multi-facet hierarchical concept dataset ( BCD ) that addresses a major limitation of a prominent class of interpretation studies . Our proposed concept pool facilitates interpretation from the perspective of the model and will serve as a common benchmark for future research . Moreover , it enriches existing linguistic and pre-defined concepts and has the potential to serve as a new classification dataset for NLP in general . BCD consists of 183 fine-grained concept labels with a total of 1M annotated instances . 2 RELATED WORK . The interpretation of DNNs in NLP has been largely dominated by the post-hoc model analysis ( Liu et al. , 2019a ; Shi et al. , 2016 ; Belinkov et al. , 2020 ; Tenney et al. , 2019a ) where the primary question is what linguistic concepts are encoded in the representation ( Radford et al. , 2018 ; Lakretz et al. , 2019 ; Dalvi et al. , 2019 ; Durrani et al. , 2020 ) . The studies have analyzed a range of concepts varying from low-level shallow concepts such as word position and sentence length to linguistically motivated concepts such as parts of speech ( POS ) , syntax and semantic tags . They provide interesting insights into what concepts are learned and encoded in the representation . One of the shortcomings of these studies is their reliance on pre-defined concepts for interpretation . Consequently , novel concepts that models learn about the language are largely undiscovered . Recently Michael et al . ( 2020 ) analyzed latent concepts learned in pre-trained models . They used a binary classification task to induce latent concepts relevant to that task and showed the presence of linguistically motivated and novel concepts in the representation . Mamou et al . ( 2020 ) analyzed representations of pre-trained models using mean-field theoretic manifold analysis and showed the emergence of linguistic structure across layer depth . Similar to Michael et al . ( 2020 ) ; Mamou et al . ( 2020 ) , we aim to analyze latent concepts learned in the representation . However different from them , we intend to analyze raw representations independent of a supervised task . The reliance on a supervision task effects the type of latent concepts found and may not fully reflect the latent concepts encoded in the raw representation . Moreover , the post-hoc classifiers require a careful evaluation to mitigate the concerns regarding the power of the probe ( Hewitt & Liang , 2019 ; Zhang & Bowman , 2018 ) . In this work , we address these limitations by analyzing latent concepts learned in BERT in an unsupervised fashion . In addition , we provide a novel multi-facet hierarchical concept dataset that significantly enrich already existing pre-defined concepts . 3 WHAT IS A CONCEPT ? . A concept represents a notion and can be viewed as a coherent fragment of knowledge . Stock ( 2010 ) defines concept as “ a class containing certain objects as elements , where the objects have certain properties ” . Deveaud et al . ( 2014 ) considers latent concepts as words that convey the most information or that are the most relevant to the initial query . A concept in language can be anything ranging from a shallow lexical property such as character bigram , to a complex syntactic phenomenon such as a adjectival phrase or a semantic property such as an event . Here , we loosely define concept as a group of words that are meaningful where meaningful can be a lexical , semantic , syntactic , morphological and/or any other relationship . For example , names of ice-hockey teams form a semantic cluster , words that occur in a certain position of a sentence represent a syntactic concept , words that are first person singular verbs form a morphological cluster , words that begin with ” anti ” form a lexical cluster . In our implementation , we consider clusters of word contextual representations in BERT as hidden concept candidates and the human-annotated candidates with linguistic meanings as our discovered hidden concepts . However , if a human is unable to identify a relation behind a group of words i.e . not-meaningful , we consider it as uninterpretable . Since , a relation may exist that humans are unable to comprehend . 4 METHODOLOGY . Consider a Neural Network ( NN ) model M with L layers { l1 , l2 , ... ll , ... , lL } , where each layer contains H hidden nodes . An input sentence consisting of M words w1 , w2 , ... wi , ... , wM is fed into NN . For the i-th word input , we can compute the node output ( after applying the activation functions ) ylh ( i ) of every hidden node h ∈ { 1 , ... , H } in each layer l , where −→y l ( i ) is the vector representation of all hidden node outputs in layer l for wi . Our goal is to cluster −→y l , the contextual representation , among all i-th input words . We call this clustering as latent concepts that map words into meaningful groups . We introduce an annotation schema and manually assign labels to each cluster following our definition of a concept . In the following , we discuss each step in detail . 4.1 CLUSTERING . Algorithm 1 ( Appendix A ) presents the clustering procedure . For each layer , we cluster the hidden nodes into K groups using agglomerative hierarchical clustering ( Gowda & Krishna , 1978 ) trained on the contextualized representation . We apply Ward ’ s minimum variance criterion that minimizes the total within-cluster variance . The distance between two vector representations is calculated with the squared Euclidean distance . The number of clustersK is a hyperparameter . We found an optimal value ofK using the Elbow method . Figure 2 shows the Elbow scores over a range of cluster values . We set K = 1000 since after that point , the inertia starts decreasing linearly . 4.2 DATA PREPARATION . The choice of a dataset is an important factor to be considered when generating latent concepts . Ideally , one would like to form latent clusters over a large collection with diverse domains . However , clustering a large number of points is computationally and memory intensive . In the case of agglomerative hierarchical clustering using Ward linkage , the memory complexity is O ( n2 ) , which translates to about 400GB of runtime memory for 200k input vectors . A potential workaround is to use a different linkage algorithm like single or average linkage , but each of these come with their own caveats regarding assumptions around how the space is structured and how noisy the inputs are . We address these problems by using a subset of a large dataset of News 2018 WMT1 ( 359M tokens ) . We randomly select 250k sentences from the dataset ( ≈5 million tokens ) . We further limit the number of tokens to ≈210k by discarding singletons , closed-class words , and tokens with a frequency higher than 10 . For every token , we extract its contextualized representations and embedding layer representation using the 12-layered BERT-base-cased model ( Devlin et al. , 2019a ) in the following way : First , we tokenize sentences using the Moses tokenizer2 and pass them through the standard pipeline of BERT as implemented in HuggingFace3 . We extract layer-wise representations of every token . If the BERT tokenizer splits an input token into subwords or multiple tokens , we mean pool over them to create an embedding of the input token . For every layer , we cluster tokens using their contextualized representations .

Summary: The paper performs a clustering analysis on the contextual representations of BERT on a subset of the News 2018 WMT corpus. These clusters are manually tagged by 3 annotators to connect these clusters to human concepts. The authors additionally release a dataset with these concepts labeled.