Discovering Latent Concepts Learned in BERT
Authors: Fahim Dalvi
Abdul Rafae Khan
Firoj Alam
Nadir Durrani
Jia Xu
Hassan Sajjad
Venue: ICLR 2022 Poster (2022)
Keywords: interpretation
BERT
NLP

Abstract: A large number of studies that analyze deep neural network models and their ability to encode various linguistic and non-linguistic concepts provide an interpretation of the inner mechanics of these models. The scope of the analyses is limited to pre-defined concepts that reinforce the traditional linguistic knowledge and do not reflect on how novel concepts are learned by the model. We address this limitation by discovering and analyzing latent concepts learned in neural network models in an unsupervised fashion and provide interpretations from the model's perspective. In this work, we study: i) what latent concepts exist in the pre-trained BERT model, ii) how the discovered latent concepts align or diverge from classical linguistic hierarchy and iii) how the latent concepts evolve across layers. 
Our findings show: i) a model learns novel concepts (e.g. animal categories and demographic groups), which do not strictly adhere to any pre-defined categorization (e.g. POS, semantic tags), ii) several latent concepts are based on multiple properties which may include semantics, syntax, and  morphology, iii) the lower layers in the model dominate in learning shallow lexical concepts while the higher layers learn semantic relations and iv) the discovered  latent concepts highlight potential biases learned in the model. We also release a novel BERT ConceptNet dataset consisting of 174 concept labels and 1M annotated instances.

Published as a conference paper at ICLR 2022
DISCOVERING LATENT CONCEPTS LEARNED IN BERT
Fahim Dalvi Abdul Rafae Khan i Firoj Alam Nadir Durrani (faimaduddin, fialam, ndurrani jaajjad]@hbbku edu. qa QQtaar Computing Research Institute, HBKU Research Complex, Doha 5825, Qatar
Jia Xut Hassan Sajjad
SScooo of Engineering and Science, Steven Institute of Technology, Hoboken, NJ 07030, USA 4 etteenessteeeeeeeee.ee.e .ddu
ABSTRACT
A large number of studies that analyze deep neural network models and their abil- ity encode various linguistic and non-linguistie concepts provide an interpreta- tion of the inner mechanics of these models. The scope of the analyses is limited to pre-defined concepts that reinforce the traditional linguistic knowledge and do not reflect how novel concepts are learned by the model. We address this lim- itation by discovering and analyzing latent concepts learned in neural network models in unsupervised fashion and provide interpretations from the model s perspective. In this work, we study: i) what latent concepts exist in the pre-trained BERT model, ii) how the discovered latent concepts align or diverge from classi- cal linguistic hierarchy and iii) how the latent concepts evolve across layers. Our findings show: model learns novel concepts (e.g. animal categories and de- mographic groups), which do not strictly adhere to any pre-defined categorization (e.g. POS, semantic tags), ii) several latent concepts are based on multiple prop- erties which may include semantics, syntax, and morphology, iii) the lower layers in the model dominate learning shallow lexical concepts while the higher layers learn semantic relations and iv) the discovered latent concepts highlight potential biases learned the model. We also release a novel BERT ConceptNet dataset (BCN) consisting of 774 concept labels and IM annotated instances.
INTRODUCTION
Interpreting deep neural networks (DNNs) is essential for understanding their inner workings and successful deployment to real-world scenarios, especially for applications where fairness, trust, ac- countability, reliability, and ethical decision-making have been devoted towards interpreting DNNs. A major line of research work has focused on DNNs in interpreting deep Natural Language Processing (NLP) models and their ability to learn various pre-defined linguistic concepts (Tenney et al., 2019b Liu et al.] 2019a). More specifically, they rely on pre-defined linguistic concepts such as: parts-of-speech tags and semantic tags, and probe whether the specific linguistic knowledge is learned in various parts of the network. For example, Belinkov al.] (2020) found that lower layers in the Neural Machine Translation (NMT) model capture word morphology and higher layers learn long-range dependencies.
critical metrics. A large number of studies
pitfall to this line of research is its study of pre-defined concepts and the ignoring of any latent concepts within these models, resulting in narrow view of what the model knows. Another weak- ness of using user-defined concepts the involvement of human bias in the selection of a concept which may result in misleading interpretation. In our work, we sidestep this issue by approach- ing interpretability from a model's perspective, specifically focusing of the discovering of latent concepts in pre- mre-trained models.
Mikolov et al. )0 Reif et al. (0199) showed that words group together in the high dimensional space based on syntactic and semantic relationships. We hypothesize that these groupings represent latent concepts, the information that model learns about the language. In our approach, we cluster
"Equal contribution
Code and dataset: http://////aaao.e.rrroo///////te html

---

Published as a conference paper at ICLR 2022
contextualized representations in high dimensional space and study the relations behind each group by using hierarchical clustering to identify groups of related words. We manually annotate the groups and assemble concept dataset. This is the first concept dataset that enables model-centric interpretation and will serve as a benchmark for analyzing these types of models. While we base our study on BERT (Devlin et al. 2019), our method can be generically applied to other models.
We study: i) what latent concepts exist in BERT, ii) how the discovered latent concepts align with pre-defined concepts, and iii) how the concepts evolve across layers. Our analysis reveals interesting findings such as: i) the model learns novel concepts like the hierarchy in animal kingdom, demo- graphic hierarchy etc. which do not strictly adhere to any pre-defined concepts, ii) the lower layers focus on shallow lexical concepts while the higher layers learn semantic relations, iii) the concepts learned at higher layers are more aligned with the linguistically motivated concepts compared the lower layers, and iv) the discovered latent concepts highlight potential biases learned in the model.
There are two main contributions of our work: i) We interpret BERT by analyzing latent concepts learned in the network, and ii) we provide a first multi-facet hierarchical BERT ConceptNet dataset (BCN) that addresses a major limitation of a prominent class of interpretation studies. Our proposed concept pool dataset not only facilitates interpretation from the perspective of the model, but also serves as a common benchmark for future research. Moreover, our findings enrich existing linguistic and pre-defined concepts and has the potential to as general. BCN consists of 1 174 fine-grained concept labels with a total of 1M annotated instances.
classification dataset for NLP in
WHAT IS A CONCEPT?
A concept represents a notion and can be viewed as a coherent fragment of knowledge, Stock (2010) defines concept as "a class containing certain objects as elements, where the objects have certain properties". Deveaud et al. (2014) considers latent concepts as words that convey the most informa- tion or that are the most relevant to the initial query. A concept in language can be anything ranging from shallow lexical property such as character bigram, to complex syntactic phenomenon such as an adjectival phrase or semantic property such as an event. Here, we loosely define concept as group of words that are meaningful i.e. can be clustered based on a linguistic relation such as lexical, semantic, syntactic, morphological etc. For example, names of ice-hockey teams form a semantic cluster, words that occur in certain position of sentence represent syntactic concept, words that are first person singular verbs form a morphological cluster, words that begin with "anti" form lexical cluster. We consider clusters of word contextual representations in BERT concept candidates and the human-annotated candidates with linguistic meanings as our discovered latent concepts. However, if a human is unable identify a relation behind group of words, we consider it as uminterpretable. It is plausible that BERT learns a relation between words that humans are unable to comprehend.
3 METHODOLOGY
Consider a Neural Network (NN) model M with layers {41, ...,,..........,. where each layer contains H hidden nodes. An input sentence consisting of M words W1, W2, W2. WM is fed into NN. For the i-th word input, we compute the node output (after applying the activation functions) vi, (t) of every hidden node h E {1,..., H} in each layer 1, where (i) is the vector representation of all hidden node outputs in layer / for w; Our goal is to cluster J' the contextual representation, among all i-th input words. We call this clustering as latent concepts that map words into meaningful groups. We introduce an annotation schema and manually assign labels to each cluster following our definition of a concept. the following, we discuss each step in detail.
3.1 CLUSTERING
Algorithm [ (Appendix A presents the clustering procedure. For each layer, we cluster the hidden nodes into K groups using agglomerative hierarchical clustering (Gowda & Krishnal 1978) trained on the contextualized representation. We apply Ward's minimum variance criterion that minimizes the total authin-cluster variance. The distance between two vector representations is calculated with the squared Fuclidean distance. The number of clusters K is hyperparameter. We empirically set
2

---

Published as a conference paper at ICLR 2022
celebrities
baseball
othletes
ployers
players
actors
Cumberbatch
Tennant
Gurriel Haniger Kambeei Hegazi
Proper
Proper
Proper
Proper
Proper
Proper
POS Tags
Noun
Noun
Noun
Noun
Noun
Noun
Lexical
Title
Title
Title
Title
Title
Title
Tags
Case
Case
Case
Case
Case
Case
WordNET
N/A
N/A
N/A
N/A
N/A
N/A
Tags
Method
Cluster
Cluster
Cluster
Cluster
Cluster
Cluster
184
184
865
865
297
297
Manual
Baseball
Baseball
Soccer
Soccer
Annotation
Player
Player
Player
Player
Figure 1: Illustration of a hierarchical concept tree. Our new concepts represent novel aspects that are not found in the pre-defined concepts.
K 1000 with an aim to avoid many large clusters of size more than 1000 tokens (under clustrrn) and large number of small clusters having less than five word types (over-clustering
3.2 DATA PREPARATION
The choice of dataset an important factor considered when generating latent concepts. Ideally, one would like to form latent clusters over large collection with diverse domains. How- ever, clustering a large number of points computationally and memory intensive. In the case of agglomerative hierarchical clustering using Ward linkage, the memory complexity is O(n²), which translates to about 400GB of runtime memory for 200k input vectors. potential workaround is to different linkage algorithm like single or average linkage, but each of these come with their own caveats regarding assumptions around how the space is structured and how noisy the inputs are.
We address these problems by using a subset of large dataset of News 2018 (M59) tokens). We randomly select 250k sentences from the dataset (=:5 million tokens). We further limit the num- ber of tokens ~210 by discarding singletons, closed-class words, and tokens with a frequency higher than 10. For every token, we extract its contextualized representations and embedding layer representation using the 12-layered BERT-base-cased model (Devlin et al. 2019) the fol- lowing way: First, we tokenize sentences using the Moses tokenizer") and pass them through the standard pipeline of BERT as implemented in HuggingFace We extract layer-wise representations of every token. If the BERT tokenizer splits an input token into subwords or multiple tokens, we mean pool over them to create an embedding of the input token. For every layer, we cluster tokens using their contextualized representations.
3.3 CONCEPT LABELS
We define hierarchical concept-tagset, starting with the core language properties i.e., se- mantics, parts-of-speech, lexical, syntax, etc. For each cluster, we assign the core properties that are represented by that cluster, and further enrich the label with finer hierarchícal infor- mation. It is worth noting that the core language properties are not mutually exclusive i.e., cluster can belong to more than one core properties at the same time. Consider group of words mentioning first name of the football players of the German team and all of the names occur at the start of a sentence, the following series of tags will be assigned to the clus- iewraanicceeneettaiiio
ter: remantic:orige Germany, f frrtmmmme,
sport: football, syntax : poos t :f:::
semant inpratiii:nnnnn namedettttt Here, preserve the hierarchy at various levels such as, sport, person name, origin, etc., which can be used to combine clusters to analyze a larger group.
We experimented with Elbow and Silhouette but they did not show reliable results.
http://cata.stattn wwtp:///////wew..//aeaaa
htta:://pypi.org/prorfo htt.://eee./mmmtttteeeeeiiiee
http://pp...ooroooooooo//////ana https///uggingfaac.coo

---

Published as a conference paper at ICLR 2022
3.4 ANNOTATION TASK
Following our definition of concepts, we formulated the annotation task using the two questions
below:
01: Is the cluster meaningful?
Q2: Can the two clusters be combined to form a meaningful group?
We show annotators the contextual information (i.e., corresponding sentences) of the words for each of these questions. The annotator can answer them as (i) yes, (ii) no, and (iii) don't know can't t judge. Q1 mainly involves identifying if the cluster is meaningful, according to our definition of a concept (Section 2) Based the yes answer of QI, annotators' task to either introduce new concept label where it belongs or classify it into an appropriate existing concept label. Note that the initial phase of the annotation, the existing concept label set was empty. As the annotators annotate and finalize the concept labels, we accumulate them into the existing concept label set and provide them the next round of annotations, in order to facilitate and speed-up the annotations. In Q2, we ask the annotators to identify whether two sibling clusters can be also combined to form meaningful super-cluster. Our motivation here to keep the hierarchical information that BERT is capturing intact. For example, cluster capturing Rock music bands in the US can be combined with its sibling that groups Rock music bands in the UK to form a Rock music bands cluster. Similarly, based the yes answer of Q2, the annotators are required to assign a concept label for the super- cluster. The annotation guidelines combined with the examples and screenshots of the annotation platform is provided in Appendix B
The annotators are asked to maintain hierarchy while annotating e.g., a cluster of ice hockey will be annotated semant i ent wwwrttiimmentttmmmmm wwn.n spoot::::::: ieeeeiceheee shccckey since the words are seman- tically grouped and the group belongs to sports, which is a subcategory of entertainment. In case of more than one possible properties that form a cluster, we annotate it with both properties e.g., title case and country names where the former is lexical property and the latter is semantic property.
Since annotation is an expensive process, we carried out annotation of the final layer of BERT only. The concept annotation is performed by three annotators followed by consolidation step to resolve disagreements if there are any. For the annotation task, we randomly selected 279 clusters of 1000 clusters obtained during the clustering step (see Section 3.1
4
ANNOTATED DATASET
4.1
INTER ANNOTATION AGREEMENT
We computed Fleiss & (Feiss et al. 2013) and found the average value for QI and Q2 to be 0.61 and 0.64, respectively, which is substantial in the A measurement (Landis & Koch 1977). The detail on measures and agreement computation using other metrics are reported in Appendix (Table
4.2 STATISTICS
For Q1, found 243 (87.1%) meaningful clusters and 36 (12.9%) non-meaningful clusters. For Q2, we found that 142 (75.9%) clusters out of 187 clusters can form a meaningful cluster when combined with their siblings. The annotation process resulted in 174 unique concept labels from the 279 clusters (See Appendix [B3 for the complete list of labels). The high number of concept labels is due to the fine- fine-grained distinctions and multi- facet nature of the clusters where a cluster can get more than one labels. The label set consists of 11 lexical labels, 10 morphological labels, 152 semantic labels and one syntactic label. The semantic labels form the richest hierarchy where the next level of hierarchy consists of 42 labels such as: entertainment, sports, geo-politics, etc.
4.3 EXAMPLES OF CONCEPT HIERARCIY
In this Section, describe the concept hierarchy that we have achieved through our annota- tion process. Figure [ illustrates an example a resulting tree found by hierarchical clus- tering. Each node is concept candidate. We capture different levels of fine-grained mean- ings of word groups, which do not exist in pre-defined concepts. For example, a SEM or NE tagger would mark name as person, but our discovery of latent concepts show
4

---

Published as a conference paper at ICLR 2022
90 20-year-ole old 5t-yrer-
year-ol old kindest bravest
uccherloveliest
Welmarschyeswis
Dresden Innsbruck Pormer
Holstein
Heizedberg Prussian
wildest
tenee some
76 7
scariest widest
grudest starkest
Brandenburg Sumeeer Freiburg
Te 70-year-old /9-yerr-old
slowar-oldd
quickest et tightest Live flercest
qulevest Weird
Smartest
MAke sweelest
harshest
Trier
Augsburg
Mainz
Stuttgart (c)
bremm
(b)
Figure 2: Example Concepts. 2a shows cluster that has lexical property (hyphenation) and
a morphological property (adjective). Similarly, 2b shares a common suffix est and additionally possesses the morphological phenomenon, superlative adjectives. Finally, a large number of clusters were grouped based on the fine-grained semantic information. For example, the cluster in (2c is named entity cluster specific to places in Germany,
that BERT uses finer hierarchy by further classifying them as "baseball players". "soc- cer players", and "athletes". Our annotation process preserves this information and provides ce relebrity:acte semantic c: samed-entityyperraa wtwrrrssnnntttteeeeeeeeee athletes soccer.. Due to the multi-faceted nature of the latent clusters, many of them annotated with more than concept label. In the case of the clusters shown in Figure [ they have a common lexical property (title case) and hence will also get the label LEX: case: : title during annotation. Figure shows word clouds of a few annotated clusters along with their descriptions 6
hierarchical labels such semantic named-entity:
and
5
ANALYSIS AND FINDINGS
5.1
QUALITATIVE ANALYSIS
Our annotated dataset unveils insightful analysis of the latent concepts learned within BERT repre- sentations. Figure 3 presents few examples of the discovered concepts. The two clusters of decimal numbers (Figure Sa 3bp look quite similar, but are semantically different. The former represents numbers appearing
percentages e.g.. 9. 6% or 2. percent and the latter captures monetary figures c.g., 9. 5 billion Euros or 2 4 millicn dollars. We found these two clusters to be sibling, which shows that BERT is learning morpho-semantic hierarchy where it grouped all the decimal numbers together (morphologically) and then further made a semantic distinction by di- viding them into percentages and monetary figures. Such a subtle difference in the usage of decimal numbers shows the importance of using fine-grained concepts for interpretation. For example, num- bers are generally analyzed as one concept (CD tag in POS) which may be less informative or even misleading given that BERT treats them in a variety of different ways. Figure 3c shows an example of cluster where the model captures an era of time, specifically 18 and 19 hundreds. Learning such information may help the model to learn relation between a particular era and the events occurring in that time period. Similarly, we found BERT learning semantic distinction of animal kingdom and separated animals into land, sea and flying groups (see Appendix B.33). These informative categories support the use of pre-trained models as knowledge base (Petroni et al. 2019).
We further observed that BERT learns aspects specific to the training data. Figure 3d shows cluster of words where female roles (Moms, Mothers, Granny, Aunt) are grouped together with job roles such as (housekeeper, Maid, Nanny). The contextual information in the sentences where these words appear, do not explicate whether the person is a male or a female, but based on the data used for the training of BERT, it has associated these roles to females. For example, the sentence "Welfare staff's s' good housekeeping praised for uncovering disability benefits error" is a general sentence. However, the contextualized representation of housekeeping this sentence associates it to a female-related concept. Similarly, in another example BERT clustered names based on demography even though the context in which the word is used, does not explicitly provide such information. An example is the sentence, "Bookings: Dzagoev, Schennikov, Musa Akinfeev" where the name Musa is used in a diverse context but it is associated a cluster of specific demography (Arab region). While natural to form clusters based on specific data aspects, we argue that the use of such concepts while making a general prediction raises concerns of bias and fairness. For example, a loan prediction application should not rely on the background of the name of the applicant.
'The size of word represents it's relative frequency in the cluster.
5

---

Published as a conference paper at ICLR 2022
1818
1800s
-1.5
m63. 9.8 9.8 13.8 21:
11.17_4.1.9 -. 8.1-:000.....00 m4
9.9.3 64. (...aaa
4..22 b
02.5 25.
7.777 4 69.4 15.3 20.4 9.4 11..22111.16666 11.6134
230
20.1120. 1011::1".......0
2117 8.
10.7
8.8 1..84477-5..11 10. 21 2x1
17.
10.5
6 6.8
8,985.77.
13.9
175
0y10
7.2
89
73
12
w..35
14.877,
ta u55 21.1
mvtt 2,04 un ur
65.4
1-18 14.2 11.3
6.111...
w 94 1=...
10.6
10.9 12.2 31.1 10.7
-5.65 1:45
7.7
17.7
1,47 47.7
9.
w1
37.9
45.6
US1.4
2.58
7.8
02 mm
16144
25:3
14.4
17.4
7.2
1846 1866181266
185218431807
1842 1845
te 1800
1825
1815
nineteenth
1841 1847 1849
1694
1850s
1836
776 70
1801
1800
= 1810
1814 1o
es 1824 1800s
18405 1787
1779
1833
1798
1729
(a) Decimal numbers referring to (b) Decimal numbers referring to (c) Eighteenth and Nineteenth Cen- value of money.
percentages.
tury.
Robar Pasha Nur Mirza Fares
Padma
Fatima Osman
Une
Amin
A
Kapoo
Hari Mami
Grareyy
MothersE Etonian
Madam housekeeping
doy hostesses
housework
Beymusa
Aad Sa im Iqbal
SAM
Yusuf
Ismail
Rao
Manon
Naira
Are
Red
Bose
Namny
PrakashJ annnnaannnjujjjjeee
housewife maids mune
HasanRana
Morri ai peqteathed stately
Shankar
widower
housekeeper Wildww
Grandratten Widowed
(c) Names representing a specific (f) Names representing a specific
(d) Feminine roles.
demography
demography
Figure 3: Examples of latent concepts learned in BERT
5.2 ALIGNMENT WITH DRE-DEFINED CONCEPTS
A number of works on interpreting DNNs study to what extent various linguistic concepts such as parts-of-speech tags, semantic tags, etc are captured within the learned representations. This is in view of one school of thought which believes that an NLP model needs to learn linguistic concepts in order to perform well and to generalize (Marasović 2018). For example, the holy grail in machine translation is that a proficient model ought to be aware of word morphology, grammatical structure, and semantics to do well (Vauquois) 1968] [Jones et al. 2012). In this section, we explored how well [968]]ones the latent concepts in BERT align with the pre-defined linguistic concepts.
We compared the resulting clusters with the following pre-defined concepts: parts of speech (POS) tags (Marcus et al.) 1993), semantic tags (SEM) using the Parallel Meaning Bank data (Abzianidze ctt al. 2017). CCG supertagging using CCGBank (Hockenmaier (2066), syntactic chunking (Chunk- ing) using CoNLL 2000 shared task dataset (Tjong Kim Sang & Buchholz 2000). WordNet (Miller 1995) and LIWC psycholinguistc-
tags (Pennebaker et al. We also introduced lexical and syntactic concepts such casing, ngram matches, affixes, first and last word in sentence etc. Appendix D provides the complete list of pre-defined concepts. For POS, SEM, Chunking, and CCG tagging. we trained a BERT-based classifier using the training data of each task and tagged the words in the selected News dataset. For WordNet and LIWC, we directly used their lexicon.
We consider a latent concept to align with corresponding pre-defined concept if 90% of the clus- ter's tokens have been annotated with that pre-defined concept. For example, Figure [a and 3b are assigned a tag POS: : CD (parts-of-speechccaddee
numbers). [able[]shows the statisties on the num- ber of clusters matched with the pre-defined concepts. Of the 1000 clusters, we found the linguistic concepts in POS to have maximum alignment (30%) with the latent concepts in BERT, followed by Casing (23%). The remaining pre-defined concepts including various linguistic categories aligned with fewer than 10% of the latent concepts. We further explored the parallelism between the hier- archy of latent and pre-defined linguistic concepts using POS tags. We merged the POS tags into 9 coarse tags and aligned them with the latent concepts. The number of latent concepts aligned to these coarse concepts increased to 50%, showing that the model has learned coarser POS concepts and does not strictly follow the same fine-grained hierarchy.
5.3 LAYER-WISH COMPARISON OF CONCEPTS
How do the latent concepts evolve in the network? To answer this question, need to manually annotate per-layer latent concepts and do a cross-layer comparison. However, as manual annotation is expensive process, we used pre-defined concepts as proxy to annotate the concepts and

---

Published as a conference paper at ICLR 2022
Lexical
Morphology and Semantics
Syntactic
Concepts Matches
Ngram
Suffix
Casing
POS
SEM
LJWC
WordNet
CCG
Chunk
FW
20
229
297
96
15
39
87
(2.0%)
(0.5%)
(23%)
(30%)
(10%)
(1.5%)
(3.9%)
(8.7%)
(6.3%)
(3.5%)
Table I: Alignment of BERT concepts (Layer 12) with the pre-defined concepts
(a) Lexical
(b) Morphology Semantics
(c) Syntactic
Nomelbod Maxhes
1.
1.0
1.3
0.0
8.8
0.8
0.6
06
0.5
0.4
04
04
02
02
5
8 7 layer
10
12
2
4
5
0
11
0
9 10 12
Layer
1.yer
Cosing (229)
Sufbor (25)
POS (308)
WORDNET (140)
Churiding (71)
CGG (105)
Nigrrna (219)
LIVO (97)
SEM (197)
FistMorr (38)
Figure 4: Layer-wise alignment of pre-defined concepts in BERT. The vertical axis represents the number of aligned clusters for given concept, normalized by the maximum across all layers. Num- bers inside parenthesis show the maximum alignment for each concept.
compare across layers. Figure summarizes the layer-wise matching of pre-defined concepts with the latent concepts. Interestingly, y, the layers differ significantly in terms of the concepts they capture. In Figure 4., the embedding layer is dominated by the concepts based on common ngram affixes The number of ngram-based shallow lexical clusters consistently decreases in the subsequent layers. The suffixes concept showed different trend, where the information peaks at the lower-middle layers and then drops in the higher layers. The dominance of ngrams-based concepts in the lower layers due to the subword segmentation. The noticeable presence in the middle layers might be due to the fact that suffixes often maintain grammatical information unlike other ngrams. For example, the ngram "ome" does not have a linguistic connotation, but suffix "ful" converts a noun into an adjective. Interestingly, casing information is least represented in the lower layers and is consistently improved for higher layers. Casing in English has a morphological role, marking a named entity, which is essential knowledge to preserve at the higher layers.
The patterns using the classical NLP tasks: POS, SEM, CCG and Chunking all showed similar trends (See Figure/4p and 4p). The concepts were predominantly captured at the middle and higher layers, with minimal representation at the lower layers. WordNet and LIWC, which are cognitive and psychology based grouping of words respectively, showed rather different trend. These concepts had a higher match at the embedding layer and a consistent drop after the middle layers. In other words, the higher and last layers of BERT are less aligned with the cognitive and psychology based grouping of words, and these concepts are better represented at the embedding and lower layers. We manually investigate a few clusters from the lower layers that match with WordNet and LIWC. We found that these clusters group words based on similar meaning e.g. all words related to "family relations" or "directions in space" form a cluster (See Appendix [ for more examples). Lastly, Figure 4 also shows that the FirstWord position information is predominantly learned at the last layers only. This somewhat unexpected, given that no clusters were based on this concept in the embedding layer, despite it having an explicitly position component.
These trends reflect the evolution of knowledge across the network: lower layers encode the lexical and reeattd knowledge and with the inclusion of context in the higher layers, the encoded concepts evolve into representing linguistic hierarchy. These findings are in line with Mamou et al. (220) where they found the emergence of linguistic manifolds across layer depth.
5.4 UNALIGNED CLUSTERS
In this section, we discuss the concepts that were not matched with the pre-defined concepts. We have divided them into 3 categories: i) latent concepts that can be explained via composition of
For every cluster we pick matching ngram with highest frequency match. Longest ngram is used when tie-breaker is required. We ignore unigrams.
7

---

Published as a conference paper at ICLR 2022
Violot mod
blueherries
herbal
Orchal
co-hoeting ee offiaae minl
cascading
Garrbeting
amounting
Cinnamon
on Siageco do prmarter
sqanr thorny
pocketed
ber
berbCoconutt
rindigo hotao
showrun Showrünner nnner
on can-Camero a
Ta ce Datle Pi Temisre
whitle
outpacing
exceeds
valuine sur surpassing
outnumber
dwarfed
verco
mango Lgs pea brusens
pineapple Lawender
hazelnut
Resper
CO- creator voiceover
Co anchor
on air
Sart
churc
niggling
Knaggngggg
(a) Verb Forms
(b) Nouns
(c) Compound Words
(d) Uninterpretable
Figure 5: Unaligned Concepts
two pre-defined concepts, ii) concepts that cannot be explained via auto-labels but are meaningful and interpretable, iii) concepts that are uninterpretable. Figure Sa and [5b] are examples of the first category mentioned above. The former describes a concept composed of different verb forms. It does not fully align with any of the fine-grained verb categories but can be deemed as a verb cluster at coarser level. Similarly the latter shows a cluster containing singular and plural nouns (described by POS:NN and POS:NNS) and WordNet concepts (WN:noun:food and :llnoun:)lants) and can be aligned with coarser noun category. Figure [c shows cluster that does not match with any human-defined concept but are interpretable in that the cluster is composed of compound words. Lastly in Figure we show cluster that we found to be uninterpretable.
6 BERT CONCEPTNET DATASET (BCN)
In addition to the manually labeled dataset, we also developed a large dataset, BCN, using a weakly supervised approach. The motivation of creating such dataset is to provide reasonable size dataset that can be effectively use for interpretation studies and other tasks. The idea is to expand the number of tokens for every concept cluster by finding new token occurrences that belong to the same cluster. There are several possible ways to develop such a dataset. Here, we explored two different approaches. First:
compute the centroid for each cluster and find the cluster that is closest to the representation of any given new token. However, this did not result in accurate cluster assignment because of the non-spherical and irregularly shaped clusters. Second: we developed logistic regression classifier (as shown in Appendix [A] Algorithm 2 using our manually annotated dataset assign new tokens to given concept clusters. We found the second approach better than the former in terms of token to cluster assignment. We trained the model using 90% of the concept clusters and evaluate its performance on the remaining 10% concept clusters (held-out set). The classifier achieved precision score of 75% on the held-out set. In order to achieve higher precision, we introduce threshold é = 0.97 on the confidence of the predicted cluster id, assigning new tokens to particular clusters only when the confidence of the classifier is higher than the threshold. This enables better precision in cluster assignment to the new tokens, at the expense of discarding some potentially good assignments, which can be offset by using more data for prediction.
Using the trained model we labeled all the tokens from the million random sentences from the unused News 2018 dataset (see Section 3. The resulting dataset: BCN is unique multi-faceted resource consisting of 174 concept labels with a total of 997.195 195 annotated instances. The average number of annotated instances per concept label are 5,731. The utility of this resource is not limited to interpretation, but can serve as a valuable fine-grained dataset for the NLP community at large. The hierarchy present in the concept labels provides fiexibility to use data with various granularity levels. Appendix provides the detailed statistics for BCN.
Case Study In order to showcase the efficacy of BCN as a useful resource in interpretation, we present a case-study based on neuron-level interpretation in this section. The goal of neuron inter- pretation is to discover a set of neurons responsible for any given concept (Sajjad et al., 2021). Most of the work in this direction Lakretz et al.) (2019); Durrani et al. (2020): Torroba Henni- gen et al. (2020) identified neurons responsible for concepts belonging to the core NLP tasks such as morphology, syntax and semantics. In this work, we found that BERT learns finer cat- egories compared to the ones defined in the core NLP tasks. For example, BCN consists of a large number of fine-grained proper noun categories, each representing unique facet such as proper-noun. person: : specifi ce--dmoorrpphe This enables identifying neurons learn- ing very fine-grained properties, thus provides more informed interpretation of models.
8

---

Published as a conference paper at ICLR 2022
In preliminary experiment, we performed neuron analysis using the PER (person) concept of SEM and the fine-grained concept of Muslim names (Figure 3e). We then compared the amount of neu- rons required for each concept. For the SEM dataset, we used the standard splits (Appendix D). For the fine-grained concept, the BCN dataset consists of 2748 Muslim name instances. We randomly selected equal amount of negative class instances and split the data into 80% train, 10% devel- opment and 10% test sets, We used Linguistic Correlation Analysis (Dalvi et al., 2019a), using the NeuroX toolkit Dalvi et al. (2019b) and identified the minimum number of neurons responsible for each concept. We found that only 19 neurons are enough to represent the fine- ame-graieed concept as opposed to 74 neurons for the PER concept of SEM, showing that the BCN dataset enables selection of specialized neurons responsible for very specific aspects of the language. The discovery of such specialized neurons also facilitate various applications such as model control (Bau et al., 2019 2021 Suau et al. 2020), domain adaptation (Gu et al. 202T and debiasing.
et al.
RELATED WORK
The interpretation of DNNs in NLP has been largely dominated by the post-hoc model analysis (Be-
linkov et al.) 2017a; Liu et al. 2019a; Tenney et al. 2019a) where the primary question is to identify encoded in the representation. Researchers have analyzed range of concepts varying from low-level shallow concepts such as word position and sentence length to linguistically motivated concepts such as morphology (Vylomova et al. 2016 Dalvi et al. 2017
which linguistic concepts
Durrani et al.) 2019), syntax (Shi et al., 2016]] Linzen et al.]]0111 and semantics (Qian et al. 2016 Belinkov al., (0176) and provide insights into what concepts are learned and encoded in the representation. One of the shortcomings of these studies is their reliance on pre-defined concepts for interpretation. Consequently, novel concepts that models learn about the language are largely undiscovered.
Recently Michael et al. (2020) analyzed latent concepts learned in pre-trained models using a bi- nary classification task to induce latent concepts relevant to the task and showed the presence of linguistically motivated and novel concepts in the representation. Mamou et al.) (2020) analyzed representations of pre-trained models using mean-field theoretic manifold analysis and showed the emergence of linguistic structure across layer depth. Similar to these two studics, we aim to analyze latent concepts learned in the representation. However different from them, analyze raw repre- sentations independent of a supervised task. The reliance on a supervision task effects the type of latent concepts found and may not fully reflect the latent concepts encoded in the raw representation. Moreover, the post-hoc classifiers require a careful evaluation to mitigate the concerns regarding the power of the probe (Hewitt& & Liang 2019 Zhang & Bowman 2018). In this work, we address these limitations by analyzing latent concepts learned in BERT in an unsupervised fashion.
8 CONCLUSION
In this work, we have provided a comprehensive analysis of latent concepts learned in the BERT model. Our analysis revealed various notable findings such as, the model learns novel concepts which do not strictly adhere to any pre-defined linguistic groups, ii) various latent concepts are composed of multiple independent properties, such as proper-nouns and first word in sentence, iii) lower layers specialize in learning shallow lexical concepts while the middle and higher layers have better representation of core linguistic and semantic concepts. We also released novel BERT- centric concept dataset that not only facilitates future interpretation studies, but will also serve as an annotated dataset similar to POS, WordNet and SEM. Different from the classical NLP datasets, provides a unique multi-facet set of concepts.
While this work has led to plenty of insight into the inner workings of BERT and led novel resource, increasing the diversity of the observed concepts is an important direction of future work. Working around the memory and algorithm constraints will allow the creation a more varied concept set. Phrase-level concepts is yet another direction that can be explored further. Finally, a few works on analyzing word representations (both static and contextual) like Reif et al. (2019) (2019
and Ethayarajh (2019) have alluded that taking the principal components into account may lead to more controlled analysis. Incorporating this line of work may yield useful insights into the type of discovered latent clusters.

---

Published as a conference paper at ICLR 2022
REFERENCES
Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hessel Haagsma, Rik van Noord, Pierre Lud- mann, Duc-Duy Nguyen, and Johan Bos. The parallel meaning bank: Towards a multilingual corpus of translations annotated with compositional meaning representations. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 17, pp. 242-242222222 242- Valencia, Spain, 2017.
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations, 2019. URLhttps: httttps://openreveee... /pppreeeeeee.....oome.oee net forum?
(d=H1z= PsR5XX
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. What do Neural Machine Translation Models Learn about Morphology? In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (ACL), Vancouver, July 2017a. Association for Computational Linguistics. URL https:://acaatthollaaaaaao coll. uni monl-saarland
de/ htsppd////P1//217-1 10800
Yonatan Belinkov, Lluís Mårquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Evaluating Layers of Representation in Neural Machine Translation Part-of-Speech and Se- mantic Tagging Tasks. In Proceedings of the 8th International Joint Conference on Natural Lan- guage Processing (ICCNLP), November 2017b.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. On the linguistic representational power of neural machine translation models. Computational Linguistics, 45(1): 1-57, March 2020.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, and Stephan Vogel. Understanding and Improving Morphological Learning in the Neural Machine Translation Decoder. In Pro- ceedings of the 8th International Joint Conference on Natural Language Processing (ICCNLP), November 2017.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, D. Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the Thirty-Thin -hird AAAI Conference on Artificial Intelligence (AAAI, Oral presen- tation), January 2019a.
Fahim Dalvi, Avery Nortonsmith, D. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Dur- rani, and James Glass. Neurox: A toolkit for analyzing individual neurons in neural networks. In AAAI Conference on Artificial Intelligence (AAAI), January 2019b.
Romain Deveaud, Eric SanJuan, and Patrice Bellot. Accurate and effective latent concept modeling for ad hoc information retrieval. Document numérique, 17(1):61-84, 2014.
Jacob Devlin, Ming- Hei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, 2019. Association for Computational Linguisties.
Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Yonatan Belinkov, and Preslav Nakov. One size does not fit all: Comparing NMT representations of different granularities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1504-1516, Minneapolis. Minnesota, June 2019, Association for Computational Linguisties. doi: 10. 10.18653/v1/N19-111 154.
URL https: www acw.eb.wwee............ org/ wttpg//nthooooyy/l1---11o-
Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. Analyzing individual neurons in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4865 4880, Online, November 2020. Association for Computational Linguistics. doi: 10. 18613//3 10200000111222m....mmmmmmm main. URL https: :tpp::///e //www.
ac 1 web. areb.org//nin antho wotthhooooyy20000 emn p- ma in 395
10

---

Published as a conference paper at ICLR 2022
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the ge- o of BERT, ELMo. and GPT-2 embeddings. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Con- ference on Natural Language Processing (FMNLP--JCNLPP), pp. 55-65, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/1//D1--100 URL
httpss// Bettp::/www.acllwebb. www.waaclweb....... www.://ww.....attoooo/////.// 1006
Joseph L Fleiss, Bruce Levin, and Myunghee Cho Paik. Statistical methods for rates and propor- tions. 2013.
K Chidananda Gowda and G Krishna. Agglomerative clustering using the concept of mutual nearest neighbourhood. Pattern recognition, 10(2):105-112, 1978.
Shuhao Gu, Yang Feng, and Wanying Xie. Pruning- then-exxandding model for domain adaptation of neural machine translation, 2021.
John Hewitt and Perey Liang. Designing and interpreting probes with control tasks. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) Hong Kong, China, November 2019. doi: 10.18653/11DD19-110 1275. URL https: www oww..e...ot//attoooooo////oooee.
pp.
Julia Hockenmaier. Creating a CCGbank and a wide-coverage CCG lexicon for German. In Pro- ceedings of the 21st International Conference on Computational Linguisties and 44th Annual Meeting of the Association for Computational Linguistics, ACL '06, pp. 505-512, Sydney, Aus- tralia, 2006.
Bevan Jones, Jacob Andreas, Daniel Bauer, Karl Moritz Hermann, and Kevin Knight. Semantics- based machine translation with hyperedge replacement grammars. In Proceedings of COLING 2012, pp. 1359-1376, Mumbai, India, December 2012. The COLING 2012 Organizing Commit- tee. URL https::/p//..........t http:///aaaatttooooco www.olllyyyyor//12111 ott/ -1883
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( EEMNLP- IJCNLP), pp. 4364-4373, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 1.118653/1/DD19-4440 URL BRLhttps::/ http://wwwwwww...b.occccoc acwweb. wwwp.//a../////tooooooooo////o// D19-1445
Klaus Krippendorff. Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1):61-70, 1970.
Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. The emergence of number and syntax units in LSTM language models. In Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp- 11-20, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 1111655//1/N19-1000 URL https:: /tt:://wwwww.. www acwweb......... org/ wwwpeeborggaattttttoooee httpg//athhollly/111--
J Richard Landis and Gary Koch. The measurement of observer agreement for categorical data. biometrics, pp. 159-174, 1977.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the Ability of LSTMs Learn Syntax-Sensitrve Dependencies. Transactions of the Association for Computational Linguistics, 4:521-535, 2016.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A. Smith. Lin- guistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1073-1094, Min- neapolis, Minnesota, June 2019a. Association for Computational Linguistics. URL https:
7/www. ac web. htw.bboog//annhooog//////1// 9-112
11

---

Published as a conference paper at ICLR 2022
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov, Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/ 1692. 2019b. URL BRLLatttp///axxv.crt absp 1907.111111
Jonathan Mamou, Hang Le, Miguel Del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, and Sueyeon Chung. Emergence of separable manifolds in deep language representations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6713-6723. PMLR, 13-18 Jul 2020. httppppppe/ppocccenn......ooo www.rr.pres//111/mmmmme... html
Ana Marasović. Nip's generalization problem, and how researchers are tackling it. The Gradient, 2018.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330, 1993. URL.
https: htps::/ww0 www.:aalllwee. wwt.:/wwwww.....oooooooo////333
Julian Michael, Jan A. Botha, and Ian Tenney. Asking without telling: Exploring latent ontologies in contextual representations. In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pp. 6792-6812, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18666.1111 http://///2222..mm-mmmem emn p- ma in. 552,
URL https://www/
aclweb. org/ htwprr////nnhhhoooooooooo
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In Proceedings of the ICLR Workshop, Scottsdale, AZ, USA, 2013.
George A. Miller. WordNet: A Lexical Database for English. Communications of the ACM, 38(11); 39-41, 1995.
James W Pennebaker, Martha E Francis, and Roger J Booth. Linguistic inquiry and word count: Liwe 2001. Mahway: Lawrence Erlbaum Associates, 71, 2001.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis. Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D15 1250. URL org/D11 1250
Peng Qian, Xipeng Qiu, and Xuanjing Huang. Investigating Language Universal and Specific Properties in Word Embeddings. In Proceedings of the 54th Annual Meeting of the Associa- fion for Computational Linguistics (Volume 1: Long Papers), pp. 1478-1488, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http:/t////// /tt:://wwwt www. www.wwcwwe......c org/ Ptccholoyy/P16-110
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL 159clffe5be 1tt.ee11b00001110000000001
hwwp://eeeeeea...ee.......s../ee Paper. pdf
Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation of deep nlp models: A survey, 2021.
Xing Shi, Inkit Padhi, and Kevin Knight. Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 16, pp. 1526-1534, Austin, TX, USA, 2016.
Wolfgang G Stock. Concepts and semantic relations in information science. Journal of the American Society for Information Science and Technology, 61(10))1951-196) 2010.
12

---

Published as a conference paper at ICLR 2022
Xavier Suau, Luca Zappella, and Nicholas Apostoloft. Finding experts in transformer models. CoRR, abs/2005.0766 077447,,.7777 2020. BRLhttps::/ /wwxia.xivvv../..i/ aww.v..org/.b//2005...om
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593-4601, Florence, Italy, July 2019a. Association for Computational Linguistics. doi: 10. 18653/v1/P19/ 1452. URL https:::
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure contextualized word representations. Interna- tional Conference Learning Representations, 2019b. URL htttts///ppperrveeer .ett/
Sorum22d==SJSSnRcRX
Erik F. Tjong Kim Sang and Sabine Buchholz. Introduction to the CoNLL-20000 shared task chunk- ing. In Fourth Conference on Compulational Natural Language Learning and the Second Learn-
ing Language in Logic Workshop, 2000. URL https: Tetttp::/www.oor wwtp://wwwe...oo/attbbo.o..
W70-0726
Lucas Torroba Hennigen, Adina Williams, and Ryan Cotterell. Intrinsic probing through dimen- sion selection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan-
guage Processing (EMNLP), pp. 197-216, Online, November 2020. Association for Computa- tional Linguisties. doi: 10.18653/v 1tt::3///2220...mmmmmmmmaaa arc//anthology/200 2000 emn -atp:/20000mmmmmmmamaaaaammmmmma 1mmain..1
URL https: http://wwwwcccc www....ebb.b..
Bernard Vauquois. A survey of formal grammars and algorithms for recognition and transformation in mechanical translation. In A. H. Morrel (ed.), IFIP Congress (2), pp. 1114-1122, 1968.
Ekaterina Vylomova, Trevor Cohn, Xuanli He, and Gholamreza Haffari. Word Representation Models for Morphologically Rich Languages in Neural Machine Translation. arXiv preprint arXiv:1606.042 04217 2016.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Achhé-Bur Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeuriPS 2019, 8-14 December 2019, Van-
couver, BC, Canada, pp. 5754-5764. 2019. URL http::/paperss.eetee///// nips cc/ppper/ for otwprr/wee.aaeaaoooooooooooneeee waw.guaggg-ddessaaddd...
8812 1-12-xinet--m
Kelly Zhang and Samuel Bowman. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. Proceedings of the 2018 EMNLP
Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 359-361, Brussels, Belgium, November 2018. Association for Computational Linguistics... doi: 10.18653/ vI/W18-5448 URL httt:tt:: / /wwww a www.w..rgg//aahhooooo//l//-----
13

---

Published as a conference paper at ICLR 2022
APPENDIX
A CLUSTERING ALGORITHM AND BCN ALGORITHM
Algorithm Learning Latent Concept with
Algorithm 2 Concept Prediction with Logis-
Clustering
tie regression
Input:
V' word representation of all
Input: Xtroin word representations for train data, Ytrain cluster id from Algorithm
ning words
Parameter:
No. output clus-
Xiest word representations for data, probability thresh-
ters
Parameter:
for each word wy do assign wy to cluster C;
old
unique cluster ids from Yeeain
3: end for
2: M train Logistic Regression model on
4: while No. of clusters * K do
Xtraso and Ytrass
for each cluster pair cu,y do di,'' inner-cluster difference of combined cluster e; and cy end for
3: for each Jr E Xtest do
4: p predict К probabilities for each
cluster using M and input
5:
i mag then
cluster pair with minimum
6: if p;
value of
assign x cluster id G;
9:
merge clusters Cj and
8: end
10: end while
9: end for
B ANNOTATION DETAILS
For the annotation, we prepared detailed instructions to guide the annotators, which they followed during the annotation tasks. Our annotation consists of two questions: (i) Q1: Is the cluster mean- ingful?, (ïi) Q2: Can the two clusters be combined to form a meaningful group?. The word cluster represented in form of word cloud, where the frequency of the word in the data represents relative size of the word in the word cloud. To understand the context of each word the cluster we also facilitated annotators with associated sentences from the dataset. For the annotation, we pro- vided specific instructions with examples for each question. With the second question, the idea is to understand whether two sibling clusters can be combined form a meaningful super-cluster, For the second question, two clusters are siblings, which we automatically identified from our hierarchi- cal clustering model. We showed the second question only a sibling of the main cluster (cluster presented in the first question) was available or identified by the clustering algorithm. Hence, anno- tators did not see the second question in all annotations. Below we provided instructions which we accompanied with the annotation platform in form of tutorial.
B. ANNOTATION INSTRUCTIONS
The annotation task was look first look group of words (i.e., representing as cluster) and answer the first question. Then, look at two groups of words to answer the second question.
Q1: IS THE CLUSTER MEANINGFUL?
word group is meaningful if it contains semantically, syntactically, lexically similar words. The example in Figure has only numbers with two digits, hence, this is a mcaningful group. The labels for this question include the following:
1. Yes (if represents a meaningful cluster)
2. No (if does not represent any meaningful cluster)
3. Don't know can't judge (if it does not have enough information to make judgment.
It is recommended to categorize the word groups using this label when the word group is not understandable all.)
14

---

Published as a conference paper at ICLR 2022
241
1 8
23
18
1518 1 2522
26
11
21
19
30 20
28
27
Figure 6: Example of a group of tokens representing a meaningful cluster.
If the answer to this question is Yes, then the task would be to assign a name to the word cluster using one more words. While assigning the name, it is important to maintain the hierarchy. For example, for the above word group in Figure we can assign name: semantic:number. This needs to be written in the text box right below this question. While deciding the name of the cluster, the priority has to be to focus on semantics first, syntax second. and followed by any other meaningful aspects. While annotating it is also important to consider (i) their relative frequency of the tokens in the cluster, which is clearly visible in the word cluster, (ii) context in a sentence where the word appears in.
Q2: CAN THE TWO CLUSTERS BE COMBINED TO FORM A MEANINGFUL. GROUP?
For this question, two clusters are shown and the task is to see if they can form a meaningful super- cluster after combining. In these clusters, the left one is the same cluster annotated for the first question. The answer (i.e., labels) of this question is similar to QI: Yes, No and Don't know or can't I judge. Depending on the answer to this question the task is to provide meaningful name similar to Q1.
In Figure 7 and g
we provide screenshots for QI and Q2, respectively.
1800s
1812
1776 18401840s
1833 1836
186518158 1810 1857,
1841 i
1825 1842 1847 18331801
1826
1818 1943
Words 1341
and associated sentences are: whit snggengge
erotornateeo Quran Vitaria ciug
the 2002 Cache Tewerrie 15.00 diecer jette Eut Charion 1922 artie Jase Zark in (ope75)) it acter proath MMioo-Mioo Enr mecolie Piemooe matic
lnlt Gemmn eaciur chemiaa One ferm
greaph (3)) music farddy Mertoner (rggln)) in tage 70 sonpatha Blger Suger laga
sgrid (oni////eeae kodjon
41 prsseear Th persoiaiity lot beeil. tarnnaaheev natherhee ron pechenberg in uosa lol.
Is cluster meaningful?
Assevriic meastnghh
TCS
NC
Don't iroun cane
Figure 7: An example of the annotation interface for Q1.
15

---

Published as a conference paper at ICLR 2022
can the two clusters be combined form meaningful group?
1886 1 168 1872 1890
1880
1818 1841
1877_ -1897
1997 &
1823
1883 18621
nineteenth
1870s
1811877 1865 1860
1801 18 these
18 1850s 1840-817 81
1776
1935
1800
123
1182 1898
196186889 8988 TM4
1833 121007
1800s
1824
1229
17
1881 1866,
1898
T065s
8631885
1891 1882 1889
18951861
Thc
beis ponte coregertiin en cortiing pir dwall agundd
senia
lacas ust und
oteer crhoc crrtrtt
on wes lniping faom sou to
da diuukyy
and essecic the bncu
attaryte nevertate sage
aucestral waaer
eren inoousti
Figure 8: An example of the annotation interface for Q2.
Agree. Pair
Fleiss
K. alpha
Avg. Agr.
Q1
CC-C
0.604
0.604
0.915
A2 C
0.622
0.623
0.921
A3-
0.604
0.604
0.915
Avg
0.610
0.610
0.917
AI
0.702
0.700
0.853
A2-C
0.625
0.621
0.814
A3
0.602
0,592
0.797
Avg
0.643
0.638
0.821
Table 2: Inter-annotator agreement using Fleiss Kappa (s). refers annotator, and refers consolidation.
B.2 ANNOTATION AGREEMENT
We computed annotation agreement using Fleiss x Fleiss et al. (2013). Krippendorff's a Krippen-
dorfi (1970) and average observed agreement Fleiss et al. (2013). In Table 2 present the anno- lation agreement with different approaches.
B.3 CONCEPT LABELS
Out annotation process resultant in 183 concept labels. In Table and we report LEX, POS and SEM concepts labels, respectively.
C ANALYSIS
Figure 9 showws example concepts. Figure shows examples LIWC and WordNet concepts found in lowe layers.
16

---

Published as a conference paper at ICLR 2022
LEX
POS
LEX:abbreviation
LEX:suffi:ties
POS:adjective
LEX:abbreviation. and acronym
LEX:unicode
POSSadiiectvvessuueraaavve
acronmm
sEX::uffi::ly
POS:adverb
LEX.case:title_cam
LEX:suffi:iion
POS:compound
dEX:dots
propperornn
POS:noun
hyhhnnttdt
veb..preennet httprreeetttiiidr.ooooooosiiiiii singulr
POn:nouns
LEX:prefix:in
POS:verb
POS:number
Table 3: Concept labels: LEX and POS.
SEM
SEM-action
SEVCenlerainmmentmusiitt pockkokkkaan and musicians
SEM:honour
SEMaationbbdyyffce
SEMeeneersinmmettpptt
SEM:language
SEM-agent
aenattrrainmennsspttteeeeeee foothall-game play
SEM:menurementteet
SHM agent passerby
SKMiccnertainmeenssppr hastatl bascball
ingerial
SEM:entersinmmensspptt haskerhall
SEMumenuremmnt length
SEM aninnalscaaammne
SEM:cnitrtaiinnsnttsptttst name
SEMumaasremmntt kongth.and_.wwight
MtMart
SEMumodaazeccal.mme media
SEMatribulchhuma
SEMMemierrainnnsuusptt..t football
SEMunnds:tvehamnl
SEMtbaby rellted
30X.:mttrrainmeenspptttggga. gamportttaamme...oo.mmm.oo score
SEM:medicine rrelated
SEMibrand
SEXCcenerrainnnennsptttt game time
meded
SEM:medicine medical condition
SEMacrimeassault
BEX::mmieraainmennsppttiii hockey
SEM:metaphysicalleee relate
SEMscrimettoo
LEM::nttrsaiimmentspttttppyoyy name
SEM:mining and.energy
SPMicnierrainneenssptttttt spottiinnnnettttttitnnnn
SF:M:ommii_emlls
JEMrdeefnneeammyttile
SEN::nnterainmentsstttt name
SEM:named_eniity encittclcctioon
SHMMonnertainmmett spert
St:M:aaedd_emii hwwp:e/eeemiilykkaitonttto
dennorappyy
W.Meenterainmeensspot winter winter.sport ssport
SF.:named, wwllitcccaatiioollly _anuucuunty
SEM:nomdd wot.im/ddnnttllllooloeo iucilyylaaaiioollllyl.. attaitt
SEM.
SEMifaashion
St:Mnnamed_emiie wet.eeeddeeillyyyoolaeoe contoyyllationncoonnty
SEM:fsshionragporrl
cmlityoraamization
SEM ElMaatotiion.... httpuaaiiieaad.roooooooo rocossry
SEM:tachionccolle
SEM:named putily:person
SEM:entetainmmnt
SEM:inancial
SEM:named enstyspersen:firs name
JEM:cmertuinmeettacen
SEM-financial money.
SEM:named eniity:prsooniiiiiaa
SEMofinancialsttee
pntti:person:llld name
enteemmtttiimmeeetiiiieeee games thrones
SR-Mffond wine related
S::Mnegusiee
SEM T.MMeeentttiimeeettttnnllllbl
SEM:food and and_plant
SEM:aunher
and
SeM feminine
St:Mmuberr
SFM erMeettaiinineeegaaaeea
D.Maggapoliiics
3FMmunbercardinallaa spellel
SEM SEMecccctiinmeetttmnii
SEM destiiistireemiiiivalll StMtiisttaiewwoo
SEM:groupstudan
SEM:mumber flooting.piin peint
SEM mwweeettiinnnnnmeeeeeeeeeo
St:Mmumber less Jundied
Table 4: Concept labels: SEM.
17

---

Published as a conference paper at ICLR 2022
SEM
SEM.number:one hundred to five hundred
SEM:origin:north america.mexico
SEMrnumberrordinal
SEMoorigntnortt.mmriiiaaa .mmrricaauuaaaaliffmiiia
SEM:uumberppercenaage
SEMooriginnpolynesia
SEMLmmmberrquaaiityoonceee
SEMoorigin:porttuuess
SFMmmumberrquunittyyvuuue
SEM:number.whole
SEMrorigin:spanish italian
S
SF.:polarity
FFAl:oggaiizziiongeeeeeneenerrr affairs
FEMrenovationn-rlltee
SEM.oorganizattonnoccca...g
EMMssiience:booloyymiee biology
SFM:ooiignnaarric
SEMrsciencerccemisstr
SEMiooiginnasiiaarrbb
JEMssieenccchhemistrymaaaiaaa
assian SE.Moorigin.asiaxeastt..
SEMsscience:geolgg
SEM.ooiginnasssiibcchnaaaaednd colony
R2M:sea.relatee
SEEMoorggnnaaiimmyaaaaa
FEMssevicc-inddusty
SEMMoriggnn:ssiassouuthh east-asiann asian
SEM:technology
SEMMoorgin:aiiassuuootinnnnn
SEMttchholgy::oommiiiiio
SEM:ooiggin:europe
SEM:rechnologyy.eecctoniii
SEMMooggin:curropcbolggguuu netherrans
SEMMoorggnn:eurgpe:duuc
SEM:time
SEMMoorgginccurrppeffracc
SEMttine:amiveessary .dwecriptorsssssrsssrooo
SEMMoorignneeunpeeggemmay
SEM:time:century: 18thtthhad _ad__nnn9l
SEMMoorignn:euunpeegemmmyyy related relattd
JEMrtine:months.of.. .oe
SEMMooiginncurrpeegrree.... ind_ltlly
SEMttimettimffrrme
SEMMtransportationnwaay
SEMBrorgginccunppepootugglll
JEMetransportaiionnwwyyyy
SEMMooiginneeurppessweeee
SEMuunamedd-eniiyyzoo
SSMoonggin:uuoppesiizrrraaal
SEM:vehicle
SEMMoorggincuroppcuuk
SEM:vchicle:bike
SEM:origin.frecch
SEM:weather
SEM:origin: america
wisaterrrddii S SYNiposition.firtt..woo
SEMM:origin/latin.aama wawaatiiaammiiarreeatttee
Table Concept labels: SEM.
18

---

Published as a conference paper at ICLR 2022
d.j...D
I.D
P.K.
T.I.
A.O.
J.B.
T.
Benfica
o
E. ON
G.M. G.M.K.J..
He
KC
I.D.T. T.M.
Oliveira a Gerals
Azores Verde Curiting
Pereirag Ferreira
C K
A.G.
D.A.
G.E.
P.
M.S.
Brasil
RlouzilO
PAULD
A.M.
C.J.T.S
Z.
Brasilia whnas Brasku00edlia Bahia
E..
D.
Jolu00e30 JANEIRO Madeira BRASILIA
J.C. J.R.
J.R.
J.W.
JA
Sousa
LEX:abbreviation. personninn:iii,,,l,
SEM-namdd LEX:case:upper case
POS:noum; SEM:namedd Pority.Pottuguese
plainly
circalest
Firstly
pirstlyvariousiio
Variously
secondly
tuac
lifetimes
Firs
consciooooooooociiioo www.yyyluuyyyyyyyo convincinlly
Historically
additionally
unhel
unelected
inscrutable
unuu linhhig relatable
ununiounrivlledd Unambiguous moree
cnaaffordable @utekrappeenn
inaliracivve unverif weveriifabllueediie Cenenn
progressively
Frankly
substantive
truthfully
creatwelyy
structurally
Infio infrequently a
unfiltered unma
checcceed unieg stee
underrecognized
factually
principally
unplanned undiscovered
sccerednännmii
whereis
chiefly alternately
Villenniia
apprxx Innreeoeer
wheresapon
(c) LEX:sufix::l, POS:adverb
(d) LEX:prefixuu, SEManegative
contraction drew
itch aches migraanos flushes
clicks screengral Server, YouTubers Chats Pag
palpitation
cramps
the
sunburn blemishes itching
itchy
rashes
Micees
blisterree dneess
pimples
renteerrennsss sweats
colds
rosebleed
Loo
webcam weets ts
retweet
subreddi
samer
Youtubev
login
Vlogger-
siaar one Tr Trending tweeters
in Files
shart
worgch Protoco chatbot
es Emieinaors Scrolling
hashtags
bloatingheartbur
sores,
Gmail
blogs
raigsliss
médiaTyper
heartbeats blemish
sringles
contractions
vieg
Cloud
seachablle ret retweeting
tesbbar
SEM:medicalmedices condition
SEM:mediasoocill media
spiders
zebra
panther
rock
wasps
Burmese
mole
wildcats
bunny
Kangaroo
Frog
Unicora
Blephanc. Paw
wasp
_Paw
Badger
Ahino
badger
bootle
boar
scor
arpo
EccorpooRee
moths
hounds
moose
Predator
buffalo
Rat
Jaws
Farrss
rosteves
crows
peac
parrot
eagles
Beaccck
plumage
ducks
ock
Jays
reds
Avian
vulture flamingo
hawk
pigeon
Hen
swaliow
feeder
Pigeons
avian
cuckoo
swans
ow
flamingos
crow
and
carrage
Goose
falcons
rouster
Owls
cackcias
(g) Land animals
Flying
animals
snail
oata crabs
eggshell
Lobster
octopus
oysters
Oyster
catfish
eel
verries
troutEgg Trout
oyster
snails
squid
Crab
(i) Sea animals
Figure 9: Example Concepts
D PRE-DEFINED CONCEPTS INFORMATION
D.1 PRE-DEFINED CONCEPTS
Table 6 provides list of all pre-defined concepts used in this work.
19

---

Published as a conference paper at ICLR 2022
Grandfathe handfathercomm
teething
paternal
Nanny
Lipinski
Earhar
Tooth
Teeth
nephews
Grandmother
aunts
lipids
earmarks
anal e
Beaker
beaked
sathary
Book
evebal
tange
Lipo
OralEars
Eyemouth
Far
mouthing beak
mouchwatering
eyewear
orphan
orphaned
Grandmothers
Lighters
widows
err
Nephew
mommy
WivessWives
orphans
aam
Aunt
momma
earmark
Mouth
lipid
eyelash
mouthed
saliva
Eyewitness
eyéliner
Grandpe
grand
Courra
Mothers
Granny
Fathers
salivating
Eyewear
Greadma
Widoww
Gr
earphones
laws
eyelid
Mommy
grandmothers
mouthplece
(a) Layer I: LIWC:bio
(b) Layer 1: LIWC:social, WordNee::NoumPerroon
awe-inspiring fondness
awe-
WOWS
admiring
a
Eoperin
fondle
Settttationn
tt eern
Eastbourne
south-wester
delights
Excelstorpraissnn
repect
fondue
admirer
fondateur
admires
WOW
excels
Fondation
esteemee
excelling
excellingPleasure
and
Praise
eeteeesteem
fonder
Pleasure
y
joyfully
delighttnng
exce
wowing
joys
joyride
Excel
Westmorelane
northerly
WEST
North East
Northcate
south
west-northwwes
werdde North east
SouthallWeste
South-Eas
EAST Eastmond
North West
Westilff
No Southwell
Westgate molterly
ghrop
Weree
Wastport
wortheo-ooorrtvvveeee north-westem
Northbourne
someess
NORTH
ecstasy
Excellency
vutiice eastere Miniieae
North-South
complimented
north-south
(c) Layer 2: LIWC:affect
(d) Layer LIWC:space
retorted
educator mentored
scoffed
Smitine
pagter
Chuckles,
grimace
smirk
giggled
chuckledwed
Lughing
giogle
wwr
wink
Teaching
Educators
Professons
Teach Trainers
tutors
tutorial
giggling
me
sarcastically
Coaching
Mentor,
cuttorialsmentrraee
frowning
frown grinsSmile
frowned
grinned
cower
instructors
tutored
educating mentors
Coaches
Educating
.tutor
glare larghssarcastie
kidding
(e) Layer 3: LIWC:affect
(I) Layer 3: LIWC:work
Figure 10: LIWC and WordNet concepts found at lower layers.
Type
Concepts
Lexical
Ngram, Prefix, Suftix, Casing
Morphology
Parts speech
Semantics
Lexical semantic tags, LIWC, WordNet, Named entity tags CCG, Chunking, First word, Last word
Syntactic
Table 6: Pre-defined concepts
D.2 DATA AND EXPERIMENTAL SETUP
We used standard splits for training, development and test data for the linguistic tasks (POS, SEM, Chunking and CCG super tagging). The splits to preprocess the data available through git
20

---

Published as a conference paper at ICLR 2022
Lexical
Morphology Semantics
Syntactic
Moonelaadddnee Matcha
7.0
1.0
1.0
0.6
0.6
0.0
0.6
0.6
0.
0.4
0.4
0.2
0.0
Caring
Suffwee
POG (134)
WORDNET
Chunking (03)
CDG (207)
Figure 11: RoBERTa (numbers inside brackets show the maximum match across all layers)
Lexical
Morphology and Semantics
Syntactic
Nomaalaed Number Maiches
1.0
1.0
1.0
0.0
0.0
0.4
0.4
0.2
02
0
3
7
10
11
2
5
6
10
11
5
12
Layer
Layer
Layer
Caxing (271)
Suftieas (250)
POS (382)
WORINNET (14)
Churking (124)
CoG (250)
Hgrrms (730)
LIWC
3EM (199)
FirsiWlord
Figure 12: XLNet (numbers inside brackets show the maximum match across all layers)
repository released with Liu et al. 2019a). See Table for statistics. We obtained the understudied pre-trained models from the authors of the paper, through personal communication.
Task
Train
Dev
Test
Tags
POS
36557
1802
1963
44
SEM
36928
5301
10600
73
Chunking
8881
1843
2011
22
CCG
39101
1908
2404
1272
Table 7: Data statistics (number of sentences) on training, development and test sets using in the experiments and the number of tags be predicted
E COMPARING PRE-TRAINED MODELS
How do models compare with respect to the pre-defined concepts? Similar BERT, cre- ated layer-wise latent clusters using RoBERTa (Liu et al., 2019b) and XLNet (Yang et al. 2019). We aligned the clusters with the pre-defined concepts. Figure IT and [2 shows the results. The overall trend of the models learning shallow concepts in lower layers and linguistic concepts in the middle-higher layers consistent across all models. few exceptions are: suffixes have consis- tently decreasing pattern for XL.Net and RoBERTa, the FirstWord information improved for every higher layer till the last few layers. Moreover, XL.Net showed significant drop in matches with the linguistic tasks for the last layers. The last layers of the models are optimized for the objective function (Kovaleva et al. 2019) and there the drop in matches for most of the concepts reflects the existence of novel task-specific concepts which may not be well aligned with the human-defined concepts.
The number of matches as shown in brackets in Figure 4 and 12 provides relative compari- son across models and pre-defined concepts. The RoBERTa and XT. found substantially more
httts://gi com http://toao...ooooooooooooooaaaa
21

---

Published as a conference paper at ICLR 2022
Concept
+Type
LEXcoloreviation acromy
4,921
LEXaabocvistion. siMMeddee-nniit----.. autitty oeganization, poopproee
8,615
1,394
LEXaabrrvisiion SEMrsume__miity peteed__miiyynoomiiii penominiial, [EXceeeegpprcco
2,039
177
LEXcusetile case. POSpoope-soon. soun, ZEM-cccctiinentsooo SLMaamed putin--enionllss nume
429
29
cEx. cise. POSsoop--tom SEM namol, jarratyeeroons SFMameed wwwp:eeeoiiiyyyoooootttoo SEMiaaiggnneuqqe germany
2,299
1,033
LEX.csee P0S.poop--oou, SEM partieeaayrrrrreereoo nost
LENcssetiilleccee POSsgope-toom, nemoii_iitt SiMtamed emity
LEXcsse:illl cus,, POS:proper soun, SEMoaggineuumppu.
1,885
LEx SEMeentstinnnenn piopioeesooonn SEM mamed patttt-pessoonllle pame
1,200
LEX.case:illle..t ctse, SEM samedaaadddde location, PUS proper--ouuu
2,622
167
IEXcssc:lilll..ss somol J locstoon, SoMoriginnottl_yem httpi//mottimmmmerrccrcaoo jmeriieneanaaa POSpeepe-toun
2,907
236
LEXcasecillee cuse, SLMzamed uniyypensos, SBMeestrtinmmetaa
P'SSproper noun
4,154
589
LEX ease SEMuumol_mti (ut.tttiyyy SEMoriign asia, POS-pcuper-nou
punty person, S SeMtatertimmmet ficionall of throne, POS geoper-nom
227
64
2.875
3,502
case, SDManigingalyeesi p0Spproper soun. namen estiy
LEx.eaeclillee case, SYNpoitiinntss word, "OSppopes-toun named, cainy perioo
3
1.772
ieperseeg,,,,, SYN pyNioiiiinnfiie first, word SEMmmeel.mmit lniittyllotiiooiityy poSpprppe-nn
3,279
doXdons
22,547
31
LEXhhyphenaeed POStadjective, POS SEMagropolticcs
2,309
205
LEXIiyppenneed.. SEM demagribbyy LEX:suttx- yyar---kd-- #OS.adjective
7,847
251
IEXhyphensddwwel
15,922
1,000
peefreii SEMingutive
LEXsulliszet aisseiiceeeeeeeeebi
LEX:uuffic.ly,
IuX adv
6.187
Liftii adVedd
75,166
1,000
LEXoofliscyy POSadvvrr.
1,943
240
LEXsufficlyy POS SaMtmmbeeraddinal FX unierdee
489
15
906
411
adjjctie, SEMuutiiblte
mom zomberouitiy SEMmumberooee hundred hundred..
SuM.esertaimmettton cnntttrtii StMtetterannmee spottguue score
gsse, entetsinmenttsuurt foothall
parttp Pf playerttame,
1,367
POSpp noun. LEX:vase:lilee case, named eatly. SEMonngin america relatted
2,50
POS moun. LEE caseriblleese saw, SEM named.emdet lmcily: STMoriigtneurnpeellgiim nellnenbes
1,740
83
POS PUS.graper-og noun. LEX case:itte case, named eanny-prrsos, SEM M.M.cnnttaamee--pptt. name.
1,051
166
POS P0Sppvoprssnous SFM meniie jenity. SEM media chammel
3,359
55
POS noun. SEMMnamdd entity. SiMoogggineegeeefffaan PoSprupp--nuun
2,165
POS,groper named entity, SLMootginseuth umericatruzal
3,524
propermuuss mamed,cnitty
lecatin. leetty- FeMarrgincurrqeefrrnn
78
POS noun. SEM contmtllenttypppe. pemsum SEM slemugmahp muslim name, LEX:asettilee.o cuuse
2,748
POS proper-nom.. noun, erMorriinurrpeegrrnne POS
SEMmaeed. locatio LEX:caertile.cos
1,951
85
7,763
1,227
POS verb
893
310
POS SEMractio, SEM-nggaive
1,858
427
POS vertprset chind person singular
Table 8: Concept labels (LEX, POS) with token and type.
matches with most of the classical NLP tasks compared to BERT. The difference is much more pro- nounced in the case of POS with RoBERTa have the highest match. This may reflect that the latent concepts of RoBERTa follow a closer hierarchy to POS compared to other models. On contrary, for WordNet and LIWC BERT showed substantially better alignment than the other two models.
F BCN DETAILS
In Table SEM.
and
10 provide concept label with the number of token and type for LEX, POS and
22

---

Published as a conference paper at ICLR 2022
Concept Label
Token
# Type
SEM-action, SEM:negative, POS:verb
2,908
364
EEMaaction:oodyfface
1.553
SEMaagen, POSsoonnssngular.
4,491
483
SEM:agent, nOSnoun:singular,
12,227
348
SEManennppastertyy moS
330
SEM-animallland anninll
2,121
419
SFM=animul-lasd_amm
1,705
129
SEMaanimal:land animal,
1,783
83
SEM:aninalsse
SEM-art-hhysical
5,041
647
SEM:boby related
2,184
SEM:utineasssull,
2,707
275
SEM:crimetool,
SEM.defenseaamyy
6,619
331
SEM.delens--amyyttitee LEXicaeetitle.cose POS:proper-noun.
255
21
SEM.defense
9,803
516
SEM.denegraphy..
POS:adjective
19,150
SEMsseenograalyyageey
27,929
274
J.Nccasettileeeee case, SFM::amedd_enity
3,774
499
SEM:donation
aecoooooornndddooovrrr...ooo LEX:case:ttte..om naMenamed contity
2,642
412
17
SEM.cntertainmenttie and_tv
2,236
S
SEM.culcttalnmentttccooooo
114
5
JEM::nletaainnnenn:lllt
6,104
SEMIzntenttinmmen:iiim and Iv.
5,007
196
JEM::oleetainmnenttgannnng
2,690
JRM:nntentaimmen:::uiicc
6,982
710
3,768
SEMeoniccttiamnenn:musisuuu reck-bands and musicians, LEX:case:tille.oom case
2.939
436
SEMeenicttainnmeeess.ppttt
3,616
154
SEMMeniirtainnmenttspptt
9.734
704
SE:Meenlcttainmmeel..pputt
15,491
2,178
STMMenttertainneenn:sppp,,
1,979
484
SEM/:ntectainnmentspprrr.
15,973
1,094
SEM:cntccttinnmenttspor..
5,619
SEMionlcctaimmmnttsppr.,
SEMooriginxeuropeuuk
18,300
232
SEM:cntccttannmeettpprr.. veciicccc.........c...........
2,335
76
MEMMuuilctaiimeenssptttannnn football-game play
14,621
388
SSMMemtttiimmettssottgggmeeeeee ycure, POS:ummee
JFMeentertiinmeentspptttttte selection
12,919
815
SEM.cnntcttiinnnntsssotttttiieee sport
6,112
SEM
2,600
149
SEM.fsshionnappprrl
707
108
SEMiaashionncloohhe
1,882
376
SEMtfinamcial, POS:noun,
7,003
308
SFM:financial, POS:noun,
907
47
SEM.finaacialsstock
995
SEM:riod_
3,601
256
SEMfooddswine_reattee
494
20
SEM.gcnkkr.fmmminee
SEMtumaamcd-enitttrrole
5.627
NEM-gruup:tudent
596
60
SEMiiustoric
2,011
S JEMiisstoricmmmeivvlll SEMooriginceuuge
621
85
SFM system, siMMooiginneumppeeuk
SEMrlanguage.
8,166
336
laM,langagee.
607
101
JEMalaggagge
foreign.wodddd word, SEMoorigineeuropoddcch
1,038
394
langaagee word, SEMsoriginffrenhh
3,321
SEM-language foreign word. SEM.orgin:feechh SEM-aamdd entity
2.993
867
TiMemmmsrremeethhhighh I.EX:hyphenated
1.397
275
SEMmmuuuremeent
3,112
673
960
156
soMimediasocial medii
42,108 108
975
mEMmmdiissocial media,
192
116
SEMmmedicine::uug related
1,625
SEMemedicinnemmdicall condition
1,340
261
SEMemetaphysical.ddaae
SEM:mining_ande energy
4,072
279
SFM L_X:casettitlle.c ccase
2,784
SEMnnaedd.cntin entity, POoppropr-, SEM.chicle:bikk.
1,353
139
SEM.named_enntiy
Poopprope--noun
4,870
79
SEMnamedd_cniip JEMMecntrtiinmennttprrr hockey.
name, LEX:case:title case
2,570
101
SEM.named entity. S.M.eannertanmeettpptttrngggg SEMeemettainmmettspoo seam name
223
SEMzmaned_ennlin entity. SEMauiiinn
POS:proper SEMmaned persoo, LExceseetiilee
835
258
SFMmaamei_enttyy aFMMariginasiirmyamme POS:proper-naa c.FXXcase:tillee
2,758
172
Table 9: Concept labels SEM with token and type.
23

---

Published as a conference paper at ICLR 2022
Concept Label
Token
Type
SEMmamell.cttty SEMaaiginsoorupegerre related, LEXcase iile.
2,958
725
SEM:nmed SEM www.:e/nnnnnnoooeeeeerrtt pumpe, POSppoppennoonn
1,159
331
SEM:aaed.caaity SEMManiginneeeeeeeweee POS noum
SEMmumoll.aaity lacallllo LEXssectillleccr siiiigggiaa Last
330
SHMmmmell_mmttty location, L.N.assclilleecss SEM originlatiin angiariatinammeriia
152
7
SEMinamed_snttpy locatty,,catto,on SEM pripiiineeepeeeerreee. and jnd,italy, LEXccasetiillecesss SEMiassnrieeeddeall
2,512
407
SEM:named, cotityllcctiionciityy.. and county. LEXccaeetirle..c
359
SEMmnmed entity lacttillactioocciilyy and county, SEMooiginnetth americausscollfmmi
19.267
297
www.ggnnnoottmmmriaaaamoo. POSgpoger-nouu, L.w.aa.e..l..eccle....... SEMinamed lncitooittt sufiinciiee
SEMinmed__sttip sat.-_etttttllaatoooaatt und, state,
1,946
123
407
15
SEM:named cutilyoorunicaatin, POSpproper
LEXacronym
2.379
266
SEMinamed estity persty
POSppmper-noon entertamnnnn
306
SEM:named, satitt-person. POS:pooper-noue noun. L.EX case
1,159
783
SEMinamed catity-persen. POS:proper noun. LEX-llyphoatedd SEManamed entiy:persomllat name, LEX.casectille case
3,107
630
SEM-named casilypersen. SEMooigannsssaarh
2,532
851
SEM canilt person, SEMoxipmasiiiinnsciiiiiii
EXcesseiitlccane popperonn
SEM:named_entity pww.actttppsssnn.iiiee. pww.eeeffissssmmm....... LeV:asettalle.as:: SYNposiion:fftt.wwat
1,024
456
SEMnammd catily-prsee: first name. POS-prope--oon SEMsmiattaimmnnn. LEXocasettid case.
479
189
SEMramedl_ctt
los.
LEXxasectikkccoo SEMesiiiiaai POS proprr-oooo
3
1,491
SEM-named
poperrooo,,,, ewe-title_nsee
SEM-named_enatty person name, poope--ooo,. L.E
case, SPMooigin:euop eeMeeeggnuuneeeeeemmyy germany_related,
GEMammell.mtig serson last name, SEMeenlertaimmnn LEXicasttillc cose, POSppuper-nom
3,884
696
SEM:named_patiyy person list, jume, SEM:coienainmeeeil hollywood, I LFXccseeiificc pase, PoSpooper-nous, SerppppeoooonSSiMMeeettiiiiiae SFMeenteriinnnnt
930
300
SEM-named_eute cari_eettttyrreeee
SEM::nierraimmentqppe POS proper-ooum, LEX case.
216
SEM:named putity-person person-
90Sppepee--noo noun, SEM:named paiilypersen.-as name, SEM:ooiginspanist italin,, POS-proper pruppee noun, LEX:casettille case
case.
707
251
SEMinamed entily-SEMMoiggiii hrM.oooiiinennngeseeeieeee
2,290
80
SEManegative, POS:noun
4,141
365
SEMsumber, aakeemetiinooootpptttaaa score, EEMeeaectiinnennssttttttett
LEX:byphentted, POSmumber
SEM:nmbberalphu numeric
1,222
387
SEM:uumber SEMmumberccrdiaa spelled, SEMmmmeerless hundred
8,203
131
LEM:nubbercadaiall speiled, SEMnumberless than hundred, LEX-hypheated,, LIWC:funet,
634
136
SEMoouber:Booingg peint, a0Smumber, =EMammberrquunity....o mumbertBooiiggppiii number, SEMalimaxiaa moncy Jigure, number
SEM:number senine
F poslippceaaeteeeeurreee surreney.symlel LEX posirionhefone surtix_illionn
4,568
798
SEM:nmixesrdiral SEMtime, SEM setertainnnn spor:same lime, SEMrenettsimmmisspottoooolall
589
103
SEMmumbcrauumiiyyyug sullixi,,, SEM mumber.
10.689
2,997
mumberruuaiiiyyyygg suffixt., SEMounnhershhoee POS number.
1,4R0
SEMoogaaniaaion goreiiaaiiinngeeemmennnen agMaaget,
14,744
516
EeMoogaaizationngoeee gov.rttoooggemmmeeeeeeenoeeoo affairs
3,139
46
orMoorgaiiatioon.oo ngo
5,132
SEMooiginnafriic
case,
antity
257
SEM:originasiaa cast asian, SEM:named, entity. propproper-ooom,
2,564
EEMorrigineeurpee proper coun
3,734
108
SEMapolaiity LEX-hyphenaied, LEX pxx.peiixcaaiiii.mmmmmmmm
6,881
662
SEMrreooationn related. pexcpreix..ne
289
biooiiieeeeoollyy.iii.. biology
2,105
403
SEM:scienz:chemiitp
1,600
131
SEM chinecctccc.,
3,138
321
SEMssiennechemmitryyn
SEMssieneeggooogg
5,453
663
16,583
287
SEMsseviae-innussry
2,743
244
SEMrechnology
4,480
149
SEMicchoology communication
4,138
463
SEMtnchholooy communication, SEMitchholooy electroni electronic
5,155
886
SEMrrechnology wla.hmaloogyeletttonisseeeeooo www.toniccttooggeeeeeeeeee hew.eeeeoooggeeeeeeeeevvaaaaa
387
21
SEMttime,
21,715
526
SEMtimessnivesarr descriptors
SEMtrimeceentary 18th soh..ad..1 19th
1.041
204
SEMrtimemmonths of year
16,041
37
SEMttieetimncraane LEXssfiixties
1,062
SEMtümetiinnciann, aljeciive LEX.hyphemmtee
DEM:toapportaicccwaayy noS.
2.637
211
MMMuramppttaioeeaayy.oo POStvoun
1,722
72
SEM:vchick,, SEM entity, POS SEMbrand
811
103
SEMissatherrdsssserr
29.698
1,631
SYN pyN.positonniinsscoo wond.
7,913
1,315
SYX positioonniiiit.ooo wond.
2,252
365
www.ppnnnfiiss.uueeee wond. I.EX cFXecesessrrleees poper, SFM
342
positionnnn,,,,,
casenille SEMMaddiaaadeeemstnnnii
posiissitin word. L caserille case. diMheerherdsssaeer
1,351
232
SYN SYN position:fisss wond. POS proper-nou,, L.EX caserttle case, SPM named entirypenooe
2,059
879
positiinnnn,,,, poggrooer LEX casettillee SEM seiimedddeeiipppsscoo. SEMoorginnasizzesss.sai SYN.positoon:inst SEMaammed eniity
134
Table 10: Concept labels SEM with token and type.
24

---

Published as a conference paper at ICLR 2022
Example of tags
Words
confine, cripple, decimate, demonised, disappoints, dislocation, disobeyed. disqualify, downsized, jeopardise, hijacked, trashed, traumatizing, undercutting. underpins. underplayed, undervalue, spoilt, spouting, spurned, stifled Gaze, LAUGH. Tears, admiting. blushing, chuckles, clap, frown, gasp. giggle, grimace,
SEMaaciionnegative
SEMFaction:body.fen
smirk, twinkle, wink
JFMaaciivsm::ddmmnsnniii
Aggression, Arguments, Armored, Attackers, Attacks, Racklash, Ceasefire, Clashes, Cops, Counterfeiting Ants, Bees, Beetle, Cobra, Frog, Jaws, Leopard, Paw, Snakes, Spider, Sumatran, frog Salmon, Shellfish. Trour, catfish, crabs, eel. berring, octopus, oyster, oysters, suail, squid, trout
anMaanimalland Janimal
animal sea, animal
Table 1I: Example of Sem tags with associated words.
25


Reviews:
[{"review_id": "WEhU5rVlMw", "reviewer": "Reviewer_3Uym", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths \n\nThe problem that the paper tackles, analysing latent concepts captured in BERT, is challenging and needed to understand the model. It is well in line with the trend of model and representation interpretation. \n\nThe proposed solution to the problem requires carefully manually annotation which was carried out thoughtfully. The innovation is that there are several syntactic and semantics aspects taken into account, including hierarchy nature of those concepts. \n\nThe analyses presented in the paper do lead to several findings about the knowledge of BERT, thus they back the claims risen in the paper.\n\n\n### Weaknesses \n\nThe paper doesn't go any further beyond 1000 clusters / concepts, which doesn't seem to have as wide coverage as e.g. wikipedia. Hence, one would question about the used clustering method, especially whether it is good enough to discover a wide range of concepts / clusters. \n\nThe methodology proposed in the paper requires expensive manual annotation. This leads to the problem of replication. Besides, although the BCD dataset has some potential, it is limited to only BERT. It does't seem useful when other models are examined. \n\nI found the BCD development (section 7.1) problematic. Although clusters were manually annotated, it is unclear how accurate an instance is assigned to a specific cluster. For instance, although annotators intuitively agreed that cluster A with N instances properly present concept \"animal\", how many instances among the N instances actually belong to concept \"animal\"?\n\n\n\n\n", "summary_of_the_paper": "The paper analyses concepts that BERT can capture. The core idea is to use agglomerative hierarchical clustering method to discover latent concepts (i.e. clusters) with hierarchy. Those clusters were manually assigned labels / meanings. The authors then used these labelled clusters to analyse concepts that BERT captures, across several syntactic and semantics aspects. The labelled clusters were also used to build a concept dataset (namely BERT concept dataset - BCD) for 1M instances (i.e. words in contexts).  ", "summary_of_the_review": "The paper tackles an important challenge with thoughtful methodology and reasonable analyses. The paper is also well written, presenting several findings about the knowledge captured in BERT. \nThe proposed methodology however has problems which limit the contribution of the paper.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VRmvf5Snm-D", "reviewer": "Reviewer_kVr9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally, although the method is simple, I think it is effective and solid. Also, I appreciate the efforts in the annotation. I think a direction for improvements is a more in-depth analysis since this paper spends a lot of writing on development (which is fair) but less on analysis. I would suggest simplifying the development details (Section 4.2, 4.4, 6.2 can be shortened, and algorithm 2 can be replaced by a simple textual description) to make space for analysis, specifically: \n\n## Alignments and Differences between discovered and pre-defined concepts\n\nThis is a point worth pursuing because the concepts learned in an unsupervised way respect patterns in data much more than human prior definition. I was expecting a detailed analysis about the concepts that are NOT aligned with pre-defined concepts in section 6.2, but there are not and the existing analysis simply reinforces what we have already known. What are these non-aligned concepts? Are they interpretable? Are they more about syntax or semantics? Why they do not align with existing annotation? I strongly encourage the authors to discuss these questions in detail because these are the questions that lead to new insights, rather than reinforcing what we have already familiar with. \n\n## Why latent concepts emerge within BERT\n\nThis is another interesting question that may be worth pursuing. This is essentially a question about how large-scale masked language modeling automatically leads to clusters of representations. My intuition is that this should trace back to the effect of contextualization, i.e., clusters emerge because words within the same cluster share the same context, and different clusters originate from different contexts. There could also be other explanations, and I would like to encourage the authors to give a more detailed mechanism about how latent concepts emerge. Also, see [1] as analysis from a manifold perspective. \n\n## Minor comments\n\n- Clustering time and memory consumption should be reported in Section 4.2\n- Word examples for each cluster should be provided in the appendix (Table 10)\n- Percentages should be provided in table 1\n\n## References\n\n[1] Emergence of Separable Manifolds in Deep Language Representations. Jonathan Mamou, Hang Le, Miguel A Del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, SueYeon Chung. ICML 2020", "summary_of_the_paper": "This paper studies the latent concept learned in BERT representations. The authors use a simple and effective clustering method to discover latent concepts in an unsupervised way. The contribution is primarily on the empirical side where the learned concepts are labeled and compared with pre-defined concepts. Then a layer-wise comparison is conducted where the results are similar to previous work but is done in an unsupervised way. The authors summarise this paper by organizing the discovered concepts into a BCD dataset. The writing is also clear. While there are many aspects deserving further pursuit, I think this is a solid paper that makes concrete contributions to the understanding of BERT. ", "summary_of_the_review": "I think this is a solid and interesting paper. My major problem is that this paper spends a lot of writing on development details that could be simplified to make place for more in-depth analysis. I am willing to further increase my score if the authors can present a more in-depth analysis. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RdoDHG1_kLT", "reviewer": "Reviewer_yqwt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper reads well, and the authors have clearly presented some interesting clusters within BERT. The authors use human intuition to hierarchically label a selection of \"concept clusters\" discovered in BERT. In this way, concept exploration in BERT is not limited to pre-defined concepts, and the authors promise to make their compiled dataset (BCD) available for additional research. Their results really do reveal some interesting clusters, and the paper's message could be bolstered if they included some tool to easily explore the concept space learned in BERT outside of the results shown in figures in the paper.\n\n\n**Weaknesses & Feedback:**\n\n1. In the main paper, the authors fail to describe the model that they use for analysis. All they say is \"We base our study on BERT\" (S1p1). But there are many flavors of BERT trained on different datasets, and the released BERT Concept Dataset (BCD) will only correspond to this model that the authors analyzed. There is not even a proper citation to BERT in the introduction of the paper (though one does appear later).\n2. In addition to not disclosing the model architecture used in the main paper, the authors do not disclose the tokenization scheme. The most common tokenization schemes for a case-sensitive BERT would likely not have the long hyphenations (e.g., \"75-year-old\"/\"90-year-old\") and all the German words (e.g., \"Mecklenburg-Western\") as single tokens in the dictionary. How would this technique combine tokens to represent a single concept?\n3. Without architecture information, I am left wondering about their statement that \"200k vectors each of size 768 require 400GB of CPU memory\". Assuming 32bit floating point, this many vectors should only occupy 0.6GB of space. Are they talking about the model memory? Why wouldn't you batch the representations and save them to hard disk to cluster later? I am also confused by this limitation.\n4. The annotation tool examples in the Appendix do not make it clear if annotators had access to the original context each token appeared in. For example, the word clouds in Fig 4(a,b) have little information in the cluster itself to inform an annotator that they one refers to percentage and the other to units of money. The annotation tool seems to show 3 or 4 different sentences that a *single token from the cluster* came from. How does that generalize to the other tokens in the cluster?\n5. (minor) There is a lot of speculation in the analysis as to how these findings could help, but I do not find the argument compelling. The authors could improve their narrative by elaborating on some of their claims. For example: \n    - Learning about the existence of a cluster where the model captures an era of time \"may help the model to learn relation between a particular era and the events occurring in that time period\" (S6.1p6).\n    - \"A clothing recommendation system may use such information to customize suggestions towards a culture or a religion\" (S6.1p6). I find this a questionable application that relies on biases learned in the language of a region/demographic.\n    - The authors show no summary of the results in  S7.2p9 \"Case Study\", and most of this section is spent persuading readers that such analysis could be helpful. It also fails to state clearly how their \"neuron analysis\" could be replicated.\n6. The annotation analysis performed in this paper is limited in that it only explores the representations in the final layer in BERT. \n7. I have a few complaints about the word cloud visualization. The size of each token reflects the frequency of that token (footnote 2), but this is strongly biased by the total frequency of that token in the sub-sampled dataset. Perhaps more meaningful would be if the size of the tokens reflected the proximity to the centroid of the cluster.", "summary_of_the_paper": "The paper performs a clustering analysis on the contextual representations of BERT on a subset of the News 2018 WMT corpus. These clusters are manually tagged by 3 annotators to connect these clusters to human concepts. The authors additionally release a dataset with these concepts labeled.", "summary_of_the_review": "In summary, I find the main paper lacking in sufficient disclosure on methods, and I do not believe the techniques or findings of the paper to be sufficiently novel, though I can see that the dataset would be useful. I still vote reject for ICLR 2022.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7yPoaN8cmvg", "reviewer": "Reviewer_We8g", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- The topic studying \"latent concepts\" is interesting and potentially impactful.\n\n- This paper involves a lot of empirical effort.\n- The dataset would be beneficial for future researches.\n\n**Weaknesses**\n\n(W1) There appears to be a potential confound in annotation. Sec 3 loosely defines \"concept\" to be \"a group of words that are meaningful\". The annotation procedure seems to let humans to decide whether the words are meaningful based on the words themselves (instead of the contexts). Ethayarajh (2019) showed that the context matter a lot for contextualized representations (e.g., BERT). This annotation procedure appears to drop some useful information. E.g., what if the words themselves are not \"meaningful\", but the way they are used in contexts appear meaningful? Do the annotators mark them as \"not meaningful\"? \n\n(W2) The generalizability of the finding is limited. The results reported in this paper appear specific to BERT. How would the clustering results compare to a static embedding (e.g., GloVe) and other contextualized representations (e.g., RoBERTa)? As far as I understand, the representation vectors cluster well because the meanings themselves follow hierarchical clusters, and that different DNN models can indeed embed their meanings fairly well -- BERT, especially, is one of the most powerful DNN models, so it is not surprising that it can do well here. Just showing \"BERT clusters them well\" does not add novel knowledge to what people already know. Comparing to other models might bring in more novel findings.\n\n(W3) The reporting of the results needs to be improved. Page 6 last paragraph: \"We compared the resulting clusters with the following pre-defined concepts: ... CCGBank, WordNet, LIWC\": Some comparison results are listed in Table 1. What is the percentage of matches? E.g., Lexical:NGram contains 20 matches, but how many concepts are there in total? In addition, Table 1 only shows the \"top-level\" match counts (& of layer 12). Among the more fine-grained concepts, how many concepts are matched? \n\n**Questions to the authors**\n\n(Q1) On the choice of concept database. In 4.3 Concept labels: there seem to be some existing databases with hierarchical concepts. E.g., WordNet. Why do you hand-engineer a concept database instead of using WordNet? Compared to WordNet, the hierarchical concepts defined in this paper appears very simplistic -- WordNet contains 155,327 words (let's say each \"concept label\" contains 100 words, then they have \"concept label\" to the magnitude of 1k), while the database of this paper contains 183 concept labels.\n\n(Q2) During the clustering, do the optimal number of clusters remain consistent for representations coming from different layers? (Am I right in assuming that Figure 2 only shows that of the final layer representation?)\n\n(Q3) On the limitation of the data size in clustering. How about first assigning each word to a \"large cluster\" (e.g., by the part-of-speech), which should be very accurate, and then run the cluster algorithm within each \"large cluster\"? This might help circumvent the computation bottleneck.", "summary_of_the_paper": "**Summary**  \n\nWhile a large number of previous papers evaluate DNN models on pre-defined concepts, this paper interprets the models (i.e., BERT) on latent concepts that are learned in an unsupervised manner. The analyses in this paper include: (1) Agglomerative hierarchical clustering from ~200k embedded tokens. (2) Comparing the clustered concepts to some hierarchical concept-tagsets. The authors find that lower layers learn many shallow lexical concepts, while higher layers learn more semantic relations. In addition, this paper releases a novel BERT concept dataset (BCD). With cross validation, the authors show that the BCD dataset has high consistency.\n\n**Contributions**\n\n1. Interpret BERT by analyzing latent concepts learned in the network\n2. Collect and annotate a hierarchical concept dataset (BCD)", "summary_of_the_review": "Interesting topic and substantial empirical effort. Methodology (clustering, dataset annotation, etc.) has potential problems. Reporting needs improvements. The generalizability and novelty of the overall finding are limited. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the latent concepts learned in BERT. In contrast to previous work which tries to map embeddings to predefined
linguistic concepts this paper sets out to discover what is inherently learned by BERT. This is however easier said that done, since 
there is no easy way to inspect the embeddings and draw conclusions on what is being learned. The authors adopt a methodology which could be used to inspect the inner workings of other pretrained models. They employ hierarchical clustering to discover latent concepts and then inspect these clusters by manually labeling them. The reviewers raised various issues regarding the number of clusters, and the amount of effort required which de facto renders the approach not very portable. The authors addressed the comments and flagged several difficulties with undertaking such an analysis. I will vote for the paper to be presented as an oral for two reasons a) it is difficult to analyze pretrained models, and although I am not convinced what the authors propose is feasible, it will at least get the discussion going, b) the manually annotated dataset is useful and will go towards allowing us to perform comparisons between models c) the annotation tool will be useful to others if the authors are considering releasing it.
Decision: Accept (Poster)